'Azure Functions - Most efficient way of saving data from API to database
At first I would like to say I'm a beginner in Azure Functions (and web development in overall xd). I'm stuck on choosing the best approach for:
- getting data from an API,
- saving that data to a database,
- combining it all in a simple and clean way.
I'm making an integration with an existing API. Let's say I need to save 10 000 offers into a database, considering that loading all of the data at once is a totally bad idea. If it's possible, I would like to synchronise the database by a schedule, so I think Timer Trigger has to be used here.
Now time for some code:
Models
public class ListOfProductsRequestRoot
{
public ProductV2GetProductListRequestFilter Filter { get; set; }
public string Last_id { get; set; }
public long Limit { get; set; }
}
public class ProductV2GetProductListRequestFilter
{
public string Visibility { get; set; } = Enum.GetName(typeof (ProductV2GetProductListRequestFilterFilterVisibility), 0);
public class ListOfProductsResponseRoot
{
public ListOfProductsResponseResult Result { get; set; }
}
public class ListOfProductsResponseResult
{
public ListOfProductsResponseItem[] Items { get; set; }
public string Last_ID { get; set; }
public long Total { get; set; }
}
public class ListOfProductsResponseItem
{
public string Offer_ID { get; set; }
public long Product_ID { get; set; }
}
public class ProductDetailsOffer
{
public string Offer_ID { get; set; }
public long Product_ID { get; set; }
public string Name { get; set; }
public string Barcode { get; set; }
}
public class ProductDetailsResponseOffer
{
public string Name { get; set; }
public string Barcode { get; set; }
}
Functions
// Where should I hold the baseURL?
private string baseURL = "####";
// Is making HttpClient static a good idea?
public static HttpClient httpClient = new HttpClient();
[FunctionName("LoadOffers")]
public async Task<List<ProductDetailsOffer>> Load(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
[TimerTrigger("0 */5 * * * *")] TimerInfo myTimer,
ILogger log)
{
var offersList = new List<ProductDetailsOffer>();
var responseBody = String.Empty;
httpClient.BaseAddress = new Uri(baseURL);
httpClient.DefaultRequestHeaders.Accept.Clear();
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
httpClient.DefaultRequestHeaders.Add("Host", "api-seller.ozon.ru");
httpClient.DefaultRequestHeaders.Add("Client-Id", "######");
httpClient.DefaultRequestHeaders.Add("Api-Key", "######-######-######-######-######");
var requestBody = JsonConvert.SerializeObject(new ListOfProductsRequestRoot()
{
Filter = new ProductV2GetProductListRequestFilter(),
Last_id = String.Empty,
Limit = 1000 // How to page using the Limit property?
});
var response = await httpClient.PostAsync("/v2/product/list", new StringContent(requestBody));
var responseJson = response.Content.ReadAsStringAsync().ToString();
dynamic responseDeserialized = JsonConvert.DeserializeObject<ListOfProductsResponseRoot>(responseJson);
// Here I get out of ListOfProductsResponseRoot model the
// collection containing offers Offer_ID and Product_ID
var responseItems = responseDeserialized.Result.Items;
for (int i = 0; i < responseItems.Length; i++)
{
// Serialize Item
requestBody = JsonConvert.SerializeObject(responseItems[i]);
// Here I get an individual offer with Offer_ID and Product_ID from responseItem
response = await httpClient.PostAsync("/v2/product/info", new StringContent(requestBody));
responseJson = response.Content.ReadAsStringAsync().ToString();
responseDeserialized = JsonConvert.DeserializeObject<ProductDetailsResponseOffer>(responseJson);
// Creating target data that I need in this model
// Using Offer_ID and Product_ID from responseItem
// Getting Name and Barcode from ProductDetailsResponseOffer
offersList.Add(new ProductDetailsOffer()
{
Offer_ID = responseItems[i].Offer_ID,
Product_ID = responseItems[i].Product_ID,
Name = responseDeserialized.Name,
Barcode = responseDeserialized.Barcode
});
}
//For debugging
if (response.IsSuccessStatusCode)
{
Console.WriteLine(responseBody);
}
else
{
Console.WriteLine("Internal server Error" + response.StatusCode);
}
return offersList;
}
//Table-Value Parameters are used since this is an Azure Function App,
//however if the target DB would be SQL Server,
// then Bulk Copy would be better, otherwise Entity Framework AddRange() instead of Add()
[FunctionName("SaveToDB")]
public async Task<string> ToDB(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
ILogger log,
ICollection<ProductDetailsOffer> offers)
{
var cmdText = @"
insert into dbo.Offers (Offer_ID, Product_ID, Name, Barcode)
select Offer_ID, Product_ID, Name, Barcode
from @offers";
using (var connection = new SqlConnection(connectionString))
{
var command = new SqlCommand(cmdText, connection);
var param = command.Parameters.AddWithValue("@offers", ToDataTable(offers));
param.TypeName = "dbo.OffersTableType";
connection.Open();
command.ExecuteNonQuery();
}
}
And my questions (or the ones I can come up with for now):
In Azure Functions, where should I hold stuff like the baseURL, Client-ID and Api-Key? (the last two are required by the API for authentication)
From what I've read in the internet, making httpClient a static variable is a good practice in terms of performance, correct me if I'm wrong.
The API method "/v2/product/list" allows a maximum amount of 1000 values to be requested at once, so I understand that through this method I can get a max of 1000 offers per request. Since I would like to load 10 000 offers, I'm thinking about something called "Paging", though I don't know how to use it. What specifically is uncertain for me is how to I get the Last_id from the previous request, so that the new request can start from that last id value.
After the method "/v2/product/list" gives me those 1000 offers (containing only Offer_ID and Product_ID), I can access an individual offers details by the one of the ID's using "/v2/product/info". So now what is the proper way to request each one of those offers to fill data for the target model ProductDetailsOffer?
Does it make sense to create another azure function just for saving the data to the database? Maybe I should just make it a private method or an extension for the main azure function?
Hope my questions are somewhat clear and understandable.
I will be grateful for any help!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
