'A way to handle write operations (600k records) to cosmos db using Entity Framework
I have a .Net Core 3.1 application. I'm trying to create an implementation that will handle ~600k writes to Cosmos database and the same amount of deletion operations. It takes a lot of time to execute the SaveChangesAsync method. Cosmos database has auto-scaling enabled with max RU/s set to 1k.
Setting AutoDetectChangesEnabled to false didn't help. Should I divide the number of operations and for each group trigger SaveChanges?
Can someone help me out finding the best way to handle high volume write operation to cosmos db using entity framework (3.1.23 version)?
Here is the code example (I renamed some values/methods):
_context.ChangeTracker.AutoDetectChangesEnabled = false;
var pohs = _context.A
.Where(x => x.Num == num)
.ToList();
var pols = _context.B.ToList();
var removed = from x in pols
join y in pohs
on x.Id equals y.id
select x;
_context.B.RemoveRange(removed.ToList());
_context.A.RemoveRange(pohs);
var anotherheaders = CreateXXX(temp.GetXXX(num));
_context.A.AddRange(anotherheaders);
_context.B.AddRange(tempLines.GetXXX(purchaseOrderHeaders));
_context.ChangeTracker.DetectChanges();
await _context.SaveChangesAsync();
UPDATE: Elements in the AddRange and RemoveRange have different partitions. I saw that requests made to Cosmos db reached the max ru/s limit. However, after I increased it to 10k it didn't really speed up the process.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
