'Maintain a large expirable cache of .NET object in external storage (with restrictive refresh)
Here's my requirement.
- I have a pool of 20 machines across which I need to share a large .NET object. The object is very large in size (~10GB).
- I want to cache this object for ~1 hour, and share among the machines. Data would change after ~1 hour (I have ability to synchronize data within 2 hour window).
- I could use .NET serializers like protobuff-net to achieve serialization to file and upload to Azure blob or other external storage that could be shared among machines.
- I have very restricted capability to recreate the .NET object (max 3-4 times every hour), because the data source is not scalable. So, I need to ensure one refresh across all machines in 1 hour.
- Also, the recreate of .NET objet is costly and takes 10 minutes or more. Hence, the need for cache of final .NET cache object.
- I don't think we deal with large number of key-value pairs, in our case two keys, and two large .NET objects associated with those keys.
Here's my thought... if I were to design this, assuming cache expiry interval of 1 hour...
- I would need a Mutex across all machines to ensure refresh is done exactly once for the lifetime of 1 hour.
- Allow read from cache (download from external storage, and deserialize to .NET object), until expiry of 1 hour.
- But, trigger one refresh of cache, in background by any one machine, as soon as 1/2 hr is reached. This will optimize and not block any machines to fetch the latest data from cache.
- I may need to shard .net objects to maximize the performance of upload/download of serialized intermediate stream of .net object.
Question: Are there any .NET libraries or established models/patterns that could achieve these out-of-box?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
