'Maintain a large expirable cache of .NET object in external storage (with restrictive refresh)

Here's my requirement.

  1. I have a pool of 20 machines across which I need to share a large .NET object. The object is very large in size (~10GB).
  2. I want to cache this object for ~1 hour, and share among the machines. Data would change after ~1 hour (I have ability to synchronize data within 2 hour window).
  3. I could use .NET serializers like protobuff-net to achieve serialization to file and upload to Azure blob or other external storage that could be shared among machines.
  4. I have very restricted capability to recreate the .NET object (max 3-4 times every hour), because the data source is not scalable. So, I need to ensure one refresh across all machines in 1 hour.
  5. Also, the recreate of .NET objet is costly and takes 10 minutes or more. Hence, the need for cache of final .NET cache object.
  6. I don't think we deal with large number of key-value pairs, in our case two keys, and two large .NET objects associated with those keys.

Here's my thought... if I were to design this, assuming cache expiry interval of 1 hour...

  1. I would need a Mutex across all machines to ensure refresh is done exactly once for the lifetime of 1 hour.
  2. Allow read from cache (download from external storage, and deserialize to .NET object), until expiry of 1 hour.
  3. But, trigger one refresh of cache, in background by any one machine, as soon as 1/2 hr is reached. This will optimize and not block any machines to fetch the latest data from cache.
  4. I may need to shard .net objects to maximize the performance of upload/download of serialized intermediate stream of .net object.

Question: Are there any .NET libraries or established models/patterns that could achieve these out-of-box?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source