Best way to cache about 4 million records with additions/deletions of about 5 thousand per day

Question

We have a streaming platform with the following sections:

• Videos
• Categories
• Actors
• Channels
• Tags

All content is translated into 13 languages ​​in separate tables: videos_translations categories_translations actors_translations channels_translations tags_translations

Every day, we have about 5000 videos added and about 2000 deleted.

A few years ago, we deleted and generated the cache every day via crons and did the same for the sitemaps.

With the database increasing day by day, this task has become impossible. Generating the sitemaps takes almost 12 hours and deleting + generating the cache takes several days.

We end up with the overlapping of the spots because they didn't have time to finish.

What's the best way to handle this?

Our biggest concern is the videos and categories sections which have several pages of paginations: https://example.com/videos

https://example.com/videos/category-slug

We thought of two solutions:

• Generate the cache 1 time per week or 2 weeks
• Put a short duration for the cache and not generate it automatically

Thanks for your help and suggestions.

Answer 1

The key word here is compatible.

You have installed one version from Nuget, yet you try to use another version.

The solution is to upgrade your Nuget package, so it matches the fixed version, you're trying to use.

To do that, in Nuget package manager, make sure you check 'Include prerelease' at the top of the window, now you should be able to select all versions, including prereleases, so the version in project and the version you run are the same.