'How to build custom cache logic in Spark Streaming

I have spark streaming application running with 4 executor, and I am looking for cache logic which is local to each executor. This cache will be used as a lookup and it will refresh every 20 min. Currently I have tried with python decorator cachetools. But looking for other solutions.

cachingpysparkstreaming


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions