'Sub directories under checkpoint directory for spark structured streaming
The checkpoint directory for spark structured streaming create four sub directories. What are each of them for?
/warehouse/test_topic/checkpointdir1/commits
/warehouse/test_topic/checkpointdir1/metadata
/warehouse/test_topic/checkpointdir1/offsets
/warehouse/test_topic/checkpointdir1/sources
Solution 1:[1]
Source folder contains the Initial kafka offset values of each partition. like if your kafka has 3 partitions 1,2,3 and starting values for each partition is 0 then it will contain value like {1:0,2:0,3:0}
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Sonu |