'In structured streaming checkpointing why offsets are not committed after foreachbatch
df
.writeStream
.trigger(Trigger.Once)
.option(checkpointKey, checkpointVal)
.foreachBatch { (batchDF: DataFrame, batchId: Long) => }
This is the sample code I am running. Observing that Structured streaming creates the offsets file at the beginning itself: checkpoints/offsets/3
Why does it not wait for the foreachBatch to complete and then write the offsets to checkpoint directory?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
