'With reliable receiver, can spark streaming loss data when worker crashed?
When I read the Fault-tolerance Semantics in spark streaming document. Combined with reliable receivers, when the worker node fails, spark streaming will not lose data.
But consider following scenario:
- Two executors get the calculated data and save them in their memory
- Receivers know that the data has been entered
- Receivers send acknowledgment to source and at the same time replica worker and current worker crashed
- But source received acknowledgment
When the receiver restarts, these data were lost! Is right?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|