'When does spark clear the scratch space
We've been seeing a couple of "No space left on device" exceptions in spark. We don't use persist() anywhere so most likely it is the spills that is being saved there
In the lifecycle of a job (spark action), when does spark decide to delete the items from the scratch space associated with the job?
Is it during the job end or application end?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|