'How to clear spark context threads without killing spark context
I have an ever-running spark context but some of the threads are in a WAITING state like(dispatcher event loop) these threads contain some information about RDD query plans at some point which leads to driver heap memory error. I want to clear those threads or what should do in that case but I don't want to kill spark context anymore. please suggest any solution.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|