'What is the right memory allocations that can be given to multiple spark streaming jobs if it is being processed in a single EMR cluster (m5.xlarge)?
I have 12 spark streaming jobs and it receives a small size data at any time. These scripts has spark transformations and joins.
What is the right memory allocations can be given to these spark streaming jobs if it is being processed in a single EMR cluster (m5.xlarge) (not using EMR steps) ? The memory allocations includes num-executors, executor-memory etc.
Please explain the working of these spark jobs in the cluster. How will the cluster split resource to these jobs? Please help me with the basics.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|