'Spark Write SIZE / Read Write
In Spark 2.0 and above, how can we calculate/derive the size of a Shuffle Write or Shuffle Read? I know Spark UI has all that info. But how can we calcualte or derive the sizes in your script (Pyspark or Scala)? Is there a way?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|