Spark shuffle and spill: what the numbers mean and what to change

Shuffle is where most Apache Spark performance goes to die. It serializes data, writes it to local disk, moves it across the network, and reads it back. Every wide transformation pays that cost: joins, group by, distinct, repartition, and window functions.

Reading the shuffle metrics

• ·Shuffle write bytes: how much a stage pushed out to be redistributed.
• ·Shuffle read bytes: how much the next stage pulled back in.
• ·Memory spill: bytes the task could not keep in memory, measured before serialization.
• ·Disk spill: what those bytes cost once written out, which is the number that hurts.

Any spill at all means tasks are working with more data than their memory allows. Persistent spill across a whole stage means the partition count and the executor memory are mismatched for this data volume.

Reduce the shuffle before you tune it

• ·Push filters and column pruning above the shuffle so less data moves.
• ·Broadcast dimension tables that fit in memory to remove the shuffle from the join entirely.
• ·Pre-aggregate before a join when the join only needs a summary.
• ·Remove manual repartition calls that sit right before an operation that shuffles anyway.

Then tune what remains

Target shuffle partitions that produce roughly 100 to 200 MB per task. Far above that and you spill. Far below and scheduling overhead dominates. Adaptive query execution coalesces partitions after the fact, which handles most of this, but it cannot rescue a plan that shuffles ten times more data than it needs.

Measure every run

SparkDoctor pulls shuffle volume, spill pressure, and partition sizing out of the event log automatically, so you can watch these numbers move between runs instead of reconstructing them by hand in the Spark UI.

sparkdoctor analyze --input ./event-logs/app-20260601 --output ./report