Spark data skew: how to detect it and how to fix it

You have seen the pattern: a stage races through 1999 of 2000 tasks and then sits on the last one for 40 minutes. That is data skew. One partition received far more data than the others, and Spark cannot finish the stage until that partition is done.

How to measure skew

Take one stage and compare the maximum task duration against the average across all tasks in that stage. Do the same for shuffle read bytes per task. Duration skew tells you the stage is stalled. Byte skew tells you why.

• ·Max duration close to average: the stage is balanced, look elsewhere.
• ·Max duration 5x to 10x the average: meaningful skew, worth fixing.
• ·Max duration above 10x the average: the stage is effectively single threaded.

Where skew comes from

Almost always a join or aggregation key with an uneven distribution. Null keys, a default tenant id, a sentinel value like 0 or unknown, or one customer that is 100 times bigger than the rest. Date keys skew too when a backfill lands everything on one partition.

Four fixes, in order of effort

• ·Enable adaptive query execution so Spark splits skewed shuffle partitions automatically at runtime.
• ·Broadcast the small side of the join so no shuffle happens on the key at all.
• ·Filter out null and sentinel keys before the join, then handle them separately.
• ·Salt the hot key by appending a random suffix, join on the salted key, then aggregate the results back.

Confirm the fix

Rerun and compare the max to average ratio for the same stage. If the ratio dropped and wall clock time did not, skew was not your bottleneck and you saved yourself from optimizing the wrong thing. SparkDoctor reports the ratio per stage on every run so the comparison takes seconds.

sparkdoctor analyze --input ./event-logs/app-20260601 --output ./report