guide

Tuning spark.sql.shuffle.partitions

This setting controls how many partitions Spark SQL creates after a shuffle: joins, group by, distinct, and window functions. The default is 200, which is a number chosen in 2015 and almost never the right one for your data.

What the value actually does

Every shuffle redistributes rows into this many buckets. Each bucket becomes one task in the next stage. So the value sets both the parallelism of that stage and the amount of data a single task must hold in memory. Too few partitions and each task is large, spills to disk, and cannot spread across the cluster. Too many and you pay task scheduling overhead thousands of times for tasks that finish in milliseconds.

A value you can defend

Aim for roughly 100 to 200 MB of shuffle data per task, then round to a multiple of total executor cores so the last wave of tasks does not leave the cluster idle.

# shuffle write bytes for the stage: 240 GB
# target per task: ~150 MB
240000 / 150 = 1600 partitions

# total cores: 200 -> 1600 is 8 clean waves
spark.conf.set("spark.sql.shuffle.partitions", 1600)

The input to that calculation is shuffle write bytes per stage, which is recorded in the event log for every run.

Symptoms of a wrong value

• ·Disk and memory spill across most tasks in a stage: partitions are too large.
• ·Task durations in the tens of milliseconds with thousands of tasks: partitions are too small.
• ·Only a fraction of executor cores busy during a stage: partition count is below total cores.
• ·One task far longer than the rest: this is skew, not partition count, and raising the value will not fix it.

Adaptive query execution changes the job

With adaptive query execution enabled, Spark coalesces shuffle partitions after seeing real statistics, so the configured value becomes an upper bound rather than a fixed count. The practical approach is to set the value generously high and let coalescing reduce it, instead of hand tuning per query.

spark.conf.set("spark.sql.adaptive.enabled", true)
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", true)
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128MB")

Adaptive execution does not rescue a plan that shuffles far more data than it needs, and it does not fix a skewed key on its own unless skew join handling is also enabled.

Measure it instead of guessing

SparkDoctor reads the event log your job already wrote and reports low shuffle parallelism, oversized shuffle partitions, and spill pressure per stage, with the byte counts behind each finding. It runs locally, so nothing is uploaded.

$ sparkdoctor analyze --input ./event-logs/app-20260601 --output ./report