Why is my Spark job slow? A checklist that actually finds the cause
A Spark job that used to finish in 20 minutes now takes two hours, and nothing obvious changed. Before you add executors, work through the list below. In practice almost every slow Apache Spark job comes down to one of seven causes, and all seven are visible in the event log the run already wrote.
1. Data skew
One partition holds far more rows than the rest, so one task runs long after every other task in the stage has finished. Confirm it by comparing the maximum task duration against the average duration inside a single stage. A ratio above roughly 5x means the stage is bound by one partition, and extra executors will do nothing.
Fixes: salt the join key, split the hot key out and union it back, or enable adaptive query execution so Spark can split skewed partitions at runtime.
2. Shuffle volume
Shuffles write to disk and move data across the network, so they dominate wall clock time on wide transformations. Look at shuffle read and write bytes per stage. If a stage moves tens of gigabytes to produce a small result, the query plan is shuffling before it filters.
Fixes: filter and project earlier, broadcast the small side of a join when it fits, and avoid repartitioning right before an operation that shuffles anyway.
3. Spill to disk
When a task cannot hold its working set in memory, Spark spills to disk and the stage slows by an order of magnitude. Memory spill and disk spill are recorded on every task end event, so spill pressure is easy to prove rather than guess.
4. Partition sizing
Too few partitions and each task is huge and cannot parallelize. Too many and you pay scheduling overhead on thousands of tasks that each run for 40 milliseconds. Both show up as poor cluster utilization with no single obvious hotspot.
5. Small files
Thousands of tiny input files turn listing and task setup into the bottleneck before any real work starts. If your read stage has far more tasks than the data size justifies, compaction upstream will beat any tuning downstream.
6. Retries and failed stages
A job can succeed while quietly burning half its runtime on retried tasks, speculative duplicates, and re-executed stages after a lost executor. Retry waste rarely shows in a dashboard, but it is explicit in the event log.
7. Query plan problems
Repeated subtrees in the physical plan mean the same work runs more than once. Missed exchange reuse is the classic version: two branches compute an identical shuffle and Spark does not share it. Caching the shared branch, or restructuring the query, removes the duplicate work outright.
Get the answer without the guesswork
SparkDoctor reads the event log your job already produced and reports which of these is actually happening, with the numbers behind it. It runs locally, so the log never leaves your environment.
sparkdoctor analyze --input ./event-logs/app-20260601 --output ./report