8 min read
Why is my Spark job slow? A checklist that actually finds the cause
The seven causes behind almost every slow Apache Spark job, how to confirm each one from the event log, and what to change once you know.
read →blog
Notes on Spark performance.
Write-ups on event log analysis, skew, shuffle waste, and what we are building into SparkDoctor.
4 min read
Why we built SparkDoctor
Spark tells you what happened. It does not tell you why the job was slow. SparkDoctor reads the event log and answers that question locally.
read →6 min read
Spark data skew: how to detect it and how to fix it
Skew is the most common reason a Spark stage stalls at 99 percent. Here is how to measure it from task metrics and the four fixes that work.
read →6 min read
Spark shuffle and spill: what the numbers mean and what to change
Shuffle read and write bytes, memory spill, and disk spill explained, plus the tuning that reduces each one without throwing more executors at the job.
read →5 min read
Catching Spark performance regressions in CI
A slow job is usually a job that got slower. Compare runs against a baseline, fail the build on regressions, and stop paying for compute waste for weeks before anyone notices.
read →5 min read
How to read a Spark event log without losing your afternoon
Where event logs live, what the Spark 4 directory layout looks like, and which fields actually matter when you are hunting a slow stage.
read →5 min read
Spark UI vs event log analysis: which one answers your question
The Spark UI is great for looking. It is bad for comparing, automating, and remembering. Here is when to open the UI and when to analyze the log instead.
read →