Why we built SparkDoctor
Every data team has the same ritual. A job that used to take 20 minutes takes 90. Someone opens the Spark UI, scrolls through stages, squints at task timelines, and guesses. A few hours later the guess is either right or expensive.
The information needed to answer the question is already in the event log. It is just buried under millions of JSON lines that nobody wants to read by hand. SparkDoctor reads those logs and turns them into a report you can act on.
Local first, by design
Event logs contain table names, paths, user names, and cluster details. Uploading them to a hosted service is a non-starter for most teams. SparkDoctor runs on your machine or in your CI runner, reads the log from disk, and writes the report next to it. Nothing leaves your environment.
What it looks for
- ·Task duration skew, where the slowest task dwarfs the average in a stage.
- ·Shuffle partition skew and oversized shuffle partitions.
- ·Memory and disk spill pressure.
- ·Low shuffle parallelism and tiny tasks.
- ·Retry waste, speculative execution, and failed stages.
- ·SQL plan issues such as repeated subtrees and possible missed exchange reuse.
Try it
sparkdoctor analyze --input ./event-logs/app-20260601 --output ./reportYou get analysis.json for tooling, recommendations.md for humans, and SQL plan graphs you can render with Graphviz.