features
What SparkDoctor finds in an event log.
Every detection below is shipped in the open-source CLI today. For team workflows · comparing runs, catching regressions, CI gates · see Pro.
Detections
- Task duration skew
- Per-stage tasks whose maximum duration runs far above the average duration.
- Shuffle partition skew
- Shuffle reads concentrated in a few partitions.
- Oversized shuffle partitions
- Individual shuffle partitions large enough to contribute to spill or OOM.
- Low shuffle parallelism
- Stages with too few shuffle partitions for the data volume.
- Spill pressure
- Stages spilling significant bytes to disk or memory.
- Memory & disk spill skew
- Spill concentrated in a small number of tasks rather than spread evenly.
- Too many tiny tasks
- Stages dominated by sub-second tasks where scheduling overhead dominates.
- Retry waste
- Time spent in failed task attempts that were later retried successfully.
- Heavy speculative execution
- Stages where speculation kicked in repeatedly, suggesting upstream skew.
- Executor & host imbalance
- Work concentrated on a subset of executors or hosts.
- SQL plans with many exchanges
- SQL executions whose physical plan contains an unusual number of exchanges.
- Repeated SQL subtrees
- Duplicated physical-plan fragments that may indicate missed reuse or caching opportunities.
- Possible missed exchange reuse
- Repeated exchange-rooted subtrees: a low-confidence physical-plan signal worth reviewing.
- Failed jobs and stages
- Failed jobs/stages with parsed error context.
Outputs
- analysis.json
- Machine-readable analysis output for tooling and CI.
- recommendations.md
- Human-readable stage hotspots, bottleneck evidence, and recommendations.
- sql-executions.md
- Readable SQL execution plan output when SQL events are present.
- sql-execution-*.dot
- Graphviz SQL plan graph files when Spark exposes structured plan data.
- Terminal summary
- Severity counts, top bottlenecks, and output paths printed to stdout.
Supported inputs
- ·Plain Spark event log files
- ·Gzip-compressed event logs (.gz)
- ·Zstandard-compressed event logs (.zstd, .zst)
- ·LZ4-compressed event logs (.lz4)
- ·Snappy-compressed event logs (.snappy)
- ·A local directory holding one Spark application's event log or its rolling parts
- ·Spark 4 event-log directories (eventlog_v2_*)
For help finding event logs in local Spark, Spark History Server, Databricks, Amazon EMR, or AWS Glue, see the event-logs guide.