open source · v0.1.5 · local-first

Stop guessing why Spark jobs are slow.

SparkDoctor reads raw Apache Spark event logs and surfaces skew, spills, wasted shuffle work, failed stages, and SQL plan issues, without a Spark History Server or hosted backend. Everything runs locally on your machine.

$ sparkdoctor analyze path/to/eventlogGitHub →

$ sparkdoctor analyze ./spill-heavy-eventlog.json --out ./report

SparkDoctor analyzed ./spill-heavy-eventlog.json
Application: spill_heavy_customer_etl
Duration: 10000 ms · Stages: 1 · Tasks: 2
Issues detected: 1 · Severity: medium=1

Top bottlenecks:
  [medium] spill_pressure (stage 9)
  → stage 9 spilled 300 MiB to disk across 2 tasks

Analysis JSON:        ./report/analysis.json
Recommendations:      ./report/recommendations.md

What it detects today

·Task duration skew
·Shuffle partition skew
·Oversized shuffle partitions
·Low shuffle parallelism
·Spill pressure
·Memory & disk spill skew
·Too many tiny tasks
·Retry waste
·Heavy speculative execution
·Executor & host imbalance
·SQL plans with many exchanges
·Repeated SQL subtrees
·Possible missed exchange reuse
·Failed jobs and stages

See full feature list →

SparkDoctor Pro

Catch Spark performance regressions before they reach production.

$24,000 / yearBilled annually

SparkDoctor Pro turns single-run Spark diagnostics into a repeatable team workflow. Compare baseline and current runs, detect regressions in runtime, shuffle, spill, failures, and bottlenecks, and fail CI when Spark jobs cross configured thresholds.

Pro keeps the same local-first model: run it inside your environment, keep Spark event logs under your control, and avoid sending sensitive Spark metadata to a SaaS platform.

·Compare baseline vs current Spark runs
·Detect runtime, shuffle, spill, failure, and bottleneck regressions
·Fail CI when configured regression thresholds are crossed
·Track performance trends across many Spark job runs
·Store run history in a local or customer-controlled database
·Batch analyze directories of Spark event logs
·Configure team thresholds and regression policies
·Generate team-ready reports for pull requests, incidents, and reviews
·Support local/self-hosted deployment workflows

Request Pro access Compare Open Source, Pro, and Enterprise →

Install

# From a GitHub release
$ curl -L -o sparkdoctor-0.1.5.zip \
    https://github.com/khodosko/sparkDoctor/releases/download/v0.1.5/sparkdoctor-0.1.5.zip
$ unzip sparkdoctor-0.1.5.zip
$ ./sparkdoctor-0.1.5/bin/sparkdoctor --help

Requires Java 17. Reads local files and directories: plain, gzip, Zstandard, LZ4, and Snappy event logs. Full docs →