docs

Install, analyze, read the report.

SparkDoctor is a local CLI. There is no daemon, no upload, no account. Point it at a Spark event log and read the generated markdown.

Install from a release

$ curl -L -o sparkdoctor-0.1.5.zip \
    https://github.com/khodosko/sparkDoctor/releases/download/v0.1.5/sparkdoctor-0.1.5.zip
$ unzip sparkdoctor-0.1.5.zip
$ ./sparkdoctor-0.1.5/bin/sparkdoctor --help

# optional: add to PATH
$ export PATH="$PWD/sparkdoctor-0.1.5/bin:$PATH"

Requires Java 17.

Build from source

$ git clone https://github.com/khodosko/sparkDoctor.git
$ cd sparkDoctor
$ ./gradlew test
$ ./gradlew installDist
$ ./build/install/sparkdoctor/bin/sparkdoctor --help

Analyze a log

$ sparkdoctor analyze path/to/eventlog --out ./sparkdoctor-report

Accepts plain, gzip, Zstandard, LZ4, and Snappy event logs, plus local directories and Spark 4 eventlog_v2_* directories.

What the report looks like

SparkDoctor analyzed ./fixtures/spill-heavy-eventlog.json
Application: spill_heavy_customer_etl
Application ID: app-spill-heavy-0001
Duration: 10000 ms · Jobs: 1 · Stages: 1 · Tasks: 2
Issues detected: 1 · Severity summary: medium=1

Top bottlenecks:
  - [medium] spill_pressure (stage 9): Stage 9 has spill pressure.

Output directory:     ./sparkdoctor-report
Analysis JSON:        ./sparkdoctor-report/analysis.json
Recommendations:      ./sparkdoctor-report/recommendations.md

recommendations.md (excerpt)

### Reduce spill pressure

- Severity: medium
- Stage ID: 9
- Related bottleneck: spill_pressure

Evidence:
- completedTasks: 2
- diskBytesSpilled: 314572800 (300 MiB)
- mediumDiskSpillThresholdBytes: 268435456 (256 MiB)

Stage 9 spilled 300 MiB to disk and 128 MiB to memory across 2
completed tasks.