Catching Spark performance regressions in CI

Most Spark performance work is reactive. Someone notices the pipeline is late, and by then the regression has been shipping for a month. The cost is not one slow run, it is 30 slow runs nobody measured.

Treat runtime as a test

Every Spark run writes an event log. That log is a complete performance record: task durations, shuffle volume, spill, retries, and the physical plan. Comparing today's log against a known good baseline turns performance into something CI can check like any other assertion.

What to compare

• ·Total wall clock time and total executor CPU time.
• ·Shuffle bytes read and written per stage.
• ·Spill volume, which usually moves first when data grows.
• ·Task duration skew ratio per stage.
• ·Physical plan shape, so a lost broadcast or a new shuffle is visible immediately.

Fail the build, not the budget

Once you have a baseline, a threshold does the rest. If shuffle volume doubles or a broadcast join silently became a sort merge join, the pull request that caused it is the one that gets flagged, while the change is still fresh in someone's head.

Run to run comparison, historical run storage, and CI failure gates are what SparkDoctor Pro adds on top of the open source analyzer. The open source version gives you the per run analysis for free.

sparkdoctor analyze --input ./event-logs/app-20260601 --output ./report