How to read a Spark event log without losing your afternoon
Event logs are the most underused artifact in a Spark deployment. They are written by default in most managed platforms, they survive the cluster, and they contain a complete record of the run.
Finding the log
The location comes from spark.eventLog.dir. On Spark 4 with rolling logs enabled, each application gets its own eventlog_v2_* directory containing an appstatus file and one or more numbered event files. Point SparkDoctor at either the application directory or the parent directory.
Fields that matter
- ·Task end events: duration, shuffle read and write bytes, memory and disk spill.
- ·Stage completion events: attempt number, task counts, and failure reasons.
- ·SQL execution events: the physical plan, which is where exchange reuse problems hide.
Comparing the maximum task duration against the average within a single stage is usually the fastest way to spot skew. If one task runs ten times longer than its peers, the stage is bound by that one partition and no amount of extra executors will help.
Automate it
sparkdoctor analyze --input ./eventlog_v2_app-20260711 --output ./reportRun it in CI on every job and the answer is waiting for you before anyone opens the Spark UI.