Spark UI vs event log analysis: which one answers your question

The Spark UI and the event log contain the same data. The UI renders it for a human, once. The log keeps it as structured JSON forever. That difference decides which one you should reach for.

The UI is good at

• ·Watching a running job and seeing which stage is active right now.
• ·Eyeballing the task timeline of a single stage.
• ·Clicking into an individual failed task and reading its stack trace.

The UI is bad at

• ·Comparing today's run against last Tuesday's run.
• ·Answering the same question across 200 jobs.
• ·Running unattended in CI.
• ·Surviving a cluster that has already been torn down, unless the history server is set up.
• ·Telling you which of the visible numbers is the actual bottleneck.

What log analysis adds

Reading the log programmatically lets you rank stages by cost, compute skew ratios, total spill across the run, quantify retry waste, and diff two runs. It also lets you keep the answer: a report file lives in your repository or artifact store long after the cluster is gone.

Use both

Analyze the log first to find out which stage matters and why, then open the UI on that one stage if you need to look at individual tasks. That order saves the afternoon you would otherwise spend scrolling.

sparkdoctor analyze --input ./event-logs/app-20260601 --output ./report