Which model, which job
Opus 4.8 · Opus 5 · Fable 5 · Sonnet 5

Measure how different models actually behave in my own Claude Code logs, then build
a metrics.json I can drop into https://3dl.dev/coding-vs-orchestration.html to
redraw its charts with my numbers.

Write the extraction script yourself. Python 3, standard library only, no installs.
Run it, sanity-check the output, and tell me what it says.

── WHERE THE DATA IS ──────────────────────────────────────────────────────────
  ~/.claude/projects/<slug>/<session-id>.jsonl              one JSON object per line
  ~/.claude/projects/<slug>/<session-id>/subagents/**/*.jsonl   subagent transcripts
Each record has: type ("user" | "assistant" | ...), timestamp, sessionId,
isSidechain, message.model, message.usage.output_tokens, and message.content as a
list of blocks typed "text" | "thinking" | "tool_use" (with .name, .input) |
"tool_result" (with .tool_use_id, .is_error, .content).
Transcripts rotate after ~30 days. If I use more than one machine, run this on each
and merge — my laptop held a quarter of what my build box did, and the small slice
gave the opposite answer on two metrics.

── WHAT COUNTS AS A TURN ──────────────────────────────────────────────────────
One human message plus every assistant message before the next human message.
A "user" record is NOT human if isSidechain or isMeta is set, if its content list
contains a tool_result, or if its text begins with "<system-reminder>",
"<local-command" or "Caveat:". Text beginning "[Request interrupted" ends the turn
and marks it interrupted. Attribute a turn to the first assistant model in it.
Skip the model "<synthetic>". Record the "effort" field if present — it is a
confound and I want to be able to filter on it.

── WHAT TO MEASURE ────────────────────────────────────────────────────────────
Per turn: total tool calls and per-tool counts; whether any dispatch tool ran
(Agent, Workflow, Task); and how many tool calls came AFTER the last dispatch.
That last one is the follow-through signal — zero means the turn ended the instant
the workers reported back. Also output tokens, and whether my next message was a
bare nudge ("continue", "keep going", "why did you stop").

On the final assistant text block of each turn — the part a human actually reads —
measure length, words per sentence, unique-word ratio, and per-1000-character
counts of bullets (^\s*[-*] ), headers (^#+ ), bold pairs, table rows (^\|),
backticks and digits, plus regex hits for:
  permission  want me to|should i|shall i|let me know|do you want|would you like|your call
  limitation  i (did not|didn't|have not|can't|cannot)|not verified|out of scope|deferred|blocked|untested
  caveat      caveat|worth flagging|to be clear|note that|heads.up|strictly speaking
  hedge       likely|probably|appears to|seems to|may be|might be|unclear|ambiguous

Per session: how many subagent transcripts were spawned, split into Workflow runs
(subagents/workflows/<run>/agent-*.jsonl — count the runs too, so you can get
agents per wave) versus plain Agent calls, and their total output tokens.

Coding, counted across the main loop AND every subagent beneath it, attributed to
the session's dominant model: Edit/Write calls, characters written, failed tool
results (is_error, or text starting "Error:", or containing "String to replace not
found"), Bash commands matching a test runner, a git commit, or a git revert, and
repeat edits to the same file path (rework). Counting only the orchestrator's own
edits will badly understate a model that delegates its typing — that mistake
reversed one of my conclusions.

── PICK THE MODELS ────────────────────────────────────────────────────────────
Find what is actually in my logs and how much of each:
    grep -ho '"model":"[^"]*"' ~/.claude/projects/*/*.jsonl | sort | uniq -c | sort -rn
Compare the two to four with the most turns. Treat anything under ~300 turns as too
small to draw a line from, and say so rather than quietly including it.

── OUTPUT SHAPE ───────────────────────────────────────────────────────────────
Write metrics.json exactly like this, or the page will not read it:

{
  "corpus": [ {"key":"turns","label":"Turns analysed","unit":"turns",
               "better":"high","note":"","v":{"model-a":3027,"model-b":1370}} ],
  "orch":  [ ... ], "esc": [ ... ], "prose": [ ... ], "code": [ ... ],
  "index": { "code": {"model-a":67.4,"model-b":91.2},
             "persist": {"model-a":83.6,"model-b":60.0} }
}

Five arrays — corpus, orch, esc, prose, code — each a list of metric rows with
key, label, unit, better ("high" or "low", meaning which direction is good), note,
and v mapping each model id to its value. Model keys must be identical everywhere.
Put cold-stop rate, work after last dispatch, agents per wave, subagents per
session, dispatch rate, tool calls per turn, dead turns and worker tokens in
"orch"; the question and phrase-density metrics in "esc"; the writing metrics in
"prose"; the edit and test metrics in "code".

For the two indices: score every model on each metric against the best performer —
100 x value / best where higher is better, 100 x best / value where lower is better
— then take the unweighted mean. "persist" is built from cold-stop rate, work after
last dispatch, agents per wave and subagents per session. "code" from edits,
characters, edit-failure rate, tool-error rate, rework, tests per edit, commits and
reverts.

── THEN TELL ME ───────────────────────────────────────────────────────────────
Which model ends the most delegated turns cold, which writes the most code per
session, and whether they are the same model. Flag every confound you can see:
models used in different months, on different projects, or at different reasoning
effort. Say which comparisons are too thin to trust. If a result contradicts what I
expected, say so plainly — do not smooth it over. Volume is not quality: edits and
commits measure throughput, and nothing here knows whether the code was correct.