--- name: model-behavior description: Measure how different models actually behave in your own Claude Code logs — orchestration follow-through, escalation, prose structure and coding output — and emit a metrics.json that redraws the charts at 3dl.dev/coding-vs-orchestration.html. Use when asked to compare models, audit which model is doing what, or check whether a model "stops early", escalates too much, or writes worse summaries. --- # Model behaviour from your own logs Measure how different models behave across the user's real Claude Code sessions, then emit `metrics.json` in the shape the published page consumes. **Write the extraction script yourself.** Python 3, standard library only, no installs. Run it, sanity-check the output, then report. Do not ask the user to download anything. ## Method ── WHERE THE DATA IS ────────────────────────────────────────────────────────── ~/.claude/projects//.jsonl one JSON object per line ~/.claude/projects///subagents/**/*.jsonl subagent transcripts Each record has: type ("user" | "assistant" | ...), timestamp, sessionId, isSidechain, message.model, message.usage.output_tokens, and message.content as a list of blocks typed "text" | "thinking" | "tool_use" (with .name, .input) | "tool_result" (with .tool_use_id, .is_error, .content). Transcripts rotate after ~30 days. If I use more than one machine, run this on each and merge — my laptop held a quarter of what my build box did, and the small slice gave the opposite answer on two metrics. ── WHAT COUNTS AS A TURN ────────────────────────────────────────────────────── One human message plus every assistant message before the next human message. A "user" record is NOT human if isSidechain or isMeta is set, if its content list contains a tool_result, or if its text begins with "", "". Record the "effort" field if present — it is a confound and I want to be able to filter on it. ── WHAT TO MEASURE ──────────────────────────────────────────────────────────── Per turn: total tool calls and per-tool counts; whether any dispatch tool ran (Agent, Workflow, Task); and how many tool calls came AFTER the last dispatch. That last one is the follow-through signal — zero means the turn ended the instant the workers reported back. Also output tokens, and whether my next message was a bare nudge ("continue", "keep going", "why did you stop"). On the final assistant text block of each turn — the part a human actually reads — measure length, words per sentence, unique-word ratio, and per-1000-character counts of bullets (^\s*[-*] ), headers (^#+ ), bold pairs, table rows (^\|), backticks and digits, plus regex hits for: permission want me to|should i|shall i|let me know|do you want|would you like|your call limitation i (did not|didn't|have not|can't|cannot)|not verified|out of scope|deferred|blocked|untested caveat caveat|worth flagging|to be clear|note that|heads.up|strictly speaking hedge likely|probably|appears to|seems to|may be|might be|unclear|ambiguous Per session: how many subagent transcripts were spawned, split into Workflow runs (subagents/workflows//agent-*.jsonl — count the runs too, so you can get agents per wave) versus plain Agent calls, and their total output tokens. Coding, counted across the main loop AND every subagent beneath it, attributed to the session's dominant model: Edit/Write calls, characters written, failed tool results (is_error, or text starting "Error:", or containing "String to replace not found"), Bash commands matching a test runner, a git commit, or a git revert, and repeat edits to the same file path (rework). Counting only the orchestrator's own edits will badly understate a model that delegates its typing — that mistake reversed one of my conclusions. ── PICK THE MODELS ──────────────────────────────────────────────────────────── Find what is actually in my logs and how much of each: grep -ho '"model":"[^"]*"' ~/.claude/projects/*/*.jsonl | sort | uniq -c | sort -rn Compare the two to four with the most turns. Treat anything under ~300 turns as too small to draw a line from, and say so rather than quietly including it. ── OUTPUT SHAPE ─────────────────────────────────────────────────────────────── Write metrics.json exactly like this, or the page will not read it: { "corpus": [ {"key":"turns","label":"Turns analysed","unit":"turns", "better":"high","note":"","v":{"model-a":3027,"model-b":1370}} ], "orch": [ ... ], "esc": [ ... ], "prose": [ ... ], "code": [ ... ], "index": { "code": {"model-a":67.4,"model-b":91.2}, "persist": {"model-a":83.6,"model-b":60.0} } } Five arrays — corpus, orch, esc, prose, code — each a list of metric rows with key, label, unit, better ("high" or "low", meaning which direction is good), note, and v mapping each model id to its value. Model keys must be identical everywhere. Put cold-stop rate, work after last dispatch, agents per wave, subagents per session, dispatch rate, tool calls per turn, dead turns and worker tokens in "orch"; the question and phrase-density metrics in "esc"; the writing metrics in "prose"; the edit and test metrics in "code". For the two indices: score every model on each metric against the best performer — 100 x value / best where higher is better, 100 x best / value where lower is better — then take the unweighted mean. "persist" is built from cold-stop rate, work after last dispatch, agents per wave and subagents per session. "code" from edits, characters, edit-failure rate, tool-error rate, rework, tests per edit, commits and reverts. ── THEN TELL ME ─────────────────────────────────────────────────────────────── Which model ends the most delegated turns cold, which writes the most code per session, and whether they are the same model. Flag every confound you can see: models used in different months, on different projects, or at different reasoning effort. Say which comparisons are too thin to trust. If a result contradicts what I expected, say so plainly — do not smooth it over. Volume is not quality: edits and commits measure throughput, and nothing here knows whether the code was correct. ## Reporting rules - Lead with the answer, not the method. - Name every confound you can see. Non-overlapping date windows, different project mixes and unmatched reasoning effort all change magnitudes and sometimes signs. - State sample sizes next to any claim built on a small one. - If the data contradicts what the user expected, say so in the first paragraph. - Never present a composite index as the finding. The underlying counts are the evidence; an index is a way of looking at them. Swapping members can reorder it — check whether yours does before leaning on it.