Prompt look-back report: what a decision model says about the author's prompts

Run on 2026-10-04 · lookback/lookback.mjs · Model typesafe-ai/jev through the Vercel AI Gateway · Prompt text is not published on this page

Result. The prompt pattern is healthy. 82% of the 113 prompts ask for research or for something to be built, and those need a language model. Only about 11% are decision-type steps at probability 0.6 or more, so there is little at the prompt level to move to a cheap decision model. No change to how the prompts are written is needed.

20most recent sessions

113prompts labelled, 0 failed

$0.0025total cost, about $0.00002 per prompt

12decision-type at p ≥ 0.6 (10.6%)

1. Method

The script reads the plain-text user prompts from the 20 most recent Claude Code session files on one machine. It skips meta lines, command tags, tool results and prompts shorter than 8 characters. Each prompt, cut to its first 300 characters, goes to Jev as one choice question: "What kind of step is this request from a user to a coding agent?" There are seven kinds. Four are decision kinds: classify_route, pick_from_list, yes_no_check and score_rank. A prompt counts as a candidate when Jev picks a decision kind at probability 0.6 or more.

2. What kind of prompts

Kind	Prompts	Share
research_or_multistep	53	46.9%
write_or_build	40	35.4%
yes_no_check	12	10.6%
pick_from_list	4	3.5%
chat	2	1.8%
score_rank	1	0.9%
classify_route	1	0.9%

Teal bars are decision kinds: 18 of 113 prompts (15.9%). Orange bars are the other kinds. Shares add to 100%.

The two large kinds, research and build, total 93 prompts (82.3%). They ask for investigation, explanation, a document, code or a multi-step task. A decision model that only picks from a given list cannot do them.

3. The 12 decision-type prompts, read by hand

Jev labels the kind of request. It does not check whether it could answer the request. I read the 12 prompts that passed p ≥ 0.6 and sorted them by whether a Jev call could do the job when given the right evidence.

Verdict	Prompts	What they are
Plausible Jev job	3	Status checks on finished work ("was it tested", "is it done"), and a lookup in a list of installed plugins. A yes/no or pick question works if Jev gets the evidence: test output, a diff, or the list.
Possible, weak	2	A conflict check between two pasted statements, and a confirmation about a list of steps whose text the script did not see. Not tested.
Not a Jev job	7	Product-research questions (4), a request to review figures (1), a question about whether the agent can do something (1), and a question about the agent's previous reply (1). They need knowledge, looking at a file, or context the script does not send.

4. What this means

The usage pattern is fine. Most prompts ask for work that needs a language model, and that is what a coding agent is for.
At the prompt level, Jev could take over about 3 of 113 prompts (2.7%) by my reading, and only with evidence supplied. The saving is small.
The better place for Jev is inside the agent's own steps: which file to open, which skill to load, did the tests run. The existing skill-router, verify and coding-reflex builds work there.

5. Limits of this report

One run, one machine, one day, 20 sessions, 113 prompts. The shares are not stable estimates.
Jev saw only the first 300 characters of each prompt and none of the replies or files. A short follow-up is labelled without its context.
The kind labels were not compared with hand labels. Only the 12 candidates were checked, by one reader.
The script reads user prompts only. It does not read the agent's tool calls, which is where most decision steps happen. That is not built.
This page contains counts only. Prompt text, file paths, project names and personal details are left out on purpose.