A pytest plugin that captures agent runs as ATIF trajectories, stores them as reviewable golden files, and fails the build when behaviour changes — not just when the answer changes.
The expensive parts of this project are already in the repository. ATIF export works, it is
event-driven, and atif_scope() hands you a typed Trajectory object in memory
with no file to parse. A hermetic FakeLLMClient exists for replaying scripted model
responses. Agents are ordinary classes, so a test is just pytest. What is missing is
small and specific: normalise a trajectory, reduce it to a comparable shape, diff two of
them, and wrap that in a fixture. That is the whole project.
The plan is therefore not "build a regression harness". It is "add four pure functions and a
pytest plugin to nooa-bench, in about 500 lines, with no new dependencies." Everything
below is either one of those pieces or a check that the piece works.
was "About 500 lines" counted only the half I was thinking about. The two source modules came to 576 lines, close to the estimate. But the tests, fixture agents and committed goldens add another 918, for 1,494 total. The estimate was not wrong about the module; it silently excluded the tests, which is the part of a testing tool you cannot skip. Read "about 500 lines" as "about 500 lines of library, and roughly as much again around it".
| Question | Answer |
|---|---|
| Where does the code live? | packages/nooa-bench/src/nooa_bench/ — an existing workspace member, already a dev dependency, plausibly upstreamable |
| New dependencies? | None. json, difflib, dataclasses, plus pydantic and pytest which are already there |
| What is a golden file? | A normalised ATIF trajectory, pretty-printed JSON, committed next to the test and reviewed in the PR diff |
| What runs in CI? | Frozen-LLM tests only — no API key, no network, deterministic. Live-model tests run nightly or on demand |
| How do you update a golden? | pytest --golden-update, the snapshot-testing convention |
| Smallest useful version | Phases 0–3. Roughly three focused days. Everything after that is leverage, not viability |
Phases 0–5 were built and run. The design survived, and the central bet — that ATIF's typed trajectories make this tractable — held. But eight specific claims below were wrong, and the pattern in how they were wrong is more useful than the list itself.
Every error came from the same source: reasoning about runtime behaviour by reading source code. The schema tells you which fields exist; it cannot tell you which ones are stable. The exporter's signature tells you it takes a path; it cannot tell you it always writes one. Nothing in the source says that adding a fixture to a test file will change the agent's system prompt. Each of these was invisible until something ran.
1 · The prompt contains memory addresses. NOOA renders the execution context —
every name in scope — into the system prompt, so any object without a custom __repr__
arrives as <function f at 0x78289b0c5d00>. That address changes every run. It is
now masked to 0xADDR. Unpredictable from source, and it invalidates every golden if
missed.
2 · Where you define a fixture agent changes its prompt. Same mechanism, worse
consequence. Adding one unrelated fixture to conftest.py broke all three goldens,
because the agent classes lived there and conftest's namespace is what gets rendered. Fixture
agents now live in a separate golden_agents.py whose namespace is deliberately small
and stable. Without this the harness is unusable: every test-scaffolding edit forces a re-record,
and re-recording on every change is how a golden suite becomes a rubber stamp.
3 · Asserting on shape alone left a hole. The plan argued — at length, and
persuasively — that comparing message content is a fool's errand. That reasoning is right for a
live model and wrong for frozen replay, where content is deterministic. A run with identical
structure but a different computed value passed while the committed golden went stale. Fixed with
a content_hash over the whole normalised trajectory, exact in frozen mode and
disabled live. It is also what caught error 1.
| # | The plan said | What happened |
|---|---|---|
| 1 | Volatile fields: timestamps, ids, cost, paths, durations | Also extra.generation_id and object addresses. Paths and durations never appeared at all |
| 2 | Fixture agents can live anywhere | Their defining module's namespace is rendered into the prompt |
| 3 | Assert on shape; never compare content | True live, wrong frozen — needs a content hash |
| 4 | "A test never needs to touch the filesystem" | atif_scope always writes a file. The first version left ~280 JSON files in the working tree |
| 5 | Call assert_atif_normative() on every capture | Dropped — it would couple the package to tests/atif, and the schema already validates |
| 6 | Register via a pytest11 entry point | A conftest.py instead — no tracked file edited. --import-mode=importlib also broke sibling imports |
| 7 | Phase 5: add testpaths, wire CI (2 hours) | Both already done upstream. Phase 5 cost nothing |
| 8 | Phase 0: uv sync and go | The repo imports fcntl and cannot run on Windows at all |
The phase-1 spike was supposed to catch #1, and its acceptance check passed anyway. The spike ran the agents from a standalone script, diffed two runs, and produced a clean list of four volatile fields — exactly the deliverable the plan asked for. It missed the addresses because a script has a different namespace than a pytest module, so the prompt it produced was not the prompt the real tests would produce.
The lesson generalises past this project: a spike that does not reproduce the real execution environment produces a confident, incomplete answer — which is worse than no answer, because the acceptance check goes green. If the plan were rewritten from scratch, phase 1 would say "run the spike as a pytest test, in the directory layout you intend to ship".
Scoping this project badly means writing a trajectory capture layer that already ships. Read these five things before writing a line.
atif_scope() — capture, already solvedAn async context manager that installs the exporter on an agent's event manager, writes the
trajectory on exit, and — the important part — yields the exporter, whose
get_trajectory() returns the live Trajectory model.
A test never needs to touch the filesystem to get its trajectory. It also records
extra.crashed = True when the wrapped block raises, so a crashed run is still a
comparable artifact rather than a missing one.
was The scope always writes a file, whether you read it or
not. Reading get_trajectory() instead of the file does not stop the write —
it only stops you parsing it. Left at its default the file lands in ./logs/atif/,
relative to wherever pytest was started. The first working version of capture()
quietly deposited about 280 JSON files into the repository before anyone noticed.
capture() now points the scope at a TemporaryDirectory, and a test
asserts the working tree is untouched afterwards. Cheap to fix, easy to miss: a stray output
directory is not something a test suite fails on.
src/nooa/atif/install.py:182 (atif_scope), :265 (enable_atif), src/nooa/atif/exporter.py:895 (get_trajectory), :850 (finalize_on_exception)
Trajectory is a Pydantic model with extra="forbid" throughout:
a root with agent, ordered steps, optional final_metrics, and
nested subagent_trajectories. Each StepObject carries
step_id, source, message, tool_calls,
observation, metrics, and llm_call_count. This is a typed tree,
not a log. model_dump() gives you a plain dict to diff; model_validate_json()
gives you back a validated object from a golden file. Both directions are free.
src/nooa/atif/schema.py — SCHEMA_VERSION = "ATIF-v1.7" at :27
tests/atif/normative.py defines nine MUST rules the structural schema does not
cover: sequential step_ids from 1 (N1), tool-call/observation joinability within a
step (N2), trajectory_id uniqueness (N3), is_copied_context propagation
across a compaction boundary (N4), llm_call_count = 0 implying absent metrics (N5),
ISO 8601 timestamps (N6), message presence (N7), final_metrics summation (N8), and
sub-trajectory reference resolvability (N9). Call assert_atif_normative() on every
captured trajectory before comparing it — a malformed capture should fail as a capture bug, not as
a spurious behavioural diff.
was Not called. The validator lives in the framework's test
tree, not its package. tests/atif/normative.py ships with the repo's tests
and is not installed, so importing it would couple nooa-bench to
tests/atif — fatal for the upstreaming pitch, since the package would no longer be
installable on its own.
Dropping it costs less than it looks. get_trajectory() constructs a Pydantic
Trajectory, so structural validation has already run by the time you hold one. The
normative rules are the exporter's contract, covered by 126 of its own tests. And a violation
that did slip through would appear in the golden diff rather than vanish. The reasoning in the
original text was sound; the packaging consequence was not considered.
tests/atif/normative.py; used by tests/atif/test_end_to_end_codeact.py
FakeLLMClient — hermetic runs, already solvedTakes scripted_responses: list[LLMResponse] and pops them in order under an
asyncio.Lock, so concurrent calls still consume the script deterministically. It
records call_count and last_messages. Exhausting the queue yields an empty
response rather than raising — worth knowing, because a behavioural change that adds an LLM
call will silently get an empty response instead of an obvious error. Phase 4 wraps it to fail
loudly instead.
src/nooa/unifiedllm/fake.py:15; tests/atif/test_end_to_end_codeact.py is a working template of a stubbed CodeAct run
asyncio_mode = "auto" means async tests need no decorator. An
integration marker already exists for "makes real API calls" — live-model trajectory
tests belong under it rather than under a new marker. packages/nooa-cli/tests is
already in testpaths, so adding packages/nooa-bench/tests follows an
established pattern.
was packages/nooa-bench/tests is already in
testpaths too — I read the list, saw the nooa-cli entry, and
stopped reading one line early. Combined with .github/workflows/ci.yml:38 already
running pytest -m "not integration and not stress" from the root, the frozen tests
are gated by the existing CI job the moment the files exist. Phase 5 turned out to be nothing.
Also unnoticed: addopts includes --import-mode=importlib, which does
not put a test directory on sys.path. Sharing a helper module between test files
needs an explicit sys.path insert in the conftest — two lines, but it took a failed
run to find.
pyproject.toml:206–230 — testpaths at :208, addopts at :225
trace_analyzer.py is not the foundation for this. The project
write-up implies it is. It is not. All 186 lines of it read OTel JSONL span files —
start_time_unix_nano, attributes["llm.token_count.prompt"],
gen_ai.usage.input_tokens — and aggregate per-model token counts and latencies into a
TaskUsageStats dataclass. It never touches ATIF. It knows nothing about steps,
tool calls, or ordering.
That does not sink the project; it relocates it. TraceAnalyzer remains useful as the
cost half of a comparison (tokens, latency, model mix), while ATIF supplies the
behaviour half (what happened, in what order). Plan to use both, but do not plan to extend
trace_analyzer.py into a trajectory differ. The honest sentence is: this project shares
a package with nooa-bench and reuses its runner conventions, but the diffing code is new.
This is the decision the whole design turns on, and getting it backwards produces a harness that either cannot run in CI or cannot detect anything.
A regression test needs a fixed baseline. But an agent run has two sources of variation — your code, and the model. If you re-run against a real model, every diff is contaminated by sampling noise, and you cannot tell "my prompt edit changed behaviour" from "the model felt different today". If you replay recorded model responses, the run is perfectly deterministic — but the model can no longer react to your prompt edit, so a prompt change produces no diff at all.
Both are useful. They detect different things, so build both and be explicit about which is which.
| Frozen mode | Live mode | |
|---|---|---|
| LLM | FakeLLMClient replaying a recorded script | The real model |
| Determinism | Total — same bytes every run | None — sampling, model drift, provider changes |
| Detects | Regressions in your code: strategy plumbing, context assembly, method dispatch, parsing, tracing, step ordering, token accounting | Regressions in behaviour: prompt edits, model swaps, strategy swaps |
| Blind to | Anything requiring the model to react — a prompt edit changes the prompt but not the scripted reply | Nothing, but it cannot distinguish signal from noise on a single run |
| Cost | Zero. No key, no network, milliseconds | Real tokens, real seconds, needs a sandbox for CodeAct |
| Where it runs | Every PR — this is the CI gate | Nightly, or on demand, under the existing integration marker |
Frozen mode is the product. It is the thing that can actually gate a merge, and it catches the class of bug that is currently invisible: a refactor of context assembly that quietly doubles the number of steps, or a strategy change that stops emitting an observation. Live mode is a research instrument — genuinely valuable for "did this prompt edit help", but it needs repetition and tolerance bands rather than equality, and it should never block a build.
Build frozen mode first even though live mode is the headline use case. Live mode without a normaliser and a differ is just running the agent twice and squinting. Both modes share the normaliser, the shape extraction, and the diff — so the shared core gets written once, proven under conditions where every difference is a real bug, and only then pointed at a noisy signal.
A pretty-printed JSON file containing the normalised ATIF trajectory, committed next to the
test that produced it: packages/nooa-bench/tests/golden/<test_name>.json. Frozen-mode
tests get a sibling <test_name>.llm.json holding the recorded model responses.
Committing the full normalised trajectory rather than a summary is the single most valuable decision in this design, and it is worth being explicit about why. When a test fails, the assertion message tells you that step count went 6 → 8. The committed file tells you what those two extra steps were — in the PR diff, in review, without re-running anything. The file is the documentation of the agent's behaviour, and a behavioural change becomes a reviewable diff rather than a number that moved.
Both files are plain JSON with sorted keys and two-space indent, so git diff is
readable and merge conflicts are resolvable by hand.
Three pure functions on Trajectory, in one new module. No classes, no config, no
plugin architecture.
normalize(traj) -> dictStrips everything that varies between two identical runs. The exact list must come from
observation, not from reading source — that is what phase 1 exists for — but the known candidates
are: timestamp on every step, session_id and trajectory_id at
every level, tool_call_id and its matching source_call_id (rewritten to
call_1, call_2… preserving the join rather than deleting it),
cost_usd, absolute paths appearing in observations, and any . duration or
latency fieldschema_version is kept — a schema bump must invalidate
goldens loudly.
was The guessed list was wrong in both directions. Two
members never appeared — no absolute path and no duration field showed up in any observed diff, so
scrubbing them would have been dead code written against an imagined failure. Two real ones were
missing: extra.generation_id (a fresh uuid4 per generation) and object-repr addresses
inside message text.
Below is what the module actually contains. cost_usd is the one entry kept on
faith — it was never observed varying, because a stubbed client reports zero cost. It is scrubbed
defensively on the assumption that live pricing moves, and that assumption is untested.
# nooa_bench/trajectory.py — as built
VOLATILE_FIELDS = (
"session_id", "trajectory_id", # fresh per run
"timestamp", # wall clock, on every step
"generation_id", # uuid4 per generation, under step.extra
"cost_usd", "total_cost_usd", # varies with live pricing (assumed)
)
# NOOA renders the execution context into the system prompt, so anything in
# scope without a __repr__ arrives as "<function f at 0x7f3c...>".
_ADDRESS = re.compile(r"0x[0-9a-fA-F]{6,16}")
def normalize(traj, *, sort_subagents=True) -> dict:
"""ATIF trajectory -> stable dict, safe to compare and to commit."""
d = traj.model_dump(mode="json", exclude_none=True)
return _normalize_one(json.loads(json.dumps(d)), sort_subagents)
def _normalize_one(d, sort_subagents):
_scrub(d) # delete volatile keys, mask 0x... in strings
_renumber_call_ids(d) # call_1, call_2, ... join preserved
subs = d.get("subagent_trajectories")
if subs:
subs = [_normalize_one(s, sort_subagents) for s in subs]
if sort_subagents: # gather completes out of order
subs.sort(key=lambda s: json.dumps(s, sort_keys=True))
d["subagent_trajectories"] = subs
return d
The recursion and the subagent sort were not in the sketch. Both are load-bearing: a five-way
asyncio.gather fan-out produces five embedded subagent trajectories that arrive in
completion order, so without a content-derived sort the golden differs on every run. The plan
flagged this as a risk; it materialised on the first try.
shape(traj) -> ShapeA small dataclass holding the things you want to assert on, extracted from the normalised dict:
step count, the ordered list of (source, llm_call_count) pairs, the ordered list of
called function names, the number of distinct subagent trajectories, whether the run crashed, and
total prompt/completion tokens. Roughly ten fields.
The split matters. Shape is what you assert on; the normalised dict is what you commit and
read. Asserting on the full dict makes every whitespace change in a model reply a test
failure. Asserting only on the shape makes the failure message legible — "step count 6 → 8,
functions called: execute_python ×3 → ×5" — while the committed file still carries the
detail for whoever investigates.
was This argument is right for live mode and wrong for frozen mode, and the plan applied it to both. "Every whitespace change in a model reply" is a problem only when the model is free to vary its wording. Under frozen replay the reply is fixed, so a content change means something in your code produced a different value — precisely the thing worth catching.
The consequence was concrete: a trajectory with identical structure but a different computed
result compared equal, the test stayed green, and the committed golden silently described
behaviour that no longer happened. Shape gained an eleventh field,
content_hash — a SHA-256 prefix over the whole normalised trajectory — compared
exactly by default and skipped via compare_content=False against a live model. Deltas
sort it last, so the named structural fields are what a reader sees first and the hash only says
"and something else moved too".
It paid for itself immediately: the memory-address leak was found by this check, not by inspection.
diff(golden, actual, *, token_tolerance=0.0, compare_content=True) -> list[Delta]Compares two shapes field by field, returns a list of Delta(field, golden, actual).
Token fields compare within a tolerance — 0.0 in frozen mode where equality is
achievable, something like 0.15 in live mode. Empty list means pass. For the human-facing
report, hand the two normalised dicts to difflib.unified_diff over their pretty-printed
JSON lines; that is the entire rendering layer.
capture(agent, awaitable) -> TrajectoryTwenty lines wrapping atif_scope: point the scope at a temporary directory, run the
awaitable inside it, return exporter.get_trajectory().
Call The temp directory is not
decoration — see the correction under assert_atif_normative() on the result.atif_scope above.
Everything above is read from source and has never been run — the same gap the doc set has. Before
writing new code, execute tests/atif/test_end_to_end_codeact.py and
examples/quickstart/14_atif_trajectory.py and confirm that a trajectory comes out where
the source says it does. This is also where the environment gets built:
uv sync --group dev, and a container if you intend to run CodeAct against anything real.
tests/atif/test_end_to_end_codeact.py, which exercises the same
export path with a stubbed client, plus a hand-written script that round-tripped a captured
trajectory through Trajectory.model_validate_json(). Adequate, but not what the plan
asked for, and it left the live path unexercised right through to the end.was "uv sync --group dev and go" does not work on
Windows. src/nooa/storage/sqlite.py:10 imports fcntl, which is
POSIX-only, so importing nooa at all fails before any test collects. The repo
declares no platform support either way; this is simply not a supported host.
Everything here therefore ran in WSL Ubuntu, with the virtualenv outside the Windows
filesystem (UV_PROJECT_ENVIRONMENT=$HOME/nooa-venv) to avoid the 9p I/O penalty and a
clash with the half-built Windows .venv. Use uv run --frozen: a plain
uv run on Linux adds an sdist entry to uv.lock, which dirties a tracked
file in what is meant to be a pristine upstream clone.
Write one throwaway script that runs the same stubbed CodeAct agent twice, dumps both trajectories
to JSON, and diffs them with difflib. Every line that differs is a field the normaliser
must handle. Do not skip this and design the normaliser from the schema — the schema tells you what
fields exist, not which ones are unstable, and the surprises live in the gap between those two.
Suspects worth watching for: REPL variable naming, asyncio.gather completion order in
fan-out, dict iteration order in arguments, and anything embedding a path or a PID.
session_id,
trajectory_id, timestamp, generation_id — out of 20 differing
lines in 135. Every one was observed rather than guessed, exactly as required. It missed the fifth
class entirely.was "One throwaway script" is the flaw. A standalone
script has a different module namespace than a pytest test module, and NOOA renders that
namespace into the system prompt. The script's prompt therefore contained no
<function ... at 0x...> reprs, so no address churn appeared in the diff — and
the omission only surfaced two phases later, when the goldens failed on a second run for reasons
the normaliser had no answer for.
Rewrite this phase as: run the spike as a pytest test, in the directory layout you
intend to ship, with the fixture agents where they will actually live. The cost is the
same two hours. The second observation from the spike was correct and important:
asyncio.gather fan-out really does emit its five subagent trajectories in
nondeterministic order, and 276 of 247 lines differed until they were content-sorted.
One new module, packages/nooa-bench/src/nooa_bench/trajectory.py, containing the four
functions above and the Shape / Delta dataclasses. Pure functions on
Pydantic models — no I/O, no globals, no async. Unit-tested against trajectories built by hand and
against the fixtures already in tests/atif/.
The tests that matter here are the negative ones: two trajectories differing only in timestamps must produce zero deltas, and two differing by one extra step must produce exactly one delta naming the step count.
normalize(t) == normalize(t) byte
for byte across two runs, for both the CodeAct case and the fan-out case, where phase 1 showed
20 and 276 differing lines respectively. 26 unit tests, all on hand-built dicts except three that
run real agents. The negative tests the plan singled out both exist and both pass.Deviation: the module is 419 lines rather than the implied ~300, mostly because
normalize recurses into subagents and check carries the golden-file I/O
that the plan had left unassigned to any function.
A golden_trajectory fixture plus a --golden-update flag,
registered through . On a miss it fails with the unified diff and the exact
command to re-record. With [project.entry-points.pytest11] in
packages/nooa-bench/pyproject.toml so it activates on install with no
conftest.py plumbing--golden-update it rewrites the file and reports what
changed rather than passing silently.
was A conftest.py instead — because the entry
point requires editing a tracked file. labs-OO-Agents is a pristine clone of
NVIDIA's repository, not a fork, and adding a pytest11 entry to its
pyproject.toml would modify tracked upstream content for a change that has not been
proposed to anyone yet. A conftest keeps the entire contribution additive: git status
shows ten new untracked files and zero modifications.
The trade the plan was avoiding is real — pytest_addoption is only honoured in an
initial conftest, so the flag could have failed on some invocations. It was tested from
the repository root and works, because testpaths already names this directory. An
GOLDEN_UPDATE=1 environment variable is wired as a fallback for invocations where it
would not.
The fixture is a plain callable rather than an object with a .check() method:
await golden_trajectory(agent, agent.run(...)). One less name to remember, and
nothing was lost.
async def test_codeact_three_turn_run(golden_trajectory, codeact_agent):
agent = codeact_agent(turns=3)
await golden_trajectory(agent, agent.run("compute 2+2"))
# miss -> unified diff + "if this change is intended: pytest --golden-update"
Missing golden file on a normal run is a failure, not an auto-create. Auto-creating means a test that has never actually asserted anything can sit green in CI forever.
content_hash with the before/after prompt visible
in the diff. Making the agent take an extra turn produced seven deltas, structural ones first:
step_count: 3 → 4, functions: ('return_result',) → ('execute_python',
'return_result'), llm_calls: 1 → 2, tokens doubled. That second message is
exactly what the summary promised this tool would print.A RecordingLLM wrapper around any UnifiedLLM that stores each
LLMResponse as it passes through — content, tool_calls,
finish_reason, assistant_message, usage; not
raw_response, which is provider-specific and may not serialise. Replay reconstructs
LLMResponse objects and hands them to FakeLLMClient.
One deliberate deviation from the existing fake: replay must raise when the script is exhausted rather than returning an empty response. An agent that makes more LLM calls than it did when recorded is exactly the regression this project exists to catch, and the current fail-soft behaviour would hide it inside a confusing downstream error.
qwen2.5:1.5b on a local Ollama, then replayed twenty times with no network — one
distinct trajectory, byte-identical to the live original. It did not pass first time: see
bug 1, where the replayed trajectory differed from its own recording on
the model_name line, a failure no amount of fake-client testing could have produced.was The strategy wraps the exhaustion error. Replay
raises ScriptExhausted as designed, but CodeActStrategy treats any
client exception as a transient API error: it retries three times and re-raises as
GenerationError. The message survives intact, which is what a person reading a CI log
needs, and the test asserts on that rather than pretending the raw exception propagates. Worth
knowing before you write pytest.raises(ScriptExhausted) and watch it fail.
Add Frozen tests run
in the existing PR job — no key, no marker, no special casing. Live tests carry the existing
packages/nooa-bench/tests to testpaths.integration marker and run on a schedule. When a golden file changes in a PR, that diff
is the review artifact; no dashboard is needed to make it visible.
was Nothing to do. testpaths already
lists packages/nooa-bench/tests, and .github/workflows/ci.yml:38 already
runs uv run pytest -q -m "not integration and not stress" from the repository root.
The tests were gated the moment the files existed. No workflow file, no config change, no edit to
anything tracked.
This is the one estimate that was wrong in the pleasant direction, and it was wrong for an
unflattering reason: I read the testpaths list, recognised the nooa-cli
entry, and stopped before the next line.
--golden-update parses correctly from a bare root invocation — the one
failure mode the conftest approach risked.Read CONTRIBUTING.md and AGENTS.md first. The pitch is narrow and
therefore credible: "trajectory diffing and a golden-file pytest fixture for
nooa-bench", one module plus one plugin, no new dependencies, following the package's
existing conventions. Land the core module before proposing the plugin — a pure-function differ is
easy to accept, and a pytest plugin that ships in a distribution is a policy decision someone else
owns.
HANDOFF.md — whether this clone should become a
fork at all — has to be answered before it can begin.Revision 3: the Ollama run recommended below has now been done, against
qwen2.5:1.5b (986 MB, no API key). It was worth an afternoon: it found two more real
bugs, one of which could not have been found any other way. Fifty-three frozen tests and four live
tests now pass.
was The bug I predicted was not the bug I had. The
original text below singled out _to_json's Pydantic branch as "the obvious way to find
out it is wrong". That branch did execute — PredictStrategy returns
content as a generated model, CapitalOfResponse(value='Tokyo') — and it
was correct. Serialising it to JSON text and letting the strategy re-parse on replay
produced an identical trajectory.
What actually broke was one line away and unguessed. See below.
NOOA copies the client's model into every agent step as model_name.
FakeLLMClient calls itself fake-model. So a trajectory recorded against
ollama_chat/qwen2.5:1.5b and then replayed differed on exactly one line —
"model_name": "ollama_chat/qwen2.5:1.5b" versus "fake-model" — which is
enough to fail its own golden. Frozen mode was, in the only way that matters, broken.
This is invisible when the recording client is itself a fake, which is why
five green record/replay tests never caught it. The script now stores the recorded model name and
replay() reports it. model_name is deliberately not scrubbed:
noticing a model swap is a headline use of this tool, so the fix has to be to carry the name
forward, not to delete the field.
Any tool-using strategy fails with
litellm.UnsupportedParamsError: ollama_chat does not support parameters:
['parallel_tool_calls'], retried three times and re-raised as GenerationError.
This is not a bug in this project — it sits between NOOA, LiteLLM and Ollama — but it contradicts
the ideas document's claim that "local models are first-class: Ollama and vLLM are both supported
through LiteLLM with no API key". That holds for plain generation. It does not hold for CodeAct,
which is the framework's headline strategy.
Setting litellm.drop_params = True fixes it, and with that CodeAct ran and returned
a real tool call. The live tests set it in a fixture and restore it afterwards, rather than in the
library — it is global mutable state on litellm and belongs to the caller.
call_3hfgfv2c; the normaliser produced call_1. Until this run every id
the harness had ever seen was one the fixtures had pinned by hand.token_tolerance=0.15, compare_content=False) — and, on this model and
prompt, produced zero differing lines even under frozen settings.usage carries fields the fake never produced —
total_tokens, completion_tokens_details,
prompt_tokens_details — and none of them destabilised the golden.Running the CodeAct live test repeatedly, roughly one run in three fails with
return_result validation failed after 3 attempts — a 1.5B model handing back something
that will not coerce to int. Nothing is wrong with the harness. The model is simply
not very good.
A live test that cannot tell "the harness broke" from "the model was weak today" is
worse than no live test, because it trains you to ignore red. The test now catches
GenerationError and skips with the reason. A rerun would have hidden the frequency; a
skip reports it. Five consecutive runs of the live suite: four clean, one skip, zero failures.
Nothing in the original plan said this. It is the single most transferable lesson here, and it is the concrete form of the claim made further up that live mode "should never block a build".
cost_usd has still never been observed varying. Ollama is free
and reports no cost, so the assumption that live pricing moves remains untested. A paid provider
would settle it.token_tolerance=0.15 has still never had
to absorb real noise. A larger model at a higher temperature would.extra. The volatile-field list is now evidence-based rather than guessed, but the
evidence is still narrow.Everything was observed against a CodeAct agent and a fan-out agent. A different strategy, a multimodal run, a context-compaction boundary, or an agent using MCP could all introduce fields that vary and are not scrubbed. The failure mode is benign — a golden that will not stabilise — but it will look like a bug in the harness rather than a missing entry in a tuple.
0x[0-9a-fA-F]{6,16} is applied to every string in the trajectory. It will also mask
a genuine hex literal in a model's reply or in code the agent wrote. That is harmless for
comparison — both sides are masked identically — but it means a golden file cannot be read as a
faithful transcript of what the model saw. If an agent's actual work involves hex, this needs
narrowing to the <... at 0x...> repr form.
Golden files rotting into rubber stamps has had nothing done about it beyond keeping the files small. The mitigation was always social, and no amount of code will supply it.
ATIF version drift is handled — a mismatched schema_version raises
with a re-record instruction, and there is a test for it — but ATIF has not actually bumped, so the
handling is unexercised against a real migration.
Each of these is a real feature that a real version of this tool might eventually want. None of them is needed to detect a regression, and every one of them is a week that produces no additional detection.
trace-explorer. Golden diffs render in git diff, which every reviewer
already has open.Four of the five below were written before anything ran. In the event, the ordering was roughly right and the biggest actual problem was not on the list at all — the prompt-namespace coupling that cost a file reorganisation. Risk registers are good at the failures you can imagine.
The one that could sink phase 2. If step ordering itself varies between two frozen runs —
most plausibly through asyncio.gather fan-out, which is exactly the pattern
method_writing_lib.py encourages — then no amount of field scrubbing produces a stable
golden. Mitigation: phase 1 finds this in two hours rather than three days. If it
bites, the shape becomes order-insensitive at the fan-out boundary — compare a multiset of sibling
calls rather than a sequence — and the committed file sorts those siblings by a stable key.
classify calls under one gather emit
five embedded subagent trajectories in completion order; 276 lines differed between two runs.
Sorting the normalised children by their canonical JSON fixed it. The ceiling is documented in the
code: content-sorting also hides a meaningful reordering, so sort_subagents=False
exists for pipelines where sequence carries information.The classic failure of every snapshot-testing system: a diff appears, nobody reads it,
--golden-update gets run, green returns. Mitigation is social, not technical
— keep golden files small enough to actually read, make the assertion message name the specific
behavioural change, and never auto-create a missing golden.
schema_version is a Literal["ATIF-v1.7"]. A bump to v1.8 invalidates
every golden file at once. Mitigation: keep schema_version in the
normalised output and fail with "golden recorded under ATIF-v1.7, running v1.8 — re-record" rather
than emitting a thousand-line diff.
Phase 4 records against a real model, and CodeAct executes model-generated Python. The README is explicit at lines 130–133 that its AST checks and deny-lists are "defense-in-depth guardrails, not a containment boundary". Mitigation: record inside a container or VM. This applies only to recording — once the script exists, replay is inert.
/mnt/c.CodeActLiteStrategy and ReflexionStrategy are gated behind a
FutureWarning and re-exported lazily so the warning cannot be bypassed. Goldens recorded
against them sit on an unstable API. Mitigation: build the fixture agents on
CodeActStrategy and PredictStrategy; treat the experimental ones as a later
test case, not a foundation.
| Phase | Estimated | What it actually left behind |
|---|---|---|
| 0 · Prove the ground | ½ day | A WSL environment (Windows cannot run this repo); 126 ATIF tests green; quickstart 14 skipped |
| 1 · Spike | 2 hours | Four of the five volatile-field classes. Green acceptance, incomplete answer |
| 2 · Core functions | 1 day | trajectory.py, 419 lines, 26 unit tests |
| 3 · pytest plugin | 1 day | The usable product — conftest fixture, 3 committed goldens, both failure demos verified |
| 4 · Frozen mode | 1 day | recording.py, 157 lines, 5 tests. Never pointed at a real model |
| 5 · CI wiring | Nothing — already wired upstream | |
| 6 · Upstream | open | Not started; blocked on the fork question in HANDOFF.md |
Roughly three focused days to something you would use, four and a half to something you
would defend in a PR. That is materially cheaper than the "substantial" label in the
project write-up, and the reason is the whole point of the ranking: NOOA already ships the expensive
half. Typed trajectories, always-on nested tracing, and re-instantiable agents are the three things
this needs, and having all three is why the project is three days here and three weeks elsewhere.
was I cannot honestly report a day count, and it would be misleading to try. Phases 0–5 were built in one agent session, which says nothing useful about how long a person would take. The estimate is left in place unrevised because the claim it supports — that this is days rather than weeks, because NOOA supplies the expensive half — did hold. Nothing in the build required inventing infrastructure; every phase was assembly.
What can be reported honestly is the shape of the work. Roughly half the elapsed effort
went into the three findings in the revision log, not into the functions the plan
described. Writing normalize, shape and diff was
straightforward. Discovering that prompts contain memory addresses, that conftest membership
changes those prompts, and that shape comparison alone leaves a hole took several failed runs each,
and each one arrived as a mysterious test failure rather than an obvious bug. Budget for that, not
for the line count.
One sequencing note that survives from the ideas document. Building project 01 first — the portfolio analyst — is still the right move, and this plan sharpens why rather than softening it. Phase 1 asks you to look at a trajectory diff and judge which differences are noise and which are behaviour. That judgment is much easier if you have already watched your own agent go wrong a few hundred times. Without it, phase 1 is guesswork dressed up as observation.
Building it first, out of order, sharpened this further rather than refuting it. Phase 1 was guesswork dressed up as observation — it produced a confident list that was missing an entire class of volatile field, and nothing in the acceptance check could tell. Two agent shapes were enough to build against and are visibly not enough to trust. Someone who had already watched a few hundred of their own traces would have looked at that clean four-item list and asked what else was in the prompt.
Ten new files under packages/nooa-bench/, all untracked; zero tracked files modified,
so git status in the clone shows only additions and the pre-existing docs/.
The clone remains pristine, which keeps the open question in HANDOFF.md — leave
untracked, branch, fork, or relocate — open rather than answered by accident.
packages/nooa-bench/
src/nooa_bench/trajectory.py 419 capture / normalize / shape / diff / render / check
src/nooa_bench/recording.py 190 RecordingLLM, strict replay, model-name carry-forward
tests/conftest.py 135 golden_trajectory fixture + --golden-update
tests/golden_agents.py 101 fixture agents — namespace kept small on purpose
tests/test_trajectory.py 314 26 tests
tests/test_recording.py 126 7 tests
tests/test_golden.py 34 3 golden tests
tests/test_live_ollama.py 185 4 live tests (deselected by default)
tests/golden/*.json 373 3 committed goldens
frozen: uv run --frozen pytest packages/nooa-bench/tests # 53 passed, 4 deselected
re-record: uv run --frozen pytest packages/nooa-bench/tests --golden-update
live: OLLAMA_API_BASE=http://<host>:11434 \
uv run --frozen pytest packages/nooa-bench/tests/test_live_ollama.py -m integration
Verified: 53 frozen tests pass and 4 live tests pass against a real Ollama; the live suite skips
cleanly when no endpoint is reachable, so CI is unaffected; the full repository suite passes (6,417
tests, 5 skipped — the four MCP collection errors are pre-existing, from the uninstalled
mcp extra); ruff check and ruff format --check clean; goldens
byte-stable across repeated runs; both failure demonstrations produce the messages quoted in
phase 3; capture() leaves the working tree clean.
Reproducing the live run on this machine. Ollama is installed at
%LOCALAPPDATA%\Programs\Ollama but had no models; ollama pull qwen2.5:1.5b
fetched 986 MB. It must be started with OLLAMA_HOST=0.0.0.0 for WSL to reach it, and
OLLAMA_API_BASE must point at the WSL gateway address from
ip route show default — not localhost, which in WSL is WSL itself.