Project 03 · explained · tutorial clone of NVIDIA-NeMo/labs-OO-Agents

Agents you can diff

Project 03 builds golden-trajectory regression testing for NOOA agents: capture a run, strip out everything random about it, and commit the result so that a change in how the agent worked shows up as a reviewable diff — even when the final answer is identical.

This page explains what the feature is, why NOOA makes it possible when most agent frameworks make it awkward, and then walks every new line of code. The companion implementation plan covers what went wrong while building it. Scoped against labs-OO-Agents at HEAD 0cccda1. Two reading depths: the green In plain terms boxes carry the whole argument on their own; the code sections are for whoever has to maintain it.

1. The problem: the answer is right, the agent is broken

An agent test normally asserts on the output. Ask the agent to compute 2+2, assert it returns 4. That test passes on the day you write it and keeps passing forever, including on the day someone refactors the prompt assembly and the agent now needs eight round-trips to the model instead of three, burns four times the tokens, and calls a tool it never used to call. Same 4. Green build.

The interesting failures in agent systems are almost never wrong answers. They are:

• a prompt edit that makes the model take a longer route to the same place;
• a strategy change that quietly drops an observation, so the model stops seeing tool results;
• a refactor of context assembly that reshapes the system prompt in a way nobody reviewed;
• a model swap that nobody announced;
• a token-cost regression that only shows up on the invoice.

What you want is a test that fails when the agent's behaviour changes. That is a snapshot test — the same idea as snapshot-testing rendered UI — except the thing being snapshotted is an execution trace, and execution traces are full of timestamps, UUIDs, memory addresses and race-dependent ordering. Snapshot a raw trace and the test fails on every run for reasons that mean nothing. That gap — between "a trace exists" and "a trace can be committed to git and diffed" — is what this project closes.

2. The NOOA feature being showcased: the run is a first-class object

NOOA emits every agent run as an ATIF trajectory — Agent Trajectory Interchange Format, version 1.7 in this tree. It is not a log file and not a vendor dashboard. It is a Pydantic schema in the library itself, at src/nooa/atif/schema.py, with a documented public surface re-exported from nooa.atif:

from nooa.atif import (
    SCHEMA_VERSION,      # "ATIF-v1.7"
    Trajectory,          # the whole run
    StepObject,          # one step
    ToolCallSchema,      # one tool call
    ObservationSchema,   # what came back
    MetricsSchema,       # tokens, cost, logprobs
    SubagentTrajectoryRef,
    atif_scope,          # context manager that captures a run
)

The shape of a Trajectory is worth reading closely, because everything Project 03 does is downstream of it (src/nooa/atif/schema.py:330):

class Trajectory(BaseModel):
    schema_version: Literal["ATIF-v1.7"]
    session_id: str | None
    trajectory_id: str | None
    agent: AgentSchema              # name, version, model_name, tool_definitions
    steps: list[StepObject]         # the run, in order
    notes: str | None
    final_metrics: FinalMetricsSchema | None
    continued_trajectory_ref: str | None
    extra: dict[str, Any] | None

and a step (schema.py:240) carries step_id, timestamp, source (system / user / agent), model_name, the full message, reasoning_content, tool_calls, the observation that came back, per-step metrics, and llm_call_count. Observations can point at subagent_trajectory_ref, so a run that fans out to sub-agents nests their trajectories inside the parent's.

Capturing it is three lines, and the capture is scoped to a block rather than installed globally (src/nooa/atif/install.py):

async with atif_scope(agent, path=Path("run.json")) as exporter:
    await agent.run("compute 2+2")
    trajectory = exporter.get_trajectory()   # a validated Trajectory

That is the feature Project 03 showcases. Not "NOOA has observability" — everything has observability. The claim is narrower and stronger: the run is available in-process, as a validated, versioned, self-describing data structure, with no exporter to configure, no collector to run, and no network hop. Which means you can write a pure function that takes a run and returns a comparable value, and put that function in a test.

3. Why the same test is awkward in other agent frameworks

The comparison worth making is structural, not a feature-checklist. Agent frameworks broadly land in one of three shapes, and each shape makes a different thing hard.

Shape	Where the run lives	What blocks a golden test
Graph / DAG frameworks nodes + edges + shared state	State snapshots per node, plus whatever the nodes chose to log.	The graph is declared, so you can assert on which nodes ran — but the prompt each node actually sent is assembled inside the node, and is not part of the recorded state unless you put it there yourself.
Callback / observer frameworks handlers fire on events	An event stream you subscribe to, usually flushed to a hosted tracing backend.	Reassembling a run from an event stream is your job, the schema is whatever the handler emitted, and the natural sink is a service — so the artifact you would diff lives outside your repo and outside your pull request.
Conversational multi-agent frameworks agents send each other messages	A message log.	Messages are the visible layer; the tool dispatch, retries and token accounting underneath are framework internals, so the log under-describes the run.
NOOA the agent is a Python class	Trajectory, a versioned Pydantic model, returned in-process by atif_scope.	Nothing structural. What is left is the mechanical work Project 03 does: scrub the volatile fields and decide what counts as a behaviour change.

There is a second, subtler difference, and it is the one that changed how this project was built. Look at what actually landed in the committed golden file — this is a real excerpt from tests/golden/test_codeact_single_turn_run.json, step 1, the system prompt:

<execution_context>
These names are already in scope inside `execute_python()` ...

```python
import asyncio
import json
from nooa import Agent, CodeActStrategy, LLMResponse, PredictStrategy, strategy
from nooa.config import CodeActConfig
from nooa.unifiedllm import FakeLLMClient, ToolCall

class CodeActAgent: ...
class FanOutAgent: ...

async def classify(item: str) -> str:
    """Classify {item} into a category."""
def codeact_script(turns: int = 3) -> list[LLMResponse]:
    """``turns`` CodeAct iterations: compute, compute, ..., return."""
...
```
</execution_context>

<self expr="doc(type(self))">
class CodeActAgent:
    """Minimal CodeAct agent."""

    async def run(self, prompt: str) -> int:
        """Solve {prompt}."""
</self>

NOOA built that prompt by introspecting the Python module the agent was defined in. The imports, the sibling functions, their signatures and docstrings, the agent's own class definition rendered through doc(type(self)) — all of it is derived from source, not written by hand.

That property cuts both ways, and Project 03 has to respect it in two places: the fixture agents live in their own module with a deliberately frozen namespace (§7), and object repr addresses that leak into the prompt have to be masked (§5).

4. The design: five pure functions and a frozen model

Two modules, 609 lines of source. trajectory.py turns a run into something comparable; recording.py makes the model itself reproducible.

File	Lines	Job
src/nooa_bench/trajectory.py	419	capture, normalise, shape, diff, golden-file check
src/nooa_bench/recording.py	190	record model replies once, replay them offline forever
tests/conftest.py	135	the golden_trajectory fixture and --golden-update
tests/golden_agents.py	101	fixture agents, in a namespace that must not move
tests/test_trajectory.py	314	26 unit tests for the normaliser and differ
tests/test_recording.py	126	7 record/replay round-trip tests
tests/test_golden.py	34	3 tests — the fixture used as intended
tests/test_live_ollama.py	185	4 live-model tests, deselected by default
tests/golden/*.json	373	3 committed golden trajectories

The plan document reports the state after building: 53 frozen tests pass, 4 live tests pass against a real Ollama.

The central design decision: shape versus content

The normalised trajectory is the artifact — the whole run, committed to git, for a human to read during review. The Shape is the assertion surface — nine fields that answer "what did the agent do". Asserting on the full dict would make every whitespace change in a model reply a red build. Asserting only on the shape would let a changed computed value slide. So the harness does both: it compares shapes, and folds a hash of the full content in as a tenth field that can be switched off when running against a live model where message text is sampling noise.

5. trajectory.py, line by line

packages/nooa-bench/src/nooa_bench/trajectory.py — 419 lines

Header and the volatile-field list

L1–2 SPDX headers, matching the rest of the repo. L3–27 Module docstring, which states the contract and — importantly — how the volatile-field list was derived: "by running identical agents twice and diffing the output, not by reading the schema — the schema says which fields exist, not which ones are stable." L29–41 Imports. Only one is from NOOA: from nooa.atif import Trajectory, atif_scope. Everything else is stdlib — difflib, hashlib, json, re, tempfile, dataclasses, pathlib. No new dependency was added for any of this. L43–55 An explicit __all__.

VOLATILE_FIELDS: tuple[str, ...] = (
    "session_id",
    "trajectory_id",
    "timestamp",
    "generation_id",
    "cost_usd",
    "total_cost_usd",
)

L58–74 Six fields, each one observed in a real diff between two byte-identical runs: session_id and trajectory_id are fresh per run (timestamp+uuid, uuid4); timestamp is wall-clock and appears on every step; generation_id is a uuid4 per generation living under step.extra; the two cost fields move with live pricing.

The comment above the tuple records a deliberate omission that is easy to get wrong: schema_version is not scrubbed, because an ATIF version bump must invalidate goldens loudly rather than silently reshaping the diff. L76 defines _SUBAGENTS = "subagent_trajectories" once, so the recursion and the scrubber cannot disagree about the key name. L79–80 GoldenMismatch subclasses AssertionError, so pytest formats it as a failed assertion rather than an error.

capture — run the agent, keep the trajectory, leave no files

async def capture(agent: Any, awaitable: Awaitable[Any]) -> Trajectory:
    with tempfile.TemporaryDirectory(prefix="nooa-capture-") as tmp:
        async with atif_scope(agent, path=Path(tmp) / "trajectory.json") as exporter:
            await awaitable
            return exporter.get_trajectory()

L88–111 Four lines of body carrying three decisions.

The temp directory (L108)

atif_scope always writes a file. Left at its default that file lands in ./logs/atif/ relative to wherever pytest was started, so a suite would quietly accumulate one per run in the working tree. Pointing the scope at a TemporaryDirectory and reading the in-memory object instead means capturing leaves nothing behind — pinned by test_capture_leaves_nothing_in_the_working_tree.

Taking the awaitable, not the agent method (L88)

The caller writes capture(agent, agent.run("compute 2+2")). Passing the already-constructed coroutine keeps capture agnostic about the method's signature — no *args forwarding, no keyword-argument plumbing, and the call site reads like the call it is wrapping.

Crashes still produce a trajectory (L97–99)

atif_scope records extra.crashed before re-raising, so a run that blows up still yields a trajectory — but the exception propagates out of capture. If a crash is the expected outcome, the caller catches it. crashed is one of the nine Shape fields, so "the agent started failing" is itself a diffable behaviour change.

Not re-validating ATIF (L101–106)

get_trajectory() constructs a Pydantic Trajectory, so its validators have already run. The ATIF normative rules — step-id sequencing, joinability — are the exporter's contract and are covered by the framework's own suite. Re-asserting them here would couple this package to tests/atif, and a violation would show up in the golden diff anyway.

normalize — make two identical runs identical

def normalize(traj: Trajectory | dict, *, sort_subagents: bool = True) -> dict:
    d = traj.model_dump(mode="json", exclude_none=True) if isinstance(traj, Trajectory) else traj
    return _normalize_one(json.loads(json.dumps(d)), sort_subagents)

L119–135 Accepts either a Trajectory or a plain dict, so unit tests can hand-build trajectories without constructing Pydantic models. mode="json" gives JSON-native types (datetimes become strings), exclude_none=True drops unset optionals so the golden file carries only what was actually populated. The json.loads(json.dumps(d)) round-trip on L135 is a deep copy that also guarantees everything downstream is a plain JSON type — the mutating scrubber never touches the caller's object.

def _normalize_one(d: dict, sort_subagents: bool) -> dict:
    _scrub(d)
    _renumber_call_ids(d)
    subs = d.get(_SUBAGENTS)
    if subs:
        normalised = [_normalize_one(s, sort_subagents) for s in subs]
        if sort_subagents:
            normalised.sort(key=lambda s: json.dumps(s, sort_keys=True))
        d[_SUBAGENTS] = normalised
    return d

L138–147 Scrub, renumber, then recurse into embedded sub-agent trajectories. The sort on L145 is the interesting line: it orders sub-agents by content rather than by arrival. A fan-out through asyncio.gather completes in nondeterministic order, so without this a five-way fan-out produces a different golden on every run. The docstring names the ceiling honestly — it also hides a meaningful reordering — and the parameter exists so a pipeline where sequence carries information can turn it off.

def _scrub(node: Any) -> None:
    if isinstance(node, dict):
        for key in VOLATILE_FIELDS:
            node.pop(key, None)
        for key, value in list(node.items()):
            if key == _SUBAGENTS:
                continue                      # handled by the caller's recursion
            if isinstance(value, str):
                node[key] = _mask(value)
            else:
                _scrub(value)
                if key == "extra" and value == {}:
                    del node[key]
    elif isinstance(node, list):
        for i, item in enumerate(node):
            if isinstance(item, str):
                node[i] = _mask(item)
            else:
                _scrub(item)

L150–170 A recursive walk with four jobs.

• L154–155 Delete every volatile key at every depth. generation_id lives under step.extra, not at the top level, so a flat pass would miss it.
• L157–158 Stop at the sub-agent boundary. _normalize_one owns that recursion because it also has to sort; letting _scrub descend would scrub them twice and break the sort key.
• L159–160, L167–168 Mask every string, in dicts and in lists alike.
• L162–163 After scrubbing a nested value, if it was an extra map that is now empty, delete the key. Otherwise every step in the golden file carries a decorative "extra": {}. The ordering matters: the emptiness check happens after the recursive call that emptied it.

_ADDRESS = re.compile(r"0x[0-9a-fA-F]{6,16}")

def _mask(text: str) -> str:
    return _ADDRESS.sub("0xADDR", text)

L173–182 This exists because of the NOOA property from §3. NOOA renders the execution context into the system prompt, so anything in scope without a __repr__ arrives at the model as <function f at 0x7f3c...> — and that address changes every run. The comment marks the shortcut and its cost: a blunt regex over every string would also mask a genuine hex literal in a message, which is harmless, because it masks it identically on both sides of the comparison.

def _renumber_call_ids(d: dict) -> None:
    mapping: dict[str, str] = {}

    def rename(raw: str) -> str:
        if raw not in mapping:
            mapping[raw] = f"call_{len(mapping) + 1}"
        return mapping[raw]

    for step in d.get("steps", []):
        for call in step.get("tool_calls") or []:
            if "tool_call_id" in call:
                call["tool_call_id"] = rename(call["tool_call_id"])
        observation = step.get("observation") or {}
        for result in observation.get("results", []):
            if result.get("source_call_id") is not None:
                result["source_call_id"] = rename(result["source_call_id"])

L185–206 Real providers emit ids like call_3hfgfv2c. Deleting them would be simpler and wrong: the join between tool_calls[i] and observation.results[j].source_call_id is behaviour — it says which result answered which call. So the ids are renumbered in first-encounter order through a shared mapping, which preserves the join while making it stable. Two tests pin the two halves: test_tool_call_ids_are_renumbered_but_the_join_survives and test_provider_id_churn_alone_is_not_a_regression.

L209–211 dumps is the single serialisation point: indent=2, sort_keys=True, trailing newline. Sorted keys make the git diff meaningful; the trailing newline keeps POSIX tools and git happy.

Shape and shape — what the agent did, minus what it said

@dataclass(frozen=True)
class Shape:
    step_count: int
    sources: tuple[str, ...]
    llm_calls: int
    functions: tuple[str, ...]
    subagents: tuple[str, ...]
    crashed: bool
    prompt_tokens: int
    completion_tokens: int
    content_hash: str

L219–231 Nine fields, frozen, all hashable — tuples rather than lists precisely so the dataclass stays immutable and comparable. Each answers a different regression question:

Field	Catches
step_count	the agent taking more or fewer turns
sources	the system/user/agent sequence changing shape
llm_calls	extra model round-trips hidden inside the same step count
functions	a different tool being called, or in a different order
subagents	fan-out appearing, disappearing, or changing width
crashed	the run starting or stopping to fail
prompt_tokens / completion_tokens	cost regressions
content_hash	everything the eight structural fields cannot see

L244–270 shape() walks the steps once, summing llm_call_count and collecting every function_name. Token counts come from final_metrics when the exporter populated it, and fall back to summing per-step metrics when it did not (L254–258, helper at L273–279) — both paths have a test. content_hash is sha256(dumps(normalised)) truncated to 12 hex characters (L269): long enough that a collision is not a practical concern, short enough to read in a failure message.

L282–288 _subagent_names flattens agent names at any depth, so a sub-agent that itself fans out is still counted. L265 sorts them, for the same nondeterministic-completion reason as the sub-agent sort in _normalize_one.

diff — the failure message

def diff(golden: Shape, actual: Shape, *,
         token_tolerance: float = 0.0,
         compare_content: bool = True) -> list[Delta]:
    deltas: list[Delta] = []
    for f in fields(Shape):
        if f.name == _CONTENT_FIELD and not compare_content:
            continue
        want, got = getattr(golden, f.name), getattr(actual, f.name)
        if f.name in _TOKEN_FIELDS and token_tolerance > 0 and _within(want, got, token_tolerance):
            continue
        if want != got:
            deltas.append(Delta(f.name, want, got))
    deltas.sort(key=lambda d: d.field == _CONTENT_FIELD)
    return deltas

L296–305 Delta is a frozen dataclass with a __str__ that renders as step_count: 1 -> 2. That string is the first line a person reads in CI, and test_mismatch_message_names_the_change asserts on it literally.

L308–337 The loop iterates fields(Shape) rather than a hand-written list, so adding a tenth field to Shape automatically adds it to the comparison — one place to change, not two. Two modes are threaded through:

token_tolerance (L331, helper L340–343)

A relative band on the token counts. 0.0 for frozen replay, where exact equality is achievable; something like 0.15 against a live model, where sampling makes exact counts meaningless. _within guards division by zero explicitly: if the golden is 0, the actual must be 0 — a tolerance band around zero is infinitely wide, and test_zero_token_golden_requires_zero pins that.

compare_content (L328)

Right for frozen replay, wrong against a live model where the message text is sampling noise. Turning it off leaves the eight structural fields doing the work.

The sort on L336

One line, and it is a usability decision: it pushes content_hash to the bottom of the list. The hash says only "something changed"; the named fields say what. Whoever reads the CI log reads the first line, so the first line should be the informative one. test_structural_change_is_reported_before_the_hash pins the order.

L346–357 render produces a unified diff of the two normalised trajectories via difflib.unified_diff, labelled golden / actual. This is the detail underneath the summary — the shape says what moved, the unified diff shows the actual text.

check — the golden-file protocol

def check(path, actual, *, update=False, token_tolerance=0.0, compare_content=True) -> None:
    if update:
        path.parent.mkdir(parents=True, exist_ok=True)
        existed = path.exists()
        before = path.read_text(encoding="utf-8") if existed else ""
        after = dumps(actual)
        path.write_text(after, encoding="utf-8")
        if existed and before != after:
            print(f"\nupdated golden: {path}\n{render(json.loads(before), actual)}")
        return

    if not path.exists():
        raise GoldenMismatch(
            f"no golden file at {path}\nrecord it with: pytest --golden-update -k <test name>")

    golden = json.loads(path.read_text(encoding="utf-8"))

    want_version = golden.get("schema_version")
    got_version = actual.get("schema_version")
    if want_version != got_version:
        raise GoldenMismatch(
            f"golden recorded under {want_version}, running {got_version} — "
            f"re-record with: pytest --golden-update")

    deltas = diff(shape(golden), shape(actual),
                  token_tolerance=token_tolerance, compare_content=compare_content)
    if not deltas:
        return

    summary = "\n".join(f"  {d}" for d in deltas)
    raise GoldenMismatch(
        f"agent behaviour changed ({len(deltas)} difference(s)):\n{summary}\n\n"
        f"{render(golden, actual)}\n\n"
        f"if this change is intended: pytest --golden-update")

L365–419 Five behaviours, each one a judgement call:

1. Update mode prints the diff it is about to erase (L386–387). Re-recording is when a real regression is most likely to be waved through; printing the old-vs-new diff at the moment of overwrite puts it in front of the person doing the waving.
2. A missing golden is a failure, never an auto-create (L390–393). If a missing file silently recorded itself, a test that has never asserted anything would sit green in CI forever. The error message carries the exact command to fix it.
3. A schema-version mismatch is its own error (L397–403). This is why schema_version was excluded from VOLATILE_FIELDS. An ATIF v1.7 → v1.8 bump would reshape the whole document and produce a diff so large it reads as noise; catching it separately turns that into one sentence.
4. Every error names the recovery command. The harness is only usable if the failure tells you what to type next.
5. The full unified diff is included in the exception (L417), so a CI log is sufficient to review the change without reproducing it locally.

6. recording.py, line by line

packages/nooa-bench/src/nooa_bench/recording.py — 190 lines

A golden trajectory recorded against a live model is only reproducible if the model's replies are reproducible. Frozen mode wraps the real client for one run, saves the replies, and replays them forever with no network and no API key.

L3–31 The docstring states both sides of that trade in the module itself:

Serialisation

def _to_json(response: LLMResponse) -> dict[str, Any]:
    content = response.content
    return {
        "content": content if isinstance(content, str) else content.model_dump_json(),
        "tool_calls": [
            {"id": c.id, "name": c.name, "arguments": c.arguments} for c in response.tool_calls
        ],
        "finish_reason": response.finish_reason,
        "assistant_message": response.assistant_message,
        "reasoning": response.reasoning,
        "usage": response.usage,
    }

L60–76 The content line handles a real asymmetry: PredictStrategy returns a parsed Pydantic model as content, not a string, so it is serialised with model_dump_json(). On reload it comes back as text and the strategy re-parses it — same trajectory, different Python type. That asymmetry is pinned by a live test (test_structured_output_survives_the_script_round_trip) because a stubbed client never produces a model in the first place.

raw_response is deliberately dropped: it is the provider's own object, is not portable, and nothing downstream of a replay reads it. test_raw_response_is_not_persisted stuffs a bare object() in there — which would not survive json.dumps — and asserts the round-trip still works. L79–90 _from_json is the inverse, reconstructing LLMResponse and its ToolCalls with raw_response=None, and using .get() for reasoning and usage so older scripts still load.

def dump_script(responses, path, *, model: str | None = None) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    payload = {"model": model, "responses": [_to_json(r) for r in responses]}
    path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")

L93–103 The model key looks like metadata and is not — see §10. L106–114 load_script and load_model read the two halves back; load_model uses .get("model") so a script written before that field existed still loads, returning None.

RecordingLLM — the tape recorder

class RecordingLLM:
    def __init__(self, inner: Any) -> None:
        self._inner = inner
        self.responses: list[LLMResponse] = []

    def __getattr__(self, name: str) -> Any:
        return getattr(self._inner, name)

    async def acall(self, *args: Any, **kwargs: Any) -> LLMResponse:
        response = await self._inner.acall(*args, **kwargs)
        self.responses.append(response)
        return response

    def save(self, path: Path) -> None:
        dump_script(self.responses, path, model=getattr(self._inner, "model", None))

L122–143 Twenty lines, and the important one is L133. __getattr__ delegating to the wrapped client means RecordingLLM works with any UnifiedLLM implementation without knowing its shape — it overrides exactly one method, acall, and forwards everything else. No interface to implement, no registry to add to, no subclassing. It is a plain Python decorator around a plain Python object, which is the whole reason it is twenty lines.

L143 getattr(self._inner, "model", None) pulls the model name off the inner client at save time. That single argument is the fix for the bug described in §10.

_StrictFake and replay — the tape player that refuses to improvise

class _StrictFake(FakeLLMClient):
    async def acall(self, *args: Any, **kwargs: Any) -> LLMResponse:
        if not self._response_queue:
            raise ScriptExhausted(
                f"agent requested reply #{self.call_count + 1} but the recorded script "
                f"has only {self.call_count}. The agent is making more model calls than "
                f"when this script was recorded — that is the regression. If it is "
                f"intended, re-record the script.")
        return await super().acall(*args, **kwargs)

L151–170 NOOA's stock FakeLLMClient returns an empty response once its queue runs dry. For a regression harness that is exactly backwards: an agent making more model calls than it did when recorded is precisely the regression being hunted, and a silent empty reply buries it inside a confusing downstream parse error. So the subclass raises instead, with a message that names the count on both sides.

The comment on L161–162 is honest about the cost: reaching into the parent's _response_queue is touching a private attribute, and a rename upstream will fail loudly here — which is the preferred failure, because the alternative is silently restoring the fail-soft behaviour this class exists to remove.

def replay(path: Path) -> FakeLLMClient:
    client = _StrictFake(scripted_responses=load_script(path))
    recorded_model = load_model(path)
    if recorded_model:
        client.model = recorded_model
    return client

L173–190 Four lines. The last two are the bug fix from §10: without them a replay reports itself as fake-model and the replayed trajectory differs from the recording on exactly one line per step. The docstring closes with a rule that ties the two modules together: "model_name is deliberately not scrubbed by the normaliser: noticing a model swap is a headline use of this tool."

7. conftest.py — the pytest surface

packages/nooa-bench/tests/conftest.py — 135 lines

Everything above is a library. This file is the two-line experience a test author gets.

async def test_codeact_three_turn_run(golden_trajectory, codeact_agent):
    agent = codeact_agent(turns=3)
    await golden_trajectory(agent, agent.run("compute 2+2"))

L3–21 Docstring: three decisions recorded

Golden files live in tests/golden/<test name>.json and are committed — the diff in a pull request is the review artifact, which is the point of the whole project. The fixture is a conftest rather than a pytest11 entry point so it needs no edit to a tracked pyproject.toml. And the fixture agents deliberately live in a separate module.

L34 sys.path.insert(0, str(Path(__file__).parent))

The repo runs pytest with --import-mode=importlib, which does not put the test directory on sys.path. This restores plain sibling imports so the tests can share golden_agents. The # noqa: E402 markers on L36–44 are the cost of needing the path set before the imports run.

L54–74 Agent factories, not agents

codeact_agent and fanout_agent are fixtures that return a factory. A test can then ask for codeact_agent(turns=1) versus codeact_agent(turns=3), and — crucially — pass llm= to substitute a recorder or a replayer. That one keyword is what lets the same fixture serve frozen tests, record/replay tests, and live tests.

L88–94 --golden-update

A standard pytest_addoption flag. L112–114 also honours GOLDEN_UPDATE=1, so re-recording works however pytest was invoked — including through a wrapper script or an IDE runner that will not forward custom flags.

L97–99 _slug

re.sub(r"[^A-Za-z0-9_.-]+", "_", name).strip("_") maps a pytest node name to a filename. Parametrised test ids keep their distinguishing part, so test_x[case-a] and test_x[case-b] get separate goldens rather than fighting over one.

L102–135 The golden_trajectory fixture

It derives the golden path from request.node.name, then returns an async callable that does capture → normalize → check. The three tuning knobs — token_tolerance, compare_content, sort_subagents — are forwarded with the same defaults as the library, so a test only names the one it needs to change.

8. golden_agents.py — a module that must not move

packages/nooa-bench/tests/golden_agents.py — 101 lines

L11–15 A second constraint, recorded in the same docstring: the fixture agents are built on CodeActStrategy and PredictStrategy rather than CodeActLiteStrategy / ReflexionStrategy, because those two are gated behind a FutureWarning and re-exported lazily through nooa.experimental — a golden recorded against them would sit on an unstable API.

L29 _DEFAULT_LLM = FakeLLMClient()

A class-level default so the agent classes are constructible without arguments; every test overrides it.

L32–56 response, exec_call, return_call

Three tiny constructors for scripted replies and tool calls. response sets finish_reason to "tool_calls" when tool calls are present and "stop" otherwise — matching what a real provider does — and defaults usage to {"prompt_tokens": 50, "completion_tokens": 10} so the token fields in Shape have something deterministic to compare.

L59–65 codeact_script(turns)

Builds n CodeAct iterations: turns - 1 replies that call execute_python, then one that calls return_result(4). Parametrising the turn count is what makes test_a_real_behavioural_change_is_still_visible possible — two runs that differ only in how much work the agent did.

L68–70 fanout_script(n)

One-liner with a comment worth its space: PredictStrategy parses the reply as structured output, so the scripted content must be JSON, not bare text.

L73–79 CodeActAgent

The whole agent:

class CodeActAgent(Agent, llm=_DEFAULT_LLM):
    """Minimal CodeAct agent."""

    @strategy(CodeActStrategy(config=CodeActConfig(max_iterations=5)))
    async def run(self, prompt: str) -> int:
        """Solve {prompt}."""
        ...

Read what is not there. No prompt template file. No graph. No tool registry. The class docstring becomes the system prompt, the method docstring becomes the task with {prompt} interpolated, the return annotation -> int is the output contract the strategy validates against, and the ... body means "the model writes this". Six lines, and the golden file in §9 is what they produce.

L82–86 classify, a standalone strategy function

Not a method on anything — a module-level async def with an ellipsis body and @strategy(PredictStrategy()). It has no LLM client of its own; standalone strategy functions cascade their client from the calling agent (src/nooa/standalone.py:161).

L88–101 FanOutAgent

The test case that justifies half the normaliser:

async def run_all(self, items: list[str]) -> list[str]:
    """Classify every item concurrently."""
    return await asyncio.gather(*(classify(item) for item in items))

run_all has a real body, so it is plain Python — it is not generated. Each classify call is its own generation and lands as an embedded sub-agent trajectory. asyncio.gather means they complete out of order, which is the case the content-sort in _normalize_one exists for.

9. The tests, and what each one is actually protecting

test_golden.py — 3 tests, the fixture used as intended

Three tests, thirty-four lines, each producing one committed golden: test_codeact_three_turn_run (two executions then a return), test_codeact_single_turn_run (the same agent doing less work — a distinct, separately-tracked shape), and test_fanout_over_five_items (five concurrent generations under one deterministic parent method). This is what a user of the harness writes.

test_trajectory.py — 26 tests for the machinery

L32–51 Two helpers, _traj and _step, hand-build trajectory dicts. Hand-building rather than capturing keeps each test isolated to one behaviour and makes them instant. The 26 tests group into five concerns:

Normalisation is sufficient

• test_volatile_fields_are_scrubbed — including the assertion that a step.extra holding only a generation_id disappears entirely rather than becoming {}.
• test_only_volatile_differences_produce_no_deltas — labelled in the source as "the central claim": two runs with different clocks and uuids, same behaviour, zero deltas.
• test_object_addresses_in_prompts_are_masked, test_subagents_are_sorted_by_content, test_subagent_sorting_can_be_turned_off, test_provider_id_churn_alone_is_not_a_regression.

Normalisation is not too aggressive

• test_one_extra_step_is_reported, test_tool_call_ids_are_renumbered_but_the_join_survives, test_content_change_with_identical_structure_is_caught — same steps, different computed value; the structural fields cannot see it, so the hash has to, "otherwise a stale golden sits green".
• test_crash_is_part_of_the_shape, test_a_real_behavioural_change_is_still_visible.

Shape and tolerance

• test_shape_reads_tokens_from_final_metrics and test_shape_falls_back_to_summing_step_metrics — both token paths.
• test_token_tolerance_bands_live_noise, test_zero_token_golden_requires_zero, test_content_comparison_is_off_against_a_live_model.

The failure message is usable

• test_structural_change_is_reported_before_the_hash — the comment says why: "the first line of the failure message is the one that gets read".
• test_render_produces_a_readable_diff, test_mismatch_message_names_the_change (asserts the literal string step_count: 1 -> 2), test_missing_golden_fails_rather_than_auto_creating, test_schema_bump_asks_for_a_re_record, test_update_writes_the_golden, test_dumps_is_stable_and_newline_terminated.

Stability against real agent runs

• test_identical_codeact_runs_normalise_identically — two real captures, byte-identical after normalisation. The docstring records the measurement that motivated the whole normaliser: before normalisation these differ on 20 of 135 lines.
• test_capture_leaves_nothing_in_the_working_tree — monkeypatch.chdir(tmp_path), run an agent, assert the directory is still empty.
• test_fanout_ordering_does_not_leak_into_the_golden — two fan-out runs, identical output, and shape(first).subagents == ("classify",) * 5.

test_recording.py — 7 round-trip tests

test_recorded_script_reproduces_the_trajectory is the core claim: record once, replay, get a byte-identical normalised trajectory. test_replay_survives_repetition runs twenty replays and asserts len(seen) == 1. The rest pin the serialisation edges — JSON round-trip fidelity, the recorded model name, the fallback when no model was recorded, and the dropped raw_response.

test_extra_model_call_raises_instead_of_padding is worth reading for its docstring alone: CodeActStrategy treats any client exception as a transient API error, retries three times and re-raises as GenerationError. So the test accepts either exception type but insists the phrase "more model calls" survives the wrapping — because that phrase is what the person reading the CI log needs.

test_live_ollama.py — 4 tests against a real model

L41 pytestmark = [pytest.mark.integration, pytest.mark.provider_compat_ollama], and the repo's addopts carries -m 'not integration and not stress', so these are deselected by default. L47–58 adds a skipif that pings {API_BASE}/api/tags with a three-second timeout — an unreachable Ollama skips rather than fails. Ollama needs no API key, and qwen2.5:1.5b is about 1 GB, so this is reproducible on a laptop.

L61–77 An autouse fixture sets litellm.drop_params = True and restores it afterwards. The docstring explains: NOOA sends parallel_tool_calls, ollama_chat rejects it outright, and without the flag any tool-using strategy fails against Ollama with UnsupportedParamsError after three retries. It is set in the test rather than in the library because it is global mutable state on litellm and belongs to the caller.

The four tests cover what a stub cannot: test_recording_replays_identically (the model_name bug's regression test), test_two_live_runs_agree_structurally (live settings: token_tolerance=0.15, compare_content=False), test_real_tool_call_ids_are_renumbered, and test_structured_output_survives_the_script_round_trip.

L143–152 contains the most disciplined thing in the file. A 1.5-billion-parameter model fails 17 * 23 + 5 roughly one run in three — it hands back something that will not coerce to int. The test pytest.skips with the model's own error rather than retrying, and the comment says why: "That is the model being weak, not the harness being broken, and a live test that cannot tell the two apart is worse than no live test. Skipping reports the frequency; a rerun would hide it." L157–159 is the same instinct — it asserts the provider's raw ids do not already look canonical, because if they did, the renumbering test would be proving nothing.

10. The artifact: what actually gets committed

Three golden files, 373 lines total. Here is the smallest one nearly in full — test_codeact_single_turn_run.json, with the long system prompt elided:

{
  "agent": {
    "name": "CodeActAgent",
    "version": "0.0.0"
  },
  "final_metrics": {
    "total_cached_tokens": 0,
    "total_completion_tokens": 10,
    "total_prompt_tokens": 50,
    "total_steps": 3
  },
  "schema_version": "ATIF-v1.7",
  "steps": [
    { "message": "<system_prompt expr=\"self._resolve_system_prompt()\">\nMinimal CodeAct
                  agent.\n</system_prompt>\n\n<strategy_prompt> ... </strategy_prompt>
                  \n\n<execution_context> ... </execution_context>\n\n<self
                  expr=\"doc(type(self))\"> ... </self>",
      "source": "system",
      "step_id": 1 },

    { "message": "## Task: run\n\nSolve compute 2+2.\n\nYou are executing `run` — code runs in
                  the Execution Context above. Calling `self.run(...)` would recurse.",
      "source": "user",
      "step_id": 2 },

    { "extra": {
        "dynamic_context": "<context>\n<state expr=\"pformat(self, max_length=50,
                            max_string=500, max_depth=4)\">\nCodeActAgent()\n</state>\n</context>"
      },
      "llm_call_count": 1,
      "message": "Done.",
      "metrics": { "cached_tokens": 0, "completion_tokens": 10, "prompt_tokens": 50 },
      "model_name": "fake-model",
      "observation": {
        "results": [ { "content": "Result accepted.", "source_call_id": "call_1" } ]
      },
      "source": "agent",
      "step_id": 3,
      "tool_calls": [
        { "arguments": { "result": 4 },
          "function_name": "return_result",
          "tool_call_id": "call_1" }
      ] }
  ]
}

Note what survived normalisation and what did not. No session_id, no trajectory_id, no timestamp on any step, no generation_id inside extra — but the dynamic_context that was in extra is still there, because it is behaviour. model_name is present and unscrubbed. The tool-call id is call_1 and the observation's source_call_id is call_1: the join survived the renumbering.

11. The two bugs only a live model could find

Both are recorded in the plan document as post-build corrections; both are worth repeating here because they are the kind of thing a stubbed test suite is structurally incapable of catching.

The fake-model bug

NOOA copies the client's model into every agent step as model_name. FakeLLMClient calls itself fake-model. So a script recorded against a real provider and replayed through the fake produced a trajectory that differed from the recording on exactly one line per step — enough to fail its own golden.

The fix is the model= argument in dump_script plus the three lines in replay that carry it forward. The point is the detection: this bug is invisible whenever the recording client is itself a fake, which is what every offline test in the package uses. Only recording against Ollama surfaced it. test_script_carries_the_recorded_model_name now pins it without needing a provider — by setting inner.model on a fake by hand.

The parallel_tool_calls bug

"Ollama works with no API key" turned out to hold for plain generation but not for CodeAct: NOOA sends parallel_tool_calls, ollama_chat rejects the parameter outright, and every tool-using strategy died with UnsupportedParamsError after three retries. The escape hatch is litellm.drop_params = True, scoped to the test module by an autouse fixture rather than set in the library.

12. Why this matters

It moves agent behaviour into code review

The normal way to notice that an agent's behaviour changed is to watch a dashboard, or to get a bill. The golden file makes it a line in a pull request. A reviewer who has never run the agent can see that a prompt refactor added two steps and 400 prompt tokens, in the same diff view as the refactor itself — because the committed artifact and the code that produced it are in the same repository, in the same commit.

It gives "behaviour" a definition you can argue with

Shape's nine fields are a written-down answer to "what counts as the agent behaving differently". You can disagree with it — maybe token counts should not be in there for your project, maybe sub-agent order is meaningful — and the disagreement is a two-line change to a frozen dataclass rather than a philosophical debate. Most agent stacks never make that definition explicit at all.

It makes CI on agents possible at all

Frozen mode is the enabling piece. Golden tests that need a live model are expensive, flaky, and need secrets, so in practice they run rarely or never. Record once, replay forever: 53 tests that need no network, no API key and no GPU, running on every push, exercising the real strategy, the real context assembly, the real REPL and the real tracing path. Only the model is a recording.

It is 609 lines with no new dependencies

The whole thing is stdlib plus one import from NOOA. That is not frugality for its own sake — it is the measurable consequence of the framework design. When the run is already a validated Pydantic model available in-process, the tooling on top of it is small. Compare the alternative: stand up a collector, define your own span schema, teach every node to emit it, reassemble runs from spans, and then start writing the normaliser.

13. What this cannot do

Stated plainly, because a regression harness that oversells itself is worse than none.

• Frozen mode cannot detect prompt-quality regressions. Edit a prompt and the recorded reply comes back unchanged — the model is a tape, it does not react. Catching "this prompt is now worse" needs a live run and a quality metric, which is a different project.
• Sub-agent sorting hides meaningful reordering. Content-sorting fan-out results is correct for asyncio.gather, where order is not meaningful by construction. For a pipeline where sequence carries information, pass sort_subagents=False — and note that the default is the unsafe choice for that case.
• The address regex is blunt. 0x[0-9a-fA-F]{6,16} masks genuine hex literals in message text too. Harmless for comparison, since both sides are masked identically, but the golden file will show 0xADDR where a real value was.
• A golden is only as good as its re-record discipline. --golden-update makes it trivial to accept a real regression. Printing the overwritten diff mitigates this; it does not solve it. The mitigation that would solve it is reviewer attention on the golden diff.
• Nothing here is wired into CI yet. The plan's phase 6 is unbuilt; the tests exist and pass, but no workflow file in this repository runs them on push. Adding one is a few lines and has not been done.
• The live tests need a local Ollama and skip otherwise. They are deselected by default, so a clean run reports 53 passed, 4 deselected — which is not the same as 57 passing.

14. Run it

# frozen — no network, no API key: 53 passed, 4 deselected
uv run --frozen pytest packages/nooa-bench/tests

# re-record the goldens after an intended behaviour change
uv run --frozen pytest packages/nooa-bench/tests --golden-update

# live, against a local Ollama (ollama pull qwen2.5:1.5b, ~1 GB)
OLLAMA_API_BASE=http://localhost:11434 \
  uv run --frozen pytest packages/nooa-bench/tests/test_live_ollama.py -m integration

Then open packages/nooa-bench/tests/golden/test_codeact_three_turn_run.json, change max_iterations in golden_agents.py, or add a helper function to that module, and re-run. The failure message names what moved.