Jev Coding Reflex for Codex

Recorded Codex demonstration

Watch a real persistent Codex session in which Jev blocks an incomplete first completion, Codex adds the missing behavior test, and a second Jev review passes.

Click the screenshot to watch the 4:16 demonstration. The browser-ready recording is 5.0 MB.

The recording is a second controlled run: its first Stop reported behavior_untested 0.83 and task_incomplete 0.78 in 988 ms; after Codex added the direct test, the second Stop was clean in 581 ms. Section 5 retains the earlier measured run as separate evidence.

1. Purpose and boundary

This is not a replacement for tests, static analysis, Git or Codex's reasoning. Version 1 does not edit files, approve dangerous operations or push code. When it finds an issue, the Stop hook continues the Codex turn with concise review feedback; Codex investigates and repairs or rejects the finding.

2. End-to-end workflow

The minimal Codex integration saves the baseline at task start, records recognized test commands and exit results after tool use, and performs the review at task stop. The post-tool recorder is local and does not call Jev. Running Jev after every edit is deliberately deferred because it would add cost, latency and repetitive findings without first proving better outcomes.

Where Jev comes into play

The key point is: Jev participates only when Codex tries to finish. It does not write code or run tests. It acts as a fast external reviewer inside the Stop hook.

Here is exactly how your five steps divide between Codex, Reflex and Jev.

1. Reflex collects evidence

When you submit the task, the UserPromptSubmit hook records:

• your original request;
• the current Git commit;
• the initial repository status.

Whenever Codex runs a recognized test command, PostToolUse records the command and its output.

When Codex tries to finish, the Stop hook runs:

node --env-file=.env hooks/coding-reflex.mjs --hook

The disabled example wiring is in .codex/hooks.example.json. Copy it to .codex/hooks.json to activate the hook.

2. This is the exact point where Jev enters

The Stop hook calls:

verifyWithQuestions(diff, QUESTIONS, {
  user_request: state.prompt,
  test_evidence: state.tests,
  initial_worktree_status: state.initialStatus
});

That happens in hooks/coding-reflex.mjs.

verifyWithQuestions then calls the shared ask() function. Inside ask(), the selected model is explicitly:

export const JEV = "typesafe-ai/jev";

The actual remote model invocation is:

evaluate({
  model: JEV,
  state,
  questions
});

See lib.mjs.

Therefore, Jev receives structured evidence resembling:

{
  user_request: "Create a trailing-whitespace utility...",
  git_diff: "all files Codex added or changed",
  test_evidence: [
    {
      command: "node --test reflex-demo/test.mjs",
      response: "3 tests passed"
    }
  ],
  initial_worktree_status: "clean"
}

3. Jev makes decisions

Jev is asked eight binary questions, including:

behavior_untested:
Does the diff add or change executable behavior without adding or
updating a direct test for that behavior?

The questions are defined in hooks/coding-reflex-questions.json.

Jev returns probabilities, conceptually:

{
  "task_incomplete": 0.78,
  "behavior_untested": 0.86,
  "test_weakened": 0.03,
  "secret": 0.01,
  "errors_swallowed": 0.04,
  "placeholder": 0.02,
  "injection": 0.01,
  "unrelated": 0.03
}

Any probability above 0.50 becomes a finding.

Jev is therefore the component that decides:

It does not write the explanation freely. It answers predefined quality questions probabilistically and very quickly.

4. Reflex returns Jev’s decision to Codex

If Jev finds a problem, the hook returns:

{
  decision: "block",
  reason: "Jev Coding Reflex found 1 issue...
           behavior_untested (p 0.86)..."
}

That line is the control mechanism. decision: "block" tells the Codex runtime:

This is why Codex resumes automatically.

5. Codex performs the repair

Codex reads Jev’s finding, examines its work, adds the missing test and runs it.

This part is performed by Codex, not Jev.

When Codex tries to finish again, the Stop hook invokes Jev a second time. Jev now sees the updated diff and test evidence. If every probability is below 0.50, Reflex returns only:

Jev Coding Reflex: no actionable findings

Codex is then allowed to finish.

So the ownership is:

You do not see a separate Jev window because Jev is operating as a hidden, low-latency decision model inside the Codex lifecycle. During the video, the visible evidence of Jev is the message:

Jev Coding Reflex found 1 issue...
behavior_untested (p 0.xx)

That is the moment Jev has prevented Codex from finishing and triggered the repair cycle.

3. Inputs and Git-diff collection

4. Worked Codex example: dry-run cleanup

diff --git a/src/cleanup.js b/src/cleanup.js
export async function cleanup(files, options = {}) {
  for (const file of files) {
    if (options.dryRun) {
      console.log(`Would delete: ${file}`);
      continue;
    }
    await fs.unlink(file);
  }
}

USER REQUEST
Add a dry-run option to the cleanup command.
It must report files without deleting them.
Add a test and update the README.

TEST RESULT
18 passed, 0 failed

GIT DIFF
[tracked changes and safe representations of new files]

QUESTIONS
1. Does the change satisfy every explicit requirement?
2. Can dry-run still delete a file through a changed path?
3. Is the dry-run behavior directly tested?
4. Was the README updated accurately?
5. Did the task introduce unrelated changes?

Codex inspects the finding, adds a test that proves the file remains present, reruns the tests and invokes the final check again. If no question crosses the reporting threshold, Codex tells you the task is complete and includes the clean Jev result.

5. Measured inside Codex

A persistent Codex CLI session ran the dry-run example against a disposable Git repository. Codex was deliberately told to omit the requested test on its first completion attempt, creating a known fault for the hook.

This proves the Codex → Jev → Codex continuation loop for one controlled omission. It does not establish general accuracy. The full privacy-safe record is eval/coding-reflex-codex.json.

Runtime finding: on this installed Codex build, lifecycle hooks ran in persistent sessions but were skipped by codex exec --ephemeral. The experiment therefore used a persistent session.

6. Jev questions and output contract

The existing twelve diff questions remain the foundation. Version 1 adds task-completion questions only where the task, diff and test outcome provide useful evidence.

Each reported finding has six fields:

{
  "id": "requested_behavior_untested",
  "title": "Dry-run behavior is not directly tested",
  "severity": "high",
  "confidence": 0.96,
  "evidence": "No changed test calls cleanup with dryRun enabled.",
  "next_step": "Add a test proving the target file remains present."
}

• Return at most three findings, ordered by severity and confidence.
• Require concrete evidence from the supplied task, diff or test result.
• Say “no actionable finding” instead of manufacturing criticism.
• Preserve Jev latency, token count, cost and probability for evaluation.

7. Trust boundary and failure behavior

Binary files, generated assets, lockfiles and oversized files are summarized rather than transmitted in full. The first version is advisory; Jev cannot push, deploy, delete, approve permissions or modify the repository.

8. Evaluation beyond the prototype

The hook should not be enabled merely because one demonstration looks plausible. Build 30–50 labelled task-change pairs with both defects and clean controls.

Those are proposed acceptance gates, not measured results. The current verifier has six held-out cases and one live synthetic demonstration; it does not yet establish these targets.

9. Roadmap from the prototype

1. Extend the existing verifier input: accept task text and test outcome in addition to the diff.
2. Add a thin Codex adapter: record the baseline at task start, capture test-result metadata after tool use and run one advisory Jev check at task stop.
3. Add local collection safeguards: include untracked text files, redact likely secret values and enforce size limits.
4. Create the labelled evaluation set: 30–50 defective and clean cases, separated into development and held-out groups.
5. Install only after the gate passes: enable the repository-local Codex hook in report-only mode.
6. Publish measured evidence: confusion matrix, individual decisions, median/p95 latency, cost and failure rate.

10. Definition of done

• A Codex task can begin and end without the user running Git commands manually.
• The collected evidence includes the instruction, tracked changes, new text files and test outcome.
• Potential credential values are redacted locally before gateway transmission.
• Jev returns zero to three evidence-backed findings and never edits files.
• Codex can repair a valid finding and request one final recheck.
• Unavailable review is clearly different from a clean review.
• The held-out evaluation meets the agreed recall and false-alarm gates.
• The public results page distinguishes measured results from design targets.