Jev model router · reliability upgrade

A router that can prove its own behavior.

The application now separates routing from generation, rejects unsafe inputs, runs without paid models, and records repeatable offline and live evaluation evidence.

Merged to main · 19 September 2026

13/13live held-out cases passed

0live evaluation errors

1.62 smedian live Jev latency

$0.000232total live evaluation cost

What changed

The change turns a working demonstration into a small application with explicit boundaries and reproducible evidence.

Boundary

Validated and capped requests

The server accepts only non-empty string messages, caps the HTTP body at 64 KiB, caps routed text at 20,000 characters, and binds to 127.0.0.1.

Separation

Routing is independently callable

chooseTier() measures the Jev decision. route() invokes the selected reply model and preserves a visible reply-model failure.

Offline

Free deterministic mode

--offline supplies local evaluator and generator functions. The real page and HTTP flow can be tested without a gateway call.

Evidence

One result row per case

The evaluator writes JSONL with input, expected and actual answers, confidence, probabilities, latency, cost, attempts, error, and mode.

Browser

Real user-flow check

Headless Chromium loads the app, enters a message, submits it, and verifies the tier, reply, and absence of console errors.

Taxonomy

Labels match the declared rubric

Trivial facts and conversions are nano; short clear work is fast; reasoning and code are balanced; high-stakes systems are frontier.

Why these changes were needed

Before	Risk	Now
The self-check called live Jev and reply models.	A basic test could spend credit, fail because a reply model was unavailable, or change between runs.	The default check is deterministic and free. Live Jev evaluation is explicit.
Routing and generation lived in one function.	A reply failure obscured whether Jev had routed correctly.	Tier selection is assessed separately; the live evaluator never calls a reply model.
The server accumulated any request body.	A local malformed request could consume unnecessary memory.	The body has a 64 KiB trust-boundary limit and clear client errors.
No held-out result contract existed.	Improvements could be asserted from anecdotes.	Thirteen versioned cases produce machine-readable, comparable evidence.

How it works

The same server supports production calls, deterministic checks, and browser automation through dependency injection.

Receive

Read a bounded JSON body from POST /chat.

Validate

Require a non-empty string and trim it to the model limit.

Classify

Jev returns one known tier plus probabilities and confidence.

Generate

The selected model answers; refusal stays visible as replyError.

Measure

Record routing latency, cost, output, and errors separately.

npm test                    # offline server check + 13 cases
npm run eval:live           # Jev only; saves eval/results-live.jsonl
node server.mjs --offline   # browser-testable local application
python tests/browser.py

What the evidence says

Check	Result	What it demonstrates
Offline evaluator	13/13	The evaluation plumbing, taxonomy, error shape, and output contract are deterministic.
Live Jev evaluator	13/13, 0 errors	Jev matched the reviewed tier rubric across trivial, reasoning, code, system-design, and high-stakes prompts.
Live latency	1,619 ms median; 4,402 ms maximum	Routing quality passed, but remote latency remains material for an interactive application.
Browser flow	Passed	The page, server, form submission, tier display, and reply rendering work together.