Jev projects · measurement and verifier upgrade

Failures now count. Late work has a cost. Full diffs get examined.

The upgrade repairs measurement errors across burst, sort, logs, and lane demonstrations, then gives the diff verifier complete staged and untracked coverage with repeatable held-out evidence.

Merged to main · 19 September 2026

6/6live verifier cases passed

50/50parallel burst calls returned

0/50even/odd decisions wrong

$0.000411live verifier evaluation cost

What changed

Each repair sits at the point where measurement or failure information was previously lost.

Shared evaluator

Invalid responses remain failures

ask() no longer converts any AI_InvalidResponseDataError with attached data into a zero-cost success. Retry attempts and required answers are explicit.

Burst

Parity follows the original input

Each fulfilled result carries its source number. Failure filtering can no longer shift array indices and create false even/odd errors.

Sort

Accuracy cannot hide failed calls

The state counts failures, the page displays them, and the check requires every item to be scored before it accepts the accuracy result.

Logs

Live concurrency is bounded

At most ten scoring calls remain in flight. Additional ticks are counted as backpressure, displayed in the UI, and paging uses one 0.5 threshold.

Lane

Late calls cannot pile up

Only one decision remains pending. A late answer is discarded as an action but its latency, cost, or error is recorded before the next Jev call begins.

Verifier

The full change is examined

Tracked, staged, and untracked text is split into 20,000-character chunks. Boolean maxima and highest risk scores are aggregated across every chunk.

Why these changes were needed

Application	Previous measurement defect	Repair
`probe-burst`	Rejected calls were removed before parity was checked, so the post-filter index could be mistaken for the original number.	Carry `n` inside each fulfilled result and compare against that value.
`sort`	Accuracy used only successful labelled rows. A run with widespread request failures could still pass.	Count failures and require `ok === total` plus complete labelled coverage.
`logs`	The live loop launched work without a cap, and UI paging used 0.8 while the check used 0.5.	Cap at ten in flight, expose backpressure, and use one threshold.
`lane`	Every timeout launched another request while the old one continued. Completed late calls disappeared from cost totals.	Retain one pending promise, repeat the previous action, then account for the settled call.
`verify`	Only the first 20,000 characters and unstaged tracked edits were visible.	Read the full base diff plus untracked files, evaluate all chunks, and aggregate results.

How full-diff verification works

The verifier remains one small script. It uses Git and the Node standard library, then asks the existing Jev question set about each bounded chunk.

Collect

Read tracked and staged changes from the selected base.

Include

Represent non-ignored untracked text as added-file diffs.

Chunk

Split complete input into bounded 20,000-character states.

Evaluate

Ask the same twelve questions about every part.

Aggregate

Keep maximum boolean probabilities and highest risk scores.

npm test
npm run eval:live
node --env-file=.env hooks/verify.mjs --base main
node --env-file=.env probe-burst.mjs 50

# Browser flow: start the server, then drive the rule-based lane
node --env-file=.env lane/lane.mjs --tick 10
python tests/browser.py

What the evidence says

Check	Result	Finding
Offline verifier suite	6/6	Secret, skipped test, dependency, swallowed error, injection, and safe documentation cases obey the expected signals.
Secret after old cutoff	Detected at p 0.98	A secret placed after 21,000 filler characters proves that later chunks are evaluated.
Live verifier suite	6/6, 0 errors	Total cost was $0.000410802; median case latency was 1,810 ms and the slowest was 10,820 ms.
Threshold boundary	0.50 no longer fires	Jev returned exactly 0.50 for an unrelated “errors swallowed” question. Requiring probability above 0.5 removed the false alarm.
Concurrent burst	50/50 returned; 0 wrong	The repaired index accounting reported a 4,206 ms burst, with 3,972 ms median per call.
Lane browser flow	60 ticks, 0 rule-driver collisions	The real page, polling loop, start action, state rendering, and new failure counters work together without console errors.