Jev projects
Every page in this repo puts the measured latency on screen. The source videos show demos from machines close to the model. From this machine one Jev decision takes 300 to 1,200 ms. A demo that hides that number is a demo of the animation, not of Jev.
What was built: one page per project
Nine pieces, each one Node file with the questions at the top and, for the six visual ones, one HTML page that polls /state and draws. The pages need the Node server and a gateway key, so this site holds the results, not the live canvases.
| Project | What Jev decides | Loop | Check result |
|---|---|---|---|
probe-burst.mjs | Nothing; 50 boolean calls fired at once | one burst | 48/50 ok, burst 7.8 s, per call min 1,079, median 1,230, max 7,739 ms (the max is the SDK's three retries on a 503) |
wiki/ | Which of up to 255 links on the current Wikipedia page leads to the target | turn-based | 5/5 races Coffee → Napoleon reached it: 4, 7, 5, 7, 4 hops in 4.6, 3.3, 4.8, 5.7, 2.3 s, $0.0004 to $0.0006 each |
town/ | What each of 50 characters does when an event is announced: ignore, investigate, join, flee, warn, plus an urgency score | one burst per event | 50/50 answered, burst 2.7 to 10.0 s, median per call 376 to 643 ms, $0.0011 per event |
chess/ | Which legal move to play (SAN), on a one-minute clock | turn-based | 10 games vs a random mover: 5 checkmates, 5 draws, 0 losses; 572 Jev moves at 465 ms mean; $0.013 |
sort/ | Which of five queues a customer message goes to | 20 calls in flight | 303 labelled messages in 13.8 s, 21.9 items/s, 93.7% agreement with the labels, $0.006 |
logs/ | Severity 0 to 3, page-the-on-call boolean, subsystem, per log line with three lines of context | streamed, 10 in flight | 2,020 lines in 119 s (second run: 573 s, see below), $0.05. Routine max 1.37, incident min 1.47: no overlap. Page boolean fired on 18/20 incidents and 3 routine lines, both runs |
lane/ | Forward, ease left, ease right, brake, or stop, once per tick | ticked, 1,000 ms | 5 runs of 60 ticks: Jev 13 collisions, forward-only 20, rule-based 0. Spec target (fewer than 3) not met |
hooks/skill-router.mjs | Which of 147 Claude Code skills fits the prompt, or none | one call per prompt | 6/6 test prompts matched; 9,920 input tokens, 408 to 6,926 ms, $0.0004 per prompt. A first run saw only 100 skills: 48 skill folders are links and readdir dirents report them as non-directories; fixed with stat |
hooks/verify.mjs | Twelve questions about a git diff in one call | one call per diff | On a synthetic diff: secret 0.99, test_weakened 0.99, debug_left 0.98, unrelated 0.88; risk 2.99 of 3; 340 ms, $0.00003 |
Findings
The gateway and the SDK
- 50 parallel calls mostly work. 48 of 50 returned; two got 503 three times in a row and failed after the SDK's own retries. The median per call in a burst (1,230 ms) is above the sequential range (300 to 700 ms). Sequential runs also see short 503 bursts: the second chess check game died on one.
lib.mjsnow retries four more times at 3, 6, 12, and 24 s. - The SDK rejects a rounding tie. Jev returns probabilities rounded to two decimals. When the chosen option shows 0.13 and another shows 0.14,
ai7.0.107 throwsAI_InvalidResponseDataError: Question "move" did not select a highest-probability option. Chess, with 20 to 40 near-equal options, hits it often.ask()reads Jev's answer from the error'sdatafield and drops the metadata for that call. - 255 options, no more. A 256-option choice is refused with "TypeSafe Choice questions support at most 255 options". Empty-string descriptions are accepted. With 255 fake titles plus "Napoleon Bonaparte" and the target "Napoleon", Jev chose the right one at p 0.97 in 337 ms.
- Cost lives at
providerMetadata.gateway.marketCost, a string in dollars. One boolean call: $0.0000116. One 255-option choice: $0.000177.
What the loops showed
- Batch-and-wait halves throughput. Sorting in batches of 20 ran at 3.5 items/s: every batch waited for its slowest call, and the tail is 5 to 7 s. Twenty workers pulling from one queue ran at 21.9 items/s on the same items.
- Investigate is the town's default. Both "Free bread at the bakery" and "A wolf is at the gate" got investigate as the modal action (36 and 29 of 50). The spread is in the tails: the wolf got 18 warn and 3 flee, the bread 7 join and 3 warn. The check asserts the tails, since the modal test from the spec cannot pass.
- The severity scale is compressed. Only 12 of 20 injected incidents scored above the spec's 2.0 line on the 0 to 3 scale. But no routine line scored above 1.37 and no incident below 1.47, so the sets separate completely at 1.4. The page-the-on-call boolean is the sharper signal: 18 of 20 incidents above 0.5, 3 of 2,000 routine lines above it.
- Jev drives badly from text. Without arithmetic help it collided 29 times in 5 runs of the fixed 60-tick script (17 obstacles). Adding a precomputed
ticks_to_impactper obstacle cut that to 13 to 16. A driver that only goes forward: 20. A rule-based driver with the same information: 0. The check now asserts that Jev beats the forward-only driver. - The gateway has slow hours. The same 2,020-line check took 119 s at about 10:00 local time and 573 s at about 10:30, with 0 failed calls both times and the same separation (routine max 1.38, incident min 1.49). One lane run in that window averaged 8,567 ms per decision against 254 to 568 ms in the others. The 503 retries at 3, 6, 12, and 24 s in
lib.mjskept every answer and paid in wall time. A page that shows latency per call shows this; a page that shows only the animation would not. - Sort accuracy is a rubric question. 88.1% → 87.5% → 93.7% across three rubric versions with Jev unchanged. The remaining misses sit between sales, billing, and other, where the labels (written once by
anthropic/claude-sonnet-5) are themselves one opinion. - Chess: no illegal moves, no losses to random. The options are the legal moves, so the schema does the legality. Five checkmates and five draws; two draws were the 200-ply cap.
Not built
- Minecraft, Doom, Melee, the trading bot, and the Sentry pipeline: out by the stack decision (Node only) or by missing hardware and accounts.
- The wiki race lane against a chat model. Chess against a chat model is wired (a gateway model id in the page's opponent field, 10 s per move) but was not measured.
- The two hooks are written and checked but not installed into
~/.claude/settings.json; each would add 300 to 1,000 ms to every prompt or edit. The README shows the line to add. - The skill router's 30-prompt evaluation against real transcripts. Six hand-written prompts stand in.