Diff verifier: twelve questions about a git diff in one call

Date: 2026-09-19  ·  Code: hooks/verify.mjs  ·  Model: typesafe-ai/jev through the Vercel AI Gateway

1. What Jev decides

For a git diff, eleven yes/no questions (a hardcoded secret, a weakened test, debugging output left in, swallowed errors, a placeholder, a changed signature, a new dependency, commented-out code, unrelated changes, string-built SQL or shell, a changed shared constant) and one risk score on four levels, all in one call. The questions live in questions.json, which is meant to grow: each bug that reaches you becomes one more question. Runs as a command over the working tree, or as a PostToolUse hook on Edit and Write. The hook is not installed by this repo.

2. The questions

questions.json. This is the block a reviewer must read; the rest of the file is plumbing.

{
  "secret": { "type": "boolean", "instructions": "Does the diff add a hardcoded secret, API key, token, password, or private key?" },
  "test_weakened": { "type": "boolean", "instructions": "Does the diff delete, skip, or loosen a test or an assertion?" },
  "debug_left": { "type": "boolean", "instructions": "Does the diff leave debugging output in place (console.log, print, debugger, dump)?" },
  "errors_swallowed": { "type": "boolean", "instructions": "Does the diff catch an error and then ignore it, so a failure would pass silently?" },
  "placeholder": { "type": "boolean", "instructions": "Does the diff add a TODO, stub, or placeholder instead of the real implementation?" },
  "signature_change": { "type": "boolean", "instructions": "Does the diff change the signature or return shape of a function that other code may call?" },
  "new_dependency": { "type": "boolean", "instructions": "Does the diff add a new third-party dependency?" },
  "dead_code": { "type": "boolean", "instructions": "Does the diff add code that is commented out?" },
  "unrelated": { "type": "boolean", "instructions": "Does the diff mix changes that are unrelated to each other?" },
  "injection": { "type": "boolean", "instructions": "Does the diff build a SQL query, shell command, or HTML string from user-supplied input without escaping it?" },
  "shared_constant": { "type": "boolean", "instructions": "Does the diff change a port, path, URL, environment variable name, or config key that other files may reference?" },
  "risk": { "type": "score", "instructions": "How likely is this diff to break something for users if merged as is?", "criteria": ["safe: docs, comments, or clearly local changes", "low: small logic change with an obvious check", "medium: touches shared code or data handling", "high: changes control flow, error handling, security, or persistence without a test"] }
}

3. Run it

node --env-file=.env hooks/verify.mjs                  # diff of the working tree vs HEAD
node --env-file=.env hooks/verify.mjs --base main
echo '{"tool_input":{"file_path":"x.mjs"}}' | node --env-file=.env hooks/verify.mjs --hook
node --env-file=.env hooks/verify.mjs --check

4. The check, verbatim

340 ms, 822 tokens, $0.00003; fired: secret 0.99, test_weakened 0.99, debug_left 0.98, unrelated 0.88; risk 2.99

The synthetic diff behind that line replaces process.env.STRIPE_KEY with a string literal, adds a console.log that prints the key, and turns a test into test.skip. The check asserts that secret and test_weakened fire and that new_dependency does not.

5. Findings

• Four questions fired on the synthetic diff, all four correctly: the secret at 0.99, the skipped test at 0.99, the debug line at 0.98, and "unrelated changes" at 0.88, which is fair for a diff that touches a payment file and a test file for different reasons. The risk score was 2.99 of 3.
• Twelve questions cost 822 input tokens and $0.00003. The assessment estimated $0.00017 for a 2,000-token diff with 100 questions. A question file ten times this size is still under a cent per diff.
• The diff is cut at 20,000 characters. A large PR would need to be split by file; the code says so in a comment.
• Not measured: a real PR from this repo. The check is synthetic because a real diff of this repo has nothing to fire on, which is also the expected result.

6. Cost and latency

$0.00003 per diff at this size. The hook would add 300 to 1,000 ms to every Edit and Write.