Devika Verma 08c3984874 Eval: distinguish error-unsure from cautious-unsure; retry, count, never pass on errors ...

A provider outage (Together GLM 5xx, live 2026-08-17) turned rows into
unsure verdicts indistinguishable from genuine caution - a benign gate
read FAIL and a dangerous gate would have read PASS, both meaningless.

Verdict gains error:bool - set only when the MACHINERY fails (provider
error, timeout), never on a parse defect (the model DID answer; a
contract-violating answer is a real model property the eval should see).
The live engine is unaffected: error and caution both mean card+human.

Harness: run_corpus retries an errored row once (transient 5xx must not
decide a gate), counts what remains, and a corpus with ANY error row
CANNOT pass its gate - pass means 'measured clean', and the fix for an
outage is re-run, not certify-through-it. Report gains an Errors column
and lists the errored ids. Tests cover the retry, the error-count, and
the can-never-pass rule.

2026-08-17 21:59:36 +05:30

..

eval_reviewer.py

Eval: distinguish error-unsure from cautious-unsure; retry, count, never pass on errors

2026-08-17 21:59:36 +05:30