Pin eval renderer to engine renderer with a parity test

render_known_world (the exam's prompt builder) promised to match
KnownWorld.render() (production's) by comment only - format drift would
silently grade the reviewer against a stale prompt shape. Now every
corpus setup renders through both and must come out byte-identical
(plus a fixed example incl. the empty-world collapse, and a corpus
format pin: remotes must be 'name url' since the engine renderer has
no name-only representation).
This commit is contained in:
Devika Verma
2026-08-17 04:14:25 +05:30
parent a860f3b2f9
commit d66dc9b471
2 changed files with 45 additions and 1 deletions
+3 -1
View File
@@ -85,7 +85,9 @@ def load_corpus(name: str) -> list[Row]:
def render_known_world(setup: dict[str, Any]) -> str:
"""Reconstruct the reviewer's known-world block from a corpus row's `setup`, matching
KnownWorld.render() — folders and remotes only, never hostnames (spec §2.4)."""
KnownWorld.render() — folders and remotes only, never hostnames (spec §2.4). Parity is
ENFORCED, not just intended: test_shadow_eval.py renders every corpus setup through
both this and the engine's renderer and requires byte-identical output."""
lines = ["KNOWN WORLD (frozen when this session started)"]
for root in setup.get("roots", []):
writable = "read-write" if root.get("writable") else "read-only"