The three committee readers of a single brief are blind and independent by
construction, so running them one after another only spent wall clock. Each
read carries the whole 424-entry shelf and takes about fifteen seconds, so
424 briefs at three readers is over five hours sequentially and under two
with the reads overlapped.
The passes are submitted to a ThreadPoolExecutor sized to COMMITTEE_READERS
and read back in submission order, because the majority vote and its
tie-break depend on that order. An exception in any pass still propagates
and stops the run. The loop over briefs stays strictly sequential: the
checkpoint, the running spend, the ceiling and the kill file all depend on
it.
The distractor gate exists to reject a brief that gives its answer away by
standing alone in its own score band. It was also dropping every brief that
shared no vocabulary with its own shelf blurb, which is the opposite case: that
brief gives nothing away and is the hardest one on the shelf.
Measured on the first 128 moves of the 424-move wave: 109 accepted, 19 dropped,
and all 19 drops carried this one reason. The dropped moves' blurbs are shorter
than the accepted ones, a median of 15 tokens against 19, and six of the 19 are
caption-family near twins, so the drop was correlated with exactly the moves and
families this round exists to score fairly.
No-shelf-overlap is now a soft rejection, owned by one predicate: the loop still
prefers an overlapping brief and only keeps the last soft-rejected candidate when
its attempts run out. Whether that brief is on target is the blind gold
committee's call, and an off-target one surfaces there visibly instead of
vanishing at generation. Every other reason still drops the move.
The soft path is never silent: the gold record carries the reason, the attempt
log records accept-soft, and the run line reads ACCEPT-SOFT, so a later audit can
report recall with and without these briefs.
Round 4 confirms end to end at several list lengths and on two models, so
the paid stage takes a grid of (arm, k, model) cells rather than a pair.
Exactly one of the three may move between two compared cells. A pairing that
would move two is refused rather than computed and qualified in the write-up:
an arm effect and a list length effect land on the same difference and nothing
in the output can say which was which. The comparison stays paired brief by
brief, and a brief only one cell ran is dropped from that pairing and from no
other, so the shared set is recomputed per pairing rather than once for the
grid.
Fixes a real defect on this path. build_report2.cell_key was model blind while
run2._record_key and run2._cell_name already carried the model, so a round 4
checkpoint holding both models would have merged model A's answers with model
B's into one cell and reported the average as a reproduction. A record that
carries no model keeps round 2's published label exactly, because round 2's
report is a committed artifact.
The temperature confound survives into the report. gpt-5.6-luna rejects
temperature 0, which every round 2 and round 3 cell used, so a model B cell
differs from its model A twin in the model and in the sampling temperature. The
pairing, the printed comparison and the rendered table all carry it, and the
cell that will not sample at 0 is named from the temperatures rather than
assumed to be the leader. provider owns that fact now, and the two backends
that send a temperature read their default from it.
The dry run prices the whole grid, cells times models, with each family priced
from its own row and a grand total that is a sum and not an average. A paid run
that names more than one model is refused instead of quietly spending on one
and reporting the grid as done.
round3/report3.html still renders byte for byte at 37794 bytes from the
committed summary and confirmation, now asserted by a test.
Round 3 applied the 95 percent floor to the strict reconstruction rate and
failed it at 75.21 percent, because a brief whose beat two near twin moves
answer equally well counted as a reconstruction failure. Round 4 scores those
briefs instead of excluding them, so the rate that blocks the grid is now the
rate over the corpus that is actually scored: a merged near twin is a success,
and a contested or an unplaceable brief is still a failure.
validate_gold.gate_rate is the single owner of that choice, so the audit status
line and the block read the same number. The strict rate is kept and printed
beside it and never decides, because it is the only number that shows how often
a blind reader landed on the constructed move, and because prune_corpus checks
its own arithmetic against it.
Wave allocation already covered all 424 shelf moves per move rather than per
source; the count is now pinned as a literal so a catalog that loses or gains a
move fails before a wave is priced against it.
Round 3 dropped every brief its blind committee could not reconstruct, 90 of
363. Those briefs were the near twins, exactly the beats where retrieval is
hardest, so every published recall number is an upper bound on the corpus that
was generated, and the 100 percent agreement the audit reported was 100 percent
by construction.
Gold becomes a set. A hit is any acceptable move being shown, a near twin joins
the acceptable set instead of ending the brief, and the committee reports its
rate over every brief it read rather than over the survivors.
gate2 owns the schema, since six loops read this field and each was doing its
own reading. A move listed as both acceptable and known-wrong is refused there
rather than scored once as the answer and once as the trap, which is what keeps
the known-wrong-pick rate from depending on read order. Clustering moves from
best[0], which a set leaves undefined, to the sorted acceptable set, so briefs
answered by the same near-twin family share one cluster.
A single-element set is round 3's scalar. The partition every clustered interval
in rounds 2 and 3 was computed over is provably unchanged on both committed
corpora, and the round 3 lexical rows regenerate from the round 3 corpus at the
published figures, which is now a test.
Step 5 of the pre-registered rule reruns the same grid on a second model
family with exactly one variable different, so the model has to be a cell
variable rather than a deployment setting.
chat now dispatches on the model: gemini ids keep going to Vertex, claude
ids go to the Messages API over the same stdlib transport, and an id that
matches no known family prefix stops the run instead of falling back to the
default backend and reporting model A's answer as a reproduction. The
entitlement probe is unchanged and covers both, because it goes through the
same dispatch; it still makes one call and caps it so it stays single digit
tokens. The second family's id is overridable, since entitlement is settled
by the probe on the account that runs the grid and not by the id's name.
Anthropic reports input tokens excluding cache reads while the cost table
subtracts cached from prompt, so the cache fields are added back before the
arithmetic sees them, and both families get their own price rows: pricing a
model B cell off the Gemini table would misread the only gate standing
between a runaway loop and an unbounded bill.
The cell identity carries the model in both directions. Keyed without it,
resume would read model A's runs as model B already done, and every report
grouping on that key would merge two models into one cell and call the
average a reproduction.
The verdict said the three hybrid weights never separated. Read across the
whole 44-value grid rather than at the headline k, that is false:
hybrid@w=0.3 is separably worse than hybrid@w=0.5 at 7 list lengths and
than hybrid@w=0.7 at 9, all of them short. What never resolves is the two
leaders, which separate at 2 of 44 list lengths and flip sign across the
grid, so the weight ordering is k-dependent in the same way the arm
ordering is. Recorded with the arithmetic that rules out a power
explanation: 231 independent moves would resolve them, against a shelf
of 424.
Rebuilt over the same 44-value k grid with all five arms. The bands are
unchanged (lexical at k=5, hybrid over k=10 to 150, lexical over k=160 to
423) and the headline still reads at k=10, so only the interval widths
moved. No cell prints a zero-width interval any more: 0 of 0.0000 to
0.0000 and 0 of 1.0000 to 1.0000 remain in the page.
A column where every cluster landed on the same value made the sandwich
return a standard error of exactly zero, so the report printed 0 refusals
in 273 runs as a rate of 0.0000 to 0.0000. That is arithmetic, not
evidence. clustered_mean now falls back to the rule of three, which bounds
the rate under 1.1% instead of pinning it, and clips every rate interval
to the range a rate can occupy so a negative refusal rate never appears as
a bound. Paired differences carry their own wider range.
Round 3: 110 tests, 0 failures. Round 2: 90 tests, 0 failures.
273 briefs and their gold, the 90 the blind committee could not reconstruct
in excluded/, the audit that decided the split, and the paid confirmation's
checkpoint. It existed on one machine, and it is the evidence every number
in VERDICT.md rests on.
vectors.json stays ignored: 24MB, regenerable by re-embedding the shelf,
which cost $0.00137 in round 2.
The report rendered the free half of the round and stopped. It now carries
the paid stage beside it: the refusal rate, the known-wrong-pick rate, and
the two differenced brief by brief against the other cell, with a column
saying which of them separated. When the paid stage has not run the page
says so rather than omitting it, because a page showing only the free
ranking reads as a finished round.
confirm3.py gains that paired comparison. Two column means over two cells
would carry brief difficulty inside the gap, since the briefs a short list
answers at all are the easy ones.
VERDICT.md is filled in. The round ends k-dependent: the rule ships lexical
at k=5, hybrid over k=10 to 150, and lexical again from k=160, converging
completely at k=423. Inside the band where the question is live hybrid
separates, and the paid confirmation agrees, ahead on fit by 0.1081 and on
pass rate by 0.1172 with neither interval touching zero. The verdict also
records what the gate does not: the committee reconstructed only 75.21% of
the generated corpus, the 90 near-twin briefs it could not were pruned, and
the surviving 100% gate is therefore true by construction.
The round 3 report carried the k-dependent verdict in summary3.json and
rendered none of it. It now leads with the rule read at every comparable k,
one row per list length, so a reader can see whether the verdict rested on
the whole curve or on one lucky k.
Read at the longest comparable k, the ranking and the paired table were
degenerate: past the point where every arm returns the whole shelf, recall
is 1.0 for all of them and every paired difference is exactly zero. Both
are now read at the shortest k that separates, which is the only list
length defensible on token cost and, being the minimum, cannot have been
picked to widen a margin.
confirm3.py adds the paid stage's two quantities, the refusal rate and the
known-wrong-pick rate, scored by gate2 unchanged and clustered by target
move. They are kept apart because they are opposite failures with opposite
fixes.
The runner kept its results file next to the code, so a later round appended
into the previous round's file. Two things went wrong at once. Resume reads
by brief id, condition and k, and a regenerated corpus numbers from 001
again, so it can inherit runs that never happened on its briefs. And the
ceiling sums every record in the file, so round 3 started its budget already
holding round 2's spend and would have stopped partway through a prefix of
the corpus instead of covering it.
The committee checkpoint moved under the corpus root for the first reason
some time ago. This is the same move for the runner, and the corpus root
test now covers both rather than the one that was noticed first.
Also stop pricing a cached rerun as if it bought everything. The projection
counted every sampled brief as unread, so a corpus already judged in full
was refused at the ceiling for work that would have cost nothing. The
projection and resume now read already judged from one definition.
The sweep read the rule at every comparable k, then printed the last k's
answer as the verdict. That presents a choice of list length as a choice
of retriever, which is the freedom reading every k was meant to remove.
When the swept ks ship different families the verdict now names no arm and
reports the bands of k instead. That is what the rule pre-registered for
this case: all arms draw from the same shelf and must converge at
sufficient k, so the finding is what k rather than which arm.
Step 2 compares the leader against its first rival from another family, so
when the leader's own family also places second, the second row and the
compared arm are different arms. Both the per-k table and the report named
the second row beside the margin, which reads as a comparison that was
never run. They now name the compared arm.
Ranking each arm at a k of its own compared different list lengths and
different token costs, so an arm could lead on budget rather than on
retrieval. Every ranking now sits at one k, which leaves the retriever as
the only thing that varies down it. The knee stays, demoted to the
question it answers: what list length to run the shipped arm at.
The rule is now read at every k below the shelf rather than at one k
picked afterwards, so agreement across ks is reported instead of a single
chosen comparison.
Both suites read their briefs through EVAL_CORPUS_ROOT, so running them in
the shell that actually runs the eval pointed them at round 3's corpus. The
round 2 equivalence proof then compared round 3 briefs against round 2's
paid grid log and failed, which reads as a broken equivalence proof rather
than a misdirected one, and against an empty corpus the round 2 harness
tests crashed in setUpClass instead of failing.
Round 2's regressions now name round 2's briefs outright, and load_briefs
takes the directory it reads so a caller can say which corpus it means
without mutating the module global. The corpus-root redirect test asserts
the root is restored to whatever it was before rather than to HERE, which
only held in a shell with no EVAL_CORPUS_ROOT set, and it now also covers
the committee checkpoint path, which is the one that actually escaped.
172 tests green with the variable unset and with it pointed elsewhere.
Recall rises with k by construction, so ranking each arm at its highest
recall selected whatever the longest swept k happened to be. Extending the
sweep past k=80 made the failure total: at k equal to the shelf size every
arm shows the model all 424 entries and scores exactly 1.0, so the ranking
tied every arm and would have shipped the whole-shelf configuration. That
configuration is round 2's control cell b, which the pre-registered rule
fixes at 0.0 and forbids changing mid-round.
The rule is silent on how k is chosen but already names the right basis,
the point where token cost starts to outrun the recall gain. That is the
knee, which the module already computed and the ranking simply did not
use. So k now comes from the knee, and any k at or past the shelf size is
kept in the reported curve, where it shows the ceiling, and kept out of
the ranking, where it decides nothing.
The report builder ranks on the same basis, so the HTML and the command
line cannot name different winners.
The committee checkpoint defaulted under HERE while the briefs, gold and
audit it belongs to had already moved under CORPUS_ROOT, so every corpus
shared one file. Resume then matched on brief_id alone. Ids are positions:
a regenerated corpus numbers from the start again, so each id collides with
an older brief that says something else, and resume answered the new corpus
with the old corpus's verdicts without spending a call to notice. The audit
prunes on those verdicts, so the corpus would have been cut on judgements of
briefs no committee ever read.
The checkpoint now sits under CORPUS_ROOT and every record carries the sha256
of the brief text it judged. A record whose digest does not match the brief
now on disk is re-read, and one written before this change cannot prove which
brief it saw, so it is re-read too.
Also splits the brief gate's distractor rejection in two. A competitor count
of zero means the brief has a score band and stands alone in it, which is a
brief that gives its move away. No overlap with its own entry means it never
scored at all, which is what a thin shelf entry does to every brief written
for it whatever the brief says. Both printed "distractor:0", which sent a
diagnosis of the second down the path of the first.
Tests cover both branches, and the cross-corpus case fails against the old
resume with new_reads 0 where it should be 3.
provider.py already documented the constraint; build_shelf.py and test_gate2.py
used the method anyway, which broke gate2's whole test module on devbox. The
guard test now checks for the 3.9 string methods, not only for 3.9 syntax.
The generator, committee, pruner, embedder and harness each carried their own
copy of the briefs path. Round 3 needs its own corpus, and round 2's briefs and
gold are what round 3's statistics are checked against, so generating in place
would have destroyed the fixtures. One EVAL_CORPUS_ROOT, derived everywhere.
Brief ids are positions in a corpus. A regenerated corpus renumbers from 001,
so every id already names a different brief in vectors.json and the resume
skipped it, handing the new corpus the old corpus's vectors with no error.
Nine parenthesized with blocks across two files are 3.9+ syntax, so both were
uncompilable on devbox while passing on a newer local interpreter. Adds a guard
test so the gap cannot reopen silently.
parse_cells validated k against the three values round 2 happened to sample,
so a sweep result past k=20 named a list length stage 2 could never be asked
for. Check the condition letter and the k arity instead.
Step 4 of the pre-registered rule runs the paid confirmation on the top two arms
only. run_grid has always taken a cells argument, but main hardcoded the full
nine cell tuple and nothing exposed it, so the only reachable run was the whole
grid. That is not a confirmation of a ranking, it is a second full round, and at
wave 1 size it costs about five times what the rule asks for.
EVAL_CELLS names the pair. The default stays the whole grid, because every
earlier round was run that way and a selection that silently narrowed it would
make two rounds' numbers incomparable with nothing saying so. An unknown cell or
a malformed k stops the run rather than being dropped: dropping it would run
fewer cells than asked for and report the result as if it were the selection.
RUNBOOK.md carries the approved wave 1 sequence, each step with its own dry run
ahead of its spend, and states two limits plainly rather than hiding them: brief
generation has no dollar meter and is bounded by attempt count instead, and the
step 5 price is an estimate until step 4 names the actual pair.
PRICE_TABLE carries a :batch row at half price and the committee projection
quoted it as the tier a real run should use. Nothing submits batch work. Every
caller reaches the model through provider.chat, which posts to the synchronous
generateContent endpoint, so the batch row had exactly one consumer and that
consumer was a projection.
The consequence was not cosmetic: a spend approval was made on the batch figure
and the real charge would have been twice it. A price nothing can pay is worse
than no price, because it reads as the cheaper of two available choices rather
than as an absent feature.
The row stays as a note on what the tier would cost, with the reason no
projection may quote it, so the next person to reach for it adds the submission
path in the same change that starts pricing it.
The grid and the gold committee both project their cost before any call is
made. Generation did not, which left the one cost that scales linearly with the
corpus as the only one invisible at the gate. A wave of 424 briefs was about to
be approved on a number nobody had.
Attempts are reported as a floor and a ceiling rather than a point. A move costs
one call if its first brief clears every gate and up to four if it keeps failing
them, and the acceptance rate of a wave that has not run is not knowable here.
The projection loads the same source text generate_corpus loads rather than
source_of's stratum label. The label is nine characters and the artifact is
hundreds, so pricing the label would have understated the input side and the
error would have looked like a plausible number rather than a bug. Reading the
files also preflights them, so a missing source fails at the gate.
Separately, the round one parity check now runs its lexical half and skips only
the semantic half when round one's gitignored vectors are absent. It used to
error, which reported a missing untracked input as a broken retriever.
Two defects found by reading the rendered report rather than the test output.
The reason an arm was not swept carries the path of the index it could not
find, which was the builder's absolute path. That dated the report to one
checkout on one machine and told the reader nothing about the sweep, so it is
now rewritten repo relative in the summary, which fixes the JSON as well as the
page.
The paired table rendered as a bare header when only one arm was available. An
empty pairing reads as a tie, and there is a difference between two arms that
did not separate and there never having been a second arm. The report now says
which one happened.
Round 3's arm ranking is decided by recall, which the retriever computes
before a token is bought, so the report that carries it should cost nothing
and be readable without the paid stage having run.
Three things drove the shape. Every table ships a clustered interval from
stats.py rather than a bare mean, because a recall number without one cannot
be read against the pre-registered rule; that forced the recall curve into
long form, since a wide arm-by-k grid has no room for error bars. The report
names which of the three stopping conditions fired, because an arm that won
on measurement, a round that ran out of corpus, and a round that ran out of
budget are three different verdicts and collapsing them into one is how a
tie gets reported as a win. And the summary file replaces infinite bounds
with null, since JSON has no infinity and a file only one language can read
back is not an auditable artifact.
VERDICT.md is deliberately pending. The paid confirmation has not run, so it
states the rule, the three stop reasons and the 424-entry shelf ceiling, and
lists every absent number beside the file that will produce it, rather than
estimating any of them. The ceiling is the load-bearing part: no corpus size
buys more than 424 independent things to be right about, so if the arms do
not separate the honest recommendation is a larger catalog, not a larger
corpus.
Tests are unittest, matching the rest of round 3, and parse the rendered
HTML rather than grepping it, since a grep passes on a caption that merely
mentions an interval.
sweep.py ranks every retrieval arm on recall alone, reading the decision rule
from DECISION-RULE.md rather than carrying its own copy, so the branch it takes
is the branch that was pre-registered. It refuses to run against a rule file
whose machine-readable block is missing, incomplete, or says an exploratory arm
may win.
weighted_hybrid exists because harness2.hybrid_topk hardcodes an equal-weight
fusion; a test pins weighted_hybrid(0.5) to that exact list so the new code is
a superset rather than a second implementation.
Rank the decision-tier arms on offline recall with intervals clustered by
target move, read the branch parameters out of DECISION-RULE.md rather than
transcribing them, and price the recall knee against round 2's measured
tokens-per-k slope.
build_report2.aggregate returned bare means with no interval at all, so
round 2's headline table had no error bars. Briefs written for the same
target move are correlated, so a per-brief interval understates the spread
and a larger corpus would have narrowed it wrongly.
Intervals are now cluster robust, collapsing onto the ordinary standard
error when every cluster holds one observation. Round 2's effective sample
comes out near 199 rather than 227.
The committee is the dominant cost of a scaled round and it has no latency
requirement, so the tier is a real decision rather than a detail. Pricing
only the interactive tier overstates the round by about two times, which
could talk the corpus size down for no reason.