Commit Graph
2413 Commits
Author SHA1 Message Date
Miguel Angel Simon Sierra fbdabb5e82 feat(evals): close round 3 with the paid confirmation on the page
The report rendered the free half of the round and stopped. It now carries
the paid stage beside it: the refusal rate, the known-wrong-pick rate, and
the two differenced brief by brief against the other cell, with a column
saying which of them separated. When the paid stage has not run the page
says so rather than omitting it, because a page showing only the free
ranking reads as a finished round.

confirm3.py gains that paired comparison. Two column means over two cells
would carry brief difficulty inside the gap, since the briefs a short list
answers at all are the easy ones.

VERDICT.md is filled in. The round ends k-dependent: the rule ships lexical
at k=5, hybrid over k=10 to 150, and lexical again from k=160, converging
completely at k=423. Inside the band where the question is live hybrid
separates, and the paid confirmation agrees, ahead on fit by 0.1081 and on
pass rate by 0.1172 with neither interval touching zero. The verdict also
records what the gate does not: the committee reconstructed only 75.21% of
the generated corpus, the 90 near-twin briefs it could not were pruned, and
the surviving 100% gate is therefore true by construction.
2026-08-01 04:52:20 +02:00
Miguel Angel Simon Sierra 47529db21f feat(evals): report the k bands and confirm what recall cannot see
The round 3 report carried the k-dependent verdict in summary3.json and
rendered none of it. It now leads with the rule read at every comparable k,
one row per list length, so a reader can see whether the verdict rested on
the whole curve or on one lucky k.

Read at the longest comparable k, the ranking and the paired table were
degenerate: past the point where every arm returns the whole shelf, recall
is 1.0 for all of them and every paired difference is exactly zero. Both
are now read at the shortest k that separates, which is the only list
length defensible on token cost and, being the minimum, cannot have been
picked to widen a margin.

confirm3.py adds the paid stage's two quantities, the refusal rate and the
known-wrong-pick rate, scored by gate2 unchanged and clustered by target
move. They are kept apart because they are opposite failures with opposite
fixes.
2026-08-01 04:36:26 +02:00
Miguel Angel Simon Sierra 58790bb5a6 fix(evals): give the grid runner the corpus its checkpoint belongs to
The runner kept its results file next to the code, so a later round appended
into the previous round's file. Two things went wrong at once. Resume reads
by brief id, condition and k, and a regenerated corpus numbers from 001
again, so it can inherit runs that never happened on its briefs. And the
ceiling sums every record in the file, so round 3 started its budget already
holding round 2's spend and would have stopped partway through a prefix of
the corpus instead of covering it.

The committee checkpoint moved under the corpus root for the first reason
some time ago. This is the same move for the runner, and the corpus root
test now covers both rather than the one that was noticed first.

Also stop pricing a cached rerun as if it bought everything. The projection
counted every sampled brief as unread, so a corpus already judged in full
was refused at the ceiling for work that would have cost nothing. The
projection and resume now read already judged from one definition.
2026-08-01 04:13:06 +02:00
Miguel Angel Simon Sierra ca378f3671 fix(evals): report the list length bands, never a ship from one k
The sweep read the rule at every comparable k, then printed the last k's
answer as the verdict. That presents a choice of list length as a choice
of retriever, which is the freedom reading every k was meant to remove.

When the swept ks ship different families the verdict now names no arm and
reports the bands of k instead. That is what the rule pre-registered for
this case: all arms draw from the same shelf and must converge at
sufficient k, so the finding is what k rather than which arm.
2026-08-01 03:58:07 +02:00
Miguel Angel Simon Sierra a55d53d851 fix(evals): name the arm a margin was actually measured against
Step 2 compares the leader against its first rival from another family, so
when the leader's own family also places second, the second row and the
compared arm are different arms. Both the per-k table and the report named
the second row beside the margin, which reads as a comparison that was
never run. They now name the compared arm.
2026-08-01 03:51:28 +02:00
Miguel Angel Simon Sierra 7bf0a25561 fix(evals): compare every arm at the same list length
Ranking each arm at a k of its own compared different list lengths and
different token costs, so an arm could lead on budget rather than on
retrieval. Every ranking now sits at one k, which leaves the retriever as
the only thing that varies down it. The knee stays, demoted to the
question it answers: what list length to run the shipped arm at.

The rule is now read at every k below the shelf rather than at one k
picked afterwards, so agreement across ks is reported instead of a single
chosen comparison.
2026-08-01 03:46:30 +02:00
Miguel Angel Simon Sierra 0a0d992d7e test(evals): pin each round's regressions to its own corpus
Both suites read their briefs through EVAL_CORPUS_ROOT, so running them in
the shell that actually runs the eval pointed them at round 3's corpus. The
round 2 equivalence proof then compared round 3 briefs against round 2's
paid grid log and failed, which reads as a broken equivalence proof rather
than a misdirected one, and against an empty corpus the round 2 harness
tests crashed in setUpClass instead of failing.

Round 2's regressions now name round 2's briefs outright, and load_briefs
takes the directory it reads so a caller can say which corpus it means
without mutating the module global. The corpus-root redirect test asserts
the root is restored to whatever it was before rather than to HERE, which
only held in a shell with no EVAL_CORPUS_ROOT set, and it now also covers
the committee checkpoint path, which is the one that actually escaped.

172 tests green with the variable unset and with it pointed elsewhere.
2026-08-01 03:31:45 +02:00
Miguel Angel Simon Sierra 842286a688 fix(evals): rank each arm at its knee, never at a shelf-length list
Recall rises with k by construction, so ranking each arm at its highest
recall selected whatever the longest swept k happened to be. Extending the
sweep past k=80 made the failure total: at k equal to the shelf size every
arm shows the model all 424 entries and scores exactly 1.0, so the ranking
tied every arm and would have shipped the whole-shelf configuration. That
configuration is round 2's control cell b, which the pre-registered rule
fixes at 0.0 and forbids changing mid-round.

The rule is silent on how k is chosen but already names the right basis,
the point where token cost starts to outrun the recall gain. That is the
knee, which the module already computed and the ranking simply did not
use. So k now comes from the knee, and any k at or past the shelf size is
kept in the reported curve, where it shows the ceiling, and kept out of
the ranking, where it decides nothing.

The report builder ranks on the same basis, so the HTML and the command
line cannot name different winners.
2026-08-01 03:20:55 +02:00
Miguel Angel Simon Sierra 10b1a89aee fix(evals): key committee resume on the brief, not its position
The committee checkpoint defaulted under HERE while the briefs, gold and
audit it belongs to had already moved under CORPUS_ROOT, so every corpus
shared one file. Resume then matched on brief_id alone. Ids are positions:
a regenerated corpus numbers from the start again, so each id collides with
an older brief that says something else, and resume answered the new corpus
with the old corpus's verdicts without spending a call to notice. The audit
prunes on those verdicts, so the corpus would have been cut on judgements of
briefs no committee ever read.

The checkpoint now sits under CORPUS_ROOT and every record carries the sha256
of the brief text it judged. A record whose digest does not match the brief
now on disk is re-read, and one written before this change cannot prove which
brief it saw, so it is re-read too.

Also splits the brief gate's distractor rejection in two. A competitor count
of zero means the brief has a score band and stands alone in it, which is a
brief that gives its move away. No overlap with its own entry means it never
scored at all, which is what a thin shelf entry does to every brief written
for it whatever the brief says. Both printed "distractor:0", which sent a
diagnosis of the second down the path of the first.

Tests cover both branches, and the cross-corpus case fails against the old
resume with new_reads 0 where it should be 3.
2026-08-01 02:32:15 +02:00
Miguel Angel Simon Sierra 7b7fbaaf40 docs(eval): runbook carries the corpus root and the real key discovery step 2026-08-01 00:19:10 +02:00
Miguel Angel Simon Sierra 73a89ff706 chore(eval): ignore the round 3 corpus vector cache 2026-08-01 00:12:10 +02:00
Miguel Angel Simon Sierra 33dff03d19 fix(eval): stop using str.removeprefix on a Python 3.8 run host
provider.py already documented the constraint; build_shelf.py and test_gate2.py
used the method anyway, which broke gate2's whole test module on devbox. The
guard test now checks for the 3.9 string methods, not only for 3.9 syntax.
2026-08-01 00:10:40 +02:00
Miguel Angel Simon Sierra ed5a2d4da4 refactor(eval): give the corpus one root so a new round cannot overwrite the fixtures
The generator, committee, pruner, embedder and harness each carried their own
copy of the briefs path. Round 3 needs its own corpus, and round 2's briefs and
gold are what round 3's statistics are checked against, so generating in place
would have destroyed the fixtures. One EVAL_CORPUS_ROOT, derived everywhere.
2026-08-01 00:08:07 +02:00
Miguel Angel Simon Sierra 168e9c8048 fix(eval): key the embedding resume on the text, not on the brief id
Brief ids are positions in a corpus. A regenerated corpus renumbers from 001,
so every id already names a different brief in vectors.json and the resume
skipped it, handing the new corpus the old corpus's vectors with no error.
2026-08-01 00:04:37 +02:00
Miguel Angel Simon Sierra abe5664114 fix(eval): stub the vector index the dry run test actually reads 2026-07-31 23:59:10 +02:00
Miguel Angel Simon Sierra 00cc85f4b1 fix(eval): make the whole test tree compile on the deployment interpreter
Nine parenthesized with blocks across two files are 3.9+ syntax, so both were
uncompilable on devbox while passing on a newer local interpreter. Adds a guard
test so the gap cannot reopen silently.
2026-07-31 23:57:16 +02:00
Miguel Angel Simon Sierra a0a2ba6eba fix(eval): keep the runner tests runnable on the machine that runs the eval 2026-07-31 23:53:29 +02:00
Miguel Angel Simon Sierra 94d5cec536 fix(eval): let the confirmation ask for the k the sweep chose
parse_cells validated k against the three values round 2 happened to sample,
so a sweep result past k=20 named a list length stage 2 could never be asked
for. Check the condition letter and the k arity instead.
2026-07-31 23:51:37 +02:00
Miguel Angel Simon Sierra 243998d729 feat(eval): let the confirmation ask for the two arms the rule names
Step 4 of the pre-registered rule runs the paid confirmation on the top two arms
only. run_grid has always taken a cells argument, but main hardcoded the full
nine cell tuple and nothing exposed it, so the only reachable run was the whole
grid. That is not a confirmation of a ranking, it is a second full round, and at
wave 1 size it costs about five times what the rule asks for.

EVAL_CELLS names the pair. The default stays the whole grid, because every
earlier round was run that way and a selection that silently narrowed it would
make two rounds' numbers incomparable with nothing saying so. An unknown cell or
a malformed k stops the run rather than being dropped: dropping it would run
fewer cells than asked for and report the result as if it were the selection.

RUNBOOK.md carries the approved wave 1 sequence, each step with its own dry run
ahead of its spend, and states two limits plainly rather than hiding them: brief
generation has no dollar meter and is bounded by attempt count instead, and the
step 5 price is an estimate until step 4 names the actual pair.
2026-07-31 23:25:32 +02:00
Miguel Angel Simon Sierra 16e0295164 fix(eval): stop the gate quoting a tier nothing can submit to
PRICE_TABLE carries a :batch row at half price and the committee projection
quoted it as the tier a real run should use. Nothing submits batch work. Every
caller reaches the model through provider.chat, which posts to the synchronous
generateContent endpoint, so the batch row had exactly one consumer and that
consumer was a projection.

The consequence was not cosmetic: a spend approval was made on the batch figure
and the real charge would have been twice it. A price nothing can pay is worse
than no price, because it reads as the cheaper of two available choices rather
than as an absent feature.

The row stays as a note on what the tier would cost, with the reason no
projection may quote it, so the next person to reach for it adds the submission
path in the same change that starts pricing it.
2026-07-31 23:09:25 +02:00
Miguel Angel Simon Sierra bd496a7ff4 feat(eval): price a generation wave before it runs
The grid and the gold committee both project their cost before any call is
made. Generation did not, which left the one cost that scales linearly with the
corpus as the only one invisible at the gate. A wave of 424 briefs was about to
be approved on a number nobody had.

Attempts are reported as a floor and a ceiling rather than a point. A move costs
one call if its first brief clears every gate and up to four if it keeps failing
them, and the acceptance rate of a wave that has not run is not knowable here.

The projection loads the same source text generate_corpus loads rather than
source_of's stratum label. The label is nine characters and the artifact is
hundreds, so pricing the label would have understated the input side and the
error would have looked like a plausible number rather than a bug. Reading the
files also preflights them, so a missing source fails at the gate.

Separately, the round one parity check now runs its lexical half and skips only
the semantic half when round one's gitignored vectors are absent. It used to
error, which reported a missing untracked input as a broken retriever.
2026-07-31 22:23:20 +02:00
Miguel Angel Simon Sierra ead878e541 fix(eval): keep the round 3 report portable and honest when an arm is alone
Two defects found by reading the rendered report rather than the test output.

The reason an arm was not swept carries the path of the index it could not
find, which was the builder's absolute path. That dated the report to one
checkout on one machine and told the reader nothing about the sweep, so it is
now rewritten repo relative in the summary, which fixes the JSON as well as the
page.

The paired table rendered as a bare header when only one arm was available. An
empty pairing reads as a tie, and there is a difference between two arms that
did not separate and there never having been a second arm. The report now says
which one happened.
2026-07-31 22:15:31 +02:00
Miguel Angel Simon Sierra a4e879f5ed feat(eval): report round 3 with intervals and a named stop reason
Round 3's arm ranking is decided by recall, which the retriever computes
before a token is bought, so the report that carries it should cost nothing
and be readable without the paid stage having run.

Three things drove the shape. Every table ships a clustered interval from
stats.py rather than a bare mean, because a recall number without one cannot
be read against the pre-registered rule; that forced the recall curve into
long form, since a wide arm-by-k grid has no room for error bars. The report
names which of the three stopping conditions fired, because an arm that won
on measurement, a round that ran out of corpus, and a round that ran out of
budget are three different verdicts and collapsing them into one is how a
tie gets reported as a win. And the summary file replaces infinite bounds
with null, since JSON has no infinity and a file only one language can read
back is not an auditable artifact.

VERDICT.md is deliberately pending. The paid confirmation has not run, so it
states the rule, the three stop reasons and the 424-entry shelf ceiling, and
lists every absent number beside the file that will produce it, rather than
estimating any of them. The ceiling is the load-bearing part: no corpus size
buys more than 424 independent things to be right about, so if the arms do
not separate the honest recommendation is a larger catalog, not a larger
corpus.

Tests are unittest, matching the rest of round 3, and parse the rendered
HTML rather than grepping it, since a grep passes on a caption that merely
mentions an interval.
2026-07-31 22:10:15 +02:00
Miguel Angel Simon Sierra 9212ed472d feat(eval): add the round 3 offline arm sweep
sweep.py ranks every retrieval arm on recall alone, reading the decision rule
from DECISION-RULE.md rather than carrying its own copy, so the branch it takes
is the branch that was pre-registered. It refuses to run against a rule file
whose machine-readable block is missing, incomplete, or says an exploratory arm
may win.

weighted_hybrid exists because harness2.hybrid_topk hardcodes an equal-weight
fusion; a test pins weighted_hybrid(0.5) to that exact list so the new code is
a superset rather than a second implementation.
2026-07-31 21:59:44 +02:00
Miguel Angel Simon Sierra 01ea003aa4 feat(evals): sweep every retrieval arm and apply the pre-registered rule
Rank the decision-tier arms on offline recall with intervals clustered by
target move, read the branch parameters out of DECISION-RULE.md rather than
transcribing them, and price the recall knee against round 2's measured
tokens-per-k slope.
2026-07-31 21:39:15 +02:00
Miguel Angel Simon Sierra e46824596d feat(eval): cluster every interval by target move
build_report2.aggregate returned bare means with no interval at all, so
round 2's headline table had no error bars. Briefs written for the same
target move are correlated, so a per-brief interval understates the spread
and a larger corpus would have narrowed it wrongly.

Intervals are now cluster robust, collapsing onto the ordinary standard
error when every cluster holds one observation. Round 2's effective sample
comes out near 199 rather than 227.
2026-07-31 21:26:46 +02:00
Miguel Angel Simon Sierra 59abcdb41c feat(eval): quote the batch tier beside the interactive one at the gate
The committee is the dominant cost of a scaled round and it has no latency
requirement, so the tier is a real decision rather than a detail. Pricing
only the interactive tier overstates the round by about two times, which
could talk the corpus size down for no reason.
2026-07-31 21:16:03 +02:00
Miguel Angel Simon Sierra 3596830187 Merge commit '16ad23020' into feat-video-primitives 2026-07-31 21:13:52 +02:00
Miguel Angel Simon Sierra 57a6687511 Merge branch 'u4-per-move-allocation' into feat-video-primitives 2026-07-31 21:13:51 +02:00
Miguel Angel Simon Sierra 94fadadf25 fix(eval): stop the hybrid arm from answering as lexical when vectors are gone
hybrid_topk fell back to the lexical list whenever vectors.json was missing
or the semantic ranking came back empty. That never fired during the round 2
grid because the vectors were present, so it shipped unnoticed. It fires
constantly in an offline sweep, where it made hybrid print lexical's exact
recall curve under its own name. An arm comparison built on that reads as a
tie between two arms rather than as a missing input.

Hybrid now exits loudly, the way semantic already did. The offline sweep
also runs arms one at a time, so an arm that needs vectors going quiet no
longer silences lexical, which needs none.
2026-07-31 21:13:36 +02:00
Miguel Angel Simon Sierra 16ad230207 feat(eval): price the gold committee before it runs
Make the committee sample rate an explicit parameter, project its cost
from context lengths alone under EVAL_DRY_RUN=1, and reuse run2's spend
ceiling, kill file and checkpoint resume so a corpus-scale audit stops
on a known number instead of discovering it. A rate below round 2's 20
percent now names the widened error bound out loud, and a rate of 0
refuses rather than shipping an unaudited corpus in silence.
2026-07-31 21:11:03 +02:00
Miguel Angel Simon Sierra dfd3a80d3a feat(eval): allocate the brief corpus per move, in waves
Round 2 split briefs equally across the three source types. At round 3
volume that would put roughly 700 briefs on 25 primitive moves, which at
the measured intra-cluster correlation of 0.33 carry about as much
information as 3 independent ones.

choose_targets now takes a wave number and brings every catalog move up
to that many generation attempts, so cluster sizes stay uniform.
Attempts are counted rather than acceptances, which is what makes a
killed wave resumable without backfilling a move the gates rejected.
wave_report prints per-source counts, the per-move count distribution
and the unbackfilled shortfall.
2026-07-31 21:11:03 +02:00
Miguel Angel Simon Sierra 4feadbc7cc Merge branch 'eval-round3-u2' into feat-video-primitives 2026-07-31 21:10:31 +02:00
Miguel Angel Simon Sierra 41fb8647cb fix(eval): correct a misquoted round 2 semantic recall figure
Semantic recall at k=10 is 0.383, not 0.282. The 0.282 figure is semantic
k=5, which collides with lexical k=10 and was misread from the cell table.
The refusal citation now names the arm it came from.

The rule itself is unchanged. Only the background citations move, and the
correction strengthens the convergence case rather than weakening it.
2026-07-31 21:10:16 +02:00
Miguel Angel Simon Sierra 2c6419fc88 feat(eval): offline recall harness proven against the round 2 paid grid 2026-07-31 21:06:29 +02:00
Miguel Angel Simon Sierra 1a4378737f feat(eval): pre-register the round 3 arm decision rule 2026-07-31 20:58:40 +02:00
Miguel Angel Simon Sierra 42985dc506 test(eval): freeze round 2's per-brief recall as an offline fixture 2026-07-31 20:56:04 +02:00
Miguel Angel Simon Sierra 6a9f4f4b50 feat(eval): record the round 2 verdict, summary and report 2026-07-31 20:52:34 +02:00
Miguel Angel Simon Sierra e0e532f56d feat(eval): report tokens and dollars per cell alongside fit 2026-07-31 16:58:44 +02:00
Miguel Angel Simon Sierra 6bfa66d2d1 feat(eval): compare cells on the same briefs, not the same column
fit when shown is conditioned on retrieval succeeding, and the briefs a short
list finds are the easy ones, so reading that column across cells compares
different brief sets and can invent an effect that is only difficulty. The
report now restricts the full shelf to each cell's own gold-shown briefs.
2026-07-31 16:42:49 +02:00
Miguel Angel Simon Sierra 5e04109a11 feat(eval): split retrieval misses from selection misses in the report
A low fit reads as the model choosing badly when the real story is often that
retrieval never showed it the answer, and those two call for opposite fixes.
Each cell now reports how often the gold move was in the list the run could
see, and its fit on just those runs.
2026-07-31 16:37:37 +02:00
Miguel Angel Simon Sierra 65949f60e0 feat(eval): aggregate the round-two grid into a report
Round one printed one card per run, which does not survive 2043 of them. This
aggregates per cell and per stratum first and shows individual runs only for
failures, keeping mountable and fit as two separate columns. It reads the
checkpoint log alone, so it runs against a grid that is still going.
2026-07-31 16:32:55 +02:00
Miguel Angel Simon Sierra 4bce4ce0ce fix(eval): dry run banner stated 50 output tokens while billing 800
The banner a reader checks before approving spend disagreed with the
arithmetic under it by a factor of sixteen, understating the output half of
every projection. The printed assumption now interpolates the constant, and a
regression pins the two together.
2026-07-31 16:09:41 +02:00
Miguel Angel Simon Sierra 7759acc5ea perf(eval): memoize the semantic ranking per brief
semantic_topk re-read and re-scored vectors.json on every call. The grid asks
for a semantic ranking 227 briefs x 4 cells, so a tens-of-megabytes parse and a
424-move cosine sweep ran roughly 900 times. The parse and the ranking are now
memoized, keyed by the vectors path so the tests that repoint VECTORS still see
their own file.
2026-07-31 16:02:43 +02:00
Miguel Angel Simon Sierra 6a2b287a2f feat(eval): drop briefs the blind committee cannot reconstruct
The gold sets are right by construction, but at 424 entries some moves are
near twins, so 45 of 272 briefs describe a beat that two moves answer equally
well. Three independent full-shelf readers reconstructed 227, below the 95
percent floor, and the grid stayed blocked.

Moves those 45 briefs and their gold into excluded/ rather than deleting them,
so the drop is auditable and reversible, and leaves gold purely constructed.
The kept corpus is easier than the generated one and its block stratum falls
from 79 to 53, which the verdict has to state.
2026-07-31 15:41:39 +02:00
Miguel Angel Simon Sierra 4ee6694be4 fix(eval): read round two briefs from round two
The runner resolved the brief directory one level up, where round one's five
briefs live. They load without error, so the whole nine cell grid would have
scored against the wrong corpus. The generator and the gold committee already
resolved it correctly, so the three stages now assert they agree.

Commits the 272 brief corpus, its gold sets and the committee audit so the
experiment is reproducible from the repository rather than from one machine.
2026-07-31 15:37:11 +02:00
Miguel Angel Simon Sierra ddcf5c0636 fix(evals): regenerate the shelf and index for the twelve merged pairs
Two moves that shipped as registry components became primitive and component
collisions, so the catalog now merges twelve pairs instead of ten. Regenerated
rather than hand edited: the shelf suite asserts these bytes come back out of
the builder unchanged.
2026-07-31 12:47:37 +02:00
Miguel Angel Simon Sierra 2372c68ba3 fix(evals): keep the formatter off generated catalog artifacts
The pre-commit formatter reformatted the staged shelf and index and re-staged
them, so the committed catalog no longer matched what the builder emits and the
regeneration assertion failed on a fresh checkout. Their generator owns those
bytes, so the formatter now skips them.
2026-07-31 12:45:17 +02:00
Miguel Angel Simon Sierra 75e5d6cd4f fix(evals): regenerate the shelf and index against the pinned collision set
The previous commit carried the source change without the artifacts it
produces, so the checked-in shelf still merged ten primitive and component
pairs while the builder now merges twelve. Regeneration is asserted byte for
byte, so a stale artifact fails the shelf suite rather than silently scoring
the eval against a catalog nobody can rebuild.
2026-07-31 12:43:02 +02:00
Miguel Angel Simon Sierra d866d092dd fix(evals): resolve shelf sources from the vendored path in tests too
The shelf builder reads primitive artifacts from evals/catalog-retrieval/sources,
but the test fixture still exported them to the old scratch layout, so the
negative controls built against a repo that had no primitive sources at all.
The location is now one named constant both sides resolve against.

Two moves that shipped as registry components in the previous commit,
caption-camera-follow and grade-split-reveal, are now real primitive plus
component collisions, so the pinned collision set and the regenerated shelf
account for them.
2026-07-31 12:36:02 +02:00