Commit Graph
2399 Commits
Author SHA1 Message Date
Miguel Angel Simon Sierra abe5664114 fix(eval): stub the vector index the dry run test actually reads 2026-07-31 23:59:10 +02:00
Miguel Angel Simon Sierra 00cc85f4b1 fix(eval): make the whole test tree compile on the deployment interpreter
Nine parenthesized with blocks across two files are 3.9+ syntax, so both were
uncompilable on devbox while passing on a newer local interpreter. Adds a guard
test so the gap cannot reopen silently.
2026-07-31 23:57:16 +02:00
Miguel Angel Simon Sierra a0a2ba6eba fix(eval): keep the runner tests runnable on the machine that runs the eval 2026-07-31 23:53:29 +02:00
Miguel Angel Simon Sierra 94d5cec536 fix(eval): let the confirmation ask for the k the sweep chose
parse_cells validated k against the three values round 2 happened to sample,
so a sweep result past k=20 named a list length stage 2 could never be asked
for. Check the condition letter and the k arity instead.
2026-07-31 23:51:37 +02:00
Miguel Angel Simon Sierra 243998d729 feat(eval): let the confirmation ask for the two arms the rule names
Step 4 of the pre-registered rule runs the paid confirmation on the top two arms
only. run_grid has always taken a cells argument, but main hardcoded the full
nine cell tuple and nothing exposed it, so the only reachable run was the whole
grid. That is not a confirmation of a ranking, it is a second full round, and at
wave 1 size it costs about five times what the rule asks for.

EVAL_CELLS names the pair. The default stays the whole grid, because every
earlier round was run that way and a selection that silently narrowed it would
make two rounds' numbers incomparable with nothing saying so. An unknown cell or
a malformed k stops the run rather than being dropped: dropping it would run
fewer cells than asked for and report the result as if it were the selection.

RUNBOOK.md carries the approved wave 1 sequence, each step with its own dry run
ahead of its spend, and states two limits plainly rather than hiding them: brief
generation has no dollar meter and is bounded by attempt count instead, and the
step 5 price is an estimate until step 4 names the actual pair.
2026-07-31 23:25:32 +02:00
Miguel Angel Simon Sierra 16e0295164 fix(eval): stop the gate quoting a tier nothing can submit to
PRICE_TABLE carries a :batch row at half price and the committee projection
quoted it as the tier a real run should use. Nothing submits batch work. Every
caller reaches the model through provider.chat, which posts to the synchronous
generateContent endpoint, so the batch row had exactly one consumer and that
consumer was a projection.

The consequence was not cosmetic: a spend approval was made on the batch figure
and the real charge would have been twice it. A price nothing can pay is worse
than no price, because it reads as the cheaper of two available choices rather
than as an absent feature.

The row stays as a note on what the tier would cost, with the reason no
projection may quote it, so the next person to reach for it adds the submission
path in the same change that starts pricing it.
2026-07-31 23:09:25 +02:00
Miguel Angel Simon Sierra bd496a7ff4 feat(eval): price a generation wave before it runs
The grid and the gold committee both project their cost before any call is
made. Generation did not, which left the one cost that scales linearly with the
corpus as the only one invisible at the gate. A wave of 424 briefs was about to
be approved on a number nobody had.

Attempts are reported as a floor and a ceiling rather than a point. A move costs
one call if its first brief clears every gate and up to four if it keeps failing
them, and the acceptance rate of a wave that has not run is not knowable here.

The projection loads the same source text generate_corpus loads rather than
source_of's stratum label. The label is nine characters and the artifact is
hundreds, so pricing the label would have understated the input side and the
error would have looked like a plausible number rather than a bug. Reading the
files also preflights them, so a missing source fails at the gate.

Separately, the round one parity check now runs its lexical half and skips only
the semantic half when round one's gitignored vectors are absent. It used to
error, which reported a missing untracked input as a broken retriever.
2026-07-31 22:23:20 +02:00
Miguel Angel Simon Sierra ead878e541 fix(eval): keep the round 3 report portable and honest when an arm is alone
Two defects found by reading the rendered report rather than the test output.

The reason an arm was not swept carries the path of the index it could not
find, which was the builder's absolute path. That dated the report to one
checkout on one machine and told the reader nothing about the sweep, so it is
now rewritten repo relative in the summary, which fixes the JSON as well as the
page.

The paired table rendered as a bare header when only one arm was available. An
empty pairing reads as a tie, and there is a difference between two arms that
did not separate and there never having been a second arm. The report now says
which one happened.
2026-07-31 22:15:31 +02:00
Miguel Angel Simon Sierra a4e879f5ed feat(eval): report round 3 with intervals and a named stop reason
Round 3's arm ranking is decided by recall, which the retriever computes
before a token is bought, so the report that carries it should cost nothing
and be readable without the paid stage having run.

Three things drove the shape. Every table ships a clustered interval from
stats.py rather than a bare mean, because a recall number without one cannot
be read against the pre-registered rule; that forced the recall curve into
long form, since a wide arm-by-k grid has no room for error bars. The report
names which of the three stopping conditions fired, because an arm that won
on measurement, a round that ran out of corpus, and a round that ran out of
budget are three different verdicts and collapsing them into one is how a
tie gets reported as a win. And the summary file replaces infinite bounds
with null, since JSON has no infinity and a file only one language can read
back is not an auditable artifact.

VERDICT.md is deliberately pending. The paid confirmation has not run, so it
states the rule, the three stop reasons and the 424-entry shelf ceiling, and
lists every absent number beside the file that will produce it, rather than
estimating any of them. The ceiling is the load-bearing part: no corpus size
buys more than 424 independent things to be right about, so if the arms do
not separate the honest recommendation is a larger catalog, not a larger
corpus.

Tests are unittest, matching the rest of round 3, and parse the rendered
HTML rather than grepping it, since a grep passes on a caption that merely
mentions an interval.
2026-07-31 22:10:15 +02:00
Miguel Angel Simon Sierra 9212ed472d feat(eval): add the round 3 offline arm sweep
sweep.py ranks every retrieval arm on recall alone, reading the decision rule
from DECISION-RULE.md rather than carrying its own copy, so the branch it takes
is the branch that was pre-registered. It refuses to run against a rule file
whose machine-readable block is missing, incomplete, or says an exploratory arm
may win.

weighted_hybrid exists because harness2.hybrid_topk hardcodes an equal-weight
fusion; a test pins weighted_hybrid(0.5) to that exact list so the new code is
a superset rather than a second implementation.
2026-07-31 21:59:44 +02:00
Miguel Angel Simon Sierra 01ea003aa4 feat(evals): sweep every retrieval arm and apply the pre-registered rule
Rank the decision-tier arms on offline recall with intervals clustered by
target move, read the branch parameters out of DECISION-RULE.md rather than
transcribing them, and price the recall knee against round 2's measured
tokens-per-k slope.
2026-07-31 21:39:15 +02:00
Miguel Angel Simon Sierra e46824596d feat(eval): cluster every interval by target move
build_report2.aggregate returned bare means with no interval at all, so
round 2's headline table had no error bars. Briefs written for the same
target move are correlated, so a per-brief interval understates the spread
and a larger corpus would have narrowed it wrongly.

Intervals are now cluster robust, collapsing onto the ordinary standard
error when every cluster holds one observation. Round 2's effective sample
comes out near 199 rather than 227.
2026-07-31 21:26:46 +02:00
Miguel Angel Simon Sierra 59abcdb41c feat(eval): quote the batch tier beside the interactive one at the gate
The committee is the dominant cost of a scaled round and it has no latency
requirement, so the tier is a real decision rather than a detail. Pricing
only the interactive tier overstates the round by about two times, which
could talk the corpus size down for no reason.
2026-07-31 21:16:03 +02:00
Miguel Angel Simon Sierra 3596830187 Merge commit '16ad23020' into feat-video-primitives 2026-07-31 21:13:52 +02:00
Miguel Angel Simon Sierra 57a6687511 Merge branch 'u4-per-move-allocation' into feat-video-primitives 2026-07-31 21:13:51 +02:00
Miguel Angel Simon Sierra 94fadadf25 fix(eval): stop the hybrid arm from answering as lexical when vectors are gone
hybrid_topk fell back to the lexical list whenever vectors.json was missing
or the semantic ranking came back empty. That never fired during the round 2
grid because the vectors were present, so it shipped unnoticed. It fires
constantly in an offline sweep, where it made hybrid print lexical's exact
recall curve under its own name. An arm comparison built on that reads as a
tie between two arms rather than as a missing input.

Hybrid now exits loudly, the way semantic already did. The offline sweep
also runs arms one at a time, so an arm that needs vectors going quiet no
longer silences lexical, which needs none.
2026-07-31 21:13:36 +02:00
Miguel Angel Simon Sierra 16ad230207 feat(eval): price the gold committee before it runs
Make the committee sample rate an explicit parameter, project its cost
from context lengths alone under EVAL_DRY_RUN=1, and reuse run2's spend
ceiling, kill file and checkpoint resume so a corpus-scale audit stops
on a known number instead of discovering it. A rate below round 2's 20
percent now names the widened error bound out loud, and a rate of 0
refuses rather than shipping an unaudited corpus in silence.
2026-07-31 21:11:03 +02:00
Miguel Angel Simon Sierra dfd3a80d3a feat(eval): allocate the brief corpus per move, in waves
Round 2 split briefs equally across the three source types. At round 3
volume that would put roughly 700 briefs on 25 primitive moves, which at
the measured intra-cluster correlation of 0.33 carry about as much
information as 3 independent ones.

choose_targets now takes a wave number and brings every catalog move up
to that many generation attempts, so cluster sizes stay uniform.
Attempts are counted rather than acceptances, which is what makes a
killed wave resumable without backfilling a move the gates rejected.
wave_report prints per-source counts, the per-move count distribution
and the unbackfilled shortfall.
2026-07-31 21:11:03 +02:00
Miguel Angel Simon Sierra 4feadbc7cc Merge branch 'eval-round3-u2' into feat-video-primitives 2026-07-31 21:10:31 +02:00
Miguel Angel Simon Sierra 41fb8647cb fix(eval): correct a misquoted round 2 semantic recall figure
Semantic recall at k=10 is 0.383, not 0.282. The 0.282 figure is semantic
k=5, which collides with lexical k=10 and was misread from the cell table.
The refusal citation now names the arm it came from.

The rule itself is unchanged. Only the background citations move, and the
correction strengthens the convergence case rather than weakening it.
2026-07-31 21:10:16 +02:00
Miguel Angel Simon Sierra 2c6419fc88 feat(eval): offline recall harness proven against the round 2 paid grid 2026-07-31 21:06:29 +02:00
Miguel Angel Simon Sierra 1a4378737f feat(eval): pre-register the round 3 arm decision rule 2026-07-31 20:58:40 +02:00
Miguel Angel Simon Sierra 42985dc506 test(eval): freeze round 2's per-brief recall as an offline fixture 2026-07-31 20:56:04 +02:00
Miguel Angel Simon Sierra 6a9f4f4b50 feat(eval): record the round 2 verdict, summary and report 2026-07-31 20:52:34 +02:00
Miguel Angel Simon Sierra e0e532f56d feat(eval): report tokens and dollars per cell alongside fit 2026-07-31 16:58:44 +02:00
Miguel Angel Simon Sierra 6bfa66d2d1 feat(eval): compare cells on the same briefs, not the same column
fit when shown is conditioned on retrieval succeeding, and the briefs a short
list finds are the easy ones, so reading that column across cells compares
different brief sets and can invent an effect that is only difficulty. The
report now restricts the full shelf to each cell's own gold-shown briefs.
2026-07-31 16:42:49 +02:00
Miguel Angel Simon Sierra 5e04109a11 feat(eval): split retrieval misses from selection misses in the report
A low fit reads as the model choosing badly when the real story is often that
retrieval never showed it the answer, and those two call for opposite fixes.
Each cell now reports how often the gold move was in the list the run could
see, and its fit on just those runs.
2026-07-31 16:37:37 +02:00
Miguel Angel Simon Sierra 65949f60e0 feat(eval): aggregate the round-two grid into a report
Round one printed one card per run, which does not survive 2043 of them. This
aggregates per cell and per stratum first and shows individual runs only for
failures, keeping mountable and fit as two separate columns. It reads the
checkpoint log alone, so it runs against a grid that is still going.
2026-07-31 16:32:55 +02:00
Miguel Angel Simon Sierra 4bce4ce0ce fix(eval): dry run banner stated 50 output tokens while billing 800
The banner a reader checks before approving spend disagreed with the
arithmetic under it by a factor of sixteen, understating the output half of
every projection. The printed assumption now interpolates the constant, and a
regression pins the two together.
2026-07-31 16:09:41 +02:00
Miguel Angel Simon Sierra 7759acc5ea perf(eval): memoize the semantic ranking per brief
semantic_topk re-read and re-scored vectors.json on every call. The grid asks
for a semantic ranking 227 briefs x 4 cells, so a tens-of-megabytes parse and a
424-move cosine sweep ran roughly 900 times. The parse and the ranking are now
memoized, keyed by the vectors path so the tests that repoint VECTORS still see
their own file.
2026-07-31 16:02:43 +02:00
Miguel Angel Simon Sierra 6a2b287a2f feat(eval): drop briefs the blind committee cannot reconstruct
The gold sets are right by construction, but at 424 entries some moves are
near twins, so 45 of 272 briefs describe a beat that two moves answer equally
well. Three independent full-shelf readers reconstructed 227, below the 95
percent floor, and the grid stayed blocked.

Moves those 45 briefs and their gold into excluded/ rather than deleting them,
so the drop is auditable and reversible, and leaves gold purely constructed.
The kept corpus is easier than the generated one and its block stratum falls
from 79 to 53, which the verdict has to state.
2026-07-31 15:41:39 +02:00
Miguel Angel Simon Sierra 4ee6694be4 fix(eval): read round two briefs from round two
The runner resolved the brief directory one level up, where round one's five
briefs live. They load without error, so the whole nine cell grid would have
scored against the wrong corpus. The generator and the gold committee already
resolved it correctly, so the three stages now assert they agree.

Commits the 272 brief corpus, its gold sets and the committee audit so the
experiment is reproducible from the repository rather than from one machine.
2026-07-31 15:37:11 +02:00
Miguel Angel Simon Sierra ddcf5c0636 fix(evals): regenerate the shelf and index for the twelve merged pairs
Two moves that shipped as registry components became primitive and component
collisions, so the catalog now merges twelve pairs instead of ten. Regenerated
rather than hand edited: the shelf suite asserts these bytes come back out of
the builder unchanged.
2026-07-31 12:47:37 +02:00
Miguel Angel Simon Sierra 2372c68ba3 fix(evals): keep the formatter off generated catalog artifacts
The pre-commit formatter reformatted the staged shelf and index and re-staged
them, so the committed catalog no longer matched what the builder emits and the
regeneration assertion failed on a fresh checkout. Their generator owns those
bytes, so the formatter now skips them.
2026-07-31 12:45:17 +02:00
Miguel Angel Simon Sierra 75e5d6cd4f fix(evals): regenerate the shelf and index against the pinned collision set
The previous commit carried the source change without the artifacts it
produces, so the checked-in shelf still merged ten primitive and component
pairs while the builder now merges twelve. Regeneration is asserted byte for
byte, so a stale artifact fails the shelf suite rather than silently scoring
the eval against a catalog nobody can rebuild.
2026-07-31 12:43:02 +02:00
Miguel Angel Simon Sierra d866d092dd fix(evals): resolve shelf sources from the vendored path in tests too
The shelf builder reads primitive artifacts from evals/catalog-retrieval/sources,
but the test fixture still exported them to the old scratch layout, so the
negative controls built against a repo that had no primitive sources at all.
The location is now one named constant both sides resolve against.

Two moves that shipped as registry components in the previous commit,
caption-camera-follow and grade-split-reveal, are now real primitive plus
component collisions, so the pinned collision set and the regenerated shelf
account for them.
2026-07-31 12:36:02 +02:00
Miguel Angel Simon Sierra bb912587b9 feat(primitives): add 25 video primitives, live docs pages, and the catalog retrieval eval
Adds the 25 video primitive compositions and their generated Mintlify pages.
Each page embeds a live <hyperframes-player> preview with editable composition
variables rather than a rendered MP4.

Adds the catalog-browsing eval under evals/catalog-retrieval: a 424-entry shelf
built from HEAD, nine retrieval conditions (no-catalog control, full read,
lexical, semantic, and hybrid across several values of k), a brief generator
that constructs each brief from a known target, a blind three-pass gold
validation committee, and a mechanical scorer that keeps mountable and fit as
two separate numbers.

The eval vendors its own shelf, primitive sources, and preview list instead of
reading them from an ignored working directory, so the shelf regenerates byte
for byte from a clean checkout. Derived artifacts stay out: the embedding cache
and the run checkpoints are regenerated rather than committed, so no run can
score against a stale copy of either.

Lint ignores now cover vendored minified player bundles by pattern instead of
one path at a time.
2026-07-31 12:27:41 +02:00
Miguel Angel Simon Sierra cda1203e4f feat(registry): wave M experiments - ten candidates on the watch wall
ascii-render-pass, ordered-dither-pass, segmentation-flood, slit-scan-reveal,
variable-font-flex, stitched-text-draw, ink-bleed-reveal, particle-text-dissolve,
modal-morph, stop-motion-cadence; all tri-order seek-deterministic; watch page
docs/experiments.html; candidates only, no catalog changes.
2026-07-18 21:48:34 +02:00
Miguel Angel Simon Sierra 39d2e4caa8 feat(registry): wave L wild-shelf candidates from the frame study
cursor-glyph-trail, echo-trail, facet-morph, light-sweep-pass,
soft-blob-touch, halftone-dissolve; all pure functions of timeline time,
tri-order seek determinism verified; watch page at docs/wild-shelf.html.
Candidates pending owner judgment, no catalog changes.
2026-07-18 08:47:20 +02:00
Miguel Angel Simon Sierra c996588d8c feat(registry): wave K - eight demand-backed primitives, shelf recomposed to 25
scroll-camera-story, iris-reveal, particle-image-reveal, telemetry-hud,
native-notification-pop, whip-pan-cut, spring-stack-shuffle, vox-annotate;
interruptible-spring and quiet-register laws encoded; marker-highlight
(absorbed by vox-annotate), constellation-hub, and typed-prompt leave the
hero catalog; scroll-feed, svg-stroke-trace, oversized-cursor protected.
2026-07-16 18:04:21 -04:00
Miguel Angel Simon Sierra 2ded4d09bd fix(registry): adversarial QA round two - ten primitives polished
kinetic-type-swap masked exchange + font-relative gaps (was unanimous
below bar); titlecard-lockup and chart-story light-safe inks; press-ripple
cursor exit, ink ripples, transform-only; before-after-wipe divider fade-in;
scroll-feed doomscroll default + gradient robustness + edge fades;
browser-device-stage fade-through swap; count-up affix optical trim;
per-word-rise landing settle; grid-card-assemble pacing spread; true-still
holds replace sub-perceptual drift shelf-wide.
2026-07-16 17:04:07 -04:00
Miguel Angel Simon Sierra 275b333195 feat(registry): quality round - redesigned closes and grid, demo mount overhaul, trust-strip demoted
cta-close and logo-brand-close rebuilt at display scale with real presence;
grid-card-assemble tiles become drawn-icon capability cards; all nine
generated demo mounts rebuilt full-frame on light token backdrops;
trust-strip leaves the hero shelf (too easily one-shot to hold a slot);
dash draws moved to attribute-driven offsets (reverse-seek stale paint).
2026-07-16 12:14:55 -04:00
Miguel Angel Simon Sierra 2b810e3518 feat(registry): wave J2 - ten workhorse primitives, helpers doc, hero catalog
screen-flow-carousel, ui-focus-zoom, state-chip-rail, morph-swap, line-swap,
radial-surround, code-terminal-run, testimonial-proof-card, notification-stack,
split-tilt-cards; docs/primitives-helpers.md (camera-servo + hold-alive);
registry/components/CATALOG.md (hero-21 index with use_when/avoid_when/pairs_with);
before-after-wipe demo accent moved off the variable-shadow collision.
2026-07-16 11:07:19 -04:00
Miguel Angel Simon Sierra f7da9e444c feat(registry): wave J shelf recomposition
Eight new hero primitives (press-ripple, typed-prompt, titlecard-lockup,
logo-brand-close, browser-device-stage, grid-card-assemble, chart-story,
trust-strip), before-after-wipe port with content slots, marker-highlight
contract rebuild, whiteboard-ink strokes slot, and the cues/exit/register
retrofit across the ten kept primitives. Gallery, theme gate, and
theme-compare board restaged for the shelf of 21.
2026-07-16 03:18:41 -04:00
Miguel Angel Simon Sierra afa3de94c6 fix(registry): whiteboard-ink choreography and SVG hidden-attribute selection 2026-07-16 00:11:50 -04:00
Miguel Angel Simon Sierra c29e936239 feat(registry): whiteboard-ink on the theme compare board 2026-07-15 23:36:27 -04:00
Miguel Angel Simon Sierra f5101a5a43 feat(registry): whiteboard-ink primitive for whiteboard-style explainer drawing 2026-07-15 23:34:06 -04:00
Miguel Angel Simon Sierra 6ae6679c6b fix(registry): coverage pipe-character row loss, candidate name mismatches, hero staging 2026-07-15 23:21:14 -04:00
Miguel Angel Simon Sierra 91840e9038 feat(registry): wave I definition-perfect rebuilds (7 units) and hero board update 2026-07-15 22:56:29 -04:00
Miguel Angel Simon Sierra abdeebf848 fix(theme-gate): partial runs no longer wipe previously staged variants 2026-07-15 22:26:56 -04:00