Files
hyperframes/evals
Miguel Angel Simon Sierra 842286a688 fix(evals): rank each arm at its knee, never at a shelf-length list
Recall rises with k by construction, so ranking each arm at its highest
recall selected whatever the longest swept k happened to be. Extending the
sweep past k=80 made the failure total: at k equal to the shelf size every
arm shows the model all 424 entries and scores exactly 1.0, so the ranking
tied every arm and would have shipped the whole-shelf configuration. That
configuration is round 2's control cell b, which the pre-registered rule
fixes at 0.0 and forbids changing mid-round.

The rule is silent on how k is chosen but already names the right basis,
the point where token cost starts to outrun the recall gain. That is the
knee, which the module already computed and the ranking simply did not
use. So k now comes from the knee, and any k at or past the shelf size is
kept in the reported curve, where it shows the ceiling, and kept out of
the ranking, where it decides nothing.

The report builder ranks on the same basis, so the HTML and the command
line cannot name different winners.
2026-08-01 03:20:55 +02:00
..