mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
Vance IngallsandClaude Opus 5
a7bb061afe
Registers two canaries that gate nothing — `calibration-10` (10%) and `calibration-50` (50%) — so the rollout mechanism can be proven against real traffic before any real feature depends on it. Zero behavioural risk: they are read by nothing. They answer what the unit tests structurally cannot. The tests bucket generated UUIDs and weight every install equally; real render volume is heavily skewed toward a few heavy installs, and real install ids churn (~25x more distinct ids over 30 days than in any single day on the desktop render population). Four checks, pre-registered in the docs so the read is not post-hoc: 1. ACCURACY — does 10% land at 10%, install-weighted AND event-weighted? 2. DRIFT — how fast does CUMULATIVE exposure climb above target as ids churn? The instantaneous share is flat by construction; the set of installs enrolled at some point is not. 3. STABILITY — does any install ever change cohort? Must be zero. Percentages are held FIXED for the window precisely so a flip is unambiguously a bug; during a real ramp a false->true flip would be correct instead. 4. CROSS-SURFACE — do the CLI and Studio bindings agree for the same install? A CLI-launched Studio adopts the CLI id, and 16,961 installs currently share an id across both surfaces, so this is measurable. Plus an independence check: overlap between the two calibration canaries should be ~p1*p2 (~5%), not ~min(p1,p2) (~10%, which would mean every canary lands on the same unlucky cohort). The docs also record what calibration CANNOT fix: per-install cohorts never flip, but a person who wipes their config gets a new id and a fresh roll. Preventing that needs stable identity across resets, and both candidates were rejected — hardware fingerprinting correlates the cohort with hardware (fatal for a rendering experiment, and it survives uninstall) and account identity covers only ~3.6% of local rendering installs. The drift is therefore a measured, accepted limit, and the point of calibrating is to size it and pick canary window lengths accordingly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
