mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-10 22:20:14 +00:00
feat(core): inert calibration canaries to validate the mechanism in the wild
Registers two canaries that gate nothing — `calibration-10` (10%) and `calibration-50` (50%) — so the rollout mechanism can be proven against real traffic before any real feature depends on it. Zero behavioural risk: they are read by nothing. They answer what the unit tests structurally cannot. The tests bucket generated UUIDs and weight every install equally; real render volume is heavily skewed toward a few heavy installs, and real install ids churn (~25x more distinct ids over 30 days than in any single day on the desktop render population). Four checks, pre-registered in the docs so the read is not post-hoc: 1. ACCURACY — does 10% land at 10%, install-weighted AND event-weighted? 2. DRIFT — how fast does CUMULATIVE exposure climb above target as ids churn? The instantaneous share is flat by construction; the set of installs enrolled at some point is not. 3. STABILITY — does any install ever change cohort? Must be zero. Percentages are held FIXED for the window precisely so a flip is unambiguously a bug; during a real ramp a false->true flip would be correct instead. 4. CROSS-SURFACE — do the CLI and Studio bindings agree for the same install? A CLI-launched Studio adopts the CLI id, and 16,961 installs currently share an id across both surfaces, so this is measurable. Plus an independence check: overlap between the two calibration canaries should be ~p1*p2 (~5%), not ~min(p1,p2) (~10%, which would mean every canary lands on the same unlucky cohort). The docs also record what calibration CANNOT fix: per-install cohorts never flip, but a person who wipes their config gets a new id and a fresh roll. Preventing that needs stable identity across resets, and both candidates were rejected — hardware fingerprinting correlates the cohort with hardware (fatal for a rendering experiment, and it survives uninstall) and account identity covers only ~3.6% of local rendering installs. The drift is therefore a measured, accepted limit, and the point of calibrating is to size it and pick canary window lengths accordingly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
a1682e1228
commit
a7bb061afe
@@ -45,6 +45,41 @@ export interface CanaryDefinition {
|
||||
}
|
||||
|
||||
export const CANARIES: readonly CanaryDefinition[] = [
|
||||
// ── Calibration ──────────────────────────────────────────────────────────
|
||||
// Two INERT canaries that gate nothing. They exist to validate the rollout
|
||||
// mechanism against real traffic before anything real depends on it, and
|
||||
// they answer questions the synthetic tests cannot:
|
||||
//
|
||||
// 1. Does a requested percentage land on target in the wild? The unit
|
||||
// tests use generated UUIDs and weight every install equally; real
|
||||
// render volume is heavily skewed toward a few heavy installs, so the
|
||||
// render-weighted share could differ from the install-weighted one.
|
||||
// 2. How fast does CUMULATIVE exposure drift above the target? Install
|
||||
// ids churn (measured: 24.7x more distinct ids over 30 days than in
|
||||
// any single day), so the set of installs enrolled AT SOME POINT grows
|
||||
// even though the instantaneous share stays flat. That drift is the
|
||||
// real limit on a canary's blast-radius guarantee.
|
||||
// 3. Are two canaries actually independent on real ids, not just on
|
||||
// generated ones? Overlap should be ~p1*p2, not ~min(p1,p2).
|
||||
//
|
||||
// Two different percentages so the answer is a line, not a point.
|
||||
// Delete both once the calibration window is read.
|
||||
{
|
||||
name: "calibration-10",
|
||||
percentage: 10,
|
||||
description: "Inert. Validates rollout accuracy and cumulative-exposure drift at 10%.",
|
||||
owner: "vance",
|
||||
sunsetAfter: "2026-09-15",
|
||||
},
|
||||
{
|
||||
name: "calibration-50",
|
||||
percentage: 50,
|
||||
description:
|
||||
"Inert. Second calibration point, and an independence check against calibration-10.",
|
||||
owner: "vance",
|
||||
sunsetAfter: "2026-09-15",
|
||||
},
|
||||
// ── Real rollouts ────────────────────────────────────────────────────────
|
||||
{
|
||||
name: "de-parallel-router",
|
||||
percentage: 0,
|
||||
|
||||
Reference in New Issue
Block a user