feat(core): inert calibration canaries to validate the mechanism in the wild

Registers two canaries that gate nothing — `calibration-10` (10%) and
`calibration-50` (50%) — so the rollout mechanism can be proven against real
traffic before any real feature depends on it. Zero behavioural risk: they are
read by nothing.

They answer what the unit tests structurally cannot. The tests bucket
generated UUIDs and weight every install equally; real render volume is
heavily skewed toward a few heavy installs, and real install ids churn (~25x
more distinct ids over 30 days than in any single day on the desktop render
population).

Four checks, pre-registered in the docs so the read is not post-hoc:

1. ACCURACY — does 10% land at 10%, install-weighted AND event-weighted?
2. DRIFT — how fast does CUMULATIVE exposure climb above target as ids churn?
   The instantaneous share is flat by construction; the set of installs
   enrolled at some point is not.
3. STABILITY — does any install ever change cohort? Must be zero. Percentages
   are held FIXED for the window precisely so a flip is unambiguously a bug;
   during a real ramp a false->true flip would be correct instead.
4. CROSS-SURFACE — do the CLI and Studio bindings agree for the same install?
   A CLI-launched Studio adopts the CLI id, and 16,961 installs currently
   share an id across both surfaces, so this is measurable.

Plus an independence check: overlap between the two calibration canaries
should be ~p1*p2 (~5%), not ~min(p1,p2) (~10%, which would mean every canary
lands on the same unlucky cohort).

The docs also record what calibration CANNOT fix: per-install cohorts never
flip, but a person who wipes their config gets a new id and a fresh roll.
Preventing that needs stable identity across resets, and both candidates were
rejected — hardware fingerprinting correlates the cohort with hardware (fatal
for a rendering experiment, and it survives uninstall) and account identity
covers only ~3.6% of local rendering installs. The drift is therefore a
measured, accepted limit, and the point of calibrating is to size it and pick
canary window lengths accordingly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-07-30 15:14:59 -07:00
co-authored by Claude Opus 5
parent a1682e1228
commit a7bb061afe
2 changed files with 163 additions and 0 deletions
+35
View File
@@ -45,6 +45,41 @@ export interface CanaryDefinition {
}
export const CANARIES: readonly CanaryDefinition[] = [
// ── Calibration ──────────────────────────────────────────────────────────
// Two INERT canaries that gate nothing. They exist to validate the rollout
// mechanism against real traffic before anything real depends on it, and
// they answer questions the synthetic tests cannot:
//
// 1. Does a requested percentage land on target in the wild? The unit
// tests use generated UUIDs and weight every install equally; real
// render volume is heavily skewed toward a few heavy installs, so the
// render-weighted share could differ from the install-weighted one.
// 2. How fast does CUMULATIVE exposure drift above the target? Install
// ids churn (measured: 24.7x more distinct ids over 30 days than in
// any single day), so the set of installs enrolled AT SOME POINT grows
// even though the instantaneous share stays flat. That drift is the
// real limit on a canary's blast-radius guarantee.
// 3. Are two canaries actually independent on real ids, not just on
// generated ones? Overlap should be ~p1*p2, not ~min(p1,p2).
//
// Two different percentages so the answer is a line, not a point.
// Delete both once the calibration window is read.
{
name: "calibration-10",
percentage: 10,
description: "Inert. Validates rollout accuracy and cumulative-exposure drift at 10%.",
owner: "vance",
sunsetAfter: "2026-09-15",
},
{
name: "calibration-50",
percentage: 50,
description:
"Inert. Second calibration point, and an independence check against calibration-10.",
owner: "vance",
sunsetAfter: "2026-09-15",
},
// ── Real rollouts ────────────────────────────────────────────────────────
{
name: "de-parallel-router",
percentage: 0,