Files
hyperframes/packages/core/src/canaryRegistry.ts
T
Vance IngallsandClaude Fable 5 19defeabfe feat(core,cli): ship the DE parallel router fleet-wide — remove the canary gate
Deletes the `de-parallel-router` canary entry and the `isCanaryEnabled` guard
in render.ts together, leaving the producer's default-ON in place. Net effect
for users: the parallel drawElement router is on for everyone again.

## Why, and why not a ramp

Gating at 5% was itself the regression. Measured 2026-08-08, the day after
v0.7.101 shipped the canary: fleet router exposure fell from 3.13-4.25% of
non-CI renders to **0.13%**, roughly 25x, because out-of-cohort installs are
explicitly disarmed and #2840 deleted the everyone-armed trial in the same
change. 2,537 installs lost a feature they already had. Severity is speed
only, never output, and nothing is persisted to disk.

PR #2840's body claimed "the canary does not make exposure smaller; it makes
it chosen and revertible." That was true of the end state and false of the
first step. This lands the end state.

Entry and guard go together deliberately: at >=100 the evaluator
short-circuits ahead of the CI/seedless exclusions, so removing only the entry
would have flipped whatever still resolved false at deletion time, unstaged.

## Both stated blockers are void

- **≤4-CPU / Docker coverage gap.** Docker renders never use drawElement — 0
  of 4,281 across every CPU tier, software GL gates it out — and the router
  requires it. No percentage could ever expose Docker, so no ramp closes that
  gap. ≤4 CPUs yields ~42 drawElement candidates in three days.
- **PRINFRA-372.** Its signature has hits on 0.4.12, 0.4.37, 0.6.52, 0.6.93,
  0.6.109 and 0.6.110 — versions predating drawElement (v0.7.38) and therefore
  this router. It is real, still live on 0.7.101, and belongs to the
  screenshot/beginframe path. 11 reproduction runs across four configurations
  on the enriched profile (darwin/arm64 25.5.0) came back clean.

## Safety unchanged

The per-install circuit breaker and the per-render self-verify are untouched;
`HF_DE_PARALLEL_ROUTER=false` remains the user-facing kill switch. Post-canary
data at 14 days: >8 CPUs 3.02% revert (177/5,857), 5-8 CPUs 2.40% (6/250) —
consistent with the 2.75-3.16% baseline.

Revert path is now a code revert rather than a registry edit. That is the
trade this shape accepts in exchange for one release instead of two.

## Corrects two claims that shipped wrong

`~17x jump in exposure onto <=4 CPUs / Docker` overstated the reach, and
`~11% of installs already route` was an OUTCOME (the share clearing
eligibility and the old 25-render cap), not an exposure setting — read as a
rollout knob it inverts the arithmetic, which is how gating at 5% came to cut
exposure rather than ramp it. Both are recorded in render.ts so they are not
reintroduced.

## Tests

Removed the core wiring assertion and the two CLI canary-gating tests, which
pinned a gate that no longer exists. Added the inverse guarantee in its place:
an ordinary install must come out of the breaker with the var UNSET so the
producer default applies — writing "false" there is precisely what disarmed
the fleet at 5%.

core 1701 passing, cli 2491 passing, studio canary 29 passing. The 2 failures
in play.test.ts reproduce on clean origin/main and are unrelated (#3114 area).
oxlint and oxfmt clean.

Note: telemetry for this rollout stops with the entry — `$feature/canary-de-parallel-router`
and `canary_reason_de_parallel_router` are emitted from the registry, so the
`Ramp —` tiles and the exposure-floor alert on PostHog dashboard 1918875 go
blank once this ships. Watch drawElement engagement on 1807532 instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-08 14:10:25 -07:00

108 lines
4.9 KiB
TypeScript

/**
* The canary registry — every staged rollout in the product, in one file.
*
* Why a registry rather than a percentage inlined at each call site: the repo
* already carries 57 loose `HF_*` / `PRODUCER_*` toggles with no index, so
* nobody can answer "what is currently rolling out, to how many people, and
* who owns it" without grepping. One table fixes that, and gives the
* telemetry and `doctor` surfaces something to enumerate.
*
* ## Adding one
*
* 1. Add an entry below. Start at `percentage: 0` and merge that — a canary
* at 0 is dead code you can land safely and ramp without a code review.
* 2. Read it at the decision point via the surface's binding (in the CLI,
* `isCanaryEnabled("your-feature")`).
* 3. Ramp by editing `percentage` in a patch release: 0 → 5 → 25 → 100.
* Widening is inclusive, so the earlier cohort stays enrolled and the
* before/after comparison survives the ramp.
* 4. At 100 and holding, DELETE the entry and the branch it guarded. That is
* the point of `sunsetAfter`.
*
* ## Overriding
*
* `HF_CANARY_<FEATURE>` with the feature name upper-snake-cased, e.g.
* `HF_CANARY_DE_PARALLEL_ROUTER=on` (also: off/true/false/1/0/yes/no).
* An override always wins over the percentage, in both directions.
*/
export interface CanaryDefinition {
/** Registry key. Kebab-case; also the hash input, so renaming reshuffles the cohort. */
name: string;
/** 0-100. Start at 0, ramp in patch releases. */
percentage: number;
/** What turning this on actually changes, in one line. */
description: string;
/** Who to ask. */
owner: string;
/**
* ISO date after which this canary is overdue for removal. A canary that
* outlives its rollout is a permanent fork of the product with none of the
* review a permanent fork would have received. The scheduled `Canary sunset`
* workflow runs `scripts/check-canary-sunset.ts` weekly and fails once the
* date passes, so this is an enforced deadline rather than a good intention.
*/
sunsetAfter: string;
}
export const CANARIES: readonly CanaryDefinition[] = [
// ── Calibration ──────────────────────────────────────────────────────────
// Two INERT canaries that gate nothing. They exist to validate the rollout
// mechanism against real traffic before anything real depends on it, and
// they answer questions the synthetic tests cannot:
//
// 1. Does a requested percentage land on target in the wild? The unit
// tests use generated UUIDs and weight every install equally; real
// render volume is heavily skewed toward a few heavy installs, so the
// render-weighted share could differ from the install-weighted one.
// 2. How fast does CUMULATIVE exposure drift above the target? Install
// ids churn (measured: 24.7x more distinct ids over 30 days than in
// any single day), so the set of installs enrolled AT SOME POINT grows
// even though the instantaneous share stays flat. That drift is the
// real limit on a canary's blast-radius guarantee.
// 3. Are two canaries actually independent on real ids, not just on
// generated ones? Overlap should be ~p1*p2, not ~min(p1,p2).
//
// Two different percentages so the answer is a line, not a point.
// Delete both once the calibration window is read.
{
name: "calibration-10",
percentage: 10,
description: "Inert. Validates rollout accuracy and cumulative-exposure drift at 10%.",
owner: "vance",
sunsetAfter: "2026-09-15",
},
{
name: "calibration-50",
percentage: 50,
description:
"Inert. Second calibration point, and an independence check against calibration-10.",
owner: "vance",
sunsetAfter: "2026-09-15",
},
] as const;
export function findCanary(name: string): CanaryDefinition | undefined {
return CANARIES.find((c) => c.name === name);
}
/** Env-var name for a feature's manual override: `de-parallel-router` → `HF_CANARY_DE_PARALLEL_ROUTER`. */
export function canaryEnvVar(name: string): string {
return `HF_CANARY_${name.toUpperCase().replace(/[^A-Z0-9]+/g, "_")}`;
}
/**
* Names of canaries whose sunset date has passed — either finish the rollout
* and delete the entry, or push the date with a reason. Exposed as a function
* (not a lint rule) so the check runs in the normal test suite.
*/
export function overdueCanaries(now: Date = new Date()): string[] {
return CANARIES.filter((c) => {
// End of the sunset day, not its start. The field is documented as the
// date AFTER which a canary is overdue, but comparing against midnight UTC
// made it overdue ON that date — and earlier still for anyone west of UTC.
const sunsetEnd = Date.parse(`${c.sunsetAfter}T23:59:59.999Z`);
return Number.isFinite(sunsetEnd) && now.getTime() > sunsetEnd;
}).map((c) => c.name);
}