feat(core): percentage-based canary rollouts

Adds a reusable staged-rollout primitive so a change can ship to a stable
slice of installs instead of all-or-nothing.

The gap it fills: the repo carries ~49 HF_*/PRODUCER_* booleans and every one
is binary — a feature is either off (and therefore unexercised on real
traffic) or on for everyone (and therefore a fleet-wide bet). The
parallel-drawElement router sat in that gap for weeks: default-off produced
almost no signal, and flipping it default-on would have exposed 100% of
eligible installs at once.

Shape:

- packages/core/src/canary.ts — pure evaluator. No fs, no network, no
  `process`; the caller supplies the unit id and overrides, so it imports
  cleanly into the CLI, producer, engine, studio-server, the browser-side
  studio bundle and the embeddable player. FNV-1a rather than node:crypto for
  the same reason.
- packages/core/src/canaryRegistry.ts — every rollout in one table (name,
  percentage, owner, description, sunsetAfter), so "what is rolling out, to
  whom, owned by whom" is answerable without grepping 49 env vars.
- packages/cli/src/telemetry/canary.ts — supplies the three things only the
  CLI knows: anonymousId, the HF_CANARY_<FEATURE> override, and is_ci.
  Day-to-day API is `isCanaryEnabled("name")`.

Three properties the tests pin, because getting them wrong is subtle:

- Slices are INDEPENDENT per feature: the bucket hashes `feature:unitId`, not
  the id alone. Bucketing on the id would hand every concurrent experiment to
  the same unlucky cohort and make two rollouts unreadable apart.
- Ramping is INCLUSIVE: `bucket < percentage`, so widening 10 -> 25 keeps the
  original cohort and before/after comparisons survive the ramp.
- It fails CLOSED: no unit id, unknown name, or CI install means not enrolled.
  A canary exists to bound blast radius, so "we don't know who this is" must
  never mean "enrol everyone".

Registry entries also carry a sunset date, and a test fails once one is past
due — a canary that outlives its rollout is a permanent fork of the product
with none of the review a permanent fork would get.

Ships with de-parallel-router registered at 0%: inert, and ready to ramp in a
patch release once #2840 lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-07-30 15:13:56 -07:00
co-authored by Claude Opus 5
parent fa564547dc
commit 3aea786687
6 changed files with 601 additions and 0 deletions
+77
View File
@@ -0,0 +1,77 @@
/**
* The canary registry — every staged rollout in the product, in one file.
*
* Why a registry rather than a percentage inlined at each call site: the repo
* already carries ~49 loose `HF_*` / `PRODUCER_*` toggles with no index, so
* nobody can answer "what is currently rolling out, to how many people, and
* who owns it" without grepping. One table fixes that, and gives the
* telemetry and `doctor` surfaces something to enumerate.
*
* ## Adding one
*
* 1. Add an entry below. Start at `percentage: 0` and merge that — a canary
* at 0 is dead code you can land safely and ramp without a code review.
* 2. Read it at the decision point via the surface's binding (in the CLI,
* `isCanaryEnabled("your-feature")`).
* 3. Ramp by editing `percentage` in a patch release: 0 → 5 → 25 → 100.
* Widening is inclusive, so the earlier cohort stays enrolled and the
* before/after comparison survives the ramp.
* 4. At 100 and holding, DELETE the entry and the branch it guarded. That is
* the point of `sunsetAfter`.
*
* ## Overriding
*
* `HF_CANARY_<FEATURE>` with the feature name upper-snake-cased, e.g.
* `HF_CANARY_DE_PARALLEL_ROUTER=on` (also: off/true/false/1/0/yes/no).
* An override always wins over the percentage, in both directions.
*/
export interface CanaryDefinition {
/** Registry key. Kebab-case; also the hash input, so renaming reshuffles the cohort. */
name: string;
/** 0-100. Start at 0, ramp in patch releases. */
percentage: number;
/** What turning this on actually changes, in one line. */
description: string;
/** Who to ask. */
owner: string;
/**
* ISO date after which this canary is overdue for removal. A canary that
* outlives its rollout is a permanent fork of the product with none of the
* review a permanent fork would have received. `assertNoOverdueCanaries`
* turns the date into a failing test rather than a good intention.
*/
sunsetAfter: string;
}
export const CANARIES: readonly CanaryDefinition[] = [
{
name: "de-parallel-router",
percentage: 0,
description:
"Route auto multi-worker renders to verified parallel drawElement streaming (HF_DE_PARALLEL_ROUTER). Ramp only alongside the per-install circuit breaker.",
owner: "vance",
sunsetAfter: "2026-10-01",
},
] as const;
export function findCanary(name: string): CanaryDefinition | undefined {
return CANARIES.find((c) => c.name === name);
}
/** Env-var name for a feature's manual override: `de-parallel-router` → `HF_CANARY_DE_PARALLEL_ROUTER`. */
export function canaryEnvVar(name: string): string {
return `HF_CANARY_${name.toUpperCase().replace(/[^A-Z0-9]+/g, "_")}`;
}
/**
* Names of canaries whose sunset date has passed — either finish the rollout
* and delete the entry, or push the date with a reason. Exposed as a function
* (not a lint rule) so the check runs in the normal test suite.
*/
export function overdueCanaries(now: Date = new Date()): string[] {
return CANARIES.filter((c) => {
const sunset = Date.parse(`${c.sunsetAfter}T00:00:00Z`);
return Number.isFinite(sunset) && now.getTime() > sunset;
}).map((c) => c.name);
}