mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
Deletes the `de-parallel-router` canary entry and the `isCanaryEnabled` guard in render.ts together, leaving the producer's default-ON in place. Net effect for users: the parallel drawElement router is on for everyone again. ## Why, and why not a ramp Gating at 5% was itself the regression. Measured 2026-08-08, the day after v0.7.101 shipped the canary: fleet router exposure fell from 3.13-4.25% of non-CI renders to **0.13%**, roughly 25x, because out-of-cohort installs are explicitly disarmed and #2840 deleted the everyone-armed trial in the same change. 2,537 installs lost a feature they already had. Severity is speed only, never output, and nothing is persisted to disk. PR #2840's body claimed "the canary does not make exposure smaller; it makes it chosen and revertible." That was true of the end state and false of the first step. This lands the end state. Entry and guard go together deliberately: at >=100 the evaluator short-circuits ahead of the CI/seedless exclusions, so removing only the entry would have flipped whatever still resolved false at deletion time, unstaged. ## Both stated blockers are void - **≤4-CPU / Docker coverage gap.** Docker renders never use drawElement — 0 of 4,281 across every CPU tier, software GL gates it out — and the router requires it. No percentage could ever expose Docker, so no ramp closes that gap. ≤4 CPUs yields ~42 drawElement candidates in three days. - **PRINFRA-372.** Its signature has hits on 0.4.12, 0.4.37, 0.6.52, 0.6.93, 0.6.109 and 0.6.110 — versions predating drawElement (v0.7.38) and therefore this router. It is real, still live on 0.7.101, and belongs to the screenshot/beginframe path. 11 reproduction runs across four configurations on the enriched profile (darwin/arm64 25.5.0) came back clean. ## Safety unchanged The per-install circuit breaker and the per-render self-verify are untouched; `HF_DE_PARALLEL_ROUTER=false` remains the user-facing kill switch. Post-canary data at 14 days: >8 CPUs 3.02% revert (177/5,857), 5-8 CPUs 2.40% (6/250) — consistent with the 2.75-3.16% baseline. Revert path is now a code revert rather than a registry edit. That is the trade this shape accepts in exchange for one release instead of two. ## Corrects two claims that shipped wrong `~17x jump in exposure onto <=4 CPUs / Docker` overstated the reach, and `~11% of installs already route` was an OUTCOME (the share clearing eligibility and the old 25-render cap), not an exposure setting — read as a rollout knob it inverts the arithmetic, which is how gating at 5% came to cut exposure rather than ramp it. Both are recorded in render.ts so they are not reintroduced. ## Tests Removed the core wiring assertion and the two CLI canary-gating tests, which pinned a gate that no longer exists. Added the inverse guarantee in its place: an ordinary install must come out of the breaker with the var UNSET so the producer default applies — writing "false" there is precisely what disarmed the fleet at 5%. core 1701 passing, cli 2491 passing, studio canary 29 passing. The 2 failures in play.test.ts reproduce on clean origin/main and are unrelated (#3114 area). oxlint and oxfmt clean. Note: telemetry for this rollout stops with the entry — `$feature/canary-de-parallel-router` and `canary_reason_de_parallel_router` are emitted from the registry, so the `Ramp —` tiles and the exposure-floor alert on PostHog dashboard 1918875 go blank once this ships. Watch drawElement engagement on 1807532 instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
108 lines
4.9 KiB
TypeScript
108 lines
4.9 KiB
TypeScript
/**
|
|
* The canary registry — every staged rollout in the product, in one file.
|
|
*
|
|
* Why a registry rather than a percentage inlined at each call site: the repo
|
|
* already carries 57 loose `HF_*` / `PRODUCER_*` toggles with no index, so
|
|
* nobody can answer "what is currently rolling out, to how many people, and
|
|
* who owns it" without grepping. One table fixes that, and gives the
|
|
* telemetry and `doctor` surfaces something to enumerate.
|
|
*
|
|
* ## Adding one
|
|
*
|
|
* 1. Add an entry below. Start at `percentage: 0` and merge that — a canary
|
|
* at 0 is dead code you can land safely and ramp without a code review.
|
|
* 2. Read it at the decision point via the surface's binding (in the CLI,
|
|
* `isCanaryEnabled("your-feature")`).
|
|
* 3. Ramp by editing `percentage` in a patch release: 0 → 5 → 25 → 100.
|
|
* Widening is inclusive, so the earlier cohort stays enrolled and the
|
|
* before/after comparison survives the ramp.
|
|
* 4. At 100 and holding, DELETE the entry and the branch it guarded. That is
|
|
* the point of `sunsetAfter`.
|
|
*
|
|
* ## Overriding
|
|
*
|
|
* `HF_CANARY_<FEATURE>` with the feature name upper-snake-cased, e.g.
|
|
* `HF_CANARY_DE_PARALLEL_ROUTER=on` (also: off/true/false/1/0/yes/no).
|
|
* An override always wins over the percentage, in both directions.
|
|
*/
|
|
|
|
export interface CanaryDefinition {
|
|
/** Registry key. Kebab-case; also the hash input, so renaming reshuffles the cohort. */
|
|
name: string;
|
|
/** 0-100. Start at 0, ramp in patch releases. */
|
|
percentage: number;
|
|
/** What turning this on actually changes, in one line. */
|
|
description: string;
|
|
/** Who to ask. */
|
|
owner: string;
|
|
/**
|
|
* ISO date after which this canary is overdue for removal. A canary that
|
|
* outlives its rollout is a permanent fork of the product with none of the
|
|
* review a permanent fork would have received. The scheduled `Canary sunset`
|
|
* workflow runs `scripts/check-canary-sunset.ts` weekly and fails once the
|
|
* date passes, so this is an enforced deadline rather than a good intention.
|
|
*/
|
|
sunsetAfter: string;
|
|
}
|
|
|
|
export const CANARIES: readonly CanaryDefinition[] = [
|
|
// ── Calibration ──────────────────────────────────────────────────────────
|
|
// Two INERT canaries that gate nothing. They exist to validate the rollout
|
|
// mechanism against real traffic before anything real depends on it, and
|
|
// they answer questions the synthetic tests cannot:
|
|
//
|
|
// 1. Does a requested percentage land on target in the wild? The unit
|
|
// tests use generated UUIDs and weight every install equally; real
|
|
// render volume is heavily skewed toward a few heavy installs, so the
|
|
// render-weighted share could differ from the install-weighted one.
|
|
// 2. How fast does CUMULATIVE exposure drift above the target? Install
|
|
// ids churn (measured: 24.7x more distinct ids over 30 days than in
|
|
// any single day), so the set of installs enrolled AT SOME POINT grows
|
|
// even though the instantaneous share stays flat. That drift is the
|
|
// real limit on a canary's blast-radius guarantee.
|
|
// 3. Are two canaries actually independent on real ids, not just on
|
|
// generated ones? Overlap should be ~p1*p2, not ~min(p1,p2).
|
|
//
|
|
// Two different percentages so the answer is a line, not a point.
|
|
// Delete both once the calibration window is read.
|
|
{
|
|
name: "calibration-10",
|
|
percentage: 10,
|
|
description: "Inert. Validates rollout accuracy and cumulative-exposure drift at 10%.",
|
|
owner: "vance",
|
|
sunsetAfter: "2026-09-15",
|
|
},
|
|
{
|
|
name: "calibration-50",
|
|
percentage: 50,
|
|
description:
|
|
"Inert. Second calibration point, and an independence check against calibration-10.",
|
|
owner: "vance",
|
|
sunsetAfter: "2026-09-15",
|
|
},
|
|
] as const;
|
|
|
|
export function findCanary(name: string): CanaryDefinition | undefined {
|
|
return CANARIES.find((c) => c.name === name);
|
|
}
|
|
|
|
/** Env-var name for a feature's manual override: `de-parallel-router` → `HF_CANARY_DE_PARALLEL_ROUTER`. */
|
|
export function canaryEnvVar(name: string): string {
|
|
return `HF_CANARY_${name.toUpperCase().replace(/[^A-Z0-9]+/g, "_")}`;
|
|
}
|
|
|
|
/**
|
|
* Names of canaries whose sunset date has passed — either finish the rollout
|
|
* and delete the entry, or push the date with a reason. Exposed as a function
|
|
* (not a lint rule) so the check runs in the normal test suite.
|
|
*/
|
|
export function overdueCanaries(now: Date = new Date()): string[] {
|
|
return CANARIES.filter((c) => {
|
|
// End of the sunset day, not its start. The field is documented as the
|
|
// date AFTER which a canary is overdue, but comparing against midnight UTC
|
|
// made it overdue ON that date — and earlier still for anyone west of UTC.
|
|
const sunsetEnd = Date.parse(`${c.sunsetAfter}T23:59:59.999Z`);
|
|
return Number.isFinite(sunsetEnd) && now.getTime() > sunsetEnd;
|
|
}).map((c) => c.name);
|
|
}
|