feat(studio): emit the canary reason on Studio events too

Review caught that the same anti-pattern was still live in the Studio binding:
canaryEventProperties destructured only `enabled` and dropped the reason. Its
own doc comment promised 'identical shape to the CLI, so a rollout spanning
both reads as one flag' — which the CLI-only fix had just made false.

This matters beyond symmetry. A CLI-launched Studio adopts the CLI's decisions
and shares its bucket seed, so a cohort flip can surface on either surface.
Emitting attribution on only one leaves Studio-observed flips unattributable
and makes the two flip counts irreconcilable — and Studio is the surface most
likely to expose a shared-seed-with-diverging-id pattern, which is the open
question the reason exists to answer.

Also adds the no_unit_id emission test the CLI side advertised but never
asserted, and a Studio pair pinning that a URL override and a cohort roll
produce the same assignment with different reasons.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-08-06 14:39:56 -07:00
co-authored by Claude Opus 5
parent 076657a639
commit b3990ac789
3 changed files with 47 additions and 4 deletions
@@ -194,6 +194,8 @@ describe("telemetry", () => {
expect(canaryEventProperties()).toEqual({
"$feature/canary-on-everywhere": "true",
"$feature/canary-off-everywhere": "false",
canary_reason_on_everywhere: "in_cohort",
canary_reason_off_everywhere: "out_of_cohort",
});
__resetStudioCanaryCacheForTests();
@@ -235,6 +237,8 @@ describe("telemetry opt-out is canary opt-out", () => {
expect(canaryEventProperties()).toEqual({
"$feature/canary-on-everywhere": "false",
"$feature/canary-off-everywhere": "false",
canary_reason_on_everywhere: "telemetry_opt_out",
canary_reason_off_everywhere: "telemetry_opt_out",
});
});
});
@@ -323,3 +327,24 @@ describe("CLI-launched Studio adopts the CLI's decisions", () => {
});
});
});
// A CLI-launched Studio shares the CLI's bucket seed, so a cohort flip can
// surface on either surface. Attribution on only one of them makes the two
// flip counts irreconcilable — which is why this is not a CLI-only property.
describe("Studio attribution matches the CLI", () => {
it("distinguishes a local URL override from a cohort roll at the same value", () => {
setSearch("?hf_canary_off_everywhere=on");
const props = canaryEventProperties();
expect(props["$feature/canary-off-everywhere"]).toBe("true");
expect(props["canary_reason_off_everywhere"]).toBe("forced_on");
// Same assignment `on-everywhere` reaches by an ordinary roll.
expect(props["$feature/canary-on-everywhere"]).toBe("true");
expect(props["canary_reason_on_everywhere"]).toBe("in_cohort");
});
it("never puts a reason inside the $feature namespace", () => {
for (const [key, value] of Object.entries(canaryEventProperties())) {
if (key.startsWith("$feature/")) expect(value).toMatch(/^(true|false)$/);
}
});
});