feat(core): emit canary assignments as PostHog flag properties

Replaces the single `canaries: "a,b"` telemetry property with PostHog's own
flag shape, one property per registered canary:

    $feature/canary-de-parallel-router: "true" | "false"

PostHog treats `$feature/<key>` as a first-class flag property, so breakdowns,
funnels split by cohort and the experiment surfaces work on a canary with
nothing configured server-side. The decision still happens locally: the render
path forbids render-time network calls, behaviour must not depend on analytics
being reachable, and neither the CLI nor Studio ships posthog-js (both
hand-roll a batch POST, so there is no SDK to evaluate a real flag with).
Decide locally, analyse natively.

Two decisions worth recording:

- BOTH ARMS ARE EMITTED. A non-enrolled install reports "false" rather than
  omitting the property. Absent means "this build predates the canary", which
  is a different fact from "this install is control" — collapsing them makes a
  ramp unreadable, because you cannot separate a control group from an old
  version.

- KEYS ARE NAMESPACED with a `canary-` infix. A real PostHog flag namespace
  already exists in this project, owned by the web app (`enable-chat-tab`, set
  by posthog-js from `$lib=web` events). Namespacing guarantees a canary key
  can never alias a real flag key and have the two fight over one property.

Values are the strings "true"/"false" to match how PostHog records boolean
flag values, so the property is directly comparable to a real flag.

98 core / 1437, 166 cli / 2194, 269 studio / 2982 green; tsc clean across all
three packages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-07-30 15:14:58 -07:00
co-authored by Claude Opus 5
parent 71ee156dac
commit a1682e1228
10 changed files with 176 additions and 49 deletions
+22 -4
View File
@@ -57,14 +57,32 @@ panic-off.
## Measuring ## Measuring
Every telemetry event carries a `canaries` property listing the cohorts the Every telemetry event carries the assignment as a PostHog flag property:
install is enrolled in, so any metric can be split by cohort:
```
$feature/canary-my-feature: "true" | "false"
```
PostHog treats `$feature/<key>` as a first-class flag, so breakdowns, funnels
split by cohort and the experiment surfaces work on a canary with **nothing
configured server-side** — the decision still happens locally and offline,
which the render path requires.
```sql ```sql
-- enrolled vs everyone else SELECT properties['$feature/canary-my-feature'] AS cohort, count()
countIf(properties.canaries LIKE '%my-feature%') AS in_canary FROM events WHERE event = 'render_complete' GROUP BY cohort
``` ```
Two details worth knowing:
- **Both arms are emitted.** A non-enrolled install reports `"false"`, not a
missing property. Absent means *this build predates the canary*, which is a
different fact from *this install is control* — collapsing them makes a ramp
unreadable.
- **Keys are namespaced with `canary-`.** A real PostHog flag namespace
already exists in this project, owned by the web app. The infix guarantees a
canary can never alias a real flag and fight it for the same property.
## Behaviour worth knowing ## Behaviour worth knowing
**Ramping is inclusive.** Widening `10 → 25` keeps everyone who was already **Ramping is inclusive.** Widening `10 → 25` keeps everyone who was already
+8 -4
View File
@@ -54,7 +54,7 @@ vi.mock("@hyperframes/core/canary-registry", async () => {
}; };
}); });
const { isCanaryEnabled, resolveCanary, activeCanaryNames, __resetCanaryCacheForTests } = const { isCanaryEnabled, resolveCanary, canaryEventProperties, __resetCanaryCacheForTests } =
await import("./canary.js"); await import("./canary.js");
beforeEach(() => { beforeEach(() => {
@@ -106,10 +106,14 @@ describe("CLI canary binding", () => {
expect(isCanaryEnabled("test-beta")).toBe(true); expect(isCanaryEnabled("test-beta")).toBe(true);
}); });
it("reports enrolled canaries for telemetry, undefined when none", () => { it("emits PostHog flag-shaped properties for every registered canary", () => {
expect(activeCanaryNames()).toBe("test-alpha"); expect(canaryEventProperties()).toEqual({
"$feature/canary-test-alpha": "true",
"$feature/canary-test-beta": "false",
});
__resetCanaryCacheForTests(); __resetCanaryCacheForTests();
process.env.HF_CANARY_TEST_ALPHA = "off"; process.env.HF_CANARY_TEST_ALPHA = "off";
expect(activeCanaryNames()).toBeUndefined(); expect(canaryEventProperties()["$feature/canary-test-alpha"]).toBe("false");
}); });
}); });
+15 -8
View File
@@ -20,7 +20,12 @@
// Leaf subpath imports, not the "@hyperframes/core" barrel: this resolves on // Leaf subpath imports, not the "@hyperframes/core" barrel: this resolves on
// the CLI startup path, and the barrel pulls the whole core surface. Same // the CLI startup path, and the barrel pulls the whole core surface. Same
// reason the producer is lazily loaded. // reason the producer is lazily loaded.
import { evaluateCanary, parseCanaryOverride, type CanaryDecision } from "@hyperframes/core/canary"; import {
canaryFeatureProperties,
evaluateCanary,
parseCanaryOverride,
type CanaryDecision,
} from "@hyperframes/core/canary";
import { CANARIES, canaryEnvVar, findCanary } from "@hyperframes/core/canary-registry"; import { CANARIES, canaryEnvVar, findCanary } from "@hyperframes/core/canary-registry";
import { readConfig } from "./config.js"; import { readConfig } from "./config.js";
import { getSystemMeta } from "./system.js"; import { getSystemMeta } from "./system.js";
@@ -74,12 +79,14 @@ export function isCanaryEnabled(name: string): boolean {
} }
/** /**
* Comma-joined names of the canaries this install is enrolled in, or * Canary assignments as PostHog flag properties — `$feature/canary-<name>`
* undefined when none — attach to telemetry so every event can be segmented * set to `"true"` / `"false"` for every registered canary. Spread onto every
* by cohort. One low-cardinality property beats a dynamic property per * event so any metric can be broken down by cohort using PostHog's native
* canary, and `contains` filtering works fine in PostHog. * flag tooling, with nothing configured server-side. See
* `canaryFeatureProperties` for why non-enrolled canaries are emitted too.
*/ */
export function activeCanaryNames(): string | undefined { export function canaryEventProperties(): Record<string, string> {
const active = CANARIES.filter((c) => resolveCanary(c.name).enabled).map((c) => c.name); return canaryFeatureProperties(
return active.length > 0 ? active.join(",") : undefined; CANARIES.map((c) => ({ name: c.name, enabled: resolveCanary(c.name).enabled })),
);
} }
+20 -10
View File
@@ -20,9 +20,12 @@ vi.mock("../utils/env.js", () => ({
// Canary enrolment is registry-driven and will change as rollouts ramp; stub // Canary enrolment is registry-driven and will change as rollouts ramp; stub
// it so this asserts the WIRING (does every event carry the cohort?) rather // it so this asserts the WIRING (does every event carry the cohort?) rather
// than whichever canaries happen to be live today. // than whichever canaries happen to be live today.
const canaryNames = vi.fn<() => string | undefined>(() => "feat-x,feat-y"); const canaryProps = vi.fn<() => Record<string, string>>(() => ({
"$feature/canary-feat-x": "true",
"$feature/canary-feat-y": "false",
}));
vi.mock("./canary.js", () => ({ vi.mock("./canary.js", () => ({
activeCanaryNames: () => canaryNames(), canaryEventProperties: () => canaryProps(),
})); }));
// Intercept the exit-time child process so flushSync delivery is assertable. // Intercept the exit-time child process so flushSync delivery is assertable.
@@ -152,27 +155,34 @@ describe("telemetry queue delivery", () => {
}); });
describe("canary cohort on every event", () => { describe("canary cohort on every event", () => {
it("attaches enrolled canaries to the event properties", async () => { it("attaches canary assignments as PostHog flag properties", async () => {
canaryNames.mockReturnValue("feat-x,feat-y"); canaryProps.mockReturnValue({
"$feature/canary-feat-x": "true",
"$feature/canary-feat-y": "false",
});
const fetchMock = vi.fn(async () => ({ ok: true, status: 200 }) as Response); const fetchMock = vi.fn(async () => ({ ok: true, status: 200 }) as Response);
vi.stubGlobal("fetch", fetchMock); vi.stubGlobal("fetch", fetchMock);
trackEvent("cli_command", { command: "render" }); trackEvent("cli_command", { command: "render" });
await flush(); await flush();
expect(eventProps(fetchMock).canaries).toBe("feat-x,feat-y"); const props = eventProps(fetchMock);
// Enrolled AND control are both emitted — absent would mean "this build
// predates the canary", a different fact from "not enrolled".
expect(props["$feature/canary-feat-x"]).toBe("true");
expect(props["$feature/canary-feat-y"]).toBe("false");
}); });
it("omits the property entirely when the install is in no canary", async () => { it("adds no canary properties when the registry is empty", async () => {
// Absent, not null/"" — PostHog treats those as real values and they would canaryProps.mockReturnValue({});
// pollute cohort filters.
canaryNames.mockReturnValue(undefined);
const fetchMock = vi.fn(async () => ({ ok: true, status: 200 }) as Response); const fetchMock = vi.fn(async () => ({ ok: true, status: 200 }) as Response);
vi.stubGlobal("fetch", fetchMock); vi.stubGlobal("fetch", fetchMock);
trackEvent("cli_command", { command: "render" }); trackEvent("cli_command", { command: "render" });
await flush(); await flush();
expect("canaries" in eventProps(fetchMock)).toBe(false); expect(Object.keys(eventProps(fetchMock)).some((k) => k.startsWith("$feature/canary-"))).toBe(
false,
);
}); });
}); });
+8 -7
View File
@@ -3,7 +3,7 @@ import { VERSION } from "../version.js";
import { c } from "../ui/colors.js"; import { c } from "../ui/colors.js";
import { diag } from "../ui/diagnostics.js"; import { diag } from "../ui/diagnostics.js";
import { getSystemMeta } from "./system.js"; import { getSystemMeta } from "./system.js";
import { activeCanaryNames } from "./canary.js"; import { canaryEventProperties } from "./canary.js";
import { enqueue, type EventProperties } from "./transport.js"; import { enqueue, type EventProperties } from "./transport.js";
import { telemetryRuntimeOverride } from "./policy.js"; import { telemetryRuntimeOverride } from "./policy.js";
@@ -80,12 +80,13 @@ export function trackEvent(
// we already knew. Absent (not false) when the config predates the // we already knew. Absent (not false) when the config predates the
// marker. Resolved after the shouldTrack guard. // marker. Resolved after the shouldTrack guard.
install_predecessor_found: readConfig().predecessorFound, install_predecessor_found: readConfig().predecessorFound,
// Canary cohorts this install is enrolled in, comma-joined (absent when // Canary assignments as `$feature/canary-<name>` — PostHog's native flag
// none). Attached to EVERY event, not just renders: a staged rollout is // property shape, so breakdowns and experiment analysis work on a canary
// only as good as the ability to split any metric by cohort. Resolved // with nothing configured server-side. On EVERY event, not just renders:
// after the shouldTrack guard above, so opted-out installs never pay for // a staged rollout is only as good as the ability to split any metric by
// it. See telemetry/canary.ts. // cohort. Resolved after the shouldTrack guard, so opted-out installs
canaries: activeCanaryNames(), // never pay for it. See telemetry/canary.ts.
...canaryEventProperties(),
agent_env_hints: sys.agent_env_hints ?? undefined, agent_env_hints: sys.agent_env_hints ?? undefined,
}, },
distinctId, distinctId,
+34
View File
@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
import { randomUUID } from "node:crypto"; import { randomUUID } from "node:crypto";
import { canaryBucket, evaluateCanary, parseCanaryOverride, type CanaryInput } from "./canary.js"; import { canaryBucket, evaluateCanary, parseCanaryOverride, type CanaryInput } from "./canary.js";
import { CANARIES, canaryEnvVar, findCanary, overdueCanaries } from "./canaryRegistry.js"; import { CANARIES, canaryEnvVar, findCanary, overdueCanaries } from "./canaryRegistry.js";
import { CANARY_FEATURE_PREFIX, canaryFeatureKey, canaryFeatureProperties } from "./canary.js";
const base = (over: Partial<CanaryInput> = {}): CanaryInput => ({ const base = (over: Partial<CanaryInput> = {}): CanaryInput => ({
feature: "test-feature", feature: "test-feature",
@@ -249,3 +250,36 @@ describe("registry", () => {
expect(overdueCanaries()).toEqual([]); expect(overdueCanaries()).toEqual([]);
}); });
}); });
describe("PostHog flag-shaped properties", () => {
it("namespaces keys so a canary can never alias a real PostHog flag", () => {
// A real flag namespace already exists in this project, owned by the web
// app (e.g. `enable-chat-tab`). Without the `canary-` infix a canary named
// after a real flag would fight it for the same property.
expect(canaryFeatureKey("de-parallel-router")).toBe("$feature/canary-de-parallel-router");
expect(CANARY_FEATURE_PREFIX.startsWith("$feature/")).toBe(true);
});
it("emits every canary, not just enrolled ones", () => {
// Absent vs "false" are different facts: absent = this build predates the
// canary, "false" = this build has it and this install is control.
// Collapsing them makes a ramp unreadable.
const props = canaryFeatureProperties([
{ name: "a", enabled: true },
{ name: "b", enabled: false },
]);
expect(props).toEqual({
"$feature/canary-a": "true",
"$feature/canary-b": "false",
});
});
it("uses string values, matching how PostHog records boolean flags", () => {
const props = canaryFeatureProperties([{ name: "a", enabled: true }]);
expect(typeof props["$feature/canary-a"]).toBe("string");
});
it("is empty when nothing is registered", () => {
expect(canaryFeatureProperties([])).toEqual({});
});
});
+44
View File
@@ -134,3 +134,47 @@ export function parseCanaryOverride(raw: string | undefined): boolean | undefine
if (v === "0" || v === "false" || v === "off" || v === "no") return false; if (v === "0" || v === "false" || v === "off" || v === "no") return false;
return undefined; return undefined;
} }
/**
* Property-name prefix for canary assignments on telemetry events.
*
* PostHog treats `$feature/<key>` as a first-class flag property: breakdowns,
* funnels split by cohort and the experiment surfaces all key on it. Emitting
* assignments in that shape means the analysis tooling works on a canary with
* nothing configured server-side — the decision still happens locally and
* offline, which the render path requires (no render-time network calls, and
* behaviour must not depend on analytics being reachable).
*
* The `canary-` infix is deliberate. A real PostHog flag namespace already
* exists in this project, owned by the web app (e.g. `enable-chat-tab`, set by
* posthog-js). Namespacing guarantees a canary key can never alias a real flag
* key and have the two fight over the same property.
*/
export const CANARY_FEATURE_PREFIX = "$feature/canary-";
/** `de-parallel-router` → `$feature/canary-de-parallel-router`. */
export function canaryFeatureKey(name: string): string {
return `${CANARY_FEATURE_PREFIX}${name}`;
}
/**
* Build the telemetry properties for a set of resolved canaries.
*
* Emits EVERY registered canary, not just the enrolled ones, because absent
* and `"false"` mean different things: absent is "this build predates the
* canary", `"false"` is "this build has it and this install is not enrolled".
* Collapsing those makes a ramp unreadable — you cannot tell a control group
* from an old version.
*
* Values are the strings `"true"` / `"false"` to match how PostHog records
* boolean flag values, so the property is directly comparable to a real flag.
*/
export function canaryFeatureProperties(
entries: ReadonlyArray<{ name: string; enabled: boolean }>,
): Record<string, string> {
const props: Record<string, string> = {};
for (const entry of entries) {
props[canaryFeatureKey(entry.name)] = entry.enabled ? "true" : "false";
}
return props;
}
+7 -4
View File
@@ -32,7 +32,7 @@ vi.mock("@hyperframes/core/canary-registry", async () => {
const { const {
isCanaryEnabled, isCanaryEnabled,
resolveCanary, resolveCanary,
activeCanaryNames, canaryEventProperties,
canaryParamName, canaryParamName,
__resetStudioCanaryCacheForTests, __resetStudioCanaryCacheForTests,
} = await import("./canary"); } = await import("./canary");
@@ -164,11 +164,14 @@ describe("cohort identity", () => {
}); });
describe("telemetry", () => { describe("telemetry", () => {
it("reports enrolled canaries, undefined when none", () => { it("emits the same PostHog flag-shaped properties as the CLI", () => {
expect(activeCanaryNames()).toBe("on-everywhere"); expect(canaryEventProperties()).toEqual({
"$feature/canary-on-everywhere": "true",
"$feature/canary-off-everywhere": "false",
});
__resetStudioCanaryCacheForTests(); __resetStudioCanaryCacheForTests();
setSearch("?hf_canary_on_everywhere=off"); setSearch("?hf_canary_on_everywhere=off");
expect(activeCanaryNames()).toBeUndefined(); expect(canaryEventProperties()["$feature/canary-on-everywhere"]).toBe("false");
}); });
}); });
+13 -7
View File
@@ -33,7 +33,12 @@
// browser bundle, and the barrel re-exports the whole core surface (parsers, // browser bundle, and the barrel re-exports the whole core surface (parsers,
// lint, studio-server); pulling that in here drags a Node-oriented dependency // lint, studio-server); pulling that in here drags a Node-oriented dependency
// graph into the bundle. These two modules are pure and leaf. // graph into the bundle. These two modules are pure and leaf.
import { evaluateCanary, parseCanaryOverride, type CanaryDecision } from "@hyperframes/core/canary"; import {
canaryFeatureProperties,
evaluateCanary,
parseCanaryOverride,
type CanaryDecision,
} from "@hyperframes/core/canary";
import { CANARIES, findCanary } from "@hyperframes/core/canary-registry"; import { CANARIES, findCanary } from "@hyperframes/core/canary-registry";
import { resolveStudioDistinctId } from "./distinctId"; import { resolveStudioDistinctId } from "./distinctId";
import { safeSessionStorage } from "../utils/safeStorage"; import { safeSessionStorage } from "../utils/safeStorage";
@@ -142,11 +147,12 @@ export function isCanaryEnabled(name: string): boolean {
} }
/** /**
* Comma-joined names of the canaries this install is enrolled in, or undefined * Canary assignments as PostHog flag properties (`$feature/canary-<name>`),
* when none — attached to every Studio event so any metric can be split by * attached to every Studio event so any metric can be split by cohort —
* cohort, exactly as the CLI does. * identical shape to the CLI, so a rollout spanning both reads as one flag.
*/ */
export function activeCanaryNames(): string | undefined { export function canaryEventProperties(): Record<string, string> {
const active = CANARIES.filter((c) => resolveCanary(c.name).enabled).map((c) => c.name); return canaryFeatureProperties(
return active.length > 0 ? active.join(",") : undefined; CANARIES.map((c) => ({ name: c.name, enabled: resolveCanary(c.name).enabled })),
);
} }
+5 -5
View File
@@ -6,7 +6,7 @@
import { getAnonymousId, hasShownNotice, isOptedOut, markNoticeShown } from "./config"; import { getAnonymousId, hasShownNotice, isOptedOut, markNoticeShown } from "./config";
import { getBrowserSystemMeta } from "./system"; import { getBrowserSystemMeta } from "./system";
import { activeCanaryNames } from "./canary"; import { canaryEventProperties } from "./canary";
// Write-only PostHog project key, safe to embed in client code. // Write-only PostHog project key, safe to embed in client code.
const POSTHOG_API_KEY = "phc_zjjbX0PnWxERXrMHhkEJWj9A9BhGVLRReICgsfTMmpx"; const POSTHOG_API_KEY = "phc_zjjbX0PnWxERXrMHhkEJWj9A9BhGVLRReICgsfTMmpx";
@@ -74,10 +74,10 @@ export function trackEvent(event: string, properties: EventProperties = {}): voi
const sys = getBrowserSystemMeta(); const sys = getBrowserSystemMeta();
eventQueue.push({ eventQueue.push({
event, event,
// `canaries` mirrors the CLI: the cohorts this install is enrolled in, on // Canary assignments as `$feature/canary-<name>`, mirroring the CLI so a
// EVERY event so any metric can be split by cohort. Resolved after the // rollout spanning both surfaces reads as one flag in PostHog. Resolved
// shouldTrack guard, so opted-out users never pay for it. // after the shouldTrack guard, so opted-out users never pay for it.
properties: { ...properties, ...sys, canaries: activeCanaryNames() }, properties: { ...properties, ...sys, ...canaryEventProperties() },
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
}); });