mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-17 23:47:33 +00:00
sync/hyperframes-codegen-e53ca0c1
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b3990ac789 |
feat(studio): emit the canary reason on Studio events too
Review caught that the same anti-pattern was still live in the Studio binding: canaryEventProperties destructured only `enabled` and dropped the reason. Its own doc comment promised 'identical shape to the CLI, so a rollout spanning both reads as one flag' — which the CLI-only fix had just made false. This matters beyond symmetry. A CLI-launched Studio adopts the CLI's decisions and shares its bucket seed, so a cohort flip can surface on either surface. Emitting attribution on only one leaves Studio-observed flips unattributable and makes the two flip counts irreconcilable — and Studio is the surface most likely to expose a shared-seed-with-diverging-id pattern, which is the open question the reason exists to answer. Also adds the no_unit_id emission test the CLI side advertised but never asserted, and a Studio pair pinning that a URL override and a cohort roll produce the same assignment with different reasons. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
076657a639 |
feat(core,cli): emit the canary decision reason alongside the assignment
The calibration contract deferred this until the stability check came back dirty. It did: the first fleet read found 304 installs (1.08%) reporting both values for a canary whose percentage never moved, and the genuinely anomalous ones could not be separated from a developer toggling HF_CANARY_*, because the assignment alone is identical in both cases. resolveCanary has always computed the reason and canaryEventProperties dropped it. Now every canary emits canary_reason_<name> beside its assignment. Deliberately outside the $feature/ namespace: PostHog treats those as flag values, and a non-boolean there would corrupt the flag's own breakdowns. Two of the six wire values are immediately useful beyond override attribution. 'excluded' identifies CI installs, which today have to be dropped by joining on is_ci — conflating them with out_of_cohort is what made the first accuracy read look like a significant failure (9.22% against a 10% target) when it was not. 'no_unit_id' surfaces the fails-closed corner. The reason is optional on the core helper so existing callers are unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3f69a2c635 |
fix(cli,core,studio): close 15 review findings + 2 R5 blockers
R5 blockers
- Negative install-state latch was cached for the process lifetime, but
only `true` is monotonic across processes. A long-lived preview server
held a stale `false` and could re-enrol after another process tripped
the breaker. Only the positive is cached now; `false` re-reads.
- The real breaker writer used writeConfig(), which collapses
{ok:true, mirrored:false} to success, so a run that mirrored nothing
reported done with the latch only on the erasable store. It consumes
writeConfigWithResult and retries until both stores carry it.
Bucketing integrity
- Storage-restricted Studio profiles all bucketed on the literal
"anonymous": computed against the shipped hash, 100% of them were
enrolled in calibration-50 rather than 50%, and they merged into one
PostHog person. Per-session random id instead — persists nothing.
- bucketSeed had read/write authority backwards: install-state is
write-once authoritative, but readConfig took config.json's blindly, so
the stores could hold different seeds until a re-mint flipped every
cohort. Merged on read, like the latch.
- An unwritable ~/.hyperframes with no config.json re-minted per call,
re-rolling the seed on every command, and the "cohorts will not be
stable" warning was unreachable on that path.
- A corrupt PRE-MOVE state file was never deleted, so a machine reset
with `rm -rf ~/.hyperframes` reported predecessorFound/stateFileCorrupt
forever — poisoning the exact metric this work exists to produce.
Opt-out honoring
- CLI canary decisions memoized per process, so `hyperframes telemetry
disable` during a running preview server was ignored for hours while
the server kept serving pre-opt-out decisions. The memo is keyed on the
telemetry posture.
- shouldTrack() memoized, contradicting policy.ts's documented "not
memoized" contract that policy.test.ts asserts.
- The Studio override path resolved the bucket unit eagerly as an
argument, minting and PERSISTING a tracking id for an opted-out profile
— a value evaluateCanary discards unread.
- Storage reads could throw out of telemetry into a post-commit catch
block, reporting an already-committed edit as failed.
- readConfig printed an unsilenceable stderr warning on every invocation
for installs that opted out of telemetry entirely.
Host split
- isLoopbackHost rejected 0.0.0.0, so the documented
HYPERFRAMES_PREVIEW_HOST LAN mode silently lost CLI→Studio identity
stitching and split one user across two PostHog persons. Identity is
now allowed when the operator explicitly opted into LAN binding.
- Corrected the comment claiming the guard refuses spoofed Hosts: a
non-browser client sets Host freely. It is a browser DNS-rebinding
mitigation, not access control, and now says so.
Semantics and test hygiene
- percentage:100 did not mean everyone — exclude and no_unit_id sat above
the fast path, so the registry's "delete the entry at 100" step was an
unstaged flip for CI and seedless installs.
- CLI cohort adoption returned before evaluateCanary, dropping Studio's
own webdriver exclusion.
- overdueCanaries() was asserted against wall-clock time, so the whole
core suite would go red on 2026-09-15 for every unrelated PR; and `>`
against midnight made a canary overdue ON its sunset date.
- Statistical assertions ran on unseeded randomUUID() populations tight
enough to fail ~1 run in 200. Seeded.
Also: broke a config -> policy -> transport -> config import cycle by
moving POSTHOG_API_KEY to a leaf module.
Tests: 2347 CLI (bundle absent), 3153 Studio, 1450 core. Fault injection
covers the latch, seed authority, LAN identity, webdriver exclusion and
the anonymous-bucket fix. Two pre-existing tests asserted behaviour these
findings identify as wrong (shouldTrack memoization, 100%-excludes-CI)
and were rewritten with the reasoning stated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
4f464dc424 |
feat(cli,studio): telemetry opt-out is canary opt-out
Both reviewers flagged the same gap: seed injection was gated on telemetryShouldTrack(), but canary EVALUATION was not. An install with DO_NOT_TRACK=1 was still bucketed and still had real code paths flipped (e.g. HF_DE_PARALLEL_ROUTER), silently and unmeasurably. A canary is a measured rollout — we enrol a slice precisely so it can be compared against everyone else. An install that sends nothing can't be compared, so enrolling it buys no signal and only changes that user's code path, on an experimental feature, without their knowledge. That is the wrong side of an opt-out. Resolves to a new `telemetry_opt_out` reason BEFORE bucketing, so no cohort is assigned at all. Distinct from `excluded` because "why is my canary off" has a very different answer for CI than for opted-out, and the reason never reaches telemetry by construction. Covers every opt-out route: persisted preference, the runtime env vars and dev/telemetry-disabled builds via policy.ts, and Studio's hyperframes-studio:telemetryDisabled. An explicit HF_CANARY_* / ?hf_canary_*= override still wins — a deliberate local choice, not silent enrolment, and the documented way to exercise a canary with telemetry off. The CLI check mirrors shouldTrack() rather than importing it: client.ts already imports canary.ts for canaryEventProperties, so depending on it would be a cycle. Both read the same two inputs, so they cannot disagree. Tests: 9 new across CLI and Studio (preference off, each runtime override, no bucket assigned, override still honoured, flag properties all-false). Fault injection: removing the CLI gate fails 6, removing the Studio gate fails 3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
8dd20ac2e8 |
feat(cli): bucket canaries on a machine-lineage seed, not the telemetry id
Cohort membership now survives a config wipe. Canaries bucket on a dedicated bucketSeed (fresh random UUID, distinct from anonymousId by design) that is mirrored write-once into the install-state file and inherited at mint: a wipe re-rolls the telemetry id but never the canary assignment. This removes cumulative-exposure drift for the recoverable churn bucket entirely — the residual drift comes only from fresh machines, containers, and genuinely new users — and keeps before/after comparisons valid across a reinstall. The seed is never emitted in telemetry (only the resulting true/false assignments are), so it does not link the old id to the new one server-side. The residual linker is the flag vector itself (k bits for k live canaries), documented as such. An explicit reset still works by deleting the state file, and the no-identity test now also asserts the seed differs from the anonymousId. Cross-surface coherence: the CLI's studio server injects the seed as window.__HF_CLI_BUCKET_SEED (same telemetry gate and script-escaping as the distinct id, and on the /api/telemetry-identity fallback), and the Studio binding buckets on it when present — without this the CLI would bucket on the seed while Studio bucketed on the distinct id, splitting one machine across cohorts (calibration check 4 would catch exactly this). Standalone Studio still buckets on its localStorage id: the browser has no second storage location, so that id doubles as the seed. Legacy configs are backfilled once (lineage seed if the state file has one, else minted) and persisted immediately — an unpersisted seed would re-roll cohorts every process. Safe to ship in the same release as the first canaries: no prior release emitted canary properties, so the bucketing-unit change is unobservable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a1682e1228 |
feat(core): emit canary assignments as PostHog flag properties
Replaces the single `canaries: "a,b"` telemetry property with PostHog's own
flag shape, one property per registered canary:
$feature/canary-de-parallel-router: "true" | "false"
PostHog treats `$feature/<key>` as a first-class flag property, so breakdowns,
funnels split by cohort and the experiment surfaces work on a canary with
nothing configured server-side. The decision still happens locally: the render
path forbids render-time network calls, behaviour must not depend on analytics
being reachable, and neither the CLI nor Studio ships posthog-js (both
hand-roll a batch POST, so there is no SDK to evaluate a real flag with).
Decide locally, analyse natively.
Two decisions worth recording:
- BOTH ARMS ARE EMITTED. A non-enrolled install reports "false" rather than
omitting the property. Absent means "this build predates the canary", which
is a different fact from "this install is control" — collapsing them makes a
ramp unreadable, because you cannot separate a control group from an old
version.
- KEYS ARE NAMESPACED with a `canary-` infix. A real PostHog flag namespace
already exists in this project, owned by the web app (`enable-chat-tab`, set
by posthog-js from `$lib=web` events). Namespacing guarantees a canary key
can never alias a real flag key and have the two fight over one property.
Values are the strings "true"/"false" to match how PostHog records boolean
flag values, so the property is directly comparable to a real flag.
98 core / 1437, 166 cli / 2194, 269 studio / 2982 green; tsc clean across all
three packages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
71ee156dac |
feat(studio): browser canary binding + leaf subpath imports
Adds the Studio (browser) binding so a canary can span the CLI and the editor,
and fixes a bundling mistake the studio test suite caught.
## The binding
Same public API as the CLI — `isCanaryEnabled("name")` — so a call site reads
identically whether it runs in Node or the browser. Three inputs differ:
- UNIT ID: `resolveStudioDistinctId()`, which already adopts
`window.__HF_CLI_DISTINCT_ID` when the CLI launched Studio. A CLI-launched
Studio therefore lands in the SAME cohort as the CLI: a rollout spanning
render and editor is coherent for that user instead of enrolling their
terminal but not their editor. A test pins that the id is passed through
unmodified — prefixing or re-hashing it would silently break that parity.
- OVERRIDE: no `process.env` in a page, so `?hf_canary_<name>=on` mirrored
into sessionStorage. Session scope is deliberate. A URL is the right carrier
(shareable — "support: open this link"), but persisting a URL-borne override
to localStorage would let one click silently pin a browser into a cohort
forever, long after anyone remembers why. Closing the tab is the reset;
`=reset` clears it explicitly.
- EXCLUSION: `navigator.webdriver` stands in for the CLI's `is_ci`. Automated
browsers mint a fresh localStorage id per run, so they would hop cohorts
between runs — noise in the signal, nothing learned about real users. An
override still reaches them, which is how you test a canary under Playwright.
Studio's `trackEvent` now attaches `canaries` to every event, mirroring the CLI.
## The bundling fix
Importing the `@hyperframes/core` barrel into studio browser code broke two
unrelated hook test files with an esbuild TextEncoder invariant violation. The
barrel re-exports the whole core surface (parsers, lint, studio-server), so it
drags a Node-oriented dependency graph into a browser bundle — the test
failure was the symptom, the bundle bloat was the bug.
`@hyperframes/core` now exposes `./canary` and `./canary-registry`, declared in
packages/core/package-subpaths.json (the generated source of truth for exports —
hand-editing package.json is reverted by the sync script) and marked
`environments: [browser, bun, node]`. Both the studio AND cli bindings import
the leaf modules; the CLI gets the same benefit for a different reason, since
this resolves on the startup path — the reason the producer is lazily loaded.
Verified: the two hook files pass again; 269 studio files / 2982 tests, 98
core / 1433, 166 cli / 2194 green, `bun run lint` clean including the subpath
check. Fault-injection confirms both design decisions are pinned — swapping
session for local storage fails the scope test, prefixing the unit id fails the
CLI/Studio cohort-parity test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
3aea786687 |
feat(core): percentage-based canary rollouts
Adds a reusable staged-rollout primitive so a change can ship to a stable
slice of installs instead of all-or-nothing.
The gap it fills: the repo carries ~49 HF_*/PRODUCER_* booleans and every one
is binary — a feature is either off (and therefore unexercised on real
traffic) or on for everyone (and therefore a fleet-wide bet). The
parallel-drawElement router sat in that gap for weeks: default-off produced
almost no signal, and flipping it default-on would have exposed 100% of
eligible installs at once.
Shape:
- packages/core/src/canary.ts — pure evaluator. No fs, no network, no
`process`; the caller supplies the unit id and overrides, so it imports
cleanly into the CLI, producer, engine, studio-server, the browser-side
studio bundle and the embeddable player. FNV-1a rather than node:crypto for
the same reason.
- packages/core/src/canaryRegistry.ts — every rollout in one table (name,
percentage, owner, description, sunsetAfter), so "what is rolling out, to
whom, owned by whom" is answerable without grepping 49 env vars.
- packages/cli/src/telemetry/canary.ts — supplies the three things only the
CLI knows: anonymousId, the HF_CANARY_<FEATURE> override, and is_ci.
Day-to-day API is `isCanaryEnabled("name")`.
Three properties the tests pin, because getting them wrong is subtle:
- Slices are INDEPENDENT per feature: the bucket hashes `feature:unitId`, not
the id alone. Bucketing on the id would hand every concurrent experiment to
the same unlucky cohort and make two rollouts unreadable apart.
- Ramping is INCLUSIVE: `bucket < percentage`, so widening 10 -> 25 keeps the
original cohort and before/after comparisons survive the ramp.
- It fails CLOSED: no unit id, unknown name, or CI install means not enrolled.
A canary exists to bound blast radius, so "we don't know who this is" must
never mean "enrol everyone".
Registry entries also carry a sunset date, and a test fails once one is past
due — a canary that outlives its rollout is a permanent fork of the product
with none of the review a permanent fork would get.
Ships with de-parallel-router registered at 0%: inert, and ready to ramp in a
patch release once #2840 lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|