There are ten creation workflows. The Workflows nav group held eight of them plus
`guides/export-and-share`, which is not a workflow at all — there is no such
skill, it is a finishing task. The tenth, the Remotion port, sat under
Developers > Composition, design & animation, a different tab entirely.
So: export-and-share moved to Build the project where the other task pages live,
the Remotion page moved into Workflows, and the chooser gained a tenth route. The
chooser rendered `<video>` unconditionally, which would have drawn a black
rectangle for a route with no clip — it now renders a neutral tile instead, and
custom.css styles it to the same box.
**The Prompt Guide and the workflow guides did not know about each other.** Level 1
is organised one page per workflow, and five of its six pages linked to no guide
at all. A reader learning to prompt for motion graphics never discovered
/guides/motion-graphics, and the traffic went nowhere in both directions. Each
Level 1 page now names its workflow and says what that page adds.
Four links pointed at redirects rather than the real page — /guides/claude-design,
/guides/common-mistakes, /guides/website-to-video, /guides/media-overlays. Seven
files now link to the destination directly.
Checked after: 0 broken links, 0 links via redirect.
**`bun unlink hyperframes` does not exist.** Bun answers "error: bun unlink
{packageName} not implemented yet" — I ran it. The command takes no package name;
it unregisters whatever directory you are standing in. The reader arrives at that
block having just been told to cd into a video project, so even the correct form
would unregister the wrong thing and leave the global link in place — which is
exactly the failure the same page's troubleshooting section then explains. The
page created the bug it diagnosed. Now `cd packages/cli && bun unlink`.
**`data-media-start` and `data-playback-start` are one value, not two features.**
The schema page gave them separate rows with different-sounding meanings, so a
reader would reasonably think they compose. Six read sites in core are all
`playbackStart ?? mediaStart` — runtime/media.ts:18 and :92, runtime/init.ts:751,
1993, 2928, 3010. One row now, alias named, and it says which wins.
**Canary rollouts contradicted itself.** Line 109 called enrolment a pure function
of `(feature, installId, percentage)`; line 195 said canaries bucket on a
dedicated `bucketSeed`, not the telemetry id. The second is right —
cli/src/telemetry/canary.ts:98 is `unitId: config.bucketSeed ?? config.anonymousId`.
Also on the Studio shortcuts page earlier: J/Shift+J for keyframe navigation,
Backspace to delete, and Ctrl+Y for redo were all missing.
R6/R7 blockers.
An already-open Studio kept emitting server-side render telemetry after
another process disabled CLI telemetry. refreshTelemetryPosture() only ran
while serving a fresh SPA document and on /api/telemetry-identity, which
Studio has no consumer for, so the render POST and its async outcome used
the posture cached when the preview server booted. It now refreshes at the
render boundary and again immediately before the completion/error event,
so an opt-out during a long render is honoured.
The identity tests were passing vacuously: their mocks omitted
readConfigFresh and resetTelemetryPostureCache, and the resulting
missing-export error was swallowed by the refresh's own catch. Mocked
properly, plus the enabled -> external disable -> next response transition
and the suppression path at the layer that drops the event.
A full reset also did not persist its new lineage in a long-lived process:
syncInstallState returned early on a process-lifetime memo even after
~/.hyperframes was deleted, so install-state was never recreated and the
next config-only re-mint rolled a third seed instead of inheriting the
second. The memo is now revalidated against the file.
Also drops a stale reference to assertNoOverdueCanaries and stops the
workflow and docs claiming the sunset job routes anything to the owner —
it names them in the run log and notifies nobody.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- A long-lived preview cached its telemetry posture in two places
(readConfig and shouldTrack). Running `telemetry disable` in another
terminal left it resolving canaries and injecting the CLI id for hours.
Both caches are now dropped together at a request boundary.
- Studio minted and shipped a telemetry id for every render regardless of
the browser profile's opt-out, and the server emitted the outcome under
CLI policy, which cannot see localStorage or DNT. The browser now sends
an explicit telemetryOptOut, distinct from an old client's omission.
- Any non-empty HYPERFRAMES_PREVIEW_HOST disabled the DNS-rebinding guard,
so even a loopback bind accepted a hostile Host. The guard now holds for
loopback binds and, on a LAN bind, admits only names this machine
answers on.
- sunsetAfter had no reader of the current date. A scheduled workflow runs
scripts/check-canary-sunset.ts weekly, so a failure lands on the
rollout's owner rather than on an unrelated PR author.
- The install-state seed memo outlived `rm -rf ~/.hyperframes`,
resurrecting a cleared cohort. Removed; it only saved a read on a
readConfig cache miss.
Docs updated for the Host rule and the 100% exclusion carve-out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
P1 — the required Test lane was red, and it was my test. The
hostile-Host SPA case asserted a 200, which only holds when
packages/studio/dist is built: true on a dev box, false in CI, so it
passed locally and failed there. The Host split moved into a pure
buildStudioHeadScriptsForHost() and is asserted directly; the route test
no longer depends on build state. Verified by running the CLI suite with
the bundle moved aside — 2330 pass.
P1 — studio:* still bypassed most privacy controls. It honoured two
localStorage keys but not navigator.doNotTrack,
VITE_HYPERFRAMES_NO_TELEMETRY, Vite dev mode or API-key eligibility, and
canary enrolment honoured a different single control. New
telemetry/policy.ts is the one answer to "may this profile be measured",
consumed by both transports and by enrolment. It imports only ./config,
so no cycle with the modules that import it. Each control is asserted
individually.
P1 — LAN/remote preview lost the authoritative decisions. Withholding
the whole head script for any non-loopback Host also dropped the safe
{enabled, forced} map, sending a supported HYPERFRAMES_PREVIEW_HOST=
0.0.0.0 Studio back to re-deriving. Identity injection is now gated
separately from decision injection: identity is loopback-only, decisions
always publish.
P1 — the breaker latch was neither authoritative nor truthfully
persisted. syncInstallState swallowed its own failures so
writeConfigWithResult always reported ok, and reads took the flag only
from config.json. The latch is now merged into every effective read,
which makes install-state authoritative and closes both the failed-mirror
and stale-concurrent-writer paths; the write additionally reports
mirrored: false rather than swallowing.
P2 — public contracts. canary-rollouts.mdx said a config wipe loses the
breaker (it does not) and documented the superseded {name: boolean} map
with unconditional CLI precedence; both corrected, with the precedence
ladder written out and the override exception stated explicitly. PR body
rewritten — it still named ~/.local/state, claimed state survives
deleting ~/.hyperframes, and carried stale counts.
Tests: 2330 CLI (bundle absent), 3151 Studio, 24 core. Fault injection:
reverting each fix alone fails 5 CLI / 5 Studio.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
P1 — SPA route bypassed the DNS-rebinding guard. Guarding only
/api/telemetry-identity left the catch-all as an open side door: a
rebound origin could fetch `/` and read __HF_CLI_DISTINCT_ID and
__HF_CLI_BUCKET_SEED straight out of the returned HTML. The SPA response
now applies the same isLoopbackHost() check; an untrusted Host still gets
a working Studio, just with no identity, seed or decisions injected.
Route-level regression added.
P1 — a CLI cohort roll could override Studio's own opt-out.
decideStudioCanary() adopted the injected decision before checking
isOptedOut(), so CLI-telemetry-on plus Studio-opted-out still enrolled
Studio. A bare boolean could not express the difference between a
deliberate override and an ordinary cohort roll, so the injected map now
carries provenance ({ enabled, forced }). Forced wins outright — it is
the documented escalation channel and must behave the same on both
surfaces — while a percentage roll now loses to this profile's opt-out.
Full interaction matrix tested.
P1 — the legacy studio:* path sat outside both contracts.
utils/studioTelemetry.ts shipped its own opt-out key and its own send
loop, so the documented hyperframes-studio:telemetryDisabled did not
silence it and its events carried no cohort assignment. It now honours
both keys (the legacy one stays, so nobody already opted out is quietly
re-enabled) and mixes in canaryEventProperties(), making "every
telemetry event carries the assignment" actually true.
P2 — partial salvage could drop a tripped breaker.
salvageInstallState() discarded the whole record when markerAt and
bucketSeed were both unusable, taking deParallelRouterTrialFired with it
and re-enrolling a machine whose router already failed. All three fields
are now independently salvageable.
Docs: canary-rollouts.mdx said "disabling telemetry disables the
reporting, not the enrolment" — exactly backwards since the opt-out gate
landed. Corrected; checked for other copies, none.
Tests: 13 new (4 opt-out precedence, 4 legacy-path opt-out and canary
props, 3 route-level host guard, 2 breaker salvage). Fault injection:
each of the four fixes reverted independently fails its own tests
(2 CLI + 1 Studio + 2 Studio).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second half of a review comment I had only half-addressed: the opt-out
behaviour shipped in 4f464dc4, but "should we add this into/connected to
the existing telemetry doc?" did not.
There is no standalone telemetry page — the canonical disclosure is the
`telemetry` command section in packages/cli. Links now run both ways so
neither page describes canaries as a separate system:
- packages/cli #telemetry: telemetry state also controls canary
enrolment, with every opt-out route named.
- contributing/canary-rollouts: a Note at the top, before any of the
how-to, saying canaries sit behind the telemetry switch and why (a
canary is a measured rollout; an install that reports nothing cannot be
compared, so enrolling it buys no signal).
- guides/feedback "Opting Out": disabling telemetry also ends canary
enrolment, alongside the feedback prompt and usage tracking.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes both cross-surface findings with one mechanism. The CLI publishes
window.__HF_CLI_CANARY_DECISIONS ({ name: boolean }); a CLI-launched
Studio takes it as authoritative over its own seed, URL override and the
registry percentage.
Studio re-deriving could not agree with the CLI in three cases:
- Telemetry off. The CLI resolves telemetry_opt_out, but Studio's
opt-out is a separate localStorage flag it cannot see, so it would
evaluate normally and could enrol on a render the CLI excluded. The
previous commit gated each surface independently; that fixed silent
enrolment per surface but NOT the disagreement between them.
- HF_CANARY_* override. Env vars never cross into the browser — Studio
reads only its URL param / sessionStorage — so a support session
forcing a canary on got the CLI forced and Studio guessing.
- No seed injected. Studio falls back to a different unit id, i.e. a
different bucket.
Shipping the decision instead of the inputs makes divergence structurally
impossible: one evaluation, two surfaces. It also exposes strictly less —
booleans about features, rather than the seed buckets derive from — which
is why it is safe to publish with telemetry off, the case it exists for.
Studio still evaluates locally when standalone, or for a canary the CLI
did not publish, and ignores a non-boolean value rather than trusting it.
Tests: 6 Studio (CLI-off wins over unset local flag, CLI-on with no URL
param, beats contradicting override, beats seed, falls back per-canary,
rejects non-boolean) and 4 CLI (decisions with telemetry off and no
identity, alongside identity when on, script-tag escaping on a hostile
canary name, throwing resolver degrades to identity only). Four existing
identity tests asserted the old "nothing when telemetry off" contract and
were updated; the registry is now mocked there so string assertions don't
move when a canary is added or ramped. Fault injection: dropping the
adoption fails 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both reviewers flagged the same gap: seed injection was gated on
telemetryShouldTrack(), but canary EVALUATION was not. An install with
DO_NOT_TRACK=1 was still bucketed and still had real code paths flipped
(e.g. HF_DE_PARALLEL_ROUTER), silently and unmeasurably.
A canary is a measured rollout — we enrol a slice precisely so it can be
compared against everyone else. An install that sends nothing can't be
compared, so enrolling it buys no signal and only changes that user's
code path, on an experimental feature, without their knowledge. That is
the wrong side of an opt-out.
Resolves to a new `telemetry_opt_out` reason BEFORE bucketing, so no
cohort is assigned at all. Distinct from `excluded` because "why is my
canary off" has a very different answer for CI than for opted-out, and
the reason never reaches telemetry by construction.
Covers every opt-out route: persisted preference, the runtime env vars
and dev/telemetry-disabled builds via policy.ts, and Studio's
hyperframes-studio:telemetryDisabled.
An explicit HF_CANARY_* / ?hf_canary_*= override still wins — a
deliberate local choice, not silent enrolment, and the documented way to
exercise a canary with telemetry off.
The CLI check mirrors shouldTrack() rather than importing it: client.ts
already imports canary.ts for canaryEventProperties, so depending on it
would be a cycle. Both read the same two inputs, so they cannot disagree.
Tests: 9 new across CLI and Studio (preference off, each runtime
override, no bucket assigned, override still honoured, flag properties
all-false). Fault injection: removing the CLI gate fails 6, removing the
Studio gate fails 3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adopts the #2904 pattern for bucketSeed. The seed rides in
install-state.json, which now lives beside config.json in ~/.hyperframes
rather than in ~/.local/state/hyperframes/.
Review rejected persisting state outside the config dir to defeat a
user's reset, and that objection is sharpest for the seed: it is the one
field that would turn install-state into a persistent pseudonymous
identifier surviving `rm -rf ~/.hyperframes`.
The carryover still earns its place, just against the churn that
actually happens. config.json is rewritten on every command and every
render, and readConfig recovers from any parse/permission/IO failure by
minting a fresh identity — so a re-mint would reshuffle cohorts
mid-rollout. A no-schema file written once at mint is decoupled from
that without leaving the directory. Config re-mint: cohorts hold.
Directory deleted: cohorts go too, deliberately.
A pre-move seed is adopted by the same migration, so installs already
carrying one do not have a live cohort reshuffled under them.
Tests: seed survives a re-mint, does NOT survive deleting the config
dir, and migrates from the pre-move path. Prose in config.ts, canary.ts
and canary-rollouts.mdx corrected — it still claimed cohorts survive a
wipe. Docs gain a removal-path section.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cohort membership now survives a config wipe. Canaries bucket on a
dedicated bucketSeed (fresh random UUID, distinct from anonymousId by
design) that is mirrored write-once into the install-state file and
inherited at mint: a wipe re-rolls the telemetry id but never the canary
assignment. This removes cumulative-exposure drift for the recoverable
churn bucket entirely — the residual drift comes only from fresh
machines, containers, and genuinely new users — and keeps before/after
comparisons valid across a reinstall.
The seed is never emitted in telemetry (only the resulting true/false
assignments are), so it does not link the old id to the new one
server-side. The residual linker is the flag vector itself (k bits for k
live canaries), documented as such. An explicit reset still works by
deleting the state file, and the no-identity test now also asserts the
seed differs from the anonymousId.
Cross-surface coherence: the CLI's studio server injects the seed as
window.__HF_CLI_BUCKET_SEED (same telemetry gate and script-escaping as
the distinct id, and on the /api/telemetry-identity fallback), and the
Studio binding buckets on it when present — without this the CLI would
bucket on the seed while Studio bucketed on the distinct id, splitting
one machine across cohorts (calibration check 4 would catch exactly
this). Standalone Studio still buckets on its localStorage id: the
browser has no second storage location, so that id doubles as the seed.
Legacy configs are backfilled once (lineage seed if the state file has
one, else minted) and persisted immediately — an unpersisted seed would
re-roll cohorts every process. Safe to ship in the same release as the
first canaries: no prior release emitted canary properties, so the
bucketing-unit change is unobservable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "what cannot be fixed" framing predated the state-file rollover: the
drift still exists, but a re-rolled install no longer re-enters a failed
path, and install_predecessor_found splits the drift into recoverable vs
unrecoverable. Window-length policy comes from the residual, not raw
turnover.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributors in the wild have no access to the project's analytics backend,
so the contributing doc must not point them at it: drop the internal
dashboard link, the backend-specific SQL blocks, and the vendor naming.
The calibration check definitions stay (public, fixed terms of the
experiment); the queries live with the dashboard tiles that run them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Telemetry carries the assignment but not the decision reason, so a manual
HF_CANARY_* toggle mid-window reads as a cohort flip. Rule it out before
treating a small-nonzero stability read as a mechanism bug; a reason
property is deliberately deferred until the check actually comes back dirty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
As written, check 4's condition (dual-surface AND value flipped anywhere)
was a strict subset of check 3's (value flipped) — if check 3 read zero,
check 4 was vacuously zero and added no independent signal. Compare the
value each surface actually reported instead, and state plainly that any
disagreement is also a check-3 flip: this check's job is attributing such
a flip to binding divergence. Matching fix applied to the live PostHog
tile (insight jQi7QdW1, dashboard 1918875).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four pre-registered checks now exist as PostHog tiles. Without the link
the doc describes queries someone has to re-type; with it the doc and the
dashboard are one artifact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Registers two canaries that gate nothing — `calibration-10` (10%) and
`calibration-50` (50%) — so the rollout mechanism can be proven against real
traffic before any real feature depends on it. Zero behavioural risk: they are
read by nothing.
They answer what the unit tests structurally cannot. The tests bucket
generated UUIDs and weight every install equally; real render volume is
heavily skewed toward a few heavy installs, and real install ids churn (~25x
more distinct ids over 30 days than in any single day on the desktop render
population).
Four checks, pre-registered in the docs so the read is not post-hoc:
1. ACCURACY — does 10% land at 10%, install-weighted AND event-weighted?
2. DRIFT — how fast does CUMULATIVE exposure climb above target as ids churn?
The instantaneous share is flat by construction; the set of installs
enrolled at some point is not.
3. STABILITY — does any install ever change cohort? Must be zero. Percentages
are held FIXED for the window precisely so a flip is unambiguously a bug;
during a real ramp a false->true flip would be correct instead.
4. CROSS-SURFACE — do the CLI and Studio bindings agree for the same install?
A CLI-launched Studio adopts the CLI id, and 16,961 installs currently
share an id across both surfaces, so this is measurable.
Plus an independence check: overlap between the two calibration canaries
should be ~p1*p2 (~5%), not ~min(p1,p2) (~10%, which would mean every canary
lands on the same unlucky cohort).
The docs also record what calibration CANNOT fix: per-install cohorts never
flip, but a person who wipes their config gets a new id and a fresh roll.
Preventing that needs stable identity across resets, and both candidates were
rejected — hardware fingerprinting correlates the cohort with hardware (fatal
for a rendering experiment, and it survives uninstall) and account identity
covers only ~3.6% of local rendering installs. The drift is therefore a
measured, accepted limit, and the point of calibrating is to size it and pick
canary window lengths accordingly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the single `canaries: "a,b"` telemetry property with PostHog's own
flag shape, one property per registered canary:
$feature/canary-de-parallel-router: "true" | "false"
PostHog treats `$feature/<key>` as a first-class flag property, so breakdowns,
funnels split by cohort and the experiment surfaces work on a canary with
nothing configured server-side. The decision still happens locally: the render
path forbids render-time network calls, behaviour must not depend on analytics
being reachable, and neither the CLI nor Studio ships posthog-js (both
hand-roll a batch POST, so there is no SDK to evaluate a real flag with).
Decide locally, analyse natively.
Two decisions worth recording:
- BOTH ARMS ARE EMITTED. A non-enrolled install reports "false" rather than
omitting the property. Absent means "this build predates the canary", which
is a different fact from "this install is control" — collapsing them makes a
ramp unreadable, because you cannot separate a control group from an old
version.
- KEYS ARE NAMESPACED with a `canary-` infix. A real PostHog flag namespace
already exists in this project, owned by the web app (`enable-chat-tab`, set
by posthog-js from `$lib=web` events). Namespacing guarantees a canary key
can never alias a real flag key and have the two fight over one property.
Values are the strings "true"/"false" to match how PostHog records boolean
flag values, so the property is directly comparable to a real flag.
98 core / 1437, 166 cli / 2194, 269 studio / 2982 green; tsc clean across all
three packages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the canary primitive.
Telemetry: every event now carries a `canaries` property listing the cohorts
the install is enrolled in, attached in `trackEvent` so it lands on ALL
events rather than renders only — a staged rollout is only as useful as the
ability to split any metric by cohort. Resolved after the shouldTrack guard,
so opted-out installs never pay for it, and omitted entirely (not null or "")
when the install is in no canary, since PostHog treats those as real values.
Test hardening, after validating the shipped code against 60k synthetic and
101 real fleet install ids:
- Pin FNV-1a against canonical vectors, AND assert the shipped canaryBucket
actually uses that hash. Without the second assertion the first is
tautological — it would only prove the test's own copy is correct while
canary.ts drifted to a different hash, silently reshuffling every live
cohort. Fault-injection confirms only this assertion catches a hash change;
the distribution tests stay green because a perturbed hash is still
well-distributed.
- Tighten the share test from a 0.6x-1.4x band to +/-1 percentage point.
Measured error was 0.16pp at n=60k, so the old band would have passed a
badly skewed hash.
- Add chi-square uniformity across all 100 buckets (chi2 89.0 vs 148.2
critical at p=0.001). A lumpy hash yields roughly the right total share
while overloading some buckets, so the share test alone cannot catch it.
- Assert N concurrent canaries enrol binomially rather than in lockstep:
8 canaries at 10% put ~43% of installs in none and zero in all eight,
matching binomial(8, 0.1). Correlated slices would put ~10% in all eight.
Also verified 88,443 of 88,448 fleet install ids are well-formed UUIDs; the
5 that are not fail closed, which is the intended direction.
Docs: docs/contributing/canary-rollouts.mdx, registered in docs.json (an
unregistered page is invisible in the nav).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
