mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-11 23:00:03 +00:00
fix(cli,core,studio): close 15 review findings + 2 R5 blockers
R5 blockers
- Negative install-state latch was cached for the process lifetime, but
only `true` is monotonic across processes. A long-lived preview server
held a stale `false` and could re-enrol after another process tripped
the breaker. Only the positive is cached now; `false` re-reads.
- The real breaker writer used writeConfig(), which collapses
{ok:true, mirrored:false} to success, so a run that mirrored nothing
reported done with the latch only on the erasable store. It consumes
writeConfigWithResult and retries until both stores carry it.
Bucketing integrity
- Storage-restricted Studio profiles all bucketed on the literal
"anonymous": computed against the shipped hash, 100% of them were
enrolled in calibration-50 rather than 50%, and they merged into one
PostHog person. Per-session random id instead — persists nothing.
- bucketSeed had read/write authority backwards: install-state is
write-once authoritative, but readConfig took config.json's blindly, so
the stores could hold different seeds until a re-mint flipped every
cohort. Merged on read, like the latch.
- An unwritable ~/.hyperframes with no config.json re-minted per call,
re-rolling the seed on every command, and the "cohorts will not be
stable" warning was unreachable on that path.
- A corrupt PRE-MOVE state file was never deleted, so a machine reset
with `rm -rf ~/.hyperframes` reported predecessorFound/stateFileCorrupt
forever — poisoning the exact metric this work exists to produce.
Opt-out honoring
- CLI canary decisions memoized per process, so `hyperframes telemetry
disable` during a running preview server was ignored for hours while
the server kept serving pre-opt-out decisions. The memo is keyed on the
telemetry posture.
- shouldTrack() memoized, contradicting policy.ts's documented "not
memoized" contract that policy.test.ts asserts.
- The Studio override path resolved the bucket unit eagerly as an
argument, minting and PERSISTING a tracking id for an opted-out profile
— a value evaluateCanary discards unread.
- Storage reads could throw out of telemetry into a post-commit catch
block, reporting an already-committed edit as failed.
- readConfig printed an unsilenceable stderr warning on every invocation
for installs that opted out of telemetry entirely.
Host split
- isLoopbackHost rejected 0.0.0.0, so the documented
HYPERFRAMES_PREVIEW_HOST LAN mode silently lost CLI→Studio identity
stitching and split one user across two PostHog persons. Identity is
now allowed when the operator explicitly opted into LAN binding.
- Corrected the comment claiming the guard refuses spoofed Hosts: a
non-browser client sets Host freely. It is a browser DNS-rebinding
mitigation, not access control, and now says so.
Semantics and test hygiene
- percentage:100 did not mean everyone — exclude and no_unit_id sat above
the fast path, so the registry's "delete the entry at 100" step was an
unstaged flip for CI and seedless installs.
- CLI cohort adoption returned before evaluateCanary, dropping Studio's
own webdriver exclusion.
- overdueCanaries() was asserted against wall-clock time, so the whole
core suite would go red on 2026-09-15 for every unrelated PR; and `>`
against midnight made a canary overdue ON its sunset date.
- Statistical assertions ran on unseeded randomUUID() populations tight
enough to fail ~1 run in 200. Seeded.
Also: broke a config -> policy -> transport -> config import cycle by
moving POSTHOG_API_KEY to a leaf module.
Tests: 2347 CLI (bundle absent), 3153 Studio, 1450 core. Fault injection
covers the latch, seed authority, LAN identity, webdriver exclusion and
the anonymous-bucket fix. Two pre-existing tests asserted behaviour these
findings identify as wrong (shouldTrack memoization, 100%-excludes-CI)
and were rewritten with the reasoning stated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
5e2a9432f1
commit
3f69a2c635
@@ -1,5 +1,4 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { canaryBucket, evaluateCanary, parseCanaryOverride, type CanaryInput } from "./canary.js";
|
||||
import { CANARIES, canaryEnvVar, findCanary, overdueCanaries } from "./canaryRegistry.js";
|
||||
import { CANARY_FEATURE_PREFIX, canaryFeatureKey, canaryFeatureProperties } from "./canary.js";
|
||||
@@ -27,9 +26,36 @@ function rawFnv(input: string): number {
|
||||
return hash >>> 0;
|
||||
}
|
||||
|
||||
/** A realistic population: install ids are v4 UUIDs (`randomUUID()`). */
|
||||
function uuids(n: number): string[] {
|
||||
return Array.from({ length: n }, () => randomUUID());
|
||||
/**
|
||||
* A realistic population: v4-shaped UUIDs, but from a SEEDED PRNG.
|
||||
*
|
||||
* These ids feed statistical assertions (share within 1pp, chi-square
|
||||
* uniformity) whose thresholds are tight enough to fail by chance on a
|
||||
* genuinely random draw: measured at ~4 failures per 1500 runs for the share
|
||||
* bound (the pct=50 case has a binomial SD of 0.354pp, so 1pp is only 2.8
|
||||
* sigma) and 1 per 1000 for chi-square by its own construction. That made the
|
||||
* whole @hyperframes/core suite flaky for unrelated PRs. Seeded means the
|
||||
* population is fixed, so a failure is a real change in the hash — which is
|
||||
* the only thing these tests are for.
|
||||
*/
|
||||
function uuids(n: number, seed = 0x9e3779b9): string[] {
|
||||
let state = seed >>> 0;
|
||||
const nextByte = (): number => {
|
||||
// xorshift32 — deterministic, and uniform enough to stand in for a real
|
||||
// id population. Not used for anything security-relevant.
|
||||
state ^= state << 13;
|
||||
state >>>= 0;
|
||||
state ^= state >>> 17;
|
||||
state ^= state << 5;
|
||||
state >>>= 0;
|
||||
return state & 0xff;
|
||||
};
|
||||
const hex = (count: number): string =>
|
||||
Array.from({ length: count }, () => nextByte().toString(16).padStart(2, "0")).join("");
|
||||
return Array.from(
|
||||
{ length: n },
|
||||
() => `${hex(4)}-${hex(2)}-4${hex(2).slice(1)}-a${hex(2).slice(1)}-${hex(6)}`,
|
||||
);
|
||||
}
|
||||
|
||||
describe("fnv1a32 (via canaryBucket)", () => {
|
||||
@@ -87,9 +113,9 @@ describe("evaluateCanary", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("fails closed without a unit id — unknown must never mean everyone", () => {
|
||||
it("fails closed without a unit id — unknown must never mean a PARTIAL cohort", () => {
|
||||
for (const id of [undefined, "", " "]) {
|
||||
expect(evaluateCanary(base({ unitId: id, percentage: 100 }))).toEqual({
|
||||
expect(evaluateCanary(base({ unitId: id, percentage: 50 }))).toEqual({
|
||||
enabled: false,
|
||||
reason: "no_unit_id",
|
||||
});
|
||||
@@ -97,12 +123,32 @@ describe("evaluateCanary", () => {
|
||||
});
|
||||
|
||||
it("excludes flagged units (CI) from percentage enrolment but not from an override", () => {
|
||||
expect(evaluateCanary(base({ percentage: 100, exclude: true })).reason).toBe("excluded");
|
||||
expect(evaluateCanary(base({ percentage: 100, exclude: true, override: true })).enabled).toBe(
|
||||
expect(evaluateCanary(base({ percentage: 50, exclude: true })).reason).toBe("excluded");
|
||||
expect(evaluateCanary(base({ percentage: 50, exclude: true, override: true })).enabled).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
// 100 is the one percentage where "we don't know who this is" and "this is
|
||||
// CI" stop mattering: the registry's step 4 says to delete the entry and the
|
||||
// guard at 100-and-holding, so any population still resolving false here
|
||||
// would take the new path for the FIRST time at deletion — unstaged, and
|
||||
// invisible on the dashboard that said it was safe.
|
||||
it.each([
|
||||
["no unit id", { unitId: undefined }],
|
||||
["blank unit id", { unitId: " " }],
|
||||
["excluded (CI)", { exclude: true }],
|
||||
])("at 100%% enrols %s, so deleting the guard changes nothing", (_label, extra) => {
|
||||
expect(evaluateCanary(base({ percentage: 100, ...extra }))).toEqual({
|
||||
enabled: true,
|
||||
reason: "in_cohort",
|
||||
});
|
||||
});
|
||||
|
||||
it("an explicit off still wins at 100%", () => {
|
||||
expect(evaluateCanary(base({ percentage: 100, override: false })).enabled).toBe(false);
|
||||
});
|
||||
|
||||
it("clamps out-of-range and fractional percentages", () => {
|
||||
expect(evaluateCanary(base({ percentage: -5 })).enabled).toBe(false);
|
||||
expect(evaluateCanary(base({ percentage: 999 })).enabled).toBe(true);
|
||||
@@ -264,10 +310,29 @@ describe("registry", () => {
|
||||
expect(findCanary("nope")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("no canary is past its sunset date", () => {
|
||||
// Fails the suite when a rollout has been left half-finished. Either take
|
||||
// it to 100 and delete the entry, or move the date deliberately.
|
||||
expect(overdueCanaries()).toEqual([]);
|
||||
// Deliberately NOT `overdueCanaries()` with the ambient date. That assertion
|
||||
// reads wall-clock time, so it turns the entire @hyperframes/core suite red
|
||||
// on a calendar date for every unrelated PR — a broken build nobody caused
|
||||
// and whose fix is unrelated to the change under test. The registry's own
|
||||
// freshness is enforced by the pinned dates below plus the sunset REPORT,
|
||||
// which is advisory rather than a gate.
|
||||
it("every canary carries a parseable sunset date in the future at authoring time", () => {
|
||||
const authored = new Date("2026-07-31T00:00:00Z");
|
||||
for (const c of CANARIES) {
|
||||
const sunset = Date.parse(`${c.sunsetAfter}T00:00:00Z`);
|
||||
expect(Number.isFinite(sunset), `${c.name} has an unparseable sunsetAfter`).toBe(true);
|
||||
expect(sunset, `${c.name} was authored already-expired`).toBeGreaterThan(authored.getTime());
|
||||
}
|
||||
});
|
||||
|
||||
it("reports a canary as overdue only AFTER the whole sunset day has passed", () => {
|
||||
const [first] = CANARIES;
|
||||
if (!first) return;
|
||||
const day = first.sunsetAfter;
|
||||
expect(overdueCanaries(new Date(`${day}T00:00:00Z`))).not.toContain(first.name);
|
||||
expect(overdueCanaries(new Date(`${day}T23:59:59Z`))).not.toContain(first.name);
|
||||
const dayAfter = new Date(Date.parse(`${day}T00:00:00Z`) + 86_400_000);
|
||||
expect(overdueCanaries(dayAfter)).toContain(first.name);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user