mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-12 07:09:59 +00:00
R5 blockers
- Negative install-state latch was cached for the process lifetime, but
only `true` is monotonic across processes. A long-lived preview server
held a stale `false` and could re-enrol after another process tripped
the breaker. Only the positive is cached now; `false` re-reads.
- The real breaker writer used writeConfig(), which collapses
{ok:true, mirrored:false} to success, so a run that mirrored nothing
reported done with the latch only on the erasable store. It consumes
writeConfigWithResult and retries until both stores carry it.
Bucketing integrity
- Storage-restricted Studio profiles all bucketed on the literal
"anonymous": computed against the shipped hash, 100% of them were
enrolled in calibration-50 rather than 50%, and they merged into one
PostHog person. Per-session random id instead — persists nothing.
- bucketSeed had read/write authority backwards: install-state is
write-once authoritative, but readConfig took config.json's blindly, so
the stores could hold different seeds until a re-mint flipped every
cohort. Merged on read, like the latch.
- An unwritable ~/.hyperframes with no config.json re-minted per call,
re-rolling the seed on every command, and the "cohorts will not be
stable" warning was unreachable on that path.
- A corrupt PRE-MOVE state file was never deleted, so a machine reset
with `rm -rf ~/.hyperframes` reported predecessorFound/stateFileCorrupt
forever — poisoning the exact metric this work exists to produce.
Opt-out honoring
- CLI canary decisions memoized per process, so `hyperframes telemetry
disable` during a running preview server was ignored for hours while
the server kept serving pre-opt-out decisions. The memo is keyed on the
telemetry posture.
- shouldTrack() memoized, contradicting policy.ts's documented "not
memoized" contract that policy.test.ts asserts.
- The Studio override path resolved the bucket unit eagerly as an
argument, minting and PERSISTING a tracking id for an opted-out profile
— a value evaluateCanary discards unread.
- Storage reads could throw out of telemetry into a post-commit catch
block, reporting an already-committed edit as failed.
- readConfig printed an unsilenceable stderr warning on every invocation
for installs that opted out of telemetry entirely.
Host split
- isLoopbackHost rejected 0.0.0.0, so the documented
HYPERFRAMES_PREVIEW_HOST LAN mode silently lost CLI→Studio identity
stitching and split one user across two PostHog persons. Identity is
now allowed when the operator explicitly opted into LAN binding.
- Corrected the comment claiming the guard refuses spoofed Hosts: a
non-browser client sets Host freely. It is a browser DNS-rebinding
mitigation, not access control, and now says so.
Semantics and test hygiene
- percentage:100 did not mean everyone — exclude and no_unit_id sat above
the fast path, so the registry's "delete the entry at 100" step was an
unstaged flip for CI and seedless installs.
- CLI cohort adoption returned before evaluateCanary, dropping Studio's
own webdriver exclusion.
- overdueCanaries() was asserted against wall-clock time, so the whole
core suite would go red on 2026-09-15 for every unrelated PR; and `>`
against midnight made a canary overdue ON its sunset date.
- Statistical assertions ran on unseeded randomUUID() populations tight
enough to fail ~1 run in 200. Seeded.
Also: broke a config -> policy -> transport -> config import cycle by
moving POSTHOG_API_KEY to a leaf module.
Tests: 2347 CLI (bundle absent), 3153 Studio, 1450 core. Fault injection
covers the latch, seed authority, LAN identity, webdriver exclusion and
the anonymous-bucket fix. Two pre-existing tests asserted behaviour these
findings identify as wrong (shouldTrack memoization, 100%-excludes-CI)
and were rewritten with the reasoning stated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
150 lines
5.4 KiB
TypeScript
150 lines
5.4 KiB
TypeScript
import { afterEach, describe, expect, it, vi } from "vitest";
|
|
|
|
const baseConfig = {
|
|
telemetryEnabled: true,
|
|
anonymousId: "test-install",
|
|
telemetryNoticeShown: true,
|
|
commandCount: 7,
|
|
renderSuccessCount: 0,
|
|
lastFeedbackPromptAt: 0,
|
|
};
|
|
|
|
async function loadTelemetryCommand(options?: {
|
|
writeSucceeds?: boolean;
|
|
configEnabled?: boolean;
|
|
devMode?: boolean;
|
|
apiKey?: string;
|
|
}) {
|
|
const config = {
|
|
...baseConfig,
|
|
telemetryEnabled: options?.configEnabled ?? true,
|
|
};
|
|
const writeConfigWithResult = vi.fn(() =>
|
|
options?.writeSucceeds === false
|
|
? { ok: false as const, error: "EACCES: permission denied" }
|
|
: { ok: true as const },
|
|
);
|
|
vi.resetModules();
|
|
vi.doMock("../telemetry/config.js", () => ({
|
|
CONFIG_PATH: "/test/.hyperframes/config.json",
|
|
STATE_PATH: "/test/.local/state/hyperframes/install-state.json",
|
|
readConfig: () => {
|
|
throw new Error("telemetry commands must bypass stale cached config");
|
|
},
|
|
readConfigFresh: () => ({ ...config }),
|
|
writeConfigWithResult,
|
|
}));
|
|
vi.doMock("../utils/env.js", () => ({
|
|
isDevMode: () => options?.devMode ?? false,
|
|
}));
|
|
// The key moved to a leaf module to break a config -> policy -> transport
|
|
// -> config import cycle; policy.ts reads it from there now.
|
|
vi.doMock("../telemetry/posthogKey.js", () => ({
|
|
POSTHOG_API_KEY: options?.apiKey ?? "phc_test",
|
|
}));
|
|
const module = await import("./telemetry.js");
|
|
return { command: module.default, writeConfigWithResult };
|
|
}
|
|
|
|
async function runSubcommand(
|
|
command: Awaited<ReturnType<typeof loadTelemetryCommand>>["command"],
|
|
subcommand: string,
|
|
): Promise<void> {
|
|
await command.run?.({
|
|
args: { subcommand },
|
|
rawArgs: [subcommand],
|
|
cmd: command,
|
|
} as never);
|
|
}
|
|
|
|
async function runWithCapturedOutput(
|
|
command: Awaited<ReturnType<typeof loadTelemetryCommand>>["command"],
|
|
subcommand: string,
|
|
): Promise<string> {
|
|
const lines: string[] = [];
|
|
vi.spyOn(console, "log").mockImplementation((...args: unknown[]) => {
|
|
lines.push(args.map(String).join(" "));
|
|
});
|
|
await runSubcommand(command, subcommand);
|
|
return lines.join("\n");
|
|
}
|
|
|
|
describe("telemetry command", () => {
|
|
afterEach(() => {
|
|
vi.doUnmock("../telemetry/config.js");
|
|
vi.doUnmock("../utils/env.js");
|
|
vi.doUnmock("../telemetry/transport.js");
|
|
vi.restoreAllMocks();
|
|
vi.resetModules();
|
|
delete process.env["HYPERFRAMES_NO_TELEMETRY"];
|
|
delete process.env["DO_NOT_TRACK"];
|
|
});
|
|
|
|
it("persists disable from a fresh config snapshot", async () => {
|
|
const { command, writeConfigWithResult } = await loadTelemetryCommand();
|
|
vi.spyOn(console, "log").mockImplementation(() => undefined);
|
|
|
|
await runSubcommand(command, "disable");
|
|
|
|
expect(writeConfigWithResult).toHaveBeenCalledWith(
|
|
expect.objectContaining({ telemetryEnabled: false }),
|
|
);
|
|
});
|
|
|
|
it("fails instead of claiming success when the preference cannot be persisted", async () => {
|
|
const { command } = await loadTelemetryCommand({ writeSucceeds: false });
|
|
const stdout = vi.spyOn(console, "log").mockImplementation(() => undefined);
|
|
const stderr = vi.spyOn(console, "error").mockImplementation(() => undefined);
|
|
|
|
await expect(runSubcommand(command, "disable")).rejects.toMatchObject({
|
|
name: "CliRuntimeError",
|
|
});
|
|
|
|
expect(stderr).toHaveBeenCalledWith(expect.stringContaining("Could not persist"));
|
|
expect(stderr).toHaveBeenCalledWith(expect.stringContaining("EACCES: permission denied"));
|
|
expect(stdout).not.toHaveBeenCalledWith(expect.stringContaining("Telemetry disabled"));
|
|
});
|
|
|
|
it("reports the effective env-var opt-out instead of the stored preference", async () => {
|
|
process.env["HYPERFRAMES_NO_TELEMETRY"] = "1";
|
|
const { command } = await loadTelemetryCommand({ configEnabled: true });
|
|
const output = await runWithCapturedOutput(command, "status");
|
|
expect(output).toContain("disabled");
|
|
expect(output).toContain("HYPERFRAMES_NO_TELEMETRY");
|
|
expect(output).toContain("Tracked commands:");
|
|
});
|
|
|
|
it("reports DO_NOT_TRACK as the effective opt-out source", async () => {
|
|
process.env["DO_NOT_TRACK"] = "1";
|
|
const { command } = await loadTelemetryCommand({ configEnabled: true });
|
|
const output = await runWithCapturedOutput(command, "status");
|
|
expect(output).toContain("DO_NOT_TRACK");
|
|
});
|
|
|
|
it.each([
|
|
["enable", "Telemetry preference"],
|
|
["disable", "Telemetry disabled"],
|
|
])("explains the effective override after telemetry %s", async (subcommand, expectedSuccess) => {
|
|
process.env["HYPERFRAMES_NO_TELEMETRY"] = "true";
|
|
const { command } = await loadTelemetryCommand({ configEnabled: subcommand === "disable" });
|
|
const output = await runWithCapturedOutput(command, subcommand);
|
|
expect(output).toContain(expectedSuccess);
|
|
expect(output).toContain("remains disabled");
|
|
expect(output).toContain("HYPERFRAMES_NO_TELEMETRY");
|
|
});
|
|
|
|
it("reports dev mode as the effective source", async () => {
|
|
const { command } = await loadTelemetryCommand({ devMode: true });
|
|
const output = await runWithCapturedOutput(command, "status");
|
|
expect(output).toContain("disabled");
|
|
expect(output).toContain("dev_mode");
|
|
});
|
|
|
|
it("reports a telemetry-disabled build as the effective source", async () => {
|
|
const { command } = await loadTelemetryCommand({ apiKey: "disabled" });
|
|
const output = await runWithCapturedOutput(command, "status");
|
|
expect(output).toContain("disabled");
|
|
expect(output).toContain("telemetry_disabled_build");
|
|
});
|
|
});
|