Files
hyperframes/packages/engine/src/config.test.ts
T
Vance IngallsandClaude Fable 5 ec06f4bf89 feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net (#1998)
* feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net

Flip useDrawElement + worker-encode defaults on (HF_DE_BATCH default 4),
clamped in resolveConfig to hosts where drawElement can engage (macOS +
hardware-GPU browser) so page-side shader compositing is untouched
everywhere else; explicit env opt-in keeps attempt-and-gate semantics.

Safety net makes default-on safe: the compile/init gates catch predictable
incompatibility; this catches the intermittent residue no static analysis
can see (stale paints, dropped background images, transient blank frames).

- engine: captureDeVerificationFrames — K=4 (HF_DE_VERIFY) ground-truth
  screenshots at init, after gates + armStaticDedup, BEFORE canvas
  injection (post-injection screenshots show the canvas bitmap, not the
  DOM). Runs the video-injection hook per sample; double-captures so
  rAF-driven text counters settle (a single immediate screenshot captures
  stale text and false-positives). Skips png, <10 frames, implausible
  __hf.duration (infinite-repeat GSAP sentinel).
- producer: guardFrame on both worker-encode drains — rolling-median blank
  guard with retry-once at drain (byte-identical retry ⇒ deterministic dark
  frame, accepted; retry save/restores the static-dedup anchor) + ffmpeg
  PSNR self-verify vs ground truth (HF_DE_VERIFY_MIN_DB, default 32dB;
  natural agreement ≥45dB, damage ≤25dB). Breach dumps the frame pair to
  tmpdir and throws DrawElementVerificationError.
- orchestrator: one-shot retry — on verification error the whole render
  re-runs with forceScreenshot (slower, never wrong); telemetry flag
  deSelfVerifyFallback.
- tooling: de-canary-suite.sh (7-comp release gate with expected verdicts),
  de-gatecheck.sh (init-only corpus routing classifier), we-render.mjs.

Validated: canary suite 7/7; 611-comp routing sample 54% drawelement /
37.5% gated / 8.3% comp-defect; 12/12 risk-band renders clean on bare
defaults (48/48 verify samples); engine suite 888 passed; caught two real
intermittent damage classes in the wild (background-image drop, root-props
offset) that previously shipped silently.

Kill switches: PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false,
HF_DE_WORKER_ENCODE=false, HF_DE_BATCH=0, HF_DE_VERIFY=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engine,producer): harden the drawElement self-verification net (max code-review findings)

15 confirmed findings from the adversarial review of the default-on flip;
the load-bearing five:

- Ground-truth capture no longer scrubs GSAP state: seek(0) + forced frame
  FIRST (lazy .from()/overlap tweens record start values on first seek —
  mid-timeline scrubs corrupted them for the whole render, and since DE
  frames and truth shared the corruption, PSNR passed on damaged output),
  then ascending even-spread fractions, page left at frame 0.
- Default-on drawElement is confined to the verified path: resolveConfig
  requires worker-encode (the drain that runs the net), the orchestrator
  disengages the default when the render takes the disk path or parallel
  capture (no drain verification there), and closes a drawElement-initialized
  probe session rather than letting the unverified path reuse it. Explicit
  PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true keeps old attempt-and-gate behavior.
- Blank-frame retry can no longer splice wrong-frame pixels: recapture goes
  through recaptureDrawElementFrameForVerify — no static-dedup shortcut
  (lastEncodeResult runs ahead of the drain) and no "No cached paint record"
  screenshot fallback (post-injection that captures the canvas = the LAST
  drawn frame); any recapture failure falls back the whole render.
- Verify indices derive from the producer-resolved duration
  (CaptureOptions.compositionDurationSeconds) instead of raw __hf.duration,
  so samples always land inside the drained range.
- The platform clamp accepts "auto" GPU mode — the stock CLI resolves auto,
  and the literal-"hardware" clamp made default-on a no-op for the primary
  audience (masked in validation by explicitly-set env).

Also: NaN-safe env parses (HF_DE_VERIFY / HF_DE_VERIFY_MIN_DB / HF_DE_BATCH);
video comps skip verification when the session has no frame injector (probe
sessions — black-video truth false-positived); psnr infrastructure failures
skip the sample instead of failing the render; boundary-saturated sample
indices are skipped; shader-transition comps prefer page-side compositing
over default drawElement and compile-gated comps get page-side compositing
restored; observability.clearFailure un-brands the recovered first streaming
attempt; canary suite exempts known-marginal "any" comps from the cross-path
PSNR gate; dead we-render options removed; clamp tests pin their env.

Validated: canary suite 7/7; auto-GPU bare render engages the full stack;
disk-path and worker-encode-off renders disengage default drawElement;
malformed HF_DE_VERIFY_MIN_DB still verifies at the default threshold;
engine suite 890 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engine,producer): review fixes — PSC intent, verify-threshold clamp, fail-closed canaries

Addresses miguel-heygen's review on #1998:

- Page-side compositing restore preserves explicit caller intent (blocker):
  resolveConfig now records pageSideCompositingAutoDisabled only when IT
  turned page-side compositing off because drawElement was on; the
  compile-time drawElement gates restore page-side compositing only when
  that flag is set. An explicit enablePageSideCompositing:false from the
  programmatic API or HF_PAGE_SIDE_COMPOSITING=false stays off. Pinned by
  two config tests.
- HF_DE_VERIFY_MIN_DB clamped to [10, 60] with a warning on out-of-range
  values: below ~10dB the check passes severe damage; above ~60dB natural
  encoder differences force a screenshot fallback on every verified render.
- de-canary-suite.sh + de-gatecheck.sh run under set -euo pipefail with
  explicit `|| true` on expected-nonzero commands (render exits handled by
  the suite's own checks, grep no-match, kill/pkill/wait races) and a hard
  FAIL when the PSNR compare produces no value — release canaries fail
  closed. Full suite re-run green (7/7) under the new flags.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 16:29:30 -07:00

368 lines
13 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { resolveConfig, DEFAULT_CONFIG, scaleProtocolTimeoutForComposition } from "./config.js";
import { isLowMemorySystem } from "./services/systemMemory.js";
describe("resolveConfig", () => {
const savedEnv = new Map<string, string | undefined>();
function setEnv(key: string, value: string) {
if (!savedEnv.has(key)) savedEnv.set(key, process.env[key]);
process.env[key] = value;
}
function unsetEnv(key: string) {
if (!savedEnv.has(key)) savedEnv.set(key, process.env[key]);
delete process.env[key];
}
beforeEach(() => {
savedEnv.clear();
});
afterEach(() => {
for (const [key, value] of savedEnv) {
if (value === undefined) {
delete process.env[key];
} else {
process.env[key] = value;
}
}
});
it("returns defaults when no overrides or env vars are set", () => {
const config = resolveConfig();
expect(config.fps).toBe(30);
expect(config.quality).toBe("standard");
expect(config.format).toBe("jpeg");
expect(config.jpegQuality).toBe(80);
expect(config.browserGpuMode).toBe("software");
expect(config.enableStreamingEncode).toBe(true);
expect(config.streamingEncodeMaxDurationSeconds).toBe(240);
expect((config as Record<string, unknown>).vp9CpuUsed).toBe(4);
expect(config.audioGain).toBe(1);
expect(config.debug).toBe(false);
});
it("applies explicit overrides over defaults", () => {
const config = resolveConfig({ fps: 60, debug: true });
expect(config.fps).toBe(60);
expect(config.debug).toBe(true);
// Non-overridden fields remain at defaults
expect(config.quality).toBe("standard");
});
it("reads numeric env vars with PRODUCER_ prefix", () => {
setEnv("PRODUCER_MAX_WORKERS", "4");
setEnv("PRODUCER_CORES_PER_WORKER", "3");
const config = resolveConfig();
expect(config.concurrency).toBe(4);
expect(config.coresPerWorker).toBe(3);
});
it("reads boolean env vars (true/false strings)", () => {
setEnv("PRODUCER_DISABLE_GPU", "true");
setEnv("PRODUCER_ENABLE_BROWSER_POOL", "true");
const config = resolveConfig();
expect(config.disableGpu).toBe(true);
expect(config.enableBrowserPool).toBe(true);
});
it("lets env vars opt out of default streaming encode", () => {
setEnv("PRODUCER_ENABLE_STREAMING_ENCODE", "false");
const config = resolveConfig();
expect(config.enableStreamingEncode).toBe(false);
});
it("reads the streaming encode duration cutoff from env", () => {
setEnv("PRODUCER_STREAMING_ENCODE_MAX_DURATION_SECONDS", "120");
const config = resolveConfig();
expect(config.streamingEncodeMaxDurationSeconds).toBe(120);
});
it("clamps negative streaming encode duration cutoff env values to zero", () => {
setEnv("PRODUCER_STREAMING_ENCODE_MAX_DURATION_SECONDS", "-1");
const config = resolveConfig();
expect(config.streamingEncodeMaxDurationSeconds).toBe(0);
});
it("reads VP9 cpu-used from env", () => {
setEnv("PRODUCER_VP9_CPU_USED", "6");
const config = resolveConfig();
expect((config as Record<string, unknown>).vp9CpuUsed).toBe(6);
});
it("falls back to the VP9 cpu-used default for invalid env values", () => {
setEnv("PRODUCER_VP9_CPU_USED", "fast");
const config = resolveConfig();
expect((config as Record<string, unknown>).vp9CpuUsed).toBe(4);
});
it("clamps VP9 cpu-used env values to libvpx's supported range", () => {
setEnv("PRODUCER_VP9_CPU_USED", "99");
expect((resolveConfig() as Record<string, unknown>).vp9CpuUsed).toBe(8);
process.env.PRODUCER_VP9_CPU_USED = "-99";
expect((resolveConfig() as Record<string, unknown>).vp9CpuUsed).toBe(-8);
});
it("treats non-'true' boolean env vars as false", () => {
setEnv("PRODUCER_DISABLE_GPU", "yes");
const config = resolveConfig();
expect(config.disableGpu).toBe(false);
});
it("reads browser GPU mode from env", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "hardware");
const config = resolveConfig();
expect(config.browserGpuMode).toBe("hardware");
});
it("accepts 'auto' as a valid browser GPU mode env value", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "auto");
const config = resolveConfig();
expect(config.browserGpuMode).toBe("auto");
});
it("falls back to software browser GPU mode for invalid env values", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "native");
const config = resolveConfig();
expect(config.browserGpuMode).toBe("software");
});
it("explicit overrides take precedence over env vars", () => {
setEnv("PRODUCER_CORES_PER_WORKER", "5");
const config = resolveConfig({ coresPerWorker: 8 });
expect(config.coresPerWorker).toBe(8);
});
it("falls back to defaults for invalid numeric env vars", () => {
setEnv("PRODUCER_CORES_PER_WORKER", "not-a-number");
const config = resolveConfig();
expect(config.coresPerWorker).toBe(DEFAULT_CONFIG.coresPerWorker);
});
it("clamps chunkSizeFrames to minimum of 120", () => {
setEnv("PRODUCER_CHUNK_SIZE_FRAMES", "50");
const config = resolveConfig();
expect(config.chunkSizeFrames).toBe(120);
});
it("clamps frameDataUriCacheLimit to minimum of 32", () => {
setEnv("PRODUCER_FRAME_DATA_URI_CACHE_LIMIT", "10");
const config = resolveConfig();
expect(config.frameDataUriCacheLimit).toBe(32);
});
describe("enablePageSideCompositing (HF_PAGE_SIDE_COMPOSITING)", () => {
it("defaults to true", () => {
const config = resolveConfig();
expect(config.enablePageSideCompositing).toBe(true);
});
it("disabled when HF_PAGE_SIDE_COMPOSITING=false", () => {
setEnv("HF_PAGE_SIDE_COMPOSITING", "false");
const config = resolveConfig();
expect(config.enablePageSideCompositing).toBe(false);
});
it("explicit override wins over the env var", () => {
setEnv("HF_PAGE_SIDE_COMPOSITING", "true");
const config = resolveConfig({ enablePageSideCompositing: false });
expect(config.enablePageSideCompositing).toBe(false);
});
});
describe("extraction cache env", () => {
it("defaults the extract cache directory to tmpdir plus uid when env is unset", () => {
unsetEnv("HYPERFRAMES_EXTRACT_CACHE_DIR");
const config = resolveConfig();
expect(config.extractCacheDir).toBe(
join(tmpdir(), `hyperframes-extract-cache-${process.getuid?.() ?? "u"}`),
);
});
it("disables the extract cache when env is an opt-out token", () => {
for (const value of ["off", "none", "false", "0", " OFF "]) {
setEnv("HYPERFRAMES_EXTRACT_CACHE_DIR", value);
expect(resolveConfig().extractCacheDir).toBeUndefined();
}
});
it("uses an explicit extract cache path from env", () => {
setEnv("HYPERFRAMES_EXTRACT_CACHE_DIR", "/tmp/custom-hf-cache");
expect(resolveConfig().extractCacheDir).toBe("/tmp/custom-hf-cache");
});
it("converts HYPERFRAMES_EXTRACT_CACHE_MAX_MB to bytes", () => {
setEnv("HYPERFRAMES_EXTRACT_CACHE_MAX_MB", "512");
expect(resolveConfig().extractCacheMaxBytes).toBe(512 * 1024 ** 2);
});
});
describe("useDrawElement (PRODUCER_EXPERIMENTAL_FAST_CAPTURE)", () => {
it("default is clamped off on software-GPU hosts (page-side compositing preserved)", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "software");
unsetEnv("PRODUCER_EXPERIMENTAL_FAST_CAPTURE");
unsetEnv("HF_DE_WORKER_ENCODE");
const config = resolveConfig();
expect(config.useDrawElement).toBe(false);
expect(config.enablePageSideCompositing).toBe(true);
});
it("default engages on macOS with a hardware-GPU browser", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "hardware");
unsetEnv("PRODUCER_EXPERIMENTAL_FAST_CAPTURE");
unsetEnv("HF_DE_WORKER_ENCODE");
const config = resolveConfig();
expect(config.useDrawElement).toBe(process.platform === "darwin");
});
it("default engages on macOS with auto GPU mode (the stock CLI path)", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "auto");
unsetEnv("PRODUCER_EXPERIMENTAL_FAST_CAPTURE");
unsetEnv("HF_DE_WORKER_ENCODE");
const config = resolveConfig();
expect(config.useDrawElement).toBe(process.platform === "darwin");
});
it("default requires worker-encode (the verified drain)", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "hardware");
setEnv("HF_DE_WORKER_ENCODE", "false");
unsetEnv("PRODUCER_EXPERIMENTAL_FAST_CAPTURE");
const config = resolveConfig();
expect(config.useDrawElement).toBe(false);
});
it("explicit env opt-in skips the platform clamp", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "software");
setEnv("PRODUCER_EXPERIMENTAL_FAST_CAPTURE", "true");
const config = resolveConfig();
expect(config.useDrawElement).toBe(true);
});
it("env kill switch wins over the default", () => {
setEnv("PRODUCER_BROWSER_GPU_MODE", "hardware");
setEnv("PRODUCER_EXPERIMENTAL_FAST_CAPTURE", "false");
const config = resolveConfig();
expect(config.useDrawElement).toBe(false);
});
it("explicit override wins over the env var", () => {
setEnv("PRODUCER_EXPERIMENTAL_FAST_CAPTURE", "true");
const config = resolveConfig({ useDrawElement: false });
expect(config.useDrawElement).toBe(false);
});
it("forces page-side compositing off when enabled (incompatible strategies)", () => {
const config = resolveConfig({ useDrawElement: true, enablePageSideCompositing: true });
expect(config.useDrawElement).toBe(true);
expect(config.enablePageSideCompositing).toBe(false);
// The auto-disable is recorded so compile-time gates can restore it.
expect(config.pageSideCompositingAutoDisabled).toBe(true);
});
it("does NOT mark auto-disabled when the caller explicitly opted out of page-side compositing", () => {
const config = resolveConfig({ useDrawElement: true, enablePageSideCompositing: false });
expect(config.enablePageSideCompositing).toBe(false);
// Explicit caller intent — a compile-time drawElement gate must not restore it.
expect(config.pageSideCompositingAutoDisabled).not.toBe(true);
});
it("leaves page-side compositing on when fast capture is off", () => {
const config = resolveConfig({ useDrawElement: false });
expect(config.enablePageSideCompositing).toBe(true);
});
});
describe("lowMemoryMode", () => {
it("forces on for truthy PRODUCER_LOW_MEMORY_MODE values", () => {
setEnv("PRODUCER_LOW_MEMORY_MODE", "true");
for (const v of ["true", "on", "1", "TRUE"]) {
process.env.PRODUCER_LOW_MEMORY_MODE = v;
expect(resolveConfig().lowMemoryMode).toBe(true);
}
});
it("forces off for falsy PRODUCER_LOW_MEMORY_MODE values", () => {
setEnv("PRODUCER_LOW_MEMORY_MODE", "false");
for (const v of ["false", "off", "0", "OFF"]) {
process.env.PRODUCER_LOW_MEMORY_MODE = v;
expect(resolveConfig().lowMemoryMode).toBe(false);
}
});
it("auto-detects from total RAM when the env var is unset", () => {
setEnv("PRODUCER_LOW_MEMORY_MODE", "");
delete process.env.PRODUCER_LOW_MEMORY_MODE;
expect(resolveConfig().lowMemoryMode).toBe(isLowMemorySystem());
});
it("explicit override beats both env and auto-detection", () => {
setEnv("PRODUCER_LOW_MEMORY_MODE", "true");
expect(resolveConfig({ lowMemoryMode: false }).lowMemoryMode).toBe(false);
});
});
});
describe("scaleProtocolTimeoutForComposition", () => {
const base = 300_000;
it("keeps the base timeout for a reference-or-smaller canvas", () => {
// 1080p == reference area → factor 1, no scale.
expect(scaleProtocolTimeoutForComposition(base, { width: 1920, height: 1080 })).toBe(base);
// Smaller than reference → still the base (never scales down).
expect(scaleProtocolTimeoutForComposition(base, { width: 1280, height: 720 })).toBe(base);
});
it("scales up proportionally with output pixel area", () => {
// 4K == 4× the reference area, which stays under the 30-minute ceiling.
const scaled = scaleProtocolTimeoutForComposition(base, { width: 3840, height: 2160 });
expect(scaled).toBeGreaterThan(base);
expect(scaled).toBe(base * 4);
});
it("clamps at the 30-minute ceiling for a pathological canvas", () => {
// 8K == 16× area → 4.8M ms, clamped to the 30-minute ceiling.
const scaled = scaleProtocolTimeoutForComposition(base, { width: 7680, height: 4320 });
expect(scaled).toBe(1_800_000);
});
it("never lowers a base timeout that already exceeds the ceiling", () => {
// Base above the 30-min ceiling + a large canvas: must not clamp below base.
const highBase = 2_400_000;
expect(
scaleProtocolTimeoutForComposition(highBase, { width: 3840, height: 2160 }),
).toBeGreaterThanOrEqual(highBase);
});
it("returns the base timeout for degenerate dimensions", () => {
expect(scaleProtocolTimeoutForComposition(base, { width: 0, height: 1080 })).toBe(base);
expect(scaleProtocolTimeoutForComposition(base, { width: 1920, height: 0 })).toBe(base);
expect(scaleProtocolTimeoutForComposition(base, { width: Number.NaN, height: 1080 })).toBe(
base,
);
});
});