ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-03 12:54:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,752 Commits 855 Branches 404 Tags
a0f2a65bc216cb69a08498bb2eb34aaaf630393d
Commit Graph
16 Commits
Author SHA1 Message Date
Vance Ingalls f44bc3a525 fix(engine,producer): drive forceScreenshot from one authoritative local 
Miguel R4 blocker on #2359: my R3 fix at renderOrchestrator only updated
the observability copy, leaving the authoritative captureForceScreenshot
local at compileResult.forceScreenshot (false for auto→software). The
frameCapture side clamped its own local and correctly routed screenshot,
but downstream orchestrator code overwrote observability back to
beginframe from the still-false local at two sites:

  - Parallel-stream label at renderOrchestrator.ts:2293 mis-labelled the
    stream as 'beginframe' when actual capture was 'screenshot'.
  - capture_strategy telemetry at renderOrchestrator.ts:2440-2450
    overwrote the earlier observability correction, so the final
    captureMode observation flipped back to 'beginframe' while the
    engine actually captured screenshot.

Fix: extract the clamp into a caller-facing helper
applyConcreteGpuScreenshotClamp(current, resolvedGpuMode, cfg) that
returns the (possibly-promoted) new boolean. Callers assign it back to
their authoritative local, so routing + telemetry + strategy code read
one value.

Changes:

  - packages/engine/src/config.ts: new exported
    applyConcreteGpuScreenshotClamp; delegates to
    shouldClampToScreenshotForConcreteGpu but computes the caller's
    final value, not just the clamp decision. Reads the programmatic
    opt-out from cfg.forceScreenshotExplicitlyOptedOut. Idempotent on
    already-true input.
  - packages/engine/src/index.ts: export the new helper.
  - packages/engine/src/services/frameCapture.ts: replace the inline
    OR expression with applyConcreteGpuScreenshotClamp.
  - packages/producer/src/services/renderOrchestrator.ts: assign result
    into the AUTHORITATIVE captureForceScreenshot local (was updating
    only observability). Downstream parallel-stream label at :2293 and
    capture_strategy telemetry at :2440-2450 now read the corrected
    value.

Tests: 6 new caller-level cases for applyConcreteGpuScreenshotClamp
covering the exact matrix Miguel called out:

  - resolved software + default false → promotes to true (screenshot)
  - resolved software + programmatic opt-out → stays false (BeginFrame)
  - resolved hardware + default false → stays false
  - resolved software + already-true → stays true (idempotent)
  - resolved software + env PRODUCER_FORCE_SCREENSHOT=false → stays false
  - resolved software + undefined cfg → promotes to true (frameCapture path)

Local: 67/67 engine config tests pass (was 61). oxfmt clean.
 2026-07-14 03:17:29 +00:00
Vance Ingalls 72daac2a1d fix(engine): carry programmatic forceScreenshot opt-out to concrete-resolved site 
Miguel R3 blocker on #2359: the runtime helper only checked the env opt-out
(PRODUCER_FORCE_SCREENSHOT=false), silently defeating the documented
programmatic escape hatch (overrides.forceScreenshot === false) on the
browserGpuMode:'auto' → software probe path. At the concrete-resolution
site the boolean forceScreenshot === false is ambiguous between default
and explicit opt-out — resolveConfig sees the provenance but the runtime
helper does not.

Fix: persist provenance on the resolved config.

- New INTERNAL EngineConfig field forceScreenshotExplicitlyOptedOut, set
  by resolveConfig when EITHER env or programmatic explicit-false is
  present. Purpose-documented in the type as 'not intended to be set by
  callers'.
- shouldClampToScreenshotForConcreteGpu gains an opts.programmaticOptOut
  parameter; returns false early when set. Env stays as the third arg
  (backward compatibility with existing tests).
- frameCapture.ts and renderOrchestrator.ts pass
  config.forceScreenshotExplicitlyOptedOut through at both call sites, so
  the auto→software probe path preserves the same escape hatches as
  literal browserGpuMode:'software'.

New tests: 5 additional cases across the helper (programmatic opt-out
alone; programmatic beats missing env) and resolveConfig provenance
(programmatic sets flag; env sets flag; neither leaves it undefined).
Local: 61/61 engine config tests pass (was 56).
 2026-07-14 03:06:34 +00:00
Vance Ingalls 2e44602f99 fix(engine,producer): apply software-GPU screenshot invariant at concrete-resolved point 
Addresses Miguel's R1 blockers:

1. `browserGpuMode: "auto"` that runtime-probes to software slipped past the
   `resolveConfig` clamp — that clamp only sees the pre-resolve string. Add
   `shouldClampToScreenshotForConcreteGpu(resolvedGpuMode, currentForceScreenshot, env)`
   in `packages/engine/src/config.ts` and apply it at BOTH concrete-resolution
   sites:
   - `packages/engine/src/services/frameCapture.ts`: downgrades `preMode`
     from "beginframe" to "screenshot" when resolved GPU is software (respects
     `PRODUCER_FORCE_SCREENSHOT=false` env opt-out), fixing the routing.
   - `packages/producer/src/services/renderOrchestrator.ts`: updates
     `captureObservability.forceScreenshot` (and thus `captureMode`) at the
     same call site, fixing the observability truth on the auto → software
     case.

2. New unit tests in `config.test.ts`:
   - Documents the auto-branch gap (resolveConfig leaves auto as
     forceScreenshot=false — the runtime companion closes it).
   - 5 branch tests on `shouldClampToScreenshotForConcreteGpu` covering
     software / hardware / already-forced / env-opt-out / non-"false" env
     values.
   Full suite: 56/56 pass.

Scope narrowing on Blocker 2: the distributed rendering path at
`packages/producer/src/services/distributed/plan.ts:753-754` and
`renderChunk.ts:462-466` explicitly hardcodes `browserGpuMode:"software",
forceScreenshot:false` post-resolveConfig and stays outside this PR's
invariant boundary. `compileStage` may still flip it to true for alpha
formats, but generic MP4 distributed renders on SwiftShader hosts remain
BeginFrame. That's a separate architectural cleanup (needs its own
behavior-change trace); the PR body now scopes the invariant to the
in-process CLI/orchestrator path.
 2026-07-13 23:13:30 +00:00
Vance Ingalls ba5168293f fix(engine): software-GPU browsers imply screenshot capture 
When `browserGpuMode === "software"`, set `forceScreenshot = true` in
`resolveConfig`. Explicit opt-outs (`PRODUCER_FORCE_SCREENSHOT=false`
or `overrides.forceScreenshot === false`) are honored.

This is defense-in-depth on top of the existing platform gates:

1. Linux + software (SwiftShader host) skips BeginFrame, avoiding the
   compositor stall on shader-heavy frames under CPU raster (same
   motivation as the closed PR #822).
2. `renderOrchestrator`'s reported `captureMode` field is derived from
   `cfg.forceScreenshot ? "screenshot" : "beginframe"` — without this
   clamp it misreports `"beginframe"` for the actual screenshot capture
   on darwin + software.
3. Any new BeginFrame or drawElement entry point that forgets to gate
   on GPU mode still routes to screenshot here.

Does NOT fix SwiftShader-on-darwin text-rasterization artifacts (an
ANGLE-SwiftShader issue on macOS text — the fix there is to use
`--browser-gpu`, which routes to `--use-angle=metal`).
 2026-07-13 21:38:10 +00:00
Vance IngallsandClaude Fable 5 ec06f4bf89 feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net (#1998) 
* feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net

Flip useDrawElement + worker-encode defaults on (HF_DE_BATCH default 4),
clamped in resolveConfig to hosts where drawElement can engage (macOS +
hardware-GPU browser) so page-side shader compositing is untouched
everywhere else; explicit env opt-in keeps attempt-and-gate semantics.

Safety net makes default-on safe: the compile/init gates catch predictable
incompatibility; this catches the intermittent residue no static analysis
can see (stale paints, dropped background images, transient blank frames).

- engine: captureDeVerificationFrames — K=4 (HF_DE_VERIFY) ground-truth
  screenshots at init, after gates + armStaticDedup, BEFORE canvas
  injection (post-injection screenshots show the canvas bitmap, not the
  DOM). Runs the video-injection hook per sample; double-captures so
  rAF-driven text counters settle (a single immediate screenshot captures
  stale text and false-positives). Skips png, <10 frames, implausible
  __hf.duration (infinite-repeat GSAP sentinel).
- producer: guardFrame on both worker-encode drains — rolling-median blank
  guard with retry-once at drain (byte-identical retry ⇒ deterministic dark
  frame, accepted; retry save/restores the static-dedup anchor) + ffmpeg
  PSNR self-verify vs ground truth (HF_DE_VERIFY_MIN_DB, default 32dB;
  natural agreement ≥45dB, damage ≤25dB). Breach dumps the frame pair to
  tmpdir and throws DrawElementVerificationError.
- orchestrator: one-shot retry — on verification error the whole render
  re-runs with forceScreenshot (slower, never wrong); telemetry flag
  deSelfVerifyFallback.
- tooling: de-canary-suite.sh (7-comp release gate with expected verdicts),
  de-gatecheck.sh (init-only corpus routing classifier), we-render.mjs.

Validated: canary suite 7/7; 611-comp routing sample 54% drawelement /
37.5% gated / 8.3% comp-defect; 12/12 risk-band renders clean on bare
defaults (48/48 verify samples); engine suite 888 passed; caught two real
intermittent damage classes in the wild (background-image drop, root-props
offset) that previously shipped silently.

Kill switches: PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false,
HF_DE_WORKER_ENCODE=false, HF_DE_BATCH=0, HF_DE_VERIFY=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engine,producer): harden the drawElement self-verification net (max code-review findings)

15 confirmed findings from the adversarial review of the default-on flip;
the load-bearing five:

- Ground-truth capture no longer scrubs GSAP state: seek(0) + forced frame
  FIRST (lazy .from()/overlap tweens record start values on first seek —
  mid-timeline scrubs corrupted them for the whole render, and since DE
  frames and truth shared the corruption, PSNR passed on damaged output),
  then ascending even-spread fractions, page left at frame 0.
- Default-on drawElement is confined to the verified path: resolveConfig
  requires worker-encode (the drain that runs the net), the orchestrator
  disengages the default when the render takes the disk path or parallel
  capture (no drain verification there), and closes a drawElement-initialized
  probe session rather than letting the unverified path reuse it. Explicit
  PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true keeps old attempt-and-gate behavior.
- Blank-frame retry can no longer splice wrong-frame pixels: recapture goes
  through recaptureDrawElementFrameForVerify — no static-dedup shortcut
  (lastEncodeResult runs ahead of the drain) and no "No cached paint record"
  screenshot fallback (post-injection that captures the canvas = the LAST
  drawn frame); any recapture failure falls back the whole render.
- Verify indices derive from the producer-resolved duration
  (CaptureOptions.compositionDurationSeconds) instead of raw __hf.duration,
  so samples always land inside the drained range.
- The platform clamp accepts "auto" GPU mode — the stock CLI resolves auto,
  and the literal-"hardware" clamp made default-on a no-op for the primary
  audience (masked in validation by explicitly-set env).

Also: NaN-safe env parses (HF_DE_VERIFY / HF_DE_VERIFY_MIN_DB / HF_DE_BATCH);
video comps skip verification when the session has no frame injector (probe
sessions — black-video truth false-positived); psnr infrastructure failures
skip the sample instead of failing the render; boundary-saturated sample
indices are skipped; shader-transition comps prefer page-side compositing
over default drawElement and compile-gated comps get page-side compositing
restored; observability.clearFailure un-brands the recovered first streaming
attempt; canary suite exempts known-marginal "any" comps from the cross-path
PSNR gate; dead we-render options removed; clamp tests pin their env.

Validated: canary suite 7/7; auto-GPU bare render engages the full stack;
disk-path and worker-encode-off renders disengage default drawElement;
malformed HF_DE_VERIFY_MIN_DB still verifies at the default threshold;
engine suite 890 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(engine,producer): review fixes — PSC intent, verify-threshold clamp, fail-closed canaries

Addresses miguel-heygen's review on #1998:

- Page-side compositing restore preserves explicit caller intent (blocker):
  resolveConfig now records pageSideCompositingAutoDisabled only when IT
  turned page-side compositing off because drawElement was on; the
  compile-time drawElement gates restore page-side compositing only when
  that flag is set. An explicit enablePageSideCompositing:false from the
  programmatic API or HF_PAGE_SIDE_COMPOSITING=false stays off. Pinned by
  two config tests.
- HF_DE_VERIFY_MIN_DB clamped to [10, 60] with a warning on out-of-range
  values: below ~10dB the check passes severe damage; above ~60dB natural
  encoder differences force a screenshot fallback on every verified render.
- de-canary-suite.sh + de-gatecheck.sh run under set -euo pipefail with
  explicit `|| true` on expected-nonzero commands (render exits handled by
  the suite's own checks, grep no-match, kill/pkill/wait races) and a hard
  FAIL when the PSNR compare produces no value — release canaries fail
  closed. Full suite re-run green (7/7) under the new flags.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-06 16:29:30 -07:00
Vance Ingalls e04f6dda37 feat(engine,cli): drawElement fast-capture config + CLI flag (#1916) 
## drawElement fast-capture — config + CLI flag (stack 1/6)

Foundation layer for the drawElement fast-capture feature: the config surface and CLI/Docker plumbing that the rest of the stack builds on.

### What this adds
- **`packages/engine/src/config.ts`** — new config fields for fast capture: `useDrawElement` / `enableDrawElementWorkerEncode` (macOS-GPU `drawElementImage` capture + worker-offloaded JPEG encode), resolved from env in `resolveConfig` (env `HF_DE_WORKER_ENCODE`). Wired alongside main's existing `staticFrameDedup` (unified downstream in 4/6).
- **`packages/cli/src/commands/render.ts`** — `--experimental-fast-capture` flag → sets `experimentalFastCapture`; `--debug` passthrough.
- **`packages/cli/src/utils/dockerRunArgs.ts`** — pass the fast-capture env through to the container.
- **`.github/workflows/fast-video-validation.yml`** — CI job validating fast-capture renders.
- `.oxlintrc.json` / `.fallowrc.jsonc` — ignore-pattern housekeeping for the new paths.

### Notes
- Config-only + entrypoint; no capture behavior yet (that's 2/6–4/6).
- Tests: `config.test.ts`, `dockerRunArgs.test.ts` added.

---
**Stack (drawElement fast-capture, rebased onto current `main`, supersedes #1295 + #1444):**
1. **#1916 config + CLI** ← you are here
2. #1917 drawElementImage capture service
3. #1918 3D projection + compositor-effect risk gate
4. #1919 frame-capture core (routing, worker-encode, static-dedup unification)
5. #1920 producer render stages + remote bg-image localizer
6. #1921 lint rule + player media sync

⚠️ Intermediate PRs (1–5) are split by package boundary for review and **do not each compile independently** (cross-file deps); the complete feature is green at the stack tip (#1921) — tsc-clean on engine + producer, 231 tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-07-06 15:57:30 -07:00
Miguel Ángel 34590649a0 perf(engine): extraction cache on by default with atomic publish and LRU gc (#1901) 
* perf(engine): extraction cache on by default with atomic publish and LRU gc

Warm re-renders now skip source-video frame extraction entirely
(video_extract 400ms -> 13ms on a 4-video composition; outputs are
pixel-identical, PSNR inf). What made default-on safe:

- Atomic entry publish: frames extract into a unique .partial-<pid>-<uuid>
  dir, the completion sentinel is written there, and the dir is renamed
  into the final key atomically. Concurrent renders sharing a cache can
  duplicate work but can never serve a torn entry (previously documented
  as single-writer only).
- Size-capped LRU gc: best-effort sweep after extraction evicts
  oldest-used entries past a 2 GiB default budget
  (HYPERFRAMES_EXTRACT_CACHE_MAX_MB) and clears crashed writers'
  partials. Entries younger than 60 min are never evicted so live
  renders keep their frames.
- Default cache dir: <tmpdir>/hyperframes-extract-cache-<uid>. Opt out
  with HYPERFRAMES_EXTRACT_CACHE_DIR=off (or none/false/0); a
  non-writable dir degrades to uncached with a single warning instead
  of failing the render.

* fix(engine): harden extraction cache publish and surface cache ops signals

Review hardening for the default-on extraction cache:

- Bypass the cache for HDR-converted intermediates: the key snapshot
  describes the original source, so publishing converted frames under
  it would poison later plain-SDR renders of the same trim. (The
  follow-up transform-keyed change re-enables caching for these.)
- publishCacheEntry TOCTOU: adopt a concurrent writer's completed
  entry both before removing an apparently-stale dir and after a
  failed retry rename, so a winner's publish is never destroyed or
  reported as a failure.
- Observability for the failure paths: cachePublishFailures,
  cacheGcEvictions, cacheGcBytesFreed, and cacheAgedPartialsCleared on
  ExtractionPhaseBreakdown; gcExtractionCache now returns sweep stats.

* fix(engine): sweep superseded cache generations in gc

After a SCHEMA_PREFIX bump, old-generation entries (hfcache-v2-*)
no longer matched the sweep's prefix filter and would orphan their
disk forever. The gc now matches any hfcache-v* generation; superseded
entries never receive sentinel touches, so the LRU evicts them first.
 2026-07-03 13:41:30 -07:00
James RussoandClaude Opus 4.8 c0c3abf0f1 fix(producer): harden capture against timeouts, transient tab deaths, and OOM (#1842) 
Four independent capture-infra hardening changes for the P2-5 failure bucket (~15K err / ~7K users):

- protocolTimeout auto-scales by device-scaled output area (applied before probe launch, since it's immutable post ppt.launch()).
- Single bounded transient retry (MAX_TRANSIENT_CAPTURE_RETRIES=1) on Target closed / Page crashed in the parallel disk-capture path; abort short-circuits before retry.
- Narrow OOM classification (Set maximum size exceeded etc., disjoint from transient) → actionable guidance naming output dims.
- StreamingEncoder.getExitError() threads FFmpeg's real exit reason into frame-0 encoder-death errors.

Render-reliability workstream P2-5. Success measured on PostHog dashboard 1783183.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-01 18:50:16 -07:00
Miguel Ángel 8408a44745 fix(engine): tune VP9 cpu-used across render paths (#1614) 
* fix(engine): tune VP9 cpu-used across render paths

* fix: address VP9 review feedback
 2026-06-20 16:36:59 -04:00
James Russo bacfb17538 feat(producer): auto low-memory safe render profile (#1225) 
## What

Adds an auto-detected **low-memory safe render profile**. On hosts at or below 8 GB total RAM, the render pipeline collapses to its cheapest shape instead of running multiple concurrent Chrome instances.

When `lowMemoryMode` is active and the user hasn't passed `--workers`, the orchestrator:
- **skips auto-worker calibration** — no throwaway second Chrome just to time 5 frames;
- **pins to a single worker** — so the probe Chrome is reused for capture, never N concurrent;
- **prefers screenshot capture over BeginFrame** — avoids the BeginFrame protocol-timeout → relaunch churn on slow hardware;
- logs a one-line explanation of what it did and how to override.

Builds on #1221 (merged), which fixed the calibration timeout cap, the `<= 8192` boundary, and added the CLI timeout flags.

## Why

Reported in #1218 / #1219: renders on 8 GB laptops sit at low progress for minutes or stall. Root cause (per the triage thread) is architectural — the default pipeline launches up to 4 Chrome instances sequentially/overlapping (probe, calibration, capture, screenshot-fallback), each ~256 MB+, on machines with ~3 GB free. The concurrent browsers drive memory pressure that makes every CDP call slow and spikes V8 GC pauses.

#1221 made the timeouts and memory flags *apply correctly*; this PR removes the expensive shape entirely on the machines that can't afford it, rather than tuning it. "Smarter by default."

## How

- **`packages/engine/src/services/systemMemory.ts`** (new): one shared `isLowMemorySystem()` / `getSystemTotalMb()`, de-duplicating the `totalmem()` reads previously copied in `config.ts` and `browserManager.ts`. Threshold is inclusive (`<= 8192 MB`) — real "8 GB" hardware reports ~7600–8192 MB after firmware/iGPU reservations, so a strict `<` would skip the optimisation on the very hardware that needs it.
- **`config.ts`**: new `lowMemoryMode` field on `EngineConfig`, resolved tri-state — explicit override → `PRODUCER_LOW_MEMORY_MODE` (on/off) → auto-detect from total RAM.
- **`renderOrchestrator.ts`**: gate calibration off, pin workers to 1, force screenshot capture, and emit a safe-mode log line when `lowMemoryMode` is set and `--workers` is absent.
- **`render.ts`**: `--low-memory-mode` / `--no-low-memory-mode` override (sets the env var the producer's `resolveConfig` reads) + docs table entry.

Fully overridable: an explicit `--workers N` restores calibration-free parallelism; `--no-low-memory-mode` / `PRODUCER_LOW_MEMORY_MODE=false` restores the full default shape.

### Deliberately deferred (separate PRs)
- **Reuse the probe session for calibration**: only executes on the tier *above* 8 GB (safe-mode skips calibration on the target boxes). A correct BeginFrame-mode reuse would lose calibration's fast-fail-to-screenshot timeout — real risk on a path the reported scenario never hits. Better scoped on its own.
- **Retuning `calculateOptimalWorkers`'s `totalmem*0.5/256` memory model**: hot path for *all* renders incl. servers/Lambda, outside this PR's local-laptop scope.

## Test plan

- [x] Unit tests added/updated — `systemMemory.test.ts` (8192 boundary cases), `config.test.ts` (tri-state env resolution + explicit-override precedence). Engine suite passes (25 relevant tests).
- [x] `tsc` clean across engine/producer/cli; `oxlint` + `oxfmt` clean; removed an unused export so the `fallow --fail-on-issues` dead-code gate stays green.
- [x] Documentation updated — `docs/packages/cli.mdx` render-flags table.
- [ ] Manual testing on a real ≤ 8 GB host — not yet run; behaviour is unit-covered and the safe path (1 worker + screenshot) is already a supported render shape.

Note: one pre-existing producer test (`rejects a maliciously crafted key…`) fails identically on `main` — environment-specific path test, unrelated to this change.
 2026-06-05 16:24:03 -07:00
Vance IngallsClaude Opus 4.6Vai
 		f84cc492de perf(engine): faster shader transitions via page-side WebGL compositing (#832) 
* fix(cli): prefer puppeteer cache + numeric version sort (staff review)

Two correctness fixes from PR #821 self-review:

1. Cache priority order. Previous order was hyperframes-managed cache →
   puppeteer cache. HF cache is pinned to CHROME_VERSION (131-era) which
   lags 17+ releases behind upstream; if a user separately installed a
   newer chrome-headless-shell via @puppeteer/browsers install, the CLI
   would silently hand engine the older HF-cache binary while engine's
   own resolveHeadlessShellPath would have picked the newer one. Flip
   the priority so puppeteer cache wins, matching engine semantics.

2. Numeric (not lexicographic) version sort. `readdirSync.sort().reverse()`
   over names like `linux-148.0.7778.97` and `linux-99.0.6533.123` would
   return `linux-99...` first because character '9' outranks '1'. Parse
   each name into integer segments and compare them numerically.

Tests: add both-caches-populated and linux-148-beats-linux-99 cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(engine): page-side compositing for shader transitions (opt-in spike)

Add an opt-in `--page-side-compositing` flag (CLI) backed by a new engine
config field `enablePageSideCompositing` and env var `HF_PAGE_SIDE_COMPOSITING`.
When set, SDR shader-transition compositions skip the Node-side layered blend
(the hf#677 chain) and instead run the shader inside Chrome via a page-side
WebGL canvas; the engine then captures ONE opaque RGB frame per output frame
via the existing streaming capture path.

This is the strongest non-beginFrame perf lever for Mac users, who cannot
take the beginFrame `~5×` path (Chromium structural limit, crbug.com/40656275).
Stacks on top of the hf#677 1.95× baseline.

Default OFF — existing fixture pins (byte-exact MP4 output) are preserved.
Opt-in path is intentionally PSNR-pinned, not byte-equal (WebGL is f32; Node
is f64). HDR content forces the existing layered path regardless.

Implementation:
- engine: new `EngineConfig.enablePageSideCompositing` (default false).
- producer/fileServer: new `HF_PAGE_SIDE_COMPOSITING_STUB` early-page script
  injected into the served HTML head when the flag is on.
- producer/renderOrchestrator: when the flag + no HDR + no png-sequence,
  route SDR transitions through the streaming path instead of the layered
  HDR stage.
- shader-transitions: new `engineModePageComposite.ts` installs a fullscreen
  WebGL compositor overlay and wraps `window.__hf.seek` so each seek inside
  a transition window captures both scenes via the Chromium
  `drawElementImage` API to GL textures, runs the fragment shader, and
  displays the composited result on the overlay canvas. The engine takes
  one screenshot per frame and sees the composited overlay.
- cli: new `--page-side-compositing` flag sets `HF_PAGE_SIDE_COMPOSITING=true`
  before producer load.
- scripts/page-side-compositing-smoke: bundled-CLI smoke that renders a
  representative fixture with and without the flag, validates the canary
  strings are in the shipped bundles, and writes a wall-time pair.

Determinism trade documented in the engine config doc-comment. The smoke
script enforces the bundled-CLI validation discipline from prior perf work
(see internal feedback note `validate_bundled_cli_not_dev_path`).

Runtime requirement: Chromium's `CanvasDrawElement` feature (already
enabled by the engine's `--enable-features=CanvasDrawElement` launch flag).
When the runtime feature is unavailable, the page-side installer logs a
warning and falls back to opacity-flip mode — the engine still takes the
streaming path; the transition window degrades to a hard scene swap. Vance
will validate on Mac Chrome where the feature is supported.

Co-Authored-By: Vai <vai@heygen.com>

* fix(shader-transitions): use html2canvas for page-side compositor capture

The original drawElementImage approach fails in engine render mode because
the virtual-time shim prevents Chromium from generating paint records for
cloned elements. drawElementImage requires a cached paint record from the
browser's compositor — clones created at capture time never receive one
because (a) shimmed rAFs deadlock inside the seek wrapper, (b) original
rAFs don't produce real paints under virtual-time control, and
(c) layoutsubtree canvases don't apply CSS stylesheet rules to children.

Switch scene capture to html2canvas (foreignObjectRendering: false), the
same JS-based renderer already used by the preview-mode fallback path in
capture.ts. html2canvas reads computed styles and renders via its own
canvas drawing pipeline with no dependency on the browser paint cycle.

Also fixes:
- Engine seek must return the result so Puppeteer awaits async seek
  promises (frameCapture.ts).
- GSAP opacity cache: compositor must restore scene opacity before seek,
  not after — GSAP caches inline values and skips re-writes.
- Support check gates on WebGL availability, not drawElementImage.

Perf: 15-scene shader-perf fixture (28s, 14 transitions, 30fps)
  Baseline (Node-side layered): 137s
  Page-side (html2canvas+WebGL): 33s → 4.1× speedup

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(shader-transitions): simplify review fixes for page-side compositor

- Use uploadTexture (zeroes canvas backing store after upload) to prevent
  ~2.2GB transient memory pressure across 280 html2canvas calls per render
- Add ignoreElements + stabilizeTransformedBoxShadows to html2canvas call,
  matching the preview-path capture.ts behavior
- Parallelize from/to scene captures with Promise.all
- Wrap post-capture render in try/finally so opacity is always restored
- Fix WebGL context leak in isPageSideCompositingSupported probe
- Remove dead ResolvedTransition.index field
- Export stabilizeTransformedBoxShadows from capture.ts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(producer): unify page-side compositing gating and Docker forwarding

Addresses three issues from staff review:

1. ignoreElements filter stripped all in-scene canvases (Chart.js, D3,
   p5.js) — narrowed to data-no-capture only since the compositor canvas
   is a body sibling never in the scene subtree.

2. Docker mode silently dropped --page-side-compositing — thread
   pageSideCompositing through DockerRenderOptions/buildDockerRunArgs
   with regression tests.

3. Fragmented gating across 4 independent sites could disagree:
   - Stub injection gated only on cfg flag (leaked into HDR/alpha)
   - Probe-created fileServer never got the stub
   - needsAlpha (WebM/MOV) not excluded from the gate
   - WebGL-unavailable fallback claimed layered path would run but
     orchestrator had already disabled it

   Fix: compute stub injection at the same site as the layered-bypass
   decision (after hasHdrContent is known), using addPreHeadScript on
   the already-running fileServer. Single predicate now gates both
   decisions, including !needsAlpha.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf(engine): two-phase drawElementImage capture for page-side compositing

Replace html2canvas with native drawElementImage for scene capture in
the page-side compositor. drawElementImage reads from the browser's own
paint cache, giving pixel-identical output to the preview path.

The blocker was that cloned elements inside layoutsubtree canvases have
no cached paint record under virtual time — the compositor only paints
when explicitly triggered. Fix: split the seek+composite into two phases
with an engine-forced paint between them.

Phase 1 (seek wrapper, page-side):
  - GSAP seek positions the timeline
  - Clone FROM/TO scenes into visible layoutsubtree staging canvases
  - Set window.__hf_page_composite_pending flag

Engine paint force (frameCapture.ts):
  - Detect pending flag after seek returns
  - Fire micro Page.captureScreenshot (1x1 clip) via CDP to force the
    browser compositor to paint all visible elements including staging
    canvas children

Phase 2 (page.evaluate, page-side):
  - drawElementImage reads the now-valid paint records
  - Upload textures to WebGL, run shader, show GL overlay

Key insight: staging canvases must be visible (not opacity:0) for the
browser to paint their children. They sit at z-index:-9998, behind
the main DOM and covered by the GL overlay during transitions.

Perf: 15-scene fixture (28s, 14 transitions, 30fps):
  Baseline (Node-side layered): 137s
  html2canvas + WebGL:           33s (3.7×)
  drawElementImage + WebGL:      21s (6.6×)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf(engine): optimize two-phase compositor hot path

- uploadTextureSource instead of uploadTexture: eliminates ~2.3GB of
  canvas buffer alloc/dealloc churn (persistent staging canvases don't
  need the one-shot zeroing behavior)
- Fold hasPending check into seek page.evaluate: eliminates one CDP
  round-trip per frame (~700 unnecessary IPC calls on non-transition
  frames)
- Fix renderShader error handling: on failure, leave source scenes
  visible as fallback instead of hiding both scenes + GL overlay
  (which produced black frames)
- Move mutable state declarations above resolveComposite to prevent
  TDZ risk on refactor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(engine): staff review — staging cleanup, pending flag, beginFrame guard

- Clear staging canvas children when leaving transition window (prevents
  visible clone bleed-through on transparent compositions)
- Clear __hf_page_composite_pending on all resolveComposite exit paths
- Guard micro-screenshot paint force against beginFrame mode (CDP
  Page.captureScreenshot conflicts with beginFrame compositor control)
- Update CLI flag description: document video/canvas limitation, remove
  stale PSNR claim

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(engine): default-on page-side compositing for SDR shader transitions

Page-side compositing is now enabled by default for SDR shader-transition
renders without video content. The 6.6× speedup applies automatically —
no flag needed.

Auto-disables when:
- HDR content detected
- Alpha output (WebM/MOV/PNG-sequence)
- Composition contains <video> elements (cloneNode loses playback state)
- beginFrame capture mode (Linux headless)

Use --no-page-side-compositing to force the Node-side layered path.

Changes:
- Engine config: enablePageSideCompositing defaults to true
- CLI: flag default flipped to true; --no-page-side-compositing disables
- Orchestrator: added composition.videos.length === 0 gate
- Docker: forwards --no-page-side-compositing when explicitly disabled
- Config tests updated for new default

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(engine): support video elements on page-side compositing fast path

Three-phase capture protocol lets shader transitions render video scenes
without falling back to the slow Node-side layered pipeline:

1. Seek → compositor records transition metadata, sets pending flag
2. onBeforeCapture → video frame injector updates <img> replacements
3. prepare → cloneNode picks up current video frames, img.decode() awaits
4. micro-screenshot → forces browser to paint cloned elements
5. resolve → drawElementImage reads paint records, shader composites

Key changes:
- Remove `composition.videos.length === 0` gate from orchestrator
- Split compositor resolve into prepare (clone) + resolve (shader)
- Move onBeforeCapture before compositor prepare in frameCapture.ts
- Await img.decode() on cloned data-URI images to prevent stale frames
- Stop manipulating scene opacity in compositor (GL canvas overlay suffices)
- Add gsap.set declaration for shader-transitions ambient types
- Add video_missing_timing_attrs lint rule for <video> without id/data-start/data-end

Performance: compositions with video now render at 7.5s (6 workers) instead
of 2m38s on the layered path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(core): auto-inject data-start on video/audio so frame extraction works without explicit attrs

The timing compiler now injects data-start="0" on <video> and <audio>
elements that lack it. This makes discoverMediaFromBrowser() find the
element (it queries video[data-start]), so the frame extraction pipeline
activates automatically. Videos "just work" without requiring authors to
add data-start, data-end, or id attributes.

Also removes the video_missing_timing_attrs lint rule — the compiler
handles the missing attributes automatically, so the lint rule would
only false-positive.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(core): add data-hf-auto-start sentinel on auto-injected video timing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(producer): add discoverVideoVisibilityFromTimeline for runtime video discovery

Seeks the GSAP timeline in Puppeteer to discover when each video's parent
scene is visible (opacity > 0). Uses coarse sampling at 100ms steps followed
by binary search refinement to frame-level precision (1/60s). Only processes
videos with the data-hf-auto-start sentinel so author-specified timing is
never overridden.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(producer): integrate runtime video visibility discovery into probe stage

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(producer): trigger browser probe for auto-start videos, remove debug logging

The probe stage was skipping browser launch when composition duration was
already known, which meant discoverVideoVisibilityFromTimeline never ran.
Now needsBrowser also checks for data-hf-auto-start sentinel in compiled HTML.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): use mkdtempSync for smoke test work directory

Replaces hardcoded /tmp/hf-page-side-smoke with a unique temp directory
via mkdtempSync to resolve CodeQL "insecure temporary file" alert.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: format smoke test script

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Vai <vai@heygen.com>
 2026-05-15 16:23:12 -07:00
James 67bb56c703 feat(engine): browserGpuMode "auto" — probe WebGL once, fall back to software 
When the host doesn't have a usable GPU (CI containers, eval rigs without
GPU passthrough, dev VMs), Chrome's hardware-mode WebGL flags
(`--use-gl=egl/metal/d3d11`) silently leave WebGL unavailable —
`getContext("webgl")` returns null, three.js' WebGLRenderer dies, the
canvas stays black. Surfaced today by Abhay's c2v-eval failing on a
docker render of an hf bundle that uses three.js + a custom fragment
shader.

The fix that's been there: `--use-gl=angle --use-angle=swiftshader` (CPU
software WebGL, ~5-50× slower but pixel-identical). The engine already
exposed `browserGpuMode: "software"` for this. The gap was discovery —
users had to know to pass `--no-browser-gpu` on no-GPU hosts.

This change adds `browserGpuMode: "auto"` (now the CLI default for local
renders): on first launch in the process, probe Chrome with hardware
args, check `canvas.getContext("webgl") !== null`, cache the result.
~1-2 s on first render, free on every subsequent render in the same
worker. Hardware GPUs keep their fast path; no-GPU hosts get SwiftShader
without ceremony.

Behaviour matrix:
- No flag, no env, local       → "auto" (NEW default)
- `--browser-gpu`              → "hardware" (force; errors if no GPU)
- `--no-browser-gpu`           → "software" (force SwiftShader)
- `PRODUCER_BROWSER_GPU_MODE`  → "hardware" / "software" / "auto" / unset
- Docker mode                  → forced "software" (unchanged)

Engine-config default stays "software" (conservative for embedders); the
"auto" default lives in the CLI's `resolveBrowserGpuForCli` so producer
embedders aren't surprised by a probe-on-launch.

Also adds `--enable-unsafe-swiftshader` to the software flag set —
Chrome 120+ deprecated implicit SwiftShader fallback and emits a
deprecation warning unless the flag is set explicitly. Despite the
"unsafe" name this is exactly the pre-deprecation behaviour; the rename
is about Chrome's threat model on the open web, not about the rendering
itself.

Verification:
- Engine 535/535 + CLI 256/256 (incl. new probe tests + tri-state CLI test)
- Empirical: probe on this no-GPU devbox returns "software" in 240 ms,
  cached 0 ms on subsequent calls
- Format / lint / typecheck clean across all packages

Refs the Abhay/Slack thread on c2v-eval rendering without a GPU node.
 2026-05-06 17:33:55 +00:00
Miguel Ángel dde26cf62d feat: default streaming encode for sequential renders (#579) 2026-05-01 04:22:26 +02:00
Miguel Ángel 395fb9c084 feat: add browser GPU render mode (#571) 
## Problem

HyperFrames already had `--gpu`, but that flag only controlled FFmpeg hardware encoding. The browser capture path still forced Chrome/WebGL through SwiftShader software GL via `--use-angle=swiftshader`, so WebGL-heavy local renders could leave the biggest bottleneck on the CPU path.

That made the existing flag naming easy to misread: `--gpu` sounded like it accelerated the whole render, but it did not change the browser frame-capture backend.

## What this fixes

- Enables host browser GPU acceleration automatically for local CLI renders.
- Adds `--no-browser-gpu` as the local opt-out for software Chrome/WebGL capture.
- Keeps `--browser-gpu` as an explicit local browser-GPU request.
- Adds `browserGpuMode: "software" | "hardware"` to engine config, with `PRODUCER_BROWSER_GPU_MODE` env support for lower-level producer users.
- Keeps Docker browser capture on the deterministic software path.
- Maps hardware browser GPU mode to platform-native Chrome backends:
  - macOS: Metal-backed ANGLE
  - Windows: D3D11-backed ANGLE
  - Linux: EGL
- Blocks explicit `--browser-gpu --docker` with a clear error because Docker browser GPU passthrough is not cross-platform.
- Clarifies docs so `--gpu` means FFmpeg encoder GPU and browser GPU means Chrome/WebGL capture GPU.
- Keeps encoder backend selection auto-detected from FFmpeg capabilities:
  - NVIDIA: NVENC
  - macOS: VideoToolbox
  - Linux: VAAPI
  - Intel: QSV

## Why two flags

There are two separate GPU surfaces in the render pipeline:

1. Browser GPU controls Chrome frame capture.
   - Affects WebGL, canvas, CSS rendering, compositing, and screenshot capture inside the browser.
   - This is enabled automatically for local CLI renders.
   - Use `--no-browser-gpu` when you want the software browser baseline.

2. `--gpu` controls FFmpeg video encoding.
   - Affects the final encode step after frames have already been captured.
   - The concrete encoder is auto-detected from the host FFmpeg build and hardware.
   - It can be faster for some machines/codecs, but it is not equivalent to browser rendering acceleration.

The controls stay independent because users may want:

- `hyperframes render` for the fast local default with browser GPU capture.
- `hyperframes render --no-browser-gpu` for the software-browser local baseline.
- `hyperframes render --gpu` for browser GPU capture plus hardware FFmpeg encoding.
- `hyperframes render --no-browser-gpu --gpu` for software browser capture plus hardware FFmpeg encoding.
- `hyperframes render --docker` for deterministic browser capture.

## Why `--gpu` does not imply browser GPU

Keeping `--gpu` scoped to FFmpeg encoding avoids a semantic break and keeps the risk profile explicit:

- `--gpu` already means encoder acceleration. Expanding it to also change Chrome capture would silently alter behavior for users who only wanted hardware encoding.
- Browser GPU and encoder GPU have different portability. Encoder GPU can work in Docker when the host exposes the right devices; browser GPU passthrough is not cross-platform, so this PR intentionally blocks explicit `--browser-gpu --docker`.
- The Apple presentation benchmark shows why the controls should stay separate: browser GPU capture was the useful improvement, while macOS VideoToolbox via `--gpu` was slower and produced larger output for this `standard` H.264 run.

If HyperFrames later wants a single umbrella acceleration control, it should be explicit, for example `--acceleration browser|encoder|all` or `--gpu=browser|encoder|all`, rather than changing the meaning of the existing boolean `--gpu`.

## Root cause

`buildChromeArgs()` always injected `--use-gl=angle --use-angle=swiftshader`. `disableGpu` only appended `--disable-gpu`; it did not provide a hardware-GPU mode. That made the public `--gpu` flag look broader than it was, because render capture stayed software-backed even when encoder GPU was requested.

## Verification

### Local checks

- `bun install`
- `bun run build:hyperframes-runtime`
- `bun run --filter @hyperframes/engine test src/config.test.ts src/services/browserManager.test.ts`
- `bun run --filter @hyperframes/cli test src/utils/dockerRunArgs.test.ts src/commands/render.test.ts`
- `bun run --filter @hyperframes/cli typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bun run --filter @hyperframes/producer typecheck`
- `cd packages/producer && bunx vitest run src/services/renderOrchestrator.test.ts`
- `bunx oxlint packages/cli/src/commands/render.ts packages/cli/src/commands/render.test.ts packages/cli/src/utils/dockerRunArgs.ts packages/cli/src/utils/dockerRunArgs.test.ts packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/browserManager.ts packages/engine/src/services/browserManager.test.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt --check ...` on changed source/docs files
- `git diff --check`
- `bun packages/cli/src/cli.ts render --help | rg -n "browser-gpu|no-browser-gpu|GPU"`
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --output /tmp/hf-auto-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
  - Render plan prints `GPU: browser GPU (auto)`.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --no-browser-gpu --output /tmp/hf-software-browser-gpu-smoke.mp4 --workers 1 --quality draft --fps 24 --strict`
  - Render plan does not print browser GPU.
- `bun packages/cli/src/cli.ts render packages/producer/tests/css-spinner-render-compat/src --docker --browser-gpu --output /tmp/should-not-render.mp4`
  - Exits 1 with `Browser GPU is local-only`.
- `buildDockerRunArgs()` regression coverage asserts Docker container args include `--no-browser-gpu`, preventing nested container renders from re-enabling browser GPU through the local CLI default.
- `resolveBrowserGpuForCli()` regression coverage asserts `PRODUCER_BROWSER_GPU_MODE=software` opts out when no CLI browser-GPU flag is supplied, while explicit `--browser-gpu` / `--no-browser-gpu` still win.
- `ffmpeg -v error -i /tmp/hf-auto-browser-gpu-smoke.mp4 -f null -`
- `ffmpeg -v error -i /tmp/hf-software-browser-gpu-smoke.mp4 -f null -`
- `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-browser-gpu-smoke.mp4` -> H.264, 1920x1080, 24fps, 5.0s

### Apple presentation benchmark

Rendered `/Users/miguel07code/Downloads/apple-presentation.zip` as supplied after extracting to `/tmp/hf-apple-profile/apple-presentation`.

Fixed settings:

- 1920x1080
- 30fps
- `standard` quality
- 4240 frames
- 141.32s duration
- 8-worker cap; render auto-calibration used 6 capture workers
- macOS host detected FFmpeg GPU encoder: `videotoolbox`

| Mode | Equivalent flags after this PR | Wall time | vs software-browser baseline | Speed | Capture | Encode | Output |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Software browser + CPU encode | `--no-browser-gpu` | 120.77s | baseline | 1.17x | 97.87s | 10.04s | 8.38MB |
| Browser GPU + CPU encode | default local render | 70.10s | 42.0% faster | 2.02x | 50.72s | 9.91s | 8.39MB |
| Software browser + encoder GPU | `--no-browser-gpu --gpu` | 133.16s | 10.3% slower | 1.06x | 103.58s | 18.31s | 25.43MB |
| Browser GPU + encoder GPU | `--gpu` | 74.12s | 38.6% faster | 1.91x | 46.69s | 17.93s | 25.45MB |

Result: browser GPU capture is the meaningful improvement for this WebGL/browser-capture-heavy presentation. VideoToolbox encoding was slower and produced larger files for this current `standard` H.264 path, so `--gpu` should stay separate and opt-in.

Why `--gpu` plus browser GPU was slower than browser GPU alone: the combined run captured about 4.0s faster than browser GPU alone, but VideoToolbox encoding was about 8.0s slower than CPU x264 encoding, so the encode loss outweighed the capture gain.

### VideoToolbox flag check

I also isolated the encode stage against the already-captured Apple frames to check whether macOS GPU encoding only needed special flags.

`ffmpeg -h encoder=h264_videotoolbox` does not expose a CRF/CQ-style quality option like x264. It exposes bitrate-oriented and VideoToolbox-specific options such as `-b:v`, `-realtime`, `-profile`, `-coder`, `-prio_speed`, `-power_efficient`, and `-allow_sw`. That means our current `-q:v` mapping is not equivalent to x264 CRF and can produce very different bitrate/size behavior.

Measured full-frame encode variants on this host:

| VideoToolbox variant | Encode wall time | Output size | Bitrate |
| --- | ---: | ---: | ---: |
| Current `-q:v 64 -allow_sw 1` | 18.76s | 25.31MB | 1.43 Mbps |
| Current without `-allow_sw 1` | 18.21s | 25.31MB | 1.43 Mbps |
| `-b:v 500k -maxrate 750k -bufsize 1000k -profile high -coder cabac -realtime 1 -prio_speed 1 -power_efficient 0` | 20.58s | 7.42MB | 0.42 Mbps |
| Same with `-b:v 1500k` | 20.84s | 16.70MB | 0.95 Mbps |
| `-b:v 500k -profile baseline -coder cavlc -realtime 1 -prio_speed 1 -power_efficient 0` | 18.11s | 8.94MB | 0.51 Mbps |

Conclusion: VideoToolbox can be made size/bitrate-predictable with explicit `--video-bitrate`, but the tested speed-oriented flags did not make it faster than CPU x264 wall time for this render. That reinforces keeping `--gpu` encoder acceleration explicit and separate from browser GPU capture.

Artifacts from the local benchmark:

- `/tmp/hf-apple-profile/results/cpu.mp4`
- `/tmp/hf-apple-profile/results/browser-gpu.mp4`
- `/tmp/hf-apple-profile/results/encoder-gpu.mp4`
- `/tmp/hf-apple-profile/results/full-gpu.mp4`
- `/tmp/hf-apple-profile/results/summary.json`

All four benchmark MP4s completed `ffprobe` and full `ffmpeg -f null` decode checks.

### Pixel comparison

Compared decoded MP4 output between software-browser and browser-GPU renders:

- Apple presentation:
  - 4240 frames compared
  - 636 exact matching decoded frame hashes
  - 3604 different decoded frame hashes
  - Average PSNR: 57.79 dB
- `css-spinner-render-compat` clean fixture:
  - 120 frames compared
  - 0 exact matching decoded frame hashes
  - Average PSNR: 61.57 dB

Interpretation: browser GPU output is not strict hash/pixel-identical to the software-browser path after lossy H.264 encode, but the measured deltas are visually tiny. Above 50 dB PSNR is typically visually indistinguishable for normal video review. Use `--no-browser-gpu` or Docker when strict cross-run/cross-machine reproducibility matters more than local speed.

### Browser verification

- Started HyperFrames Studio preview for `packages/producer/tests/css-spinner-render-compat/src`.
- Used `agent-browser` to open `http://localhost:5191#project/src` and verify the composition loaded in Studio.
- Screenshots:
  - `/tmp/hf-gpu-browser-proof/preview-loaded.png`
  - `/tmp/hf-gpu-browser-proof/preview-playing.png`
  - `/tmp/hf-gpu-browser-proof/preview-frame-60.png`
- Agent-browser recordings:
  - `/tmp/hf-gpu-browser-proof/preview-playback.webm`
  - `/tmp/hf-gpu-browser-proof/preview-seek.webm`

## Notes

- Browser GPU is enabled automatically for local CLI renders and disabled in Docker.
- `--no-browser-gpu` is the opt-out for software Chrome/WebGL capture.
- `--gpu` remains encoder-only and opt-in.
- The Apple presentation zip has existing lint errors around unmanaged nested videos and imperative media `play()` calls. The benchmark still compares the same supplied source across modes, but it should not be treated as a clean deterministic-composition fixture.
 2026-04-30 06:46:14 +02:00
Miguel Ángel b98093aa1c fix: remove hidden audio gain in renders (#362) 
## Summary

This fixes a render-time audio correctness bug where Hyperframes applied a hidden post-mix gain to every rendered output, boosting audio by about +2.6 dB and causing clipping on normally leveled sources.

It also fixes a related mute bug where `data-volume="0"` was treated as falsy and silently converted back to full volume during audio track preparation.

Additionally, this PR fixes the Studio workspace typecheck path for `@hyperframes/player`, so local pre-commit/typecheck flows no longer depend on the Player package having been built first.

## Root Cause

The issue report measured a near-constant gain increase and suspected a hidden normalization step. After tracing the engine audio path, the root cause turned out to be explicit code, not FFmpeg behavior:

- `packages/engine/src/config.ts` defaulted `audioGain` to `1.35`
- `packages/engine/src/services/audioMixer.ts` always appended a post-mix FFmpeg filter:
  - `[mixed]volume=${masterOutputGain}[out]`
- with the default config, that meant every render got multiplied by `1.35`

That exactly matches the issue reporter's measured scalar boost.

While investigating the workaround, I also found a second correctness bug:

- `processCompositionAudio()` used `element.volume || 1.0`
- that coerced `0` to `1.0`
- so `data-volume="0"` did not actually mute the track in rendered output

Separately, the repo-level Studio typecheck could fail before any build step because:

- `packages/studio/src/player/components/Player.tsx` imports `@hyperframes/player`
- `packages/player/package.json` points TypeScript at built `dist/*` outputs
- in a fresh workspace, those built outputs may not exist yet
- Studio therefore failed type resolution for `@hyperframes/player` during pre-commit/typecheck

## What Changed

1. Set the engine default `audioGain` back to unity (`1`)
2. Preserve explicit zero volumes by changing `element.volume || 1.0` to `element.volume ?? 1.0`
3. Added regression coverage for both behaviors
4. Updated the producer-side config fixture to reflect the corrected default
5. Added a Studio tsconfig path mapping for `@hyperframes/player` to the local workspace source and widened `rootDir` so workspace typecheck succeeds without requiring a prior Player build

## Why These Changes Are Needed

This is not a UX preference issue; it is a correctness and API contract issue.

- The docs describe `data-volume` as a direct 0-1 control.
- Rendered output should preserve source levels unless the author explicitly changes them.
- Hidden global gain makes output non-deterministic from the author's perspective.
- `data-volume="0"` must mean silence, not full-volume playback.
- Local workspace typecheck should not require unrelated package build artifacts to exist first.

Leaving the current behavior in place means:

- voice recordings near normal peak levels can clip during render
- authors need undocumented manual compensation (`0.75`-ish scaling) to get unity output
- mute semantics in docs and code diverge
- local pre-commit/typecheck can fail for reasons unrelated to the actual diff being committed

## Testing

### Focused regression tests

Ran:

- `packages/engine/node_modules/.bin/vitest run packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.test.ts`

Result:

- `10 passed`

These tests specifically verify:

- default resolved `audioGain` is `1`
- a track with `volume: 0` stays `volume=0` in the FFmpeg filter graph
- the post-mix output filter stays at unity gain (`[mixed]volume=1[out]`)

### Broader package verification

Ran:

- `bun run --filter @hyperframes/engine test`
- `bun run --filter @hyperframes/engine build`
- `packages/engine/node_modules/.bin/vitest run packages/producer/src/services/renderOrchestrator.test.ts`
- `bun run --filter @hyperframes/producer typecheck`
- `bun run --filter @hyperframes/studio typecheck`
- `bunx oxlint packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.ts packages/engine/src/services/audioMixer.test.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.ts packages/engine/src/services/audioMixer.test.ts packages/producer/src/services/renderOrchestrator.test.ts packages/studio/tsconfig.json`
- `bunx lefthook run pre-commit`

Results:

- full engine test suite passed (`309 passed`)
- engine build passed
- touched producer test file passed (`7 passed`)
- producer typecheck passed
- studio typecheck passed
- oxlint passed with `0 warnings, 0 errors`
- formatting passed
- pre-commit hook no longer hits the prior `@hyperframes/player` module-resolution blocker

## Known Verification Limitation

There is no meaningful browser UI flow for this bug: the defect is in the engine/CLI audio render pipeline rather than an interactive browser surface. Because of that, verification was done at the renderer and test level rather than through an agent-browser flow.

## User Impact

After this change:

- rendered audio matches source level by default
- authors no longer need to compensate for a hidden +2.6 dB boost
- `data-volume="0"` correctly mutes rendered audio
- the documented volume contract matches engine behavior again
- local workspace typecheck no longer depends on prebuilt `@hyperframes/player` artifacts

Closes #361.
 2026-04-21 20:33:25 +02:00
Vance IngallsandClaude Opus 4.6 9f8e5ba5a1 initial code (#2) 
* feat: initial code port from hyperframes-internal

Port all OSS-ready packages from the internal monorepo:
- @hyperframes/core — shared types, HTML generation, GSAP utilities, runtime
- @hyperframes/cli — CLI for creating, previewing, and rendering compositions
- @hyperframes/engine — framework-agnostic rendering engine (BeginFrame + FFmpeg)
- @hyperframes/producer — video rendering pipeline (Puppeteer + FFmpeg)
- @hyperframes/ui-player — browser-based video player component
- @hyperframes/studio — composition editor (React frontend + Hono backend)

Includes regression test suite with Docker-based test harness.

All HeyGen-internal references, deployment infrastructure, and
proprietary assets have been removed. Package names migrated
from @app/* to @hyperframes/*.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: scrub internal codenames and stale references from OSS port

- Replace static.heygen.ai runtime URLs in test fixtures
- Remove internal CDN publish script (publish-hyperframe-runtime.ts)
- Replace sandbox-studio, sandbox-interceptor, __magicEditRuntime
  with neutral names (studio, hyperframe-runtime, __hyperframeRuntime)
- Fix stale Vault API / localhost references in docs
- Remove broken deprecated_studio link

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove remaining internal codenames and stale references

- Delete stale producer README.md and PIPELINE.md (referenced nonexistent files)
- Replace "Cerberus" codename with "HyperFrames" in test design reviews
- Replace magic-edit postMessage identifiers with hf-preview/hf-parent
- Rename debug-magic-edit-timeline.ts to debug-timeline.ts
- Replace "Motion Cut" with "HyperFrames" in Timeline comments
- Fix studio/CLI references to nonexistent archive package
  (use local data/projects/ dir, stub render proxy)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-21 22:43:56 -07:00