Move the SDR / DOM-only-HDR disk-capture body out of `executeRenderJob`
into `services/render/stages/captureStage.ts`. Covers both branches of
the disk path: parallel capture via `executeDiskCaptureWithAdaptiveRetry`
(`workerCount > 1`) and sequential per-process capture (`workerCount === 1`,
reusing `probeSession` when available).
The HDR layered branch (`useLayeredComposite === true`) and the streaming
encode fusion path (`useStreamingEncode === true` with successful encoder
spawn) stay inline in the sequencer — they will be extracted by the next
two PRs in the stack.
Hard constraints preserved verbatim:
- `probeSession` is closed (and the sequencer's `let probeSession`
nulled via the returned result) at the same points.
- `captureAttempts` is mutated in place — the parallel retry loop still
pushes each attempt onto the array the sequencer owns.
- `workerCount` reassignment from adaptive retry survives via the
returned result.
- `lastBrowserConsole` is set to the buffer of whichever session was
active last (probe close path or sequential capture finally).
- `job.framesRendered` is updated at the same per-frame / per-progress
points; `Capturing frame N/M [(K workers)]` `updateJobStatus` payloads
fire at the same 30-frame and completion checkpoints.
- `perfStages.captureMs` is still computed by the sequencer from the
outer `stage4Start` so its window covers both the in-sequencer setup
(fileServer init, calibration, worker resolution, preset selection)
AND the capture call.
Two small new exports on `renderOrchestrator.ts`:
- `executeDiskCaptureWithAdaptiveRetry` — was a private helper; the
stage calls it directly.
- `updateJobStatus` — was a private helper; the stage uses it for the
per-frame progress callbacks so the `completedAt` branch matches.
These re-introduce a small runtime cycle between the stage and the
orchestrator (orchestrator imports `runCaptureStage`; stage imports
helpers back). The cycle is safe (both modules finish loading before
any stage function is invoked at runtime) and will be flattened in a
follow-up PR that consolidates capture helpers into a shared module.
Removes the now-orphaned `captureFrame` import from the orchestrator.
Verified inside `Dockerfile.test`:
- `font-variant-numeric`: audio correlation 1.000
- `many-cuts`: 0 failed frames, audio correlation 0.994
- `variables-prod`: PSNR ~69 dB, audio correlation 0.975
- `sub-composition-video`: PSNR ~43-52 dB, audio correlation 0.947
(exercises video extraction + capture end-to-end)
- `gsap-letters-render-compat`: PSNR ~53-55 dB, audio correlation 1.000
(exercises the parallel capture path; 5/5 PASS overall)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the audio mixing sub-stage of `executeRenderJob` into
`services/render/stages/audioStage.ts`. Trivial wrapper around
`processCompositionAudio` with the same skip-when-empty path.
No behavior change:
- `audioOutputPath` is still `join(workDir, "audio.aac")` regardless of
whether the composition has audio.
- `hasAudio` still reflects `audioResult.success` (false when no audio
elements or when the mixer returns success: false).
- `perfStages.audioProcessMs` is set at the same end-of-stage point
whether or not the mixer ran.
- The "Processing audio tracks" progress callback fires at 20% at the
same code point.
Removes the now-unused `processCompositionAudio` import from the
orchestrator (oxlint flagged it).
Verified inside `Dockerfile.test` against `font-variant-numeric`,
`many-cuts`, `variables-prod` — 3/3 pass with audio correlations
1.000 / 0.994 / 0.975 (identical to prior PRs in the stack).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the video frame extraction sub-stage out of `executeRenderJob` into
`services/render/stages/extractVideosStage.ts`. The stage covers HDR
color-space pre-detection for videos and images, the
`extractAllVideoFrames` call, frame-lookup-table construction, video
readiness skip-id collection, video metadata hints, and the auto-detect
of audio tracks from video files.
Hard constraints preserved verbatim:
- `composition.audios` is still mutated in place to add audio entries
auto-discovered from video files via ffprobe.
- `perfStages.videoExtractMs` is set at the same end-of-stage point.
- `materializeExtractedFramesForCompiledDir` is still called once when
`extractionResult.extracted` is non-empty.
- `force-sdr` mode still skips ALL ffprobe overhead.
New for distributed mode (`materializeSymlinks: boolean`, default false):
- Plumbs through to `materializeExtractedFramesForCompiledDir` via a new
option of the same name. When `true`, the helper invokes
`cpSync(recursive)` instead of `symlinkSync` so the staged frames are
real files inside `compiledDir`. Symlinks don't survive S3 / GCS
round-trips, so distributed `plan()` will pass `true` once it lands.
Default `false` preserves the in-process renderer's symlink behavior.
- New unit test covers the copy path and asserts `symlinkSync` is NOT
invoked; the existing symlink test was updated with the parallel
guard that `cpSync` is NOT invoked.
Removes the imports the orchestrator no longer needs after the
extraction: `extractAllVideoFrames`, `resolveProjectRelativeSrc`,
`createFrameLookupTable`, `FrameLookupTable`, `detectTransfer`,
`isHdrColorSpace`, `extractMediaMetadata`, `VideoColorSpace` (oxlint
flagged each).
Verified:
- `bunx oxlint` + `bunx oxfmt --check` clean
- `bun run --filter @hyperframes/producer typecheck` + `build` clean
- `bun test packages/producer/src/services/` — 176 pass, 1 pre-existing
unrelated failure
- `docker run hyperframes-producer:test` against `font-variant-numeric`,
`many-cuts`, `variables-prod`, `sub-composition-video` — 4/4 PASS with
correlations 1.000 / 0.994 / 0.947 / 0.975 (sub-composition-video is
the one that exercises video frame extraction end-to-end)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small follow-ups on the shared.ts extraction, addressing review
feedback on #720:
- Hoist `BROWSER_MEDIA_EPSILON` from `probeStage.ts` into `shared.ts`
so any future stage that reconciles browser media (chunked rendering
re-probe, for instance) doesn't have to redeclare it.
- Migrate `renderOrchestrator.test.ts` to import the five moved symbols
(`applyRenderModeHints`, `projectBrowserEndToCompositionTimeline`,
`resolveDeviceScaleFactor`, `writeCompiledArtifacts`,
`CompositionMetadata`) directly from `./render/shared.js`. This is
the clean end state — the back-compat re-export through
`renderOrchestrator.ts` was a stepping-stone.
- Drop the back-compat re-export block from `renderOrchestrator.ts`.
No remaining importers go through it (verified via grep across
`packages/`). The five symbols now have exactly one path:
`./render/shared.js`.
No behavior change. Renderer smoke-tested inside `Dockerfile.test`
against `font-variant-numeric`, `many-cuts`, and `variables-prod` —
audio correlations 1.000 / 0.994 / 0.975, matching every prior PR in
the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Breaks the runtime circular import between `renderOrchestrator.ts` and
the stage files under `services/render/stages/`. Before this change the
stages imported runtime helpers (`writeCompiledArtifacts`,
`applyRenderModeHints`, `resolveDeviceScaleFactor`,
`projectBrowserEndToCompositionTimeline`) and types (`CompositionMetadata`)
back from `renderOrchestrator.ts`, which itself imports the stage
functions. The cycle resolved at build time because both modules
finished initializing before any stage was invoked, but it was fragile
and would keep growing as more stages were extracted.
This PR:
- Adds `packages/producer/src/services/render/shared.ts` and moves the
four functions plus the `CompositionMetadata` interface into it.
- Has `renderOrchestrator.ts` re-export everything from `shared.ts`, so
external callers (the existing `renderOrchestrator.test.ts`, any code
importing `applyRenderModeHints` etc. from the orchestrator) keep
working with no churn on their side.
- Updates `compileStage.ts` and `probeStage.ts` to import the runtime
helpers from `../shared.js`. The only remaining import from
`renderOrchestrator.ts` in the stages is `import type { RenderJob }`,
which is erased at runtime and creates no cycle.
- Removes the imports the orchestrator no longer needs after losing
the four function definitions (`CANVAS_DIMENSIONS`, `VideoElement`,
`AudioElement`, `ImageElement`).
No behavior change. Renderer smoke-tested inside `Dockerfile.test`
against `font-variant-numeric`, `many-cuts`, and `variables-prod` —
all PSNR / audio-correlation baselines match PR 1.3 exactly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The probe stage previously assigned `job.duration` and `job.totalFrames`
inside its body AND the sequencer re-asserted them after the call to
restore TS narrowing. Two writers for the same field is a maintenance
hazard — a future refactor could drop one and create a silent skew.
Move ownership: the stage computes `duration` and `totalFrames` and
returns them; the sequencer is the sole writer onto the `RenderJob`.
This also aligns with the eventual chunk-worker model where a chunk
running in a separate process cannot mutate the orchestrator's `job`.
No observable behavior change. `job.duration` / `job.totalFrames` end
up with the same values; the zero-duration `throw` still happens
inside the stage (now using the local `duration` constant) before any
sequencer-side assignment. Verified by:
- `bun run --filter @hyperframes/producer typecheck` clean
- `bun test packages/producer/src/services/` 175 pass / 1 pre-existing
unrelated failure on `main`
Review feedback addressed: vanceingalls on #719.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment-only cleanup. Removes "PR 1.x", "Phase 1 PR", and "Phase 3 PR 3.1"
references from JSDoc blocks in `compileStage.ts`, `probeStage.ts`,
`planHash.ts`, and `freezePlan.ts`. Track / PR identifiers rot quickly and
belong in PR descriptions, not in source. Design-doc section citations
(DISTRIBUTED-RENDERING-PLAN.md §X.Y) are kept — those reference a stable
external artifact.
Also tightens the `probeStage.ts` `browserProbeMs` doc string to say
"near-zero when `needsBrowser` was false" instead of "0" — the Date.now()
delta around the function body is sub-ms but not literally zero.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the browser probe / duration discovery / recompile / media
reconciliation block out of `executeRenderJob` into
`services/render/stages/probeStage.ts`. No behavior change. The sequencer
calls `runProbeStage` at the same code point with identical inputs and
outputs.
The probe stage owns the `FileServerHandle` and the `CaptureSession` it
creates and returns them to the sequencer. The sequencer still tracks
them in its `let fileServer` / `let probeSession` bindings and closes
them in its `finally` block — the resource lifetime is unchanged.
`recompileWithResolutions` lives inside this stage because it depends on
browser-resolved durations even though §2.1 of the distributed plan
lists recompile as a sibling phase.
Preserved invariants:
- `composition` is mutated in place (videos / audios / duration) so
downstream stages see the reconciled view through the same reference.
- `job.duration` and `job.totalFrames` end up with the same values at
the same code points. The result type carries `duration: number`
alongside `totalFrames: number`, and the sequencer re-asserts the
assignments after the call so TypeScript's control-flow narrowing
works for the rest of `executeRenderJob`.
- `perfStages.browserProbeMs` and `perfStages.compileMs` are written at
the same code points with the same values.
- The "Composition duration is 0" diagnostic builds the same hint string
from the same console-buffer regex and `__timelines` probe.
- The post-probe "failed network requests" warning fires with the same
regex, the same first-10/first-5 slicing, and the same `console.warn`
prefix.
Renderer smoke-tested inside `Dockerfile.test` against `font-variant-numeric`,
`many-cuts`, and `variables-prod` — all PSNR / audio correlation baselines
match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a `TODO(distributed-render):` comment near the `applyRenderModeHints`
call documenting that this caller-owned-object mutation needs to move
into the result type before `freezePlan` wires up. The mutation pattern
works in-process but won't survive across processes / replays from a
frozen plan — the value belongs in `LockedRenderConfig`, not on a
mutated `EngineConfig`.
No behavior change. Comment-only.
Review feedback addressed: vanceingalls on #718.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the pure compile sub-stage (`compileForRender` + `applyRenderModeHints`
+ `writeCompiledArtifacts` + `CompositionMetadata` build + DPR resolution)
out of `executeRenderJob` into `services/render/stages/compileStage.ts`.
No behavior change. The sequencer calls `runCompileStage` at the same code
point with identical inputs and outputs. The following invariants are
preserved verbatim:
- `cfg.forceScreenshot` is still mutated by `applyRenderModeHints`.
- `perfStages.compileOnlyMs` is set to the same wall-clock interval (around
the `compileForRender` call only).
- The "Compiled composition metadata" log line is emitted after artifact
writes with the same payload shape.
- The "Supersampling composition via deviceScaleFactor" log line is emitted
only when `deviceScaleFactor > 1`.
- `stage1Start`, `updateJobStatus(..., "Compiling composition", 5, ...)`,
and `perfStages.compileMs` (set at the end of probe) remain at their
current code points in the sequencer.
The probe sub-stage (`if (needsBrowser)`) is unchanged — it is extracted
separately in PR 1.3. `recompileWithResolutions` lives inside the probe
block because it depends on browser-resolved durations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add `PLAN_HASH_SCHEMA_PREFIX = "hyperframes-plan-hash-v1\x00"` mixed
into every digest. Future framing changes must bump the trailing
integer; this makes cross-version mismatches visible at the wire
format instead of producing silent collisions. Impossible to backfill
later, easy to bake in now.
- Hoist the `0x00` field delimiter to module scope (`FIELD_DELIMITER`).
- Document the UTF-8 encoding contract for all string-typed input
fields in the file-level JSDoc, so external verifiers know the
encoding without reading the implementation.
- Add a known-digest test for one fixed reference input
(`995b4105...`). If the framing changes silently this test fails,
forcing the developer to also bump the schema prefix.
- Add an explicit `canonicalJsonStringify(undefined) → TypeError` test
to pin the contract.
No callers yet, so no behavior change in any code path.
Review feedback addressed: vanceingalls on #717.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seed PR for the Phase 1 staged refactor of `executeRenderJob`. Creates the
new `packages/producer/src/services/render/stages/` directory and adds two
files that subsequent stage-extraction PRs build on:
- `planHash.ts`: a content-addressed sha256 helper plus a canonical-JSON
serializer, with unit tests covering determinism, asset-order
independence, sensitivity to each hashed component, and a delimiter-
framing test against path/sha boundary collisions.
- `freezePlan.ts`: signature-only skeleton (throws "not implemented") for
the eventual plan-freeze step. No callers; the body lands later when the
distributed-render primitives compose the Phase 1 stages.
Zero behavior change. No code in `executeRenderJob` is touched and no
existing exports move. The new files are not yet referenced anywhere
outside the `stages/` directory.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four existing presets only cover 16:9 (landscape) and 9:16 (portrait)
aspect ratios. A 1080×1080 square comp had nowhere to land at any scale:
"Auto" rendered at the comp's authored 1080×1080, and picking 1080p or 4K
mapped to a landscape/portrait preset whose aspect ratio mismatched, which
the producer's resolveDeviceScaleFactor validator rejects with
"does not match the aspect ratio of the composition".
Add `square` (1080×1080) and `square-4k` (2160×2160) to CANVAS_DIMENSIONS
in core. The existing `keyof typeof CANVAS_DIMENSIONS` derivation
extends the `CanvasResolution` union and `VALID_CANVAS_RESOLUTIONS` array
automatically, so the producer's validator, the render API route, and
the CLI `--resolution` flag pick the new presets up without further
changes.
- core: extend CANVAS_DIMENSIONS, RESOLUTION_ALIASES, and the
htmlParser to recognize `data-resolution="square|square-4k"` and to
infer square from equal width/height (vs. the prior "square defaults
to portrait" tie-breaker).
- studio: extend the local ResolutionPreset / CANVAS_DIMENSIONS mirrors;
collapse isPortraitComp into a 3-way `compAspect` helper so
resolveResolution returns the square preset for square comps.
- cli: update --resolution help text on `init` and `render` to mention
the new presets.
- tests: add square cases to renderOrchestrator's resolveDeviceScaleFactor
suite (returns 1 for square→square, 2 for square→square-4k, rejects
landscape preset on square comp), update the htmlParser test that
previously pinned the "square→portrait" tiebreaker.
Cleanup from the /simplify pass on PR #715.
- App.tsx: subscribe to the runtime's `stage-size` message (which
carries authoritative width/height post-applyCompositionSizing)
instead of re-parsing data-width/data-height from the iframe DOM.
Drops the cross-origin try/catch, querySelector, and parseInt logic,
and fires once per comp load instead of on every state/timeline tick.
- App.tsx: import CompositionDimensions from RenderQueue instead of
inlining the shape.
- RenderQueue.tsx: replace scaleLabel() with a SCALE_LABEL record,
inline the one-call formatDims helper, and trim the type comment to
the WHY.
Two bugs in getSharedBrowser() could take down the entire Vite dev
server:
1. Unhandled rejection from puppeteer.launch() — the timeout error
surfaces through puppeteer's internal RxJS chain, and any uncaught
path crashes the Node process. The thumbnail route's try/catch
doesn't always intercept it.
2. _browserLaunchPromise was never reset on failure, so subsequent
thumbnail requests reused a stale rejected promise instead of
retrying.
Wrap the IIFE in try/catch, return null on any failure (the thumbnail
route already handles a null adapter result with a 500), and reset
_browserLaunchPromise in a finally block so a transient launch failure
doesn't poison the singleton. Also drop the launch timeout from
puppeteer's 30s default to 10s so a wedged handshake fails fast instead
of stalling every pending thumbnail.
Verified locally: the dev server now logs
"[Studio] puppeteer launch failed — thumbnails disabled: ..." and
keeps serving the studio UI after a thumbnail request fails.
Orientation is a property of the composition, not a user choice — the
backend's portrait/landscape presets are tied to the comp's authored
aspect ratio. Letting users pick "1080p portrait" for a landscape
composition just produces a wrong-aspect render.
The dropdown now exposes three scale choices (Auto / 1080p / 4K) and
maps to the correct portrait/landscape preset based on the active
composition's data-width / data-height. Native <select title> tooltips
are unreliable across browsers, so the resolved dimensions render
inline in each option label (e.g. "1080p · 1920×1080") — always
visible, no hover needed.
App.tsx tracks the active comp's dimensions by listening for the
existing hf-preview state/timeline postMessages (same source the
caption-detection logic uses) and passes them to RenderQueue. The
useRenderQueue / backend contract is unchanged: RenderQueue still emits
"landscape" | "portrait" | "landscape-4k" | "portrait-4k" | "auto".
- Hoist duplicated test mock helpers (createMockAudioContext / setupTransport /
mockBuffer / mockEl) from the two describe blocks to module scope.
- Drop redundant math-derivation comments in schedulePlayback; the dedicated
rate-aware tests are the canonical proof.
- Tighten setRate JSDoc.
- Add no-op guard in setRate when the new rate equals the current rate, so a
duplicate set-playback-rate postMessage doesn't re-anchor or walk active
sources for nothing.
- Add a regression test for the no-op guard, and strengthen the clamp test
to schedule at rate=2 first so the clamp-to-1 assertion is non-vacuous.
WebAudioTransport scheduled AudioBufferSourceNodes with the implicit
default playbackRate of 1, so non-1x transport rates desynced visuals
from audio: GSAP timelines, the transport clock, and native <video>
all sped up while WebAudio-routed <audio> clips kept playing at 1x.
- schedulePlayback now accepts a rate, sets sourceNode.playbackRate,
and scales the future-clip start delay by the rate (the in-progress
buffer offset stays elapsed + mediaStart, which is rate-independent).
- New setRate() updates active sources in place and rebases the
getTime() reference frame so the audio-master clock stays continuous
across mid-playback rate changes.
- Runtime onSetPlaybackRate now forwards into webAudio.setRate, and
player.play() schedules each clip with state.playbackRate.
Fixes#713
Adds a new docs page under Getting Started that links to the
heygen-com/hyperframes-launches repo — open-source HyperFrames
compositions behind HeyGen's product launch videos. Includes a brief
catalog of the 5 projects currently in there, framing on why these are
useful (multi-composition shape, real adapter mix, production-grade
timing), and the LFS-aware clone recipe.
Cross-linked from `docs/examples.mdx`'s Next Steps and from
`docs/community/adopters.mdx`.
Came from a Discord ask via blackNoir (forwarded by James) — users
landing on the docs want to see how the internal team builds their own
videos through HyperFrames; this surfaces that source in one click.
The agent skill loader rejects SKILL.md files whose frontmatter description
exceeds 1024 characters, so remotion-to-hyperframes was being skipped at
startup with a "exceeds maximum length of 1024 characters" warning.
Trimmed the description from 1240 to 896 characters by collapsing the
trigger-phrase examples and tightening prose, while preserving every
trigger / no-trigger guardrail. Moved the detailed list of trigger phrases
and the 4 negative cases into a new "## When to use" section in the body
so the guidance is not lost.
Fixes#688
GitHub strips autoplay/loop from <video> tags in markdown, so the MP4
required a click to play. WebP autoplays via <img> while preserving full
HD source quality (1280x720 vs the previous GIF's 400x225).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The autobuild-2026-04-23-13-16 release was rotated out of BtbN's
recent-dailies window, returning 404 on the Windows test/render jobs.
Switch to the 2026-04-30 month-end snapshot, which BtbN keeps long-term
(visible in the persistent monthly-snapshot history).
The first dynamic `await import("./render.js")` cold-load takes >5 s on
Windows runners — long enough to blow vitest's default 5 s timeout in
whichever test ran it first. Subsequent imports are <10 ms because the
module is now cached, so only test #1 ever times out.
The downstream failure is more subtle: when test #1 times out, vitest
moves on, but its leaked async function eventually hits the synchronous
`producer.createRenderJob(...)` line and pushes a stale config to
`producerState.createdJobs`. That push lands AFTER test #2's `beforeEach`
clears the array, so test #2's `createdJobs[0]` is the leaked test #1
entry instead of its own. That's why test #2 saw `browserGpuMode: 'software'`
when it expected `'auto'`.
Hoist the import into `beforeAll` (matching the pattern the existing
`parseVariablesArg` and `validateVariablesAgainstProject` describe blocks
in this file already use). Cold-load happens once outside any test's
timeout window, every test stays fast, no leaked promise can corrupt
state.
Failing run: https://github.com/heygen-com/hyperframes/actions/runs/25470257972/job/74732502915
Started failing on main with the merge of #642 (auto-detect-browser-gpu),
which added the "forwards browserGpuMode='auto'" test as test #2.
Drive-by fix: hf#631 (composition flag) merged with two test calls
using `browserGpu: false`, but hf#642 (browserGpuMode auto) merged
shortly after and removed that field from RenderOptions in favour of
the tri-state `browserGpuMode`. Main has been failing typecheck since
hf#642 landed (every PR inherits the failure).
Renaming `browserGpu: false` → `browserGpuMode: "software"` matches
the new shape; both tests still verify what they were written for
(forwards entryFile / omits entryFile to createRenderJob).
After hf#641 inlined the runtime IIFE into every bundle, lint tools
inspecting bundled output (including Abhay's c2v eval) started flagging
empty `catch {}` blocks across the runtime. The source had explanatory
comments inside, but esbuild's minifier strips them — the IIFE ships
~10 visible patterns of `}catch{}` and consumers' linters fire on each.
Each empty catch is intentional best-effort error swallowing —
postMessage to a parent frame that may not exist, `media.play()` /
`pause()` that throw under autoplay restrictions, timeline `seek()` on
a disposed timeline, anime.js / lottie feature detection on hosts that
don't load those libraries, etc. The right behaviour stays "tried,
didn't work, move on", but doing it visibly improves three things:
- lint clean: helper call is a real statement; no `no-empty` warnings
survive minification
- debuggable: flip `window.__hfDebug = true` in DevTools to see every
swallow site with `console.debug` (silent in prod by default)
- observable: studio / embeddings can install
`window.__hf.onSwallowed = handler` to collect runtime swallow
events without polluting the page console
Implementation: `packages/core/src/runtime/diagnostics.ts` exports
`swallow(label, err?)`. 41 catch sites across 12 runtime files
converted via mechanical pass (auto-generated `runtime.<module>.siteN`
labels — labels can be tightened site-by-site as a follow-up; the
shape of the change is what matters here).
Verification:
- core 674/674 (incl. 6 new diagnostics tests covering silent default,
__hfDebug logging, legacy __HYPERFRAMES_DEBUG flag, handler hook,
handler-throws-doesn't-recurse, both-active)
- typecheck clean
- format / lint clean
- runtime IIFE rebuilds successfully (`bun run build:hyperframes-runtime`)
Refs Abhay's c2v eval — bundler artefacts now lint-clean with the
runtime body inlined.
Remove dollar amounts, render-time figures, and credit allowances from the
Vercel/Cloudflare tabs. These were sourced from the template READMEs but go
stale fast (pricing changes) and are load-bearing on a single composition's
quirks (perf isn't proportional to duration). Keep qualitative framing and
link out to the canonical pricing pages instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Reword frontmatter description to describe capability, not providers
- Soften "deploy in one click" — Cloudflare requires Workers Paid plan
- Qualify Cloudflare ~25s perf number as a local-Docker measurement
on a 6-vCPU host, not a standard-4 production figure
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaces the two official one-click deployment templates
(heygen-com/hyperframes-vercel-template, heygen-com/hyperframes-cloudflare-template)
in the docs site. Previously they only existed as GitHub READMEs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two files on main fail `bun run format:check`:
- `registry/registry.json` (missing trailing newline)
- `registry/blocks/vfx-liquid-glass/vfx-liquid-glass.html` (whitespace
+ quote-style)
Local format-check on the auto-detect-browser-gpu branch only flagged
these once rebased onto current main (post-#647 / v0.5.2). Pure
whitespace fix; no semantic change.
Three follow-ups from Vai's staff-eng review:
1. Concurrent-probe race (real bug): the parallel coordinator runs N
workers via Promise.all, so `--workers 4` on a no-GPU host fired 4
simultaneous probe Chromes — each paying the same 240 ms launch cost.
Cache the *Promise* (not the resolved value): first caller assigns
the in-flight Promise, every other concurrent caller awaits the same
one. Verified with a new test asserting all concurrent callers get
the identical Promise reference.
2. Stale rendering.md (lines 23, 29): user-visible contract said
"browser GPU enabled by default", which was wrong post-auto. Now
describes the auto / hardware / software trichotomy explicitly.
3. Silent fallback: auto-mode produced no output, so a regression to
"always falls back to software even with GPU present" would have
been invisible in production logs. Added a single stderr line per
process when the probe resolves: `[hyperframes] browserGpuMode auto
→ <mode> (<reason>)`. Cache hits don't re-log.
Verification:
- Engine 536/536 (incl. new concurrent-dedup test asserting Promise
reference equality across simultaneous callers)
- CLI 256/256
- Format / lint / typecheck clean
Initial PR carried a backwards-compat shim where RenderOptions had both
`browserGpu?: boolean` (for docker) and `browserGpuMode?` (for local).
Since renderLocal/renderDocker have no external callers, simplify to a
single field. The boolean → docker-args conversion now happens inline at
the one site that needs it (`browserGpu: options.browserGpuMode === "hardware"`
when handing off to dockerRunArgs).
No behaviour change. 535/535 engine + 256/256 CLI still pass.
When the host doesn't have a usable GPU (CI containers, eval rigs without
GPU passthrough, dev VMs), Chrome's hardware-mode WebGL flags
(`--use-gl=egl/metal/d3d11`) silently leave WebGL unavailable —
`getContext("webgl")` returns null, three.js' WebGLRenderer dies, the
canvas stays black. Surfaced today by Abhay's c2v-eval failing on a
docker render of an hf bundle that uses three.js + a custom fragment
shader.
The fix that's been there: `--use-gl=angle --use-angle=swiftshader` (CPU
software WebGL, ~5-50× slower but pixel-identical). The engine already
exposed `browserGpuMode: "software"` for this. The gap was discovery —
users had to know to pass `--no-browser-gpu` on no-GPU hosts.
This change adds `browserGpuMode: "auto"` (now the CLI default for local
renders): on first launch in the process, probe Chrome with hardware
args, check `canvas.getContext("webgl") !== null`, cache the result.
~1-2 s on first render, free on every subsequent render in the same
worker. Hardware GPUs keep their fast path; no-GPU hosts get SwiftShader
without ceremony.
Behaviour matrix:
- No flag, no env, local → "auto" (NEW default)
- `--browser-gpu` → "hardware" (force; errors if no GPU)
- `--no-browser-gpu` → "software" (force SwiftShader)
- `PRODUCER_BROWSER_GPU_MODE` → "hardware" / "software" / "auto" / unset
- Docker mode → forced "software" (unchanged)
Engine-config default stays "software" (conservative for embedders); the
"auto" default lives in the CLI's `resolveBrowserGpuForCli` so producer
embedders aren't surprised by a probe-on-launch.
Also adds `--enable-unsafe-swiftshader` to the software flag set —
Chrome 120+ deprecated implicit SwiftShader fallback and emits a
deprecation warning unless the flag is set explicitly. Despite the
"unsafe" name this is exactly the pre-deprecation behaviour; the rename
is about Chrome's threat model on the open web, not about the rendering
itself.
Verification:
- Engine 535/535 + CLI 256/256 (incl. new probe tests + tri-state CLI test)
- Empirical: probe on this no-GPU devbox returns "software" in 240 ms,
cached 0 ms on subsequent calls
- Format / lint / typecheck clean across all packages
Refs the Abhay/Slack thread on c2v-eval rendering without a GPU node.
Per CodeQL's `js/bad-tag-filter` recommendation, replace the regex-based
`<script>` body extraction with a `parseHTML` + `querySelectorAll`
walk. The rule explicitly says "use a parser library" — and linkedom
is already imported in this file, so the diff is small.
This eliminates the regex entirely, so the rule can no longer fire on
this site (instead of chasing whitespace / case / trailing-content
edge cases one at a time).
CodeQL still flagged `</script\s*>` as too narrow — the rule wants
tolerance for `</script\t\n bar>` (HTML parser treats trailing content
in a close tag as part of the tag). Switched to `</script[^>]*>` for
full coverage.
The bundler still always emits the canonical `</script>`; this is
test-side hardening, not a runtime fix.
CodeQL's `js/bad-tag-filter` rule flagged `</script>` as too strict —
`</script >` (with whitespace before `>`) is valid HTML and would slip
past the matcher. Changed to `</script\s*>` for full defense-in-depth.
The bundler always emits the canonical form, so no real-traffic miss —
this is hardening the test's parse-loop, not fixing a downstream bug.
Addresses CodeQL alert on #641.
CodeQL flagged the inline `<script>...</script>` regex as case-sensitive,
which would miss `<SCRIPT>` tags. The bundler always emits lowercase, so
this is a defense-in-depth fix matching the `/i` flag already used by the
sibling regexes in this file (lines 37 & 75).
Addresses CodeQL review on #641.
- Extract applyMask helper from postprocess and add 5 unit tests pinning
the contract this PR is selling: fg.alpha + bg.alpha === 255 per pixel,
RGB triples byte-identical between fg and bg, and bg=null path leaves
the bg buffer untouched. Without these, a future postprocess change
(mask threshold, premultiplied alpha, gamma) could silently break the
inverse-alpha relationship and the existing plumbing tests would all
still pass.
- Add stdin 'error' listener inside spawnFfmpeg. If either encoder dies
mid-render, Node emits an unhandled error on the dead writable on the
next .write() and crashes the CLI before waitForExit's reject path
can surface the encoder's stderr tail. Doubled encoder count = doubled
failure surface, so this is worth pinning down.
- Tighten stdio param to a 3-tuple so an accidental 1-element array fails
at type-check.
- Sharpen backpressure comment: write→true means "highWaterMark not
exceeded," not "libuv flushed." Reuse-without-corruption is safe only
because session.process is slow enough that libuv drains in between.
Addresses review on PR #637.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Emit an inverse-alpha background plate alongside the cutout in a single
inference pass. Same source RGB, alpha = 255 − mask. Dual-encoder pipeline
runs in parallel; both outputs share the same --quality preset.
This is a hole-cut plate (subject region transparent), not an inpainted
clean plate — composite something opaque under it to fill the hole.
Docs and skill cover when each is the right tool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four regression tests (font-variant-numeric, many-cuts, missing-host-comp-id,
variables-prod) failed on this PR with `Unable to parse PSNR output at <last
checkpoint>s`. Root cause: the harness derived all 100 checkpoints from the
*rendered* video's container duration, then asked ffmpeg's PSNR filter to
compare the same frame index from both videos.
The encoder changes earlier in this PR add `-avoid_negative_ts make_zero` to
the mux step. With AAC audio that shifts the first audio sample to t=0
instead of the encoder-delay offset, extending reported container duration
by ~20ms without changing video frame count. For the four failing tests,
the i=99 checkpoint then landed on a frame index that exists in the rendered
video but not in the snapshot baseline (e.g. round(2.98998 * 24) = 72 in a
72-frame baseline). ffmpeg's PSNR filter ran on zero matched frames and
emitted no `average:` line, so the parser threw.
Fix: probe both videos and use min(rendered, snapshot) duration when
spreading checkpoints. This is the correct semantics for symmetric PSNR
comparison anyway — both videos must have a frame at every sampled time.
The change is local to the harness; no encoder behavior changes, no
baselines regenerated.
Other regression tests with audio (chat, sub-composition-video,
vignelli-stacking) passed because their checkpoint-99 frame index landed
inside the baseline's frame range with several frames of slack. The four
failing tests had round-number durations where a 20ms drift was enough to
push the last checkpoint past `nb_frames - 1`.
The model removes background from any video with a person — we tested
with avatars because they were convenient, but anyone can bring a
talking-head clip, presenter footage, vlog, etc. Replace avatar-specific
filenames (avatar.mp4 / brandon.mp4) with neutral subject.mp4 (or
presenter.mp4 in the text-behind-subject example) and rephrase
copy that read as if avatars were the only use case.
Touches docs/guides/remove-background.mdx, hyperframes-media SKILL.md,
and hyperframes/patterns.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- engine/chunkEncoder, engine/streamingEncoder: extend `-bf 0` to GPU h264
paths (nvenc, qsv, vaapi) and `-b_strategy 0` for qsv so GPU-encoded
outputs avoid negative-DTS freezes too — not just SW libx264.
- engine/videoFrameExtractor: detect mid-path traversal (e.g.
`assets/../../foo.mp4`) by normalizing first and re-anchoring at the
project root. Adds a regression test.
- engine/videoFrameExtractor: dedupe stderr "src not resolvable" warnings
by `video.src` so a comp with N broken sources logs once, not N times.
- engine/videoFrameExtractor.test: drop dynamic `require("node:fs")`,
use ES `import { writeFileSync } from "node:fs"`.
- engine/ffprobe: extract `readTagCI` helper for case-insensitive ffprobe
tag reads (will recur for other libavformat-versioned sidecar tags).
- cli/background-removal/pipeline: collapse Quality / QUALITIES /
QUALITY_CRF / DEFAULT_QUALITY / isQuality surface using
`Quality = keyof typeof QUALITY_CRF`.
- producer/renderOrchestrator: replace `v.src.startsWith("/")` with
`isAbsolute(v.src)` in the HDR probe path so Windows absolute paths
(`C:\...`) aren't treated as relative — matches the audioMixer guard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tag-based alpha detection (alpha_mode / ALPHA_MODE / pix_fmt yuva*) is
fundamentally brittle. Failure modes seen in the wild:
- case-sensitivity across ffmpeg versions (alpha_mode vs ALPHA_MODE)
- older muxers that omit the sidecar tag entirely
- mp4-as-webm rewraps that drop the tag
- ffprobe reporting yuv420p for VP9-with-alpha because the alpha plane
lives in a Matroska BlockAdditional sidecar, not the main pix_fmt
Each of those silently strips alpha at extraction time. The bug doesn't
surface until the rendered output is missing layers — frustrating to debug,
silent in stdout. The previous case-insensitive fix patched one of the
failure modes; this commit removes the class.
The robust alternative is codec-based: any bitstream that CAN carry alpha
(VP9, VP8, ProRes 4444) gets the alpha-aware decoder and PNG output by
default, regardless of what the tag says. The cost is a small file-size
increase on opaque VP9/VP8 sources (cached PNGs vs JPGs); the benefit is
no class of silent alpha loss from tag misdetection.
- Adds codecMayHaveAlpha() + decoderForCodec() helpers and exports them.
- Updates extractVideoFramesRange to force libvpx-vp9 / libvpx for VP9 / VP8
unconditionally (was: only when metadata.hasAlpha).
- Updates resolveFrameFormat to default to PNG for any alpha-capable codec
(was: only when metadata.hasAlpha).
- +4 unit tests covering the codec table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Locks in the case-insensitive behavior alongside the existing alpha_mode
(lowercase) test. If either path regresses, the producer would silently
extract alpha-having webms as opaque JPGs and the injected <img> overlays
would cover every element below them on the z-stack — a bug that doesn't
surface in the studio preview, only in production renders.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Newer libavformat builds write the VP9-alpha sidecar tag as 'ALPHA_MODE'
(uppercase); older builds write 'alpha_mode'. ffprobe.ts only checked the
lowercase form, so files produced by recent ffmpeg encoders (including the
output of 'hyperframes remove-background' itself) were misclassified as
having no alpha channel. Knock-on effect: the producer extracted them as
JPGs (no alpha), the injected <img> overlays were fully opaque rectangles,
and any element below them on the z-stack (text, captions, other layers)
silently disappeared from the rendered output — even though the studio
preview rendered the same composition correctly via native <video> playback.
Symptom in our repro: a text-behind-subject composition showed the
headline correctly in studio preview but the production render covered
the headline entirely with the opaque avatar image.
Fix: read videoStream.tags.alpha_mode OR videoStream.tags.ALPHA_MODE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skill (hyperframes-cli): three-pattern table (cutout-over-different-scene
vs over-its-own-source vs over-different-take) + the two non-obvious rules
(wrap video in non-timed div for opacity control, both videos data-start=0
for sync). Skill (hyperframes/patterns): worked text-behind-subject example.
Docs: --quality flag, compositing pitfalls section, quality preset table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A <video src='../assets/foo.mp4'> inside a sub-composition silently dropped
from extraction; the rendered output froze on the first decoded frame for
the entire clip, with no error in stdout.
Root cause: browser URL resolver clamps '..' at origin root (studio preview
loads fine), but path.join(projectDir, '../assets/foo.mp4') normalizes to
parent-of-project/assets/foo.mp4, which usually doesn't exist. existsSync
returns false, extraction is skipped, no frame lookup is built, the
per-frame injector has nothing to swap, and the <video> element's first
decoded frame paints every screenshot.
- Adds resolveProjectRelativeSrc in videoFrameExtractor that mirrors browser
clamping (literal join first, then leading '..' stripped).
- Surfaces a loud stderr warning when the resolver misses.
- Mirrors fix in audioMixer.ts (same bug for <audio src='../'>) and
renderOrchestrator HDR probe loop.
- +6 regression tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related render robustness fixes:
1. frameCapture.ts: bump videos-ready check from `readyState >= 1`
(HAVE_METADATA — only dimensions known) to `>= 2` (HAVE_CURRENT_DATA —
first frame is rasterized). Without this, when two `<video>` elements
with different codecs (h264 mp4 + VP9 webm) decode at different rates,
the faster one passes readiness while the slower one still hasn't
painted, producing a black "first frame" for the slower clip.
2. chunkEncoder.ts (libx264 path) + streamingEncoder.ts: disable B-frames
for h264 (`-bf 0`). Standard libx264 with B-frames produces negative
DTS at stream start (the first B-frame's decode order is "before" the
first I-frame's presentation time). VS Code preview, several browser
<video> implementations, and some HW decoders freeze on the first
frame and only audio plays. -bf 0 makes PTS == DTS at every frame,
eliminating the issue at the source. Quality cost is ~5–10% larger
files at the same CRF — worthwhile for "the file plays everywhere".
3. chunkEncoder.ts (encoder + mux paths): add `-avoid_negative_ts make_zero`
as belt-and-suspenders against negative DTS sneaking back in via
`-c:v copy` mux passes when audio/video PTS bases differ.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- inference.ts: force `.toColourspace("b-w")` on the resized mask. Sharp upcasts
the 1-channel raw input to RGB-interleaved during resize, so `fullMask[i]`
was reading R,G,B,R,G,B... of pixels 0..691199 instead of the alpha for all
2,073,600 pixels. Visible symptom: horizontal scanline alpha artifact in
every transparent webm — the avatar appeared semi-transparent throughout.
- pipeline.ts: add BT.709 + limited-range colorspace tags so Chrome's YUV→RGB
matches the source mp4 (without these, ffmpeg's default RGB→YUV is BT.601
and skin tones drift visibly when the cutout is overlaid on its source).
- pipeline.ts: add Quality preset type ("fast"/"balanced"/"best" → CRF 30/18/12).
Default raised from CRF 30 → 18 ("balanced") so the most common pattern
(text-behind-subject) works out of the box without visible doubling.
- remove-background.ts: wire `--quality` flag with validation, +2 examples.
- Tests: BT.709 tags present, quality preset → CRF mapping, default is balanced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review on PR #619 caught two places that still pointed transcribe/tts
at hyperframes-cli — directly undercutting the description-trigger
goal of the split:
- skills/hyperframes/SKILL.md description ended with "For CLI commands
(init, lint, preview, render, transcribe, tts) see the
hyperframes-cli skill." Now splits the redirect: dev-loop commands
(init, lint, inspect, preview, render) → hyperframes-cli; asset
preprocessing (tts, transcribe, remove-background) →
hyperframes-media.
- packages/cli/src/templates/_shared/CLAUDE.md is the skills table
baked into every project bootstrapped by `hyperframes init`. Its
hyperframes-cli row still listed transcribe/tts. Trimmed to the
dev-loop commands and added a hyperframes-media row beside it, so
new projects pick up the correct mapping.
Also caught by greppping for stale skill lists:
- .codex-plugin/plugin.json longDescription bundled transcribe/tts
into "use the CLI for ...". Split into "use the CLI for the dev
loop (init/preview/render), preprocess assets
(tts/transcribe/remove-background)" so the Codex plugin store
surface matches reality.
Confirmed `npx hyperframes skills` shells out to `npx skills add
heygen-com/hyperframes --all` (packages/cli/src/commands/skills.ts),
so the skill list is read dynamically from the repo and picks up
hyperframes-media without code changes.
Code review found the new hyperframes-media skill was parallel
content with skills/hyperframes/references/tts.md and the "Whisper
Model Guide" section of transcript-guide.md — same voice table, same
.en-translates-non-English warning, same TTS→transcribe chain in
both places. Plus some scope creep in hyperframes-media (audio/video
HTML snippets that duplicate the canonical track docs in
hyperframes/SKILL.md:265+).
Consolidation:
- hyperframes-media is now the single source of truth for CLI
invocation, voice selection, multilingual phonemization, whisper
model selection, and the .en gotcha. Picked up the multilingual
prefix decoding from the deleted tts.md.
- skills/hyperframes/references/tts.md deleted; the bullet in
hyperframes/SKILL.md is removed (no replacement — agents land on
hyperframes-media via its own description).
- skills/hyperframes/references/transcript-guide.md keeps only the
caption-side concerns: input-format table, mandatory quality
check, cleaning JS, external-API import path, and the
"if no transcript exists" flow. The intro bash recipe and Whisper
Model Guide section both moved to hyperframes-media. Top of the
file now points to hyperframes-media for CLI/model details.
Other tightening in hyperframes-media:
- Dropped WHAT-narration filler and the inline <audio>/<video> HTML
snippets — they duplicate the canonical track-attribute docs in
hyperframes/SKILL.md.
- Added the `id` field (`w0`, `w1`, ...) to the transcript output
shape — the actual Word interface in
packages/cli/src/whisper/normalize.ts includes it (optional for
backwards compat), used by caption override logic.
- Compressed the TTS → Transcribe → Captions chain section.
Net: hyperframes-media 147 → 136 lines, transcript-guide.md 152 →
106 lines, tts.md gone (-75 lines).
Move tts/transcribe/remove-background guidance into a new
hyperframes-media sibling skill so the CLI skill stays focused on
the dev loop (init/lint/inspect/preview/render/doctor).
Two motivations:
1. Description bloat. The CLI skill listed every subcommand as a
trigger keyword, which made agents auto-load it for any mention
of audio, transcription, or backgrounds — even when the task
was just rendering a composition.
2. Body bloat. Voice tables, the .en-translates-non-English
whisper rule, and codec selection guidance all loaded on
every CLI invocation. With three preprocessing commands now
in the CLI (tts, transcribe, remove-background), this is only
going to grow.
The split keeps a single sibling (hyperframes-media), not three:
the commands share a workflow (preprocess asset → drop into
composition) and the same first-run-downloads-a-model pattern,
so they belong together. CLI skill now references hyperframes-media
from a one-paragraph "Asset Preprocessing" stub.
Doc references updated in README.md, CLAUDE.md,
docs/quickstart.mdx, and docs/guides/prompting.mdx.
Align the no-op timeline duration with the root's data-duration. The
fixture's root has data-duration="3" but the placeholder timeline tween
was still { duration: 2 } — leftover from when I bumped the duration
from 2s to 3s to dodge the PSNR-checkpoint-at-1.99s parse edge case.
The tween is a no-op (no targets, no visible effect) so rendered pixels
don't change; baseline still passes Docker regression at 100/100
checkpoints.
Reuse + efficiency reviews otherwise clean. Two findings deferred:
silence.wav duplication is real but only 2 fixtures share it today —
worth extracting to tests/_shared/ when the third fixture lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
