Pure service for Phase -1 of the long-form render plan: merges per-pid RSS
rows into browser/renderer peaks, last-sample total and GPU presence on an
unref'd interval with a reentrancy guard, so a CDP or ps failure can never
reach the capture loop. CaptureOptions.onMemorySample is the live route the
producer will wire so a mid-render crash still reports memory state. Nothing
instantiates the sampler yet (Task 3); no behaviour change.
Beyond the plan's tests, pins the double-start, empty-sample and catch
guards directly, since the interval-based error test stays green when the
catch is removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(capture): never silently render a webgpu composition blank
A composition declaring data-requires-webgpu only got the Chrome WebGPU
flag when hardware GPU auto-detection succeeded. On a software-GPU host
it launched without the flag and rendered blank with no error.
compositionRequiresWebGpu (static declaration check) and
assertWebGpuAdapterAvailable (live per-page adapter probe) now live once
in packages/engine's browserManager, the shared choke point every launch
site already goes through. All six launch sites (frame capture, the four
CLI diagnostic commands, the studio thumbnail server, and the render
pipeline's capture-options builder) route through those two functions;
the old CLI-only static-heuristic guard is deleted, not kept alongside
the new one. A software host that cannot obtain any WebGPU adapter now
fails loudly naming the requirement instead of producing a blank frame.
* fix(capture): thread requiresWebGpu through the distributed chunk-render path
The distributed renderChunk() path built its CaptureOptions without
requiresWebGpu, so it fell through to frameCapture's own fetch-based
detection, whose failure path silently treats "requires webgpu" as
"doesn't" if that fetch ever fails. Every other launch site (frame
capture, all four CLI diagnostic commands, the studio thumbnail server,
and the in-process render pipeline) already reads the value from HTML
already in memory or on disk instead of relying on that fallback.
Reads the compiled composition's index.html the same way probeStage
already does and derives requiresWebGpu from it before building
CaptureOptions. Adds a regression test that pins the exact path
renderChunk() reads at render time, since the composition doesn't
actually exercise WebGPU so an end-to-end render can't distinguish the
two cases on this host.
* test(producer): observe the real CaptureOptions renderChunk() builds
The previous regression test called plan() and compositionRequiresWebGpu()
directly, bypassing renderChunk() entirely, so it could not detect a
regression to the wiring it was meant to protect (confirmed: it stayed
green with the fix reverted).
Exports the existing createCaptureSession dependency seam so a test can
substitute a spy that delegates to the real implementation while
capturing the CaptureOptions object renderChunk() actually constructs.
Verified non-vacuous by reverting the fix and confirming the test fails.
* fix(gcp-cloud-run): resolve puppeteer-core from a real file in the webgpu probe
The image build's WebGPU adapter probe imported puppeteer-core from an
inline `bun -e` eval string. The eval's synthetic /app/[eval] location
can't resolve the workspace-hoisted dependency, so every build of this
stage failed closed with "Cannot find package 'puppeteer-core'" before
the probe itself ever ran.
Moves the probe into a real script file next to probe-beginframe.ts,
which already imports puppeteer-core dynamically the same way and
resolves fine from its real workspace location. Verified locally against
a cached chrome-headless-shell binary.
* fix(engine): bound the webgpu adapter probe so a stuck driver can't hang the caller
requestAdapter() has no native timeout. A broken software GPU driver
that never settles it would hang assertWebGpuAdapterAvailable
indefinitely, including inside the GCP image build's probe RUN step,
producing an opaque build timeout instead of a clean probe failure.
Races the in-page request against a bounded timeout so a stuck adapter
reads as unavailable and fails the same way a genuinely absent one does.
* test(cli): mock the engine's webgpu exports in the browser-fallback test
* test(producer): run the typegpu adapter fixture only on hosts with a hardware WebGPU adapter
The fixture declares data-requires-webgpu, so a software-only host now refuses it
instead of silently rendering the shader blank. The old golden had no shader in it.
Tag it webgpu, skip that tag in the CI image and the test scripts, and re-record the
golden from a Metal render with the shader visible.
* fix(engine): bound the composition fetch that decides the webgpu flag
createCaptureSession fetched the served page with no timeout before launching the
browser, so a hung server stalled a render forever and hung a fake-timer test. Give it a
five second limit, and let that test say it has no WebGPU requirement the way the
producer does.
* fix(capture): refuse a software fallback WebGPU adapter instead of rendering on it
swiftshader reports a WebGPU adapter but cannot render canvas WebGPU, so a
data-requires-webgpu composition on a software-only host crashed on its first frame.
Refuse fallback adapters with the same clear WebGPU-unavailable error. The GCP image
probe now serves its page over http (file:// hides navigator.gpu) and reports a
software-only host instead of failing the image build.
* fix(engine): parse the composition root tag without a backtracking regex
* fix(engine): keep quoted attribute values whole when reading the composition root tag
* feat(core): hold render-ready on declared compute and a build-ready hold, not just duration
Adds the compute and paint-and-idle inputs to the composition readiness gate,
plus window.__hf.buildReady: a promise registry a piece registers when it has
setup work no adapter can see (mesh building, shader compiles). The runtime
holds render-ready on it and the player shows its loading state instead of a
blank frame while it resolves.
* refactor(core): dedupe the paint-and-idle readiness test setup
Simplify pass on 23810da1b: fallow flagged the two paintAndIdleReadinessInput
tests as a 12-line clone; extracted the shared doc/tracking setup.
* fix(core): stop the readiness settled-tracker hanging on 2+ pending promises
Promise.all allocates a new promise every call, so comparing ITS identity
across repeat polls (isAdapterReadinessSettled, isBuildReadinessSettled) never
observes settled once 2+ promises are combined: every poll rebuilds a new
combined promise, re-arms, and republishes forever without the caller ever
seeing true. Reproduced with two already-resolved window.__hf.buildReady
entries, which is the exact multi-piece usage the new authoring doc describes.
Fixes both trackers at once with a shared createSettledTracker that compares
the source promise array by element instead of the Promise.all wrapper.
* fix(core): cancel readiness inputs once settleCompositionReadiness settles
computeReadinessInput's poll timer and paintAndIdleReadinessInput's rAF chain kept
running after the shared timeout won the race, since nothing told the losing
inputs to stop. One AbortSignal now threads through all three inputs and is
aborted the instant the race settles.
* docs(core): fix a stale sync-callback claim in settleCompositionReadiness
paintAndIdleReadinessInput always has a frame to wait on for any real
document, so the sync fast path this docstring described is no longer
reachable with the default inputs except for a viewless document.
Found by the simplify pass on this branch.
* fix(core): clear a stale buildReady entry on runtime teardown
A never-resolved window.__hf.buildReady promise from a torn-down
composition survived into the next init in the same window (window.__hf
is never reset), permanently blocking render-ready for whatever loaded
next -- the exact black-screen class this branch fixes, reproduced
through its own new mechanism. Found by the independent review.
Also fixes a stale hyperframes-player.ts docstring (media/images/fonts
had not been updated to compute + paint-and-idle) and documents the
buildReady key-collision hazard in three.md.
* fix(core): skip the compute readiness wait when no runtime ever ran
computeReadinessInput polled window.__renderReady for the full shared
timeout on any same-origin document with no HyperFrames runtime
injected (a probe-only composition using native video/duration, or a
blank test iframe) — nothing was ever going to set that flag. Checking
window.__hf first (the same signal composition-probe.ts already treats
as "has a runtime") lets it return null instead of waiting.
* fix(player): debounce the loading overlay and diagnose compute timeouts
paintAndIdleReadinessInput always has a frame to wait on for an
attached iframe, so settleCompositionReadiness's synchronous-settle
path is unreachable in practice now — the loading overlay showed on
every single Play, even ones with nothing genuinely pending. Debounce
the overlay itself instead: a wait that resolves fast never shows it.
Also names compute (window.__renderReady) in the stuck-assets timeout
diagnostic, which previously only re-scanned media, images and fonts.
* fix(engine): name a stuck buildReady key in the zero-duration diagnostic
A build promise a composition registers on window.__hf.buildReady that
never resolves left the export path's zero-duration timeout naming
nothing useful — hasHf/hasSeek/duration all looked fine, since the
runtime was genuinely present and just waiting on that one promise.
Probes buildReady's keys for pending ones and names them in the hint.
* fix(engine): stop the buildReady diagnostic from crashing on a rejection
A microtask race against a bare resolved promise rejected the whole
diagnostic expression on any rejected buildReady entry (the runtime's
own createSettledTracker rejects deliberately) and mis-sorted a settled
non-native thenable as pending, since its extra adoption tick loses the
race. Both measured directly, not inferred.
Settle each key onto its own record past one macrotask instead: no
throw, and pending/resolved/rejected are distinguished correctly. A
rejected key gets its own hint, separate from a merely pending one.
* fix(player): report a backgrounded document instead of overclaiming a paint bound
MAX_PAINT_WAIT_MS only bounds a document that keeps producing frames; one
that stops entirely (backgrounded tab suspends rAF, or fewer than 3 frames
ever fire) still rides the full shared 8s timeout, same as before this
input existed. Corrected the comments that claimed otherwise.
That gap is exactly why the stuck-assets diagnostic left paint-and-idle
unreported: there was no accurate way to say it might be the cause. Added
documentHidden (doc.hidden) to the diagnostic instead, since a backgrounded
tab is the reachable, directly observable case.
* refactor(engine): simplify the buildReady diagnostic rewrite
Skip the macrotask wait entirely when a composition registers no
buildReady keys — the common case, since it's opt-in. Nothing to settle
means nothing to wait on.
Collapse buildZeroDurationDiagnostic's inline parameter type into the
existing HfDiagnostic interface instead of maintaining two copies of
the same shape.
* docs(engine): note the setTimeout(0)-thenable gap the review found
An independent adversarial review, executing hostile inputs rather than
tracing, found the macrotask wait can still misreport a thenable that
resolves via its own internal setTimeout(0), since the diagnostic's own
timer is registered before the thenable's. Bounded and self-correcting
(only feeds the message text, and the diagnostic re-runs every ~1s), so
documenting the boundary rather than chasing an unbounded fix.
These tests spawn a bun process to check platform-specific chrome
binary resolution, and have hit the default test timeout on Windows
CI runners before. The log only ever shows the bare timeout, with no
way to tell whether the spawn itself was slow or something else in
the test body stalled. Wrapping each spawn in console.time/
console.timeEnd costs nothing when the test is fast and names the
slow step the next time it isn't. No behavior change.
* feat(engine): adaptive per-frame sample count for motion blur
Measured against the accuracy bar set in #4010 (72.4dB mean PSNR vs the
960fps shutter integral), the fixed default of 16 samples misses it for
hard-edge content while paying nothing extra for static/slow content is
still important. This picks a sample count per frame from measured
motion, floor 16 ceiling 64 (issue #4029).
Two probe captures at the true shutter-window edges (never a sample that
enters the average) measure motion magnitude, mapped through a monotone
step table to a sample count. Frames confirmed to move only non-spatial
properties (opacity, colour) skip the probe and use the floor directly,
since a fade's shutter integral is a smooth blend that does not need many
samples. MotionBlurPlan's window (from shutter angle/phase alone) is now
independent of how many samples fill it, which is what lets the count
vary per frame without moving the window; motionBlurWindowIsStatic is
bounded by the widest sample inset any chosen K could produce so the
static-dedup safety guarantee holds for every K in [16, 64].
* refactor(engine): simplify the adaptive motion-blur sample selection
Two findings from an independent code-reuse/quality/efficiency review of
the previous commit:
- armStaticDedup ran the GSAP-timeline walk (computeStaticFrameSet)
twice on the same page when adaptive motion blur and static dedup
both apply, once for each purpose. Shares the one result instead.
- The non-spatial floor path read the sample count from
adaptiveSampleCount(0), which only worked because the step table
happens to be sorted with the first bucket covering zero. Reads
the now-exported DEFAULT_SAMPLES_PER_FRAME directly instead.
* fix(engine): read raw options for the adaptive motion-blur gate
Independent review caught it: every armStaticDedup call site runs
before finalizeSessionInit resolves session.motionBlur, so the
adaptive-classification gate's `session.motionBlur &&
session.motionBlur.fixedSamplesPerFrame === null` was always false at
runtime. The non-spatial probe-skip described in the PR never engaged
outside tests that hand-set motionBlurNonSpatialFrames directly,
bypassing real init order.
Reads session.options.motionBlur (set unconditionally at session
creation) instead, which carries the same "adaptive requested"
information the resolved plan would have. Adds a source-order lock
test plus direct coverage of computeStaticFrameSet's
nonSpatialOnlyFrameSet, mutation-proved against the exact regression.
* fix(engine): add xPercent/yPercent to the spatial-property list
Round 2 of the independent review: the spatial-property list deciding
whether a frame's motion needs the adaptive probe omitted GSAP's
percentage-based position properties (xPercent/yPercent) and two 3D
transform properties (perspective/transformPerspective) that two other
lists already in this repo (packages/parsers/src/gsapConstants.ts,
packages/studio/.../gsapAnimatesProperty.ts) both carry. A composition
positioned via xPercent/yPercent would have been silently floored to
K=16 with no probe and no log line, the same failure shape as the
round-1 bug through a data gap instead of a dead gate.
Moves the list to an exported SPATIAL_TWEEN_PROPERTIES array in
motionBlur.ts, passed into computeStaticFrameSet's page.evaluate as
data rather than duplicated inline, so it stays a single list and is
directly testable. Adds tests that execute the real page-side closure
(the existing tests all mocked the whole result and never exercised
the matching code), mutation-proved against the exact gap found.
* fix(engine): add rotate to the spatial-property list
Round 3 of the independent review, comparing SPATIAL_TWEEN_PROPERTIES
side by side against the two sibling lists rather than trusting the
prior commit's summary: GSAP's individual "rotate" transform property
(distinct from "rotation", carried in
packages/studio/.../gsapAnimatesProperty.ts's GSAP_TRANSFORM_PROPS)
was still missing, same failure shape as the two prior rounds.
The same pass also raised transformOrigin, which is deliberately NOT
added: gsapConstants.ts's own classifyTweenPropertyGroup excludes it
as a pivot-point modifier with no independent visual effect, and a
test now locks that reasoning in rather than leaving it to be
"fixed" again by a future pass that does not read that far.
The static-frame dedup reuses the previous frame's buffer when a frame is
byte-identical to its predecessor. With motion blur on that test is not
enough: the shutter window reads content either side of the frame instant,
so a still frame next to a moving one reused a buffer that carries none of
that motion. At 30 fps and 16 samples frame 11's window runs 10.75 to 11.25
at a 180 degree shutter and 10.06 to 11.94 at 720, so both reach past the
frame instant and the default is affected too.
motionBlurWindowIsStatic now owns whether a blurred frame may reuse its
predecessor's buffer. It requires every frame the window reads from to be
static, and covers the previous frame's window as well, because reuse claims
the two blurred outputs are equal. The range is derived from the plan's own
sample tick offsets, so it cannot drift from where the samples land.
The guard's contract comes from #4013 by Dante-dan, which carried it before
we did. One existing test fixture is widened: it asserted reuse with only the
frame itself marked static, which the guard now correctly rejects.
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* feat(producer): make sub-frame motion blur reachable from a render
Second deliverable for issue #4010. The engine gained the accumulation path, but
nothing set CaptureOptions.motionBlur, so the feature could not be switched on
from a real render. RenderConfig now carries it and the producer threads it into
the capture options.
Doing that surfaced the defect that would have made it fail silently. "Does this
render capture PNG frames" was spelled four times, every one of them re-derived
from needsAlpha, which describes the OUTPUT container rather than the capture:
probeStage picked the probe capture format
renderOrchestrator picked the render capture format
encodeStage built the encoder's frame_%06d input pattern, twice
captureStage built the disk worker's frame extension
Motion blur is a second, independent reason to capture PNG: the sub-frame samples
are averaged pixel by pixel, so JPEG samples would be averaged after a lossy
quantization. Add that reason to only one of those four and the encoder goes
looking for frame_%06d.jpg against files written as .png, and the render finds no
frames at all rather than reporting an error.
So the rule now has one owner, resolveCaptureImageFormat, which names both
reasons. Everyone downstream reads the resolved capture format instead of
re-deriving it: the encoder and the disk worker share frameFileExtension with
writeCapturedFrame, which is what actually names the files. needsAlpha keeps only
the question it really answers, whether the output preserves alpha.
captureStage no longer needs needsAlpha at all, which is the tell that the
re-derivation was its only reason to know.
No JPEG encoder is involved. The PNG capture path is already load-bearing, since
every transparent-output render uses it and the streaming encoder already selects
png as its input codec.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* fix(engine): resolve the motion-blur plan on every initialization exit
A real render produced no blur at all. The plan was correct, the producer
wiring was correct, and the sub-frame sample times were genuinely distinct, but
K=16 and K=64 rendered identically to no blur because neither ever ran.
initializeSession has two exits: screenshot mode finishes its own block and
returns, every other capture mode falls through to the end. The plan was
resolved at the fall-through exit only, and screenshot is the one capture mode
motion blur supports, so on exactly the path the feature runs on
session.motionBlur was never assigned. captureFrameCore then read it as
undefined and took the single-capture branch: one capture per output frame, no
accumulation, no error.
finalizeSessionInit is now the single owner of what finishing initialization
means. It sets the plan and isInitialized together and is called at both exits,
so a third exit cannot mark a session ready while forgetting the plan.
Why the existing tests passed, which is the part worth keeping: the accumulation
test set session.motionBlur directly on a stub session. It exercised the
mechanism while bypassing the only code that decides whether the mechanism runs
at all. A test that proves something works and never asks whether it is
reachable is not end to end, however much behaviour it drives.
Two tests. A structural guard that session.isInitialized = true appears exactly
once and that the same place resolves the plan, which fails if the defect is
reintroduced at either exit. And a sample-distinctness assertion that hashes the
K buffers captured within one output frame and requires K distinct ones, which
covers the different failure of every sub-frame seek landing on one instant.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* feat(engine): add packageHls for HLS VOD packaging
Adds `packageHls` next to `muxVideoWithAudio`: stream-copies an
already-encoded H.264 video and optional AAC sidecar into an HLS VOD
directory (`master.m3u8` + split video/audio renditions, MPEG-TS
segments) with `-c copy`, so no re-encode happens at package time.
Depends on the locked GOP from the previous commit. `-hls_time` can only
cut at existing keyframes, so `gopSize = segmentSeconds x fps` is what
makes every segment exactly `segmentSeconds` long and IDR-aligned. A
test packages an unlocked encode to pin that: the same call yields one
4s segment instead of four 1s ones.
Deliberately omits the mp4 mux's provenance tags, which MPEG-TS drops
and whose `-movflags` is invalid for `-f hls`, and `-avoid_negative_ts`,
which would discard the AAC priming and shift audio against video (see
issue 3487). A real-ffmpeg test asserts the first presentable audio
sample lands on the first video frame.
No caller yet; `format: "hls"` in the producer follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(engine): zero HLS mux base and reject % in packageHls outputDir
- Pass -muxdelay 0 -muxpreload 0 so the first segment starts at PTS 0
instead of the mpegts muxer's 1.4 s base; A/V alignment through the
AAC priming packet is unchanged
- Throw on an outputDir containing %, which ffmpeg expands as a printf
conversion in -hls_segment_filename and fails with a misleading
"No such file or directory"
- Assert alignment on relative spacing rather than on how one ffmpeg
build clamps the priming packet's negative PTS
- Encode the test fixture with the full software GOP lock
(lockedGopCodecParams), not just appendLockedGopArgs
- Trim comments to the non-obvious rationale
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(engine): satisfy the ffprobe argv contract in packageHls test
`producer/src/utils/ffprobeArgvContract.test.ts` sweeps every package at
the source level and requires `--` to be the argv entry immediately before
an ffprobe input path. The `probe()` helper here passed
`join(outputDir, HLS_VIDEO_PLAYLIST)` in that slot, which the contract's
comma-splitting parser reads as two entries, so the producer suite was red
on this branch while the engine suite stayed green.
`probe()` now takes the input path as a parameter and appends the
terminator itself. The sibling ffmpeg `run()` helper switches from a
spread-in-array-literal to `.concat`, because a verbosity-flag-then-spread
literal is indistinguishable from the ffprobe wrapper shape the contract
demands a terminator on. The matcher reads raw source including comments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(engine): drop the audio-only variant from the HLS master playlist
With `a:0,agroup:aud` in `-var_stream_map`, ffmpeg lists the audio
rendition twice in `master.m3u8`: as the `#EXT-X-MEDIA:TYPE=AUDIO` group
the video variant references, and again as a standalone `#EXT-X-STREAM-INF`
variant with no `RESOLUTION`. That is valid HLS, but a player choosing by
bandwidth can select it and play sound with no picture, and the VOD
consumer asked for a single rendition.
`stripAudioOnlyVariants` removes any variant tag without a `RESOLUTION`
attribute together with its URI line; `packageHls` applies it to the
master after a successful audio+video run. The rendition group and the
audio playlist are untouched, so `audio.m3u8` is still referenced from the
master. Pinned by a unit test on the playlist text and by the real-ffmpeg
test, which now asserts exactly one `#EXT-X-STREAM-INF`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(producer): extract hdrMode warning and decision-log helpers
`resolveEffectiveHdrMode` was already over the fallow complexity gate on
`main` (21 cyclomatic / 25 cognitive / CRAP 462) and only surfaces now
because the HLS format work needs to touch its downgrade warning. Pull the
downgrade message into `formatDowngradeWarning` and the "which mode and
why" info line into `logHdrDecision`, which puts the function under the
threshold.
No behavior change: every log string is byte-identical and `hdrMode.test.ts`
asserts them unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(producer): add hls output format
Adds `format: "hls"` to `@hyperframes/producer`. An HLS render is the same
opaque H.264 + AAC encode as `mp4`, stream-copied at assemble time (no
re-encode) into a directory: `master.m3u8`, `video.m3u8` + `video_%05d.ts`,
and `audio.m3u8` + `audio_%05d.ts` when the composition has audio.
The load-bearing part is that `-c copy -f hls` can only cut a segment at an
existing keyframe, so the encoder's GOP is locked to
`hlsSegmentSeconds x fps` frames. `resolveHlsEncoderGopLock` returns that
pair as one value spread into both encoder inputs, so the streaming and
disk/chunked paths cannot disagree — a render reaching only one of them
still succeeds, it just emits variable-length segments silently.
- `renderFormat.ts`: `hls` is opaque (`outputNeedsAlpha` is now
`!== "mp4" && !== "hls"` — missing this flips HLS to RGBA screenshot
capture with nothing failing), supports page-side compositing, and
`outputUsesH264Pipeline` replaces the three `outputFormat === "mp4"`
router gates.
- `hlsConfig.ts`: `hlsSegmentSeconds` (default 4, positive integer), the
GOP derivation, and the fail-fast gates. `force-hdr` is rejected (v1 is
SDR-only; HDR10 needs HEVC in fMP4), and so is `useGpu` — the GOP lock is
software-only, so a GPU HLS render segments correctly by accident on
videotoolbox and wrong on nvenc.
- `assembleStage.ts`: `hls` branches to `packageHls` instead of
mux/faststart, after the same `padOrTrimAudioToVideoFrameCount` the mp4
mux uses.
- `artifactTransaction.ts`: a directory artifact can now carry a
`probeTarget` + `requiredFiles`, so the #3395 truncation gate still runs
for HLS against `video.m3u8`. `expectedFrames` is inert there (ffprobe
reports no frame count through the hls demuxer); duration carries it.
- `renderRequest.ts`: accepts `hls` + `hlsSegmentSeconds`; distributed
conversion rejects it alongside `gif`. `DistributedFormat` is unchanged.
`RENDER_REQUEST_VERSION` stays at 1: the new field is optional, the
parser ignores unknown option keys, and an older worker already rejects
`format: "hls"` with "Render request format is invalid".
- `hdrMode.ts`: the non-mp4 SDR downgrade warning no longer claims
"HDR + alpha" for `hls`; the helper extraction landed in the preceding
refactor commit.
Verified: `hlsRender.integration.test.ts` renders a real 10 s composition
and asserts 3 segments, the first two exactly 4.000 s, every segment
starting on an IDR frame, and PTS 0. Removing the GOP wiring collapses it
to 2 segments, so the test is load-bearing. Both producer lanes (unit and
integration) run on every PR in `ci.yml`, so that wiring check is on the
PR gate.
CLI flags and docs are deliberately left to the next PR in the series.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(engine): close the HLS master playlist rewrite race
CodeQL flagged two high-severity alerts on the `stripAudioOnlyVariants`
rewrite: a check-then-use race (`existsSync` then a separate path-based
read and write) and insecure creation of a file at a predictable path in
the OS temp dir.
Both came from resolving `master.m3u8` three times. Open it once with
`r+` and owner-only mode — which neither creates the playlist nor widens
its permissions, ffmpeg owns that — then read, truncate and write through
that one descriptor. Absence is still tolerated for the argument-level
tests that stub ffmpeg, now via ENOENT rather than a prior existence
check.
No behavior change: `hlsRender.integration.test.ts` still renders a real
composition with audio and asserts the stripped master playlist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deliverable one of issue #4010: average K sub-frame captures into each output
frame so rotation, scale and opacity blur the way After Effects does, which the
translation-only catalog component (#4008) cannot do. Opt in per render with
CaptureOptions.motionBlur; absent means the capture path is unchanged.
The blocker was quantizeTimeToFrame flooring every seek onto the output frame
grid, which collapsed all K samples onto one instant and made the averaged frame
byte-identical to the unblurred one. quantizeSeekTime now owns that decision at
all three quantize sites (runtime player, its no-timeline fallback, and the init
transport) and refines the grid by an integer subdivision count, which can only
contain the output grid, never move a frame-time instant off it. The finer grid
rounds rather than floors: a sample time round-trips through the page bridge as
a float, and past a few million ticks that error exceeds the floor epsilon.
Accumulation sits inside captureFrameCore AFTER the static-dedup early return,
so a static frame still costs one reused buffer and only moving frames pay K. It
is not a before-capture hook, which would have disabled dedup outright.
Callback invariant: one eventful seek per output frame at the frame time, then K
suppressed sample seeks, then a suppressed restore to the frame time. The
restore matters because a suppressed seek still moves the playhead, so without
it the next frame's eventful seek would swallow the callbacks inside the shutter
window.
Scope of this deliverable: screenshot capture mode and PNG output. beginframe,
drawelement and JPEG encoding are rejected with a named error rather than
silently rendering unblurred. Video content is rejected for the same reason it
is a documented limitation: injected frames are extracted per output frame and
cannot follow a sub-frame seek.
Blend defaults to sRGB, matching AE's non-linearised 8-bpc working space and
the catalog component. Linear light is an explicit opt-in.
Samples are folded into a running average as they are captured rather than held
until the end, so peak memory is one accumulator plus one decoded sample instead
of all K: at 1080p that is a constant 41 MB rather than 141 MB at 16 samples and
540 MB at the 64 maximum, on every frame of the render.
PNG chunk CRC32 now has one owner in utils/crc32.ts, shared by the new encoder
and ffprobe's metadata reader. The encoder previously imported zlib's crc32 by
name, which throws at module evaluation on Node 22.0 and 22.1; ffprobe already
carried the guarded version and the comment explaining why. The sRGB transfer
function is likewise exported once and shared.
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
## What this does
Video encoders normally choose their own keyframe positions. A keyframe is a frame that can be decoded on its own, and you can only cut a video cleanly at one. The engine already has a flag, `lockGopForChunkConcat`, that forces a keyframe every `gopSize` frames so chunk files can be joined with `ffmpeg -c copy`. Today that logic is copy-pasted in two places inside `buildEncoderArgs` and does not exist at all in the streaming encoder.
This PR:
1. **Extracts the locked-GOP logic into three small shared helpers** in `chunkEncoder.ts`: `resolveLockedGopSize` (validation), `appendLockedGopArgs` (the `-g` / `-keyint_min` / `-sc_threshold` / `-force_key_frames` args), and `lockedGopCodecParams` (the x264/x265 param fragment). `buildEncoderArgs` now calls them instead of inlining the same block twice.
2. **Adds the same flag to the streaming encoder** (`StreamingEncoderOptions.lockGopForChunkConcat` + `gopSize`), which is the path most renders under four minutes actually use. It emits the identical args as the disk encoder, including `-bf 0` for h265 under a lock so B-frame reordering cannot reintroduce negative DTS at a boundary.
## What does not change
- Any render that does not set the flag produces a byte-identical ffmpeg arg list. The flag defaults to `false` everywhere.
- Nothing in the tree sets the flag yet except the distributed chunk worker, which already used the disk encoder. No user-facing behavior changes.
- GPU encoders, VP9 and ProRes streaming encodes continue to ignore the flag, exactly as `EncoderOptions` documents. An earlier revision of this PR made GPU encoders honor it; that was dropped because the path is unreachable until HLS lands and was verified only on videotoolbox.
## Why
Groundwork for `format: "hls"` in `@hyperframes/producer`. HLS packaging stream-copies the encoded video into fixed-length segments with `-f hls -c copy`, and the segmenter can only cut on keyframes the encoder already produced. Locking the GOP at encode time is what makes every segment land exactly on a 4-second boundary without re-encoding. The follow-up PR adds the format itself and should reject `--gpu` for `hls` in v1.
## Testing
- `chunkEncoder.test.ts`: the three helpers tested directly; existing locked-GOP and GPU no-op tests unchanged.
- `streamingEncoder.test.ts`: new `lockGopForChunkConcat` block covering default off, libx264, libx265, and invalid `gopSize`.
- Engine typecheck, oxlint, oxfmt clean. Full engine suite matches main's baseline (the 4 `ffprobe.test.ts` HDR fixture failures are pre-existing on `main`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Add regression coverage for finite SDR video slots that start past EOF in mixed HDR timelines, and verify lint exclusions with successful local duration probes.
Co-authored-by: heygengenesis[bot] <262951085+heygengenesis[bot]@users.noreply.github.com>
Co-authored-by: miguel.sierra <229591595+miguel-heygen@users.noreply.github.com>
Allow explicit finite non-looping video slots to use the existing held-final-frame extraction path when data-media-start is at or beyond source duration. Warn during project lint for local static slots that will use this behavior.
Co-authored-by: heygengenesis[bot] <262951085+heygengenesis[bot]@users.noreply.github.com>
Co-authored-by: miguel.sierra <229591595+miguel-heygen@users.noreply.github.com>
The shared, size-capped video-frame extraction cache evicts entries whose
LRU-clock sentinel hasn't been touched in over an hour. That clock was set
once, at cache-hit lookup time, and never again — so a render holding a
compiled-dir symlink (or a still-open partial writer dir) into an entry for
longer than an hour could lose it to a concurrent render's GC sweep on the
same shared machine, leaving its frame reads a dangling ENOENT with no
retry.
Every active video's captured frame now renews its cache entry's clock,
throttled to once per 5 minutes per directory. touchCacheDir renews both
signals gcExtractionCache actually reads: the directory's own mtime (which
gates a still-open partial writer dir) and the completion sentinel's mtime
(which gates a published entry).
Co-authored-by: Miguel Angel <miguel.sierra@heygen.com>
Chromium's CDP layer can refuse a screenshot capture outright ("Protocol
error (Page.captureScreenshot): Unable to capture screenshot") with no
timeout wording at all. classifyCaptureFailure() didn't recognize the
message, so it fell through to the fatal "authoring" bucket and neither
capture-retry path (the disk-capture retry loop, or the streaming-capture
pinned-fallback retry) ever attempted a retry — including under
--low-memory-mode, whose single-worker plain-screenshot capture never
engages either path's other retry conditions.
Adds the literal error text to classifyCaptureFailure()'s transient_browser
bucket, and threads the resulting classification into
shouldRetryViaPinnedFallback() as a routing-independent retry condition, so
the streaming path recovers on a fresh session the same way the disk path
already does for other transient browser failures.
Co-authored-by: Miguel Angel <miguel.sierra@heygen.com>
Review feedback on #3805.
captureFrameToBufferPipelined checked isRecoverableDrawElementError and
threw DrawElementCaptureError before reaching captureFrameErrorDiagnostics,
so the NCPR/canvas failures that now abort the whole attempt produced no
frame-error PNG/HTML/JSON bundle — the exact case worth debugging, and the
one the adjacent comment promised mirrored the serial path. The serial path
was unaffected because its own DrawElementCaptureError throw propagates
through captureFrameCore's outer diagnostics catch.
Run diagnostics first, then the recoverable wrapper. Bounded to at most one
bundle per attempt, since a recoverable error fails the whole attempt, and
captureFrameErrorDiagnostics self-catches, so a dead page cannot mask the
structural error the producer's fresh-page retry depends on.
Covered by a new test asserting the bundle lands for a recoverable pipelined
failure; verified by mutation (restoring the old order fails it).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The static-frame dedup predictor walked tween intervals from
window.__timelines but was blind to onUpdate callbacks — motion driven
from a timeline's onUpdate in a tween-free window was predicted static.
The verifier compounded the gap by seeking with suppressEvents: true,
so the onUpdate never fired and the frozen frame passed verification.
Two changes:
1. Predictor: when a timeline carries vars.onUpdate, mark its full span
as animated so those frames are never predicted static.
2. Verifier: seek with suppressEvents: false so the verification page
behaves identically to the capture page. The verification page is
already isolated (separate Page instance), so out-of-order event
side effects cannot corrupt sequential capture.
Fixes#3793
* test(producer): pin unpinned drawElement stall retry
* fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing (#3172)
* fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing
A single drawElement frame could kill an entire render. On one comp
(caption-editorial-emphasis) drawElementImage returns normally and the renderer
then stops draining its task queue: the setTimeout(…, 0) that schedules
toDataURL never fires, the capture page.evaluate never settles, and 60s later
the stage watchdog fails the whole render. Deterministic on 152.0.7977.30,
always the same frame. Root-cause detail in PRINFRA-488.
Two gaps, both closed here.
1. Nothing below the 60s stage watchdog bounded a frame. Adds a per-frame
deadline (HF_DE_FRAME_TIMEOUT_MS, default 15000, 0 disables) around the WHOLE
frame operation, not just the drawElementImage call — the stall surfaces at
whichever page round-trip comes next, and was observed at both the capture
and the seek's background-image decode. Bounding one call missed it.
Deliberately NO per-frame screenshot fallback for this error: once the
renderer stops scheduling it is wedged for every subsequent round-trip on
that page. Measured — the screenshot fallback blew the same deadline. The
frame fails fast instead, so the recovery that actually works can run.
2. That recovery was ineligible. shouldRetryViaPinnedFallback only retried a
generic capture failure when the worker count was PINNED by the inversion or
the router; a comp that engaged drawElement on the ordinary single-worker
path had no whole-render fallback at all. A renderer stall is now retryable
on any routing — the failure is a property of drawElement itself, and the
retry re-renders on a fresh page via screenshot. Reported as
de_fallback_reason "de_renderer_stall" so it is distinguishable from
capture_error in telemetry, and counted per-session as deFrameTimeouts.
Verified end to end on the repro: previously exit 1 with "stalled: no frame
progress for 60000ms"; now the deadline fires, the render retries via
screenshot, and RENDER_OK with a valid 1920x1080 / 240-frame / 8.0s MP4.
Tests cover the predicate both ways plus the cross-package error match. Engine
1,481 and producer 587 green (audioPadTrim.integration flakes only under
full-suite parallel load — passes in isolation, and on clean HEAD it is green
too, so it is not from this change).
Refs PRINFRA-488
* fix(engine,producer,cli): count drawElement frame timeouts where they happen
`session.deFrameTimeouts` was incremented inside `captureFrameCore`'s catch,
but the deadline races that function from OUTSIDE it — `withFrameDeadline` is
applied by `captureFrameToBuffer`, and `DeFrameTimeoutError` is constructed
nowhere else. So the branch was unreachable: on a wedged renderer the counter
stayed 0, the engine-side stall log never printed, and
`CapturePerfSummary.deFrameTimeouts` reported 0 on every affected render. The
end-to-end recovery worked throughout — `isDeRendererStallError` matches by
name at the producer — only the observability was dead.
Moved the increment and the log into an `onTimeout` hook that fires from the
deadline itself, deleted the unreachable branch and the now-unused
`isDeFrameTimeoutError`, and threaded the count out to
`render_completed.de_frame_timeouts` so the rate is graphable apart from the
other `capture_error` fallbacks. A stall always costs a whole-render re-run,
which is why it deserves its own series.
Also corrected the `fallbackReason` docstring, which still listed only
`psnr | blank | oom | capture_error` after this branch added
`de_renderer_stall`.
`withFrameDeadline` is exported for a two-case fake-timer test: the deadline
fires and calls `onTimeout` once, and a resolving inner promise passes its
value through with no late timer. The first case fails if the counter moves
back inside the work promise.
(cherry picked from commit ec2b2b5b2a)
* fix(render): retry sequential screenshot stalls
* fix(render): bound parallel worker capture phases
---------
Co-authored-by: Vance Ingalls <vance@heygen.com>
* fix(producer): transport safe extraction failure metadata
* refactor(producer): generalize public error metadata
* test(producer): use vendor-neutral media hosts
The ENAMETOOLONG regression writes 150 real clip files and the mixer
existence-checks each one: ~58ms on Linux, but past vitest's 5s default on
the Windows lane. packages/engine sets no global testTimeout, so heavy
tests here carry an explicit one.
On timeout its abandoned async work kept calling the shared runFfmpegMock
after afterEach cleared it, so the next test saw 5 calls instead of 3 and
lost its queued once-implementations to the leak. mockReset stops an
aborted test from handing leftovers to the next one.
When a composition script throws during execution, the GSAP timeline
registration never arrives and pollSubCompositionTimelines times out.
Previously the render continued with a degenerate 2-frame output and
reported success — now it fails loudly.
Two changes:
1. Detect composition script runtime errors in the browser console
handler and feed them into scriptLoadFailures, triggering the
existing fail-fast path (same as script load 404s).
2. Make sub_timeline_script_failure a fatal warning in
applyRenderWarningPolicy, alongside audio_processing_failed.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`muxVideoWithAudio` passed `-avoid_negative_ts make_zero` unless the caller
set `preserveAudioPrimingEditList`. In practice the dominant path is an AAC
sidecar copied into mp4, where that flag is actively harmful: ffmpeg's
default is `auto`, which the mp4/mov muxers (AVFMT_TS_NEGATIVE) already
resolve to `disabled`. Forcing `make_zero` overrides the correct default,
discards the priming edit list the sidecar encode created, shifts the video
start_time forward by one AAC frame and writes an empty video edit at t=0 —
which edit-list-honoring players (QuickTime/Safari) render as a black first
frame.
Verified with ffprobe on a copy mux of a 30fps h264 mp4 and an AAC sidecar:
with `make_zero` video start_time 0.066000, elst: [media time -1,
dur 5940] + [media time 6000, dur 180000]
audio start_time 0.042993, elst: [media time -1, ...]
without (this fix) video start_time 0.000000, elst: [media time 6000,
dur 180000]
audio start_time 0.000000, elst: [media time 1024, ...]
The empty leading edit and the offset both disappear, and the audio keeps
its 1024-sample priming edit.
The flag is now never passed for a mux, in any mode. `preserveAudioPrimingEditList`
is part of the exported engine API, so it stays on `MuxVideoWithAudioOptions`
as `@deprecated` and no-op rather than being removed; the two internal callers
that set it (`assembleStage`, distributed `assemble`) drop it.
`buildEncoderArgs` and `streamingEncoder` still pass the flag for video-only
output and are deliberately left alone — those chunks are consumed as
intermediates, not as a delivered mp4/mov.
Fixes#3487
Co-authored-by: Alexandru Mincu <alex@mountsoftware.ro>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parent src-only scans skipped multi-format <video>/<audio> markup, so those
elements were never extracted, downloaded, or mixed and rendered blank/silent.
Lint now accepts a child <source src> as a resolvable media src.
lookupCacheEntry reported a hit purely on the presence of the
.hf-complete sentinel. The sentinel records that extraction finished,
not that the frames survived, so any per-file cleanup that empties the
directory leaves an entry that rehydrates with zero frames.
rehydrateCacheEntry then returns totalFrames: 0, the clip reaches the
coverage gate with nothing, and the render aborts with a message about
capture coverage. Because the poison is on disk rather than in the
composition, every later render of the project fails the same way with
nothing the user can change to fix it.
An entry now counts as a hit only when it carries the sentinel AND
still holds at least one frame file, so an emptied entry re-extracts.
The check is format-agnostic: a hit must be usable whatever extension
the frames carry.
Addresses the cache half of #3372.
compileTimingAttrs/injectDurations used parseFloat, so data-start="intro"
wrote a NaN data-end and extract preferred that over duration; parseNumeric
now skips the id-ref (parseVideoElements already resolves it).
collectRenderMedia's resolveHostWindow likewise read host data-start with
parseFloat, so chained sub-composition slots (data-start="hook") stacked at
0-2s and every scene after the first rendered black. It now resolves host
starts through the shared resolveReferencedStart, matching the media parsers.
Fixes#3361.
* fix(producer): fall back to screenshot capture on drawElement canvas-not-initialized
The fast-capture drawElement path only special-cased the "No cached
paint record" error to trigger a per-frame screenshot fallback; every
other error (including "drawElement canvas not initialized", seen at
frame 0 on some macOS/Chrome combinations) was rethrown, hard-failing
the whole render even though the docs promise automatic fallback on
incompatible compositions.
Extend the existing fallback branch (in both captureFrameCore and
captureFrameToBufferPipelined) to also catch canvas-not-initialized
errors via a shared isRecoverableDrawElementError predicate, with a
diagnostic message identifying which case triggered the fallback.
Closes#3423
Co-Authored-By: Miga <noreply@anthropic.com>
* fix(producer): address review — tighten error matching, audit batch path, add fallback-ratio guard
* fix(engine): add prepareFrameForCapture to batch screenshot fallback loop
* fix(engine): split canvas-not-initialized from composition-root-missing errors
drawElementService threw the same HF_DE_CANVAS_NOT_INITIALIZED error for
both !canvas and !root. Missing composition root (navigated/broken page)
was classified recoverable and fell back to pageScreenshotCapture, which
captured blank or wrong content silently.
Now:
- !root → HF_DE_COMPOSITION_ROOT_MISSING (not recoverable, hard fail)
- !canvas → HF_DE_CANVAS_NOT_INITIALIZED (recoverable, screenshot fallback)
Split applied at all 3 emit sites (serial, pipelined, batch).
Co-Authored-By: miga-heygen <miguel.sierra_miga@heygen.com>
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Co-authored-by: Miga <noreply@anthropic.com>
Root cause: the per-worker capture calls in captureFrameRange
(parallelCoordinator.ts) take no abort signal of their own, and only
checked `signal.aborted` BEFORE starting each frame — a no-op once a
worker is already awaiting an in-flight call. On WSL2, the native
drawElement/BeginFrame capture call can hang indefinitely at frame 0
with no error. The DE parallel-router's existing stall watchdog
(captureStreamingStage.ts) correctly fires `stallController.abort()`
after HF_DE_STALL_MS, but that abort had no way to reach a
worker already wedged inside a hung capture call — so
executeParallelCapture's Promise.all waited forever, the render hung
indefinitely, and the CLI's circuit breaker (which only runs after
executeRenderJob settles) never got a chance to trip.
Fix: race each per-frame capture call against the signal actually
firing (raceAgainstAbort), the same "can't cancel, only race" pattern
already used by the sequential capture path. Once the watchdog's abort
is observed, the wedged worker rejects, executeParallelCapture settles,
and the existing pinned-fallback retry / "reverted" outcome / circuit
breaker machinery (already correct) runs end to end.
Also widen the CLI breaker's trip condition from the literal string
"reverted" to "not a clean routed success", so any future non-success
outcome the observability layer records also latches the breaker
instead of silently falling through.
Closes#3441
Co-authored-by: Miga <noreply@anthropic.com>
* fix(core): harden audio FX and group identity
* fix(core): address audio group review feedback
* fix(core): align preview transport with grouped audio
* test(core): pin audio group gain ceiling
* fix(core): preserve solo bridge through stack
* fix(engine): harden grouped audio rendering
* docs(engine): explain grouped mix fallback invariant
* test(engine): allow grouped mixes to finish on Windows
## What
Adds a one-shot ffmpeg-psnr filter probe at drawElement session bootstrap.
When the resident ffmpeg is missing or lacks libpostproc (no `psnr` filter),
the capture-session router now force-fallbacks to the screenshot capture
path and emits a `de_gate_reason = "ffmpeg_no_psnr_filter"` telemetry
signal via the existing `render_complete` breakdown.
Also tightens `psnrForDiskSample`'s catch: infrastructure-class ffmpeg
failures (ENOENT, "No such filter") no longer silently skip the sample —
they abort the render so the safety net cannot fail-open post-preflight.
## Why
The drawElement self-verify safety net (parallelCoordinator's
`psnrForDiskSample` → `psnrDb`) shells to `ffmpeg -lavfi psnr`. If ffmpeg
is missing, or was compiled without libpostproc (so the `psnr` filter is
absent), every per-sample compare throws. The existing catch swallows the
error and returns `null` — callers treat that as "skip this sample" and
the render completes with the safety net inoperative.
Field signal (⭐ 9/10 CLI feedback, Slack ts=1787380767.210079,
hyperframes 0.8.7, darwin/arm64, tid=93ff9910-2207-45c2-bc1f-54c0b347d4fe):
> "host ffmpeg lacked psnr filter used by drawElement self-verification,
> but render completed."
The user's frames happened to be byte-identical so no visual damage
shipped — but the safety net silently wasn't running. Any future
compositor-damage bug on that host would have shipped straight through.
## How
Two-part fix, both in `packages/engine`:
1. New `utils/psnrFilterAvailability.ts` — cached probe that runs
`ffmpeg -hide_banner -filters` once per process and word-boundary-
matches `psnr` in the output. Any failure (ENOENT, non-zero exit,
timeout, unparseable output) returns `false`; never rejects.
2. Wired into `services/frameCapture.ts` `initDrawElementOrTransparentBackground`
right after the Chrome capability probe: when useDrawElement resolves
true and the preflight returns false, set
`session.deGateReason = "ffmpeg_no_psnr_filter"` (same low-cardinality
bucket every other DE gate uses; flows through `getCapturePerfSummary`
→ `render_complete.de_gate_reason` in PostHog), emit a stderr warning
naming what's missing, and call `routeToFallback()` — the same
fail-graceful shape as the SwiftShader / CSS-effect / at-risk-timeline
gates. Skipped under `HF_FORCE_DRAWELEMENT=1` (matches the diagnostic
knob's policy of bypassing every other gate).
Belt-and-braces: `psnrForDiskSample` now discriminates infrastructure-
class failures (ENOENT / "No such filter" / "Unknown filter") from
per-sample noise (readFile races, transient EPERM). Only the former
re-throw — per-sample noise still returns `null` (skipped sample). The
preflight normally catches this at bootstrap; the re-throw covers
ffmpeg-swapped-mid-render.
## Test plan
- [x] Unit tests added:
`packages/engine/src/utils/psnrFilterAvailability.test.ts` — mocked
`execFile` covers: `psnr` present → true; `psnr` absent → false; ENOENT
→ false; non-zero exit → false; result memoized + reset works;
substring-not-word-boundary → false.
- [x] Unit tests added:
`isFfmpegInfrastructureFailure` in
`packages/engine/src/services/parallelCoordinator.test.ts` covers
ENOENT, "No such filter", "Unknown filter", per-sample EACCES, parse
errors, null/non-object.
- [x] `bun run test` — `packages/engine/src/utils/psnrFilterAvailability.test.ts`
(6 tests) + `packages/engine/src/services/parallelCoordinator.test.ts`
(50 tests) + `frameCapture.test.ts` (26 tests) all pass. Pre-existing
ffprobe test failures (4) on the base commit are unrelated (missing PNG
fixture bytes — the file is 129 B on disk, likely LFS-stored).
- [x] `bunx tsc --noEmit -p packages/engine/tsconfig.json` — clean.
- [x] `bunx oxlint <files>` — 0 warnings, 0 errors.
- [x] `bunx oxfmt --check <files>` — clean.
Not covered here: an integration test that boots
`initDrawElementOrTransparentBackground` end-to-end. That path is
Puppeteer-driven and has no unit-scale bootstrap harness in the
repository — the pure preflight + pure discriminator coverage above are
what this PR can prove at the vitest layer.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
`places a delayed track on its authored start` timed out on the windows runner
and failed an unrelated PR, the second time this week an ffmpeg audio test has
done that.
The previous fix raised the budget in `audioMixer.grouping.test.ts`, which was
the file the symptom named. It was the wrong scope: that file was the only audio
suite with explicit timeouts at all. `audioMixer.level.test.ts` had none, so its
two real-ffmpeg tests ran on vitest's 5s default. The failing one takes ~137ms
locally, so the runner is not 36x slower — but 5s was never a budget anyone
chose for a full mix.
Applied to the suite rather than to each test, so there is one home for it, and
scoped to the ffmpeg-gated describe: the sibling parsing suites are pure and
should keep failing fast at 5s.
Headroom alone would only have delayed an undiagnosable failure. The ffmpeg
process timeout is 5 minutes by default, far above any test budget, so a stalled
mix could only ever surface as a bare "Test timed out" with no stderr and no
failing stage. Tests now cap it at 20s and assert through a helper that reports
`failures` instead of collapsing to `expected false to be true`.
Both claims verified rather than asserted: a deliberately 6s test now passes
where the 5s default would have killed it, and forcing the process timeout to
1ms reports `stage: "prepare", reason: "ffmpeg_timeout"` instead of a timeout.
Reviewing with whitespace ignored is much smaller: adding the third argument to
`describe` reindents the suite body, so 131/101 is really 34/4.
`a group FX chain fully cutting its members leaves an ungrouped track
untouched` timed out on the windows runner, failing an unrelated PR. The whole
file runs in ~4s locally and that test in ~1.1s, so 30s was not generous — but
the runner is roughly 10x slower and this test drives more ffmpeg than any of
its siblings, two full mixes plus a group FX chain. 30s was the tightest budget
in the package; 60s is what the rest of the ffmpeg-driven engine tests use.
Headroom alone would only have moved the same undiagnosable failure later,
because nothing here could report why. The production ffmpeg process timeout is
5 minutes, far above any test budget, so a stalled mix could only ever surface
as "Test timed out in 30000ms" with no stderr and no failing stage. Tests now
cap it at 20s, and the mix wrapper throws the recorded failures instead of
returning `success: false` into an `expect(...).toBe(true)` that reports
`expected false to be true` and discards the reason.
Verified by forcing the process timeout to 1ms: the failure goes from a 30s
wall-clock timeout to a 150ms error naming the stage, reason and element
(`stage: "prepare", reason: "ffmpeg_timeout", elementId: "a"`).
This does not explain the Windows stall itself, which I could not reproduce on
macOS. It makes the next occurrence report what it was doing.
audioMixer.grouping.test.ts spawns real ffmpeg per assertion and ran on
vitest's 5s default; on slow Windows runners the FX-chain and envelope
cases land right at the line and fail runs that touch nothing in the
engine. The other ffmpeg-bound engine suites (videoFrameExtractor)
already carry a per-test 30_000 timeout; this brings the grouping suite
in line.
B5: mute and solo, on groups and tracks (track mute already shipped by A2 —
nothing to build there).
Group mute — persisted as data-hidden on the <hf-audio-group> element itself
(never written onto members, per design doc §2.1's state-restoration
warning). Studio action reuses B7's generic setAudioGroupAttribute
(setQuiet/setLive split) rather than duplicating toggleTimelineTrackHidden's
shape — same one-atomic-patch/one-undo-entry contract, already built for
exactly this purpose. Render: B4 already drops every member of a
data-hidden group (confirmed by a new audioMixer.test.ts case — no
production change needed there). Preview: a dedicated muteGain node
(groupInput -> [fx] -> muteGain -> output -> master) so a mute toggle
never fights scheduleVolumeLane's ramps on the same param — the same
hazard B7's volume fader was split out to avoid. Mid-playback toggles
sync via a new syncAudioGroupMute pass in init.ts (a group carries no
data-start, so it's invisible to the existing visibility-node query).
Members of a muted group render the strikethrough label treatment
(TimelineTrackPlainHeader's isGroupMuted, sourced from
TimelineElement.audioGroupHidden) — display only, no attribute touched.
Solo — "Hear only this": a new session-only store slice (audioSoloSlice,
soloed: ReadonlySet<string> of clip/group ids, never track numbers, never
serialized). Predicate (isAudibleUnderSolo, packages/core/src/audioGroups.ts
so both the store and the preview transport share one definition): an
element is audible while any solo is active only if it or its own group is
soloed. "Siblings, never ancestors" lives in the graph, not the predicate —
solo gain is a per-element stage only; group buses are never attenuated by
solo, so a soloed member's path through its group stays open by
construction. Preview: a dedicated per-element soloGain in
webAudioTransport.ts (parallel to the mute mechanics), pushed via
window.__hf.setAudioSolo — a direct call, not an attribute write, so it
can't ride the visibility-diff path mute uses. media.ts's HTMLMedia
fallback folds the same predicate into its per-tick volume computation
(the same seam A2 used for data-hidden). Half-lit group indicator
(isGroupHalfLitUnderSolo) for "not soloed itself, but a member is".
Exclusive-by-default toggle, ⌘/Ctrl-click to add/remove, TimelineSoloButton
(⌗) beside mute on both track and group headers. Transport-bar banner
("Hearing only <label> — your export is not affected", Clear button) added
in PlayerControls.tsx, reading labels straight off the live preview DOM.
Export-safety, the most important property here: toggling/adding/clearing
solo never calls setAttribute/removeAttribute on any element and never
invokes the project save path (both asserted directly via spies in
audioSoloSlice.test.ts) — solo cannot reach an export by construction, not
by convention.
Also: extracted useHydrateActiveCompPathFromUrl out of App.tsx (a
pre-existing, unrelated effect) to stay under the 600-line filesize cap
after wiring useAudioSoloBridge in; and fixed a circular dependency the
solo-banner wiring introduced (useAudioSoloBridge.ts now imports
usePlayerStore from its concrete module instead of the player/ barrel,
which re-exports PlayerControls.tsx — the barrel path is what closed the
cycle).
Gates: bun run build clean; packages/core full suite 2379/2379; packages/
studio full suite 4276/4294 (18 pre-existing todo); packages/engine
audioMixer.grouping.test.ts 5/5; oxfmt/oxlint clean on all 23 touched
files; fallow clean (0 new circular deps, 0 new filesize/complexity
findings).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(core): route grouped audio through a group bus in preview
An audio element carrying `data-audio-group` no longer lands its gain on
the master bus directly — it feeds a per-group `GainNode` (built lazily on
first use, one per group id) which itself feeds master, so members of the
same group sum before the ear, ready for a group-level FX chain and
volume/mute in later steps. An id with no matching `<hf-audio-group>`
element still gets a plain, unprocessed bus rather than losing the track.
The group's own chain and volume lane are wired through the same
`attachElementFxChain`/`scheduleVolumeLane` every element already uses,
against the group's clock — composition time (design doc §1.3), since a
group has no `data-start` and a missing one parses as 0. The bus persists
across `stopAll()` (mirroring `_masterGain`'s own lifecycle) so replaying a
group does not rebuild its chain; only `destroy()` disposes it.
Render is untouched — stays flat until B4; `audio-groups` is still a 0%
canary so nothing ships this to a real composition without hand-authoring
`data-audio-group`.
Also: `audioGroupOf` (B1) crashed on any element lacking a real `tagName`/
`getAttribute` — exactly the shape of most `HTMLMediaElement` test doubles
in this suite, including this file's own `mockEl`. Made it tolerant, same
style as `readChain`'s existing guard in `runtime/audioFx.ts`.
`schedulePlayback` was already 110 lines pre-existing before this diff;
extracted `resolveDestination` and `handleSourceEnded` to shrink it to 92,
then suppressed the remainder (inherently sequential graph wiring, not a
decision tree) per the same precedent B2 used on `TimelineLogicalRow`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(studio,lint): carve targets voiceover groups — always, when plural
Plural voiceover carve now targets a group instead of naming each clip:
`resolveCarveSourceIds` (core `audioGroups.ts`) expands a group id to its
current members at analysis time, so a clip added to the group later is
covered without touching `sources`. The picker (`useFxCarve.ts`) offers a
grouped voice as one option instead of one row per member, tests overlap
as a union of member spans (a group overlaps the bed if ANY member does),
and prefers a qualifying group over its individual members in
`autoSourceIds`.
Picking two or more ungrouped voice clips in the carve flow now mints a
group behind them (`mintGroupId`, de-duped against every id in the
document) and writes `data-audio-group` on each picked clip atomically,
one undo entry — `createAudioGroupAndAssignMembers` in
`timelineTrackVisibility.ts` copies `setElementsHidden`'s multi-target
write shape. The DSP is untouched: `mixCarveSources` already sums
multiple sources correctly (verified in the design doc's own
investigation) — this only fixes the picker.
New lint rule `audio_carve_ungrouped_sources` (`packages/lint/src/rules/
media.ts`, alongside `audio_volume_double_automation`) warns when a
`data-fx-carve`'s `sources` names two or more plain clip ids instead of a
group — the shape that silently rots when a clip is added. `/hyperframes-
audio` states the same rule as an invariant, not a tip, with the grouped-
narration HTML example from the design doc.
The group-matching and auto-group logic (`withAutoGroupedSources`,
`collectCarveCandidates`) is split into `useFxCarveGrouping.ts` —
`useFxCarve.ts` was pushing past the 600-line cap. `resolveNextCarveSettings`
is deliberately NOT an `async function`: wrapping it in one would force a
microtask on every call, including the synchronous branch — the exact bug
`withAutoGroupedSources`'s own sync-when-possible contract exists to avoid,
and one caught via `propertyPanelAudioFxGroup.test.tsx` (10 failures)
before fixing it back to a plain function the caller conditionally awaits.
Also extracted `useEffectiveTimelineDuration` out of `App.tsx` and
`useRemoveBackground` out of `StudioRightPanel.tsx` (both pushed past 600
lines from an added prop wire), and decomposed `useFxCarve.ts`'s picker
IIFE to clear fallow's complexity gate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(engine): render grouped audio through a summed, FX-processed bus
Renders what B3 already routes in preview: a group's members sub-mix into
one PCM WAV at full composition length (adelay already places each member
at its composition position, so the group WAV's t=0 IS composition time),
run through the group's own FX chain and automation via the same
applyAudioFxChain/envelope-bake path a member uses, then fold into the flat
track list as one processed AudioTrack — the final mixAudioTracks call
never has to know groups exist.
Gain law verified against plans/spikes/amix-nesting-spike.sh (brought over
from the plans branch, along with audioMixer.grouping.test.ts, since both
were committed there and never merged to origin/main — every step branch in
this stack descends from origin/main): the sub-mix's own amix prefers
normalize=0 (nulls exactly against a flat mix), falling back to per-node
compensation by the group's OWN member count only when this ffmpeg build's
amix rejects the option. Carrying any other count into a nested amix node
is the exact +2.499 dB silent failure the spike measured — confirmed by a
manual mutation check (wrong-count compensation landed 3.5 dB hot, exactly
20*log10(3/2) for a 2-member group compensated as 3; reverted after
confirming the level test catches it).
A group element carrying data-hidden drops every member before the sub-mix
ever runs (RULES: mute-by-drop, never mute-by-volume-0) — parseAudioElements
now resolves groups once per parse and skips hidden-group members the same
way it already skips data-hidden ancestors.
HfAudioGroup (packages/core/src/audioGroups.ts, from B1) gains fxChain,
automation, volume and hidden, read off the group element the same way
resolveAudioGroups already reads data-label — audioGroups.test.ts updated
for the wider shape plus new coverage for the added reads.
it.todo("mixes a grouped composition at the same level as the ungrouped
one") is now a real, passing test; two more added per the step doc (FX
routing isolation, member-level envelope survives grouping).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>