* feat(engine): adaptive per-frame sample count for motion blur
Measured against the accuracy bar set in #4010 (72.4dB mean PSNR vs the
960fps shutter integral), the fixed default of 16 samples misses it for
hard-edge content while paying nothing extra for static/slow content is
still important. This picks a sample count per frame from measured
motion, floor 16 ceiling 64 (issue #4029).
Two probe captures at the true shutter-window edges (never a sample that
enters the average) measure motion magnitude, mapped through a monotone
step table to a sample count. Frames confirmed to move only non-spatial
properties (opacity, colour) skip the probe and use the floor directly,
since a fade's shutter integral is a smooth blend that does not need many
samples. MotionBlurPlan's window (from shutter angle/phase alone) is now
independent of how many samples fill it, which is what lets the count
vary per frame without moving the window; motionBlurWindowIsStatic is
bounded by the widest sample inset any chosen K could produce so the
static-dedup safety guarantee holds for every K in [16, 64].
* refactor(engine): simplify the adaptive motion-blur sample selection
Two findings from an independent code-reuse/quality/efficiency review of
the previous commit:
- armStaticDedup ran the GSAP-timeline walk (computeStaticFrameSet)
twice on the same page when adaptive motion blur and static dedup
both apply, once for each purpose. Shares the one result instead.
- The non-spatial floor path read the sample count from
adaptiveSampleCount(0), which only worked because the step table
happens to be sorted with the first bucket covering zero. Reads
the now-exported DEFAULT_SAMPLES_PER_FRAME directly instead.
* fix(engine): read raw options for the adaptive motion-blur gate
Independent review caught it: every armStaticDedup call site runs
before finalizeSessionInit resolves session.motionBlur, so the
adaptive-classification gate's `session.motionBlur &&
session.motionBlur.fixedSamplesPerFrame === null` was always false at
runtime. The non-spatial probe-skip described in the PR never engaged
outside tests that hand-set motionBlurNonSpatialFrames directly,
bypassing real init order.
Reads session.options.motionBlur (set unconditionally at session
creation) instead, which carries the same "adaptive requested"
information the resolved plan would have. Adds a source-order lock
test plus direct coverage of computeStaticFrameSet's
nonSpatialOnlyFrameSet, mutation-proved against the exact regression.
* fix(engine): add xPercent/yPercent to the spatial-property list
Round 2 of the independent review: the spatial-property list deciding
whether a frame's motion needs the adaptive probe omitted GSAP's
percentage-based position properties (xPercent/yPercent) and two 3D
transform properties (perspective/transformPerspective) that two other
lists already in this repo (packages/parsers/src/gsapConstants.ts,
packages/studio/.../gsapAnimatesProperty.ts) both carry. A composition
positioned via xPercent/yPercent would have been silently floored to
K=16 with no probe and no log line, the same failure shape as the
round-1 bug through a data gap instead of a dead gate.
Moves the list to an exported SPATIAL_TWEEN_PROPERTIES array in
motionBlur.ts, passed into computeStaticFrameSet's page.evaluate as
data rather than duplicated inline, so it stays a single list and is
directly testable. Adds tests that execute the real page-side closure
(the existing tests all mocked the whole result and never exercised
the matching code), mutation-proved against the exact gap found.
* fix(engine): add rotate to the spatial-property list
Round 3 of the independent review, comparing SPATIAL_TWEEN_PROPERTIES
side by side against the two sibling lists rather than trusting the
prior commit's summary: GSAP's individual "rotate" transform property
(distinct from "rotation", carried in
packages/studio/.../gsapAnimatesProperty.ts's GSAP_TRANSFORM_PROPS)
was still missing, same failure shape as the two prior rounds.
The same pass also raised transformOrigin, which is deliberately NOT
added: gsapConstants.ts's own classifyTweenPropertyGroup excludes it
as a pivot-point modifier with no independent visual effect, and a
test now locks that reasoning in rather than leaving it to be
"fixed" again by a future pass that does not read that far.
The static-frame dedup reuses the previous frame's buffer when a frame is
byte-identical to its predecessor. With motion blur on that test is not
enough: the shutter window reads content either side of the frame instant,
so a still frame next to a moving one reused a buffer that carries none of
that motion. At 30 fps and 16 samples frame 11's window runs 10.75 to 11.25
at a 180 degree shutter and 10.06 to 11.94 at 720, so both reach past the
frame instant and the default is affected too.
motionBlurWindowIsStatic now owns whether a blurred frame may reuse its
predecessor's buffer. It requires every frame the window reads from to be
static, and covers the previous frame's window as well, because reuse claims
the two blurred outputs are equal. The range is derived from the plan's own
sample tick offsets, so it cannot drift from where the samples land.
The guard's contract comes from #4013 by Dante-dan, which carried it before
we did. One existing test fixture is widened: it asserted reuse with only the
frame itself marked static, which the guard now correctly rejects.
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* feat(producer): make sub-frame motion blur reachable from a render
Second deliverable for issue #4010. The engine gained the accumulation path, but
nothing set CaptureOptions.motionBlur, so the feature could not be switched on
from a real render. RenderConfig now carries it and the producer threads it into
the capture options.
Doing that surfaced the defect that would have made it fail silently. "Does this
render capture PNG frames" was spelled four times, every one of them re-derived
from needsAlpha, which describes the OUTPUT container rather than the capture:
probeStage picked the probe capture format
renderOrchestrator picked the render capture format
encodeStage built the encoder's frame_%06d input pattern, twice
captureStage built the disk worker's frame extension
Motion blur is a second, independent reason to capture PNG: the sub-frame samples
are averaged pixel by pixel, so JPEG samples would be averaged after a lossy
quantization. Add that reason to only one of those four and the encoder goes
looking for frame_%06d.jpg against files written as .png, and the render finds no
frames at all rather than reporting an error.
So the rule now has one owner, resolveCaptureImageFormat, which names both
reasons. Everyone downstream reads the resolved capture format instead of
re-deriving it: the encoder and the disk worker share frameFileExtension with
writeCapturedFrame, which is what actually names the files. needsAlpha keeps only
the question it really answers, whether the output preserves alpha.
captureStage no longer needs needsAlpha at all, which is the tell that the
re-derivation was its only reason to know.
No JPEG encoder is involved. The PNG capture path is already load-bearing, since
every transparent-output render uses it and the streaming encoder already selects
png as its input codec.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* fix(engine): resolve the motion-blur plan on every initialization exit
A real render produced no blur at all. The plan was correct, the producer
wiring was correct, and the sub-frame sample times were genuinely distinct, but
K=16 and K=64 rendered identically to no blur because neither ever ran.
initializeSession has two exits: screenshot mode finishes its own block and
returns, every other capture mode falls through to the end. The plan was
resolved at the fall-through exit only, and screenshot is the one capture mode
motion blur supports, so on exactly the path the feature runs on
session.motionBlur was never assigned. captureFrameCore then read it as
undefined and took the single-capture branch: one capture per output frame, no
accumulation, no error.
finalizeSessionInit is now the single owner of what finishing initialization
means. It sets the plan and isInitialized together and is called at both exits,
so a third exit cannot mark a session ready while forgetting the plan.
Why the existing tests passed, which is the part worth keeping: the accumulation
test set session.motionBlur directly on a stub session. It exercised the
mechanism while bypassing the only code that decides whether the mechanism runs
at all. A test that proves something works and never asks whether it is
reachable is not end to end, however much behaviour it drives.
Two tests. A structural guard that session.isInitialized = true appears exactly
once and that the same place resolves the plan, which fails if the defect is
reintroduced at either exit. And a sample-distinctness assertion that hashes the
K buffers captured within one output frame and requires K distinct ones, which
covers the different failure of every sub-frame seek landing on one instant.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* feat(engine): add packageHls for HLS VOD packaging
Adds `packageHls` next to `muxVideoWithAudio`: stream-copies an
already-encoded H.264 video and optional AAC sidecar into an HLS VOD
directory (`master.m3u8` + split video/audio renditions, MPEG-TS
segments) with `-c copy`, so no re-encode happens at package time.
Depends on the locked GOP from the previous commit. `-hls_time` can only
cut at existing keyframes, so `gopSize = segmentSeconds x fps` is what
makes every segment exactly `segmentSeconds` long and IDR-aligned. A
test packages an unlocked encode to pin that: the same call yields one
4s segment instead of four 1s ones.
Deliberately omits the mp4 mux's provenance tags, which MPEG-TS drops
and whose `-movflags` is invalid for `-f hls`, and `-avoid_negative_ts`,
which would discard the AAC priming and shift audio against video (see
issue 3487). A real-ffmpeg test asserts the first presentable audio
sample lands on the first video frame.
No caller yet; `format: "hls"` in the producer follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(engine): zero HLS mux base and reject % in packageHls outputDir
- Pass -muxdelay 0 -muxpreload 0 so the first segment starts at PTS 0
instead of the mpegts muxer's 1.4 s base; A/V alignment through the
AAC priming packet is unchanged
- Throw on an outputDir containing %, which ffmpeg expands as a printf
conversion in -hls_segment_filename and fails with a misleading
"No such file or directory"
- Assert alignment on relative spacing rather than on how one ffmpeg
build clamps the priming packet's negative PTS
- Encode the test fixture with the full software GOP lock
(lockedGopCodecParams), not just appendLockedGopArgs
- Trim comments to the non-obvious rationale
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(engine): satisfy the ffprobe argv contract in packageHls test
`producer/src/utils/ffprobeArgvContract.test.ts` sweeps every package at
the source level and requires `--` to be the argv entry immediately before
an ffprobe input path. The `probe()` helper here passed
`join(outputDir, HLS_VIDEO_PLAYLIST)` in that slot, which the contract's
comma-splitting parser reads as two entries, so the producer suite was red
on this branch while the engine suite stayed green.
`probe()` now takes the input path as a parameter and appends the
terminator itself. The sibling ffmpeg `run()` helper switches from a
spread-in-array-literal to `.concat`, because a verbosity-flag-then-spread
literal is indistinguishable from the ffprobe wrapper shape the contract
demands a terminator on. The matcher reads raw source including comments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(engine): drop the audio-only variant from the HLS master playlist
With `a:0,agroup:aud` in `-var_stream_map`, ffmpeg lists the audio
rendition twice in `master.m3u8`: as the `#EXT-X-MEDIA:TYPE=AUDIO` group
the video variant references, and again as a standalone `#EXT-X-STREAM-INF`
variant with no `RESOLUTION`. That is valid HLS, but a player choosing by
bandwidth can select it and play sound with no picture, and the VOD
consumer asked for a single rendition.
`stripAudioOnlyVariants` removes any variant tag without a `RESOLUTION`
attribute together with its URI line; `packageHls` applies it to the
master after a successful audio+video run. The rendition group and the
audio playlist are untouched, so `audio.m3u8` is still referenced from the
master. Pinned by a unit test on the playlist text and by the real-ffmpeg
test, which now asserts exactly one `#EXT-X-STREAM-INF`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(producer): extract hdrMode warning and decision-log helpers
`resolveEffectiveHdrMode` was already over the fallow complexity gate on
`main` (21 cyclomatic / 25 cognitive / CRAP 462) and only surfaces now
because the HLS format work needs to touch its downgrade warning. Pull the
downgrade message into `formatDowngradeWarning` and the "which mode and
why" info line into `logHdrDecision`, which puts the function under the
threshold.
No behavior change: every log string is byte-identical and `hdrMode.test.ts`
asserts them unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(producer): add hls output format
Adds `format: "hls"` to `@hyperframes/producer`. An HLS render is the same
opaque H.264 + AAC encode as `mp4`, stream-copied at assemble time (no
re-encode) into a directory: `master.m3u8`, `video.m3u8` + `video_%05d.ts`,
and `audio.m3u8` + `audio_%05d.ts` when the composition has audio.
The load-bearing part is that `-c copy -f hls` can only cut a segment at an
existing keyframe, so the encoder's GOP is locked to
`hlsSegmentSeconds x fps` frames. `resolveHlsEncoderGopLock` returns that
pair as one value spread into both encoder inputs, so the streaming and
disk/chunked paths cannot disagree — a render reaching only one of them
still succeeds, it just emits variable-length segments silently.
- `renderFormat.ts`: `hls` is opaque (`outputNeedsAlpha` is now
`!== "mp4" && !== "hls"` — missing this flips HLS to RGBA screenshot
capture with nothing failing), supports page-side compositing, and
`outputUsesH264Pipeline` replaces the three `outputFormat === "mp4"`
router gates.
- `hlsConfig.ts`: `hlsSegmentSeconds` (default 4, positive integer), the
GOP derivation, and the fail-fast gates. `force-hdr` is rejected (v1 is
SDR-only; HDR10 needs HEVC in fMP4), and so is `useGpu` — the GOP lock is
software-only, so a GPU HLS render segments correctly by accident on
videotoolbox and wrong on nvenc.
- `assembleStage.ts`: `hls` branches to `packageHls` instead of
mux/faststart, after the same `padOrTrimAudioToVideoFrameCount` the mp4
mux uses.
- `artifactTransaction.ts`: a directory artifact can now carry a
`probeTarget` + `requiredFiles`, so the #3395 truncation gate still runs
for HLS against `video.m3u8`. `expectedFrames` is inert there (ffprobe
reports no frame count through the hls demuxer); duration carries it.
- `renderRequest.ts`: accepts `hls` + `hlsSegmentSeconds`; distributed
conversion rejects it alongside `gif`. `DistributedFormat` is unchanged.
`RENDER_REQUEST_VERSION` stays at 1: the new field is optional, the
parser ignores unknown option keys, and an older worker already rejects
`format: "hls"` with "Render request format is invalid".
- `hdrMode.ts`: the non-mp4 SDR downgrade warning no longer claims
"HDR + alpha" for `hls`; the helper extraction landed in the preceding
refactor commit.
Verified: `hlsRender.integration.test.ts` renders a real 10 s composition
and asserts 3 segments, the first two exactly 4.000 s, every segment
starting on an IDR frame, and PTS 0. Removing the GOP wiring collapses it
to 2 segments, so the test is load-bearing. Both producer lanes (unit and
integration) run on every PR in `ci.yml`, so that wiring check is on the
PR gate.
CLI flags and docs are deliberately left to the next PR in the series.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(engine): close the HLS master playlist rewrite race
CodeQL flagged two high-severity alerts on the `stripAudioOnlyVariants`
rewrite: a check-then-use race (`existsSync` then a separate path-based
read and write) and insecure creation of a file at a predictable path in
the OS temp dir.
Both came from resolving `master.m3u8` three times. Open it once with
`r+` and owner-only mode — which neither creates the playlist nor widens
its permissions, ffmpeg owns that — then read, truncate and write through
that one descriptor. Absence is still tolerated for the argument-level
tests that stub ffmpeg, now via ENOENT rather than a prior existence
check.
No behavior change: `hlsRender.integration.test.ts` still renders a real
composition with audio and asserts the stripped master playlist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deliverable one of issue #4010: average K sub-frame captures into each output
frame so rotation, scale and opacity blur the way After Effects does, which the
translation-only catalog component (#4008) cannot do. Opt in per render with
CaptureOptions.motionBlur; absent means the capture path is unchanged.
The blocker was quantizeTimeToFrame flooring every seek onto the output frame
grid, which collapsed all K samples onto one instant and made the averaged frame
byte-identical to the unblurred one. quantizeSeekTime now owns that decision at
all three quantize sites (runtime player, its no-timeline fallback, and the init
transport) and refines the grid by an integer subdivision count, which can only
contain the output grid, never move a frame-time instant off it. The finer grid
rounds rather than floors: a sample time round-trips through the page bridge as
a float, and past a few million ticks that error exceeds the floor epsilon.
Accumulation sits inside captureFrameCore AFTER the static-dedup early return,
so a static frame still costs one reused buffer and only moving frames pay K. It
is not a before-capture hook, which would have disabled dedup outright.
Callback invariant: one eventful seek per output frame at the frame time, then K
suppressed sample seeks, then a suppressed restore to the frame time. The
restore matters because a suppressed seek still moves the playhead, so without
it the next frame's eventful seek would swallow the callbacks inside the shutter
window.
Scope of this deliverable: screenshot capture mode and PNG output. beginframe,
drawelement and JPEG encoding are rejected with a named error rather than
silently rendering unblurred. Video content is rejected for the same reason it
is a documented limitation: injected frames are extracted per output frame and
cannot follow a sub-frame seek.
Blend defaults to sRGB, matching AE's non-linearised 8-bpc working space and
the catalog component. Linear light is an explicit opt-in.
Samples are folded into a running average as they are captured rather than held
until the end, so peak memory is one accumulator plus one decoded sample instead
of all K: at 1080p that is a constant 41 MB rather than 141 MB at 16 samples and
540 MB at the 64 maximum, on every frame of the render.
PNG chunk CRC32 now has one owner in utils/crc32.ts, shared by the new encoder
and ffprobe's metadata reader. The encoder previously imported zlib's crc32 by
name, which throws at module evaluation on Node 22.0 and 22.1; ffprobe already
carried the guarded version and the comment explaining why. The sRGB transfer
function is likewise exported once and shared.
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
## What this does
Video encoders normally choose their own keyframe positions. A keyframe is a frame that can be decoded on its own, and you can only cut a video cleanly at one. The engine already has a flag, `lockGopForChunkConcat`, that forces a keyframe every `gopSize` frames so chunk files can be joined with `ffmpeg -c copy`. Today that logic is copy-pasted in two places inside `buildEncoderArgs` and does not exist at all in the streaming encoder.
This PR:
1. **Extracts the locked-GOP logic into three small shared helpers** in `chunkEncoder.ts`: `resolveLockedGopSize` (validation), `appendLockedGopArgs` (the `-g` / `-keyint_min` / `-sc_threshold` / `-force_key_frames` args), and `lockedGopCodecParams` (the x264/x265 param fragment). `buildEncoderArgs` now calls them instead of inlining the same block twice.
2. **Adds the same flag to the streaming encoder** (`StreamingEncoderOptions.lockGopForChunkConcat` + `gopSize`), which is the path most renders under four minutes actually use. It emits the identical args as the disk encoder, including `-bf 0` for h265 under a lock so B-frame reordering cannot reintroduce negative DTS at a boundary.
## What does not change
- Any render that does not set the flag produces a byte-identical ffmpeg arg list. The flag defaults to `false` everywhere.
- Nothing in the tree sets the flag yet except the distributed chunk worker, which already used the disk encoder. No user-facing behavior changes.
- GPU encoders, VP9 and ProRes streaming encodes continue to ignore the flag, exactly as `EncoderOptions` documents. An earlier revision of this PR made GPU encoders honor it; that was dropped because the path is unreachable until HLS lands and was verified only on videotoolbox.
## Why
Groundwork for `format: "hls"` in `@hyperframes/producer`. HLS packaging stream-copies the encoded video into fixed-length segments with `-f hls -c copy`, and the segmenter can only cut on keyframes the encoder already produced. Locking the GOP at encode time is what makes every segment land exactly on a 4-second boundary without re-encoding. The follow-up PR adds the format itself and should reject `--gpu` for `hls` in v1.
## Testing
- `chunkEncoder.test.ts`: the three helpers tested directly; existing locked-GOP and GPU no-op tests unchanged.
- `streamingEncoder.test.ts`: new `lockGopForChunkConcat` block covering default off, libx264, libx265, and invalid `gopSize`.
- Engine typecheck, oxlint, oxfmt clean. Full engine suite matches main's baseline (the 4 `ffprobe.test.ts` HDR fixture failures are pre-existing on `main`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Add regression coverage for finite SDR video slots that start past EOF in mixed HDR timelines, and verify lint exclusions with successful local duration probes.
Co-authored-by: heygengenesis[bot] <262951085+heygengenesis[bot]@users.noreply.github.com>
Co-authored-by: miguel.sierra <229591595+miguel-heygen@users.noreply.github.com>
Allow explicit finite non-looping video slots to use the existing held-final-frame extraction path when data-media-start is at or beyond source duration. Warn during project lint for local static slots that will use this behavior.
Co-authored-by: heygengenesis[bot] <262951085+heygengenesis[bot]@users.noreply.github.com>
Co-authored-by: miguel.sierra <229591595+miguel-heygen@users.noreply.github.com>
The shared, size-capped video-frame extraction cache evicts entries whose
LRU-clock sentinel hasn't been touched in over an hour. That clock was set
once, at cache-hit lookup time, and never again — so a render holding a
compiled-dir symlink (or a still-open partial writer dir) into an entry for
longer than an hour could lose it to a concurrent render's GC sweep on the
same shared machine, leaving its frame reads a dangling ENOENT with no
retry.
Every active video's captured frame now renews its cache entry's clock,
throttled to once per 5 minutes per directory. touchCacheDir renews both
signals gcExtractionCache actually reads: the directory's own mtime (which
gates a still-open partial writer dir) and the completion sentinel's mtime
(which gates a published entry).
Co-authored-by: Miguel Angel <miguel.sierra@heygen.com>
Chromium's CDP layer can refuse a screenshot capture outright ("Protocol
error (Page.captureScreenshot): Unable to capture screenshot") with no
timeout wording at all. classifyCaptureFailure() didn't recognize the
message, so it fell through to the fatal "authoring" bucket and neither
capture-retry path (the disk-capture retry loop, or the streaming-capture
pinned-fallback retry) ever attempted a retry — including under
--low-memory-mode, whose single-worker plain-screenshot capture never
engages either path's other retry conditions.
Adds the literal error text to classifyCaptureFailure()'s transient_browser
bucket, and threads the resulting classification into
shouldRetryViaPinnedFallback() as a routing-independent retry condition, so
the streaming path recovers on a fresh session the same way the disk path
already does for other transient browser failures.
Co-authored-by: Miguel Angel <miguel.sierra@heygen.com>
Review feedback on #3805.
captureFrameToBufferPipelined checked isRecoverableDrawElementError and
threw DrawElementCaptureError before reaching captureFrameErrorDiagnostics,
so the NCPR/canvas failures that now abort the whole attempt produced no
frame-error PNG/HTML/JSON bundle — the exact case worth debugging, and the
one the adjacent comment promised mirrored the serial path. The serial path
was unaffected because its own DrawElementCaptureError throw propagates
through captureFrameCore's outer diagnostics catch.
Run diagnostics first, then the recoverable wrapper. Bounded to at most one
bundle per attempt, since a recoverable error fails the whole attempt, and
captureFrameErrorDiagnostics self-catches, so a dead page cannot mask the
structural error the producer's fresh-page retry depends on.
Covered by a new test asserting the bundle lands for a recoverable pipelined
failure; verified by mutation (restoring the old order fails it).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The static-frame dedup predictor walked tween intervals from
window.__timelines but was blind to onUpdate callbacks — motion driven
from a timeline's onUpdate in a tween-free window was predicted static.
The verifier compounded the gap by seeking with suppressEvents: true,
so the onUpdate never fired and the frozen frame passed verification.
Two changes:
1. Predictor: when a timeline carries vars.onUpdate, mark its full span
as animated so those frames are never predicted static.
2. Verifier: seek with suppressEvents: false so the verification page
behaves identically to the capture page. The verification page is
already isolated (separate Page instance), so out-of-order event
side effects cannot corrupt sequential capture.
Fixes#3793
* test(producer): pin unpinned drawElement stall retry
* fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing (#3172)
* fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing
A single drawElement frame could kill an entire render. On one comp
(caption-editorial-emphasis) drawElementImage returns normally and the renderer
then stops draining its task queue: the setTimeout(…, 0) that schedules
toDataURL never fires, the capture page.evaluate never settles, and 60s later
the stage watchdog fails the whole render. Deterministic on 152.0.7977.30,
always the same frame. Root-cause detail in PRINFRA-488.
Two gaps, both closed here.
1. Nothing below the 60s stage watchdog bounded a frame. Adds a per-frame
deadline (HF_DE_FRAME_TIMEOUT_MS, default 15000, 0 disables) around the WHOLE
frame operation, not just the drawElementImage call — the stall surfaces at
whichever page round-trip comes next, and was observed at both the capture
and the seek's background-image decode. Bounding one call missed it.
Deliberately NO per-frame screenshot fallback for this error: once the
renderer stops scheduling it is wedged for every subsequent round-trip on
that page. Measured — the screenshot fallback blew the same deadline. The
frame fails fast instead, so the recovery that actually works can run.
2. That recovery was ineligible. shouldRetryViaPinnedFallback only retried a
generic capture failure when the worker count was PINNED by the inversion or
the router; a comp that engaged drawElement on the ordinary single-worker
path had no whole-render fallback at all. A renderer stall is now retryable
on any routing — the failure is a property of drawElement itself, and the
retry re-renders on a fresh page via screenshot. Reported as
de_fallback_reason "de_renderer_stall" so it is distinguishable from
capture_error in telemetry, and counted per-session as deFrameTimeouts.
Verified end to end on the repro: previously exit 1 with "stalled: no frame
progress for 60000ms"; now the deadline fires, the render retries via
screenshot, and RENDER_OK with a valid 1920x1080 / 240-frame / 8.0s MP4.
Tests cover the predicate both ways plus the cross-package error match. Engine
1,481 and producer 587 green (audioPadTrim.integration flakes only under
full-suite parallel load — passes in isolation, and on clean HEAD it is green
too, so it is not from this change).
Refs PRINFRA-488
* fix(engine,producer,cli): count drawElement frame timeouts where they happen
`session.deFrameTimeouts` was incremented inside `captureFrameCore`'s catch,
but the deadline races that function from OUTSIDE it — `withFrameDeadline` is
applied by `captureFrameToBuffer`, and `DeFrameTimeoutError` is constructed
nowhere else. So the branch was unreachable: on a wedged renderer the counter
stayed 0, the engine-side stall log never printed, and
`CapturePerfSummary.deFrameTimeouts` reported 0 on every affected render. The
end-to-end recovery worked throughout — `isDeRendererStallError` matches by
name at the producer — only the observability was dead.
Moved the increment and the log into an `onTimeout` hook that fires from the
deadline itself, deleted the unreachable branch and the now-unused
`isDeFrameTimeoutError`, and threaded the count out to
`render_completed.de_frame_timeouts` so the rate is graphable apart from the
other `capture_error` fallbacks. A stall always costs a whole-render re-run,
which is why it deserves its own series.
Also corrected the `fallbackReason` docstring, which still listed only
`psnr | blank | oom | capture_error` after this branch added
`de_renderer_stall`.
`withFrameDeadline` is exported for a two-case fake-timer test: the deadline
fires and calls `onTimeout` once, and a resolving inner promise passes its
value through with no late timer. The first case fails if the counter moves
back inside the work promise.
(cherry picked from commit ec2b2b5b2a)
* fix(render): retry sequential screenshot stalls
* fix(render): bound parallel worker capture phases
---------
Co-authored-by: Vance Ingalls <vance@heygen.com>
* fix(producer): transport safe extraction failure metadata
* refactor(producer): generalize public error metadata
* test(producer): use vendor-neutral media hosts
The ENAMETOOLONG regression writes 150 real clip files and the mixer
existence-checks each one: ~58ms on Linux, but past vitest's 5s default on
the Windows lane. packages/engine sets no global testTimeout, so heavy
tests here carry an explicit one.
On timeout its abandoned async work kept calling the shared runFfmpegMock
after afterEach cleared it, so the next test saw 5 calls instead of 3 and
lost its queued once-implementations to the leak. mockReset stops an
aborted test from handing leftovers to the next one.
When a composition script throws during execution, the GSAP timeline
registration never arrives and pollSubCompositionTimelines times out.
Previously the render continued with a degenerate 2-frame output and
reported success — now it fails loudly.
Two changes:
1. Detect composition script runtime errors in the browser console
handler and feed them into scriptLoadFailures, triggering the
existing fail-fast path (same as script load 404s).
2. Make sub_timeline_script_failure a fatal warning in
applyRenderWarningPolicy, alongside audio_processing_failed.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`muxVideoWithAudio` passed `-avoid_negative_ts make_zero` unless the caller
set `preserveAudioPrimingEditList`. In practice the dominant path is an AAC
sidecar copied into mp4, where that flag is actively harmful: ffmpeg's
default is `auto`, which the mp4/mov muxers (AVFMT_TS_NEGATIVE) already
resolve to `disabled`. Forcing `make_zero` overrides the correct default,
discards the priming edit list the sidecar encode created, shifts the video
start_time forward by one AAC frame and writes an empty video edit at t=0 —
which edit-list-honoring players (QuickTime/Safari) render as a black first
frame.
Verified with ffprobe on a copy mux of a 30fps h264 mp4 and an AAC sidecar:
with `make_zero` video start_time 0.066000, elst: [media time -1,
dur 5940] + [media time 6000, dur 180000]
audio start_time 0.042993, elst: [media time -1, ...]
without (this fix) video start_time 0.000000, elst: [media time 6000,
dur 180000]
audio start_time 0.000000, elst: [media time 1024, ...]
The empty leading edit and the offset both disappear, and the audio keeps
its 1024-sample priming edit.
The flag is now never passed for a mux, in any mode. `preserveAudioPrimingEditList`
is part of the exported engine API, so it stays on `MuxVideoWithAudioOptions`
as `@deprecated` and no-op rather than being removed; the two internal callers
that set it (`assembleStage`, distributed `assemble`) drop it.
`buildEncoderArgs` and `streamingEncoder` still pass the flag for video-only
output and are deliberately left alone — those chunks are consumed as
intermediates, not as a delivered mp4/mov.
Fixes#3487
Co-authored-by: Alexandru Mincu <alex@mountsoftware.ro>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parent src-only scans skipped multi-format <video>/<audio> markup, so those
elements were never extracted, downloaded, or mixed and rendered blank/silent.
Lint now accepts a child <source src> as a resolvable media src.
lookupCacheEntry reported a hit purely on the presence of the
.hf-complete sentinel. The sentinel records that extraction finished,
not that the frames survived, so any per-file cleanup that empties the
directory leaves an entry that rehydrates with zero frames.
rehydrateCacheEntry then returns totalFrames: 0, the clip reaches the
coverage gate with nothing, and the render aborts with a message about
capture coverage. Because the poison is on disk rather than in the
composition, every later render of the project fails the same way with
nothing the user can change to fix it.
An entry now counts as a hit only when it carries the sentinel AND
still holds at least one frame file, so an emptied entry re-extracts.
The check is format-agnostic: a hit must be usable whatever extension
the frames carry.
Addresses the cache half of #3372.
compileTimingAttrs/injectDurations used parseFloat, so data-start="intro"
wrote a NaN data-end and extract preferred that over duration; parseNumeric
now skips the id-ref (parseVideoElements already resolves it).
collectRenderMedia's resolveHostWindow likewise read host data-start with
parseFloat, so chained sub-composition slots (data-start="hook") stacked at
0-2s and every scene after the first rendered black. It now resolves host
starts through the shared resolveReferencedStart, matching the media parsers.
Fixes#3361.
* fix(producer): fall back to screenshot capture on drawElement canvas-not-initialized
The fast-capture drawElement path only special-cased the "No cached
paint record" error to trigger a per-frame screenshot fallback; every
other error (including "drawElement canvas not initialized", seen at
frame 0 on some macOS/Chrome combinations) was rethrown, hard-failing
the whole render even though the docs promise automatic fallback on
incompatible compositions.
Extend the existing fallback branch (in both captureFrameCore and
captureFrameToBufferPipelined) to also catch canvas-not-initialized
errors via a shared isRecoverableDrawElementError predicate, with a
diagnostic message identifying which case triggered the fallback.
Closes#3423
Co-Authored-By: Miga <noreply@anthropic.com>
* fix(producer): address review — tighten error matching, audit batch path, add fallback-ratio guard
* fix(engine): add prepareFrameForCapture to batch screenshot fallback loop
* fix(engine): split canvas-not-initialized from composition-root-missing errors
drawElementService threw the same HF_DE_CANVAS_NOT_INITIALIZED error for
both !canvas and !root. Missing composition root (navigated/broken page)
was classified recoverable and fell back to pageScreenshotCapture, which
captured blank or wrong content silently.
Now:
- !root → HF_DE_COMPOSITION_ROOT_MISSING (not recoverable, hard fail)
- !canvas → HF_DE_CANVAS_NOT_INITIALIZED (recoverable, screenshot fallback)
Split applied at all 3 emit sites (serial, pipelined, batch).
Co-Authored-By: miga-heygen <miguel.sierra_miga@heygen.com>
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Co-authored-by: Miga <noreply@anthropic.com>
Root cause: the per-worker capture calls in captureFrameRange
(parallelCoordinator.ts) take no abort signal of their own, and only
checked `signal.aborted` BEFORE starting each frame — a no-op once a
worker is already awaiting an in-flight call. On WSL2, the native
drawElement/BeginFrame capture call can hang indefinitely at frame 0
with no error. The DE parallel-router's existing stall watchdog
(captureStreamingStage.ts) correctly fires `stallController.abort()`
after HF_DE_STALL_MS, but that abort had no way to reach a
worker already wedged inside a hung capture call — so
executeParallelCapture's Promise.all waited forever, the render hung
indefinitely, and the CLI's circuit breaker (which only runs after
executeRenderJob settles) never got a chance to trip.
Fix: race each per-frame capture call against the signal actually
firing (raceAgainstAbort), the same "can't cancel, only race" pattern
already used by the sequential capture path. Once the watchdog's abort
is observed, the wedged worker rejects, executeParallelCapture settles,
and the existing pinned-fallback retry / "reverted" outcome / circuit
breaker machinery (already correct) runs end to end.
Also widen the CLI breaker's trip condition from the literal string
"reverted" to "not a clean routed success", so any future non-success
outcome the observability layer records also latches the breaker
instead of silently falling through.
Closes#3441
Co-authored-by: Miga <noreply@anthropic.com>
* fix(core): harden audio FX and group identity
* fix(core): address audio group review feedback
* fix(core): align preview transport with grouped audio
* test(core): pin audio group gain ceiling
* fix(core): preserve solo bridge through stack
* fix(engine): harden grouped audio rendering
* docs(engine): explain grouped mix fallback invariant
* test(engine): allow grouped mixes to finish on Windows
## What
Adds a one-shot ffmpeg-psnr filter probe at drawElement session bootstrap.
When the resident ffmpeg is missing or lacks libpostproc (no `psnr` filter),
the capture-session router now force-fallbacks to the screenshot capture
path and emits a `de_gate_reason = "ffmpeg_no_psnr_filter"` telemetry
signal via the existing `render_complete` breakdown.
Also tightens `psnrForDiskSample`'s catch: infrastructure-class ffmpeg
failures (ENOENT, "No such filter") no longer silently skip the sample —
they abort the render so the safety net cannot fail-open post-preflight.
## Why
The drawElement self-verify safety net (parallelCoordinator's
`psnrForDiskSample` → `psnrDb`) shells to `ffmpeg -lavfi psnr`. If ffmpeg
is missing, or was compiled without libpostproc (so the `psnr` filter is
absent), every per-sample compare throws. The existing catch swallows the
error and returns `null` — callers treat that as "skip this sample" and
the render completes with the safety net inoperative.
Field signal (⭐ 9/10 CLI feedback, Slack ts=1787380767.210079,
hyperframes 0.8.7, darwin/arm64, tid=93ff9910-2207-45c2-bc1f-54c0b347d4fe):
> "host ffmpeg lacked psnr filter used by drawElement self-verification,
> but render completed."
The user's frames happened to be byte-identical so no visual damage
shipped — but the safety net silently wasn't running. Any future
compositor-damage bug on that host would have shipped straight through.
## How
Two-part fix, both in `packages/engine`:
1. New `utils/psnrFilterAvailability.ts` — cached probe that runs
`ffmpeg -hide_banner -filters` once per process and word-boundary-
matches `psnr` in the output. Any failure (ENOENT, non-zero exit,
timeout, unparseable output) returns `false`; never rejects.
2. Wired into `services/frameCapture.ts` `initDrawElementOrTransparentBackground`
right after the Chrome capability probe: when useDrawElement resolves
true and the preflight returns false, set
`session.deGateReason = "ffmpeg_no_psnr_filter"` (same low-cardinality
bucket every other DE gate uses; flows through `getCapturePerfSummary`
→ `render_complete.de_gate_reason` in PostHog), emit a stderr warning
naming what's missing, and call `routeToFallback()` — the same
fail-graceful shape as the SwiftShader / CSS-effect / at-risk-timeline
gates. Skipped under `HF_FORCE_DRAWELEMENT=1` (matches the diagnostic
knob's policy of bypassing every other gate).
Belt-and-braces: `psnrForDiskSample` now discriminates infrastructure-
class failures (ENOENT / "No such filter" / "Unknown filter") from
per-sample noise (readFile races, transient EPERM). Only the former
re-throw — per-sample noise still returns `null` (skipped sample). The
preflight normally catches this at bootstrap; the re-throw covers
ffmpeg-swapped-mid-render.
## Test plan
- [x] Unit tests added:
`packages/engine/src/utils/psnrFilterAvailability.test.ts` — mocked
`execFile` covers: `psnr` present → true; `psnr` absent → false; ENOENT
→ false; non-zero exit → false; result memoized + reset works;
substring-not-word-boundary → false.
- [x] Unit tests added:
`isFfmpegInfrastructureFailure` in
`packages/engine/src/services/parallelCoordinator.test.ts` covers
ENOENT, "No such filter", "Unknown filter", per-sample EACCES, parse
errors, null/non-object.
- [x] `bun run test` — `packages/engine/src/utils/psnrFilterAvailability.test.ts`
(6 tests) + `packages/engine/src/services/parallelCoordinator.test.ts`
(50 tests) + `frameCapture.test.ts` (26 tests) all pass. Pre-existing
ffprobe test failures (4) on the base commit are unrelated (missing PNG
fixture bytes — the file is 129 B on disk, likely LFS-stored).
- [x] `bunx tsc --noEmit -p packages/engine/tsconfig.json` — clean.
- [x] `bunx oxlint <files>` — 0 warnings, 0 errors.
- [x] `bunx oxfmt --check <files>` — clean.
Not covered here: an integration test that boots
`initDrawElementOrTransparentBackground` end-to-end. That path is
Puppeteer-driven and has no unit-scale bootstrap harness in the
repository — the pure preflight + pure discriminator coverage above are
what this PR can prove at the vitest layer.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
`places a delayed track on its authored start` timed out on the windows runner
and failed an unrelated PR, the second time this week an ffmpeg audio test has
done that.
The previous fix raised the budget in `audioMixer.grouping.test.ts`, which was
the file the symptom named. It was the wrong scope: that file was the only audio
suite with explicit timeouts at all. `audioMixer.level.test.ts` had none, so its
two real-ffmpeg tests ran on vitest's 5s default. The failing one takes ~137ms
locally, so the runner is not 36x slower — but 5s was never a budget anyone
chose for a full mix.
Applied to the suite rather than to each test, so there is one home for it, and
scoped to the ffmpeg-gated describe: the sibling parsing suites are pure and
should keep failing fast at 5s.
Headroom alone would only have delayed an undiagnosable failure. The ffmpeg
process timeout is 5 minutes by default, far above any test budget, so a stalled
mix could only ever surface as a bare "Test timed out" with no stderr and no
failing stage. Tests now cap it at 20s and assert through a helper that reports
`failures` instead of collapsing to `expected false to be true`.
Both claims verified rather than asserted: a deliberately 6s test now passes
where the 5s default would have killed it, and forcing the process timeout to
1ms reports `stage: "prepare", reason: "ffmpeg_timeout"` instead of a timeout.
Reviewing with whitespace ignored is much smaller: adding the third argument to
`describe` reindents the suite body, so 131/101 is really 34/4.
`a group FX chain fully cutting its members leaves an ungrouped track
untouched` timed out on the windows runner, failing an unrelated PR. The whole
file runs in ~4s locally and that test in ~1.1s, so 30s was not generous — but
the runner is roughly 10x slower and this test drives more ffmpeg than any of
its siblings, two full mixes plus a group FX chain. 30s was the tightest budget
in the package; 60s is what the rest of the ffmpeg-driven engine tests use.
Headroom alone would only have moved the same undiagnosable failure later,
because nothing here could report why. The production ffmpeg process timeout is
5 minutes, far above any test budget, so a stalled mix could only ever surface
as "Test timed out in 30000ms" with no stderr and no failing stage. Tests now
cap it at 20s, and the mix wrapper throws the recorded failures instead of
returning `success: false` into an `expect(...).toBe(true)` that reports
`expected false to be true` and discards the reason.
Verified by forcing the process timeout to 1ms: the failure goes from a 30s
wall-clock timeout to a 150ms error naming the stage, reason and element
(`stage: "prepare", reason: "ffmpeg_timeout", elementId: "a"`).
This does not explain the Windows stall itself, which I could not reproduce on
macOS. It makes the next occurrence report what it was doing.
audioMixer.grouping.test.ts spawns real ffmpeg per assertion and ran on
vitest's 5s default; on slow Windows runners the FX-chain and envelope
cases land right at the line and fail runs that touch nothing in the
engine. The other ffmpeg-bound engine suites (videoFrameExtractor)
already carry a per-test 30_000 timeout; this brings the grouping suite
in line.
B5: mute and solo, on groups and tracks (track mute already shipped by A2 —
nothing to build there).
Group mute — persisted as data-hidden on the <hf-audio-group> element itself
(never written onto members, per design doc §2.1's state-restoration
warning). Studio action reuses B7's generic setAudioGroupAttribute
(setQuiet/setLive split) rather than duplicating toggleTimelineTrackHidden's
shape — same one-atomic-patch/one-undo-entry contract, already built for
exactly this purpose. Render: B4 already drops every member of a
data-hidden group (confirmed by a new audioMixer.test.ts case — no
production change needed there). Preview: a dedicated muteGain node
(groupInput -> [fx] -> muteGain -> output -> master) so a mute toggle
never fights scheduleVolumeLane's ramps on the same param — the same
hazard B7's volume fader was split out to avoid. Mid-playback toggles
sync via a new syncAudioGroupMute pass in init.ts (a group carries no
data-start, so it's invisible to the existing visibility-node query).
Members of a muted group render the strikethrough label treatment
(TimelineTrackPlainHeader's isGroupMuted, sourced from
TimelineElement.audioGroupHidden) — display only, no attribute touched.
Solo — "Hear only this": a new session-only store slice (audioSoloSlice,
soloed: ReadonlySet<string> of clip/group ids, never track numbers, never
serialized). Predicate (isAudibleUnderSolo, packages/core/src/audioGroups.ts
so both the store and the preview transport share one definition): an
element is audible while any solo is active only if it or its own group is
soloed. "Siblings, never ancestors" lives in the graph, not the predicate —
solo gain is a per-element stage only; group buses are never attenuated by
solo, so a soloed member's path through its group stays open by
construction. Preview: a dedicated per-element soloGain in
webAudioTransport.ts (parallel to the mute mechanics), pushed via
window.__hf.setAudioSolo — a direct call, not an attribute write, so it
can't ride the visibility-diff path mute uses. media.ts's HTMLMedia
fallback folds the same predicate into its per-tick volume computation
(the same seam A2 used for data-hidden). Half-lit group indicator
(isGroupHalfLitUnderSolo) for "not soloed itself, but a member is".
Exclusive-by-default toggle, ⌘/Ctrl-click to add/remove, TimelineSoloButton
(⌗) beside mute on both track and group headers. Transport-bar banner
("Hearing only <label> — your export is not affected", Clear button) added
in PlayerControls.tsx, reading labels straight off the live preview DOM.
Export-safety, the most important property here: toggling/adding/clearing
solo never calls setAttribute/removeAttribute on any element and never
invokes the project save path (both asserted directly via spies in
audioSoloSlice.test.ts) — solo cannot reach an export by construction, not
by convention.
Also: extracted useHydrateActiveCompPathFromUrl out of App.tsx (a
pre-existing, unrelated effect) to stay under the 600-line filesize cap
after wiring useAudioSoloBridge in; and fixed a circular dependency the
solo-banner wiring introduced (useAudioSoloBridge.ts now imports
usePlayerStore from its concrete module instead of the player/ barrel,
which re-exports PlayerControls.tsx — the barrel path is what closed the
cycle).
Gates: bun run build clean; packages/core full suite 2379/2379; packages/
studio full suite 4276/4294 (18 pre-existing todo); packages/engine
audioMixer.grouping.test.ts 5/5; oxfmt/oxlint clean on all 23 touched
files; fallow clean (0 new circular deps, 0 new filesize/complexity
findings).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(core): route grouped audio through a group bus in preview
An audio element carrying `data-audio-group` no longer lands its gain on
the master bus directly — it feeds a per-group `GainNode` (built lazily on
first use, one per group id) which itself feeds master, so members of the
same group sum before the ear, ready for a group-level FX chain and
volume/mute in later steps. An id with no matching `<hf-audio-group>`
element still gets a plain, unprocessed bus rather than losing the track.
The group's own chain and volume lane are wired through the same
`attachElementFxChain`/`scheduleVolumeLane` every element already uses,
against the group's clock — composition time (design doc §1.3), since a
group has no `data-start` and a missing one parses as 0. The bus persists
across `stopAll()` (mirroring `_masterGain`'s own lifecycle) so replaying a
group does not rebuild its chain; only `destroy()` disposes it.
Render is untouched — stays flat until B4; `audio-groups` is still a 0%
canary so nothing ships this to a real composition without hand-authoring
`data-audio-group`.
Also: `audioGroupOf` (B1) crashed on any element lacking a real `tagName`/
`getAttribute` — exactly the shape of most `HTMLMediaElement` test doubles
in this suite, including this file's own `mockEl`. Made it tolerant, same
style as `readChain`'s existing guard in `runtime/audioFx.ts`.
`schedulePlayback` was already 110 lines pre-existing before this diff;
extracted `resolveDestination` and `handleSourceEnded` to shrink it to 92,
then suppressed the remainder (inherently sequential graph wiring, not a
decision tree) per the same precedent B2 used on `TimelineLogicalRow`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(studio,lint): carve targets voiceover groups — always, when plural
Plural voiceover carve now targets a group instead of naming each clip:
`resolveCarveSourceIds` (core `audioGroups.ts`) expands a group id to its
current members at analysis time, so a clip added to the group later is
covered without touching `sources`. The picker (`useFxCarve.ts`) offers a
grouped voice as one option instead of one row per member, tests overlap
as a union of member spans (a group overlaps the bed if ANY member does),
and prefers a qualifying group over its individual members in
`autoSourceIds`.
Picking two or more ungrouped voice clips in the carve flow now mints a
group behind them (`mintGroupId`, de-duped against every id in the
document) and writes `data-audio-group` on each picked clip atomically,
one undo entry — `createAudioGroupAndAssignMembers` in
`timelineTrackVisibility.ts` copies `setElementsHidden`'s multi-target
write shape. The DSP is untouched: `mixCarveSources` already sums
multiple sources correctly (verified in the design doc's own
investigation) — this only fixes the picker.
New lint rule `audio_carve_ungrouped_sources` (`packages/lint/src/rules/
media.ts`, alongside `audio_volume_double_automation`) warns when a
`data-fx-carve`'s `sources` names two or more plain clip ids instead of a
group — the shape that silently rots when a clip is added. `/hyperframes-
audio` states the same rule as an invariant, not a tip, with the grouped-
narration HTML example from the design doc.
The group-matching and auto-group logic (`withAutoGroupedSources`,
`collectCarveCandidates`) is split into `useFxCarveGrouping.ts` —
`useFxCarve.ts` was pushing past the 600-line cap. `resolveNextCarveSettings`
is deliberately NOT an `async function`: wrapping it in one would force a
microtask on every call, including the synchronous branch — the exact bug
`withAutoGroupedSources`'s own sync-when-possible contract exists to avoid,
and one caught via `propertyPanelAudioFxGroup.test.tsx` (10 failures)
before fixing it back to a plain function the caller conditionally awaits.
Also extracted `useEffectiveTimelineDuration` out of `App.tsx` and
`useRemoveBackground` out of `StudioRightPanel.tsx` (both pushed past 600
lines from an added prop wire), and decomposed `useFxCarve.ts`'s picker
IIFE to clear fallow's complexity gate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(engine): render grouped audio through a summed, FX-processed bus
Renders what B3 already routes in preview: a group's members sub-mix into
one PCM WAV at full composition length (adelay already places each member
at its composition position, so the group WAV's t=0 IS composition time),
run through the group's own FX chain and automation via the same
applyAudioFxChain/envelope-bake path a member uses, then fold into the flat
track list as one processed AudioTrack — the final mixAudioTracks call
never has to know groups exist.
Gain law verified against plans/spikes/amix-nesting-spike.sh (brought over
from the plans branch, along with audioMixer.grouping.test.ts, since both
were committed there and never merged to origin/main — every step branch in
this stack descends from origin/main): the sub-mix's own amix prefers
normalize=0 (nulls exactly against a flat mix), falling back to per-node
compensation by the group's OWN member count only when this ffmpeg build's
amix rejects the option. Carrying any other count into a nested amix node
is the exact +2.499 dB silent failure the spike measured — confirmed by a
manual mutation check (wrong-count compensation landed 3.5 dB hot, exactly
20*log10(3/2) for a 2-member group compensated as 3; reverted after
confirming the level test catches it).
A group element carrying data-hidden drops every member before the sub-mix
ever runs (RULES: mute-by-drop, never mute-by-volume-0) — parseAudioElements
now resolves groups once per parse and skips hidden-group members the same
way it already skips data-hidden ancestors.
HfAudioGroup (packages/core/src/audioGroups.ts, from B1) gains fxChain,
automation, volume and hidden, read off the group element the same way
resolveAudioGroups already reads data-label — audioGroups.test.ts updated
for the wider shape plus new coverage for the added reads.
it.todo("mixes a grouped composition at the same level as the ungrouped
one") is now a real, passing test; two more added per the step doc (FX
routing isolation, member-level envelope survives grouping).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
ffmpeg and ffprobe are console-subsystem binaries and Node defaults
windowsHide to false, so every spawn opened a visible console window on
Windows. A render shells out dozens of times across parallel workers,
which flashed a burst of windows across the user's desktop.
Applied at every production spawn site rather than only the two named in
the report, since they all share the cause: runFfmpeg, both gpuEncoder
probes, ffprobe, streamingEncoder, audioExtractor and the distributed
version check. windowsHide is a no-op on macOS and Linux.
The dev-only parity and regression harnesses are left alone; they never
run on a user's desktop.
Closes#3379
Adds hf-pitchshift alongside the four existing dynamics worklets: a dual-tap
granular delay line, 100 ms grain, taps 180° apart so one is always
crossfading in as the other resets — hides the splice each tap makes on
wrap. Read-tap speed relative to the write head tracks the semitone ratio,
so pitch shifts without changing duration.
Registered through the same workletBuilder/dispose-message path the other
four use (so shapeOf never rebuilds on a param tweak, and a chain drop
retires it), wired into the registry with a plain-language copy entry and a
~0.2s chain tail (two grains). One implementation, shared by preview (Web
Audio in the page) and render (the same worklet run inside an
OfflineAudioContext in the headless browser) — confirmed by a browser-render
test that measures the actual output frequency, not just that it differs
from input.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
On FFmpeg 5.x through 8.0.x the samples `apad` appends carry timestamps
the following `atrim` misreads. A delayed branch then sounds at t=0
instead of its offset and, once four or more branches are mixed, the last
one disappears from the output entirely. No error is raised; the render
succeeds with wrong audio.
Reverting to `apad=whole_dur=` is not an option: #2769 moved off that
form because some builds reject the option outright ("Error applying
option 'whole_dur': Option not found"). Inserting `asetpts=N/SR/TB`
between the pad and the trim rebuilds the timestamps from the sample
count using only filters every build ships, so it fixes the misplacement
without giving up the portability that change bought.
Verified on FFmpeg 4.2.7, 7.0.2, an 8.x nightly and 8.1.1: the current
form is wrong on the middle two, the new form is correct on all four.
audioPadTrim.ts also pads with apad+atrim but has no adelay and is
correct on every version tested, so it is left alone.
Closes#3344
* fix(producer): give inlined media a document-unique render id
Element ids are unique per composition file, but the render document is
the inlined union of every file. The producer merged the per-file media
lists and deduplicated by id, so clips that shared an id collapsed into a
single entry, and every id-keyed stage (extract, inject, visibility,
bounds) resolved to whichever element came first in the document. The
surviving clip's frames landed on the wrong element and the visible scene
rendered without footage.
Two shapes hit this, and neither is author error:
- Two scenes that each declare `<video id="clip">`. Legal per file, and
unavoidable when a scene is duplicated into a copy with inner ids
kept, or when one file is mounted twice.
- Two scenes that each declare a bare `<video>`. The timing compiler
numbers auto-ids per file, so both arrive as `hf-video-0` with no
authored id involved at all.
Stamp a document-unique `data-hf-render-id` while inlining, and read the
media list off the inlined document instead of merging per-file lists.
The render id equals the element id whenever that id is already unique,
so documents without a collision keep identical pipeline keys.
Author `id` attributes are left alone: 158 of the 161 registry blocks
reference their own ids from `#id` CSS or getElementById, so renaming
would trade broken footage for broken styling. The engine resolves media
elements through the render id instead, falling back to getElementById
for documents the producer never compiled.
Collecting from the inlined document also retires the per-file media
extraction in parseSubCompositions along with its offset bookkeeping;
host offsets are recovered from the composition hosts the clip sits in.
* fix(core): resolve render-frame siblings by render id in the runtime
The injector creates each `__render_frame_<id>__` sibling from the media
element's render id, but four runtime readers still built that id from the
plain `el.id`. On a document where two compositions share a media id, all
of them resolved the first collider's frame.
colorGrading is the one that changes pixels: findRenderFrameImage returns
the image the grading pass samples, with no class check to catch the
mismatch, so the second video was graded from the first one's frame.
media, mediaProxy and video-texture-compat use it as a render-mode or
substitute-source signal, where both colliders happen to agree during
render, but none of them should rest on that.
Add renderFrameSibling as the single owner of "which frame belongs to
this element" and route all four through it. It reads the stamped render
id and falls back to the author id, so a collision-free document resolves
exactly as before and an uncompiled one (preview, snapshot, check) is
unchanged.
The engine's in-page bridge keeps its own copy of the rule because code
serialized into page.evaluate cannot import; it now names core as the
definition, and a test pins the sibling-id format both sides build so
they cannot drift apart silently.
* refactor(engine): build render-frame sibling ids from core's definition
The drift guard named both sides but pinned one. renderFrameSibling.test
asserts core's format, while the engine rebuilt the same id from a literal
template at six independent sites. Changing the format on either side left
the test green and every runtime reader silently unable to find its frame —
this PR's own failure mode, one level up.
Export the affixes and renderFrameIdForRenderId from core, and take the id
from there at all six. Four sites resolve it on the Node side, where the
engine can import; the two that iterate the DOM in-page receive the affixes
as evaluate arguments, which avoids depending on bridge install order.
Also switch two `__hfMediaId?.(el) ?? el.id` reads to `||`. The bridge
returns "" for an element with neither id, so `??` kept the empty string
and built `__render_frame___`, which no reader looks for. Inert today
because the compiler assigns positional ids to id-less timed media, but it
made the two sides disagree in the one case they could.
Miguel Ángel