* feat(engine): adaptive per-frame sample count for motion blur
Measured against the accuracy bar set in #4010 (72.4dB mean PSNR vs the
960fps shutter integral), the fixed default of 16 samples misses it for
hard-edge content while paying nothing extra for static/slow content is
still important. This picks a sample count per frame from measured
motion, floor 16 ceiling 64 (issue #4029).
Two probe captures at the true shutter-window edges (never a sample that
enters the average) measure motion magnitude, mapped through a monotone
step table to a sample count. Frames confirmed to move only non-spatial
properties (opacity, colour) skip the probe and use the floor directly,
since a fade's shutter integral is a smooth blend that does not need many
samples. MotionBlurPlan's window (from shutter angle/phase alone) is now
independent of how many samples fill it, which is what lets the count
vary per frame without moving the window; motionBlurWindowIsStatic is
bounded by the widest sample inset any chosen K could produce so the
static-dedup safety guarantee holds for every K in [16, 64].
* refactor(engine): simplify the adaptive motion-blur sample selection
Two findings from an independent code-reuse/quality/efficiency review of
the previous commit:
- armStaticDedup ran the GSAP-timeline walk (computeStaticFrameSet)
twice on the same page when adaptive motion blur and static dedup
both apply, once for each purpose. Shares the one result instead.
- The non-spatial floor path read the sample count from
adaptiveSampleCount(0), which only worked because the step table
happens to be sorted with the first bucket covering zero. Reads
the now-exported DEFAULT_SAMPLES_PER_FRAME directly instead.
* fix(engine): read raw options for the adaptive motion-blur gate
Independent review caught it: every armStaticDedup call site runs
before finalizeSessionInit resolves session.motionBlur, so the
adaptive-classification gate's `session.motionBlur &&
session.motionBlur.fixedSamplesPerFrame === null` was always false at
runtime. The non-spatial probe-skip described in the PR never engaged
outside tests that hand-set motionBlurNonSpatialFrames directly,
bypassing real init order.
Reads session.options.motionBlur (set unconditionally at session
creation) instead, which carries the same "adaptive requested"
information the resolved plan would have. Adds a source-order lock
test plus direct coverage of computeStaticFrameSet's
nonSpatialOnlyFrameSet, mutation-proved against the exact regression.
* fix(engine): add xPercent/yPercent to the spatial-property list
Round 2 of the independent review: the spatial-property list deciding
whether a frame's motion needs the adaptive probe omitted GSAP's
percentage-based position properties (xPercent/yPercent) and two 3D
transform properties (perspective/transformPerspective) that two other
lists already in this repo (packages/parsers/src/gsapConstants.ts,
packages/studio/.../gsapAnimatesProperty.ts) both carry. A composition
positioned via xPercent/yPercent would have been silently floored to
K=16 with no probe and no log line, the same failure shape as the
round-1 bug through a data gap instead of a dead gate.
Moves the list to an exported SPATIAL_TWEEN_PROPERTIES array in
motionBlur.ts, passed into computeStaticFrameSet's page.evaluate as
data rather than duplicated inline, so it stays a single list and is
directly testable. Adds tests that execute the real page-side closure
(the existing tests all mocked the whole result and never exercised
the matching code), mutation-proved against the exact gap found.
* fix(engine): add rotate to the spatial-property list
Round 3 of the independent review, comparing SPATIAL_TWEEN_PROPERTIES
side by side against the two sibling lists rather than trusting the
prior commit's summary: GSAP's individual "rotate" transform property
(distinct from "rotation", carried in
packages/studio/.../gsapAnimatesProperty.ts's GSAP_TRANSFORM_PROPS)
was still missing, same failure shape as the two prior rounds.
The same pass also raised transformOrigin, which is deliberately NOT
added: gsapConstants.ts's own classifyTweenPropertyGroup excludes it
as a pivot-point modifier with no independent visual effect, and a
test now locks that reasoning in rather than leaving it to be
"fixed" again by a future pass that does not read that far.
The static-frame dedup reuses the previous frame's buffer when a frame is
byte-identical to its predecessor. With motion blur on that test is not
enough: the shutter window reads content either side of the frame instant,
so a still frame next to a moving one reused a buffer that carries none of
that motion. At 30 fps and 16 samples frame 11's window runs 10.75 to 11.25
at a 180 degree shutter and 10.06 to 11.94 at 720, so both reach past the
frame instant and the default is affected too.
motionBlurWindowIsStatic now owns whether a blurred frame may reuse its
predecessor's buffer. It requires every frame the window reads from to be
static, and covers the previous frame's window as well, because reuse claims
the two blurred outputs are equal. The range is derived from the plan's own
sample tick offsets, so it cannot drift from where the samples land.
The guard's contract comes from #4013 by Dante-dan, which carried it before
we did. One existing test fixture is widened: it asserted reuse with only the
frame itself marked static, which the guard now correctly rejects.
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* feat(producer): make sub-frame motion blur reachable from a render
Second deliverable for issue #4010. The engine gained the accumulation path, but
nothing set CaptureOptions.motionBlur, so the feature could not be switched on
from a real render. RenderConfig now carries it and the producer threads it into
the capture options.
Doing that surfaced the defect that would have made it fail silently. "Does this
render capture PNG frames" was spelled four times, every one of them re-derived
from needsAlpha, which describes the OUTPUT container rather than the capture:
probeStage picked the probe capture format
renderOrchestrator picked the render capture format
encodeStage built the encoder's frame_%06d input pattern, twice
captureStage built the disk worker's frame extension
Motion blur is a second, independent reason to capture PNG: the sub-frame samples
are averaged pixel by pixel, so JPEG samples would be averaged after a lossy
quantization. Add that reason to only one of those four and the encoder goes
looking for frame_%06d.jpg against files written as .png, and the render finds no
frames at all rather than reporting an error.
So the rule now has one owner, resolveCaptureImageFormat, which names both
reasons. Everyone downstream reads the resolved capture format instead of
re-deriving it: the encoder and the disk worker share frameFileExtension with
writeCapturedFrame, which is what actually names the files. needsAlpha keeps only
the question it really answers, whether the output preserves alpha.
captureStage no longer needs needsAlpha at all, which is the tell that the
re-derivation was its only reason to know.
No JPEG encoder is involved. The PNG capture path is already load-bearing, since
every transparent-output render uses it and the streaming encoder already selects
png as its input codec.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* fix(engine): resolve the motion-blur plan on every initialization exit
A real render produced no blur at all. The plan was correct, the producer
wiring was correct, and the sub-frame sample times were genuinely distinct, but
K=16 and K=64 rendered identically to no blur because neither ever ran.
initializeSession has two exits: screenshot mode finishes its own block and
returns, every other capture mode falls through to the end. The plan was
resolved at the fall-through exit only, and screenshot is the one capture mode
motion blur supports, so on exactly the path the feature runs on
session.motionBlur was never assigned. captureFrameCore then read it as
undefined and took the single-capture branch: one capture per output frame, no
accumulation, no error.
finalizeSessionInit is now the single owner of what finishing initialization
means. It sets the plan and isInitialized together and is called at both exits,
so a third exit cannot mark a session ready while forgetting the plan.
Why the existing tests passed, which is the part worth keeping: the accumulation
test set session.motionBlur directly on a stub session. It exercised the
mechanism while bypassing the only code that decides whether the mechanism runs
at all. A test that proves something works and never asks whether it is
reachable is not end to end, however much behaviour it drives.
Two tests. A structural guard that session.isInitialized = true appears exactly
once and that the same place resolves the plan, which fails if the defect is
reintroduced at either exit. And a sample-distinctness assertion that hashes the
K buffers captured within one output frame and requires K distinct ones, which
covers the different failure of every sub-frame seek landing on one instant.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* feat(engine): add packageHls for HLS VOD packaging
Adds `packageHls` next to `muxVideoWithAudio`: stream-copies an
already-encoded H.264 video and optional AAC sidecar into an HLS VOD
directory (`master.m3u8` + split video/audio renditions, MPEG-TS
segments) with `-c copy`, so no re-encode happens at package time.
Depends on the locked GOP from the previous commit. `-hls_time` can only
cut at existing keyframes, so `gopSize = segmentSeconds x fps` is what
makes every segment exactly `segmentSeconds` long and IDR-aligned. A
test packages an unlocked encode to pin that: the same call yields one
4s segment instead of four 1s ones.
Deliberately omits the mp4 mux's provenance tags, which MPEG-TS drops
and whose `-movflags` is invalid for `-f hls`, and `-avoid_negative_ts`,
which would discard the AAC priming and shift audio against video (see
issue 3487). A real-ffmpeg test asserts the first presentable audio
sample lands on the first video frame.
No caller yet; `format: "hls"` in the producer follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(engine): zero HLS mux base and reject % in packageHls outputDir
- Pass -muxdelay 0 -muxpreload 0 so the first segment starts at PTS 0
instead of the mpegts muxer's 1.4 s base; A/V alignment through the
AAC priming packet is unchanged
- Throw on an outputDir containing %, which ffmpeg expands as a printf
conversion in -hls_segment_filename and fails with a misleading
"No such file or directory"
- Assert alignment on relative spacing rather than on how one ffmpeg
build clamps the priming packet's negative PTS
- Encode the test fixture with the full software GOP lock
(lockedGopCodecParams), not just appendLockedGopArgs
- Trim comments to the non-obvious rationale
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(engine): satisfy the ffprobe argv contract in packageHls test
`producer/src/utils/ffprobeArgvContract.test.ts` sweeps every package at
the source level and requires `--` to be the argv entry immediately before
an ffprobe input path. The `probe()` helper here passed
`join(outputDir, HLS_VIDEO_PLAYLIST)` in that slot, which the contract's
comma-splitting parser reads as two entries, so the producer suite was red
on this branch while the engine suite stayed green.
`probe()` now takes the input path as a parameter and appends the
terminator itself. The sibling ffmpeg `run()` helper switches from a
spread-in-array-literal to `.concat`, because a verbosity-flag-then-spread
literal is indistinguishable from the ffprobe wrapper shape the contract
demands a terminator on. The matcher reads raw source including comments.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(engine): drop the audio-only variant from the HLS master playlist
With `a:0,agroup:aud` in `-var_stream_map`, ffmpeg lists the audio
rendition twice in `master.m3u8`: as the `#EXT-X-MEDIA:TYPE=AUDIO` group
the video variant references, and again as a standalone `#EXT-X-STREAM-INF`
variant with no `RESOLUTION`. That is valid HLS, but a player choosing by
bandwidth can select it and play sound with no picture, and the VOD
consumer asked for a single rendition.
`stripAudioOnlyVariants` removes any variant tag without a `RESOLUTION`
attribute together with its URI line; `packageHls` applies it to the
master after a successful audio+video run. The rendition group and the
audio playlist are untouched, so `audio.m3u8` is still referenced from the
master. Pinned by a unit test on the playlist text and by the real-ffmpeg
test, which now asserts exactly one `#EXT-X-STREAM-INF`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(producer): extract hdrMode warning and decision-log helpers
`resolveEffectiveHdrMode` was already over the fallow complexity gate on
`main` (21 cyclomatic / 25 cognitive / CRAP 462) and only surfaces now
because the HLS format work needs to touch its downgrade warning. Pull the
downgrade message into `formatDowngradeWarning` and the "which mode and
why" info line into `logHdrDecision`, which puts the function under the
threshold.
No behavior change: every log string is byte-identical and `hdrMode.test.ts`
asserts them unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(producer): add hls output format
Adds `format: "hls"` to `@hyperframes/producer`. An HLS render is the same
opaque H.264 + AAC encode as `mp4`, stream-copied at assemble time (no
re-encode) into a directory: `master.m3u8`, `video.m3u8` + `video_%05d.ts`,
and `audio.m3u8` + `audio_%05d.ts` when the composition has audio.
The load-bearing part is that `-c copy -f hls` can only cut a segment at an
existing keyframe, so the encoder's GOP is locked to
`hlsSegmentSeconds x fps` frames. `resolveHlsEncoderGopLock` returns that
pair as one value spread into both encoder inputs, so the streaming and
disk/chunked paths cannot disagree — a render reaching only one of them
still succeeds, it just emits variable-length segments silently.
- `renderFormat.ts`: `hls` is opaque (`outputNeedsAlpha` is now
`!== "mp4" && !== "hls"` — missing this flips HLS to RGBA screenshot
capture with nothing failing), supports page-side compositing, and
`outputUsesH264Pipeline` replaces the three `outputFormat === "mp4"`
router gates.
- `hlsConfig.ts`: `hlsSegmentSeconds` (default 4, positive integer), the
GOP derivation, and the fail-fast gates. `force-hdr` is rejected (v1 is
SDR-only; HDR10 needs HEVC in fMP4), and so is `useGpu` — the GOP lock is
software-only, so a GPU HLS render segments correctly by accident on
videotoolbox and wrong on nvenc.
- `assembleStage.ts`: `hls` branches to `packageHls` instead of
mux/faststart, after the same `padOrTrimAudioToVideoFrameCount` the mp4
mux uses.
- `artifactTransaction.ts`: a directory artifact can now carry a
`probeTarget` + `requiredFiles`, so the #3395 truncation gate still runs
for HLS against `video.m3u8`. `expectedFrames` is inert there (ffprobe
reports no frame count through the hls demuxer); duration carries it.
- `renderRequest.ts`: accepts `hls` + `hlsSegmentSeconds`; distributed
conversion rejects it alongside `gif`. `DistributedFormat` is unchanged.
`RENDER_REQUEST_VERSION` stays at 1: the new field is optional, the
parser ignores unknown option keys, and an older worker already rejects
`format: "hls"` with "Render request format is invalid".
- `hdrMode.ts`: the non-mp4 SDR downgrade warning no longer claims
"HDR + alpha" for `hls`; the helper extraction landed in the preceding
refactor commit.
Verified: `hlsRender.integration.test.ts` renders a real 10 s composition
and asserts 3 segments, the first two exactly 4.000 s, every segment
starting on an IDR frame, and PTS 0. Removing the GOP wiring collapses it
to 2 segments, so the test is load-bearing. Both producer lanes (unit and
integration) run on every PR in `ci.yml`, so that wiring check is on the
PR gate.
CLI flags and docs are deliberately left to the next PR in the series.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(engine): close the HLS master playlist rewrite race
CodeQL flagged two high-severity alerts on the `stripAudioOnlyVariants`
rewrite: a check-then-use race (`existsSync` then a separate path-based
read and write) and insecure creation of a file at a predictable path in
the OS temp dir.
Both came from resolving `master.m3u8` three times. Open it once with
`r+` and owner-only mode — which neither creates the playlist nor widens
its permissions, ffmpeg owns that — then read, truncate and write through
that one descriptor. Absence is still tolerated for the argument-level
tests that stub ffmpeg, now via ENOENT rather than a prior existence
check.
No behavior change: `hlsRender.integration.test.ts` still renders a real
composition with audio and asserts the stripped master playlist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deliverable one of issue #4010: average K sub-frame captures into each output
frame so rotation, scale and opacity blur the way After Effects does, which the
translation-only catalog component (#4008) cannot do. Opt in per render with
CaptureOptions.motionBlur; absent means the capture path is unchanged.
The blocker was quantizeTimeToFrame flooring every seek onto the output frame
grid, which collapsed all K samples onto one instant and made the averaged frame
byte-identical to the unblurred one. quantizeSeekTime now owns that decision at
all three quantize sites (runtime player, its no-timeline fallback, and the init
transport) and refines the grid by an integer subdivision count, which can only
contain the output grid, never move a frame-time instant off it. The finer grid
rounds rather than floors: a sample time round-trips through the page bridge as
a float, and past a few million ticks that error exceeds the floor epsilon.
Accumulation sits inside captureFrameCore AFTER the static-dedup early return,
so a static frame still costs one reused buffer and only moving frames pay K. It
is not a before-capture hook, which would have disabled dedup outright.
Callback invariant: one eventful seek per output frame at the frame time, then K
suppressed sample seeks, then a suppressed restore to the frame time. The
restore matters because a suppressed seek still moves the playhead, so without
it the next frame's eventful seek would swallow the callbacks inside the shutter
window.
Scope of this deliverable: screenshot capture mode and PNG output. beginframe,
drawelement and JPEG encoding are rejected with a named error rather than
silently rendering unblurred. Video content is rejected for the same reason it
is a documented limitation: injected frames are extracted per output frame and
cannot follow a sub-frame seek.
Blend defaults to sRGB, matching AE's non-linearised 8-bpc working space and
the catalog component. Linear light is an explicit opt-in.
Samples are folded into a running average as they are captured rather than held
until the end, so peak memory is one accumulator plus one decoded sample instead
of all K: at 1080p that is a constant 41 MB rather than 141 MB at 16 samples and
540 MB at the 64 maximum, on every frame of the render.
PNG chunk CRC32 now has one owner in utils/crc32.ts, shared by the new encoder
and ffprobe's metadata reader. The encoder previously imported zlib's crc32 by
name, which throws at module evaluation on Node 22.0 and 22.1; ffprobe already
carried the guarded version and the comment explaining why. The sRGB transfer
function is likewise exported once and shared.
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
## What this does
Video encoders normally choose their own keyframe positions. A keyframe is a frame that can be decoded on its own, and you can only cut a video cleanly at one. The engine already has a flag, `lockGopForChunkConcat`, that forces a keyframe every `gopSize` frames so chunk files can be joined with `ffmpeg -c copy`. Today that logic is copy-pasted in two places inside `buildEncoderArgs` and does not exist at all in the streaming encoder.
This PR:
1. **Extracts the locked-GOP logic into three small shared helpers** in `chunkEncoder.ts`: `resolveLockedGopSize` (validation), `appendLockedGopArgs` (the `-g` / `-keyint_min` / `-sc_threshold` / `-force_key_frames` args), and `lockedGopCodecParams` (the x264/x265 param fragment). `buildEncoderArgs` now calls them instead of inlining the same block twice.
2. **Adds the same flag to the streaming encoder** (`StreamingEncoderOptions.lockGopForChunkConcat` + `gopSize`), which is the path most renders under four minutes actually use. It emits the identical args as the disk encoder, including `-bf 0` for h265 under a lock so B-frame reordering cannot reintroduce negative DTS at a boundary.
## What does not change
- Any render that does not set the flag produces a byte-identical ffmpeg arg list. The flag defaults to `false` everywhere.
- Nothing in the tree sets the flag yet except the distributed chunk worker, which already used the disk encoder. No user-facing behavior changes.
- GPU encoders, VP9 and ProRes streaming encodes continue to ignore the flag, exactly as `EncoderOptions` documents. An earlier revision of this PR made GPU encoders honor it; that was dropped because the path is unreachable until HLS lands and was verified only on videotoolbox.
## Why
Groundwork for `format: "hls"` in `@hyperframes/producer`. HLS packaging stream-copies the encoded video into fixed-length segments with `-f hls -c copy`, and the segmenter can only cut on keyframes the encoder already produced. Locking the GOP at encode time is what makes every segment land exactly on a 4-second boundary without re-encoding. The follow-up PR adds the format itself and should reject `--gpu` for `hls` in v1.
## Testing
- `chunkEncoder.test.ts`: the three helpers tested directly; existing locked-GOP and GPU no-op tests unchanged.
- `streamingEncoder.test.ts`: new `lockGopForChunkConcat` block covering default off, libx264, libx265, and invalid `gopSize`.
- Engine typecheck, oxlint, oxfmt clean. Full engine suite matches main's baseline (the 4 `ffprobe.test.ts` HDR fixture failures are pre-existing on `main`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Report HDR auto-promotion consistently from the render media set, including remote media, and make its regression coverage portable across supported paths.
Co-authored-by: heygengenesis[bot] <262951085+heygengenesis[bot]@users.noreply.github.com>
Co-authored-by: miguel.sierra <229591595+miguel-heygen@users.noreply.github.com>
Add regression coverage for finite SDR video slots that start past EOF in mixed HDR timelines, and verify lint exclusions with successful local duration probes.
Co-authored-by: heygengenesis[bot] <262951085+heygengenesis[bot]@users.noreply.github.com>
Co-authored-by: miguel.sierra <229591595+miguel-heygen@users.noreply.github.com>
Allow explicit finite non-looping video slots to use the existing held-final-frame extraction path when data-media-start is at or beyond source duration. Warn during project lint for local static slots that will use this behavior.
Co-authored-by: heygengenesis[bot] <262951085+heygengenesis[bot]@users.noreply.github.com>
Co-authored-by: miguel.sierra <229591595+miguel-heygen@users.noreply.github.com>
The shared, size-capped video-frame extraction cache evicts entries whose
LRU-clock sentinel hasn't been touched in over an hour. That clock was set
once, at cache-hit lookup time, and never again — so a render holding a
compiled-dir symlink (or a still-open partial writer dir) into an entry for
longer than an hour could lose it to a concurrent render's GC sweep on the
same shared machine, leaving its frame reads a dangling ENOENT with no
retry.
Every active video's captured frame now renews its cache entry's clock,
throttled to once per 5 minutes per directory. touchCacheDir renews both
signals gcExtractionCache actually reads: the directory's own mtime (which
gates a still-open partial writer dir) and the completion sentinel's mtime
(which gates a published entry).
Co-authored-by: Miguel Angel <miguel.sierra@heygen.com>
Chromium's CDP layer can refuse a screenshot capture outright ("Protocol
error (Page.captureScreenshot): Unable to capture screenshot") with no
timeout wording at all. classifyCaptureFailure() didn't recognize the
message, so it fell through to the fatal "authoring" bucket and neither
capture-retry path (the disk-capture retry loop, or the streaming-capture
pinned-fallback retry) ever attempted a retry — including under
--low-memory-mode, whose single-worker plain-screenshot capture never
engages either path's other retry conditions.
Adds the literal error text to classifyCaptureFailure()'s transient_browser
bucket, and threads the resulting classification into
shouldRetryViaPinnedFallback() as a routing-independent retry condition, so
the streaming path recovers on a fresh session the same way the disk path
already does for other transient browser failures.
Co-authored-by: Miguel Angel <miguel.sierra@heygen.com>
Review feedback on #3805.
captureFrameToBufferPipelined checked isRecoverableDrawElementError and
threw DrawElementCaptureError before reaching captureFrameErrorDiagnostics,
so the NCPR/canvas failures that now abort the whole attempt produced no
frame-error PNG/HTML/JSON bundle — the exact case worth debugging, and the
one the adjacent comment promised mirrored the serial path. The serial path
was unaffected because its own DrawElementCaptureError throw propagates
through captureFrameCore's outer diagnostics catch.
Run diagnostics first, then the recoverable wrapper. Bounded to at most one
bundle per attempt, since a recoverable error fails the whole attempt, and
captureFrameErrorDiagnostics self-catches, so a dead page cannot mask the
structural error the producer's fresh-page retry depends on.
Covered by a new test asserting the bundle lands for a recoverable pipelined
failure; verified by mutation (restoring the old order fails it).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(build): stop generated files from being read half-written during the build
Two unrelated pull requests kept failing CI for reasons that had nothing to do
with their changes.
Typecheck failed with "TS1002: Unterminated string literal" pointing at a
generated file. The root build compiles several packages at the same time, and
more than one of them regenerates files under a package's src/generated while a
sibling's tsc is already reading them. A plain write empties the file and then
refills it, so for a few milliseconds what is on disk is the first half of the
new file. Whichever build read it in that gap saw a truncated file and stopped.
It never happened locally because locally nothing else is reading.
Every generator now writes the new version under a temporary name and then
renames it over the target, which the filesystem does in one step. A reader
either gets the whole previous version or the whole new one; there is no moment
where it can see half of either. Content that has not changed is not rewritten
at all, so repeat builds no longer touch these files. Four generators were
affected and all four now go through one shared helper, which is where the rule
lives from now on.
The build also rebuilt the core package a second time, in parallel with the
packages that read it. That second rebuild was already redundant, and removing
it takes the writers out of the window entirely. Renaming is what makes the file
safe; dropping the duplicate build makes the build shorter as well.
Separately, a linter performance test failed on four of the last ten failed runs
on main. It timed a scan with a stopwatch and demanded the result come in under
two seconds; one run took 3.7s. That is a statement about how busy the shared
runner was, not about the code, because a stopwatch also counts the time the
machine spent running someone else's job. The test now counts only the processor
time this process actually used, and checks that the cost grows in step with the
input rather than against a fixed number of milliseconds. Measured on a machine
under load, the stopwatch ratio for the same input reached 45x while the
processor-time ratio stayed at 20x.
Both fixes come with a check that fails if the fix is removed.
* fix(build): keep the shared write helper inside the core package
The helper the generators call had been placed in the repo-root scripts folder.
Every container image that builds a package copies the packages folders whole
but cherry-picks root scripts one file at a time, so the image builds failed on
a missing module. The repo already shows both halves of that convention: the one
root script a package build imports has a matching copy line in the image, and
cross-package imports into the core package need none.
The helper now lives with the core package's other build scripts, and the one
generator outside that package reaches it the way the engine package already
reaches core.
* test(engine): keep the probe cache bound test off the filesystem
The eviction test wrote, stat'ed and deleted 129 temp files to exercise an
in-memory LRU rule. Its runtime tracked filesystem contention rather than
the code under test, and on a busy Windows runner the file churn alone
pushed a ~300ms test past the 5s timeout, failing unrelated pull requests.
Cache identity comes from stat, so the test now synthesises stat results
and never touches disk. While here, the test also asserts the LRU touch:
re-probing an entry before the bound is hit must keep it resident and
evict the next-oldest one instead. The previous shape never hit the cache
during the fill, so that branch was untested.
* test(lint): keep the scaling check inside the scanner's linear range
The CPU-ratio version sampled 320k characters, where the output string's
own growth dominates and the whole test cost seconds of CPU; on a shared
runner that tripped the default test timeout, the failure this change
exists to remove. Sample 10k and 80k characters instead, where 8x input
measures ~8x and a quadratic scan still measures 60x or more, and give
the test an explicit timeout so a slow runner reports the ratio.
* test(core): snapshot file identity through a descriptor
The before/after inode and mtime checks read the file through a path stat
and then exercised the writer on the same path, which reads as a
check-then-use race to static analysis. Snapshot through an open
descriptor instead; the assertions are unchanged.
The static-frame dedup predictor walked tween intervals from
window.__timelines but was blind to onUpdate callbacks — motion driven
from a timeline's onUpdate in a tween-free window was predicted static.
The verifier compounded the gap by seeking with suppressEvents: true,
so the onUpdate never fired and the frozen frame passed verification.
Two changes:
1. Predictor: when a timeline carries vars.onUpdate, mark its full span
as animated so those frames are never predicted static.
2. Verifier: seek with suppressEvents: false so the verification page
behaves identically to the capture page. The verification page is
already isolated (separate Page instance), so out-of-order event
side effects cannot corrupt sequential capture.
Fixes#3793
* test(producer): pin unpinned drawElement stall retry
* fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing (#3172)
* fix(engine): bound drawElement frames so a wedged renderer falls back instead of failing
A single drawElement frame could kill an entire render. On one comp
(caption-editorial-emphasis) drawElementImage returns normally and the renderer
then stops draining its task queue: the setTimeout(…, 0) that schedules
toDataURL never fires, the capture page.evaluate never settles, and 60s later
the stage watchdog fails the whole render. Deterministic on 152.0.7977.30,
always the same frame. Root-cause detail in PRINFRA-488.
Two gaps, both closed here.
1. Nothing below the 60s stage watchdog bounded a frame. Adds a per-frame
deadline (HF_DE_FRAME_TIMEOUT_MS, default 15000, 0 disables) around the WHOLE
frame operation, not just the drawElementImage call — the stall surfaces at
whichever page round-trip comes next, and was observed at both the capture
and the seek's background-image decode. Bounding one call missed it.
Deliberately NO per-frame screenshot fallback for this error: once the
renderer stops scheduling it is wedged for every subsequent round-trip on
that page. Measured — the screenshot fallback blew the same deadline. The
frame fails fast instead, so the recovery that actually works can run.
2. That recovery was ineligible. shouldRetryViaPinnedFallback only retried a
generic capture failure when the worker count was PINNED by the inversion or
the router; a comp that engaged drawElement on the ordinary single-worker
path had no whole-render fallback at all. A renderer stall is now retryable
on any routing — the failure is a property of drawElement itself, and the
retry re-renders on a fresh page via screenshot. Reported as
de_fallback_reason "de_renderer_stall" so it is distinguishable from
capture_error in telemetry, and counted per-session as deFrameTimeouts.
Verified end to end on the repro: previously exit 1 with "stalled: no frame
progress for 60000ms"; now the deadline fires, the render retries via
screenshot, and RENDER_OK with a valid 1920x1080 / 240-frame / 8.0s MP4.
Tests cover the predicate both ways plus the cross-package error match. Engine
1,481 and producer 587 green (audioPadTrim.integration flakes only under
full-suite parallel load — passes in isolation, and on clean HEAD it is green
too, so it is not from this change).
Refs PRINFRA-488
* fix(engine,producer,cli): count drawElement frame timeouts where they happen
`session.deFrameTimeouts` was incremented inside `captureFrameCore`'s catch,
but the deadline races that function from OUTSIDE it — `withFrameDeadline` is
applied by `captureFrameToBuffer`, and `DeFrameTimeoutError` is constructed
nowhere else. So the branch was unreachable: on a wedged renderer the counter
stayed 0, the engine-side stall log never printed, and
`CapturePerfSummary.deFrameTimeouts` reported 0 on every affected render. The
end-to-end recovery worked throughout — `isDeRendererStallError` matches by
name at the producer — only the observability was dead.
Moved the increment and the log into an `onTimeout` hook that fires from the
deadline itself, deleted the unreachable branch and the now-unused
`isDeFrameTimeoutError`, and threaded the count out to
`render_completed.de_frame_timeouts` so the rate is graphable apart from the
other `capture_error` fallbacks. A stall always costs a whole-render re-run,
which is why it deserves its own series.
Also corrected the `fallbackReason` docstring, which still listed only
`psnr | blank | oom | capture_error` after this branch added
`de_renderer_stall`.
`withFrameDeadline` is exported for a two-case fake-timer test: the deadline
fires and calls `onTimeout` once, and a resolving inner promise passes its
value through with no late timer. The first case fails if the counter moves
back inside the work promise.
(cherry picked from commit ec2b2b5b2a)
* fix(render): retry sequential screenshot stalls
* fix(render): bound parallel worker capture phases
---------
Co-authored-by: Vance Ingalls <vance@heygen.com>
* fix(producer): transport safe extraction failure metadata
* refactor(producer): generalize public error metadata
* test(producer): use vendor-neutral media hosts
The ENAMETOOLONG regression writes 150 real clip files and the mixer
existence-checks each one: ~58ms on Linux, but past vitest's 5s default on
the Windows lane. packages/engine sets no global testTimeout, so heavy
tests here carry an explicit one.
On timeout its abandoned async work kept calling the shared runFfmpegMock
after afterEach cleared it, so the next test saw 5 calls instead of 3 and
lost its queued once-implementations to the leak. mockReset stops an
aborted test from handing leftovers to the next one.
When a composition script throws during execution, the GSAP timeline
registration never arrives and pollSubCompositionTimelines times out.
Previously the render continued with a degenerate 2-frame output and
reported success — now it fails loudly.
Two changes:
1. Detect composition script runtime errors in the browser console
handler and feed them into scriptLoadFailures, triggering the
existing fail-fast path (same as script load 404s).
2. Make sub_timeline_script_failure a fatal warning in
applyRenderWarningPolicy, alongside audio_processing_failed.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(producer): assert render artifact duration and frame count before commit
Refuse to publish an artifact that is significantly shorter or has fewer frames
than the capture pipeline just reported. Adds a duration/frame-count gate on top
of the existing readable-non-empty check inside ArtifactTransaction.validate(),
keyed off the values the orchestrator already carries. Closes#3395.
* fix(producer): wire ffprobe frame count into the artifact duration probe
The frame-count gate added in #3395 accepts an expectedFrames value from
the orchestrator, but defaultArtifactDurationProbe was still returning
only durationSeconds - so the wire was half-built and the assertion
short-circuited on undefined for every real render. Forward meta.frames
from ffprobe so the field-packet case the issue names (container duration
correct, stream shorter) is actually caught by the frame-count check,
not just the duration one.
extractMediaMetadata now populates a new frames field from the video
stream's nb_frames tag, returning undefined when the demuxer did not
report one (fragmented MP4, malformed streams, muxes that require
-count_packets). Callers that gate on the count must treat undefined as
no answer; the assertion already does.
The previous CI run (#32589981916) cancelled shard-6 at the 1h job
timeout after bun install failed to extract the aws-cdk-lib tarball
mid-Docker-build - a cache flake, not a code regression. Pushing a
follow-up commit retriggers CI against the now-populated cache layer;
the regression should clear without further code changes.
---------
Co-authored-by: Santhi Prakash <b.santhiprakash@gmail.com>
`muxVideoWithAudio` passed `-avoid_negative_ts make_zero` unless the caller
set `preserveAudioPrimingEditList`. In practice the dominant path is an AAC
sidecar copied into mp4, where that flag is actively harmful: ffmpeg's
default is `auto`, which the mp4/mov muxers (AVFMT_TS_NEGATIVE) already
resolve to `disabled`. Forcing `make_zero` overrides the correct default,
discards the priming edit list the sidecar encode created, shifts the video
start_time forward by one AAC frame and writes an empty video edit at t=0 —
which edit-list-honoring players (QuickTime/Safari) render as a black first
frame.
Verified with ffprobe on a copy mux of a 30fps h264 mp4 and an AAC sidecar:
with `make_zero` video start_time 0.066000, elst: [media time -1,
dur 5940] + [media time 6000, dur 180000]
audio start_time 0.042993, elst: [media time -1, ...]
without (this fix) video start_time 0.000000, elst: [media time 6000,
dur 180000]
audio start_time 0.000000, elst: [media time 1024, ...]
The empty leading edit and the offset both disappear, and the audio keeps
its 1024-sample priming edit.
The flag is now never passed for a mux, in any mode. `preserveAudioPrimingEditList`
is part of the exported engine API, so it stays on `MuxVideoWithAudioOptions`
as `@deprecated` and no-op rather than being removed; the two internal callers
that set it (`assembleStage`, distributed `assemble`) drop it.
`buildEncoderArgs` and `streamingEncoder` still pass the flag for video-only
output and are deliberately left alone — those chunks are consumed as
intermediates, not as a delivered mp4/mov.
Fixes#3487
Co-authored-by: Alexandru Mincu <alex@mountsoftware.ro>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parent src-only scans skipped multi-format <video>/<audio> markup, so those
elements were never extracted, downloaded, or mixed and rendered blank/silent.
Lint now accepts a child <source src> as a resolvable media src.
lookupCacheEntry reported a hit purely on the presence of the
.hf-complete sentinel. The sentinel records that extraction finished,
not that the frames survived, so any per-file cleanup that empties the
directory leaves an entry that rehydrates with zero frames.
rehydrateCacheEntry then returns totalFrames: 0, the clip reaches the
coverage gate with nothing, and the render aborts with a message about
capture coverage. Because the poison is on disk rather than in the
composition, every later render of the project fails the same way with
nothing the user can change to fix it.
An entry now counts as a hit only when it carries the sentinel AND
still holds at least one frame file, so an emptied entry re-extracts.
The check is format-agnostic: a hit must be usable whatever extension
the frames carry.
Addresses the cache half of #3372.
compileTimingAttrs/injectDurations used parseFloat, so data-start="intro"
wrote a NaN data-end and extract preferred that over duration; parseNumeric
now skips the id-ref (parseVideoElements already resolves it).
collectRenderMedia's resolveHostWindow likewise read host data-start with
parseFloat, so chained sub-composition slots (data-start="hook") stacked at
0-2s and every scene after the first rendered black. It now resolves host
starts through the shared resolveReferencedStart, matching the media parsers.
Fixes#3361.
* fix(producer): fall back to screenshot capture on drawElement canvas-not-initialized
The fast-capture drawElement path only special-cased the "No cached
paint record" error to trigger a per-frame screenshot fallback; every
other error (including "drawElement canvas not initialized", seen at
frame 0 on some macOS/Chrome combinations) was rethrown, hard-failing
the whole render even though the docs promise automatic fallback on
incompatible compositions.
Extend the existing fallback branch (in both captureFrameCore and
captureFrameToBufferPipelined) to also catch canvas-not-initialized
errors via a shared isRecoverableDrawElementError predicate, with a
diagnostic message identifying which case triggered the fallback.
Closes#3423
Co-Authored-By: Miga <noreply@anthropic.com>
* fix(producer): address review — tighten error matching, audit batch path, add fallback-ratio guard
* fix(engine): add prepareFrameForCapture to batch screenshot fallback loop
* fix(engine): split canvas-not-initialized from composition-root-missing errors
drawElementService threw the same HF_DE_CANVAS_NOT_INITIALIZED error for
both !canvas and !root. Missing composition root (navigated/broken page)
was classified recoverable and fell back to pageScreenshotCapture, which
captured blank or wrong content silently.
Now:
- !root → HF_DE_COMPOSITION_ROOT_MISSING (not recoverable, hard fail)
- !canvas → HF_DE_CANVAS_NOT_INITIALIZED (recoverable, screenshot fallback)
Split applied at all 3 emit sites (serial, pipelined, batch).
Co-Authored-By: miga-heygen <miguel.sierra_miga@heygen.com>
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Co-authored-by: Miga <noreply@anthropic.com>
Root cause: the per-worker capture calls in captureFrameRange
(parallelCoordinator.ts) take no abort signal of their own, and only
checked `signal.aborted` BEFORE starting each frame — a no-op once a
worker is already awaiting an in-flight call. On WSL2, the native
drawElement/BeginFrame capture call can hang indefinitely at frame 0
with no error. The DE parallel-router's existing stall watchdog
(captureStreamingStage.ts) correctly fires `stallController.abort()`
after HF_DE_STALL_MS, but that abort had no way to reach a
worker already wedged inside a hung capture call — so
executeParallelCapture's Promise.all waited forever, the render hung
indefinitely, and the CLI's circuit breaker (which only runs after
executeRenderJob settles) never got a chance to trip.
Fix: race each per-frame capture call against the signal actually
firing (raceAgainstAbort), the same "can't cancel, only race" pattern
already used by the sequential capture path. Once the watchdog's abort
is observed, the wedged worker rejects, executeParallelCapture settles,
and the existing pinned-fallback retry / "reverted" outcome / circuit
breaker machinery (already correct) runs end to end.
Also widen the CLI breaker's trip condition from the literal string
"reverted" to "not a clean routed success", so any future non-success
outcome the observability layer records also latches the breaker
instead of silently falling through.
Closes#3441
Co-authored-by: Miga <noreply@anthropic.com>
* fix(core): harden audio FX and group identity
* fix(core): address audio group review feedback
* fix(core): align preview transport with grouped audio
* test(core): pin audio group gain ceiling
* fix(core): preserve solo bridge through stack
* fix(engine): harden grouped audio rendering
* docs(engine): explain grouped mix fallback invariant
* test(engine): allow grouped mixes to finish on Windows
## What
Adds a one-shot ffmpeg-psnr filter probe at drawElement session bootstrap.
When the resident ffmpeg is missing or lacks libpostproc (no `psnr` filter),
the capture-session router now force-fallbacks to the screenshot capture
path and emits a `de_gate_reason = "ffmpeg_no_psnr_filter"` telemetry
signal via the existing `render_complete` breakdown.
Also tightens `psnrForDiskSample`'s catch: infrastructure-class ffmpeg
failures (ENOENT, "No such filter") no longer silently skip the sample —
they abort the render so the safety net cannot fail-open post-preflight.
## Why
The drawElement self-verify safety net (parallelCoordinator's
`psnrForDiskSample` → `psnrDb`) shells to `ffmpeg -lavfi psnr`. If ffmpeg
is missing, or was compiled without libpostproc (so the `psnr` filter is
absent), every per-sample compare throws. The existing catch swallows the
error and returns `null` — callers treat that as "skip this sample" and
the render completes with the safety net inoperative.
Field signal (⭐ 9/10 CLI feedback, Slack ts=1787380767.210079,
hyperframes 0.8.7, darwin/arm64, tid=93ff9910-2207-45c2-bc1f-54c0b347d4fe):
> "host ffmpeg lacked psnr filter used by drawElement self-verification,
> but render completed."
The user's frames happened to be byte-identical so no visual damage
shipped — but the safety net silently wasn't running. Any future
compositor-damage bug on that host would have shipped straight through.
## How
Two-part fix, both in `packages/engine`:
1. New `utils/psnrFilterAvailability.ts` — cached probe that runs
`ffmpeg -hide_banner -filters` once per process and word-boundary-
matches `psnr` in the output. Any failure (ENOENT, non-zero exit,
timeout, unparseable output) returns `false`; never rejects.
2. Wired into `services/frameCapture.ts` `initDrawElementOrTransparentBackground`
right after the Chrome capability probe: when useDrawElement resolves
true and the preflight returns false, set
`session.deGateReason = "ffmpeg_no_psnr_filter"` (same low-cardinality
bucket every other DE gate uses; flows through `getCapturePerfSummary`
→ `render_complete.de_gate_reason` in PostHog), emit a stderr warning
naming what's missing, and call `routeToFallback()` — the same
fail-graceful shape as the SwiftShader / CSS-effect / at-risk-timeline
gates. Skipped under `HF_FORCE_DRAWELEMENT=1` (matches the diagnostic
knob's policy of bypassing every other gate).
Belt-and-braces: `psnrForDiskSample` now discriminates infrastructure-
class failures (ENOENT / "No such filter" / "Unknown filter") from
per-sample noise (readFile races, transient EPERM). Only the former
re-throw — per-sample noise still returns `null` (skipped sample). The
preflight normally catches this at bootstrap; the re-throw covers
ffmpeg-swapped-mid-render.
## Test plan
- [x] Unit tests added:
`packages/engine/src/utils/psnrFilterAvailability.test.ts` — mocked
`execFile` covers: `psnr` present → true; `psnr` absent → false; ENOENT
→ false; non-zero exit → false; result memoized + reset works;
substring-not-word-boundary → false.
- [x] Unit tests added:
`isFfmpegInfrastructureFailure` in
`packages/engine/src/services/parallelCoordinator.test.ts` covers
ENOENT, "No such filter", "Unknown filter", per-sample EACCES, parse
errors, null/non-object.
- [x] `bun run test` — `packages/engine/src/utils/psnrFilterAvailability.test.ts`
(6 tests) + `packages/engine/src/services/parallelCoordinator.test.ts`
(50 tests) + `frameCapture.test.ts` (26 tests) all pass. Pre-existing
ffprobe test failures (4) on the base commit are unrelated (missing PNG
fixture bytes — the file is 129 B on disk, likely LFS-stored).
- [x] `bunx tsc --noEmit -p packages/engine/tsconfig.json` — clean.
- [x] `bunx oxlint <files>` — 0 warnings, 0 errors.
- [x] `bunx oxfmt --check <files>` — clean.
Not covered here: an integration test that boots
`initDrawElementOrTransparentBackground` end-to-end. That path is
Puppeteer-driven and has no unit-scale bootstrap harness in the
repository — the pure preflight + pure discriminator coverage above are
what this PR can prove at the vitest layer.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
`places a delayed track on its authored start` timed out on the windows runner
and failed an unrelated PR, the second time this week an ffmpeg audio test has
done that.
The previous fix raised the budget in `audioMixer.grouping.test.ts`, which was
the file the symptom named. It was the wrong scope: that file was the only audio
suite with explicit timeouts at all. `audioMixer.level.test.ts` had none, so its
two real-ffmpeg tests ran on vitest's 5s default. The failing one takes ~137ms
locally, so the runner is not 36x slower — but 5s was never a budget anyone
chose for a full mix.
Applied to the suite rather than to each test, so there is one home for it, and
scoped to the ffmpeg-gated describe: the sibling parsing suites are pure and
should keep failing fast at 5s.
Headroom alone would only have delayed an undiagnosable failure. The ffmpeg
process timeout is 5 minutes by default, far above any test budget, so a stalled
mix could only ever surface as a bare "Test timed out" with no stderr and no
failing stage. Tests now cap it at 20s and assert through a helper that reports
`failures` instead of collapsing to `expected false to be true`.
Both claims verified rather than asserted: a deliberately 6s test now passes
where the 5s default would have killed it, and forcing the process timeout to
1ms reports `stage: "prepare", reason: "ffmpeg_timeout"` instead of a timeout.
Reviewing with whitespace ignored is much smaller: adding the third argument to
`describe` reindents the suite body, so 131/101 is really 34/4.
`a group FX chain fully cutting its members leaves an ungrouped track
untouched` timed out on the windows runner, failing an unrelated PR. The whole
file runs in ~4s locally and that test in ~1.1s, so 30s was not generous — but
the runner is roughly 10x slower and this test drives more ffmpeg than any of
its siblings, two full mixes plus a group FX chain. 30s was the tightest budget
in the package; 60s is what the rest of the ffmpeg-driven engine tests use.
Headroom alone would only have moved the same undiagnosable failure later,
because nothing here could report why. The production ffmpeg process timeout is
5 minutes, far above any test budget, so a stalled mix could only ever surface
as "Test timed out in 30000ms" with no stderr and no failing stage. Tests now
cap it at 20s, and the mix wrapper throws the recorded failures instead of
returning `success: false` into an `expect(...).toBe(true)` that reports
`expected false to be true` and discards the reason.
Verified by forcing the process timeout to 1ms: the failure goes from a 30s
wall-clock timeout to a 150ms error naming the stage, reason and element
(`stage: "prepare", reason: "ffmpeg_timeout", elementId: "a"`).
This does not explain the Windows stall itself, which I could not reproduce on
macOS. It makes the next occurrence report what it was doing.
audioMixer.grouping.test.ts spawns real ffmpeg per assertion and ran on
vitest's 5s default; on slow Windows runners the FX-chain and envelope
cases land right at the line and fail runs that touch nothing in the
engine. The other ffmpeg-bound engine suites (videoFrameExtractor)
already carry a per-test 30_000 timeout; this brings the grouping suite
in line.
Miguel Ángel