* fix(engine,producer): render media readers treat private project files as missing
A video, audio, GIF or HEIC src that names a file the file servers refuse
(git metadata, env, credentials, tool state, app records), as written or
through a link, is now treated as missing by every render-time reader:
the compile-time duration probe and keyframe advisory, GIF and HEIC prep,
the media drop step, the HDR probe, frame extraction and the audio mixer.
The engine's isPrivateUnderAssetRoots judges a path from every asset root
that holds it, so a render's own work folder inside the project's
.hyperframes/ keeps its downloaded and prepared files.
* fix(engine): probe refuses a text file that ffmpeg reads as text-art video
ffmpeg's tty and bintext demuxers turn a text file (such as .asc or .nfo)
into a video stream that draws the text into frames. Every probe now
refuses an input whose container is tty or whose video codec is ansi,
bintext, xbin or idf, as not media. Probing, frame extraction and audio
reads all go through the same probe, on every render. No demuxer for real
video or audio produces these, so local renders keep every other format.
* fix(engine,producer,cli): playlist entries and check probes skip private files
- A media playlist (HLS segment, key or map URI, ffconcat file) that names a
private file inside its own folder is refused as not media, like one that
names a file outside it.
- `hyperframes check` no longer probes a media source outside the project or
private in it: mediaPath returns null for such a file and the audio,
heard-clip and HDR checks skip it.
- The compiler's duration probe and keyframe check judge privacy from every
asset root, as the render-time readers do.
* fix(cli): check hears a WAV whose header misstates its byte rate, as render plays it
A PCM WAV whose header byte rate is wrong (some speech generators write 2x) sent the silence decode's input seek past the file's end, so every clip past the middle was reported audio_clip_silent. Render already cuts such a file by decoded time (#7541); the byte-rate rule now lives in parsers and the CLI decoder seeks after decoding starts for those files.
* test(cli): keep the misstated byte-rate case with the other single-clip checks
* style(cli): format the WAV test helper
* style(parsers): keep the byte-rate rule's comment within the line limit
* fix(engine): offset clips from a WAV with a wrong byte rate play the right sound to their end
A WAV header that states twice its real byte rate sent the mixer's input seek
to twice the clip's in-point: the clip played later words, then silence. Such a
source is now cut by decoded timestamps instead of by a byte seek.
* refactor(engine): let the byte-rate flag's name carry the seek rule
* fix(engine): read a video's length from its packets when its headers state none
* fix(engine): ignore the cut first line of a long packet scan and count N/A as a frame
* chore(engine): trim the packet scan comments and put its doc back on the function
* style(engine): format the packet duration test
* fix(engine): a video on a non-square pixel grid renders at its display shape
* chore(engine): drop comments the names already carry
* fix(engine): a rotated clip with a non-square pixel grid keeps its display shape
* test(engine): the frame probes end their options with -- like every ffprobe call
* fix(engine): late-starting videos play to their last frame
A video whose first frame is after timestamp 0 could end early (frozen
tail) or late: ffprobe's stream duration does not run from the first
frame in every container (fragmented MP4, Matroska). Such a video's end
is now read from its last packet, once, where its facts are probed.
* test(engine): the WebM-with-audio case measures its first frame from the file
* test(engine): the first-frame probe ends its arguments with the input file
* fix(engine): late-starting MP4 and WebM videos render every frame the preview shows
The browser counts an MP4 or WebM from timestamp 0, but FFmpeg seeks from the file's
earliest stream start. A file whose first frame comes after 0 (a fragment cut from a
longer stream, an edit-list offset, a WebM with a start offset) was extracted early and
short, so renders tripped the video coverage gate. Every video seek now maps media time
to FFmpeg's timeline, and the video begins where the browser shows it.
* fix(engine): audio from late-starting MP4 and WebM plays when the preview plays it
The mixer seeked and trimmed audio by FFmpeg's clock, which starts at the file's earliest
stream. In an MP4 or WebM whose streams all start after 0, the browser plays that audio
later, so the render heard it early by the same amount. The mixer now maps media time onto
FFmpeg's clock with the same offset frame extraction uses.
* fix(engine): a late-starting file's picture shows on the clock its audio plays on
* test(producer): vfr-screen-recording baseline shows each frame at its timestamp
* ci(windows): probe that a render opens no visible window
* ci(windows): the probe names what opened each window
* ci(windows): the probe spawns each binary alone, with and without detached
* ci(windows): the probe logs the processes started during the render
* ci(windows): the probe also spawns Chrome one consoleless hop down
* ci(windows): the probe logs each child started without windowsHide
* ci(windows): the probe logs every child process with its windowsHide and detached
* fix(engine): Chrome keeps the CLI's console on Windows, so a render opens no window
* fix(cli): every child the CLI starts on Windows is hidden unless it opts out
* ci(windows): the probe bisects Chrome's flags and keeps each command's output
* ci(windows): the probe names the Chrome process behind each window
* ci(windows): the probe loads a page in Chrome directly and one hidden hop down
* ci(windows): the probe prints the Chrome main command line and renders with the reporter's flags
* ci(windows): the probe narrows the CLI's Chrome arguments to the ones that open a window
* ci(windows): the probe launches the CLI's Chrome arguments one hidden hop down
* ci(windows): the probe launches Chrome through the engine's browser pool
* ci(windows): the probe varies what the engine passes to Puppeteer
* fix(engine): renders on Windows launch Chrome with a Puppeteer that keeps it attached
* refactor(cli): hideChildConsoles copies its arguments and keeps one comment line
* fix(engine): extraction waits out the decode up to a deep seek in a long-GOP video
* chore(engine): shorten the keyframe gap doc comment
* fix(engine): cancelling an extraction during the keyframe probe reports a cancel
* fix(engine): a segment stopped by another's failure reads as stopped, not cancelled
* fix: phone videos with an undecodable first audio track check, preview and render
* fix(engine): a rate-lane clip bakes its ramp from the decodable audio track
* test: phone-clip tests end ffprobe options before the input
* fix(producer): convert HEIC images to JPEG at compile time
Tiled HEIC photos probe as several video streams, so the media-type
preflight rejected them as image:video, and Chrome cannot decode HEIC.
Sniff the ftyp brands of every <img>, join and rotate the tile grid with
ffmpeg, verify the declared size, and swap in a cached JPEG.
* fix(producer): shorten the HEIC prep comments
* fix(producer): name host tool failures and keep a failed HEIC preview image as authored
Also: HEIC tests run in the integration lane; the primary grid is picked by default disposition.
* chore(producer): raise the package size budget for the HEIC prep
A VP8/VP9 source with no alpha tag has one frame decoded with the alpha-aware decoder when its
metadata is read; if that frame has no alpha plane, its frames extract as jpeg instead of png.
Tagged alpha, a decoded alpha plane, a failed probe, or a plan written without the field keep png.
* perf(engine): a remote source's tail decodes run many at a time, not one per half-core
* perf(producer): the plan logs where its extract stage spends its time
* perf(engine): a window that ends inside its source takes its frame count without a decode
* test(engine): pin the tail-decode slot test to the windows that decode
* style(engine): format the tail-probe window check
* test(engine): deferred counts match extraction on a B-frame NTSC source; VFR is not deferred
* style(engine): format the tail-probe tests
* perf(aws-lambda): a site stores each video's ffprobe JSON, and the plan reads it instead of probing
* test(aws-lambda): a site stores and hands out each video's probe
* style(aws-lambda): format the probe-facts upload and tests
* perf(engine): a site's stored probe also holds the final-frame timestamp the plan would probe for
* test(engine): drop an unused import
* test(aws-lambda): the stored probe holds the ffprobe JSON and the final frame
* test(aws-lambda): the stored probe holds the ffprobe JSON and the final frame
* perf(producer): a distributed chunk encodes while it captures
* perf(producer): the audio task logs where its mix spends its time
One line per audio stage: prepare, groups, mix, master, encode and the peak check, in ms.
* perf(engine): locked warmup ticks stop sleeping once the page is ready
* perf(engine): a window reaching a stored video's end takes its frame count from stored facts
* fix(producer): the cloud audio task logs its phases
* perf(engine): stored facts keep the last packet length so late-start windows skip the tail decode
* fix(engine): a probe spawns without waiting when no stored facts are declared
* chore(aws-lambda): shorten comments over the line limit
* fix(engine): warmup keeps pacing after a failed tick; stored final frames check ffprobe major
* test(engine): the recorded ffprobe major matches the build's own banner
* test(engine): the version-banner probes go through a named binary like the other spawns
* perf(engine): remote sources of a site keep their probe results for the whole render
* test(engine): probe a content-addressed URL once per render
* style(engine): format the audio probe split
* perf(aws-lambda): a render reads a video's audio from a small file stored beside it
A video's audio is interleaved with its pictures, so mixing it from the cloud read the whole video (2.5 GB for a 20-minute 4K film) at the Lambda's network speed. lambda sites create now stores each large video's audio packets as a separate object keyed by the video's hash; the audio task reads that when it exists.
* chore(producer): shorten the audio sidecar header comment
* fix(aws-lambda): no audio sidecar for a video whose audio starts late or that has several tracks
* test(producer): format the audio sidecar tests and end their ffprobe options with --
* fix(studio-server): keep transparency in proxies of VP8/VP9 alpha webm
A webm whose alpha lives in the VP8/VP9 side channel (alpha_mode=1,
pix_fmt yuv420p) was proxied to opaque H.264, and the VP8 proxy path
decoded it with ffmpeg's native decoder, which drops alpha. The alpha
rule and the alpha-reading decoder now live once in
@hyperframes/parsers/media-alpha, used by engine and studio-server.
* build(parsers): raise the packed size budget for the media-alpha subpath
parsers already sat at its packed budget on main; the shared alpha rule
adds 676 B of JS and 803 B of declarations (190958 B packed in CI).
* fix(engine): long 4K video exports no longer time out while frames are still coming
* test(engine): the paced extraction test keeps a wide margin on a loaded runner
* fix(engine): long phone-video exports probe each source once instead of once per cut
* fix(engine): a failed source probe is retried, and probes share the extraction slots
* test(engine): synthesize the tone-mapped concurrency fixture with known input colour
* perf(producer): distributed chunks download only the videos they show
A project video under compiled/ that chunks extract their own frames from
now goes only to the chunks that show it, like video-sources/ files. A
chunk skips the readiness wait for a deferred video it does not show, so
it no longer logs media_load_failed for a file it was never given.
* fix(producer): cloud site uploads pack the files their id hashes, symlinks included
* perf(aws-lambda): chunks read only the byte ranges of a video they need
A chunk that extracts its own frames from a deferred project video now
reads it from S3 by byte range through a short-lived pre-signed URL,
instead of downloading and hashing the whole object first. S3 attests the
object's stored SHA-256 before the URL is signed; every other artifact is
still downloaded and verified as before.
* perf(aws-lambda): upload project files once to a shared content-addressed area
* chore(producer): keep planV2 comment share at the base level
* fix(engine): verify the TLS certificate of https video inputs and never surface their signed query
* fix(aws-lambda): renders re-upload a site's expired files instead of failing
* perf(aws-lambda): the plan step probes project videos in place instead of downloading them
* fix(engine): https video reads reconnect and time out, and redaction covers wrapped causes
Signed-URL reads now pass -rw_timeout, -reconnect and -reconnect_delay_max (both Lambda builds accept them) and ffmpeg also -reconnect_on_network_error, so a transient S3 error or a stalled read fails fast or recovers instead of waiting out the process timeout. Query redaction now descends into error.cause.
* fix(aws-lambda): share entry bundles and ratchet package budget
* fix(aws-lambda): site ids uploaded before files.json still render
A site id from an earlier sites create has only sites/<id>/project.tar.gz.
renderToLambda now falls back to that tarball, which the plan step still
reads, and when neither key exists it fails before starting with both keys
named and the fix (run sites create again).
* fix(aws-lambda): renders fail before upload on a stack too old to read sites
A stack deployed before files.json sites untars the site file in its plan
step and fails after four retries. renderToLambda now reads the state
machine definition first and stops with STACK_TOO_OLD and the redeploy
command when the plan task does not list SITE_MANIFEST_INVALID, which new
SAM and CDK stacks now treat as terminal. The README's upgrade section
drops the planProtocol v1 fallback, which hits the same failure.
* docs(aws-lambda): document shared content storage and the upgrade path
The deploy guide still described the tarball HeadObject short-circuit. It
now covers the cas/sha256/ store and its 30-day lifecycle rule, the
STACK_TOO_OLD, SITE_FILES_MISSING and SITE_NOT_FOUND errors, that old site
ids keep working, and that every project uploads once more after the
upgrade. A test keeps a troubleshooting entry for each pre-submit error.
* fix(producer): chunks log each video they read by URL, without its query
A chunk reading a video straight from the bucket left no trace in its log.
It now logs one line per remote source with the path and the URL minus its
query string, so signatures never reach the log.
* fix(engine): ffmpeg errors never keep part of a signed URL's query
A process's stderr tail was trimmed before its URLs were redacted, so a
cut inside a signed URL left a query fragment the URL pattern no longer
matched, and the fragment reached the error. The stderr tail is now
redacted before every trim, and the redactor also strips a signing
parameter cut off from its URL.
* fix(aws-lambda): comments name the site manifest instead of a file path
* fix(engine): stderr redaction survives chunk splits, and every ffmpeg spawn uses it
The stderr tail was redacted on each chunk and then cut, so once a cut
severed a long URL's scheme, the rest of its signature in the next chunk
matched neither pattern. The tail is now kept raw with 2x slack and
redacted only when the text is produced, then cut to size. The unused
producer audioExtractor (its own spawn, unredacted stderr) is deleted and
the parity harness's PCM decode goes through ManagedChildProcess; a test
fails on any ffmpeg/ffprobe spawn in engine, producer or aws-lambda that
does not.
* chore(producer): record the audioExtractor deletion as intended
* fix(engine): drop a signature fragment the stderr tail cut from its URL
The raw tail is kept at twice the cap, but that cut can itself land inside a
signature, and when redaction shrinks the rest below the cap the fragment
survived. After a cut, the severed token is dropped before redaction.
* refactor(engine): name the severed-token drop instead of commenting it
* test(engine): a cut-off URL leaves nothing behind; aws-lambda size budget to measured
* fix(producer): chunk URL-read tests use main's frame list; producer size budget to measured
* chore(producer): size budget at the measured size, no headroom
* chore(producer): size budget back to measured x1.05, the check's own rule
* fix(engine): exports place sound and picture right when a source's audio and video start apart
* fix(engine): a late video stream plays to its end and distributed and HDR renders hide its lead
* test(engine): pin a late video stream's trimmed start, held tail and HDR trim
* fix(engine): recover audio duration from packet timestamps before clamping
An authored `data-duration` on an <audio> element could be silently
shortened when a source file's container-summary duration (mvhd/tkhd/mdhd)
undercounts the real audio length while the packet stream itself is
intact. Add `probeAudioDurationFromPackets()`, which re-derives the true
duration from the last packet's own pts_time + duration_time — figures
ffprobe computes from the stream's own timestamps, not the container's
summary header — and wire it into the compiler's existing clamp check as a
rescue attempt before accepting a shorter value. Works regardless of
codec/profile; verified against real HE-AAC/SBR fixtures and a
binary-patched "lying container" MP4 built for the regression test.
The rescue only fires for candidates the existing clamp predicate already
flagged (no added cost on the common path), runs in parallel across
candidates rather than serially, and requires the recovered duration to
exceed the existing clamp epsilon (absorbing the few milliseconds of
priming-delay drift raw packet timestamps carry relative to an
edit-list-corrected container duration) before overriding the shorter
value.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* fix(engine): move audio duration correction into extractAudioMetadata
The previous approach compared the compiler's own packet-scan probe
against the already-corrected duration returned by extractAudioMetadata,
which meant the correction only applied to one caller and the regression
fixture happened to exercise a pre-existing AAC-LC-specific refinement
instead of the new code path.
Move the fix to the single canonical place duration is derived: replace
the old AAC-LC-only packet-count refinement (a profile allowlist assuming
a fixed 1024-samples-per-frame constant) with a codec-agnostic mechanism
that decodes the final audio frame and reads its own timestamp plus real
sample count. This has no per-codec assumption, so it applies uniformly
regardless of codec or profile, and every caller of extractAudioMetadata
benefits automatically. The compiler's own clamp logic is now unchanged
from its pre-existing form.
Verified against a real, previously-untouched case: a FLAC-in-MP4 file
(a codec the old AAC-LC-only path never corrected) with its mvhd/tkhd/mdhd
duration atoms binary-patched to half their true value while every real
frame stays intact. Confirmed this reproduces on the prior code (by
literally substituting its actual content) and is fixed by the new code.
The decode probe reads CSV output (each line stands alone) rather than
JSON (a single top-level array), and retains only the output tail under a
small cap — a lying container's true duration can be arbitrarily larger
than its declared one, and JSON truncated from the front isn't valid JSON,
so an unbounded scan of a long file silently fell back to the wrong
duration once ffprobe's stdout cap was hit.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* test(producer): update style-7-prod golden for corrected AAC duration probe
The deleted AAC-LC packet-count refinement assumed every demuxed packet
decodes to exactly 1024 samples. For this suite's real audio fixture
(a 48kHz stereo AAC-LC MP4), that assumption overcounts: ffprobe reports
753 demuxed packets, but only 752 of them actually decode into a real
audio frame (verified via raw PCM sample count: 770048 samples / 48000Hz
= 16.0426666...s). The old mechanism's 753 * 1024 / 48000 = 16.064s was
therefore wrong; the new decode-based probe correctly derives ~16.043s
from the real final frame.
Regression-shards caught this as a compilation mismatch on the previous
commit. Verified the corrected golden against the real source: full
compilation, visual (0 failed frames), and audio (correlation 0.966)
checks all pass locally with the actual LFS-hosted golden video, and a
standalone compileForRender() check confirms all other audio-bearing
fixture suites in the repo are unaffected.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* perf(engine): seek the audio decode probe instead of scanning the whole file
probeDecodedAudioDuration ran a full, single-threaded decode of the
entire audio stream on every extractAudioMetadata call (once per audio
element in the HTML compiler, plus audio mixing/padding/encoding
steps). On real 30-minute files this took over a second, and could
approach ffprobe's 30s deadline on multi-hour files while holding a
shared probe slot the whole time.
Add -read_intervals so the probe seeks to ~1 second before the
container's already-known claimed duration and decodes only that
tail. This is safe because the correction only ever raises the
duration: an honest or over-claiming container's seek lands at or past
the real end and finds nothing extra, while an under-claiming
container (the only case this exists to fix) still has real frames
past the seek point. If the tail read finds nothing at all, fall back
to a full unseeked decode before giving up, protecting files whose
seek index is unreliable (an index-less or non-seekably-sourced
container) from silently missing a real correction — mirroring the
existing seek-then-fallback pattern already used by the sibling
final-video-frame probe in this file.
Also replaces the real-file regression test's runtime ffmpeg-spawning
fixture generation with two small, committed binary fixtures (an
honest FLAC-in-MP4 file and a copy with its container duration atoms
halved) plus the ffprobe-static package, so this test's real-file
proof actually runs in CI instead of being silently skipped when no
system ffmpeg is on PATH.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* refactor(engine): trim the audio duration probe comments to the invariant
* test(engine): cache tests queue the decode probe and count metadata probes only
* test(engine): probe for ffprobe with a literal argv so the argv contract classifies it
* test(producer): regenerate the speed-ramp golden for the single-encode mix
The audio probe now reads the mixed AAC as exactly 4.000 s instead of the
packet-count 4.032 s (which counted the priming frame), so the final mix
is stream-copied instead of re-encoded once more. Same waveform (sample
correlation 0.9996 vs the old golden), 3.1 dB less true-peak attenuation;
video frames are byte-identical.
* fix(engine): measure decoded audio from the stream start and trust it for raw ADTS
The decoded last-frame end is an absolute timestamp, so subtract the audio stream's start time
(format start as fallback) before comparing it with the start-relative container duration, and
start the tail read at start + duration - 1 s. MPEG-TS audio (1.4 s default start) and offset
MP4s were over-extended by their start time.
Raw ADTS (format aac) has no duration field; ffprobe estimates it from the opening bitrate and
overclaims after a quiet opening. Its decoded timestamps are sample-counted, so for raw ADTS the
decoded end now wins in both directions beyond the margin. Other containers stay upward-only so
an MP4 edit list is never extended.
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* fix(engine): declare SDR extracted frames as sRGB so Chrome does not darken footage
* fix(engine): encode SDR mp4 frames with the BT.709 matrix they are tagged with
Chrome hands the SDR encoder BT.601 full-range JPEG captures (or RGB PNGs).
The encoders only converted the range, so swscale kept the BT.601 matrix under
the BT.709 tag (ffmpeg 5.1 to 7.0: CSS #C83C28 delivered as #D44824) or
converted YUV to YUV through a low-precision step (ffmpeg 8+: #101010 delivered
as #0B0D0D). Decode to RGB with the capture matrix, then encode with BT.709.
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* fix(engine): tag SDR frames plain sRGB so every ffmpeg writes the sRGB chunk
ffmpeg writes a PNG sRGB chunk only for BT.709 primaries and writes cICP only
from 6.1. On ffmpeg 5.1 (Debian bookworm) a BT.601-tagged source therefore got
cHRM plus gAMA 1/2.2, which Chrome decodes as a pure 2.2 power: 0x10 shadows
rendered as 0x07. Declaring BT.709 primaries too gives the sRGB chunk on every
version, so Chrome shows the frame code values for every SDR source. The test
now reads the PNG chunks instead of ffprobe, which reports no transfer for an
sRGB chunk on ffmpeg 5.1.
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* fix(engine): extract SDR jpg frames as the BT.601 full range JPEG readers assume
The default frame format for opaque SDR sources is JPEG. ffmpeg converted the
BT.709 source to full-range YUV without changing the matrix, and every JPEG
reader (Chrome included) decodes BT.601, so footage shifted: 0xC83C28 showed as
0xBB2D29. The old mp4 encode made the inverse mistake and hid it; with the
encode fixed, it would reach the output. Decode through RGB with the source
matrix, then encode BT.601 full range, keeping 4:2:0.
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* fix(engine): round the SDR mp4 colour conversion accurately
swscale's default rounding moved saturated colours up to 3 codes (pure red 254
delivered as 251 on ffmpeg 5.1 to 7.0); accurate_rnd brings that to 1. The
chunk encoder now decides the HDR-or-SDR filter once, and the 10-bit GPU pad
test sets hdr, which its name always claimed.
* style(engine): keep the colour filter reasons in names and tests, not source comments
The comment ratchet caps a file's comment share: the new filter module and the
extractor carry their intent in constant names, and the encode test states why
the conversion goes through RGB.
* fix(engine): keep colour edges sharp through the SDR colour conversions
Going through RGB upsampled chroma with swscale's default filter, so ffmpeg 4.4
to 7.x smeared colour across edges by about 2 px (red 254 next to blue came out
8 inside the blue). Nearest-neighbour resampling maps the half-resolution chroma
back one to one, keeps the flat-colour results, and makes accurate_rnd redundant.
The pixel tests now also check a red to blue edge.
* test(producer): regenerate goldens whose colours the old mp4 matrix bent
These four fixtures compare at 40 to 45 dB, and their goldens were rendered
with the BT.601-under-BT.709 encode, so the corrected colours score 34 to 37 dB.
Regenerated in the Dockerfile.test image. Each new golden is closer to the
colour its page declares, e.g. #00d4aa was 00bcab and is now 00d3aa, and
#087443 was 006643 and is now 087241.
* fix(engine): keep VFR frame timing when extraction applies colour filters
ffmpeg before 6.1 ignores frame durations in CFR resampling once any -vf is
set, so a VFR window ending on a still lost frames. Filtered VFR extraction
now resamples in one process exactly as before and filters in a second.
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* fix(engine): encode raw SDR layered frames in BT.709
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* test(engine): keep the new colour tests inside the format, comment and argv checks
* fix(engine): keep HDR and unusual pixel formats out of the two-process VFR path
The raw handoff dropped HDR10 light levels and misread semi-planar formats, and
decoder options rejected some ffprobe colour names. Only SDR sources in formats
nut carries unchanged now take it, their colour tags come back through setparams,
and a process that cannot start releases the other.
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* fix(engine): read untagged footage with the matrix Chrome plays it with
Chrome decodes an untagged matrix as BT.709 from 720 coded lines up and as
BT.601 below; extraction assumed BT.601 at every size, so HD footage without
colour tags shifted hue against Studio's own preview.
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* fix(engine): untagged VP9 and AV1 footage keeps BT.601, as Chrome plays it
Chrome applies the 720-line rule only to codecs it decodes through ffmpeg
(H.264, VP8); untagged VP9, VP9 alpha and AV1 play as BT.601 at every size.
Also widens the raw-pipe pixel formats to all verified ones, and reports the
pipeline process that failed first by settle order, not by per-process clocks.
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
* test(producer): regenerate goldens for corrected page colours and untagged SD footage
* docs(engine): say which untagged codecs the matrix rule was measured on
---------
Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
When ffprobe omits the video stream's own `duration` field,
`extractMediaMetadata` falls back to the container duration
(`format.duration`), which includes the longest stream in the
container. Stock and looping clips commonly have an audio track
2x the video length, inflating `videoStreamDurationSeconds` and
causing the downstream coverage gate to expect twice the frames
FFmpeg can actually produce — the "50% cluster" failure pattern.
Cross-check with `nb_frames / fps` when the stream duration is
absent. If the frame-derived duration is >10% shorter than the
container, use it instead. This corrects the inflation at probe
time without requiring a second ffprobe pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(engine): classify a single-keyframe video as problematic, not healthy
analyzeKeyframeIntervals short-circuited to isProblematic:false whenever
it found fewer than 2 keyframes, but exactly one keyframe is the worst
case for seek performance, not a healthy result: every seek past frame 0
must decode the entire file.
The single-keyframe branch treats the rest of the video stream's own
duration (not a container duration, which a longer audio track can
inflate) as one uninterrupted GOP. ffprobe's frame timestamps are
absolute presentation timestamps rather than relative to stream start,
so a re-muxed or trimmed clip with a nonzero start time is normalized
the same way extractFinalVideoFrameTimestamp already does for the same
quirk. The cache key now includes both values, matching that sibling
function's convention, since a stale cache entry from a different
duration/start pair would otherwise win silently.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* refactor(engine): share stream-window normalization between keyframe check and final-frame lookup
extractFinalVideoFrameTimestamp and analyzeKeyframeIntervals each
independently computed the same stream-start fallback, finite guard, and
cache-key format from the same metadata shape. Extracted resolveStreamWindow
so the two can't drift out of sync with each other.
Also: match ffprobe's real trailing-comma csv output in one keyframe mock,
add a test at the exact 2s threshold boundary, and comment the
noUncheckedIndexedAccess-only fallback at the single-keyframe branch.
Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>
* style(engine): trim long comments to one-to-three lines
---------
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Deliverable one of issue #4010: average K sub-frame captures into each output
frame so rotation, scale and opacity blur the way After Effects does, which the
translation-only catalog component (#4008) cannot do. Opt in per render with
CaptureOptions.motionBlur; absent means the capture path is unchanged.
The blocker was quantizeTimeToFrame flooring every seek onto the output frame
grid, which collapsed all K samples onto one instant and made the averaged frame
byte-identical to the unblurred one. quantizeSeekTime now owns that decision at
all three quantize sites (runtime player, its no-timeline fallback, and the init
transport) and refines the grid by an integer subdivision count, which can only
contain the output grid, never move a frame-time instant off it. The finer grid
rounds rather than floors: a sample time round-trips through the page bridge as
a float, and past a few million ticks that error exceeds the floor epsilon.
Accumulation sits inside captureFrameCore AFTER the static-dedup early return,
so a static frame still costs one reused buffer and only moving frames pay K. It
is not a before-capture hook, which would have disabled dedup outright.
Callback invariant: one eventful seek per output frame at the frame time, then K
suppressed sample seeks, then a suppressed restore to the frame time. The
restore matters because a suppressed seek still moves the playhead, so without
it the next frame's eventful seek would swallow the callbacks inside the shutter
window.
Scope of this deliverable: screenshot capture mode and PNG output. beginframe,
drawelement and JPEG encoding are rejected with a named error rather than
silently rendering unblurred. Video content is rejected for the same reason it
is a documented limitation: injected frames are extracted per output frame and
cannot follow a sub-frame seek.
Blend defaults to sRGB, matching AE's non-linearised 8-bpc working space and
the catalog component. Linear light is an explicit opt-in.
Samples are folded into a running average as they are captured rather than held
until the end, so peak memory is one accumulator plus one decoded sample instead
of all K: at 1080p that is a constant 41 MB rather than 141 MB at 16 samples and
540 MB at the 64 maximum, on every frame of the render.
PNG chunk CRC32 now has one owner in utils/crc32.ts, shared by the new encoder
and ffprobe's metadata reader. The encoder previously imported zlib's crc32 by
name, which throws at module evaluation on Node 22.0 and 22.1; ffprobe already
carried the guarded version and the comment explaining why. The sRGB transfer
function is likewise exported once and shared.
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
* fix(producer): assert render artifact duration and frame count before commit
Refuse to publish an artifact that is significantly shorter or has fewer frames
than the capture pipeline just reported. Adds a duration/frame-count gate on top
of the existing readable-non-empty check inside ArtifactTransaction.validate(),
keyed off the values the orchestrator already carries. Closes#3395.
* fix(producer): wire ffprobe frame count into the artifact duration probe
The frame-count gate added in #3395 accepts an expectedFrames value from
the orchestrator, but defaultArtifactDurationProbe was still returning
only durationSeconds - so the wire was half-built and the assertion
short-circuited on undefined for every real render. Forward meta.frames
from ffprobe so the field-packet case the issue names (container duration
correct, stream shorter) is actually caught by the frame-count check,
not just the duration one.
extractMediaMetadata now populates a new frames field from the video
stream's nb_frames tag, returning undefined when the demuxer did not
report one (fragmented MP4, malformed streams, muxes that require
-count_packets). Callers that gate on the count must treat undefined as
no answer; the assertion already does.
The previous CI run (#32589981916) cancelled shard-6 at the 1h job
timeout after bun install failed to extract the aws-cdk-lib tarball
mid-Docker-build - a cache flake, not a code regression. Pushing a
follow-up commit retriggers CI against the now-populated cache layer;
the regression should clear without further code changes.
---------
Co-authored-by: Santhi Prakash <b.santhiprakash@gmail.com>
ffmpeg and ffprobe are console-subsystem binaries and Node defaults
windowsHide to false, so every spawn opened a visible console window on
Windows. A render shells out dozens of times across parallel workers,
which flashed a burst of windows across the user's desktop.
Applied at every production spawn site rather than only the two named in
the report, since they all share the cause: runFfmpeg, both gpuEncoder
probes, ffprobe, streamingEncoder, audioExtractor and the distributed
version check. windowsHide is a no-op on macOS and Linux.
The dev-only parity and regression harnesses are left alone; they never
run on a user's desktop.
Closes#3379
* feat(engine): stamp rendered files with hidden renderer provenance
* fix(engine,producer): re-assert provenance at every container writer
Review found that a no-audio MOV render still shipped untagged. The concat
step is the last container write on that path (mux is skipped without audio,
and applyFaststart only copies mov/webm), and the concat demuxer does not
carry the chunks' container metadata through.
The same hole applies to no-audio WebM, and to the in-process chunked encode
in chunkEncoder, not just the distributed assemble path. mp4 was masked
throughout because applyFaststart re-runs ffmpeg for that format and re-tagged
the output.
Tags the four remaining writers: the chunked-encode concat, and assemble's
single-chunk remux, concat and cfr re-encode.
Also corrects the trust claim. These are unsigned, freely writable keys, so a
present tag means the file claims to be HyperFrames output, not that
HyperFrames wrote it. Documented as an unauthenticated diagnostic hint rather
than an authenticity or attribution boundary.
Tests assert on the assembled file through the real assemble() path for both
mov and webm; both fail without the concat fix.
* test(engine): pin provenance through the in-process chunked concat
Review noted the distributed writers are mutation-pinned but the
encodeFramesChunkedConcat fix had no real-file regression of its own.
Encodes 70 frames at a 30-frame chunk size so the concat step actually runs,
then asserts the tags on the resulting no-audio mov. Fails without the concat
fix, passes with it.
* fix: bound HDR and video extraction resources
* fix: trim negative video extraction preroll
* fix: skip invisible video extraction windows
* fix: preserve negative-start loop and held tails
* fix: cap finite video slots to source duration
* fix: bound held-tail frame extraction
* fix: plan from playable video duration
* fix: preserve open-ended held video tails
* fix: resolve held tails from decoded frames
* fix: normalize final-frame probe timestamps
* fix: handle unseekable final-frame sources
* fix: dedupe final-frame probes per render
* refactor: clarify output dynamic range contract
Three issues in runFfprobe's process and stream handling.
A filePath of exactly "-" hung for 30 seconds. `--` stops option parsing,
so "-intro.mp4" is safe, but ffprobe rewrites "-" to `fd:` AFTER option
parsing and reads stdin — and stdin was an inherited pipe the parent
never writes to and never ends. The probe ran to the deadline and failed
with an empty diagnostic, because ffprobe never errored so stderr was
blank: 30010 ms and no message, against 28 ms for a normal missing-file
error. Rejected up front, and the child now gets stdio ["ignore", ...]
so no future invocation can block on stdin either.
stdout was decoded per chunk. `stdout += data.toString()` decodes each
64 KiB pipe chunk independently, so a multi-byte character straddling a
boundary became U+FFFD on both sides — verified: 200 KB of 3-byte
characters produced 15 replacements and a string 9 characters longer
than the source. -show_format output above ~64 KiB with non-ASCII tag
text returns silently mangled values, since JSON.parse still succeeds.
Now accumulated through StringDecoder.
Note on testing that one: U+FFFD is valid JSON string content, and
nothing on extractMediaMetadata's public surface exposes a tag value, so
there is no assertion that fails against the old implementation. Rather
than add a test that cannot fail, it is stated here and the bound below
is what the new test covers.
stdout was unbounded. stderr is capped by ManagedChildProcess but stdout
was not, and analyzeKeyframeIntervals emits one line per frame — an
all-intra ProRes proxy can produce an arbitrarily large string. Capped
at 8M characters, which real -show_streams JSON is nowhere near.
Tests: "-" rejected without spawning, the stdio shape, and the size
bound. Reverting the stdin guards fails 1. The first draft of the bound
checked before appending, so a single oversized chunk passed — the test
caught it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous gate was a HE-AAC DENYLIST, so every other profile still
got the 1024-sample formula. ffprobe reports codec_name "aac" for all of
them; the framing lives in the profile:
LC 1024 samples/frame <- the only one this maths fits
HE-AAC v1/v2 2048 output samples against a doubled sample_rate
LD 512
ELD 480
Main/SSR/LTP 1024 nominally, unverified here
xHE-AAC variable
LD and ELD therefore had their already-correct container duration
overwritten with a value 2x / ~2.13x too large, and an unknown or
missing profile fell through — so an unrecognised HE spelling preserved
the exact truncation the previous commit set out to close.
Now an affirmative match on LC. Skipping the refinement is harmless:
format.duration is already correct before it runs.
Tests: 11 non-LC profiles (including LD, ELD, xHE-AAC, empty and
unrecognised) assert the container duration is kept AND that the second
probe is not launched; LC still refines, with whitespace tolerated. The
pre-existing duration table asserted that an UNPROFILED "aac" stream
refines — the behaviour under review — so it now states LC explicitly
and adds an unprofiled row that must not refine.
Also strengthened the `--` separator test while it was failing: it
compared a flattened count of 3 across three spawns, which one call
emitting three terminators would satisfy. Now asserts the last two argv
entries per call.
Reverting the allowlist fails 8.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The packet-count probe is a refinement — durationSeconds is already
correct from format.duration before it runs — but it was written as if
it were load-bearing.
It could fail the whole call. No try/catch, and `-count_packets` demuxes
the entire container against runFfprobe's fixed 30s deadline, so a long
AAC file on slow or network storage timed out and extractAudioMetadata
rejected. htmlCompiler catches that under the comment "Source file has
no audio stream", returns duration 0, drops the audio element, and the
render ships silent with no warning. Now caught, keeping the container
duration.
It ignored the caller's AbortSignal. Only the first probe received it,
so aborting during the packet probe let the child run to completion and
the call resolved with full metadata after cancellation — while
audioPadTrim's comment claims the wrapper preserves cancellation. The
signal is forwarded, and an abort still propagates rather than being
swallowed as a refinement failure.
It halved HE-AAC durations. ffprobe reports codec_name "aac" for
HE-AAC v1/v2 as well — the marker is in the profile field — and with SBR
each packet carries 2048 output samples against the doubled output
sample_rate, so the 1024 assumption computed exactly half. A 10:00
podcast became 5:00 and htmlCompiler truncated the audio there. Gated on
profile, with `profile` added to FFProbeStream.
Tests: probe failure, junk output, three HE-AAC profile spellings (which
also assert the second probe is not attempted), and that plain AAC-LC is
still refined. Reverting the guards fails 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two paths the previous guards still let through.
Rounding could recreate Infinity after the finite check. `raw * 100`
overflows for a finite-but-huge rate — "1e307", "1e307/1" — so `rounded`
became Infinity and passed the positivity check, reaching exactly the
`-r Infinity` failure the finite guard exists to prevent. The rounded
result is now checked too.
The rational operands still used parseFloat. The plain-number path
switched to Number() so trailing garbage fails the whole string, but the
numerator and denominator did not, so "60fps/1", "60/1fps" and
"30garbage/1garbage" returned valid rates while the contract says
malformed frame rates fail closed. Both operands are now parsed strictly,
and an empty operand ("/", "/1", "30/") is rejected rather than coerced.
Tests: 8 malformed inputs and 3 overflow cases in the direct table.
Reverting either fix fails 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
parseFrameRate guarded its operands but not its result, so several
inputs produced values that are not usable frame rates — and nothing
downstream catches them, because callers use `meta.fps || 30`, which
only rescues 0 and NaN. Everything below was truthy and flowed into
buildEncoderArgs as `-r <value>` (rejected by ffmpeg mid-render) and
into frameCount arithmetic.
"1e308/1e-10", "2/1e-320" -> Infinity (finite operands, infinite quotient)
"-30/1", "30/-1", "-60" -> negative (sign never checked)
"30/1/2" -> 30 (parts.length !== 2 fell through)
"60fps" -> 60 (parseFloat stops at garbage)
Now: the quotient is checked rather than the operands, non-positive is
rejected, more than two parts is rejected, and the single-part path uses
Number() rather than parseFloat so trailing garbage fails the whole
string.
Separately, 2dp rounding collapsed any rate below 0.005 to exactly 0,
and the caller's 30fps default then re-encoded a 300-second 1/300-fps
timelapse as a ~1/30-second clip with frameCount 9000 for a 1-frame
file. Those floor to 0.01 instead.
parseFrameRate is now exported and tested directly. The previous table
drove it through extractMediaMetadata behind a spawn mock, costing a
vi.resetModules() plus a re-import of core's 238-file barrel per row
(74.9 ms vs 0.094 ms) — and 4 of its 7 rows produced identical values
against the pre-fix implementation, so it could not fail for the bugs it
existed to catch. The replacement fails 9 against that implementation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defects in how ffprobe output and the PNG fallback are combined.
The cICP fallback was unreachable. `ffprobeColorSpace ?? stillImageMeta
?.colorSpace` discarded the PNG result whenever ffprobe returned ANY
colour field — and ffprobe emits color_space "gbr" for every PNG,
including a plain rgb24 with no colour metadata. So on the build the
parser exists for (reports gbr, does not decode cICP) an HDR PQ PNG
resolved colorTransfer "" , isHdrColorSpace() returned false, and the
still graded SDR. Now merged per field.
hasAlpha's anchor bound to one alternative. In
/(^|[^a-z])yuva|rgba|.../ the `|` is looser than concatenation, so
(^|[^a-z]) guarded `yuva` and nothing else. The list also omitted abgr,
ya8, ya16 and ayuv64, and `gray[a-z0-9]*a` matched only gray8a/gray16a —
names FFmpeg renamed to ya8/ya16 in 2013, so dead against modern builds.
A ya8 grayscale-plus-alpha PNG reported hasAlpha:false, resolveFrameFormat
picked jpg and the overlay flattened to an opaque rectangle. Replaced
with the start-anchored form studio-server already uses, extracted as
exported pixelFormatHasAlpha so the test asserts the shipped predicate
rather than a copy of the pattern.
The PNG parse ran eagerly and was discarded. It sat before the first
await, so readFileSync plus the CRC walk executed for every file before
a single ffprobe was spawned — a caller fanning out over
composition.images with Promise.all serialised entirely: 12 4K PNGs took
2649 ms against 170 ms probe-only, 2.5 s of event-loop stall that also
blocks Puppeteer IPC. On the happy path the value was then thrown away.
Now lazily memoized behind the paths that actually consult it.
Tests: 18 pix_fmt cases against the real predicate. Reverting the regex
fails 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zlib.crc32 landed in Node 22.2.0, but engine and cli both declare
`"node": ">=22"` and the runtime gate is major-only, so 22.0 and 22.1
are supported. A NAMED import of a missing export throws at module
EVALUATION — ffprobe.ts would have failed to load at all on those
runtimes, before any PNG was touched, taking every probe with it.
Namespace import plus a capability check, with the previous
bit-at-a-time implementation retained as the fallback. Modern runtimes
keep the 210ms -> 1.3ms win; older ones keep working.
Raising the floor to >=22.2.0 was the alternative, but that is a
user-facing support change and does not belong in a PNG bug fix.
Tests: the same HDR PNG parses identically with the native export
absent, and a corrupt chunk still rejects on the fallback path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defects in the PNG metadata fallback, all introduced when the
cICP early return became an accumulator.
Corrupt trailing chunk nulls a good result. cICP must precede IDAT, so
continuing past it only visits chunks this parser ignores — while making
whole-file integrity a precondition for returning anything. A truncated
or bad-CRC chunk after cICP in an otherwise-good HDR PNG returned null,
and extractMediaMetadata then re-throws the ffprobe error it had
swallowed instead of using the fallback it just computed: the render
dies on a host without FFmpeg, or grades SDR on a build that does not
decode cICP. Now stops once dimensions and colour are known.
A second IHDR overwrote the dimensions. PNG permits exactly one, first,
but nothing enforced that here — a trailing [IHDR 1x1] replaced a real
3840x2160 and the producer laid out a one-pixel image. Anchored to the
first. The length guard was also `>= 8` against a spec length of 13,
which accepted a truncated header and read height out of the CRC bytes.
crc32 was hand-rolled bit-at-a-time and fed a Buffer.concat per chunk.
Since the walk no longer stops early it CRC'd whole files: 210 ms on a
12 MiB PNG, 647 ms on a 35 MiB 4K one, synchronously on the event loop,
plus ~11 MB of garbage per parse from concatenating a 4-byte type tag
onto every chunk. node:zlib's crc32 is native and takes a running seed,
so type and data hash in sequence with no copy. 210.28 ms -> 1.291 ms.
Tests: 5 regressions — corrupt-after-cICP, truncation after cICP,
second IHDR, short IHDR, and that a corrupt IHDR/cICP still rejects.
Reverting the break or the anchor fails 3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- parseFrameRate now rejects malformed ratios (e.g. "30/", "30/0") instead of NaN.
- Add "--" before file paths so names starting with "-" are not parsed as options.
- cICP PNG chunk no longer returns before IHDR supplies width and height.
- Add regression tests for option injection, frame rates, and cICP ordering.
Probes the rendered output for video and audio stream durations after
render and fails the test if they differ by more than 0.5s. Catches
mux-level truncation regressions like the ffmpeg -shortest bug (#1648)
where one stream gets silently cut short.
Runs on all non-png-sequence fixtures with audio — no new meta.json
field needed since this is a universal invariant, not a per-fixture
threshold.
## Problem
Windows renders commonly fail with environment errors before any real work starts:
- `Browser was not found at the configured executablePath (...chrome-headless-shell.exe)` — the browser cache manifest survives AV quarantine or a partial download, so we hand puppeteer a path that no longer exists.
- `[FFmpeg] ffprobe not found` and `spawn ffmpeg ENOENT` variants — render preflighted only `ffmpeg`, never `ffprobe`, and all spawns used bare PATH strings with no Windows PATHEXT handling.
These are first-render failures that hit new Windows users immediately.
## Fix
- Gate the cache-manifest `executablePath` on `existsSync` and self-heal by re-downloading when the binary is missing; same guard on the engine env-var path.
- New shared environment preflight (`packages/cli/src/browser/preflight.ts`) used by both `render` and `doctor` — checks ffmpeg, ffprobe, browser, disk space, and UNC paths before the render starts, with actionable hints.
- Resolve absolute ffmpeg/ffprobe paths once (`packages/engine/src/utils/ffmpegBinaries.ts`) and pass them to every engine spawn instead of relying on PATH.
- Map opaque Windows ffmpeg exit codes to actionable messages.
## Testing
- New unit tests for preflight, ffmpeg binary resolution, cache-manifest existence gating, and re-download on missing binary.
- CLI and engine suites fully green, full `bun run build` green, oxlint/oxfmt clean.
- Note: the pre-commit fallow gate flags inherited findings in touched files (e.g. `audioExtractor.ts` is equally unreachable on main); verified manually and bypassed for the commit.