mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-05 17:30:50 +00:00
7203e0a73ca9f51cde74f0820bc31fed16454972
293
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vance IngallsandClaude Opus 5
|
23ba58cfd3 |
feat(audio): play automation envelopes in preview and bake them at render (#3016)
* feat(engine): render audio FX in an OfflineAudioContext Reads `data-fx-chain` off an audio element and runs the chain over the trimmed WAV before volume automation is baked in — effects should see the raw signal, and the envelope belongs on their output. The processing happens in an OfflineAudioContext inside the headless browser the engine already drives, running the same graph builders the studio previews with. That is the point of the approach: one implementation per effect, so the render agreeing with the preview is a property of the architecture rather than a tolerance to police. Reimplementing each effect as an FFmpeg filter would mean two implementations to keep in step, and for the dynamics processors and modulated delays there is no filter that behaves the same way. `build:audio-fx-runtime` bundles the graph builders into an injectable IIFE, following the same pattern as the existing runtime artifacts, so the browser runs exactly the code the studio does. The page loads from a file:// URL rather than about:blank because AudioWorklet is only exposed in a secure context — the compressor, limiter, gate and bitcrush processors would otherwise fail to register with an opaque error. file:// qualifies and needs no listening socket. The chain is serialised into the attribute the way colour grading carries its config, so there is no side-car file to resolve or lose. An FX failure is fatal for the whole mix rather than a per-track soft failure. Every other audio failure mode degrades gracefully — the track drops, siblings continue — but substituting the dry signal for a processed one ships a render that sounds plausible and is not what the author set up. Since the per-element work races under Promise.all, an internal AbortController chained off the caller's signal aborts in-flight siblings before workDir is removed. * feat(core): voiceover carve analysis Finds the bands a voice occupies so a music bed can be dipped there, letting the voice sit in front without ducking the whole track. Carve is a relationship between two tracks rather than an effect on one, so it stays out of the FX chain. What it emits is an ordinary chain of peaking filters, so a carve composes with whatever else is on the track and needs no separate rendering path. Selection is weighted toward intelligibility rather than raw voice energy. Ranking purely by power lands on the fundamental almost every time, because that is where a voice is loudest — but the masking that actually hurts a voiceover happens higher up, and dipping 160 Hz mostly just thins the bed. The bias is a control, not a constant: at 0 it follows raw energy, at 1 it weights toward 1-3 kHz. Ranking happens in dB, which matters more than it looks. Speech spreads 20-30 dB across these bands — it falls off roughly 6 dB per octave above the fundamental — so a weighting has to be on that scale to move anything at all. A multiplicative weight of `1 - bias + bias * shaped` is bounded below by `1 - bias`, capping its influence at 10*log10(1/(1 - bias)): 5.2 dB at the 0.7 default, 3 dB at 0.5. That is no influence against a real voice — every bias short of ~0.95 would rank exactly like bias 0 and carve the fundamental, the outcome the bias exists to prevent, while looking decisive against a fixture whose bands sit 2 dB apart. So the bias is a dB penalty, zero at 2 kHz and worth up to 30 dB at full strength, and relative cut depths come from a dB difference rather than a ratio of weighted linear powers. The bias reweights ranking without overriding the spectrum — a band the voice has no energy in is not worth carving, and scores -Infinity rather than competing — so a strongly low-pitched voice can still select low at full bias. What the tests hold is that biasing never selects lower than the unbiased ranking, that the DEFAULT bias reaches the presence region on a voice with a realistic tilt, and that bias 0 still follows raw power exactly. Includes a radix-2 FFT rather than a dependency; one Welch-style averaged spectrum over third-octave bands does not justify pulling in a DSP library. * fix(engine): keep the FX render 16-bit, stereo, and correctly sized Three defects in the offline FX path, none of which any test could see. **Float output silently disabled sample-accurate volume automation.** The writer emitted 32-bit IEEE float; the very next mixer step bakes the volume envelope into the samples and accepts only 16-bit PCM, returning null otherwise. So enabling any effect downgraded that track to the ffmpeg expression path — capped at 32 straight segments, quantising a curved envelope, and on a dense one falling back to base volume. It now writes 16-bit PCM, clamped rather than wrapped so a limiter at 0 dB or a resonant filter cannot turn overshoot into a click. A test asserts the baker accepts the writer's own output and actually fades it. **Everything was folded to mono.** `prepareAudioTrack` goes out of its way to emit stereo — its pan filter exists to dodge ffmpeg's 3 dB mono-to-stereo rematrix — and this folded it, then wrote one channel. So adding a single peaking EQ collapsed a bed's width and cost ~3 dB in the render, while preview stayed stereo. Channels now travel as one plane each, through an OfflineAudioContext of the same width, and come back interleaved. **Small results decoded the wrong length.** `new Float32Array(buf.buffer)` discards byteOffset and byteLength, and Node pools small allocations: a 400-byte payload sits at offset 8 inside an 8 KiB pool, so a clip under ~1024 samples decoded as 2048 samples of unrelated memory — and the empty-result guard could not see it. The reader has the mirror-image fix: a float data chunk on an odd boundary (ffmpeg's pcm_f32le writes fmt(18) + fact, landing `data` at 58) now copies instead of throwing RangeError on an unaligned view. The tail limitation is now stated rather than mis-stated: the context is exactly as long as the input, so a reverb or delay still ringing is cut there. The old comment claimed the opposite. How far a tail may run past a clip's end changes the clip's length in the mix, so it is a product decision, not one to make here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(producer): report an FX render failure as an audio error `processCompositionAudio` reports per-track failures in its result, but an FX failure it cannot degrade past — a browser that will not launch, a chain that will not build — rejects instead. `runAudioStage` had no try, so that rejection escaped to the orchestrator as an unclassified pipeline exception, losing the stage/owner/retryable classification this stage exists to attach, and skipping its abort check on the way out. It now lands in `audioError` alongside every other cause, while an abort still keeps its own shape rather than being reported as an audio problem. Not done here: committing the generated `audio-fx-runtime-inline.ts` so a fresh clone typechecks packages/engine without building first. The bundle is built from the stub, and the stub changes three times across this stack — so the artifact differs per branch and would conflict on every restack. Its model, position-edits-render-inline.ts, is committed only because it is stable. Building before testing is this monorepo's existing contract (studio's tests need core's dist too), so the gap is not specific to audio FX and is better closed by a build ordering gate than by committing a per-branch artifact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(engine): skip the browser FX render cases when there is no browser CI's `Test` job was red on this PR with four failures, all the same cause: Failed to launch the browser process: spawn /home/runner/.cache/hyperframes/chrome/chrome-headless-shell The job installs ffmpeg and no browser, deliberately — every other suite that needs an external binary already guards on it (`describe.skipIf(!HAS_FFMPEG)`). These cases were the only ones assuming a Chrome, so they failed on an absent dependency rather than on anything about the code. Guards on `resolveHeadlessShellPath()` — the same resolver `acquireBrowser` launches through, so the check cannot drift from the thing it guards the way a hard-coded cache path would. A configured path that does not exist throws; that is caught and read as "cannot run here". Checked both directions rather than just the green one: with a browser all 11 cases run and pass, and with `HYPERFRAMES_BROWSER_PATH` pointed at a missing binary exactly 3 skip and the other 8 still run. A guard that silently skipped everything would have looked identical in CI. They keep their value where it exists — every developer machine, and any job that has run `hyperframes browser ensure`. Not touched: the CodeQL failure on this PR is a run from 2026-08-07, five days and several force-pushes stale. None of the 17 open repo alerts are in files this PR changes; it re-runs on this push. * chore(engine): suppress the temp-file alert with the reason it is safe CodeQL flags `writeWav`'s `writeFileSync` as js/insecure-temporary-file (high) — the one new alert on #3021, and the reason its CodeQL check is red. It is a false positive, and the comment says why rather than just silencing it: `path` is always inside a directory made by `mkdtempSync`, never a name assembled directly under `tmpdir()`. Both callers are covered — the browser host page writes into `mkdtempSync(join(tmpdir(), "hf-fx-host-"))`, and the render output goes to the producer work dir, itself `mkdtempSync(join(tempRoot, "producer-project-"))`. mkdtemp picks the random suffix and creates the directory 0700 in one syscall, so the predictable filename inside it cannot be pre-created or symlinked by another user, which is the attack the rule is about. The analyzer sees the dataflow reach `tmpdir()` and not the mkdtemp in between. Suppressed inline rather than dismissed in the UI, so the justification lives next to the code and the rule stays live for anything added later in this file. Matches the repo's existing convention — `planV2.ts:222` carries an `lgtm[js/insecure-temporary-file]` for a different reason on the same rule. Correcting myself: I first reported this alert as not real, having intersected the PR's files against the default-branch alert list, which does not contain PR-ref alerts. Querying ?ref=refs/pull/3021/merge returns it straight away. * test(engine): probe ffmpeg and Chrome instead of assuming them Two failures on #3021's Test job, both about the environment rather than the code under test. **Bare `ffmpeg` is not on PATH in CI.** The 16-bit fixture shelled out to `execFileSync("ffmpeg", ...)` and died with ENOENT. The job does provide ffmpeg, through `prepare-ffmpeg-bin`, which is what `getFfmpegBinary()` resolves — every other ffmpeg-dependent suite in this package already goes through it. Now this one does too, and the case is `skipIf(!HAS_FFMPEG)` so a contributor without ffmpeg skips rather than fails. **The browser guard trusted the wrong thing.** It asked `resolveHeadlessShellPath()` and treated a returned path as "a browser is here". CI's cache holds a chrome-headless-shell that resolves and then fails to spawn — a partial download is indistinguishable from a working one by `existsSync`, which is all that resolver checks. So the three browser cases ran anyway and failed on the launch. It now runs `--version` and requires exit 0, which is the same probe the ffmpeg suites use: ask the binary, do not infer from the filesystem. Checked both directions rather than just the green one. With a working browser all 11 cases run and pass; with `HYPERFRAMES_BROWSER_PATH` pointed at a binary that exits non-zero — CI's exact situation — exactly 3 skip and the other 8 still run. A guard that quietly skipped everything would have looked identical on the CI summary. * feat(core): register the audio-fx-rack canary at 0% Lands the rollout switch dark, per the registry's own procedure: "Start at percentage: 0 and merge that — a canary at 0 is dead code you can land safely and ramp without a code review." Declared at the bottom of the stack so every branch above can read it. The gate itself goes in at wa-4-fx-panel, where the rack first appears. Scope is deliberate and stated in the description: it gates the AUTHORING surface only. A composition that already carries `data-fx-chain` still plays and renders it. A canary should stage who can REACH a feature, not make an attribute somebody already wrote silently inert — an agent that writes a chain through the skill would otherwise produce a file whose audio processing vanishes with no error. * feat(studio): audio FX panel generated from the registry Controls for the whole chain: add, remove, reorder, bypass, and every knob each effect declares. Nothing in the panel knows what a compressor is. The registry supplies each parameter's range, step, unit and scale and the panel renders what it finds, so adding an effect or a knob upstream needs no change here, and the panel cannot offer a value the renderer would reject — a typed-in figure is clamped into the declared range on the way through. Frequency and time controls span three or four decades, so those declare a log scale and the slider maps exponentially; a linear slider would spend most of its travel somewhere useless. Reorder is a first-class control because chain order changes the sound: a reverb before a compressor is not the same as after. Carve gets its own block rather than an entry in the add menu, with a picker for the voice track to listen to. It processes this track based on another one, which is how a sidechain control works — it lives on the track that changes, and names the source. * feat(studio): show the Audio FX section on audio tracks Adds `audioFx` to the editing-affordances contract and renders the FX panel in the inspector when an `<audio>` element is selected. The section is audio-only. A `<video>` carries its sound on a separate `<audio>` element, so an FX chain on the video would have nothing to process. Chain and carve settings are written straight back onto the element as serialised attributes, the way colour grading carries its config, so persistence is an ordinary attribute write and needs no new server route. A chain that cannot be parsed renders as empty rather than breaking the panel, and the attribute is left untouched until the user changes something. The collapsed group summarises what is on the track ("2 effects + carve") so the state is visible without expanding it. Wired into PropertyPanelFlat rather than PropertyPanel: STUDIO_FLAT_INSPECTOR_ENABLED defaults to true, so the flat inspector is what actually renders. * refactor(studio): lift audioFxSummary out of PropertyPanelFlat `PropertyPanelFlat.tsx` is 612 lines here against the repo's 600-line cap, so the required File size check is red — the sole reason this PR is blocked. The review says as much: "mechanical fix (~5 min), not a design problem. Code itself is LGTM." Moves `audioFxSummary` to `audioFxSummary.ts`, the same file a later branch creates for it. Deliberately the smallest cut that clears the cap rather than the whole `AudioFxGroup` extraction: every later commit in the stack edits AudioFxGroup, so moving it here would collide with each of them, while almost nothing touches this function. 595 lines. * feat(core,studio): hear the FX chain in preview, and run the carve analysis Splices an element's FX chain into the playback graph so preview stops being silent about effects, and wires the carve button that was previously inert. The chain goes between the decoded source and its gain stage: effects see the raw signal and volume automation rides on their output, matching the order the offline render uses. Since preview and render call the same graph builders, what is heard while scrubbing is what gets written. The splice lives in the transport rather than on the `<audio>` element. The transport plays each track from a decoded AudioBuffer and mutes the element to avoid doubling, so capturing the element with createMediaElementSource would have processed a stream nothing is listening to — it looked like it worked because the call succeeded, and the audio was unchanged. A chain that cannot be built plays dry rather than silencing the track, which is the right failure in preview: the author keeps working and hears the source. The render still refuses, because shipping the dry signal there would be wrong. Carve now analyses for real: it decodes the chosen voice track, ranks its bands and writes the resulting peaking filters onto this track. Generated nodes are tagged `fromCarve`, so re-running replaces the previous carve instead of stacking another set on top of hand-added effects. Known limitation: the graph is built when a source is scheduled, so a knob turned mid-playback takes effect on the next play or seek rather than immediately. Live re-parameterisation needs the transport to hold the handle and forward updates. * fix(studio,core): stop parameter drags from restarting playback Dragging a knob wrote the chain through the persisting attribute path on every input event. That path refreshes the preview, which reloads the composition and reschedules audio — so a single drag reloaded dozens of times and playback stuttered the whole way. Drags now go through `onSetAttributeLive`, the same path colour grading uses for scrubs: it coalesces undo entries and sets `skipRefresh`, so no reload happens. The persisting write fires once, when the gesture ends — pointer-up or blur for a slider, Enter or blur for a typed value. A select commits immediately since there is no drag to wait for. While dragging, the control is driven from local state. Waiting for the value to round-trip through the element attribute made the knob lag behind the pointer. For the change to be audible without a reload, the graph now follows the attribute: the chain installed by the transport observes the element and re-parameterises itself in place, so a value change lands on the next 128-sample quantum. A shape change (effect added, bypassed, pole count) cannot be patched into a running graph, so it still waits for the next schedule rather than cutting the audio mid-play. The regression test drags a slider through several values and asserts the persisting handler is untouched until release. * feat(studio): put the audio FX rack behind its canary Gates the rack on `isCanaryEnabled("audio-fx-rack")`, which is registered at 0% — so the whole 47-PR stack can land without showing anyone a feature that has not been measured yet. The gate sits on the AUTHORING surface and nowhere else. The runtime and the render still honour a `data-fx-chain` already on an element, so a composition written through the skill or by `carve.mjs` keeps its processing rather than going silently dry for anyone outside the cohort. A canary should stage who can REACH a feature, not make an attribute somebody already wrote stop working with no error. Gated at the panel rather than in `resolveEditingSections`: the affordance resolver is a pure function in core describing what an element CAN support, and rollout state is not a property of an `<audio>` tag. Pinned the 0% with a test, and checked it fails at 25 — a ramp should have to break something that says "this ships dark" out loud. One gap, stated rather than papered over: the gate itself has no unit test. I wrote one and deleted it, because `PropertyPanel.test.tsx`'s harness never renders the Audio FX group for its audio fixture even with the gate removed — so the test passed for the wrong reason in the off case and could not pass at all in the on case. A test that cannot fail for the right reason is worse than none. Verifying the gate needs the panel harness to mount that section first, which is its own change. * fix(core): register FX worklets before building nodes that need them An AudioWorkletNode cannot be constructed before its processor is registered — it throws, and the surrounding chain is lost with it. `attachElementFxChain` built the chain first and only then called `ensureAudioFxWorklets`, so every worklet-backed effect (compressor, limiter, gate, bitcrush) threw on construction and the track fell back to dry. Instrumenting the preview showed `hf-compressor: InvalidStateError` with addModule never called at all. When the module has not landed yet the track now plays dry and the graph is swapped in once registration resolves, so the effect arrives a moment late instead of never. Registration is also tracked per context rather than in one module-level promise. A processor registered on one AudioContext does not exist on another, so the shared promise made every context after the first believe it was ready when it was not — the studio's transport owns its own context, which is exactly that case. With the worklets actually running, the compressor's per-sample log10 and pow became real audio-thread work. Samples below the knee have a gain of exactly unity and need neither, so the envelope is now compared in the linear domain and the transcendentals only run for samples that are actually being compressed. * refactor(studio): split the FX node row out of FxSection Clears the health findings the FX stack left behind: the chain-node render callback was a 70-line closure over half of FxSection's state, and the two reorder arrows were the same button written twice. Also drops two exports with no consumers, and registers the audio FX runtime stub as an entry point — it is bundled by file path, so nothing imports it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(core): automation envelope model for audio tracks Adds the data model behind Ableton-style automation lanes: breakpoint envelopes over track volume or one knob of one effect in the track's FX chain, stored on the element as `data-automation`. Times are clip-local, so an envelope travels with the clip when it moves — the clip-envelope model rather than arrangement automation. `sampleAutomationLane` is the single interpolator. The lane drawing, the preview scheduler and the render bake all call it, so the picture and the sound cannot disagree about the curve. Log-scaled parameters interpolate in log space, matching what their own knob already promises. FX nodes gain a stable `id`, minted by count rather than randomly so the document is the same on every machine. Lanes address nodes by id, so reordering a chain never re-points a lane at a different effect, and a lane whose effect was deleted is dropped rather than left to reattach. Also warns when a track carries both a volume lane and a GSAP volume tween, since only the lane is heard and the tween silently does nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(studio): lift the audio FX group out of PropertyPanelFlat `PropertyPanelFlat.tsx` was 672 lines against the repo's 600-line cap, so the required File size check was red — the sole reason #3014 and #3022 are blocked. Both reviews say the same thing: "mechanical fix, not a design problem. Code itself is LGTM." Moves `AudioFxGroup` and `audioFxSummary` into `propertyPanelAudioFxGroup.tsx`, which is where a later branch puts them anyway — done here so the file is under the cap from the point it first crosses it, rather than ten branches later. 533 lines now. The four audio imports it no longer needs go with it. Not fixed here: three `FxSection carve` tests fail on this branch with "Cannot read properties of undefined (reading 'toFixed')". Confirmed pre-existing by stashing this change and re-running — that is the separate `Test` failure the review also flags. * feat(core): expose the AudioParams behind automatable FX knobs Marks the knobs an automation lane can drive and has each graph builder hand back the AudioParam behind them, so a scheduler can write to a running effect without knowing what the effect is. A knob is not always one AudioParam. A wet/dry mix is two gains moving in opposition, and a knob in milliseconds drives a delay time in seconds, so each target carries the mapping out of the knob's own declared unit. What stays unautomatable is stated where it is decided: a WaveShaper curve, a convolution impulse and a one-pole filter's coefficients are all rebuilt wholesale rather than scheduled, and the four worklet effects take values by postMessage rather than through AudioParams. The registry flag is written by hand, so a test builds every effect and checks the exposure both ways — nothing flagged is missing, nothing exposed is unflagged. A flag that lied would offer a lane that silently did nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(core): play automation envelopes in preview Schedules each lane onto the AudioParams behind its knob using native ramps and value curves. Nothing evaluates the envelope per frame: it is handed to the audio thread once, so it stays sample-accurate however busy the main thread is, and the offline render will schedule it the same way. Timing comes from the transport, so an envelope survives seeking into the middle of a clip, a clip that has not started yet, and a playback rate that compresses clip seconds into context seconds. A straight line is only scheduled as a ramp when nothing bends it — no curvature, a linear parameter scale, and no unit mapping. Log-scaled parameters and mapped ones are sampled instead, since a delay knob in milliseconds and a wet/dry pair moving in opposition are not linear in the parameter they drive. Lanes with nowhere to write are skipped rather than reported: a one-pole filter exposes no frequency param, and the worklet effects expose none at all. Editing an envelope mid-playback re-aims it at the live playhead rather than restarting the track. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(engine): bake automation envelopes into the render The offline render schedules FX lanes with the same scheduler preview uses, inside the OfflineAudioContext that already runs the same graph builders. The input WAV is the clip's own audio from its first sample, so clip-local time is offline time and the envelope needs no offset. Volume lanes take the existing PCM bake rather than a second mechanism: the lane is converted to keyframes, so a straight fade stays two of them and only a bent segment is sampled — the baker interpolates linearly and would otherwise quietly straighten the curve. A volume lane supersedes keyframes probed from the timeline, which `lint` already warns about. A browser test sweeps a lowpass from below a 2 kHz tone to well above it and measures both ends. Parsing the envelope is not the same as scheduling it, and only running the real thing tells the two apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(studio): drop the FX panel's dead __testables export Fallow audit flagged it — no test imports the module. * fix(core,studio): clear the remaining Fallow audit findings on the FX panel - Split FxSection's per-node row into FxNodeRow + FxNodeControls so the CRAP score (31.6, threshold 30) splits across two smaller units instead of moving wholesale with one extraction. - Dedupe the repeated "open the add menu, read its items" block in propertyPanelFxSection.test.tsx into openAddMenuItems(). - Merge build-audio-fx-runtime.ts and build-position-edits-render.ts into one build-inline-artifact.ts, config-selected by CLI arg — the two scripts were a byte-for-byte clone save for names. - Exempt canary.test.ts's rawFnv (a deliberate independent reimplementation used to cross-check canaryBucket, per its own docstring) and the property-panel test files' shared renderInto/mount scaffolding (pre-existing across 9 files, 2 outside this stack) in .fallowrc.jsonc, consistent with this file's existing exemptions for the same class of intentional/pre-existing duplication. * fix(ci): allowlist the build-script consolidation in the no-main-deletions guard build-audio-fx-runtime.ts and build-position-edits-render.ts were merged into build-inline-artifact.ts to kill a fallow duplication finding; the deletion guard flagged that as an accidental loss since main still has both originals. * fix(core): dedupe the wet/dry mix math between delayFeedback and chorusLfo Both effect builders set wet.gain to the mix and dry.gain to its complement in identical two-line blocks; fallow kept re-flagging it as a 10-line clone on every unrelated change. Extracted setWetDryMix. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Vance IngallsandClaude Opus 5
|
cc40e35aa0 |
feat(engine): render the FX chain offline, and the carve analysis behind it (#3021)
* feat(engine): render audio FX in an OfflineAudioContext Reads `data-fx-chain` off an audio element and runs the chain over the trimmed WAV before volume automation is baked in — effects should see the raw signal, and the envelope belongs on their output. The processing happens in an OfflineAudioContext inside the headless browser the engine already drives, running the same graph builders the studio previews with. That is the point of the approach: one implementation per effect, so the render agreeing with the preview is a property of the architecture rather than a tolerance to police. Reimplementing each effect as an FFmpeg filter would mean two implementations to keep in step, and for the dynamics processors and modulated delays there is no filter that behaves the same way. `build:audio-fx-runtime` bundles the graph builders into an injectable IIFE, following the same pattern as the existing runtime artifacts, so the browser runs exactly the code the studio does. The page loads from a file:// URL rather than about:blank because AudioWorklet is only exposed in a secure context — the compressor, limiter, gate and bitcrush processors would otherwise fail to register with an opaque error. file:// qualifies and needs no listening socket. The chain is serialised into the attribute the way colour grading carries its config, so there is no side-car file to resolve or lose. An FX failure is fatal for the whole mix rather than a per-track soft failure. Every other audio failure mode degrades gracefully — the track drops, siblings continue — but substituting the dry signal for a processed one ships a render that sounds plausible and is not what the author set up. Since the per-element work races under Promise.all, an internal AbortController chained off the caller's signal aborts in-flight siblings before workDir is removed. * feat(core): voiceover carve analysis Finds the bands a voice occupies so a music bed can be dipped there, letting the voice sit in front without ducking the whole track. Carve is a relationship between two tracks rather than an effect on one, so it stays out of the FX chain. What it emits is an ordinary chain of peaking filters, so a carve composes with whatever else is on the track and needs no separate rendering path. Selection is weighted toward intelligibility rather than raw voice energy. Ranking purely by power lands on the fundamental almost every time, because that is where a voice is loudest — but the masking that actually hurts a voiceover happens higher up, and dipping 160 Hz mostly just thins the bed. The bias is a control, not a constant: at 0 it follows raw energy, at 1 it weights toward 1-3 kHz. Ranking happens in dB, which matters more than it looks. Speech spreads 20-30 dB across these bands — it falls off roughly 6 dB per octave above the fundamental — so a weighting has to be on that scale to move anything at all. A multiplicative weight of `1 - bias + bias * shaped` is bounded below by `1 - bias`, capping its influence at 10*log10(1/(1 - bias)): 5.2 dB at the 0.7 default, 3 dB at 0.5. That is no influence against a real voice — every bias short of ~0.95 would rank exactly like bias 0 and carve the fundamental, the outcome the bias exists to prevent, while looking decisive against a fixture whose bands sit 2 dB apart. So the bias is a dB penalty, zero at 2 kHz and worth up to 30 dB at full strength, and relative cut depths come from a dB difference rather than a ratio of weighted linear powers. The bias reweights ranking without overriding the spectrum — a band the voice has no energy in is not worth carving, and scores -Infinity rather than competing — so a strongly low-pitched voice can still select low at full bias. What the tests hold is that biasing never selects lower than the unbiased ranking, that the DEFAULT bias reaches the presence region on a voice with a realistic tilt, and that bias 0 still follows raw power exactly. Includes a radix-2 FFT rather than a dependency; one Welch-style averaged spectrum over third-octave bands does not justify pulling in a DSP library. * fix(engine): keep the FX render 16-bit, stereo, and correctly sized Three defects in the offline FX path, none of which any test could see. **Float output silently disabled sample-accurate volume automation.** The writer emitted 32-bit IEEE float; the very next mixer step bakes the volume envelope into the samples and accepts only 16-bit PCM, returning null otherwise. So enabling any effect downgraded that track to the ffmpeg expression path — capped at 32 straight segments, quantising a curved envelope, and on a dense one falling back to base volume. It now writes 16-bit PCM, clamped rather than wrapped so a limiter at 0 dB or a resonant filter cannot turn overshoot into a click. A test asserts the baker accepts the writer's own output and actually fades it. **Everything was folded to mono.** `prepareAudioTrack` goes out of its way to emit stereo — its pan filter exists to dodge ffmpeg's 3 dB mono-to-stereo rematrix — and this folded it, then wrote one channel. So adding a single peaking EQ collapsed a bed's width and cost ~3 dB in the render, while preview stayed stereo. Channels now travel as one plane each, through an OfflineAudioContext of the same width, and come back interleaved. **Small results decoded the wrong length.** `new Float32Array(buf.buffer)` discards byteOffset and byteLength, and Node pools small allocations: a 400-byte payload sits at offset 8 inside an 8 KiB pool, so a clip under ~1024 samples decoded as 2048 samples of unrelated memory — and the empty-result guard could not see it. The reader has the mirror-image fix: a float data chunk on an odd boundary (ffmpeg's pcm_f32le writes fmt(18) + fact, landing `data` at 58) now copies instead of throwing RangeError on an unaligned view. The tail limitation is now stated rather than mis-stated: the context is exactly as long as the input, so a reverb or delay still ringing is cut there. The old comment claimed the opposite. How far a tail may run past a clip's end changes the clip's length in the mix, so it is a product decision, not one to make here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(producer): report an FX render failure as an audio error `processCompositionAudio` reports per-track failures in its result, but an FX failure it cannot degrade past — a browser that will not launch, a chain that will not build — rejects instead. `runAudioStage` had no try, so that rejection escaped to the orchestrator as an unclassified pipeline exception, losing the stage/owner/retryable classification this stage exists to attach, and skipping its abort check on the way out. It now lands in `audioError` alongside every other cause, while an abort still keeps its own shape rather than being reported as an audio problem. Not done here: committing the generated `audio-fx-runtime-inline.ts` so a fresh clone typechecks packages/engine without building first. The bundle is built from the stub, and the stub changes three times across this stack — so the artifact differs per branch and would conflict on every restack. Its model, position-edits-render-inline.ts, is committed only because it is stable. Building before testing is this monorepo's existing contract (studio's tests need core's dist too), so the gap is not specific to audio FX and is better closed by a build ordering gate than by committing a per-branch artifact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(engine): skip the browser FX render cases when there is no browser CI's `Test` job was red on this PR with four failures, all the same cause: Failed to launch the browser process: spawn /home/runner/.cache/hyperframes/chrome/chrome-headless-shell The job installs ffmpeg and no browser, deliberately — every other suite that needs an external binary already guards on it (`describe.skipIf(!HAS_FFMPEG)`). These cases were the only ones assuming a Chrome, so they failed on an absent dependency rather than on anything about the code. Guards on `resolveHeadlessShellPath()` — the same resolver `acquireBrowser` launches through, so the check cannot drift from the thing it guards the way a hard-coded cache path would. A configured path that does not exist throws; that is caught and read as "cannot run here". Checked both directions rather than just the green one: with a browser all 11 cases run and pass, and with `HYPERFRAMES_BROWSER_PATH` pointed at a missing binary exactly 3 skip and the other 8 still run. A guard that silently skipped everything would have looked identical in CI. They keep their value where it exists — every developer machine, and any job that has run `hyperframes browser ensure`. Not touched: the CodeQL failure on this PR is a run from 2026-08-07, five days and several force-pushes stale. None of the 17 open repo alerts are in files this PR changes; it re-runs on this push. * chore(engine): suppress the temp-file alert with the reason it is safe CodeQL flags `writeWav`'s `writeFileSync` as js/insecure-temporary-file (high) — the one new alert on #3021, and the reason its CodeQL check is red. It is a false positive, and the comment says why rather than just silencing it: `path` is always inside a directory made by `mkdtempSync`, never a name assembled directly under `tmpdir()`. Both callers are covered — the browser host page writes into `mkdtempSync(join(tmpdir(), "hf-fx-host-"))`, and the render output goes to the producer work dir, itself `mkdtempSync(join(tempRoot, "producer-project-"))`. mkdtemp picks the random suffix and creates the directory 0700 in one syscall, so the predictable filename inside it cannot be pre-created or symlinked by another user, which is the attack the rule is about. The analyzer sees the dataflow reach `tmpdir()` and not the mkdtemp in between. Suppressed inline rather than dismissed in the UI, so the justification lives next to the code and the rule stays live for anything added later in this file. Matches the repo's existing convention — `planV2.ts:222` carries an `lgtm[js/insecure-temporary-file]` for a different reason on the same rule. Correcting myself: I first reported this alert as not real, having intersected the PR's files against the default-branch alert list, which does not contain PR-ref alerts. Querying ?ref=refs/pull/3021/merge returns it straight away. * test(engine): probe ffmpeg and Chrome instead of assuming them Two failures on #3021's Test job, both about the environment rather than the code under test. **Bare `ffmpeg` is not on PATH in CI.** The 16-bit fixture shelled out to `execFileSync("ffmpeg", ...)` and died with ENOENT. The job does provide ffmpeg, through `prepare-ffmpeg-bin`, which is what `getFfmpegBinary()` resolves — every other ffmpeg-dependent suite in this package already goes through it. Now this one does too, and the case is `skipIf(!HAS_FFMPEG)` so a contributor without ffmpeg skips rather than fails. **The browser guard trusted the wrong thing.** It asked `resolveHeadlessShellPath()` and treated a returned path as "a browser is here". CI's cache holds a chrome-headless-shell that resolves and then fails to spawn — a partial download is indistinguishable from a working one by `existsSync`, which is all that resolver checks. So the three browser cases ran anyway and failed on the launch. It now runs `--version` and requires exit 0, which is the same probe the ffmpeg suites use: ask the binary, do not infer from the filesystem. Checked both directions rather than just the green one. With a working browser all 11 cases run and pass; with `HYPERFRAMES_BROWSER_PATH` pointed at a binary that exits non-zero — CI's exact situation — exactly 3 skip and the other 8 still run. A guard that quietly skipped everything would have looked identical on the CI summary. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Miguel Ángel
|
dc4383113c |
fix(producer): mix audio into a container that can record encoder delay (#3200)
* fix(producer): mix audio into a container that can record encoder delay Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz) after its authored `data-start`, against a frame-accurate video track. The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix was written to a raw ADTS `.aac` file, which has nowhere to record that delay, so it decoded as real leading silence and every stage downstream preserved it faithfully. Measuring each intermediate localises it precisely: the source WAV is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and mux stages inherit it unchanged. The filter graph itself is correct - run by hand to PCM it lands on the authored start. Switch the artifact to an MP4-family container, which stores the delay as an edit list that decoders strip. Same codec, same bitrate, so no size or quality change. The filename is a contract shared by three consumers - the mux input, the distributed plan artifact, and the PNG-sequence sidecar handed to users for NLE ingest - and its extension is what selects the muxer. Give it one owner in the engine rather than five literals, so those consumers cannot drift onto different containers. Note for reviewers: this renames the distributed plan's audio artifact, which is an on-disk contract between the plan writer and the assembler. Both move together here, but a plan written by an older build would not be found by a newer assembler. Flagging in case that mixed-version window matters for how these are deployed. * fix(cloud): read the plan audio artifact name from the producer contract The aws-lambda and gcp-cloud-run adapters each restated the plan's audio filename in five places, so renaming it in the producer left them looking for a file that is no longer written. CI caught it: the gcp dispatch test asserting a plan has no audio artifact started seeing one. Export the name from `@hyperframes/producer/distributed` and consume it in both adapters. This is the same failure the constant exists to prevent, one package boundary further out: a literal that drifts from the writer's is a silently missing audio track rather than a loud error, because both call sites only ever ask whether the file exists. * fix(cloud): accept a legacy plan's audio artifact name for one release Review raised a rolling-deploy window I had flagged but left undecided: `plan` and `assemble` are separate invocations bridged by object storage, so a pre-rollout planner can be paired with a post-rollout assembler. Both readers locate the artifact by existence alone, which makes that pairing a silently muted video rather than an error. That is reachable enough to be worth two lines, so reads now accept the old name while writes only ever emit the new one. Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`) rather than four call sites, marked for deletion one release out. Also fixes a hole in the first pass of this: the plan-v2 materializer matched either name but then joined the CURRENT one, so a legacy plan resolved to a path that was never written. It now joins the artifact's own name. Review nits in the same pass: correct the pad-branch docstring, which still described a concat-copy shape the pad branch stopped using when it moved to apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so it cannot model a shape that reintroduces the priming delay. * test(producer): rebake the missing-host-comp-id golden without the audio delay The pinned reference was rendered before this branch, so it carries the 1024 sample encoder-priming delay in its audio. With the delay gone the correct audio now sits ahead of the reference and the harness's envelope correlation drops below its floor. Cross-correlating the old and new references at native 48 kHz gives a lag of exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved by exactly the amount this branch removes. Regenerated inside the CI container (Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches the encoder CI will compare against - the container reproduced CI's failure to the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and passes at correlation 1.0 after it. Note for archaeology: the new reference is also 3 dB louder than the old one. That gap is not from this branch - `main` and this branch render the fixture at the same level - it is pre-existing drift the reference had accumulated, which a scale-invariant correlator could never see. The rebake absorbs it. Only output.mp4 is updated. `--update` also rewrites compiled.html, but that diff is embedded-font churn with no bearing on the comparison, which reports "Failed at compilation: 0" either way. * test(producer): rebake the variables-prod golden without the audio delay Same cause as the missing-host-comp-id rebake, caught by shard-8 once the earlier shard stopped failing and the rest of the matrix could run: this reference also carries the encoder-priming delay this branch removes. Reproduced in the CI container to the digit (correlation 0.42704173048439215, lagWindows -12), rebaked there, and it now passes at correlation 1.0. Worth recording: the shift here is 2048 samples (42.67 ms) at correlation 0.99983, exactly twice the 1024 of the other fixture. The delay compounds once per un-compensated AAC generation, and this fixture's audio needs its duration normalized, so it takes the pad/trim branch's re-encode and picks up a second frame of priming on top of the mixer's. So the pre-fix error was not a fixed 21 ms - it grew with the number of times the audio was re-encoded. All nine shards ran in that CI round with only this one failing, so the matrix has now covered every fixture against this change. |
||
|
Miguel Ángel
|
b57dc13cb6 |
fix(engine): stop SwiftShader ghosting in software screenshot captures (#3096)
Apply the --disable-gpu-compositing workaround to every software capture, not just BeginFrame ones. SwiftShader's compositor re-presents stale raster for a partially invalidated layer, so successive screenshot captures accumulate copies of earlier seeks; alpha renders are forced onto the screenshot path and were the only ones left unprotected. Refreshes the byte-strict png-sequence alpha baseline for the resulting antialiasing delta (content unchanged, min PSNR 41.3 dB). Fixes #3049. |
||
|
Vance Ingalls
|
c03cc2c52c |
Merge branch 'main' into via/studio-5433-html-sniff-defense
Both conflicts were import/export unions in the engine package, resolved by keeping both sides: - packages/engine/src/index.ts — main widened the urlDownloader re-export (fetchPublicHttpsText, safeDownloadUrlIdentity, writeUrlDownloadTelemetry and their types) while this branch added the notMediaPayload exports. - packages/engine/src/services/audioMixer.ts — main added UrlDownloadError and writeUrlDownloadTelemetry to the urlDownloader import; this branch added isNotMediaPayload. In audioMixer's prepare path both intents compose in order: main's download telemetry and typed download failure, then the STUDIO-5433 non-media sniff before the probe. |
||
|
Vance IngallsandClaude Opus 5
|
349c066a83 |
fix(producer): classify JSON error bodies as non-media sources too
A source that answers with a JSON error body still reached ffprobe and
produced `moov atom not found`. Replicate returns
`{"detail": "requested file not found"}` for a dead asset, and a gateway
in front of it can relay that body with a success status.
The sniff now treats `<`, `{`, or `[` as the opening byte of a text
document. No supported container starts with any of them, so this is the
same trade as before: three bytes instead of an allowlist that grows one
entry per payload shape observed in production.
Renamed accordingly, since the class now covers JSON as well as markup:
MARKUP_NOT_MEDIA -> NOT_MEDIA_PAYLOAD, MarkupNotMediaError ->
NotMediaPayloadError, markupPayload.ts -> notMediaPayload.ts. Registry
entries in the Lambda name map, the CDK and SAM plan lists, the Cloud Run
set, and SAFE_RENDER_ERROR_CODES move with it.
Also documents the reachability boundary on the error class: only a 2xx
response gets here. `downloadToTemp` rejects 404/410 as `http_not_found`
before writing a byte, and every ffprobe input is local because
videoFrameExtractor downloads http srcs first. So the shapes this
classifies are soft-404 and interstitial HTML, S3/CloudFront error
documents, and JSON API error bodies -- each served with a success
status. A genuine 404 surfaces as a download failure, not as this error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Varo
|
b390b71bde |
fix(engine): keep BeginFrame time monotonic across frame rates (#3011)
Fixes #3012 ## What Fixes `HeadlessExperimental.beginFrame` hanging at 60fps and other high frame rates. ## Why Warmup always advances Chrome in 33ms steps, but the first capture timestamp was calculated from the output fps. At 60fps, the last warmup tick is 1947ms, while the next commit tick used to jump back to about 1067ms. Chrome sees time moving backwards and `beginFrame` can stall. ## How Keep the old timestamps when they are already safe. If the commit tick would go backwards, move the capture baseline past warmup first. The init commit and both producer probes now use the same timestamp helpers, so they cannot calculate different values. ## Test plan - [x] Added tests for 24, 30, 31, 32, 33, 60, 120, 240, and 59.94fps - [x] Added coverage for the actual session baseline, commit parameters, and both producer probe paths - [x] Engine tests: 1383 passed, 3 skipped - [x] Producer unit tests passed - [x] Engine and producer typechecks and builds passed - [ ] Manual render test |
||
|
James Russo
|
bbdfee1166 | fix(engine): validate remote download integrity (#2938) | ||
|
Vance IngallsandClaude Opus 5
|
f3689c1481 |
fix(producer): scope and harden the markup-payload sniff
Review follow-up on the STUDIO-5433 defense.
Correctness
- The sniff ran above the documented video/audio failure split, so an
<audio> src that resolved to an HTML payload aborted the whole render
instead of degrading to duration 0. It now runs inside the same try, so
video surfaces the typed error while audio still drops out, with a
warning naming the element.
- Raw fs errors (EISDIR on a directory src, EACCES, the existsSync->open
ENOENT race, EMFILE) escaped and failed the compile with an unclassified
error carrying an unredacted temp path. The sniff is now a classifier that
never throws: an unreadable file reports "not markup" and the real probe
produces the real error.
- Elements whose duration the compiler never resolves (a data-end video, a
looping audio) skipped the sniff entirely, so the original ffprobe error
still escaped, and looping audio was reported as owner "system" after
every frame had been captured. Video is now caught in the asset preflight,
which sees every local src regardless of authored timing; audio is
classified per-element in audioMixer as source/invalid_media/owner "user",
keeping audio failures non-fatal as they already were.
- Detection is a byte-level check for a leading "<" (BOM-, whitespace- and
NUL-tolerant, looped read) instead of a <!doctype|<html|<?xml string
prefix, which missed a NUL-prefixed payload, >256B of leading whitespace,
UTF-16-encoded HTML, and a prolog-less <svg. No supported container starts
with "<", so the allowlist no longer grows per payload shape.
- finally { await fh.close() } could replace the in-flight typed error with
the close error.
Routing and privacy
- MARKUP_NOT_MEDIA is now in SAFE_RENDER_ERROR_CODES, the Lambda terminal
name map, the CDK and SAM non-retryable plan lists, and the Cloud Run
non-retryable set, and the class carries owner/retryable. Previously the
API emitted errorCode: undefined and a deterministic authoring bug burned
the full distributed retry budget.
- The message no longer carries 32 raw payload bytes or the src.
redactTelemetryString preserves host and path for HTTP srcs, so
per-tenant CDN paths reached a message the server forwards to clients.
Correlation is a sha256 element fingerprint, matching
AssetMediaTypeMismatchError.
- The message names both causes (unresolved nested-composition URL, or an
HTML/XML error page served as 200) rather than misdiagnosing an S3 403
body as an authoring bug.
Tests
- Byte-level detection is unit-tested in engine: markup shapes, BOMs,
UTF-16, nine container signatures, unreadable inputs.
- Replaced the tautological assertions. The old checks for "html" in and
"moov" absent from a fixed message template could not fail for any input.
- New coverage for audio degradation, the audioMixer classification, the
preflight video/image/audio split, and the API error metadata.
- The sibling htmlCompiler.mediaType failure was a vitest-under-bun runner
mismatch, not a missing ffmpeg binary. It passes, including the 4-wide
probe-semaphore invariant the sniff now runs inside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Miguel Angel Simon Sierra
|
f69c4a0e3a |
fix(engine): distinguish probe failure from a genuinely absent GPU
A probe that could not run is no evidence about the GPU, so pointing the operator at GPU passthrough hid broken Chrome installs behind a phantom problem. Carry a cause off the probe and emit the matching remediation. Also un-exports buildUnverifiedHardwareGpuWarning (Fallow: engine test files are not audit entry points, so a test-only import would not have counted as a consumer) and covers the non-linux branch via the spy. |
||
|
Miguel Angel Simon Sierra
|
6703ea7e04 | fix(engine): warn once per process about unverified hardware GPU | ||
|
Miguel Angel Simon Sierra
|
131780fe96 |
fix(engine): verify explicit browserGpuMode=hardware instead of trusting it
Chrome's hardware GL args are advisory: with no usable GPU it silently falls back to software WebGL and the capture runs at CPU speed. Run the existing WebGL probe for explicit hardware mode too and warn loudly with the platform's remediation. The requested mode is still honoured. Fixes #2967 |
||
|
James Russo
|
127eb19371 |
fix(producer): correct short VFR frame coverage (#2936)
* fix(producer): correct short VFR frame coverage * fix(engine): keep VFR extraction seek-local * fix(engine): match ffmpeg decimal frame boundaries * fix(engine): preserve exact extraction frame rates * fix(engine): key frame cache by exact rate |
||
|
James Russo
|
2339757377 |
fix: bound HDR and video extraction resources (#2955)
* fix: bound HDR and video extraction resources * fix: trim negative video extraction preroll * fix: skip invisible video extraction windows * fix: preserve negative-start loop and held tails * fix: cap finite video slots to source duration * fix: bound held-tail frame extraction * fix: plan from playable video duration * fix: preserve open-ended held video tails * fix: resolve held tails from decoded frames * fix: normalize final-frame probe timestamps * fix: handle unseekable final-frame sources * fix: dedupe final-frame probes per render * refactor: clarify output dynamic range contract |
||
|
James Russo
|
3a6b7f0612 |
fix: align local WebGPU capture behavior (#2907)
* fix: align local WebGPU capture behavior * fix: address WebGPU capture review feedback * fix: retain overlapping GPU seek work * fix: satisfy runtime seek completion types * fix: drain concurrent GPU seek work * fix: prevent WebGPU capture barrier starvation * fix: keep WebGPU presentation active during render seeks |
||
|
Vance IngallsandClaude Opus 5
|
c61a24b510 |
fix(producer): unbias the static element count and stop zeroing failures
Review findings on #2891. Two of them bite directly on this PR's own purpose — making the fleet element-count distribution readable — so they are fixed rather than noted. countElementTags counted `</` + letter anywhere, including inside inline JS. A compiled comp containing `const h = "</div>"` or a template literal building `</span>` inflated the count once per occurrence. Compiled comps embed large inline scripts, so the bias is systematic, not noise, and it lands entirely on the ~83% of renders with no probe session — precisely the cohort this PR exists to characterize. Script and style bodies are now stripped before matching; losing their own closing tags costs 1-2 counts against a threshold in the thousands. The new elementCount fell back to 0 when its page.evaluate threw, following the tweenCount pattern beside it. For this field that pattern is wrong: evaluate failures concentrate on the huge-DOM compositions the field is meant to observe, and a 0 there is indistinguishable from a legitimately empty comp, so the fleet p50/p99 would absorb both silently. It is now undefined on failure, the INIT console line omits the token entirely rather than emitting a zero, and the parser reports absent — mirroring the live/static provenance split the routing resolver already uses. Also documented: the "every render reaches this path" claim holds only for renders that survive to end of init, so the tail is survivor-biased and should be read as a lower bound; and the two element-count fields now say plainly which is which — composition_element_count gates routing, observability_init_element_count is the observational counterpart — so the follow-up analysis can't query the wrong one. Nits: envInt is integer-only per its name, both live-DOM reads use getElementsByTagName (live collection length, no NodeList materialized on the 40k-node tail), and the attribution block notes that it runs with routing off by design. Fault injection confirms the new tests bite: disabling script stripping fails 4, and the zero-vs-undefined case is pinned separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Vance IngallsandClaude Opus 5
|
d74afc7b7d |
feat(engine): measure live DOM size on every render, not just probed ones
The short-comp routing gate can only read a live element count when a
probe session exists, and the first v0.7.83 data shows that is far rarer
than estimated: 17% of renders (86/503), not the ">=28%" the video-presence
proxy suggested. The other 83% fall back to a static source scan, which
is exactly blind to the shape that motivated the live count — small
markup, thousands of script-created nodes.
That leaves the fleet element-count distribution unknowable for most
renders, and the observed distribution is already surprising: p99 ~900,
max 1,420 against a 2,500 ceiling calibrated on 7k/20k/40k synthetic
nodes. Either the ceiling is close to irrelevant, or the large-DOM tail
is hiding in the 83% we cannot see. Both readings change what PR B
should do, and neither is decidable from probed renders alone (they are
a biased sample — they got a probe *because* they carry media or
unresolved compositions).
So measure it where every render already goes: capture-session init.
`collectSessionInitTelemetry` gains a querySelectorAll("*") count beside
the tween count it already collects, riding the same channel to
`observability_init_element_count`. This is observational only — capture
has begun, far too late to route on — and it deliberately does not feed
the gate. It answers the distribution question the gate cannot.
Coverage for this channel is proven rather than assumed: the tween-count
fix that shipped in v0.7.83 took the clamped-parallel bucket from 0/272
renders to 217/217, and 23.1% -> 100% overall.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Vance Ingalls
|
b2e7d76d67 |
Merge pull request #2875 from heygen-com/feat/de-short-inversion
feat(producer): short-comp DE inversion band — baseline release (telemetry only, routing off) |
||
|
Miguel Ángel
|
b7160f69bb |
fix: exclude hidden audio from render mix (#2870)
* fix: exclude hidden audio from render mix * fix(engine): honor hidden media ancestry in audio mix |
||
|
Miguel Ángel
|
85f0c9d354 |
fix(cli): select host-compatible cached browser (#2861)
* fix(cli): select host-compatible cached browser * test(engine): make browser cache fixture portable * fix(browser): reject foreign ARM cache binaries |
||
|
Vance IngallsandClaude Opus 5
|
23854f7c6a |
feat(producer): surface init telemetry from parallel workers — the band's missing motion axis
The routing surface the short-comp benchmarks validated is (motion x DOM size x frames). After the baseline release, fleet telemetry carries DOM size (composition_element_count) and frames on every render — but the motion proxy, observability_init_tween_count, has 0% coverage on the exact renders the band routes: parallel workers' console buffers (and so the [FrameCapture:INIT] line the summary parses) only propagate to the orchestrator on FAILURE. Single-worker screenshot renders report it; the multi-worker clamp bucket never does. Verified against 7d of fleet data: 35k screenshot renders carry tween counts, 0 of 9,600 band renders. Fix rides the one channel parallel workers already return on success — the per-worker CapturePerfSummary. Sessions record initTelemetry on every init path; the perf summary now carries it; the orchestrator max-merges across workers (same multi-session semantics the console parser uses) and feeds it to the observability summary as a structured fallback, console lines still refining when present. With this, every band render carries full coordinates — (elements, tweens, frames, path, speed) — which buys two reads: regressing wild DE speed against element count on the existing 900+ inversions validates the bench's 0.50ms/element slope BEFORE the routing flip, and any post-flip misroute can be reproduced locally by feeding its telemetry row straight into gen-crossover-comp's knobs (--movers ~ tween count, --static ~ element count) and re-benching. (Also drops a now-stale fallow suppression in render.ts — the test-only reset export it guarded gained real test importers, so the issue it suppressed no longer exists and the gate flags the leftover.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Vance IngallsandClaude Opus 5
|
4520cd240b |
fix(engine): bucket gpu_renderer + cover the failure cohort (review)
Three review findings on the win32 drawElement PR: 1. gpu_renderer shipped the raw UNMASKED_RENDERER_WEBGL string — unbounded, driver-authored, GPU-model-specific, and |-joined across parallel sessions, i.e. high cardinality by construction, against this file's own convention of sanitizing engine-sourced strings (deGateReason is a bucket; error messages go through redactTelemetryString). Now bucketed at the source by classifyGpuRenderer to <backend>/<vendor> (metal/apple, d3d11/nvidia, swiftshader/other, ...), which is the whole analytic signal the win32 rollout needs and nothing else. The raw string never leaves the engine. 2. gpu_renderer reached render_complete only, so a crashed render — the cohort the field exists to attribute — carried no backend. It now rides RenderCaptureObservability (deGpuRenderer, sourced from the live probe session like the de_* counters), so both render_complete and render_error carry it and a hard failure still reports its GPU backend. On render_complete the perfSummary value still wins by spread order. 3. Restore the fallow-ignore-next-line suppression above __resetDeParallelRouterTrialStateForTests: CLI test files are not fallow entry points, so removing it fails the CI dead-code audit (local pre-commit passed only because of its changed-file scope). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Vance IngallsandClaude Opus 5
|
cb30157ebb |
feat(engine): open drawElement fast capture to Windows hardware GPU
Widen the default-on drawElement clamp from darwin-only to darwin|win32 (still requiring a non-software-GPU browser). The darwin restriction was a validation envelope, not an architectural limit — the CanvasDrawElement Chrome flag ships on every platform, and every safety layer that made the macOS default-on release (v0.7.38) survivable is platform-neutral: compile-time gates, the SwiftShader init gate, per-render worker-encode self-verification with screenshot fallback, and the blank guard. Worst case on an unvalidated D3D11 backend is the same as on Metal: verify catches a bad frame and the render re-runs on the screenshot baseline. Why now: 30-day telemetry shows ~206k non-CI hardware-GPU Windows renders (~78% of the win32 fleet, 18k installs) held on the slow screenshot path by the clamp — the second-largest perf population after macOS, carrying ~1,550 capture-hours/month in the DE-eligible >=700-frame band alone at a measured ~2x speedup opportunity. Instrumentation for the new cohort: drawElement session init now records the raw WebGL UNMASKED_RENDERER_WEBGL string (detectSwiftShader generalized to detectGpuBackend — same single evaluate, the string was previously read and discarded) and threads it session -> CapturePerfSummary -> RenderPerfSummary -> render_complete as `gpu_renderer`. drawElement damage proved compositor-backend-specific throughout the macOS rollout, so D3D11-cohort failures must cluster by ANGLE backend + GPU vendor (NVIDIA/AMD/Intel), not just `os`. The two DE clamp branches are extracted into a pure, unit-tested `resolveDefaultDrawElement` (platform + GPU mode + worker-encode + explicit opt-in), which also drops resolveConfig's cyclomatic complexity. The win32 streaming-encode compound tests collapse onto one shared helper. Linux stays excluded: that fleet is headless/Docker SwiftShader, where DE has no speedup and known rendering defects. Kill switches unchanged: PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false, --experimental-fast-capture=false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
James
|
96cafb47c6 | fix(producer): fallback distributed capture safely | ||
|
James
|
2a284a8e3a | fix(gcp): enforce effective BeginFrame capture | ||
|
James
|
33ca1de063 | fix(render): aggregate extraction launch failures | ||
|
James
|
c01e1a5f96 | fix(producer): narrow extraction error shapes honestly | ||
|
James
|
9b63646c8a | fix(producer): type video extraction failures | ||
|
James
|
c01f6b4468 | fix(engine): make video downloads atomic and retry transient failures | ||
|
Miguel Ángel
|
4b116b9880 | fix(producer): normalize padded audio on sample timeline | ||
|
Miguel Ángel
|
63bc525ca9 | fix(engine): stop mux at shortest normalized stream | ||
|
Miguel Ángel
|
20188d637b | test(render): cover duration-capped audio mux | ||
|
Miguel Ángel
|
19258ea5ba | fix(render): cap final mux to video duration | ||
|
Miguel Ángel
|
9289551e98 | fix(render): scope M4A priming preservation to trims | ||
|
Miguel Ángel
|
59c56d3257 | fix(render): preserve normalized M4A edit timing | ||
|
James Russo
|
37b88688e7 |
fix(audio): preserve causes and use portable padding (#2769)
* fix(audio): preserve causes and use portable padding * fix(audio): address failure taxonomy review |
||
|
ukimsanov
|
944640c328 | feat(runtime): render media treatments deterministically | ||
|
Vance Ingalls
|
9fc1c2f159 | fix(producer): verify sequential disk drawElement samples too | ||
|
Vance Ingalls
|
060b6f8ae5 | fix(engine): self-verify parallel disk drawElement samples (PRINFRA-352) | ||
|
Vance Ingalls
|
84e4eafacd |
Merge pull request #2723 from heygen-com/07-21-fix_engine_worker_autoscaler_memory_budget
fix(engine): realistic worker memory budget + sizing/feedback telemetry |
||
|
Vance Ingalls
|
c6462a0a22 | fix(producer): emit heap advisory at orchestrator, lock message + telemetry props with tests | ||
|
Xuanru Li
|
69446e7726 |
Revert "feat(producer): renderStretch to re-time short compositions across longer scenes (#2676)" (#2730)
This reverts commit
|
||
|
Vance Ingalls
|
12e599a6ba | fix(engine): realistic worker memory budget + sizing/feedback telemetry | ||
|
Vance Ingalls
|
ed32898439 |
Merge pull request #2681 from heygen-com/07-21-feat_engine_warn_on_live_map_viewports_at_capture_init
feat(engine): warn when a live map viewport is detected at capture init |
||
|
Xuanru Li
|
e786b78b33 |
feat(producer): renderStretch to re-time short compositions across longer scenes (#2676)
Linear: VA-1859 ## Problem For a `fit_to_scene` B-roll where the composition's intrinsic timeline (e.g. `data-duration=1.0s` → 30 frames) is shorter than the scene it fills (e.g. 4.8s narration), the producer renders only the intrinsic 30 frames and the downstream compositor frame-holds/PTS-stretches that fixed clip to the scene length. Spreading 30 unique frames over 4.8s starves motion to ~6 effective fps → a visibly choppy result. Root cause: the producer welds one `composition.duration` to both the frame count and the 1:1 seek mapping, with no notion of a target output length. ## Fix Add optional `renderStretch: number` (default `1.0` = no-op), `renderStretch = intrinsic / target`: - **Frame count** comes from the target: `outputDuration = intrinsic / renderStretch`, `totalFrames = outputDuration × fps` (`probeStage.ts`). `composition.duration` stays intrinsic (drives video/audio windows). - **Per-frame seek** is scaled: `time = (frameIndex / fps) × renderStretch`, so the N output frames map across `[0, intrinsic]` — a fresh frame per output frame. All seek sites go through a single shared `outputFrameToTimelineSeconds(frameIndex, fps, renderStretch)` helper (`core.types.ts`), consumed by every capture path so none can silently diverge: - parallel (`parallelCoordinator.ts`), `sdr_streaming` (`captureStreamingStage.ts` ×3), `sdr_disk` (`captureStage.ts`), HDR loops. - DrawElement + static self-verify (`frameCapture.ts`) — ground-truth seek uses the same mapping, so PSNR compares like-for-like (no spurious verification failure on stretched comps). - Distributed path: `renderStretch` threaded through `DistributedRenderConfig` → chunk workers, and **folded into the plan hash only when `!= 1`** so a pre-stretch cached plan is never reused. With `renderStretch = 1` (or omitted → `?? 1`): every seek is `×1.0` (IEEE-754 identity), frame counts unchanged, and the plan hash is byte-identical — a provable no-op. `player.ts` absolute-seek is untouched. ## Verify - typecheck (core + engine + producer): pass. lint/format/fallow/commitlint: pass. `planHash` + `renderRequest` unit suites: pass. - Adversarial self-review found + fixed three capture-path gaps (streaming, self-verify, distributed) before this revision. - **Not yet runtime-verified** on a real render — needs a fit_to_scene render at `renderStretch < 1` confirming N distinct frames over the target length (draft until then). Paired with experiment-framework#42766, which computes and forwards `renderStretch = hf intrinsic / scene duration`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Vance Ingalls
|
30ca51c615 | feat(engine): warn when a live map viewport is detected at capture init | ||
|
Vance Ingalls
|
4f53dd4f2c | fix(engine): carry frameStride onto WorkerResult (fixes interleaved worker false-positive) | ||
|
James
|
344d9c0a87 | fix: bound invalid render durations | ||
|
James
|
a03d40ef4c | test(engine): isolate peer abort regression | ||
|
Miguel Ángel
|
2be8a62c00 |
fix(engine): stop compositing phantom duplicates on captureBeyondViewport (#2607)
* fix(core): stop the async media-metadata rebind once render capture starts seeking scheduleMetadataDurationHydration re-resolves and can swap the captured GSAP timeline off a debounced loadedmetadata/durationchange event, fully uncoordinated with the producer's own per-frame renderSeek calls. When a full-length <video>'s metadata resolves after capture has already begun (slow I/O, Docker), this races the deterministic BeginFrame capture loop and can reflow sub-composition state mid-render, producing phase-offset duplicate content in captured frames (#2550). Render-mode duration correction already happens deterministically during the probe stage before capture starts, so once renderSeek has been called once there is nothing left for this self-correction to do — gate it off for the rest of the session. * fix(core): scope the metadata-rebind guard to actual render/export pages renderSeek isn't capture-exclusive — Studio's own preview iframe falls back to it for compositions whose timeline overhangs every native adapter's duration. Gating the HF#2550 fix on renderCaptureSeekStarted alone silently disabled the metadata-driven duration self-correction for that live-scrub case too, where it's still needed. Require the render/ export page signal (window.__HF_EXPORT_RENDER_SEEK_CONFIG, set only by the producer's fileServer.ts) alongside it, and add a regression test covering the Studio-preview case. * fix(engine): stop requesting beyond-viewport capture for video comps that don't need it Root-caused HF#2550 by reproducing the reporter's public repro end-to-end (not just the timeline-rebind mechanism from the earlier commits in this branch) on native Linux: instrumented the actual DOM state during a real capture session and confirmed the sub-composition never double-mounts — getBoundingClientRect and the timeline's own local time both match the single, correct DOM tree throughout. The phantom second copy only exists in the captured screenshot pixels. Bisected it to captureBeyondViewport: resolveVideoCaptureBeyondViewport (#1094's tall-portrait fix) forces `Page.captureScreenshot`'s beyond-viewport path on for any render with a native <video>, regardless of whether the page's content actually overflows the declared capture height. On SwiftShader that beyond-viewport path can composite a stale, vertically offset paint of the page alongside the fresh one for content that fits entirely within the viewport — producing exactly the reported phase-offset duplicate. Disabling captureBeyondViewport (repro's video still present) eliminates the duplicate outright; re-enabling it reproduces the duplicate byte-for-byte, isolating it as the actual cause. Adds pageContentExceedsCaptureHeight, a ground-truth measurement of the page's actual scrollHeight against the requested capture height, and wires it into initializeSession to downgrade captureBeyondViewport back to false once the page is settled and it's confirmed unnecessary — the "reliable clip predictor" the original #1094 fix's ponytail comment flagged as missing. This keeps #1094's fix intact for content that genuinely overflows while closing the SwiftShader ghosting hazard for the (common) case of video that fits inside its own viewport. * test(producer): add HF#2550 video+sub-composition regression fixture Checks in the reporter's confirmed real-world reproduction (media regenerated via ffmpeg testsrc2, matching their public repro repo) as a regression fixture, with a golden baseline rendered against the fix. Verified end-to-end via the project's own Docker regression harness: - Rendering this fixture with the fix produces the golden baseline (clean, single flowchart instance, captureBeyondViewport correctly downgraded). - Direct CLI renders (not through this harness) against unpatched code reproduce the reported phantom-duplicate artifact reliably (10/10). Caveat documented in meta.json: the underlying bug is timing-dependent. Two harness runs against unpatched code, using this same fixture, did not reproduce the artifact (0/2) — the harness's in-process render path apparently doesn't hit the same race window a direct CLI process does on this host. This fixture is a best-effort regression guard and a preserved real-world repro, not the sole protection — the deterministic guard is packages/engine/src/services/screenshotService.test.ts's pageContentExceedsCaptureHeight unit tests, which exercise the actual fix logic directly. Also adds an .gitattributes LFS rule for this fixture's source index.html (744 KB — carries the real project's embedded base64 assets, over the largefiles hook's 500 KB non-LFS limit). * fix: route HF#2550 fixture binaries through LFS (were committed raw) filter.lfs.clean/smudge were locally configured as a no-op "cat" in this repo's shared .git/config, silently disabling LFS filtering for every worktree. The previous commit's large binaries (output.mp4, compiled.html, source index.html, source video) landed as raw blobs instead of LFS pointers as a result. Ran `git lfs install --local --force` to restore the correct filter commands, then re-staged the affected files so they commit as proper LFS pointers. * fix(engine): address capture viewport review feedback |