Commit Graph
382 Commits
Author SHA1 Message Date
Vance IngallsandClaude Opus 5 cc40e35aa0 feat(engine): render the FX chain offline, and the carve analysis behind it (#3021)
* feat(engine): render audio FX in an OfflineAudioContext

Reads `data-fx-chain` off an audio element and runs the chain over the trimmed
WAV before volume automation is baked in — effects should see the raw signal,
and the envelope belongs on their output.

The processing happens in an OfflineAudioContext inside the headless browser
the engine already drives, running the same graph builders the studio previews
with. That is the point of the approach: one implementation per effect, so the
render agreeing with the preview is a property of the architecture rather than
a tolerance to police. Reimplementing each effect as an FFmpeg filter would
mean two implementations to keep in step, and for the dynamics processors and
modulated delays there is no filter that behaves the same way.

`build:audio-fx-runtime` bundles the graph builders into an injectable IIFE,
following the same pattern as the existing runtime artifacts, so the browser
runs exactly the code the studio does.

The page loads from a file:// URL rather than about:blank because AudioWorklet
is only exposed in a secure context — the compressor, limiter, gate and
bitcrush processors would otherwise fail to register with an opaque error.
file:// qualifies and needs no listening socket.

The chain is serialised into the attribute the way colour grading carries its
config, so there is no side-car file to resolve or lose.

An FX failure is fatal for the whole mix rather than a per-track soft failure.
Every other audio failure mode degrades gracefully — the track drops, siblings
continue — but substituting the dry signal for a processed one ships a render
that sounds plausible and is not what the author set up. Since the per-element
work races under Promise.all, an internal AbortController chained off the
caller's signal aborts in-flight siblings before workDir is removed.

* feat(core): voiceover carve analysis

Finds the bands a voice occupies so a music bed can be dipped there, letting
the voice sit in front without ducking the whole track.

Carve is a relationship between two tracks rather than an effect on one, so it
stays out of the FX chain. What it emits is an ordinary chain of peaking
filters, so a carve composes with whatever else is on the track and needs no
separate rendering path.

Selection is weighted toward intelligibility rather than raw voice energy.
Ranking purely by power lands on the fundamental almost every time, because
that is where a voice is loudest — but the masking that actually hurts a
voiceover happens higher up, and dipping 160 Hz mostly just thins the bed. The
bias is a control, not a constant: at 0 it follows raw energy, at 1 it weights
toward 1-3 kHz.

Ranking happens in dB, which matters more than it looks. Speech spreads 20-30 dB
across these bands — it falls off roughly 6 dB per octave above the fundamental
— so a weighting has to be on that scale to move anything at all. A
multiplicative weight of `1 - bias + bias * shaped` is bounded below by
`1 - bias`, capping its influence at 10*log10(1/(1 - bias)): 5.2 dB at the 0.7
default, 3 dB at 0.5. That is no influence against a real voice — every bias
short of ~0.95 would rank exactly like bias 0 and carve the fundamental, the
outcome the bias exists to prevent, while looking decisive against a fixture
whose bands sit 2 dB apart. So the bias is a dB penalty, zero at 2 kHz and worth
up to 30 dB at full strength, and relative cut depths come from a dB difference
rather than a ratio of weighted linear powers.

The bias reweights ranking without overriding the spectrum — a band the voice
has no energy in is not worth carving, and scores -Infinity rather than
competing — so a strongly low-pitched voice can still select low at full bias.
What the tests hold is that biasing never selects lower than the unbiased
ranking, that the DEFAULT bias reaches the presence region on a voice with a
realistic tilt, and that bias 0 still follows raw power exactly.

Includes a radix-2 FFT rather than a dependency; one Welch-style averaged
spectrum over third-octave bands does not justify pulling in a DSP library.

* fix(engine): keep the FX render 16-bit, stereo, and correctly sized

Three defects in the offline FX path, none of which any test could see.

**Float output silently disabled sample-accurate volume automation.** The writer
emitted 32-bit IEEE float; the very next mixer step bakes the volume envelope
into the samples and accepts only 16-bit PCM, returning null otherwise. So
enabling any effect downgraded that track to the ffmpeg expression path — capped
at 32 straight segments, quantising a curved envelope, and on a dense one falling
back to base volume. It now writes 16-bit PCM, clamped rather than wrapped so a
limiter at 0 dB or a resonant filter cannot turn overshoot into a click. A test
asserts the baker accepts the writer's own output and actually fades it.

**Everything was folded to mono.** `prepareAudioTrack` goes out of its way to
emit stereo — its pan filter exists to dodge ffmpeg's 3 dB mono-to-stereo
rematrix — and this folded it, then wrote one channel. So adding a single peaking
EQ collapsed a bed's width and cost ~3 dB in the render, while preview stayed
stereo. Channels now travel as one plane each, through an OfflineAudioContext of
the same width, and come back interleaved.

**Small results decoded the wrong length.** `new Float32Array(buf.buffer)`
discards byteOffset and byteLength, and Node pools small allocations: a 400-byte
payload sits at offset 8 inside an 8 KiB pool, so a clip under ~1024 samples
decoded as 2048 samples of unrelated memory — and the empty-result guard could
not see it. The reader has the mirror-image fix: a float data chunk on an odd
boundary (ffmpeg's pcm_f32le writes fmt(18) + fact, landing `data` at 58) now
copies instead of throwing RangeError on an unaligned view.

The tail limitation is now stated rather than mis-stated: the context is exactly
as long as the input, so a reverb or delay still ringing is cut there. The old
comment claimed the opposite. How far a tail may run past a clip's end changes
the clip's length in the mix, so it is a product decision, not one to make here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(producer): report an FX render failure as an audio error

`processCompositionAudio` reports per-track failures in its result, but an FX
failure it cannot degrade past — a browser that will not launch, a chain that
will not build — rejects instead. `runAudioStage` had no try, so that rejection
escaped to the orchestrator as an unclassified pipeline exception, losing the
stage/owner/retryable classification this stage exists to attach, and skipping
its abort check on the way out.

It now lands in `audioError` alongside every other cause, while an abort still
keeps its own shape rather than being reported as an audio problem.

Not done here: committing the generated `audio-fx-runtime-inline.ts` so a fresh
clone typechecks packages/engine without building first. The bundle is built from
the stub, and the stub changes three times across this stack — so the artifact
differs per branch and would conflict on every restack. Its model,
position-edits-render-inline.ts, is committed only because it is stable. Building
before testing is this monorepo's existing contract (studio's tests need core's
dist too), so the gap is not specific to audio FX and is better closed by a build
ordering gate than by committing a per-branch artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(engine): skip the browser FX render cases when there is no browser

CI's `Test` job was red on this PR with four failures, all the same cause:

  Failed to launch the browser process: spawn
  /home/runner/.cache/hyperframes/chrome/chrome-headless-shell

The job installs ffmpeg and no browser, deliberately — every other suite
that needs an external binary already guards on it
(`describe.skipIf(!HAS_FFMPEG)`). These cases were the only ones assuming
a Chrome, so they failed on an absent dependency rather than on anything
about the code.

Guards on `resolveHeadlessShellPath()` — the same resolver
`acquireBrowser` launches through, so the check cannot drift from the
thing it guards the way a hard-coded cache path would. A configured path
that does not exist throws; that is caught and read as "cannot run here".

Checked both directions rather than just the green one: with a browser all
11 cases run and pass, and with `HYPERFRAMES_BROWSER_PATH` pointed at a
missing binary exactly 3 skip and the other 8 still run. A guard that
silently skipped everything would have looked identical in CI.

They keep their value where it exists — every developer machine, and any
job that has run `hyperframes browser ensure`.

Not touched: the CodeQL failure on this PR is a run from 2026-08-07, five
days and several force-pushes stale. None of the 17 open repo alerts are
in files this PR changes; it re-runs on this push.

* chore(engine): suppress the temp-file alert with the reason it is safe

CodeQL flags `writeWav`'s `writeFileSync` as js/insecure-temporary-file
(high) — the one new alert on #3021, and the reason its CodeQL check is
red.

It is a false positive, and the comment says why rather than just silencing
it: `path` is always inside a directory made by `mkdtempSync`, never a
name assembled directly under `tmpdir()`. Both callers are covered — the
browser host page writes into `mkdtempSync(join(tmpdir(), "hf-fx-host-"))`,
and the render output goes to the producer work dir, itself
`mkdtempSync(join(tempRoot, "producer-project-"))`. mkdtemp picks the
random suffix and creates the directory 0700 in one syscall, so the
predictable filename inside it cannot be pre-created or symlinked by
another user, which is the attack the rule is about. The analyzer sees the
dataflow reach `tmpdir()` and not the mkdtemp in between.

Suppressed inline rather than dismissed in the UI, so the justification
lives next to the code and the rule stays live for anything added later in
this file. Matches the repo's existing convention — `planV2.ts:222`
carries an `lgtm[js/insecure-temporary-file]` for a different reason on
the same rule.

Correcting myself: I first reported this alert as not real, having
intersected the PR's files against the default-branch alert list, which
does not contain PR-ref alerts. Querying ?ref=refs/pull/3021/merge returns
it straight away.

* test(engine): probe ffmpeg and Chrome instead of assuming them

Two failures on #3021's Test job, both about the environment rather than
the code under test.

**Bare `ffmpeg` is not on PATH in CI.** The 16-bit fixture shelled out to
`execFileSync("ffmpeg", ...)` and died with ENOENT. The job does provide
ffmpeg, through `prepare-ffmpeg-bin`, which is what `getFfmpegBinary()`
resolves — every other ffmpeg-dependent suite in this package already goes
through it. Now this one does too, and the case is `skipIf(!HAS_FFMPEG)`
so a contributor without ffmpeg skips rather than fails.

**The browser guard trusted the wrong thing.** It asked
`resolveHeadlessShellPath()` and treated a returned path as "a browser is
here". CI's cache holds a chrome-headless-shell that resolves and then
fails to spawn — a partial download is indistinguishable from a working
one by `existsSync`, which is all that resolver checks. So the three
browser cases ran anyway and failed on the launch.

It now runs `--version` and requires exit 0, which is the same probe the
ffmpeg suites use: ask the binary, do not infer from the filesystem.

Checked both directions rather than just the green one. With a working
browser all 11 cases run and pass; with `HYPERFRAMES_BROWSER_PATH` pointed
at a binary that exits non-zero — CI's exact situation — exactly 3 skip
and the other 8 still run. A guard that quietly skipped everything would
have looked identical on the CI summary.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 09:13:12 -07:00
Vance IngallsandClaude Opus 5 5752d22492 feat(core): the audio FX registry (#3019)
* feat(core): audio FX registry

One declarative description of every effect that can be applied to an audio
track: fourteen across filters, dynamics, non-linear and time, each exposing
its full parameter surface rather than a curated subset.

Parameters carry the range, step, unit and scale a control needs, so a panel
can generate its UI from this rather than hard-coding a form per effect, and a
value that survives `normalizeAudioFxParams` is always safe to realise.
Everything is declared in the units a person thinks in — dB, ms, Hz.

Parsing rejects an unknown effect id rather than skipping the node. A chain
that quietly loses an effect renders something other than what was authored,
which is worse than refusing to load it.

Data only: no audio is produced here. The graph that realises each effect is
referenced by the `web` id and lands in the next change, which keeps this
module free of browser globals so the engine and the linter can import it.

* fix(core): stop declaring knobs that move nothing

Three parameters were declared with ranges, defaults and hints, and read by no
builder — dials an author could turn with no audible result.

- `chorus.decay` and `bitcrush.aa`: removed. FFmpeg's chorus feeds a decay back
  into its delay line and a bitcrusher's anti-alias needs a real filter; adding
  either is new DSP, not a fix, so the honest move is to stop advertising them.
- `lowshelf.q` / `highshelf.q`: removed. The Web Audio spec leaves Q unused for
  shelving filters, so the control moved nothing — and because the shared Q
  helper marks it automatable, an author could draw an envelope on it and hear
  nothing at all.

`phaser.decay` and `gate.knee` stay: the first drives the sweep depth, and the
second is now read by the gate's processor.

A test asserts each of these directly, since the existing exposure invariant only
checks that a flagged parameter reaches an AudioParam — a parameter the node then
ignores passes it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 00:09:59 -07:00
Miguel Ángel e0ba41c024 chore: release v0.7.107 (#3228) 2026-08-11 16:19:20 -04:00
Miguel Ángel 636dc042a7 feat(core): sanitize rich text on the way into a composition (#3141)
* feat(core): sanitize rich text on the way into a composition

Studio's patch vocabulary was inline-style, attribute, html-attribute and
text-content. text-content assigns textContent, and the text-field model
escapes markup on the way out and refuses a change in child structure, so a
styled span had no route into a composition file.

Adds a rich-text operation with one, guarded by a single sanitizer called on
both ends of the trip: in the browser so the preview shows what will be
saved, and on the server because that is where the file is written. Tags and
style properties are a small allowlist, and an unexpected tag loses its
formatting rather than its words. Spans an edit adds get their ids in the
same write, so a follow-up write cannot race it.

No UI yet — this is the persistence contract the editor is built on.

* fix(core): document and test the sanitizer boundary

* fix(core): harden rich text sanitizer traversal
2026-08-11 02:03:47 -04:00
Miguel Ángel c9dd8413c3 chore: release v0.7.106 (#3197) 2026-08-10 22:53:34 -04:00
Vance Ingalls bd1c1af291 chore: release v0.7.105 (#3152) 2026-08-10 08:08:16 -07:00
Miguel Ángel c96b30c717 chore: release v0.7.104 (#3147) 2026-08-09 17:19:31 -07:00
Vance Ingalls adb13ce125 chore: release v0.7.103 (#3127) 2026-08-09 03:14:27 -07:00
Miguel Ángel b1f7d8881b chore: release v0.7.102 (#3119) 2026-08-08 13:13:50 -07:00
Vance Ingalls eba96feda7 chore: release v0.7.101 2026-08-07 19:50:48 -07:00
Vance IngallsandMiguel Ángel d9b00e57eb chore: release v0.7.100 (#3093)
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
2026-08-07 16:12:25 -07:00
Miguel Ángel 9aa90f6e3e chore: release v0.7.99 2026-08-07 15:31:57 +00:00
Vance Ingalls 6114749d8e chore: release v0.7.98 2026-08-06 23:37:23 -07:00
Miguel Ángel c4b41072c7 chore: release v0.7.97 2026-08-07 05:16:27 +00:00
Vance Ingalls 026e6941ac chore: release v0.7.96 2026-08-06 14:53:26 -07:00
Miguel Ángel fc0298de4a chore: release v0.7.95 (#3068) 2026-08-06 10:12:20 -07:00
Miguel Ángel 29f004cfc0 chore: release v0.7.94 (#3045) 2026-08-04 20:42:17 -07:00
Miguel Ángel a4eb602ee2 chore: release v0.7.93 (#3044) 2026-08-04 20:27:07 -07:00
Vance Ingalls df8b9604fe chore: release v0.7.92 2026-08-04 01:49:05 -07:00
Vance Ingalls 8c6cf90ff9 chore: release v0.7.91 2026-08-03 22:48:00 -07:00
Vance Ingalls 71fd96bbf1 Merge pull request #2854 from heygen-com/feat/canary-rollouts
feat(core): percentage-based canary rollouts + calibration experiment
2026-08-03 22:35:37 -07:00
James Russo 1e51eaec2c chore: release v0.7.90 (#2958) 2026-08-02 21:57:19 -07:00
Miguel Ángel 411ada0d90 chore: release v0.7.89 (#2953) 2026-08-02 20:25:14 +02:00
Vance Ingalls 74fadf69c4 chore: release v0.7.88 2026-08-01 16:53:08 -07:00
Miguel Ángel 89e970fbd7 chore: release v0.7.87 (#2935) 2026-07-31 22:36:46 +02:00
Vance IngallsandClaude Opus 5 71ee156dac feat(studio): browser canary binding + leaf subpath imports
Adds the Studio (browser) binding so a canary can span the CLI and the editor,
and fixes a bundling mistake the studio test suite caught.

## The binding

Same public API as the CLI — `isCanaryEnabled("name")` — so a call site reads
identically whether it runs in Node or the browser. Three inputs differ:

- UNIT ID: `resolveStudioDistinctId()`, which already adopts
  `window.__HF_CLI_DISTINCT_ID` when the CLI launched Studio. A CLI-launched
  Studio therefore lands in the SAME cohort as the CLI: a rollout spanning
  render and editor is coherent for that user instead of enrolling their
  terminal but not their editor. A test pins that the id is passed through
  unmodified — prefixing or re-hashing it would silently break that parity.

- OVERRIDE: no `process.env` in a page, so `?hf_canary_<name>=on` mirrored
  into sessionStorage. Session scope is deliberate. A URL is the right carrier
  (shareable — "support: open this link"), but persisting a URL-borne override
  to localStorage would let one click silently pin a browser into a cohort
  forever, long after anyone remembers why. Closing the tab is the reset;
  `=reset` clears it explicitly.

- EXCLUSION: `navigator.webdriver` stands in for the CLI's `is_ci`. Automated
  browsers mint a fresh localStorage id per run, so they would hop cohorts
  between runs — noise in the signal, nothing learned about real users. An
  override still reaches them, which is how you test a canary under Playwright.

Studio's `trackEvent` now attaches `canaries` to every event, mirroring the CLI.

## The bundling fix

Importing the `@hyperframes/core` barrel into studio browser code broke two
unrelated hook test files with an esbuild TextEncoder invariant violation. The
barrel re-exports the whole core surface (parsers, lint, studio-server), so it
drags a Node-oriented dependency graph into a browser bundle — the test
failure was the symptom, the bundle bloat was the bug.

`@hyperframes/core` now exposes `./canary` and `./canary-registry`, declared in
packages/core/package-subpaths.json (the generated source of truth for exports —
hand-editing package.json is reverted by the sync script) and marked
`environments: [browser, bun, node]`. Both the studio AND cli bindings import
the leaf modules; the CLI gets the same benefit for a different reason, since
this resolves on the startup path — the reason the producer is lazily loaded.

Verified: the two hook files pass again; 269 studio files / 2982 tests, 98
core / 1433, 166 cli / 2194 green, `bun run lint` clean including the subpath
check. Fault-injection confirms both design decisions are pinned — swapping
session for local storage fails the scope test, prefixing the unit id fails the
CLI/Studio cohort-parity test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:14:35 -07:00
Vance Ingalls 12ee861123 chore: release v0.7.85 2026-07-30 10:38:10 -07:00
Vance Ingalls 065293ecf3 chore: release v0.7.84 2026-07-30 01:39:54 -07:00
Vance Ingalls 5244dde5f1 chore: release v0.7.83 2026-07-29 18:37:58 -07:00
Miguel Ángel b904343949 chore: release v0.7.82 2026-07-29 11:25:04 +00:00
James 0d42d65525 chore: release v0.7.81 2026-07-29 06:11:11 +00:00
Miguel Ángel 5829515932 chore: release v0.7.80 2026-07-29 02:22:54 +00:00
Miguel Angel Simon Sierra 659e22656e fix(studio): switch keyframe ease modes optimistically 2026-07-29 03:44:31 +02:00
Miguel Ángel a68729bcf1 chore: release v0.7.79 2026-07-28 23:06:45 +00:00
Vance Ingalls c691869e22 chore: release v0.7.78 2026-07-28 05:27:29 -07:00
Miguel Ángel 3c857d768b chore: release v0.7.77 2026-07-28 02:12:41 +00:00
James 5cad2bc312 chore: release v0.7.76 2026-07-27 05:08:40 +00:00
James 45b458c007 chore: release v0.7.75 2026-07-27 03:38:16 +00:00
James Russo 0cc78c1d42 chore: release v0.7.74 (#2818) 2026-07-26 21:57:54 -04:00
James 527751c871 chore: release v0.7.73 2026-07-26 21:25:03 +00:00
Ular Kimsanov e2e61b0767 Merge pull request #2798 from heygen-com/feat/agent-native-color-grading
feat(cli): expose agent-native color grading
2026-07-26 03:23:29 -07:00
ukimsanov c1dde28980 fix(cli): address media treatment review findings 2026-07-26 01:25:42 -07:00
James Russo 72ae39e3d0 chore: release v0.7.72 (#2800) 2026-07-26 02:01:10 -04:00
Miguel Angel Simon Sierra 5acbf240cb feat(core): add deterministic keyframe ease runtime 2026-07-25 14:12:16 +02:00
James d07d1874b6 chore: release v0.7.71 2026-07-24 19:03:16 +00:00
Vance Ingalls b9c3e3a4b6 chore: release v0.7.70 2026-07-23 20:59:30 -07:00
James f0034228f5 chore: release v0.7.69 2026-07-23 12:03:36 -07:00
James 71d84ff27f chore: release v0.7.68 2026-07-22 04:41:54 +00:00
James 4b6bb8ffa9 chore: release v0.7.67 2026-07-21 23:51:05 +00:00
Miguel Angel Simon Sierra 84841b8dae chore: release v0.7.66 2026-07-21 14:36:22 +02:00