ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-01 19:42:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(audio): play automation envelopes in preview and bake them at render (#3016)

Browse Source 
* feat(engine): render audio FX in an OfflineAudioContext

Reads `data-fx-chain` off an audio element and runs the chain over the trimmed
WAV before volume automation is baked in — effects should see the raw signal,
and the envelope belongs on their output.

The processing happens in an OfflineAudioContext inside the headless browser
the engine already drives, running the same graph builders the studio previews
with. That is the point of the approach: one implementation per effect, so the
render agreeing with the preview is a property of the architecture rather than
a tolerance to police. Reimplementing each effect as an FFmpeg filter would
mean two implementations to keep in step, and for the dynamics processors and
modulated delays there is no filter that behaves the same way.

`build:audio-fx-runtime` bundles the graph builders into an injectable IIFE,
following the same pattern as the existing runtime artifacts, so the browser
runs exactly the code the studio does.

The page loads from a file:// URL rather than about:blank because AudioWorklet
is only exposed in a secure context — the compressor, limiter, gate and
bitcrush processors would otherwise fail to register with an opaque error.
file:// qualifies and needs no listening socket.

The chain is serialised into the attribute the way colour grading carries its
config, so there is no side-car file to resolve or lose.

An FX failure is fatal for the whole mix rather than a per-track soft failure.
Every other audio failure mode degrades gracefully — the track drops, siblings
continue — but substituting the dry signal for a processed one ships a render
that sounds plausible and is not what the author set up. Since the per-element
work races under Promise.all, an internal AbortController chained off the
caller's signal aborts in-flight siblings before workDir is removed.

* feat(core): voiceover carve analysis

Finds the bands a voice occupies so a music bed can be dipped there, letting
the voice sit in front without ducking the whole track.

Carve is a relationship between two tracks rather than an effect on one, so it
stays out of the FX chain. What it emits is an ordinary chain of peaking
filters, so a carve composes with whatever else is on the track and needs no
separate rendering path.

Selection is weighted toward intelligibility rather than raw voice energy.
Ranking purely by power lands on the fundamental almost every time, because
that is where a voice is loudest — but the masking that actually hurts a
voiceover happens higher up, and dipping 160 Hz mostly just thins the bed. The
bias is a control, not a constant: at 0 it follows raw energy, at 1 it weights
toward 1-3 kHz.

Ranking happens in dB, which matters more than it looks. Speech spreads 20-30 dB
across these bands — it falls off roughly 6 dB per octave above the fundamental
— so a weighting has to be on that scale to move anything at all. A
multiplicative weight of `1 - bias + bias * shaped` is bounded below by
`1 - bias`, capping its influence at 10*log10(1/(1 - bias)): 5.2 dB at the 0.7
default, 3 dB at 0.5. That is no influence against a real voice — every bias
short of ~0.95 would rank exactly like bias 0 and carve the fundamental, the
outcome the bias exists to prevent, while looking decisive against a fixture
whose bands sit 2 dB apart. So the bias is a dB penalty, zero at 2 kHz and worth
up to 30 dB at full strength, and relative cut depths come from a dB difference
rather than a ratio of weighted linear powers.

The bias reweights ranking without overriding the spectrum — a band the voice
has no energy in is not worth carving, and scores -Infinity rather than
competing — so a strongly low-pitched voice can still select low at full bias.
What the tests hold is that biasing never selects lower than the unbiased
ranking, that the DEFAULT bias reaches the presence region on a voice with a
realistic tilt, and that bias 0 still follows raw power exactly.

Includes a radix-2 FFT rather than a dependency; one Welch-style averaged
spectrum over third-octave bands does not justify pulling in a DSP library.

* fix(engine): keep the FX render 16-bit, stereo, and correctly sized

Three defects in the offline FX path, none of which any test could see.

**Float output silently disabled sample-accurate volume automation.** The writer
emitted 32-bit IEEE float; the very next mixer step bakes the volume envelope
into the samples and accepts only 16-bit PCM, returning null otherwise. So
enabling any effect downgraded that track to the ffmpeg expression path — capped
at 32 straight segments, quantising a curved envelope, and on a dense one falling
back to base volume. It now writes 16-bit PCM, clamped rather than wrapped so a
limiter at 0 dB or a resonant filter cannot turn overshoot into a click. A test
asserts the baker accepts the writer's own output and actually fades it.

**Everything was folded to mono.** `prepareAudioTrack` goes out of its way to
emit stereo — its pan filter exists to dodge ffmpeg's 3 dB mono-to-stereo
rematrix — and this folded it, then wrote one channel. So adding a single peaking
EQ collapsed a bed's width and cost ~3 dB in the render, while preview stayed
stereo. Channels now travel as one plane each, through an OfflineAudioContext of
the same width, and come back interleaved.

**Small results decoded the wrong length.** `new Float32Array(buf.buffer)`
discards byteOffset and byteLength, and Node pools small allocations: a 400-byte
payload sits at offset 8 inside an 8 KiB pool, so a clip under ~1024 samples
decoded as 2048 samples of unrelated memory — and the empty-result guard could
not see it. The reader has the mirror-image fix: a float data chunk on an odd
boundary (ffmpeg's pcm_f32le writes fmt(18) + fact, landing `data` at 58) now
copies instead of throwing RangeError on an unaligned view.

The tail limitation is now stated rather than mis-stated: the context is exactly
as long as the input, so a reverb or delay still ringing is cut there. The old
comment claimed the opposite. How far a tail may run past a clip's end changes
the clip's length in the mix, so it is a product decision, not one to make here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(producer): report an FX render failure as an audio error

`processCompositionAudio` reports per-track failures in its result, but an FX
failure it cannot degrade past — a browser that will not launch, a chain that
will not build — rejects instead. `runAudioStage` had no try, so that rejection
escaped to the orchestrator as an unclassified pipeline exception, losing the
stage/owner/retryable classification this stage exists to attach, and skipping
its abort check on the way out.

It now lands in `audioError` alongside every other cause, while an abort still
keeps its own shape rather than being reported as an audio problem.

Not done here: committing the generated `audio-fx-runtime-inline.ts` so a fresh
clone typechecks packages/engine without building first. The bundle is built from
the stub, and the stub changes three times across this stack — so the artifact
differs per branch and would conflict on every restack. Its model,
position-edits-render-inline.ts, is committed only because it is stable. Building
before testing is this monorepo's existing contract (studio's tests need core's
dist too), so the gap is not specific to audio FX and is better closed by a build
ordering gate than by committing a per-branch artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(engine): skip the browser FX render cases when there is no browser

CI's `Test` job was red on this PR with four failures, all the same cause:

  Failed to launch the browser process: spawn
  /home/runner/.cache/hyperframes/chrome/chrome-headless-shell

The job installs ffmpeg and no browser, deliberately — every other suite
that needs an external binary already guards on it
(`describe.skipIf(!HAS_FFMPEG)`). These cases were the only ones assuming
a Chrome, so they failed on an absent dependency rather than on anything
about the code.

Guards on `resolveHeadlessShellPath()` — the same resolver
`acquireBrowser` launches through, so the check cannot drift from the
thing it guards the way a hard-coded cache path would. A configured path
that does not exist throws; that is caught and read as "cannot run here".

Checked both directions rather than just the green one: with a browser all
11 cases run and pass, and with `HYPERFRAMES_BROWSER_PATH` pointed at a
missing binary exactly 3 skip and the other 8 still run. A guard that
silently skipped everything would have looked identical in CI.

They keep their value where it exists — every developer machine, and any
job that has run `hyperframes browser ensure`.

Not touched: the CodeQL failure on this PR is a run from 2026-08-07, five
days and several force-pushes stale. None of the 17 open repo alerts are
in files this PR changes; it re-runs on this push.

* chore(engine): suppress the temp-file alert with the reason it is safe

CodeQL flags `writeWav`'s `writeFileSync` as js/insecure-temporary-file
(high) — the one new alert on #3021, and the reason its CodeQL check is
red.

It is a false positive, and the comment says why rather than just silencing
it: `path` is always inside a directory made by `mkdtempSync`, never a
name assembled directly under `tmpdir()`. Both callers are covered — the
browser host page writes into `mkdtempSync(join(tmpdir(), "hf-fx-host-"))`,
and the render output goes to the producer work dir, itself
`mkdtempSync(join(tempRoot, "producer-project-"))`. mkdtemp picks the
random suffix and creates the directory 0700 in one syscall, so the
predictable filename inside it cannot be pre-created or symlinked by
another user, which is the attack the rule is about. The analyzer sees the
dataflow reach `tmpdir()` and not the mkdtemp in between.

Suppressed inline rather than dismissed in the UI, so the justification
lives next to the code and the rule stays live for anything added later in
this file. Matches the repo's existing convention — `planV2.ts:222`
carries an `lgtm[js/insecure-temporary-file]` for a different reason on
the same rule.

Correcting myself: I first reported this alert as not real, having
intersected the PR's files against the default-branch alert list, which
does not contain PR-ref alerts. Querying ?ref=refs/pull/3021/merge returns
it straight away.

* test(engine): probe ffmpeg and Chrome instead of assuming them

Two failures on #3021's Test job, both about the environment rather than
the code under test.

**Bare `ffmpeg` is not on PATH in CI.** The 16-bit fixture shelled out to
`execFileSync("ffmpeg", ...)` and died with ENOENT. The job does provide
ffmpeg, through `prepare-ffmpeg-bin`, which is what `getFfmpegBinary()`
resolves — every other ffmpeg-dependent suite in this package already goes
through it. Now this one does too, and the case is `skipIf(!HAS_FFMPEG)`
so a contributor without ffmpeg skips rather than fails.

**The browser guard trusted the wrong thing.** It asked
`resolveHeadlessShellPath()` and treated a returned path as "a browser is
here". CI's cache holds a chrome-headless-shell that resolves and then
fails to spawn — a partial download is indistinguishable from a working
one by `existsSync`, which is all that resolver checks. So the three
browser cases ran anyway and failed on the launch.

It now runs `--version` and requires exit 0, which is the same probe the
ffmpeg suites use: ask the binary, do not infer from the filesystem.

Checked both directions rather than just the green one. With a working
browser all 11 cases run and pass; with `HYPERFRAMES_BROWSER_PATH` pointed
at a binary that exits non-zero — CI's exact situation — exactly 3 skip
and the other 8 still run. A guard that quietly skipped everything would
have looked identical on the CI summary.

* feat(core): register the audio-fx-rack canary at 0%

Lands the rollout switch dark, per the registry's own procedure: "Start at
percentage: 0 and merge that — a canary at 0 is dead code you can land
safely and ramp without a code review."

Declared at the bottom of the stack so every branch above can read it. The
gate itself goes in at wa-4-fx-panel, where the rack first appears.

Scope is deliberate and stated in the description: it gates the AUTHORING
surface only. A composition that already carries `data-fx-chain` still
plays and renders it. A canary should stage who can REACH a feature, not
make an attribute somebody already wrote silently inert — an agent that
writes a chain through the skill would otherwise produce a file whose audio
processing vanishes with no error.

* feat(studio): audio FX panel generated from the registry

Controls for the whole chain: add, remove, reorder, bypass, and every knob each
effect declares.

Nothing in the panel knows what a compressor is. The registry supplies each
parameter's range, step, unit and scale and the panel renders what it finds, so
adding an effect or a knob upstream needs no change here, and the panel cannot
offer a value the renderer would reject — a typed-in figure is clamped into the
declared range on the way through.

Frequency and time controls span three or four decades, so those declare a log
scale and the slider maps exponentially; a linear slider would spend most of
its travel somewhere useless.

Reorder is a first-class control because chain order changes the sound: a
reverb before a compressor is not the same as after.

Carve gets its own block rather than an entry in the add menu, with a picker
for the voice track to listen to. It processes this track based on another one,
which is how a sidechain control works — it lives on the track that changes,
and names the source.

* feat(studio): show the Audio FX section on audio tracks

Adds `audioFx` to the editing-affordances contract and renders the FX panel in
the inspector when an `<audio>` element is selected.

The section is audio-only. A `<video>` carries its sound on a separate
`<audio>` element, so an FX chain on the video would have nothing to process.

Chain and carve settings are written straight back onto the element as
serialised attributes, the way colour grading carries its config, so
persistence is an ordinary attribute write and needs no new server route. A
chain that cannot be parsed renders as empty rather than breaking the panel,
and the attribute is left untouched until the user changes something.

The collapsed group summarises what is on the track ("2 effects + carve") so
the state is visible without expanding it.

Wired into PropertyPanelFlat rather than PropertyPanel: STUDIO_FLAT_INSPECTOR_ENABLED
defaults to true, so the flat inspector is what actually renders.

* refactor(studio): lift audioFxSummary out of PropertyPanelFlat

`PropertyPanelFlat.tsx` is 612 lines here against the repo's 600-line cap,
so the required File size check is red — the sole reason this PR is
blocked. The review says as much: "mechanical fix (~5 min), not a design
problem. Code itself is LGTM."

Moves `audioFxSummary` to `audioFxSummary.ts`, the same file a later
branch creates for it. Deliberately the smallest cut that clears the cap
rather than the whole `AudioFxGroup` extraction: every later commit in the
stack edits AudioFxGroup, so moving it here would collide with each of
them, while almost nothing touches this function.

595 lines.

* feat(core,studio): hear the FX chain in preview, and run the carve analysis

Splices an element's FX chain into the playback graph so preview stops being
silent about effects, and wires the carve button that was previously inert.

The chain goes between the decoded source and its gain stage: effects see the
raw signal and volume automation rides on their output, matching the order the
offline render uses. Since preview and render call the same graph builders,
what is heard while scrubbing is what gets written.

The splice lives in the transport rather than on the `<audio>` element. The
transport plays each track from a decoded AudioBuffer and mutes the element to
avoid doubling, so capturing the element with createMediaElementSource would
have processed a stream nothing is listening to — it looked like it worked
because the call succeeded, and the audio was unchanged.

A chain that cannot be built plays dry rather than silencing the track, which
is the right failure in preview: the author keeps working and hears the source.
The render still refuses, because shipping the dry signal there would be wrong.

Carve now analyses for real: it decodes the chosen voice track, ranks its bands
and writes the resulting peaking filters onto this track. Generated nodes are
tagged `fromCarve`, so re-running replaces the previous carve instead of
stacking another set on top of hand-added effects.

Known limitation: the graph is built when a source is scheduled, so a knob
turned mid-playback takes effect on the next play or seek rather than
immediately. Live re-parameterisation needs the transport to hold the handle
and forward updates.

* fix(studio,core): stop parameter drags from restarting playback

Dragging a knob wrote the chain through the persisting attribute path on every
input event. That path refreshes the preview, which reloads the composition and
reschedules audio — so a single drag reloaded dozens of times and playback
stuttered the whole way.

Drags now go through `onSetAttributeLive`, the same path colour grading uses for
scrubs: it coalesces undo entries and sets `skipRefresh`, so no reload happens.
The persisting write fires once, when the gesture ends — pointer-up or blur for
a slider, Enter or blur for a typed value. A select commits immediately since
there is no drag to wait for.

While dragging, the control is driven from local state. Waiting for the value to
round-trip through the element attribute made the knob lag behind the pointer.

For the change to be audible without a reload, the graph now follows the
attribute: the chain installed by the transport observes the element and
re-parameterises itself in place, so a value change lands on the next
128-sample quantum. A shape change (effect added, bypassed, pole count) cannot
be patched into a running graph, so it still waits for the next schedule rather
than cutting the audio mid-play.

The regression test drags a slider through several values and asserts the
persisting handler is untouched until release.

* feat(studio): put the audio FX rack behind its canary

Gates the rack on `isCanaryEnabled("audio-fx-rack")`, which is registered
at 0% — so the whole 47-PR stack can land without showing anyone a feature
that has not been measured yet.

The gate sits on the AUTHORING surface and nowhere else. The runtime and
the render still honour a `data-fx-chain` already on an element, so a
composition written through the skill or by `carve.mjs` keeps its
processing rather than going silently dry for anyone outside the cohort. A
canary should stage who can REACH a feature, not make an attribute somebody
already wrote stop working with no error.

Gated at the panel rather than in `resolveEditingSections`: the affordance
resolver is a pure function in core describing what an element CAN support,
and rollout state is not a property of an `<audio>` tag.

Pinned the 0% with a test, and checked it fails at 25 — a ramp should have
to break something that says "this ships dark" out loud.

One gap, stated rather than papered over: the gate itself has no unit test.
I wrote one and deleted it, because `PropertyPanel.test.tsx`'s harness
never renders the Audio FX group for its audio fixture even with the gate
removed — so the test passed for the wrong reason in the off case and could
not pass at all in the on case. A test that cannot fail for the right
reason is worse than none. Verifying the gate needs the panel harness to
mount that section first, which is its own change.

* fix(core): register FX worklets before building nodes that need them

An AudioWorkletNode cannot be constructed before its processor is registered —
it throws, and the surrounding chain is lost with it. `attachElementFxChain`
built the chain first and only then called `ensureAudioFxWorklets`, so every
worklet-backed effect (compressor, limiter, gate, bitcrush) threw on
construction and the track fell back to dry. Instrumenting the preview showed
`hf-compressor: InvalidStateError` with addModule never called at all.

When the module has not landed yet the track now plays dry and the graph is
swapped in once registration resolves, so the effect arrives a moment late
instead of never.

Registration is also tracked per context rather than in one module-level
promise. A processor registered on one AudioContext does not exist on another,
so the shared promise made every context after the first believe it was ready
when it was not — the studio's transport owns its own context, which is exactly
that case.

With the worklets actually running, the compressor's per-sample log10 and pow
became real audio-thread work. Samples below the knee have a gain of exactly
unity and need neither, so the envelope is now compared in the linear domain
and the transcendentals only run for samples that are actually being
compressed.

* refactor(studio): split the FX node row out of FxSection

Clears the health findings the FX stack left behind: the chain-node render
callback was a 70-line closure over half of FxSection's state, and the two
reorder arrows were the same button written twice.

Also drops two exports with no consumers, and registers the audio FX runtime
stub as an entry point — it is bundled by file path, so nothing imports it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(core): automation envelope model for audio tracks

Adds the data model behind Ableton-style automation lanes: breakpoint
envelopes over track volume or one knob of one effect in the track's FX
chain, stored on the element as `data-automation`.

Times are clip-local, so an envelope travels with the clip when it moves —
the clip-envelope model rather than arrangement automation.

`sampleAutomationLane` is the single interpolator. The lane drawing, the
preview scheduler and the render bake all call it, so the picture and the
sound cannot disagree about the curve. Log-scaled parameters interpolate in
log space, matching what their own knob already promises.

FX nodes gain a stable `id`, minted by count rather than randomly so the
document is the same on every machine. Lanes address nodes by id, so
reordering a chain never re-points a lane at a different effect, and a lane
whose effect was deleted is dropped rather than left to reattach.

Also warns when a track carries both a volume lane and a GSAP volume tween,
since only the lane is heard and the tween silently does nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(studio): lift the audio FX group out of PropertyPanelFlat

`PropertyPanelFlat.tsx` was 672 lines against the repo's 600-line cap, so
the required File size check was red — the sole reason #3014 and #3022 are
blocked. Both reviews say the same thing: "mechanical fix, not a design
problem. Code itself is LGTM."

Moves `AudioFxGroup` and `audioFxSummary` into
`propertyPanelAudioFxGroup.tsx`, which is where a later branch puts them
anyway — done here so the file is under the cap from the point it first
crosses it, rather than ten branches later.

533 lines now. The four audio imports it no longer needs go with it.

Not fixed here: three `FxSection carve` tests fail on this branch with
"Cannot read properties of undefined (reading 'toFixed')". Confirmed
pre-existing by stashing this change and re-running — that is the separate
`Test` failure the review also flags.

* feat(core): expose the AudioParams behind automatable FX knobs

Marks the knobs an automation lane can drive and has each graph builder hand
back the AudioParam behind them, so a scheduler can write to a running effect
without knowing what the effect is.

A knob is not always one AudioParam. A wet/dry mix is two gains moving in
opposition, and a knob in milliseconds drives a delay time in seconds, so
each target carries the mapping out of the knob's own declared unit.

What stays unautomatable is stated where it is decided: a WaveShaper curve, a
convolution impulse and a one-pole filter's coefficients are all rebuilt
wholesale rather than scheduled, and the four worklet effects take values by
postMessage rather than through AudioParams.

The registry flag is written by hand, so a test builds every effect and
checks the exposure both ways — nothing flagged is missing, nothing exposed
is unflagged. A flag that lied would offer a lane that silently did nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(core): play automation envelopes in preview

Schedules each lane onto the AudioParams behind its knob using native ramps
and value curves. Nothing evaluates the envelope per frame: it is handed to
the audio thread once, so it stays sample-accurate however busy the main
thread is, and the offline render will schedule it the same way.

Timing comes from the transport, so an envelope survives seeking into the
middle of a clip, a clip that has not started yet, and a playback rate that
compresses clip seconds into context seconds.

A straight line is only scheduled as a ramp when nothing bends it — no
curvature, a linear parameter scale, and no unit mapping. Log-scaled
parameters and mapped ones are sampled instead, since a delay knob in
milliseconds and a wet/dry pair moving in opposition are not linear in the
parameter they drive.

Lanes with nowhere to write are skipped rather than reported: a one-pole
filter exposes no frequency param, and the worklet effects expose none at
all. Editing an envelope mid-playback re-aims it at the live playhead rather
than restarting the track.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(engine): bake automation envelopes into the render

The offline render schedules FX lanes with the same scheduler preview uses,
inside the OfflineAudioContext that already runs the same graph builders. The
input WAV is the clip's own audio from its first sample, so clip-local time
is offline time and the envelope needs no offset.

Volume lanes take the existing PCM bake rather than a second mechanism: the
lane is converted to keyframes, so a straight fade stays two of them and only
a bent segment is sampled — the baker interpolates linearly and would
otherwise quietly straighten the curve. A volume lane supersedes keyframes
probed from the timeline, which `lint` already warns about.

A browser test sweeps a lowpass from below a 2 kHz tone to well above it and
measures both ends. Parsing the envelope is not the same as scheduling it,
and only running the real thing tells the two apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(studio): drop the FX panel's dead __testables export

Fallow audit flagged it — no test imports the module.

* fix(core,studio): clear the remaining Fallow audit findings on the FX panel

- Split FxSection's per-node row into FxNodeRow + FxNodeControls so the
  CRAP score (31.6, threshold 30) splits across two smaller units instead
  of moving wholesale with one extraction.
- Dedupe the repeated "open the add menu, read its items" block in
  propertyPanelFxSection.test.tsx into openAddMenuItems().
- Merge build-audio-fx-runtime.ts and build-position-edits-render.ts into
  one build-inline-artifact.ts, config-selected by CLI arg — the two
  scripts were a byte-for-byte clone save for names.
- Exempt canary.test.ts's rawFnv (a deliberate independent
  reimplementation used to cross-check canaryBucket, per its own
  docstring) and the property-panel test files' shared renderInto/mount
  scaffolding (pre-existing across 9 files, 2 outside this stack) in
  .fallowrc.jsonc, consistent with this file's existing exemptions for
  the same class of intentional/pre-existing duplication.

* fix(ci): allowlist the build-script consolidation in the no-main-deletions guard

build-audio-fx-runtime.ts and build-position-edits-render.ts were merged into
build-inline-artifact.ts to kill a fallow duplication finding; the deletion
guard flagged that as an accidental loss since main still has both originals.

* fix(core): dedupe the wet/dry mix math between delayFeedback and chorusLfo

Both effect builders set wet.gain to the mix and dry.gain to its complement
in identical two-line blocks; fallow kept re-flagging it as a 10-line clone
on every unrelated change. Extracted setWetDryMix.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-08-12 15:53:52 -07:00
committed by GitHub
co-authored by Claude Opus 5
parent fd97926cf6
commit 23ba58cfd3
11 changed files with 959 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/core/src/audio/audioFxAutomation.test.ts
+353
View File
@@ -0,0 +1,353 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import {
cancelParamLane,
scheduleChainAutomation,
scheduleParamLane,
volumeLane,
type AutomationTiming,
} from "./audioFxAutomation.js";
import type { FxParamTarget } from "./audioFxGraph.js";
import type { HfAutomationLane } from "../audioAutomation.js";
import type { HfAudioFxChain } from "../audioFx.js";
/**
* happy-dom has no Web Audio. These record what the scheduler asks an
* AudioParam to do — the shape of the envelope handed to the audio thread —
* which is the part that has to be right; whether Chrome then plays a ramp
* accurately is not ours to test.
*/
type Call =
| { op: "set"; value: number; time: number }
| { op: "ramp"; value: number; time: number }
| { op: "curve"; values: number[]; time: number; duration: number }
| { op: "cancel"; time: number }
| { op: "hold"; time: number };
class FakeParam {
calls: Call[] = [];
value = 0;
setValueAtTime(value: number, time: number): void {
this.calls.push({ op: "set", value, time });
}
linearRampToValueAtTime(value: number, time: number): void {
this.calls.push({ op: "ramp", value, time });
}
setValueCurveAtTime(values: Float32Array, time: number, duration: number): void {
this.calls.push({ op: "curve", values: Array.from(values), time, duration });
}
cancelScheduledValues(time: number): void {
this.calls.push({ op: "cancel", time });
}
cancelAndHoldAtTime(time: number): void {
this.calls.push({ op: "hold", time });
}
}
const fake = (): { target: FxParamTarget; param: FakeParam } => {
const param = new FakeParam();
return { target: { param: param as unknown as AudioParam }, param };
};
const at = (elapsed = 0, rate = 1, scheduledAt = 10): AutomationTiming => ({
scheduledAt,
elapsed,
rate,
});
const ramp: HfAutomationLane = {
target: "volume",
points: [
{ t: 1, v: 0.2 },
{ t: 3, v: 0.8 },
],
};
describe("scheduleParamLane", () => {
it("seeds the current value, then ramps to each later point", () => {
const { target, param } = fake();
scheduleParamLane([target], ramp, "linear", at(0));
expect(param.calls).toEqual([
{ op: "cancel", time: 10 },
// Before the first point the envelope holds that point's value.
{ op: "set", value: 0.2, time: 10 },
{ op: "ramp", value: 0.8, time: 13 },
]);
});
it("enters a segment mid-way when the playhead landed inside it", () => {
const { target, param } = fake();
scheduleParamLane([target], ramp, "linear", at(2));
// Half way along a 0.2 → 0.8 ramp.
expect(param.calls[1]).toEqual({ op: "set", value: 0.5, time: 10 });
expect(param.calls[2]).toEqual({ op: "ramp", value: 0.8, time: 11 });
});
it("holds the last value once the envelope is behind the playhead", () => {
const { target, param } = fake();
scheduleParamLane([target], ramp, "linear", at(9));
expect(param.calls).toEqual([
{ op: "cancel", time: 10 },
{ op: "set", value: 0.8, time: 10 },
]);
});
it("waits for a clip that has not started yet", () => {
const { target, param } = fake();
scheduleParamLane([target], ramp, "linear", at(-2));
// Clip time 1 is three seconds of context time away.
expect(param.calls[1]).toEqual({ op: "set", value: 0.2, time: 10 });
expect(param.calls[2]).toEqual({ op: "ramp", value: 0.8, time: 15 });
});
it("compresses the envelope by the playback rate", () => {
const { target, param } = fake();
scheduleParamLane([target], ramp, "linear", at(0, 2));
// Clip time 3 arrives after 1.5 s of context time at double speed.
expect(param.calls[2]).toEqual({ op: "ramp", value: 0.8, time: 11.5 });
});
it("sets a constant lane once instead of scheduling it", () => {
const { target, param } = fake();
scheduleParamLane(
[target],
{
target: "volume",
points: [
{ t: 0, v: 0.4 },
{ t: 5, v: 0.4 },
],
},
"linear",
at(0),
);
expect(param.calls).toEqual([
{ op: "cancel", time: 10 },
{ op: "set", value: 0.4, time: 10 },
]);
});
it("samples a curved segment rather than ramping through the bend", () => {
const { target, param } = fake();
scheduleParamLane(
[target],
{
target: "volume",
points: [
{ t: 0, v: 0, curve: 1 },
{ t: 2, v: 1 },
],
},
"linear",
at(0),
);
const curve = param.calls.find((c) => c.op === "curve");
expect(curve).toBeTruthy();
if (curve?.op !== "curve") throw new Error("expected a curve");
expect(curve.time).toBe(10);
expect(curve.duration).toBe(2);
expect(curve.values[0]).toBeCloseTo(0, 6);
expect(curve.values[curve.values.length - 1]).toBeCloseTo(1, 6);
// Curved, so the midpoint is not halfway.
expect(curve.values[Math.floor(curve.values.length / 2)]).toBeLessThan(0.4);
});
it("samples a log-scaled sweep, which a linear ramp would get wrong", () => {
const { target, param } = fake();
scheduleParamLane(
[target],
{
target: "fx.n1.frequency",
points: [
{ t: 0, v: 200 },
{ t: 2, v: 8000 },
],
},
"log",
at(0),
);
const curve = param.calls.find((c) => c.op === "curve");
if (curve?.op !== "curve") throw new Error("expected a curve");
// Halfway is the geometric mean, not the arithmetic one. The array has an
// even length so no sample sits exactly on it; both neighbours bracket it.
const half = curve.values.length / 2;
const geometric = Math.sqrt(200 * 8000);
expect(curve.values[half - 1] ?? 0).toBeLessThan(geometric);
expect(curve.values[half] ?? 0).toBeGreaterThan(geometric);
expect((curve.values[half] ?? 0) / geometric).toBeCloseTo(1, 1);
// A linear ramp would have been at 4100 by now — nowhere near.
expect(curve.values[half] ?? 0).toBeLessThan(2000);
});
it("does not seed on top of a curve that starts at the same instant", () => {
const { target, param } = fake();
scheduleParamLane(
[target],
{
target: "volume",
points: [
{ t: 0, v: 0, curve: 1 },
{ t: 2, v: 1 },
],
},
"linear",
at(0),
);
// A value curve may not overlap another event, so there is no set at 10.
expect(param.calls.filter((c) => c.op === "set")).toEqual([]);
expect(param.calls[1]?.op).toBe("curve");
});
it("writes every AudioParam behind one knob, each through its own mapping", () => {
const wet = new FakeParam();
const dry = new FakeParam();
scheduleParamLane(
[
{ param: wet as unknown as AudioParam },
{ param: dry as unknown as AudioParam, map: (v) => 1 - v },
],
{
target: "fx.n1.mix",
points: [
{ t: 0, v: 0.25 },
{ t: 4, v: 0.75 },
],
},
"linear",
at(0),
);
expect(wet.calls[1]).toEqual({ op: "set", value: 0.25, time: 10 });
// The dry side moves the opposite way. A mapping may be non-linear, so the
// segment is sampled rather than ramped.
const dryCurve = dry.calls.find((c) => c.op === "curve");
if (dryCurve?.op !== "curve") throw new Error("expected a curve");
expect(dryCurve.values[0]).toBeCloseTo(0.75, 6);
expect(dryCurve.values[dryCurve.values.length - 1]).toBeCloseTo(0.25, 6);
});
it("ignores an empty lane", () => {
const { target, param } = fake();
scheduleParamLane([target], { target: "volume", points: [] }, "linear", at(0));
expect(param.calls).toEqual([]);
});
});
describe("cancelParamLane", () => {
it("holds the value the envelope had reached rather than snapping back", () => {
const { target, param } = fake();
cancelParamLane([target], 12);
expect(param.calls).toEqual([{ op: "hold", time: 12 }]);
});
});
describe("scheduleChainAutomation", () => {
const chain: HfAudioFxChain = {
version: 1,
nodes: [{ type: "peaking", id: "n1", enabled: true, params: {} }],
};
const nodeWith = (automation: Record<string, FxParamTarget[]> | undefined) => [
{
id: "n1",
handle: {
input: {} as AudioNode,
output: {} as AudioNode,
update: () => {},
dispose: () => {},
automation,
},
},
];
it("routes a lane to the AudioParam its node exposes", () => {
const param = new FakeParam();
const nodes = nodeWith({ frequency: [{ param: param as unknown as AudioParam }] });
const scheduled = scheduleChainAutomation(
{
version: 1,
lanes: [
{
target: "fx.n1.frequency",
points: [
{ t: 0, v: 200 },
{ t: 2, v: 8000 },
],
},
],
},
chain,
nodes,
at(0),
);
expect(scheduled.length).toBe(1);
expect(param.calls.length).toBeGreaterThan(1);
});
it("skips a lane whose node exposes nothing, instead of failing", () => {
const scheduled = scheduleChainAutomation(
{
version: 1,
lanes: [
{
target: "fx.n1.frequency",
points: [
{ t: 0, v: 200 },
{ t: 2, v: 900 },
],
},
],
},
chain,
nodeWith(undefined),
at(0),
);
expect(scheduled).toEqual([]);
});
it("skips a lane addressed to a node that is not in the chain", () => {
const param = new FakeParam();
const scheduled = scheduleChainAutomation(
{
version: 1,
lanes: [
{
target: "fx.gone.frequency",
points: [
{ t: 0, v: 200 },
{ t: 2, v: 900 },
],
},
],
},
chain,
nodeWith({ frequency: [{ param: param as unknown as AudioParam }] }),
at(0),
);
expect(scheduled).toEqual([]);
expect(param.calls).toEqual([]);
});
it("leaves the volume lane alone — the transport owns the fader", () => {
const param = new FakeParam();
const automation = {
version: 1,
lanes: [
{
target: "volume",
points: [
{ t: 0, v: 1 },
{ t: 2, v: 0 },
],
},
],
};
const scheduled = scheduleChainAutomation(
automation,
chain,
nodeWith({ frequency: [{ param: param as unknown as AudioParam }] }),
at(0),
);
expect(scheduled).toEqual([]);
expect(volumeLane(automation)?.points.length).toBe(2);
});
});
packages/core/src/audio/audioFxAutomation.ts
+213
View File
@@ -0,0 +1,213 @@
/**
* Plays automation envelopes through native AudioParam scheduling.
*
* Nothing here runs per frame. The envelope is handed to the audio thread once,
* as ramps and curves, so it stays sample-accurate no matter what the main
* thread is doing — and the offline render schedules it the same way, which is
* why preview and render agree by construction rather than by tolerance.
*/
import {
isConstantLane,
resolveAutomationRange,
sampleAutomationCurve,
sampleAutomationLane,
type HfAutomation,
type HfAutomationLane,
} from "../audioAutomation.js";
import { parseAutomationTarget, VOLUME_TARGET } from "../audioAutomation.js";
import type { HfAudioFxChain } from "../audioFx.js";
import type { FxNodeHandle, FxParamTarget } from "./audioFxGraph.js";
/** Where the clip sits in context time when its source is scheduled. */
export interface AutomationTiming {
/** Context time the source was scheduled at. */
scheduledAt: number;
/** Clip-local seconds already elapsed at that moment; negative before it starts. */
elapsed: number;
/** Playback rate, which compresses clip seconds into context seconds. */
rate: number;
}
/**
* Sampling density for segments that cannot be expressed as a ramp. Dense
* enough that a curve is inaudible from a continuous one, capped so a long
* segment cannot allocate without bound.
*/
const CURVE_POINTS_PER_SEC = 100;
const MIN_CURVE_POINTS = 8;
const MAX_CURVE_POINTS = 4096;
const identity = (v: number): number => v;
type Op =
| { kind: "set"; time: number; value: number }
| { kind: "ramp"; time: number; value: number }
| { kind: "curve"; time: number; duration: number; values: Float32Array };
/**
* A segment is a straight line only when nothing bends it: no curvature on the
* point it leaves, a linear parameter scale, and no unit mapping that could be
* non-linear. Everything else is sampled.
*/
function isStraight(
curve: number | undefined,
scale: "linear" | "log",
map: FxParamTarget["map"],
): boolean {
return !curve && scale === "linear" && !map;
}
function curvePointCount(duration: number): number {
const wanted = Math.ceil(duration * CURVE_POINTS_PER_SEC);
return Math.min(MAX_CURVE_POINTS, Math.max(MIN_CURVE_POINTS, wanted));
}
/**
* One segment of the envelope, as the event that plays it. Returns null when
* the segment is already behind the playhead.
*/
function segmentOp(
lane: HfAutomationLane,
index: number,
scale: "linear" | "log",
timing: AutomationTiming,
map: FxParamTarget["map"],
): Op | null {
const { scheduledAt, elapsed, rate } = timing;
const apply = map ?? identity;
const toCtx = (t: number): number => scheduledAt + (t - elapsed) / rate;
const a = lane.points[index]!;
const b = lane.points[index + 1]!;
const toTime = toCtx(b.t);
if (toTime <= scheduledAt) return null;
const fromTime = Math.max(scheduledAt, toCtx(a.t));
const duration = toTime - fromTime;
if (duration <= 0) return null;
if (isStraight(a.curve, scale, map)) {
return { kind: "ramp", time: toTime, value: apply(b.v) };
}
// Sample from wherever the segment is entered, which is mid-segment when the
// playhead landed inside it.
const count = curvePointCount(duration);
const raw = sampleAutomationCurve(lane, Math.max(a.t, elapsed), b.t, count, scale);
const values = new Float32Array(count);
for (let k = 0; k < count; k += 1) values[k] = apply(raw[k] ?? 0);
return { kind: "curve", time: fromTime, duration, values };
}
/** Turn a lane into the events one AudioParam should receive. */
function planOps(
lane: HfAutomationLane,
scale: "linear" | "log",
timing: AutomationTiming,
map: FxParamTarget["map"],
): Op[] {
const ops: Op[] = [];
for (let i = 0; i + 1 < lane.points.length; i += 1) {
const op = segmentOp(lane, i, scale, timing, map);
if (op) ops.push(op);
}
// Hold the envelope's value at the moment playback starts. This is what makes
// a lane correct before its first point and after its last, and it gives a
// following ramp something to ramp from.
//
// A value curve may not overlap another event, so the seed is dropped when a
// curve already begins at that instant — the curve sets the value itself.
const first = ops[0];
if (!(first?.kind === "curve" && first.time <= timing.scheduledAt)) {
ops.unshift({
kind: "set",
time: timing.scheduledAt,
value: (map ?? identity)(sampleAutomationLane(lane, timing.elapsed, scale)),
});
}
return ops;
}
function emit(param: AudioParam, ops: readonly Op[]): void {
for (const op of ops) {
if (op.kind === "set") param.setValueAtTime(op.value, op.time);
else if (op.kind === "ramp") param.linearRampToValueAtTime(op.value, op.time);
else param.setValueCurveAtTime(op.values, op.time, op.duration);
}
}
/**
* Schedule one lane onto every AudioParam behind its knob.
*
* Existing automation from an earlier schedule is cancelled first, so calling
* this again after an edit replaces the envelope rather than layering on it.
*/
export function scheduleParamLane(
targets: readonly FxParamTarget[],
lane: HfAutomationLane,
scale: "linear" | "log",
timing: AutomationTiming,
): void {
if (lane.points.length === 0) return;
for (const target of targets) {
target.param.cancelScheduledValues(timing.scheduledAt);
if (isConstantLane(lane)) {
const value = (target.map ?? identity)(lane.points[0]!.v);
target.param.setValueAtTime(value, timing.scheduledAt);
continue;
}
emit(target.param, planOps(lane, scale, timing, target.map));
}
}
/** Stop an envelope, leaving the parameter wherever it currently sits. */
export function cancelParamLane(targets: readonly FxParamTarget[], from: number): void {
for (const target of targets) {
// Keep the value the ramp had reached rather than snapping back to the
// last explicitly-set one, which would click.
if (typeof target.param.cancelAndHoldAtTime === "function") {
target.param.cancelAndHoldAtTime(from);
} else {
target.param.cancelScheduledValues(from);
}
}
}
/** One built effect, paired with the chain node id lanes address it by. */
export interface AutomatableNode {
id?: string;
handle: FxNodeHandle;
}
/**
* Schedule every FX lane in an automation set against a built chain.
*
* Lanes with nowhere to write are skipped rather than reported: a one-pole
* filter exposes no frequency param, and a worklet effect exposes none at all.
* Returns the targets that were scheduled, so they can be cancelled together.
*/
export function scheduleChainAutomation(
automation: HfAutomation,
chain: HfAudioFxChain,
nodes: readonly AutomatableNode[],
timing: AutomationTiming,
): FxParamTarget[] {
const byId = new Map(nodes.filter((n) => n.id).map((n) => [n.id as string, n.handle]));
const scheduled: FxParamTarget[] = [];
for (const lane of automation.lanes) {
const parsed = parseAutomationTarget(lane.target);
if (!parsed || parsed.kind !== "fx") continue;
const targets = byId.get(parsed.nodeId)?.automation?.[parsed.param];
if (!targets || targets.length === 0) continue;
const range = resolveAutomationRange(lane.target, chain);
if (!range) continue;
scheduleParamLane(targets, lane, range.scale, timing);
scheduled.push(...targets);
}
return scheduled;
}
/** The track's volume lane, if it has one. */
export function volumeLane(automation: HfAutomation): HfAutomationLane | null {
return automation.lanes.find((lane) => lane.target === VOLUME_TARGET) ?? null;
}
packages/core/src/audio/audioFxGraph.ts
+5 -2
View File
@@ -435,6 +435,8 @@ export function buildFxNode(
export interface FxChainHandle {
input: AudioNode;
output: AudioNode;
/** Built effects in chain order, carrying the node ids lanes address. */
nodes: { id?: string; type: string; handle: FxNodeHandle }[];
/** Re-parameterise in place when the shape is unchanged; false if a rebuild is needed. */
update(chain: HfAudioFxChain): boolean;
dispose(): void;
@@ -469,7 +471,7 @@ function shapeOf(chain: HfAudioFxChain): string {
export function buildFxChain(ctx: BaseAudioContext, chain: HfAudioFxChain): FxChainHandle {
const input = ctx.createGain();
const output = ctx.createGain();
const handles: { type: string; handle: FxNodeHandle }[] = [];
const handles: { id?: string; type: string; handle: FxNodeHandle }[] = [];
let tail: AudioNode = input;
for (const node of chain.nodes) {
@@ -477,7 +479,7 @@ export function buildFxChain(ctx: BaseAudioContext, chain: HfAudioFxChain): FxCh
const handle = buildFxNode(ctx, node.type, node.params ?? {});
tail.connect(handle.input);
tail = handle.output;
handles.push({ type: node.type, handle });
handles.push({ ...(node.id ? { id: node.id } : {}), type: node.type, handle });
}
tail.connect(output);
@@ -486,6 +488,7 @@ export function buildFxChain(ctx: BaseAudioContext, chain: HfAudioFxChain): FxCh
return {
input,
output,
nodes: handles,
update(next) {
if (shapeOf(next) !== shape) return false;
const active = next.nodes.filter((node) => node.enabled !== false);
packages/core/src/runtime/audioFx.ts
+83 -2
View File
@@ -11,6 +11,18 @@
*/
import { HF_AUDIO_FX_ATTR, parseAudioFxChain, type HfAudioFxChain } from "../audioFx.js";
import {
HF_AUDIO_AUTOMATION_ATTR,
parseAutomation,
resolveAutomation,
type HfAutomation,
} from "../audioAutomation.js";
import {
cancelParamLane,
scheduleChainAutomation,
type AutomationTiming,
} from "../audio/audioFxAutomation.js";
import type { FxParamTarget } from "../audio/audioFxGraph.js";
import {
audioFxWorkletsReady,
buildFxChain,
@@ -20,6 +32,24 @@ import {
import type { FxChainHandle } from "../audio/audioFxGraph.js";
const EMPTY: HfAudioFxChain = { version: 1, nodes: [] };
const NO_AUTOMATION: HfAutomation = { version: 1, lanes: [] };
function readAutomation(
el: { getAttribute?(name: string): string | null },
chain: HfAudioFxChain,
): HfAutomation {
const raw =
(typeof el.getAttribute === "function" ? el.getAttribute(HF_AUDIO_AUTOMATION_ATTR) : null) ??
"";
if (!raw) return NO_AUTOMATION;
try {
return resolveAutomation(parseAutomation(raw), chain);
} catch {
// Unreadable automation plays the track flat rather than silencing it,
// matching how an unreadable chain plays dry. The render refuses instead.
return NO_AUTOMATION;
}
}
function readChain(el: { getAttribute?(name: string): string | null }): {
chain: HfAudioFxChain;
@@ -38,6 +68,18 @@ function readChain(el: { getAttribute?(name: string): string | null }): {
}
}
/**
* An element's automation lanes, bound to whatever chain it carries.
*
* FX lanes need the chain to resolve their target's range, so they are dropped
* for an element with no chain; a volume lane is always readable.
*/
export function readElementAutomation(el: {
getAttribute?(name: string): string | null;
}): HfAutomation {
return readAutomation(el, readChain(el).chain);
}
/**
* Splice an element's FX chain between a decoded source and its gain stage.
*
@@ -48,12 +90,16 @@ function readChain(el: { getAttribute?(name: string): string | null }): {
*
* Returns null when the element carries no chain, leaving the original
* source-to-gain connection in place.
*
* With `timing`, the element's automation lanes are scheduled onto the built
* effects as AudioParam ramps, and rescheduled when the attribute is edited.
*/
export function attachElementFxChain(
ctx: BaseAudioContext,
el: { getAttribute?(name: string): string | null },
source: AudioNode,
destination: AudioNode,
timing?: AutomationTiming,
): { dispose(): void } | null {
const { chain } = readChain(el);
if (chain.nodes.length === 0) {
@@ -103,6 +149,31 @@ export function attachElementFxChain(
source.connect(handle.input);
handle.output.connect(destination);
let automated: FxParamTarget[] = timing
? scheduleChainAutomation(readAutomation(el, chain), chain, handle.nodes, timing)
: [];
/**
* Re-aim the envelope at the live playhead. An edit lands mid-playback, so
* the clip has advanced past the offset the source was scheduled with.
*/
const timingNow = (): AutomationTiming | null => {
if (!timing) return null;
const now = typeof ctx.currentTime === "number" ? ctx.currentTime : timing.scheduledAt;
return {
scheduledAt: now,
elapsed: timing.elapsed + (now - timing.scheduledAt) * timing.rate,
rate: timing.rate,
};
};
const rescheduleAutomation = (next: HfAudioFxChain): void => {
const at = timingNow();
if (!at) return;
cancelParamLane(automated, at.scheduledAt);
automated = scheduleChainAutomation(readAutomation(el, next), next, handle.nodes, at);
};
// Follow the attribute while the source plays, so dragging a knob is heard
// without rescheduling the track. Values-only changes re-parameterise the
// running graph and land on the next 128-sample quantum; a shape change
@@ -116,14 +187,24 @@ export function attachElementFxChain(
) {
observer = new MutationObserver(() => {
const next = readChain(el);
if (next.chain.nodes.length > 0) handle.update(next.chain);
if (next.chain.nodes.length === 0) return;
handle.update(next.chain);
// Values pushed by `update` would fight a running envelope, so the lanes
// are re-scheduled on top of them from the current playhead.
rescheduleAutomation(next.chain);
});
observer.observe(target, {
attributes: true,
attributeFilter: [HF_AUDIO_FX_ATTR, HF_AUDIO_AUTOMATION_ATTR],
});
observer.observe(target, { attributes: true, attributeFilter: [HF_AUDIO_FX_ATTR] });
}
return {
dispose: () => {
observer?.disconnect();
if (automated.length > 0) {
cancelParamLane(automated, typeof ctx.currentTime === "number" ? ctx.currentTime : 0);
}
handle.dispose();
},
};
packages/core/src/runtime/webAudioTransport.ts
+28 -4
View File
@@ -1,4 +1,10 @@
import { attachElementFxChain } from "./audioFx.js";
import { attachElementFxChain, readElementAutomation } from "./audioFx.js";
import {
scheduleParamLane,
volumeLane,
type AutomationTiming,
} from "../audio/audioFxAutomation.js";
import { VOLUME_RANGE } from "../audioAutomation.js";
import { swallow } from "./diagnostics";
import { getDebugSurface } from "./globals.js";
@@ -57,6 +63,20 @@ function startBoundedSource(
return true;
}
/**
* The volume lane rides the fader, after the effects — where a DAW puts it,
* and the order the render bakes it in.
*/
function scheduleVolumeLane(
el: HTMLMediaElement,
gainNode: GainNode,
timing: AutomationTiming,
): void {
const lane = volumeLane(readElementAutomation(el));
if (!lane) return;
scheduleParamLane([{ param: gainNode.gain }], lane, VOLUME_RANGE.scale, timing);
}
export type ScheduledSource = {
el: HTMLMediaElement;
sourceNode: AudioBufferSourceNode;
@@ -188,15 +208,19 @@ export class WebAudioTransport {
const gainNode = this._ctx.createGain();
gainNode.gain.value = volume;
const elapsed = compositionTime - compositionStart;
const scheduledAt = this._ctx.currentTime;
const timing: AutomationTiming = { scheduledAt, elapsed, rate: safeRate };
// Splice the element's FX chain between the decoded source and its gain,
// so effects see the raw signal and volume automation rides on their
// output — the same order the offline render uses. Preview and render run
// the identical graph builders, so what is heard here is what is written.
const fx = attachElementFxChain(this._ctx, el, sourceNode, gainNode);
const fx = attachElementFxChain(this._ctx, el, sourceNode, gainNode, timing);
gainNode.connect(this._masterGain);
const elapsed = compositionTime - compositionStart;
const scheduledAt = this._ctx.currentTime;
scheduleVolumeLane(el, gainNode, timing);
this._rate = safeRate;
this._rateAnchorCtx = scheduledAt;
this._rateAnchorComp = compositionTime;
packages/core/stubs/audio-fx-runtime-entry.ts
+20 -2
View File
@@ -7,8 +7,8 @@
* ends share a single implementation rather than two that have to be kept in
* agreement.
*
* Exposes `window.__HF_AUDIO_FX.render(pcm, sampleRate, chain)`, which returns
* the processed samples.
* Exposes `window.__HF_AUDIO_FX.render(pcm, sampleRate, chain, automation)`,
* which returns the processed samples.
*/
import {
@@ -16,6 +16,8 @@ import {
chainNeedsWorklets,
ensureAudioFxWorklets,
} from "../src/audio/audioFxGraph.js";
import { scheduleChainAutomation } from "../src/audio/audioFxAutomation.js";
import { parseAutomation, resolveAutomation } from "../src/audioAutomation.js";
import { parseAudioFxChain, type HfAudioFxChain } from "../src/audioFx.js";
declare global {
@@ -25,6 +27,7 @@ declare global {
planes: Float32Array[],
sampleRate: number,
chainJson: string,
automationJson?: string,
): Promise<Float32Array[]>;
};
}
@@ -42,6 +45,7 @@ declare global {
* mix, so how far a tail may run past a clip's end is a product decision rather
* than something to pick here.
*/
/** The clip's audio as an AudioBuffer, a plane per channel. */
function toBuffer(
ctx: OfflineAudioContext,
@@ -71,6 +75,7 @@ async function render(
planes: Float32Array[],
sampleRate: number,
chainJson: string,
automationJson?: string,
): Promise<Float32Array[]> {
const chain: HfAudioFxChain = parseAudioFxChain(chainJson);
const channels = Math.max(1, planes.length);
@@ -83,6 +88,19 @@ async function render(
source.buffer = toBuffer(ctx, planes, channels, frames, sampleRate);
const fx = buildFxChain(ctx, chain);
// The input WAV is the clip's own audio from its first sample, so clip-local
// time is offline time — the envelope needs no offset here. Same scheduler as
// preview, which is what makes the two agree.
if (automationJson) {
const automation = resolveAutomation(parseAutomation(automationJson), chain);
scheduleChainAutomation(automation, chain, fx.nodes, {
scheduledAt: 0,
elapsed: 0,
rate: 1,
});
}
source.connect(fx.input);
fx.output.connect(ctx.destination);
source.start();
packages/engine/src/services/audioAutomationRender.test.ts
+99
View File
@@ -0,0 +1,99 @@
import { describe, expect, it } from "vitest";
import { parseAudioElements, volumeLaneKeyframes } from "./audioMixer.js";
import type { HfAutomationLane } from "@hyperframes/core/audio-automation";
const lanes = (points: HfAutomationLane["points"], target = "volume") => ({
lanes: [{ target, points }],
});
describe("volumeLaneKeyframes", () => {
it("keeps a straight fade at two keyframes, in composition time", () => {
const out = volumeLaneKeyframes(
lanes([
{ t: 0, v: 1 },
{ t: 2, v: 0 },
]),
5,
2,
);
expect(out).toEqual([
{ time: 5, volume: 1 },
{ time: 7, volume: 0 },
]);
});
it("holds the first value before the envelope starts", () => {
const out = volumeLaneKeyframes(
lanes([
{ t: 1, v: 0.3 },
{ t: 2, v: 1 },
]),
0,
2,
);
// Not `data-volume` at t=0 — the lane's own first value.
expect(out?.[0]).toEqual({ time: 0, volume: 0.3 });
});
it("holds the last value out to the clip end", () => {
const out = volumeLaneKeyframes(
lanes([
{ t: 0, v: 1 },
{ t: 1, v: 0.5 },
]),
0,
4,
);
expect(out?.at(-1)).toEqual({ time: 4, volume: 0.5 });
});
it("samples a bent segment, which the linear baker would straighten", () => {
const straight = volumeLaneKeyframes(
lanes([
{ t: 0, v: 0 },
{ t: 2, v: 1 },
]),
0,
2,
);
const bent = volumeLaneKeyframes(
lanes([
{ t: 0, v: 0, curve: 1 },
{ t: 2, v: 1 },
]),
0,
2,
);
expect(straight?.length).toBe(2);
expect(bent?.length ?? 0).toBeGreaterThan(30);
// Same endpoints, different path between them.
expect(bent?.[0]).toEqual({ time: 0, volume: 0 });
expect(bent?.at(-1)).toEqual({ time: 2, volume: 1 });
const mid = bent?.find((k) => Math.abs(k.time - 1) < 0.02);
expect(mid?.volume ?? 1).toBeLessThan(0.4);
});
it("has nothing to bake for a track whose only lane is an FX one", () => {
expect(volumeLaneKeyframes(lanes([{ t: 0, v: 200 }], "fx.n1.frequency"), 0, 2)).toBeNull();
});
});
describe("parseAudioElements", () => {
it("carries the automation attribute through to the mixer", () => {
const automation = '{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":1}]}]}';
const html = `<!DOCTYPE html><html><body>
<div id="root" data-composition-id="main" data-start="0" data-width="1920" data-height="1080" data-duration="10">
<audio id="bgm" src="a.wav" data-start="0" data-duration="10" data-automation='${automation}'></audio>
</div></body></html>`;
const [el] = parseAudioElements(html);
expect(el?.automation).toBe(automation);
});
it("leaves automation unset when the element has none", () => {
const html = `<!DOCTYPE html><html><body>
<div id="root" data-composition-id="main" data-start="0" data-width="1920" data-height="1080" data-duration="10">
<audio id="bgm" src="a.wav" data-start="0" data-duration="10"></audio>
</div></body></html>`;
expect(parseAudioElements(html)[0]?.automation).toBeUndefined();
});
});
packages/engine/src/services/audioFxRender.test.ts
+48
View File
@@ -67,6 +67,15 @@ function tone(path: string, seconds = 0.3, freq = 440): void {
writeWav(path, s, SR);
}
/** RMS of one slice, for comparing how loud a moment is against another. */
const sliceRms = (s: Float32Array, from: number, to: number): number => {
const a = Math.max(0, Math.floor(from * SR));
const b = Math.min(s.length, Math.floor(to * SR));
let sum = 0;
for (let i = a; i < b; i++) sum += (s[i] ?? 0) * (s[i] ?? 0);
return Math.sqrt(sum / Math.max(1, b - a));
};
const rms = (s: Float32Array): number =>
Math.sqrt(s.reduce((a, x) => a + x * x, 0) / Math.max(1, s.length));
const db = (x: number): number => 20 * Math.log10(x + 1e-30);
@@ -235,6 +244,45 @@ describe.skipIf(!HAS_BROWSER)("browser render", () => {
expect(db(rms(readWav(outPath).samples))).toBeLessThan(db(rms(readWav(input).samples)) - 3);
}, 180_000);
it("sweeps a filter across the clip when a lane automates it", async () => {
// A 2 kHz tone under a lowpass whose cutoff rises from below it to well
// above: the start should be attenuated and the end should not. This is
// the whole point of the render path — the envelope has to be *scheduled*
// offline, not merely parsed.
const input = join(dir, "sweep-in.wav");
tone(input, 1.5, 2000);
const outPath = join(dir, "sweep-out.wav");
await applyAudioFxChain(
input,
{
version: 1,
nodes: [{ type: "lowpass", id: "n1", enabled: true, params: { frequency: 300, q: 0.707 } }],
},
outPath,
{
trackId: "t",
automation: {
version: 1,
lanes: [
{
target: "fx.n1.frequency",
points: [
{ t: 0, v: 300 },
{ t: 1.5, v: 16000 },
],
},
],
},
},
);
const after = readWav(outPath).samples;
const head = db(sliceRms(after, 0.05, 0.25));
const tail = db(sliceRms(after, 1.2, 1.45));
// Opening the filter past the tone has to leave it far louder than when
// the cutoff sat an octave and a half below it.
expect(tail).toBeGreaterThan(head + 15);
}, 180_000);
it("renders a multi-effect chain including reverb", async () => {
const input = join(dir, "in.wav");
tone(input);
packages/engine/src/services/audioFxRender.ts
+23 -6
View File
@@ -18,6 +18,7 @@ import { join } from "node:path";
import { pathToFileURL } from "node:url";
import { getAudioFxRuntimeScript } from "@hyperframes/core/audio-fx-runtime";
import { enabledAudioFxNodes, type HfAudioFxChain } from "@hyperframes/core/audio-fx";
import { serializeAutomation, type HfAutomation } from "@hyperframes/core/audio-automation";
import { acquireBrowser } from "./browserManager.js";
export class AudioFxRenderError extends Error {
@@ -195,7 +196,7 @@ export async function applyAudioFxChain(
inputWav: string,
chain: HfAudioFxChain,
outputWav: string,
options: { trackId: string; signal?: AbortSignal },
options: { trackId: string; signal?: AbortSignal; automation?: HfAutomation },
): Promise<string> {
if (enabledAudioFxNodes(chain).length === 0) return inputWav;
if (!existsSync(inputWav)) {
@@ -227,7 +228,12 @@ export async function applyAudioFxChain(
await page.addScriptTag({ content: getAudioFxRuntimeScript() });
const rendered = (await page.evaluate(
async ([channelB64, rate, chainJson]: [string[], number, string]) => {
async ([channelB64, rate, chainJson, automationJson]: [
string[],
number,
string,
string,
]) => {
const decode = (b64: string): Float32Array => {
const bin = atob(b64);
const bytes = new Uint8Array(bin.length);
@@ -237,12 +243,22 @@ export async function applyAudioFxChain(
const api = (
window as unknown as {
__HF_AUDIO_FX?: {
render(p: Float32Array[], r: number, c: string): Promise<Float32Array[]>;
render(
p: Float32Array[],
r: number,
c: string,
a?: string,
): Promise<Float32Array[]>;
};
}
).__HF_AUDIO_FX;
if (!api) throw new Error("audio FX runtime failed to load");
const out = await api.render(channelB64.map(decode), rate, chainJson);
const out = await api.render(
channelB64.map(decode),
rate,
chainJson,
automationJson || undefined,
);
const encode = (plane: Float32Array): string => {
const u8 = new Uint8Array(plane.buffer, plane.byteOffset, plane.length * 4);
let s = "";
@@ -260,7 +276,8 @@ export async function applyAudioFxChain(
),
sampleRate,
JSON.stringify(chain),
] as [string[], number, string],
options.automation ? serializeAutomation(options.automation) : "",
] as [string[], number, string, string],
)) as string[];
// byteOffset and byteLength matter: Node pools small allocations, so a
@@ -289,4 +306,4 @@ export async function applyAudioFxChain(
}
}
export type { HfAudioFxChain };
export type { HfAudioFxChain, HfAutomation };
packages/engine/src/services/audioMixer.ts
+85 -4
View File
@@ -30,7 +30,16 @@ import type {
} from "./audioMixer.types.js";
import { applyVolumeEnvelopeToWav } from "./audioVolumeEnvelope.js";
import { HF_AUDIO_FX_ATTR, parseAudioFxChain } from "@hyperframes/core/audio-fx";
import {
HF_AUDIO_AUTOMATION_ATTR,
parseAutomation,
resolveAutomation,
sampleAutomationLane,
VOLUME_TARGET,
type HfAutomationLane,
} from "@hyperframes/core/audio-automation";
import { applyAudioFxChain, AudioFxRenderError } from "./audioFxRender.js";
import type { AudioVolumeKeyframe } from "./audioMixer.types.js";
export type { AudioElement, MixResult } from "./audioMixer.types.js";
@@ -331,6 +340,56 @@ function ffmpegFailure(
};
}
/** Extra samples per second inside a segment the baker cannot draw straight. */
const CURVED_SEGMENT_SAMPLES_PER_SEC = 30;
/**
* A volume lane as keyframes for the PCM baker.
*
* The baker interpolates linearly between keyframes, so a straight segment
* needs only its two ends — a simple fade stays two keyframes. A bent one is
* sampled, or the bake would quietly straighten it.
*
* Times come out in composition seconds, which is what the baker subtracts the
* track start from. Returns null when the track has no volume lane.
*/
export function volumeLaneKeyframes(
automation: { lanes: HfAutomationLane[] },
trackStart: number,
duration: number,
): AudioVolumeKeyframe[] | null {
const lane = automation.lanes.find((l) => l.target === VOLUME_TARGET);
if (!lane || lane.points.length === 0) return null;
const out: AudioVolumeKeyframe[] = [];
const push = (t: number, v: number): void => {
out.push({ time: trackStart + t, volume: v });
};
// The envelope holds its first value before the first point, rather than
// falling back to `data-volume`.
const first = lane.points[0]!;
if (first.t > 0) push(0, first.v);
for (let i = 0; i < lane.points.length; i += 1) {
const a = lane.points[i]!;
push(a.t, a.v);
const b = lane.points[i + 1];
if (!b || !a.curve) continue;
const steps = Math.max(2, Math.ceil((b.t - a.t) * CURVED_SEGMENT_SAMPLES_PER_SEC));
for (let k = 1; k < steps; k += 1) {
const t = a.t + ((b.t - a.t) * k) / steps;
push(t, sampleAutomationLane(lane, t));
}
}
// Hold the last value to the clip's end, so the baker does not ramp away
// from it toward whatever it would otherwise assume.
const last = lane.points[lane.points.length - 1]!;
if (duration > last.t) push(duration, last.v);
return out;
}
export function parseAudioElements(html: string): AudioElement[] {
const elements: AudioElement[] = [];
const { document } = parseHTML(unwrapTemplate(html));
@@ -363,12 +422,14 @@ export function parseAudioElements(html: string): AudioElement[] {
};
// <audio> and <video data-has-audio> tracks differ only in the emitted id
// and `type`; everything else (timing, layer, volume) is read identically.
const build = (el: RefResolverEl, id: string, type: AudioElement["type"]): AudioElement => {
const mediaStartAttr = el.getAttribute("data-media-start");
const layerAttr = el.getAttribute("data-layer");
const volumeAttr = el.getAttribute("data-volume");
const fxChain = el.getAttribute(HF_AUDIO_FX_ATTR);
const automation = el.getAttribute(HF_AUDIO_AUTOMATION_ATTR);
return {
id,
src: el.getAttribute("src") as string,
@@ -378,6 +439,7 @@ export function parseAudioElements(html: string): AudioElement[] {
layer: layerAttr ? parseInt(layerAttr) : 0,
volume: volumeAttr ? parseFloat(volumeAttr) : 1.0,
...(fxChain ? { fxChain } : {}),
...(automation ? { automation } : {}),
type,
};
};
@@ -875,6 +937,13 @@ export async function processCompositionAudio(
// envelope belongs on their output. A missing or broken chain is fatal
// for the whole mix rather than a per-track warning — quietly rendering
// the dry signal ships a mix that sounds plausible and is wrong.
const automation = element.automation
? resolveAutomation(
parseAutomation(element.automation),
element.fxChain ? parseAudioFxChain(element.fxChain) : undefined,
)
: null;
if (element.fxChain) {
// The chain is serialised into the attribute, the same way colour
// grading carries its config, so there is no side-car file to find,
@@ -884,7 +953,11 @@ export async function processCompositionAudio(
audioSrcPath,
chain,
join(workDir, `${element.id}-fx.wav`),
{ trackId: element.id, signal: effectiveSignal },
{
trackId: element.id,
signal: effectiveSignal,
...(automation ? { automation } : {}),
},
);
}
@@ -892,11 +965,19 @@ export async function processCompositionAudio(
// (sample-accurate, no keyframe ceiling). If the WAV isn't the expected
// 16-bit PCM, fall back to the ffmpeg expression path by leaving the
// keyframes on the track for buildVolumeExpression to handle.
//
// A volume lane supersedes keyframes probed from the timeline: the two
// would fight, and the lane is the explicit one. `lint` warns when a
// track carries both.
const laneKeyframes = automation
? volumeLaneKeyframes(automation, element.start, element.end - element.start)
: null;
const envelopeKeyframes = laneKeyframes ?? element.volumeKeyframes;
let bakedEnvelope = false;
if (element.volumeKeyframes && element.volumeKeyframes.length > 0) {
if (envelopeKeyframes && envelopeKeyframes.length > 0) {
bakedEnvelope = applyVolumeEnvelopeToWav(
audioSrcPath,
element.volumeKeyframes,
envelopeKeyframes,
element.start,
element.volume ?? 1.0,
);
@@ -910,7 +991,7 @@ export async function processCompositionAudio(
duration: element.end - element.start,
// Gain is already in the samples when baked, so mix at unity.
volume: bakedEnvelope ? 1.0 : (element.volume ?? 1.0),
volumeKeyframes: bakedEnvelope ? undefined : element.volumeKeyframes,
volumeKeyframes: bakedEnvelope ? undefined : (envelopeKeyframes ?? undefined),
});
} catch (err: unknown) {
// An FX failure is fatal for the whole mix. Every other failure mode
packages/engine/src/services/audioMixer.types.ts
+2
View File
@@ -14,6 +14,8 @@ export interface AudioElement {
volumeKeyframes?: AudioVolumeKeyframe[];
/** Serialised FX chain JSON from `data-fx-chain`, when set. */
fxChain?: string;
/** Serialised automation JSON from `data-automation`, when set. */
automation?: string;
type: "audio" | "video";
}