docs(skills): add /hyperframes-audio, and key the waveform cache by file (#3211)

* feat(studio): show every automated knob at the playhead, and carve as one module

An automated parameter has two values: the number sitting in the chain, which is
only the seed a lane replaced, and the number the envelope is on right now. The
second is the true one, so the panel shows it — on the carve rack's readouts and
on every effect's own fader and number field. A rack that showed the seed stood
still while the carve was audibly working.

Off the clip it keeps sampling rather than falling back to the stored number: a
lane holds its first value backwards and its last forwards, so before the clip
starts it already knows what it will open on, and the stored seed is a value
nothing will ever play. Showing it made the fader jump the moment the clip came
under the playhead.

The playhead comes off the liveTime channel, throttled to 30 Hz — the RAF loop
deliberately keeps frames out of the store, so a panel watching only the store
would sit still for a whole take. PropertyPanel had that subscription inline;
it is now one shared hook with two callers.

Readouts reserve the width their parameter can need rather than what its current
value takes, because an updating value one character narrower shunted everything
after it sideways 30 times a second.

The carve's effects are presented as one module: an author switched on a carve,
and the peaking filters plus the level stage are how it is built, not six things
to remove one at a time. Opening it lists every member's settings as readouts,
since strength is what sets them. No carve control is offered on a track another
track already carves against — that track is the voice, not the bed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(studio-server): key the waveform cache on the file, not just its path

Two takes written to the same path returned the first one's waveform, so a
re-recorded track drew the shape of the audio it replaced. The key now carries
size and mtime, which is enough to notice the bytes changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(engine): render audio FX in an OfflineAudioContext

Reads `data-fx-chain` off an audio element and runs the chain over the trimmed
WAV before volume automation is baked in — effects should see the raw signal,
and the envelope belongs on their output.

The processing happens in an OfflineAudioContext inside the headless browser
the engine already drives, running the same graph builders the studio previews
with. That is the point of the approach: one implementation per effect, so the
render agreeing with the preview is a property of the architecture rather than
a tolerance to police. Reimplementing each effect as an FFmpeg filter would
mean two implementations to keep in step, and for the dynamics processors and
modulated delays there is no filter that behaves the same way.

`build:audio-fx-runtime` bundles the graph builders into an injectable IIFE,
following the same pattern as the existing runtime artifacts, so the browser
runs exactly the code the studio does.

The page loads from a file:// URL rather than about:blank because AudioWorklet
is only exposed in a secure context — the compressor, limiter, gate and
bitcrush processors would otherwise fail to register with an opaque error.
file:// qualifies and needs no listening socket.

The chain is serialised into the attribute the way colour grading carries its
config, so there is no side-car file to resolve or lose.

An FX failure is fatal for the whole mix rather than a per-track soft failure.
Every other audio failure mode degrades gracefully — the track drops, siblings
continue — but substituting the dry signal for a processed one ships a render
that sounds plausible and is not what the author set up. Since the per-element
work races under Promise.all, an internal AbortController chained off the
caller's signal aborts in-flight siblings before workDir is removed.

* feat(core): voiceover carve analysis

Finds the bands a voice occupies so a music bed can be dipped there, letting
the voice sit in front without ducking the whole track.

Carve is a relationship between two tracks rather than an effect on one, so it
stays out of the FX chain. What it emits is an ordinary chain of peaking
filters, so a carve composes with whatever else is on the track and needs no
separate rendering path.

Selection is weighted toward intelligibility rather than raw voice energy.
Ranking purely by power lands on the fundamental almost every time, because
that is where a voice is loudest — but the masking that actually hurts a
voiceover happens higher up, and dipping 160 Hz mostly just thins the bed. The
bias is a control, not a constant: at 0 it follows raw energy, at 1 it weights
toward 1-3 kHz.

Ranking happens in dB, which matters more than it looks. Speech spreads 20-30 dB
across these bands — it falls off roughly 6 dB per octave above the fundamental
— so a weighting has to be on that scale to move anything at all. A
multiplicative weight of `1 - bias + bias * shaped` is bounded below by
`1 - bias`, capping its influence at 10*log10(1/(1 - bias)): 5.2 dB at the 0.7
default, 3 dB at 0.5. That is no influence against a real voice — every bias
short of ~0.95 would rank exactly like bias 0 and carve the fundamental, the
outcome the bias exists to prevent, while looking decisive against a fixture
whose bands sit 2 dB apart. So the bias is a dB penalty, zero at 2 kHz and worth
up to 30 dB at full strength, and relative cut depths come from a dB difference
rather than a ratio of weighted linear powers.

The bias reweights ranking without overriding the spectrum — a band the voice
has no energy in is not worth carving, and scores -Infinity rather than
competing — so a strongly low-pitched voice can still select low at full bias.
What the tests hold is that biasing never selects lower than the unbiased
ranking, that the DEFAULT bias reaches the presence region on a voice with a
realistic tilt, and that bias 0 still follows raw power exactly.

Includes a radix-2 FFT rather than a dependency; one Welch-style averaged
spectrum over third-octave bands does not justify pulling in a DSP library.

* fix(engine): keep the FX render 16-bit, stereo, and correctly sized

Three defects in the offline FX path, none of which any test could see.

**Float output silently disabled sample-accurate volume automation.** The writer
emitted 32-bit IEEE float; the very next mixer step bakes the volume envelope
into the samples and accepts only 16-bit PCM, returning null otherwise. So
enabling any effect downgraded that track to the ffmpeg expression path — capped
at 32 straight segments, quantising a curved envelope, and on a dense one falling
back to base volume. It now writes 16-bit PCM, clamped rather than wrapped so a
limiter at 0 dB or a resonant filter cannot turn overshoot into a click. A test
asserts the baker accepts the writer's own output and actually fades it.

**Everything was folded to mono.** `prepareAudioTrack` goes out of its way to
emit stereo — its pan filter exists to dodge ffmpeg's 3 dB mono-to-stereo
rematrix — and this folded it, then wrote one channel. So adding a single peaking
EQ collapsed a bed's width and cost ~3 dB in the render, while preview stayed
stereo. Channels now travel as one plane each, through an OfflineAudioContext of
the same width, and come back interleaved.

**Small results decoded the wrong length.** `new Float32Array(buf.buffer)`
discards byteOffset and byteLength, and Node pools small allocations: a 400-byte
payload sits at offset 8 inside an 8 KiB pool, so a clip under ~1024 samples
decoded as 2048 samples of unrelated memory — and the empty-result guard could
not see it. The reader has the mirror-image fix: a float data chunk on an odd
boundary (ffmpeg's pcm_f32le writes fmt(18) + fact, landing `data` at 58) now
copies instead of throwing RangeError on an unaligned view.

The tail limitation is now stated rather than mis-stated: the context is exactly
as long as the input, so a reverb or delay still ringing is cut there. The old
comment claimed the opposite. How far a tail may run past a clip's end changes
the clip's length in the mix, so it is a product decision, not one to make here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(producer): report an FX render failure as an audio error

`processCompositionAudio` reports per-track failures in its result, but an FX
failure it cannot degrade past — a browser that will not launch, a chain that
will not build — rejects instead. `runAudioStage` had no try, so that rejection
escaped to the orchestrator as an unclassified pipeline exception, losing the
stage/owner/retryable classification this stage exists to attach, and skipping
its abort check on the way out.

It now lands in `audioError` alongside every other cause, while an abort still
keeps its own shape rather than being reported as an audio problem.

Not done here: committing the generated `audio-fx-runtime-inline.ts` so a fresh
clone typechecks packages/engine without building first. The bundle is built from
the stub, and the stub changes three times across this stack — so the artifact
differs per branch and would conflict on every restack. Its model,
position-edits-render-inline.ts, is committed only because it is stable. Building
before testing is this monorepo's existing contract (studio's tests need core's
dist too), so the gap is not specific to audio FX and is better closed by a build
ordering gate than by committing a per-branch artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(engine): skip the browser FX render cases when there is no browser

CI's `Test` job was red on this PR with four failures, all the same cause:

  Failed to launch the browser process: spawn
  /home/runner/.cache/hyperframes/chrome/chrome-headless-shell

The job installs ffmpeg and no browser, deliberately — every other suite
that needs an external binary already guards on it
(`describe.skipIf(!HAS_FFMPEG)`). These cases were the only ones assuming
a Chrome, so they failed on an absent dependency rather than on anything
about the code.

Guards on `resolveHeadlessShellPath()` — the same resolver
`acquireBrowser` launches through, so the check cannot drift from the
thing it guards the way a hard-coded cache path would. A configured path
that does not exist throws; that is caught and read as "cannot run here".

Checked both directions rather than just the green one: with a browser all
11 cases run and pass, and with `HYPERFRAMES_BROWSER_PATH` pointed at a
missing binary exactly 3 skip and the other 8 still run. A guard that
silently skipped everything would have looked identical in CI.

They keep their value where it exists — every developer machine, and any
job that has run `hyperframes browser ensure`.

Not touched: the CodeQL failure on this PR is a run from 2026-08-07, five
days and several force-pushes stale. None of the 17 open repo alerts are
in files this PR changes; it re-runs on this push.

* chore(engine): suppress the temp-file alert with the reason it is safe

CodeQL flags `writeWav`'s `writeFileSync` as js/insecure-temporary-file
(high) — the one new alert on #3021, and the reason its CodeQL check is
red.

It is a false positive, and the comment says why rather than just silencing
it: `path` is always inside a directory made by `mkdtempSync`, never a
name assembled directly under `tmpdir()`. Both callers are covered — the
browser host page writes into `mkdtempSync(join(tmpdir(), "hf-fx-host-"))`,
and the render output goes to the producer work dir, itself
`mkdtempSync(join(tempRoot, "producer-project-"))`. mkdtemp picks the
random suffix and creates the directory 0700 in one syscall, so the
predictable filename inside it cannot be pre-created or symlinked by
another user, which is the attack the rule is about. The analyzer sees the
dataflow reach `tmpdir()` and not the mkdtemp in between.

Suppressed inline rather than dismissed in the UI, so the justification
lives next to the code and the rule stays live for anything added later in
this file. Matches the repo's existing convention — `planV2.ts:222`
carries an `lgtm[js/insecure-temporary-file]` for a different reason on
the same rule.

Correcting myself: I first reported this alert as not real, having
intersected the PR's files against the default-branch alert list, which
does not contain PR-ref alerts. Querying ?ref=refs/pull/3021/merge returns
it straight away.

* test(engine): probe ffmpeg and Chrome instead of assuming them

Two failures on #3021's Test job, both about the environment rather than
the code under test.

**Bare `ffmpeg` is not on PATH in CI.** The 16-bit fixture shelled out to
`execFileSync("ffmpeg", ...)` and died with ENOENT. The job does provide
ffmpeg, through `prepare-ffmpeg-bin`, which is what `getFfmpegBinary()`
resolves — every other ffmpeg-dependent suite in this package already goes
through it. Now this one does too, and the case is `skipIf(!HAS_FFMPEG)`
so a contributor without ffmpeg skips rather than fails.

**The browser guard trusted the wrong thing.** It asked
`resolveHeadlessShellPath()` and treated a returned path as "a browser is
here". CI's cache holds a chrome-headless-shell that resolves and then
fails to spawn — a partial download is indistinguishable from a working
one by `existsSync`, which is all that resolver checks. So the three
browser cases ran anyway and failed on the launch.

It now runs `--version` and requires exit 0, which is the same probe the
ffmpeg suites use: ask the binary, do not infer from the filesystem.

Checked both directions rather than just the green one. With a working
browser all 11 cases run and pass; with `HYPERFRAMES_BROWSER_PATH` pointed
at a binary that exits non-zero — CI's exact situation — exactly 3 skip
and the other 8 still run. A guard that quietly skipped everything would
have looked identical on the CI summary.

* feat(core): register the audio-fx-rack canary at 0%

Lands the rollout switch dark, per the registry's own procedure: "Start at
percentage: 0 and merge that — a canary at 0 is dead code you can land
safely and ramp without a code review."

Declared at the bottom of the stack so every branch above can read it. The
gate itself goes in at wa-4-fx-panel, where the rack first appears.

Scope is deliberate and stated in the description: it gates the AUTHORING
surface only. A composition that already carries `data-fx-chain` still
plays and renders it. A canary should stage who can REACH a feature, not
make an attribute somebody already wrote silently inert — an agent that
writes a chain through the skill would otherwise produce a file whose audio
processing vanishes with no error.

* feat(studio): audio FX panel generated from the registry

Controls for the whole chain: add, remove, reorder, bypass, and every knob each
effect declares.

Nothing in the panel knows what a compressor is. The registry supplies each
parameter's range, step, unit and scale and the panel renders what it finds, so
adding an effect or a knob upstream needs no change here, and the panel cannot
offer a value the renderer would reject — a typed-in figure is clamped into the
declared range on the way through.

Frequency and time controls span three or four decades, so those declare a log
scale and the slider maps exponentially; a linear slider would spend most of
its travel somewhere useless.

Reorder is a first-class control because chain order changes the sound: a
reverb before a compressor is not the same as after.

Carve gets its own block rather than an entry in the add menu, with a picker
for the voice track to listen to. It processes this track based on another one,
which is how a sidechain control works — it lives on the track that changes,
and names the source.

* feat(studio): show the Audio FX section on audio tracks

Adds `audioFx` to the editing-affordances contract and renders the FX panel in
the inspector when an `<audio>` element is selected.

The section is audio-only. A `<video>` carries its sound on a separate
`<audio>` element, so an FX chain on the video would have nothing to process.

Chain and carve settings are written straight back onto the element as
serialised attributes, the way colour grading carries its config, so
persistence is an ordinary attribute write and needs no new server route. A
chain that cannot be parsed renders as empty rather than breaking the panel,
and the attribute is left untouched until the user changes something.

The collapsed group summarises what is on the track ("2 effects + carve") so
the state is visible without expanding it.

Wired into PropertyPanelFlat rather than PropertyPanel: STUDIO_FLAT_INSPECTOR_ENABLED
defaults to true, so the flat inspector is what actually renders.

* refactor(studio): lift audioFxSummary out of PropertyPanelFlat

`PropertyPanelFlat.tsx` is 612 lines here against the repo's 600-line cap,
so the required File size check is red — the sole reason this PR is
blocked. The review says as much: "mechanical fix (~5 min), not a design
problem. Code itself is LGTM."

Moves `audioFxSummary` to `audioFxSummary.ts`, the same file a later
branch creates for it. Deliberately the smallest cut that clears the cap
rather than the whole `AudioFxGroup` extraction: every later commit in the
stack edits AudioFxGroup, so moving it here would collide with each of
them, while almost nothing touches this function.

595 lines.

* feat(core,studio): hear the FX chain in preview, and run the carve analysis

Splices an element's FX chain into the playback graph so preview stops being
silent about effects, and wires the carve button that was previously inert.

The chain goes between the decoded source and its gain stage: effects see the
raw signal and volume automation rides on their output, matching the order the
offline render uses. Since preview and render call the same graph builders,
what is heard while scrubbing is what gets written.

The splice lives in the transport rather than on the `<audio>` element. The
transport plays each track from a decoded AudioBuffer and mutes the element to
avoid doubling, so capturing the element with createMediaElementSource would
have processed a stream nothing is listening to — it looked like it worked
because the call succeeded, and the audio was unchanged.

A chain that cannot be built plays dry rather than silencing the track, which
is the right failure in preview: the author keeps working and hears the source.
The render still refuses, because shipping the dry signal there would be wrong.

Carve now analyses for real: it decodes the chosen voice track, ranks its bands
and writes the resulting peaking filters onto this track. Generated nodes are
tagged `fromCarve`, so re-running replaces the previous carve instead of
stacking another set on top of hand-added effects.

Known limitation: the graph is built when a source is scheduled, so a knob
turned mid-playback takes effect on the next play or seek rather than
immediately. Live re-parameterisation needs the transport to hold the handle
and forward updates.

* fix(studio,core): stop parameter drags from restarting playback

Dragging a knob wrote the chain through the persisting attribute path on every
input event. That path refreshes the preview, which reloads the composition and
reschedules audio — so a single drag reloaded dozens of times and playback
stuttered the whole way.

Drags now go through `onSetAttributeLive`, the same path colour grading uses for
scrubs: it coalesces undo entries and sets `skipRefresh`, so no reload happens.
The persisting write fires once, when the gesture ends — pointer-up or blur for
a slider, Enter or blur for a typed value. A select commits immediately since
there is no drag to wait for.

While dragging, the control is driven from local state. Waiting for the value to
round-trip through the element attribute made the knob lag behind the pointer.

For the change to be audible without a reload, the graph now follows the
attribute: the chain installed by the transport observes the element and
re-parameterises itself in place, so a value change lands on the next
128-sample quantum. A shape change (effect added, bypassed, pole count) cannot
be patched into a running graph, so it still waits for the next schedule rather
than cutting the audio mid-play.

The regression test drags a slider through several values and asserts the
persisting handler is untouched until release.

* feat(studio): put the audio FX rack behind its canary

Gates the rack on `isCanaryEnabled("audio-fx-rack")`, which is registered
at 0% — so the whole 47-PR stack can land without showing anyone a feature
that has not been measured yet.

The gate sits on the AUTHORING surface and nowhere else. The runtime and
the render still honour a `data-fx-chain` already on an element, so a
composition written through the skill or by `carve.mjs` keeps its
processing rather than going silently dry for anyone outside the cohort. A
canary should stage who can REACH a feature, not make an attribute somebody
already wrote stop working with no error.

Gated at the panel rather than in `resolveEditingSections`: the affordance
resolver is a pure function in core describing what an element CAN support,
and rollout state is not a property of an `<audio>` tag.

Pinned the 0% with a test, and checked it fails at 25 — a ramp should have
to break something that says "this ships dark" out loud.

One gap, stated rather than papered over: the gate itself has no unit test.
I wrote one and deleted it, because `PropertyPanel.test.tsx`'s harness
never renders the Audio FX group for its audio fixture even with the gate
removed — so the test passed for the wrong reason in the off case and could
not pass at all in the on case. A test that cannot fail for the right
reason is worse than none. Verifying the gate needs the panel harness to
mount that section first, which is its own change.

* fix(core): register FX worklets before building nodes that need them

An AudioWorkletNode cannot be constructed before its processor is registered —
it throws, and the surrounding chain is lost with it. `attachElementFxChain`
built the chain first and only then called `ensureAudioFxWorklets`, so every
worklet-backed effect (compressor, limiter, gate, bitcrush) threw on
construction and the track fell back to dry. Instrumenting the preview showed
`hf-compressor: InvalidStateError` with addModule never called at all.

When the module has not landed yet the track now plays dry and the graph is
swapped in once registration resolves, so the effect arrives a moment late
instead of never.

Registration is also tracked per context rather than in one module-level
promise. A processor registered on one AudioContext does not exist on another,
so the shared promise made every context after the first believe it was ready
when it was not — the studio's transport owns its own context, which is exactly
that case.

With the worklets actually running, the compressor's per-sample log10 and pow
became real audio-thread work. Samples below the knee have a gain of exactly
unity and need neither, so the envelope is now compared in the linear domain
and the transcendentals only run for samples that are actually being
compressed.

* refactor(studio): split the FX node row out of FxSection

Clears the health findings the FX stack left behind: the chain-node render
callback was a 70-line closure over half of FxSection's state, and the two
reorder arrows were the same button written twice.

Also drops two exports with no consumers, and registers the audio FX runtime
stub as an entry point — it is bundled by file path, so nothing imports it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(core): automation envelope model for audio tracks

Adds the data model behind Ableton-style automation lanes: breakpoint
envelopes over track volume or one knob of one effect in the track's FX
chain, stored on the element as `data-automation`.

Times are clip-local, so an envelope travels with the clip when it moves —
the clip-envelope model rather than arrangement automation.

`sampleAutomationLane` is the single interpolator. The lane drawing, the
preview scheduler and the render bake all call it, so the picture and the
sound cannot disagree about the curve. Log-scaled parameters interpolate in
log space, matching what their own knob already promises.

FX nodes gain a stable `id`, minted by count rather than randomly so the
document is the same on every machine. Lanes address nodes by id, so
reordering a chain never re-points a lane at a different effect, and a lane
whose effect was deleted is dropped rather than left to reattach.

Also warns when a track carries both a volume lane and a GSAP volume tween,
since only the lane is heard and the tween silently does nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(studio): lift the audio FX group out of PropertyPanelFlat

`PropertyPanelFlat.tsx` was 672 lines against the repo's 600-line cap, so
the required File size check was red — the sole reason #3014 and #3022 are
blocked. Both reviews say the same thing: "mechanical fix, not a design
problem. Code itself is LGTM."

Moves `AudioFxGroup` and `audioFxSummary` into
`propertyPanelAudioFxGroup.tsx`, which is where a later branch puts them
anyway — done here so the file is under the cap from the point it first
crosses it, rather than ten branches later.

533 lines now. The four audio imports it no longer needs go with it.

Not fixed here: three `FxSection carve` tests fail on this branch with
"Cannot read properties of undefined (reading 'toFixed')". Confirmed
pre-existing by stashing this change and re-running — that is the separate
`Test` failure the review also flags.

* feat(core): expose the AudioParams behind automatable FX knobs

Marks the knobs an automation lane can drive and has each graph builder hand
back the AudioParam behind them, so a scheduler can write to a running effect
without knowing what the effect is.

A knob is not always one AudioParam. A wet/dry mix is two gains moving in
opposition, and a knob in milliseconds drives a delay time in seconds, so
each target carries the mapping out of the knob's own declared unit.

What stays unautomatable is stated where it is decided: a WaveShaper curve, a
convolution impulse and a one-pole filter's coefficients are all rebuilt
wholesale rather than scheduled, and the four worklet effects take values by
postMessage rather than through AudioParams.

The registry flag is written by hand, so a test builds every effect and
checks the exposure both ways — nothing flagged is missing, nothing exposed
is unflagged. A flag that lied would offer a lane that silently did nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(core): play automation envelopes in preview

Schedules each lane onto the AudioParams behind its knob using native ramps
and value curves. Nothing evaluates the envelope per frame: it is handed to
the audio thread once, so it stays sample-accurate however busy the main
thread is, and the offline render will schedule it the same way.

Timing comes from the transport, so an envelope survives seeking into the
middle of a clip, a clip that has not started yet, and a playback rate that
compresses clip seconds into context seconds.

A straight line is only scheduled as a ramp when nothing bends it — no
curvature, a linear parameter scale, and no unit mapping. Log-scaled
parameters and mapped ones are sampled instead, since a delay knob in
milliseconds and a wet/dry pair moving in opposition are not linear in the
parameter they drive.

Lanes with nowhere to write are skipped rather than reported: a one-pole
filter exposes no frequency param, and the worklet effects expose none at
all. Editing an envelope mid-playback re-aims it at the live playhead rather
than restarting the track.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(core): make the volume lane audible in preview

The envelope was scheduled onto the transport's gain AudioParam, but the
runtime rewrites that gain every tick from `data-volume` and the GSAP-seeked
value — so it was erased within a frame. Volume automation was correct in the
render and inaudible while previewing.

The lane now feeds the per-tick path where the probed volume keyframes already
sit, checked ahead of them so the two cannot fight, and the transport no
longer schedules volume at all: one mechanism instead of two racing.

The cost is honest — in preview the level steps per tick rather than per
sample, exactly as the existing keyframe path does. The render still bakes it
into the PCM sample-accurately, and FX parameters are still scheduled on their
own AudioParams, since nothing rewrites those.

Parsed lanes are cached by attribute text: the runtime asks once per tick per
track, and parsing there would run the JSON parser 60 times a second for a
value that only changes on an edit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(engine): bake automation envelopes into the render

The offline render schedules FX lanes with the same scheduler preview uses,
inside the OfflineAudioContext that already runs the same graph builders. The
input WAV is the clip's own audio from its first sample, so clip-local time
is offline time and the envelope needs no offset.

Volume lanes take the existing PCM bake rather than a second mechanism: the
lane is converted to keyframes, so a straight fade stays two of them and only
a bent segment is sampled — the baker interpolates linearly and would
otherwise quietly straighten the curve. A volume lane supersedes keyframes
probed from the timeline, which `lint` already warns about.

A browser test sweeps a lowpass from below a 2 kHz tone to well above it and
measures both ends. Parsing the envelope is not the same as scheduling it,
and only running the real thing tells the two apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(core): apply chain edits to the running graph

A structural edit — an effect added, removed, bypassed, or a filter's pole
count switched — was dropped. `buildFxChain`'s update reports false when the
change is not merely new values, and the attribute observer ignored that, so
the edit only took hold when the persisting write reloaded the composition.
That reload restarted every playing track, which is what was heard as the
audio chopping.

The graph is now swapped in place: the old effects are detached, the new ones
built and connected between the same source and gain, and any lanes
re-scheduled onto the new nodes. The source node is never touched, so playback
does not restart.

A track with no chain is watched too, rather than wired through and forgotten,
so adding its first effect is heard the same way. That means the function
always returns a disposer instead of null for the empty case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(studio): drop the FX panel's dead __testables export

Fallow audit flagged it — no test imports the module.

* fix(core,studio): clear the remaining Fallow audit findings on the FX panel

- Split FxSection's per-node row into FxNodeRow + FxNodeControls so the
  CRAP score (31.6, threshold 30) splits across two smaller units instead
  of moving wholesale with one extraction.
- Dedupe the repeated "open the add menu, read its items" block in
  propertyPanelFxSection.test.tsx into openAddMenuItems().
- Merge build-audio-fx-runtime.ts and build-position-edits-render.ts into
  one build-inline-artifact.ts, config-selected by CLI arg — the two
  scripts were a byte-for-byte clone save for names.
- Exempt canary.test.ts's rawFnv (a deliberate independent
  reimplementation used to cross-check canaryBucket, per its own
  docstring) and the property-panel test files' shared renderInto/mount
  scaffolding (pre-existing across 9 files, 2 outside this stack) in
  .fallowrc.jsonc, consistent with this file's existing exemptions for
  the same class of intentional/pre-existing duplication.

* fix(ci): allowlist the build-script consolidation in the no-main-deletions guard

build-audio-fx-runtime.ts and build-position-edits-render.ts were merged into
build-inline-artifact.ts to kill a fallow duplication finding; the deletion
guard flagged that as an accidental loss since main still has both originals.

* fix(core): dedupe the wet/dry mix math between delayFeedback and chorusLfo

Both effect builders set wet.gain to the mix and dry.gain to its complement
in identical two-line blocks; fallow kept re-flagging it as a 10-line clone
on every unrelated change. Extracted setWetDryMix.

* fix(core): remove the build-audio-fx-runtime.ts stray resurrected by a main merge

An earlier merge with main brought this deleted file back (git's merge/delete
handling on an unchanged-on-one-side file); package.json already points at
build-inline-artifact.ts, so it sat unreachable and duplicating that file's
config, both of which fallow flagged.

* fix(studio): pull TimelineLanes under the 600-line cap

TimelineLanes.tsx hit 620 lines. Extracted the three per-clip pointer
gestures (resize-start, pointer-down move-arm, click/razor-split) into
createClipGestureHandlers — one factory call per rendered clip instead of
~120 lines of inline handler bodies in the render loop. 529 lines now.

* fix(studio): split the extracted pointerdown handler under the CRAP threshold

Moving the ~120-line gesture logic into timelineClipGestureHandlers.ts
concentrated it into two functions fallow flagged (onPointerDown at CRAP
63.6, onResizeStart at 31.6). Split the decision logic (which gesture a
pointerdown implies) into a pure resolvePointerDownAction, then split
its own intent-blocking check into isIntentBlocked. onResizeStart's guard
moved into canStartResize. Every function now scores under 30.

* fix(studio): drop the unused DomEditSelection import in PropertyPanelFlat

CI caught it on PR #3026 (wa-12-panel-params); a later refactor in the
stack removed the last use of the type here without removing the import.

* fix(studio): close the typecheck and fallow gaps wa-18b-reschedule opened

useAutomationLanes.ts's write() assumed gesture-scoped coalescing and a
preview-only commit that useDomEditAttributeCommits.ts never grew — backported
that option support from its own later commit so the two sides of the API
agree. The paste path and its tests were missing the box selection's v0/v1
bounds a sibling commit added to AutomationSelection. The FX panel's carve
controls still edited the six mechanism numbers (maxCutDb, bands,
intelligibilityBias) after carveProfile() collapsed authoring to one Strength
knob, so those fields no longer existed on HfCarveSettings; UI now edits
strength, and analyseCarveBands is called with carveProfile(strength).

Also closes fallow's complexity, dead-code and duplication findings on this
PR's diff: extracted automationLaneDragMath.ts (pure group/point-move math)
and useAutomationRangeDrag.ts (the marquee-select gesture) out of
useAutomationLaneGestures.ts, pulled a couple of render-loop ternaries and a
resolver into named functions, dropped an export nothing outside its file
used, and shared a step-simplifier between audioCarve's two envelope
builders.

The edge-stretch vs. box-select priority test in TimelineAutomationLane.test
was still pinning the pre-box-select rule (edge wins over a point sitting on
it) that a sibling commit deliberately reversed — a point inside the box is
now selected content, so grabbing it drags the group instead. Updated the
test to the shipped rule instead of the old one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(core): cap the via conic's weight so an edge-clamped via point can't NaN

A via point pulled out past the segment (viaX: 5, viaY: -3) clamps to
(0.999, 0.001) — exactly on the steady region's edge, where edge - viaX is 0.
viaConic divided by that zero to get an infinite weight, and shapeVia turned
Infinity into NaN a few steps later (Infinity - Infinity in the quadratic
coefficient). NaN reaching setValueCurveAtTime silences the automated
parameter for the rest of the render.

Capped the weight at 1e6 instead of leaving it unbounded — past that point
the arc already reads as touching the via point, so nothing visible is lost.
Also hardened shapeVia's existing denominator guard (`<= 0`) to `!(> 0)`,
since NaN fails the original comparison and fell through it.

Review by Miga (PR #3208).

* fix(studio-server): fingerprint the proactive waveform cache key too

The route already keys the waveform cache on the asset's size and mtime as
well as its path, so a rebuilt-in-place file gets fresh peaks instead of
stale ones. generateWaveformCache — the proactive path that runs on upload —
still called buildWaveformCacheKey with the path alone, so it wrote to a
different key than the route reads from (making the pre-generated cache
never found) and kept the exact collision bug this fingerprint exists to fix
on its own path.

Review by Miga (PR #3211).

* style(docs): run oxfmt on the /hyperframes-audio skill docs

Table column widths had drifted out of alignment with oxfmt's own rules,
failing format:check and blocking the Preflight gate every downstream
branch inherits. Whitespace only, no content change.

* fix(studio): widen PropertyPanel's resetModules render timeout again

The 20s margin (already once widened for the same reason) is timing out in
CI's full-monorepo Test run — the resetModules()+fresh-import render this
test needs is uncached and competes with every other package's test suite
for the same worker pool, and the same test passes in well under 2s
standalone. Went to 45s rather than re-tuning to whatever number happens to
clear the current CI load, since that number moves every time CI gains a
package.

* feat(core): carve against every voice over a bed, always (#3212)

* feat(core): carve against every voice over a bed, always dynamically

A bed usually runs under a whole sequence — a narrator, an interview answer, a
second presenter — and carving against one of them left the others fighting it.
`source` becomes `sources`, and `mixCarveSources` sums every voice onto the BED's
clock before anything is measured. That is what keeps one analysis sufficient: the
chain is fixed, so there is no per-voice filter to switch between, and bands drawn
from all the speech there is with envelopes that rise wherever any of it happens
answer the actual question — where and when is speech masking this bed.

Summed rather than averaged: two people talking at once mask more than either
alone. Audio before the bed starts is dropped rather than folded in at zero, since
it plays over nothing and shifting it would put a cut where there is no voice.

`dynamic` is gone. A fixed depth thins the bed through every pause, and once both
have been heard there is no reason to want it, so every carve follows the speech.

Two helpers the panel and the headless script now share instead of each carrying a
copy — two definitions of "what does this name suggest" drift, and then the two
disagree about which track is the voice:

- `classifyAudioName` reads a track's kind from its id and filename together.
  `unknown` is deliberately common: treating an unrecognised name as "not a voice"
  would hide the one track somebody needs to pick.
- `clipsOverlap` keeps out a voice that never plays while the bed does. An unwritten
  duration counts as unbounded, not zero — refusing a clip whose length the
  composition leaves to the media would drop the commonest case there is.

Files written before this still load: a single `source` reads as a one-voice list,
a stored `dynamic` is ignored, and an absent attribute means the defaults whole.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(core): stop \b from missing underscore-separated names, guard clipsOverlap's negative duration

\b treats `_` as a word character, so \bbed\b never matched bed_01,
music_bed_loop, or theme_song, and \bvo\b/\bvox\b/\btts\b had the same gap —
an underscore-separated bed classified as "unknown" and could end up offered
as its own carve source. Replaced the short hints with a boundary that
actually excludes letters and digits on both sides.

clipsOverlap computed end = start + duration without guarding sign, so a
negative duration put end before start — an interval that does not describe
anything, and one specific case showed it silently dropping a real overlap
(a shorter, earlier broken end rejected a clip that genuinely contained the
point). Duration clamps to zero instead: a clip cannot un-play time, and a
zero-length clip at its start is the sane reading of "duration nobody wrote
down as positive."

Review by Miga (PR #3212).

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(studio): port the carve UI off the removed source/dynamic fields

#3212 (accidentally squash-merged into this branch instead of main) changed
HfCarveSettings from a single `source` + `dynamic` toggle to a `sources`
list with dynamic mode removed outright — the multi-voice UI consumer that
goes with that shape lands in the very next PR, so this branch was left with
a type that no longer matched its own code.

Minimal port, not the multi-voice redesign that PR does properly: the
"Listen to" picker and analyse() treat sources[0] as the one voice this UI
still understands, and every dynamic-mode branch (the automated envelope
lanes, the toggle, the checkbox) is gone along with the field — a carve is
now always the static value the analysis computes, matching what the type
change made permanent. Test suite trimmed the same way: the automation-lane
and toggle tests covered behavior that no longer exists.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-08-13 02:16:36 -07:00
committed by GitHub
co-authored by Claude Opus 5
parent 7d38fed97f
commit 56d8df65ca
21 changed files with 1337 additions and 294 deletions
+1
View File
@@ -15,6 +15,7 @@
"skills": [
"./skills/hyperframes",
"./skills/hyperframes-animation",
"./skills/hyperframes-audio",
"./skills/hyperframes-cli",
"./skills/hyperframes-core",
"./skills/hyperframes-creative",
+5 -4
View File
@@ -4,12 +4,12 @@ Open-source video rendering framework: write HTML, render video.
## Skills
This repo ships 19 AI agent skills via [vercel-labs/skills](https://github.com/vercel-labs/skills). Install them before writing compositions — they encode framework-specific patterns that generic docs don't cover. **Default to the core set**: the `/hyperframes` router installs each creation workflow on demand; install all 19 only when the user explicitly asks for the full set.
This repo ships 20 AI agent skills via [vercel-labs/skills](https://github.com/vercel-labs/skills). Install them before writing compositions — they encode framework-specific patterns that generic docs don't cover. **Default to the core set**: the `/hyperframes` router installs each creation workflow on demand; install all 20 only when the user explicitly asks for the full set.
```bash
npx hyperframes skills update # default: installs/refreshes the core set — workflows install on demand
npx skills add heygen-com/hyperframes --full-depth # interactive picker (terminal only — non-interactive without --skill installs all 19)
npx skills add heygen-com/hyperframes --all --full-depth # all 19 at once — only on explicit request
npx skills add heygen-com/hyperframes --full-depth # interactive picker (terminal only — non-interactive without --skill installs all 20)
npx skills add heygen-com/hyperframes --all --full-depth # all 20 at once — only on explicit request
npx skills add heygen-com/hyperframes --skill <name> --full-depth # just one (bare name, no leading slash)
```
@@ -39,6 +39,7 @@ Atomic capabilities the creation workflows compose against — pull one when you
- `/hyperframes-keyframes` — seek-safe keyframe authoring across runtimes: GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, text trails, 3D depth; plus `hyperframes keyframes` diagnostics for surfacing and verifying rendered motion.
- `/hyperframes-creative` — non-animation creative direction: `frame.md` / `design.md` handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns.
- `/media-use` — the media OS: resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record; generate via TTS / music / image models when the catalog misses; transcribe, caption, remove backgrounds, and reuse assets across projects. One shared `scripts/audio.mjs` engine + manifest tracking; keeps search noise on disk.
- `/hyperframes-audio` — mix the audio already placed in a composition: voiceover carve (dip a music bed only in the bands the voice occupies, static or dynamic, level match included), the effect chain (EQ, compressor, limiter, gate, saturation, delay, reverb, chorus, phaser, bitcrush), and automation envelopes on volume or any effect parameter. Sourcing the audio is `/media-use`; this is what happens to it afterwards.
- `/hyperframes-cli` — CLI dev loop: `init`, `add`, `lint`, `check`, `snapshot`, `preview`, `render`, `publish`, `doctor`, `lambda` (AWS Lambda cloud rendering).
- `/hyperframes-registry` — install and wire registry blocks and components into compositions via `hyperframes add`. Covers authoring a new block or component to contribute upstream.
- `/figma` — import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition.
@@ -51,7 +52,7 @@ When adding a new skill, or substantially renaming / repurposing an existing one
2. The scaffolded project template `packages/cli/src/templates/_shared/CLAUDE.md` + `AGENTS.md` — written into every `hyperframes init` project, so a stale entry there ships to users. The two template files must stay byte-identical.
3. If the skill changes the routing surface for "make a video" requests, also update the routing table + intent layer in `skills/hyperframes/SKILL.md` AND that workflow's own route file, `skills/hyperframes/references/routes/<workflow>.md`. One file carries both halves: the input/output/trigger contract the router reads before the workflow is installed, and its interview entry (must-haves, conditionals, deferred asks, run-shape). The older `references/workflow-catalog.md` and `references/route-briefs.md` are now "moved" stubs pointing at `routes/` — don't edit them.
4. Mirror the Router / Creation workflows / Domain skills grouping across all surfaces so a skill always lives in the same column.
5. Skill count appears in the README and CLAUDE.md intro lines ("19 AI agent skills…") — update on add/remove. The `docs/guides/skills.mdx` page and the CLI templates deliberately omit a count to avoid drift; keep them count-free.
5. Skill count appears in the README and CLAUDE.md intro lines ("20 AI agent skills…") — update on add/remove. The `docs/guides/skills.mdx` page and the CLI templates deliberately omit a count to avoid drift; keep them count-free.
The skill's own `SKILL.md` frontmatter `description:` is the source of truth for the one-line "use when" blurb; copy from there into the catalog rather than paraphrasing.
+14 -13
View File
@@ -41,7 +41,7 @@ Install the HyperFrames skills, then describe the video you want:
npx skills add heygen-com/hyperframes --full-depth
```
> The picker opens with nothing pre-selected — the **Core Skills** group is all you need: the `/hyperframes` router installs each creation workflow on demand. Agents and non-interactive runs should use `npx hyperframes skills update` instead — it installs exactly the core set, whereas a non-interactive `skills add` without `--skill` installs all 19.
> The picker opens with nothing pre-selected — the **Core Skills** group is all you need: the `/hyperframes` router installs each creation workflow on demand. Agents and non-interactive runs should use `npx hyperframes skills update` instead — it installs exactly the core set, whereas a non-interactive `skills add` without `--skill` installs all 20.
>
> `--full-depth` does a full clone of the repo's current `main`. Without it, `skills add` fetches the skills.sh registry blob, which lags `main` by hours — you'd get an older copy of a skill. (`hyperframes skills update` already installs full-depth.)
@@ -53,9 +53,9 @@ The skills teach agents the HyperFrames production loop: plan the video, write v
## Skills
HyperFrames ships 19 skills agents load on demand. Read `/hyperframes` first — it's the router and capability map; it picks a workflow for any "make me a…" request — video, deck, or composition port — and points to the domain skills below.
HyperFrames ships 20 skills agents load on demand. Read `/hyperframes` first — it's the router and capability map; it picks a workflow for any "make me a…" request — video, deck, or composition port — and points to the domain skills below.
Default to the **core set** — the router installs each creation workflow on demand. `npx hyperframes skills update` installs exactly that from anywhere; the interactive picker (`npx skills add heygen-com/hyperframes --full-depth`) lists it as the "Core Skills" group, nothing pre-selected. The picker is interactive-only — a non-interactive or agent run without `--skill` installs all 19. Use `npx skills add heygen-com/hyperframes --all --full-depth` to install all 19 deliberately (skips the picker), or `npx skills add heygen-com/hyperframes --skill <name> --full-depth` for just one (bare name, no leading `/`). Keep `--full-depth` — it installs the current `main`; without it `skills add` fetches the skills.sh blob, which lags by hours.
Default to the **core set** — the router installs each creation workflow on demand. `npx hyperframes skills update` installs exactly that from anywhere; the interactive picker (`npx skills add heygen-com/hyperframes --full-depth`) lists it as the "Core Skills" group, nothing pre-selected. The picker is interactive-only — a non-interactive or agent run without `--skill` installs all 20. Use `npx skills add heygen-com/hyperframes --all --full-depth` to install all 20 deliberately (skips the picker), or `npx skills add heygen-com/hyperframes --skill <name> --full-depth` for just one (bare name, no leading `/`). Keep `--full-depth` — it installs the current `main`; without it `skills add` fetches the skills.sh blob, which lags by hours.
Installs stay lean after that: `npx hyperframes init` keeps the **core set** fresh (the router, the `hyperframes-*` domain skills, and `media-use` — plus whatever is already installed; `/figma` stays on demand) and never expands a partial install; the creation workflows install **on demand** — the router runs `npx hyperframes skills update <workflow>` before entering one. Nothing re-pulls the full set behind your back.
@@ -94,16 +94,17 @@ This writes `dist/hyperframes-plugin.zip` with a `hyperframes/` root folder and
Atomic capabilities the creation workflows compose against — pull one when you need that specific layer.
| Skill | Covers |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `/hyperframes-core` | The composition contract — `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, determinism rules. |
| `/hyperframes-animation` | All animation knowledge — atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). |
| `/hyperframes-keyframes` | Seek-safe keyframe authoring across runtimes — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth — plus `hyperframes keyframes` diagnostics for rendered motion. |
| `/hyperframes-creative` | Non-animation creative direction — `frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. |
| `/media-use` | The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. |
| `/hyperframes-cli` | CLI dev loop — `init`, `lint`, `check`, `snapshot`, `preview`, `render`, `publish`, `doctor`, plus HeyGen-hosted cloud rendering (`cloud render`) and AWS Lambda rendering (`lambda deploy / render / progress`). |
| `/hyperframes-registry` | Install and wire registry blocks and components into compositions via `hyperframes add`. Authoring a new block or component to contribute upstream. |
| `/figma` | Import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition. |
| Skill | Covers |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/hyperframes-core` | The composition contract — `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, determinism rules. |
| `/hyperframes-animation` | All animation knowledge — atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). |
| `/hyperframes-keyframes` | Seek-safe keyframe authoring across runtimes — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth — plus `hyperframes keyframes` diagnostics for rendered motion. |
| `/hyperframes-creative` | Non-animation creative direction — `frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. |
| `/media-use` | The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. |
| `/hyperframes-cli` | CLI dev loop — `init`, `lint`, `check`, `snapshot`, `preview`, `render`, `publish`, `doctor`, plus HeyGen-hosted cloud rendering (`cloud render`) and AWS Lambda rendering (`lambda deploy / render / progress`). |
| `/hyperframes-audio` | Mix the audio already placed in a composition — voiceover carve (dip a music bed only in the bands the voice occupies, static or dynamic, level match included), the effect chain (EQ, compressor, limiter, gate, saturation, delay, reverb, chorus, phaser, bitcrush), and automation envelopes on volume or any effect parameter. Sourcing the audio is `/media-use`. |
| `/hyperframes-registry` | Install and wire registry blocks and components into compositions via `hyperframes add`. Authoring a new block or component to contribute upstream. |
| `/figma` | Import Figma assets, tokens, components, and storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI) plus Motion animations (MCP) and shaders (MCP source / native export) into a composition. |
For visual design handoff workflows, see the [Claude Design guide](https://hyperframes.heygen.com/guides/claude-design) and [Open Design guide](https://hyperframes.heygen.com/guides/open-design).
+1 -1
View File
@@ -18,7 +18,7 @@
**Porting an existing composition?** `/remotion-to-hyperframes` translates a Remotion (React) composition into HyperFrames HTML — a source migration, separate from the creation workflows above.
The domain skills (`/hyperframes-core`, `/hyperframes-animation`, `/hyperframes-keyframes`, `/hyperframes-creative`, `/hyperframes-cli`, `/media-use`, `/hyperframes-registry`, `/figma`) and the full capability map live inside `/hyperframes` — it is the single source of truth for which skill handles which intent.
The domain skills (`/hyperframes-core`, `/hyperframes-animation`, `/hyperframes-keyframes`, `/hyperframes-creative`, `/hyperframes-cli`, `/media-use`, `/hyperframes-audio`, `/hyperframes-registry`, `/figma`) and the full capability map live inside `/hyperframes` — it is the single source of truth for which skill handles which intent.
**Changing how real footage or images look or reveal?** Load `/media-use` and read its `references/media-treatments.md` before editing, even when the request only says dark, flat, boring, retro, private, or “make the reveal cooler.” It governs how footage is treated, never whether media may be used. Use canonical media treatments and seek-safe motion; do not improvise equivalent CSS/SVG filters or overlays.
+1 -1
View File
@@ -18,7 +18,7 @@
**Porting an existing composition?** `/remotion-to-hyperframes` translates a Remotion (React) composition into HyperFrames HTML — a source migration, separate from the creation workflows above.
The domain skills (`/hyperframes-core`, `/hyperframes-animation`, `/hyperframes-keyframes`, `/hyperframes-creative`, `/hyperframes-cli`, `/media-use`, `/hyperframes-registry`, `/figma`) and the full capability map live inside `/hyperframes` — it is the single source of truth for which skill handles which intent.
The domain skills (`/hyperframes-core`, `/hyperframes-animation`, `/hyperframes-keyframes`, `/hyperframes-creative`, `/hyperframes-cli`, `/media-use`, `/hyperframes-audio`, `/hyperframes-registry`, `/figma`) and the full capability map live inside `/hyperframes` — it is the single source of truth for which skill handles which intent.
**Changing how real footage or images look or reveal?** Load `/media-use` and read its `references/media-treatments.md` before editing, even when the request only says dark, flat, boring, retro, private, or “make the reveal cooler.” It governs how footage is treated, never whether media may be used. Use canonical media treatments and seek-safe motion; do not improvise equivalent CSS/SVG filters or overlays.
+1
View File
@@ -150,6 +150,7 @@ export function isCoreSkill(name: string): boolean {
export const FALLBACK_CORE_SKILLS: readonly string[] = [
"hyperframes",
"hyperframes-animation",
"hyperframes-audio",
"hyperframes-cli",
"hyperframes-core",
"hyperframes-creative",
+153
View File
@@ -6,6 +6,10 @@ import {
analyseCarveDynamics,
carveBandsToChain,
carveProfile,
classifyAudioName,
clipsOverlap,
mixCarveSources,
couldBeCarveSource,
DEFAULT_CARVE,
normalizeCarveSettings,
} from "./audioCarve.js";
@@ -458,3 +462,152 @@ describe("analyseCarveDynamics", () => {
expect(analyseCarveDynamics(new Float32Array(0), SR, [BAND])).toEqual([]);
});
});
describe("classifyAudioName", () => {
it("reads a track's kind from its id and its filename together", () => {
// Either can be the informative one: elements named a1/a2 may still have
// narration.mp3 and bgm.mp3 behind them.
expect(classifyAudioName("narration")).toBe("voice");
expect(classifyAudioName("a1", "voiceover-take3.wav")).toBe("voice");
expect(classifyAudioName("music-bed")).toBe("music");
expect(classifyAudioName("a2", "bgm_loop.m4a")).toBe("music");
expect(classifyAudioName("sfx-explosion")).toBe("sfx");
expect(classifyAudioName("whoosh-01")).toBe("sfx");
});
it("says nothing about a name that says nothing", () => {
// The common case, and the reason nothing downstream may treat "unknown" as
// "not a voice": it would hide the one track somebody needs to pick.
expect(classifyAudioName("a1")).toBe("unknown");
expect(classifyAudioName("clip-2", "0f9c1a.mp3")).toBe("unknown");
expect(classifyAudioName(undefined, null)).toBe("unknown");
});
it("prefers voice when a name carries both hints", () => {
// A file called voiceover-over-music-bed.wav is the voiceover, and a track
// matching both is better offered than hidden.
expect(classifyAudioName("voiceover-over-music-bed.wav")).toBe("voice");
});
it("offers speech and unnamed tracks as carve sources, never music or effects", () => {
expect(couldBeCarveSource("recap-audio")).toBe(true);
expect(couldBeCarveSource("a1")).toBe(true);
expect(couldBeCarveSource("music-bed")).toBe(false);
expect(couldBeCarveSource("sfx-explosion")).toBe(false);
});
it("treats underscores as separators, not word characters, for short hints", () => {
// `\b` treats `_` as a word character, so `\bbed\b` used to miss `bed_01` —
// an underscore-separated bed classified as "unknown" and could end up
// offered as its own carve source.
expect(classifyAudioName("bed_01")).toBe("music");
expect(classifyAudioName("my_bed")).toBe("music");
expect(classifyAudioName("music_bed_loop")).toBe("music");
expect(classifyAudioName("theme_song")).toBe("music");
expect(classifyAudioName("vo_take3")).toBe("voice");
expect(classifyAudioName("main_vox")).toBe("voice");
expect(couldBeCarveSource("bed_01")).toBe(false);
});
});
describe("clipsOverlap", () => {
it("overlaps when the spans genuinely share time", () => {
expect(clipsOverlap({ start: 0, duration: 5 }, { start: 3, duration: 5 })).toBe(true);
});
it("does not overlap when one span ends before the other starts", () => {
expect(clipsOverlap({ start: 0, duration: 5 }, { start: 5, duration: 5 })).toBe(false);
expect(clipsOverlap({ start: 10, duration: 5 }, { start: 0, duration: 5 })).toBe(false);
});
it("does not overlap two clips that only touch at an edge", () => {
// Half-open: an end exactly at the other's start shares no time to carve.
expect(clipsOverlap({ start: 0, duration: 5 }, { start: 5, duration: 5 })).toBe(false);
});
it("treats a null or undefined duration as unbounded", () => {
expect(clipsOverlap({ start: 0, duration: null }, { start: 100, duration: 1 })).toBe(true);
expect(clipsOverlap({ start: 0 }, { start: 100, duration: 1 })).toBe(true);
// Symmetric: the unbounded span can be on either side.
expect(clipsOverlap({ start: 100, duration: 1 }, { start: 0, duration: undefined })).toBe(true);
});
it("gives a zero-duration clip a single instant, not a span", () => {
expect(clipsOverlap({ start: 5, duration: 0 }, { start: 5, duration: 5 })).toBe(false);
expect(clipsOverlap({ start: 5, duration: 0 }, { start: 4, duration: 5 })).toBe(true);
});
it("clamps a negative duration to zero rather than inverting the interval", () => {
// The regression: end = start + duration puts a negative-duration clip's
// end BEFORE its start, and end(a) is what the other clip's start gets
// compared against — so a smaller (earlier) broken end silently rejects
// real overlaps too. {start:10, duration:-5} clamped is a zero-length
// clip AT t=10, which genuinely sits inside {start:6, duration:20}'s
// [6, 26) span; the unclamped math missed it (end(a) came out to 5).
expect(clipsOverlap({ start: 10, duration: -5 }, { start: 6, duration: 20 })).toBe(true);
// And it stays correct where it isn't inside anything.
expect(clipsOverlap({ start: 10, duration: -5 }, { start: 20, duration: 5 })).toBe(false);
});
});
describe("mixCarveSources", () => {
const tone = (seconds: number, level: number, sampleRate = 48000) =>
new Float32Array(Math.round(seconds * sampleRate)).fill(level);
it("places every voice where it starts on the bed's clock", () => {
// Three people talking at different times is still one question — where and
// when is speech masking this bed — so they become one signal.
const mixed = mixCarveSources(
[
{ samples: tone(1, 0.5), offsetSeconds: 1 },
{ samples: tone(1, 0.25), offsetSeconds: 3 },
],
48000,
);
expect(mixed.length).toBe(4 * 48000);
const at = (t: number) => mixed[Math.round(t * 48000)];
expect(at(0.5)).toBe(0); // before anyone speaks
expect(at(1.5)).toBeCloseTo(0.5, 5);
expect(at(2.5)).toBe(0); // the gap between them
expect(at(3.5)).toBeCloseTo(0.25, 5);
});
it("sums voices that overlap, because two at once mask more than one", () => {
const mixed = mixCarveSources(
[
{ samples: tone(1, 0.3), offsetSeconds: 0 },
{ samples: tone(1, 0.3), offsetSeconds: 0 },
],
48000,
);
expect(mixed[0]).toBeCloseTo(0.6, 5);
});
it("drops the part of a voice that plays before the bed starts", () => {
// It masks nothing there, and folding it in at zero would put a cut where
// there is no voice.
const mixed = mixCarveSources([{ samples: tone(1, 0.5), offsetSeconds: -0.5 }], 48000);
expect(mixed.length).toBe(0.5 * 48000);
expect(mixed[0]).toBeCloseTo(0.5, 5);
});
it("has nothing to mix when there are no voices", () => {
expect(mixCarveSources([], 48000)).toHaveLength(0);
});
});
describe("carve settings written before this took a list of voices", () => {
it("reads a single `source` as a one-voice list, and forgets `dynamic`", () => {
// Every carve is dynamic now: a static one thinned the bed through every pause,
// and nobody wanted that once they had heard both.
const read = normalizeCarveSettings({ source: "vo", strength: 0.4, dynamic: false } as never);
expect(read.sources).toEqual(["vo"]);
expect(read.strength).toBe(0.4);
expect("dynamic" in read).toBe(false);
});
it("drops empty ids rather than carrying a source that names nothing", () => {
expect(normalizeCarveSettings({ sources: ["", "vo", ""] } as never).sources).toEqual(["vo"]);
expect(normalizeCarveSettings({ source: "" } as never).sources).toEqual([]);
});
});
+170 -12
View File
@@ -46,19 +46,28 @@ export interface HfCarveBand {
* once.
*/
export interface HfCarveSettings {
/** Element id of the voice track to analyse. */
source: string;
/**
* Element ids of every voice track this bed makes room for.
*
* More than one because a bed usually runs under a whole sequence: a narrator, an
* interview answer, a second presenter. Each occupies its own stretch of the bed,
* and carving against only one of them leaves the others fighting it. They are
* analysed together see `mixCarveSources` so the cuts follow whoever is
* speaking rather than averaging strangers.
*/
sources: string[];
/** How hard to carve, 0..1. */
strength: number;
/**
* Follow the voice rather than sitting at a fixed depth.
* Whether the carve is applied at all.
*
* A static carve holds its cuts for the whole clip, including every pause the
* bed is thinned where there is nothing to make room for. Dynamic turns every
* value into an envelope of the voice's own level, so silence leaves the bed
* alone and a loud passage pushes the carve to full depth.
* A bed under a voice wants carving, so a track that has never been configured
* is treated as on and carved without being asked. That default needs an off
* switch that survives: with "off" represented by having no settings at all,
* selecting the clip again would read it as never-configured and re-apply. So
* switching it off writes `enabled: false` and the default stops applying.
*/
dynamic: boolean;
enabled: boolean;
}
/** The numbers the analysis actually works in, all derived from `strength`. */
@@ -82,13 +91,110 @@ export interface HfCarveProfile {
headroomDb: number;
}
/**
* What a track's name suggests it holds.
*
* Only ever a hint a name is what the author called something, not what is in the
* file so this is used to order and to filter a list of candidates, never to
* decide alone. `unknown` is deliberately common: a track called `a1` could be
* anything, and treating an unrecognised name as "not a voice" would hide the one
* track somebody needs to pick.
*/
export type HfAudioNameKind = "voice" | "music" | "sfx" | "unknown";
/** Short, deliberately dull effects. Nothing here is ever a voiceover. */
const SFX_NAME =
/sfx|foley|whoosh|impact|riser|stinger|swoosh|thud|boom|click|ding|beep|ambien|room[-_ ]?tone/i;
// `\b` treats `_` as a word character, so `\bbed\b` does not match `bed_01`
// or `music_bed_loop` — exactly the separator an asset name is likely to use.
// These short words need a boundary that actually excludes letters and
// digits on both sides; everything else here is long enough that a
// substring match is already the intent (`music` inside `bgmusic` is fine).
const NOT_WORD = "(?<![a-z0-9])";
const NOT_WORD_END = "(?![a-z0-9])";
const wordish = (term: string): string => `${NOT_WORD}${term}${NOT_WORD_END}`;
/** A bed, which is the thing being carved rather than the thing carving it. */
const MUSIC_NAME = new RegExp(
`music|bgm|${wordish("bed")}|soundtrack|score|${wordish("song")}|theme|instrumental|track\\d`,
"i",
);
/** Speech. */
const VOICE_NAME = new RegExp(
`voice|${wordish("vo")}|${wordish("vox")}|narrat|speech|dialog|monolog|announce|` +
`${wordish("tts")}|talk|interview|podcast|recap|script`,
"i",
);
/**
* Classify a track from its id and filename together.
*
* Both, because either can be the informative one: an author naming elements `a1`
* and `a2` may still have `narration.mp3` and `bgm.mp3` as their sources, and one
* naming them `voice` and `music` may have opaque hashes for filenames.
*
* Voice is tested first: a file called `voiceover-music-bed.wav` is more likely the
* voiceover than the bed, and a track matching both hints is better offered than
* hidden.
*/
export function classifyAudioName(
...parts: readonly (string | null | undefined)[]
): HfAudioNameKind {
const text = parts.filter(Boolean).join(" ");
if (VOICE_NAME.test(text)) return "voice";
if (SFX_NAME.test(text)) return "sfx";
if (MUSIC_NAME.test(text)) return "music";
return "unknown";
}
/** A clip's place on the timeline. A duration that is not a number is unbounded. */
export interface HfClipSpan {
start: number;
duration?: number | null;
}
/**
* Do these two clips share any time at all?
*
* A voice that never plays while the bed does cannot mask it, so it has no business
* in the carve: it would contribute silence to the analysis and, worse, invite the
* author to wonder why including it changed nothing.
*
* An unknown duration counts as unbounded rather than as zero. Refusing a track
* because its length is not written down would drop the commonest case there is a
* clip whose duration the composition leaves to the media itself.
*/
export function clipsOverlap(a: HfClipSpan, b: HfClipSpan): boolean {
const end = (clip: HfClipSpan): number =>
typeof clip.duration === "number" && Number.isFinite(clip.duration)
? // Negative is clamped to zero-length rather than passed through: a
// clip cannot un-play time, and letting it through inverts the
// interval (end before start), which reads as overlapping everything
// it is nowhere near.
clip.start + Math.max(0, clip.duration)
: Number.POSITIVE_INFINITY;
return a.start < end(b) && b.start < end(a);
}
/**
* Could this track be the voice a carve listens to?
*
* Music and SFX are out: a bed is the thing being carved, and a 200 ms whoosh has
* no speech to make room for. Everything else stays in, including names that say
* nothing see `HfAudioNameKind`.
*/
export function couldBeCarveSource(...parts: readonly (string | null | undefined)[]): boolean {
const kind = classifyAudioName(...parts);
return kind === "voice" || kind === "unknown";
}
export const DEFAULT_CARVE: HfCarveSettings = {
source: "",
enabled: true,
sources: [],
// A quarter, because the knob's range was doubled and this is the point on the
// new scale that produces what the panel has always defaulted to. Switching
// carve on sounds the same as it did; the extra range is above, not under.
strength: 0.25,
dynamic: false,
};
/**
@@ -133,10 +239,16 @@ export function carveProfile(strength: number): HfCarveProfile {
export function normalizeCarveSettings(
raw: Partial<HfCarveSettings & HfCarveProfile> | undefined,
): HfCarveSettings {
// `source` and `dynamic` are gone from the type but still out there in files.
const legacy = raw as (Partial<HfCarveSettings> & { source?: unknown }) | undefined;
const num = (v: unknown): number | null => {
const n = typeof v === "number" ? v : Number(v);
return Number.isFinite(n) ? n : null;
};
// No attribute at all is not a carve to read, it is the absence of one — so the
// defaults apply whole, dynamic included. Only a stored object gets the reading
// below, where a missing `dynamic` means the static carve it was written as.
if (raw === undefined || raw === null) return { ...DEFAULT_CARVE };
const strength = num(raw?.strength);
const legacyDepth = num(raw?.maxCutDb);
const resolved =
@@ -147,13 +259,59 @@ export function normalizeCarveSettings(
// reads back as the strength that produces 6 dB.
(legacyDepth - 2) / 16
: DEFAULT_CARVE.strength;
// A carve written before this took a list names its one voice in `source`.
const stored = Array.isArray(raw?.sources)
? raw.sources
: typeof legacy?.source === "string"
? [legacy.source]
: [];
return {
source: typeof raw?.source === "string" ? raw.source : "",
// Absent means on: every carve written before the flag existed was applied.
enabled: raw?.enabled !== false,
sources: stored.filter((id): id is string => typeof id === "string" && id !== ""),
strength: Math.min(1, Math.max(0, resolved)),
dynamic: raw?.dynamic === true,
};
}
/**
* Every voice as one signal on the BED's clock.
*
* The analysis asks one question where and when is speech masking this bed and
* that question has one answer even when three people are talking at different
* times. Summing them onto the bed's timeline first means the existing analysis
* needs no notion of "which voice": bands come out of all the speech there is, and
* the envelopes rise wherever any of it is happening.
*
* `offsetSeconds` is where each voice starts relative to the bed. Audio before the
* bed begins is dropped rather than folded in at zero: it plays over nothing and
* cannot mask anything, and shifting it would put a cut where there is no voice.
*
* Summed, not averaged. Two people speaking at once mask more than either alone,
* which is exactly what the carve should answer to.
*/
export function mixCarveSources(
parts: readonly { samples: Float32Array; offsetSeconds: number }[],
sampleRate: number,
): Float32Array {
const placed = parts.map((part) => ({
samples: part.samples,
at: Math.round(part.offsetSeconds * sampleRate),
}));
const length = placed.reduce((max, p) => Math.max(max, p.at + p.samples.length), 0);
if (length <= 0) return new Float32Array(0);
const mixed = new Float32Array(length);
for (const { samples, at } of placed) {
// A voice starting before the bed contributes only the part that overlaps it.
const from = at < 0 ? -at : 0;
for (let i = from; i < samples.length; i += 1) {
const target = at + i;
if (target < 0 || target >= length) continue;
mixed[target] = (mixed[target] ?? 0) + (samples[i] ?? 0);
}
}
return mixed;
}
/** Averaged power spectrum, Welch-style. */
function powerSpectrum(
mono: Float32Array,
@@ -0,0 +1,36 @@
import { describe, expect, it } from "vitest";
import { buildWaveformCacheKey } from "./waveform.js";
describe("buildWaveformCacheKey", () => {
it("is stable for the same file", () => {
const a = buildWaveformCacheKey("assets/music-bed.m4a", { size: 4187869, mtimeMs: 1000 });
const b = buildWaveformCacheKey("assets/music-bed.m4a", { size: 4187869, mtimeMs: 1000 });
expect(a).toBe(b);
});
it("changes when the file behind the path is replaced", () => {
// The case this exists for: an asset rebuilt in place — same name, new
// content. Keyed on the path alone the cache served the old peaks forever,
// so a bed whose ducking had just been removed still drew as ducked.
const before = buildWaveformCacheKey("assets/music-bed.m4a", { size: 4187869, mtimeMs: 1000 });
const after = buildWaveformCacheKey("assets/music-bed.m4a", { size: 3900000, mtimeMs: 2000 });
expect(after).not.toBe(before);
});
it("separates two files of the same size edited at different times, and vice versa", () => {
const base = { size: 100, mtimeMs: 1000 };
expect(buildWaveformCacheKey("a.m4a", base)).not.toBe(
buildWaveformCacheKey("a.m4a", { ...base, mtimeMs: 1001 }),
);
expect(buildWaveformCacheKey("a.m4a", base)).not.toBe(
buildWaveformCacheKey("a.m4a", { ...base, size: 101 }),
);
});
it("keeps distinct assets apart and stays a plain filename", () => {
const fp = { size: 10, mtimeMs: 5 };
expect(buildWaveformCacheKey("a/b.m4a", fp)).not.toBe(buildWaveformCacheKey("a/c.m4a", fp));
expect(buildWaveformCacheKey("a/b.m4a", fp)).not.toMatch(/[/\\]/);
expect(buildWaveformCacheKey("a/b.m4a", fp)).toMatch(/\.json$/);
});
});
+25 -4
View File
@@ -1,5 +1,5 @@
import { spawn } from "node:child_process";
import { existsSync, writeFileSync, mkdirSync } from "node:fs";
import { existsSync, writeFileSync, mkdirSync, statSync } from "node:fs";
import { join } from "node:path";
import { findFfBinary } from "@hyperframes/parsers/ff-binaries";
@@ -7,8 +7,28 @@ const SAMPLE_RATE = 4000;
const PEAK_COUNT = 4000;
const WAVEFORM_CACHE_VERSION = "v2";
export function buildWaveformCacheKey(assetPath: string): string {
return `${WAVEFORM_CACHE_VERSION}_${assetPath.replace(/[/\\]/g, "_")}.json`;
/**
* Cache filename for one asset's peaks, keyed on its content as well as its name.
*
* The path alone is not an identity. An asset rebuilt in place a bed
* re-encoded without its ducking, a plate swapped for the right one keeps its
* name and gets new samples, and a path-keyed entry then served the old peaks
* for the rest of the project's life: the timeline drew a duck that was no
* longer in the file, which reads as the render having done it. Size and mtime
* are what a rebuild always changes, and both are already on the stat the route
* takes to check the file exists.
*
* Without a fingerprint it falls back to the old path-only key, so a caller that
* cannot stat still gets caching rather than an error.
*/
export function buildWaveformCacheKey(
assetPath: string,
fingerprint?: { size: number; mtimeMs: number },
): string {
const name = assetPath.replace(/[/\\]/g, "_");
if (!fingerprint) return `${WAVEFORM_CACHE_VERSION}_${name}.json`;
const stamp = `${fingerprint.size}-${Math.round(fingerprint.mtimeMs)}`;
return `${WAVEFORM_CACHE_VERSION}_${name}_${stamp}.json`;
}
function computePeaks(floats: Float32Array, count: number): number[] {
@@ -74,8 +94,9 @@ export async function generateWaveformCache(projectDir: string, assetPath: strin
const audioPath = join(projectDir, assetPath);
if (!existsSync(audioPath)) return;
const stats = statSync(audioPath);
const cacheDir = join(projectDir, ".waveform-cache");
const cachePath = join(cacheDir, buildWaveformCacheKey(assetPath));
const cachePath = join(cacheDir, buildWaveformCacheKey(assetPath, stats));
if (existsSync(cachePath)) return;
const peaks = await decodeAudioPeaks(audioPath);
@@ -1,4 +1,4 @@
import { existsSync, readFileSync, writeFileSync, mkdirSync } from "node:fs";
import { existsSync, readFileSync, writeFileSync, mkdirSync, statSync } from "node:fs";
import { join } from "node:path";
import type { Hono } from "hono";
import type { StudioApiAdapter } from "../types.js";
@@ -13,10 +13,13 @@ export function registerWaveformRoutes(api: Hono, adapter: StudioApiAdapter): vo
c.req.path.replace(`/projects/${project.id}/waveform/`, "").split("?")[0] ?? "",
);
const audioPath = join(project.dir, assetPath);
if (!existsSync(audioPath)) return c.json({ error: "file not found" }, 404);
const stats = statSync(audioPath, { throwIfNoEntry: false });
if (!stats) return c.json({ error: "file not found" }, 404);
const cacheDir = join(project.dir, ".waveform-cache");
const cachePath = join(cacheDir, buildWaveformCacheKey(assetPath));
// Keyed on the file's size and mtime as well as its name, so re-encoding an
// asset in place invalidates its peaks instead of drawing the old ones.
const cachePath = join(cacheDir, buildWaveformCacheKey(assetPath, stats));
if (existsSync(cachePath)) {
try {
@@ -246,15 +246,15 @@ describe("AudioFxGroup carve", () => {
});
});
describe("AudioFxGroup dynamic carve", () => {
describe("AudioFxGroup carve analysis", () => {
const carvedChain = JSON.stringify({
version: 1,
nodes: [{ type: "lowpass", id: "n1", params: { frequency: 400, q: 0.9, poles: "2" } }],
});
// Strength 0 carves frequencies only — no level ducking — so the spectral
// cases measure just the spectral half. A case that wants the duck raises it.
const settings = (dynamic: boolean, over: Record<string, unknown> = {}) =>
JSON.stringify({ source: "vo", strength: 0, dynamic, ...over });
const settings = (over: Record<string, unknown> = {}) =>
JSON.stringify({ sources: ["vo"], strength: 0, ...over });
/** The value written for one attribute, whatever order the writes landed in. */
const writeFor = (calls: unknown[][], attr: string) =>
@@ -267,9 +267,6 @@ describe("AudioFxGroup dynamic carve", () => {
select.dispatchEvent(new Event("change", { bubbles: true }));
};
const dynamicBox = (host: HTMLElement) =>
host.querySelector<HTMLInputElement>(".hf-fx-carve-dynamic")!;
/** A voice with a pause in it, decoded through a stubbed offline context. */
function stubDecode(): void {
const sampleRate = 48000;
@@ -292,114 +289,11 @@ describe("AudioFxGroup dynamic carve", () => {
afterEach(() => vi.unstubAllGlobals());
it("records the choice in the carve settings", () => {
const { host, onSetAttributeQuiet } = mount({
"fx-chain": carvedChain,
"fx-carve": settings(false),
});
act(() => dynamicBox(host).click());
const write = onSetAttributeQuiet.mock.calls.find((c) => c[0] === "data-fx-carve");
expect(JSON.parse(String(write![1])).dynamic).toBe(true);
});
it("automates the carve filters' gain from the voice, in the bed's own time", async () => {
stubDecode();
// Voice starts 10s into the composition, bed at 0: the envelope is measured
// against the voice but read from the start of the bed, so it has to shift.
const { host, onSetAttributeQuiet } = mount({
"fx-chain": carvedChain,
// No source yet: picking one is what applies the carve.
"fx-carve": settings(true, { source: "" }),
start: "0",
});
const vo = document.getElementById("vo")!;
vo.setAttribute("data-start", "10");
vo.setAttribute("src", "voice.wav");
await act(async () => {
pickSource(host, "vo");
});
// Chain first, then automation: a lane naming a node the chain does not
// carry yet is dropped when it is read back.
const order = onSetAttributeQuiet.mock.calls.map((c) => c[0]);
// The settings land first, then the filters they imply, then the envelopes.
expect(order.indexOf("data-fx-chain")).toBeLessThan(order.indexOf("data-automation"));
const carved = writeFor(onSetAttributeQuiet.mock.calls, "data-fx-chain").nodes;
const carveNode = carved.find((n: { fromCarve?: boolean }) => n.fromCarve);
expect(carveNode.id).toBeTruthy();
const lanes = writeFor(onSetAttributeQuiet.mock.calls, "data-automation").lanes;
const lane = lanes.find((l: { target: string }) => l.target === `fx.${carveNode.id}.gain`) as {
points: { t: number; v: number }[];
};
expect(lane).toBeTruthy();
// Flat at the bed's own start, before the voice exists at all.
expect(lane.points[0]).toMatchObject({ t: 0, v: 0 });
// The voice's pause is at 0-1s of its own clip, so 10-11s of the bed's.
expect(lane.points.find((p) => p.t > 10.5 && p.t < 11)?.v ?? 0).toBe(0);
// And it cuts once the voice speaks, a second later. Depth is per band and
// relative to that band's own peak in the voice, so the invariant is that the
// envelope gets most of the way to what the analysis put on the node — not a
// fixed number of dB, which changes with the band the analysis chose.
const bandGain = Number(carveNode.params?.gain ?? 0);
// At least half the depth the analysis put on the node; the exact floor
// depends on which band it chose and how the envelope was thinned.
expect(Math.min(...lane.points.map((p) => p.v))).toBeLessThanOrEqual(bandGain * 0.5);
// Ends back at no cut, so the bed is not left dipped for the rest of the clip.
expect(lane.points.at(-1)!.v).toBe(0);
});
it("adds a gain stage that ducks the bed under the voice, automated when dynamic", async () => {
// Carving frequencies cannot beat a bed that is simply louder than the
// voice. The level half rides a gain node the carve owns, so the track's own
// volume lane is left alone.
it("holds one measured value from the voice and bed", async () => {
stubDecode();
const { host, onSetAttributeQuiet } = mount({
"fx-chain": carvedChain,
"fx-carve": settings(true, { strength: 1, source: "" }),
start: "0",
});
const vo = document.getElementById("vo")!;
vo.setAttribute("data-start", "0");
vo.setAttribute("src", "voice.wav");
// The bed is measured too — "how far over the voice is it" needs both.
document.getElementById("bed")!.setAttribute("src", "bed.m4a");
await act(async () => {
pickSource(host, "vo");
});
const nodes = writeFor(onSetAttributeQuiet.mock.calls, "data-fx-chain").nodes;
const gain = nodes.find((n: { type: string }) => n.type === "gain");
expect(gain).toBeTruthy();
expect(gain.fromCarve).toBe(true);
// Dynamic hands the value to the envelope, so the static one stays at unity.
expect(gain.params.gain).toBe(0);
const lanes = writeFor(onSetAttributeQuiet.mock.calls, "data-automation").lanes;
const duckLane = lanes.find((l: { target: string }) => l.target === `fx.${gain.id}.gain`);
expect(duckLane).toBeTruthy();
expect(Math.min(...duckLane.points.map((p: { v: number }) => p.v))).toBeLessThan(0);
// Every carved band gets an envelope reaching that band's own analysed depth.
for (const node of nodes.filter((n: { type: string }) => n.type === "peaking")) {
const lane = lanes.find((l: { target: string }) => l.target === `fx.${node.id}.gain`) as
| { points: { v: number }[] }
| undefined;
expect(lane, `band ${node.id} has no envelope`).toBeTruthy();
const deepest = Math.min(...lane!.points.map((p) => p.v));
expect(deepest).toBeLessThanOrEqual(0);
expect(deepest).toBeGreaterThanOrEqual(node.params.gain - 0.2);
expect(deepest).toBeLessThanOrEqual(node.params.gain * 0.5);
}
// The author's own volume lane is not something a carve gets to touch.
expect(lanes.some((l: { target: string }) => l.target === "volume")).toBe(false);
});
it("holds one measured value when the carve is not dynamic", async () => {
stubDecode();
const { host, onSetAttributeQuiet } = mount({
"fx-chain": carvedChain,
"fx-carve": settings(false, { strength: 1, source: "" }),
"fx-carve": settings({ strength: 1, sources: [] }),
start: "0",
});
const vo = document.getElementById("vo")!;
@@ -413,7 +307,7 @@ describe("AudioFxGroup dynamic carve", () => {
const nodes = writeFor(onSetAttributeQuiet.mock.calls, "data-fx-chain").nodes;
const gain = nodes.find((n: { type: string }) => n.type === "gain");
expect(gain.params.gain).toBeLessThan(0);
// Nothing to schedule: a static carve is a value, not an envelope.
// Nothing to schedule: a carve is a value, not an envelope.
expect(onSetAttributeQuiet.mock.calls.some((c) => c[0] === "data-automation")).toBe(false);
});
@@ -421,7 +315,7 @@ describe("AudioFxGroup dynamic carve", () => {
stubDecode();
const { host, onSetAttributeQuiet } = mount({
"fx-chain": carvedChain,
"fx-carve": settings(true, { strength: 0, source: "" }),
"fx-carve": settings({ strength: 0, sources: [] }),
start: "0",
});
document.getElementById("vo")!.setAttribute("src", "voice.wav");
@@ -439,7 +333,7 @@ describe("AudioFxGroup dynamic carve", () => {
stubDecode();
const { host, onSetAttributeQuiet } = mount({
"fx-chain": JSON.stringify({ version: 1, nodes: [] }),
"fx-carve": JSON.stringify({ source: "", strength: 0.25, dynamic: true }),
"fx-carve": JSON.stringify({ sources: [], strength: 0.25 }),
start: "0",
});
document.getElementById("vo")!.setAttribute("src", "voice.wav");
@@ -448,44 +342,17 @@ describe("AudioFxGroup dynamic carve", () => {
pickSource(host, "vo");
});
const written = onSetAttributeQuiet.mock.calls.map((c) => c[0]);
expect(written).toEqual(["data-fx-carve", "data-fx-chain", "data-automation"]);
expect(written).toEqual(["data-fx-carve", "data-fx-chain"]);
const nodes = JSON.parse(
String(onSetAttributeQuiet.mock.calls.find((c) => c[0] === "data-fx-chain")![1]),
).nodes;
expect(nodes.every((n: { fromCarve?: boolean }) => n.fromCarve)).toBe(true);
});
it("re-applies when dynamic is switched on, not just when strength moves", async () => {
stubDecode();
const carvedAlready = JSON.stringify({
version: 1,
nodes: [
{
type: "peaking",
id: "n1",
fromCarve: true,
params: { frequency: 1000, gain: -6, q: 1.4 },
},
],
});
const { host, onSetAttributeQuiet } = mount({
"fx-chain": carvedAlready,
"fx-carve": settings(false, { strength: 0.25 }),
start: "0",
});
document.getElementById("vo")!.setAttribute("src", "voice.wav");
document.getElementById("bed")!.setAttribute("src", "bed.m4a");
await act(async () => {
host.querySelector<HTMLInputElement>(".hf-fx-carve-dynamic")!.click();
});
// Static and dynamic are different chains, so the switch has to rebuild them.
expect(onSetAttributeQuiet.mock.calls.some((c) => c[0] === "data-fx-chain")).toBe(true);
});
it("re-applies an existing carve when strength moves", async () => {
// Strength is the whole control surface, so it has to act on what is already
// applied. Left to the button alone, a carve kept the filters and envelopes
// its old strength produced and the knob silently described nothing.
// applied. Left to the button alone, a carve kept the filters its old
// strength produced and the knob silently described nothing.
stubDecode();
const carvedAlready = JSON.stringify({
version: 1,
@@ -501,7 +368,7 @@ describe("AudioFxGroup dynamic carve", () => {
});
const { host, onSetAttributeQuiet } = mount({
"fx-chain": carvedAlready,
"fx-carve": settings(true, { strength: 0.25 }),
"fx-carve": settings({ strength: 0.25 }),
start: "0",
});
document.getElementById("vo")!.setAttribute("src", "voice.wav");
@@ -516,9 +383,8 @@ describe("AudioFxGroup dynamic carve", () => {
const written = onSetAttributeQuiet.mock.calls.map((c) => c[0]);
expect(written).toContain("data-fx-carve");
// The settings land first, then the filters they imply, then the envelopes.
// The settings land first, then the filters they imply.
expect(written.indexOf("data-fx-carve")).toBeLessThan(written.indexOf("data-fx-chain"));
expect(written.indexOf("data-fx-chain")).toBeLessThan(written.indexOf("data-automation"));
const chainWrite = onSetAttributeQuiet.mock.calls.find((c) => c[0] === "data-fx-chain");
const nodes = JSON.parse(String(chainWrite![1])).nodes;
@@ -537,7 +403,7 @@ describe("AudioFxGroup dynamic carve", () => {
stubDecode();
const { host, onSetAttributeQuiet } = mount({
"fx-chain": JSON.stringify({ version: 1, nodes: [] }),
"fx-carve": settings(true, { strength: 0.25, source: "" }),
"fx-carve": settings({ strength: 0.25, sources: [] }),
start: "0",
});
const dial = host.querySelector<HTMLInputElement>(".hf-fx-carve input[type=range]")!;
@@ -566,7 +432,7 @@ describe("AudioFxGroup dynamic carve", () => {
});
const { host, onSetAttributeQuiet, onSetAttributeLive } = mount({
"fx-chain": carvedAlready,
"fx-carve": settings(true, { strength: 0.25 }),
"fx-carve": settings({ strength: 0.25 }),
start: "0",
});
document.getElementById("vo")!.setAttribute("src", "voice.wav");
@@ -580,36 +446,6 @@ describe("AudioFxGroup dynamic carve", () => {
expect(onSetAttributeLive.mock.calls.every((c) => c[0] === "data-fx-carve")).toBe(true);
expect(onSetAttributeQuiet.mock.calls.some((c) => c[0] === "data-fx-chain")).toBe(false);
});
it("drops the envelopes when dynamic is switched back off", async () => {
// An automated gain ignores the panel's depth, so leaving the lanes behind
// would keep the filters following a voice with nothing saying they do.
const automation = JSON.stringify({
version: 1,
lanes: [
{ target: "fx.n2.gain", points: [{ t: 0, v: 0 }] },
{ target: "volume", points: [{ t: 0, v: 1 }] },
],
});
const withCarveNode = JSON.stringify({
version: 1,
nodes: [
{ type: "peaking", id: "n2", fromCarve: true, params: { frequency: 1000, gain: -6 } },
],
});
const { host, onSetAttributeQuiet } = mount({
"fx-chain": withCarveNode,
"fx-carve": settings(true),
automation,
});
await act(async () => {
dynamicBox(host).click();
});
const write = onSetAttributeQuiet.mock.calls.find((c) => c[0] === "data-automation");
expect(write).toBeTruthy();
const lanes = JSON.parse(String(write![1])).lanes;
expect(lanes.map((l: { target: string }) => l.target)).toEqual(["volume"]);
});
});
describe("AudioFxGroup successive edits", () => {
@@ -20,7 +20,6 @@ import {
import {
analyseCarveBands,
analyseCarveDuck,
analyseCarveDynamics,
carveBandsToChain,
carveProfile,
HF_AUDIO_CARVE_ATTR,
@@ -31,7 +30,6 @@ import {
fxAutomationTarget,
sampleAutomationLane,
type HfAutomation,
type HfAutomationLane,
} from "@hyperframes/core/audio-automation";
import {
automatedTargetsOf,
@@ -185,10 +183,7 @@ export function AudioFxGroup({
* commit, which does not exist yet.
*/
const setCarve = async (next: HfCarveSettings | null): Promise<void> => {
// Envelopes the carve wrote outlive it otherwise, and an automated gain
// ignores the panel's own depth — so switching dynamic off would leave the
// filters still following the voice with nothing saying they do.
if (!next || (carve?.dynamic && !next.dynamic)) {
if (!next) {
const carriedOver = withoutCarveLanes(automation, chain);
if (carriedOver.lanes.length !== automation.lanes.length) {
await onSetAttributeQuiet(
@@ -215,11 +210,10 @@ export function AudioFxGroup({
// is already there. A carve with no source yet has nothing to analyse.
const changed =
next &&
next.source &&
next.sources.length > 0 &&
(!carve ||
next.source !== carve.source ||
next.strength !== carve.strength ||
next.dynamic !== carve.dynamic);
next.sources.join("") !== carve.sources.join("") ||
next.strength !== carve.strength);
if (next && changed) await analyse(next);
};
@@ -258,7 +252,7 @@ export function AudioFxGroup({
if (other.id === element.id) continue;
try {
const raw = other.getAttribute(HF_AUDIO_CARVE_ATTR);
if (raw && normalizeCarveSettings(JSON.parse(raw)).source === element.id) {
if (raw && normalizeCarveSettings(JSON.parse(raw)).sources.includes(element.id)) {
return other.id || "another track";
}
} catch {
@@ -284,9 +278,10 @@ export function AudioFxGroup({
* hand-added effects alone, so re-analysing does not discard other work.
*/
const analyse = async (active: HfCarveSettings | null = carve): Promise<void> => {
if (!active?.source) return;
const activeSource = active?.sources[0];
if (!activeSource) return;
const doc = element.element?.ownerDocument;
const voice = doc?.getElementById(active.source) as HTMLAudioElement | null;
const voice = doc?.getElementById(activeSource) as HTMLAudioElement | null;
const src = voice?.getAttribute("src");
if (!src) return;
setAnalysing(true);
@@ -348,14 +343,13 @@ export function AudioFxGroup({
};
const carvedNodes: HfAudioFxNode[] = carved.nodes.map(mint);
// The gain stage sits after the filters, and only exists when the carve was
// asked to make level room. Dynamic drives it from the envelope; static
// holds the one value above.
// asked to make level room, holding the one value computed above.
const duckNode =
duck.length > 0
? mint({
type: "gain",
enabled: true,
params: { ...defaultAudioFxParams("gain"), gain: active.dynamic ? 0 : staticDuckDb },
params: { ...defaultAudioFxParams("gain"), gain: staticDuckDb },
})
: null;
const next = {
@@ -372,42 +366,11 @@ export function AudioFxGroup({
// pruned when it is read back.
await onSetAttributeQuiet(HF_AUDIO_FX_ATTR, serializeAudioFxChain(next));
/**
* One carve envelope as a lane on this bed's clock.
*
* Everything the analysis returns is timed from the start of the voice, so
* it shifts by the gap between the two clips; and a lane holds its first
* value backwards to the start of its own clip, so a bed that begins before
* the voice needs an explicit "no cut" at zero or it starts out ducked.
*/
const laneFor = (id: string, points: { t: number; v: number }[]): HfAutomationLane[] => {
const shifted = points
.map((p) => ({ t: Number((p.t + offset).toFixed(3)), v: p.v }))
.filter((p) => p.t >= 0);
if ((shifted[0]?.t ?? 0) > 0) shifted.unshift({ t: 0, v: 0 });
return shifted.length > 1
? [{ target: fxAutomationTarget(id, "gain"), points: shifted }]
: [];
};
// Dynamic carve: each filter's depth becomes an envelope of the voice's
// level in that band, so pauses leave the bed alone.
const lanes: HfAutomationLane[] = active.dynamic
? analyseCarveDynamics(buffer.getChannelData(0), buffer.sampleRate, bands).flatMap(
(dyn, i) => {
const id = carvedNodes[i]?.id;
if (!id) return [];
return laneFor(id, dyn.points);
},
)
: [];
// The level envelope rides the gain stage, on the same clock as the bands.
if (active.dynamic && duckNode?.id && duck.length > 0) {
lanes.push(...laneFor(duckNode.id, duck));
}
// A carve written before dynamic mode was removed may still carry the
// envelope lanes it automated; a re-run is static now, so they are stale.
const carriedOver = withoutCarveLanes(automation, chain);
if (lanes.length > 0 || carriedOver.lanes.length !== automation.lanes.length) {
writeAutomation({ version: 1, lanes: [...carriedOver.lanes, ...lanes] });
if (carriedOver.lanes.length !== automation.lanes.length) {
writeAutomation(carriedOver);
}
} catch {
// Leave the chain as it was; the button simply re-enables.
@@ -346,13 +346,13 @@ describe("FxSection carve", () => {
it("offers no analyse button — picking a voice is the whole gesture", () => {
// A carve with a source and no filters is a setting nobody applied; the
// button was a second step for something the panel already knew to do.
const { host } = mount({ carve: { ...DEFAULT_CARVE, source: "vo" } });
const { host } = mount({ carve: { ...DEFAULT_CARVE, sources: ["vo"] } });
expect(host.querySelector(".hf-fx-analyse")).toBeNull();
expect(host.textContent).not.toMatch(/Analyse/i);
});
it("says when it is working, since there is no button to grey out", () => {
const { host } = mount({ carve: { ...DEFAULT_CARVE, source: "vo" }, analysing: true });
const { host } = mount({ carve: { ...DEFAULT_CARVE, sources: ["vo"] }, analysing: true });
expect(host.querySelector(".hf-fx-carve-working")?.textContent).toMatch(/Analysing/i);
});
@@ -516,7 +516,7 @@ describe("voiceover carve visibility", () => {
const { host } = mount({
chain: chainOf("lowpass"),
sourceOptions: [],
carve: { ...DEFAULT_CARVE, source: "vo" },
carve: { ...DEFAULT_CARVE, sources: ["vo"] },
});
expect(carveBlock(host)).toBeTruthy();
});
@@ -748,9 +748,9 @@ export function FxSection({
</span>
<select
className="hf-fx-select min-w-0 flex-1 rounded-[3px] bg-panel-surface px-1 py-0.5 font-mono text-[10px] text-panel-text-0"
value={carve.source}
value={carve.sources[0] ?? ""}
disabled={disabled}
onChange={(e) => onCarveChange({ ...carve, source: e.target.value })}
onChange={(e) => onCarveChange({ ...carve, sources: [e.target.value] })}
>
<option value="">Select a voice track</option>
{sourceOptions.map((o) => (
@@ -783,26 +783,6 @@ export function FxSection({
onChange={(_k, v) => previewCarve({ ...carve, strength: Number(v) })}
onCommit={(_k, v) => onCarveChange({ ...carve, strength: Number(v) })}
/>
{/* A static carve holds its cuts for the whole clip, pauses
included. Dynamic hands every value to an envelope of the voice's
own level, so the bed is only worked on while there is something
to make room for. Written as ordinary automation, which is why the
lanes show up in the timeline and can be edited afterwards. */}
<label className="hf-fx-row flex min-h-6 items-center gap-2">
<span className="hf-fx-label w-[86px] flex-shrink-0 truncate text-[10px] text-panel-text-4">
Dynamic
</span>
<input
type="checkbox"
className="hf-fx-carve-dynamic h-3 w-3 accent-panel-accent"
checked={carve.dynamic}
disabled={disabled}
onChange={(e) => onCarveChange({ ...carve, dynamic: e.target.checked })}
/>
<span className="min-w-0 flex-1 truncate text-[9px] text-panel-text-4">
follows the voice, flat where it is silent
</span>
</label>
{analysing ? (
<p className="hf-fx-carve-working py-1 text-center text-[10px] text-panel-text-4">
Analysing
+5 -1
View File
@@ -18,13 +18,17 @@
"files": 4
},
"hyperframes": {
"hash": "3a3ea01fe52e1600",
"hash": "756a72f58fa3739b",
"files": 17
},
"hyperframes-animation": {
"hash": "5bc2ce098387a547",
"files": 121
},
"hyperframes-audio": {
"hash": "08f593a7c159ce04",
"files": 4
},
"hyperframes-cli": {
"hash": "e042fcaaa3f9767f",
"files": 11
+272
View File
@@ -0,0 +1,272 @@
---
name: hyperframes-audio
description: >
Use when audio already placed in a HyperFrames composition needs to be mixed:
a music bed that fights a voiceover (voiceover carve), effects on a track
(EQ, compressor, limiter, gate, saturation, delay, reverb, chorus, phaser,
bitcrush), or automation envelopes drawn on a track's volume or any effect
parameter.
Don't use for sourcing or generating audio — finding BGM, SFX, or making a
voiceover is `/media-use`. Don't use for clip timing or track layout, which is
`/hyperframes-core`.
---
# HyperFrames Audio
A mix is a set of relationships, not a stack of processors. Two tracks that each
sound right alone can be unlistenable together, and the fix is almost never "turn
one down" — it is finding what they are fighting over and giving it to whichever
one needs it. Every tool here exists to express one of those relationships.
Effects live on the element as `data-fx-chain`, and preview and render run the
same Web Audio graph — the studio in a live context, the engine in an offline one
inside the browser it already drives. There is one implementation of each effect,
so what you hear while scrubbing is what gets written. You never tune twice.
Three attributes carry everything, all on the audio/video element itself:
| Attribute | Holds |
| ----------------- | --------------------------------------------------------- |
| `data-fx-chain` | the effects, in signal order |
| `data-automation` | envelopes on this track's volume or its effect parameters |
| `data-fx-carve` | the carve's own settings, so it can be re-derived |
Exact JSON for each, and the rules a lane must satisfy: `references/attributes.md`.
Every effect with its parameters, ranges and units: `references/fx-registry.md`.
## How it fits together
Two authoring surfaces write those attributes; two runtimes read them through the
same builders. That shared middle is why preview predicts the render.
```mermaid
flowchart TB
voice["voice track<br/>media file"]
bed["music bed<br/>media file"]
subgraph AUTHOR["Authoring — the only things that write attributes"]
panel["Studio<br/>Voiceover carve control"]
script["scripts/carve.mjs<br/>detects the pair, dynamic by default"]
analysis["core/audioCarve.ts<br/>carveProfile · analyseCarveBands<br/>analyseCarveDuck · analyseCarveDynamics"]
panel --> analysis
script --> analysis
end
voice --> analysis
bed --> analysis
subgraph ATTRS["Written onto the bed element"]
carveAttr["data-fx-carve<br/>source · strength · dynamic"]
chainAttr["data-fx-chain<br/>peaking xN + gain, tagged fromCarve"]
autoAttr["data-automation<br/>a lane per carved parameter"]
end
analysis --> carveAttr
analysis --> chainAttr
analysis --> autoAttr
subgraph SHARED["One implementation, read by both"]
build["audioFxGraph.ts · buildFxChain"]
sched["audioFxAutomation.ts · scheduleChainAutomation"]
end
chainAttr --> build
autoAttr --> sched
build --> preview["Preview<br/>live AudioContext<br/>attachElementFxChain"]
sched --> preview
build --> render["Render<br/>OfflineAudioContext in the headless browser<br/>applyAudioFxChain"]
sched --> render
preview --> heard["what you hear while scrubbing"]
render --> wav["processed WAV<br/>+ chainTailSeconds so the mix lets the tail through"]
wav --> mix["engine · audioMixer<br/>volume lane baked into the PCM here, not in the graph"]
mix --> out["the rendered mix"]
edit["editing the attribute mid-playback"] -.->|MutationObserver| preview
```
The carve's own settings are never read at playback — the chain and lanes it
produced are what play. `data-fx-carve` exists so strength can be changed on an
existing carve instead of guessed back out of the filters.
Inside a carved bed the signal runs through the dips first, then the level match,
then anything you built yourself — which is why a limiter you add still acts as
the last ceiling:
```mermaid
flowchart LR
src["decoded bed"] --> p1["peaking<br/>400 Hz"]
p1 --> p2["peaking<br/>1 kHz"]
p2 --> p3["peaking<br/>1.6 kHz"]
p3 --> g["gain<br/>level match"]
g --> hand["your own effects<br/>e.g. limiter"]
hand --> dest["track gain, then out"]
l1["lane fx.n1.gain"] -.->|"envelope of the voice's<br/>level in that band"| p1
l4["lane fx.n4.gain"] -.->|"how far the bed<br/>ducks overall"| g
```
A static carve is the same graph with fixed values and no lanes at all.
## Reach for a family by the problem, not the name
**Filters** (`highpass`, `lowpass`, `peaking`, `lowshelf`, `highshelf`) decide
which frequencies a track is allowed to occupy. This is the first tool for two
sources colliding, because collisions happen in bands: a bed and a voice both
want 13 kHz, and taking that from the bed costs the bed far less than turning
the whole thing down costs the mix. A high-pass on a voice is the standard fix
for rumble; a low-pass darkens or muffles deliberately.
**Dynamics** (`gain`, `compressor`, `limiter`, `gate`) decide how a track's level
behaves over time. Compression narrows the distance between loud and quiet so the
quiet parts can come up. A limiter is a ceiling — it does not shape anything, it
guarantees nothing gets past. A gate removes what is below a threshold, which is
how you silence room tone between phrases. `gain` is a plain level stage, and it
is what an automation lane rides when a track has to move out of the way.
**Nonlinear** (`saturate`, `bitcrush`) changes the waveform's shape, which adds
harmonics that were not there. Reach for it when a track needs character or
grit rather than correction — and remember it is generative: it makes a thin
source denser, not cleaner.
**Time** (`delay`, `reverb`, `chorus`, `phaser`) puts a track in a space or gives
it width. These are the ones that most easily wreck a mix, because a tail or a
detuned copy occupies the same room a voice needs. Use them on the thing that
should sit _behind_ something else, and keep the wet amount lower than sounds
right in isolation.
The chain is serial: each effect processes what the one before it produced. So
corrective filtering goes early, character in the middle, and a limiter last
where it can actually act as a ceiling.
## Voiceover carve
**The problem it solves.** A music bed under a voice makes the voice hard to
follow. The reflex is to duck the whole bed, which works and costs the bed all of
its presence — the music goes limp for the entire voiceover. But the voice does
not need the whole spectrum. It needs the few bands it actually occupies. Carve
takes only those, and the bed keeps its low end and its top, so it is still music
while the voice is still intelligible.
**It is a relationship, not an effect.** The settings live on the _bed_ — the
track that gets processed — and they name the voice to listen to, exactly as a
sidechain compressor does: you select the track that gets quieter and pick what
makes it quieter. **Never put a carve on the voice track.** A voice carved
against itself is a bug, not a subtle mix choice.
**One knob.** `strength` is 0..1 and derives everything: how deep to cut, how
many bands, how wide, how far to favour intelligibility over raw voice energy,
how far the level may drop, how far under the voice to aim. Those six move
together in any real mix — a gentle carve is a shallow cut in few bands with
little ducking, a hard one is deeper in more bands with more — so they are one
relationship written once, in `carveProfile`. Default is `0.25` — a 6 dB dip in
three bands with 6 dB of level room, audible without sounding like a hole. At
`0.5` the dip reaches 10 dB, which is where a carve starts being heard as an
effect rather than as room for the voice; above that is deliberate territory for
a loud bed under a quiet voice. `0` is spectral only — one band, no level match
at all.
**Carve by default.** A bed playing under narration wants a carve; it is not a
polish step to get to if there is time. Place both tracks, run the command below,
listen. Skip it only when there is no narration for the music to sit under — a
music video, a title card, a montage cut to the track.
**Static or dynamic — dynamic unless you know otherwise.** A static carve holds
its cuts for the whole clip, including every pause, so the bed is thinned where
there is nothing to make room for. Dynamic turns every value into an envelope of
the voice's own level: silence leaves the bed alone, a loud passage pushes the
carve to full depth. That is what almost every voiceover wants, so it is the
default. Reach for `--static` only for wall-to-wall narration with no real gaps,
where an envelope is hundreds of breakpoints describing a constant.
**Level matching is part of it.** Spectral carving cannot fix a bed that is
simply louder than the voice. So the carve also measures how far over the voice
the bed sits and writes a `gain` stage: held at one value for a static carve,
driven by an envelope for a dynamic one. That envelope releases slowly on
purpose — music that snaps back to full the instant a word ends sounds like a
machine doing it.
**Running it.** In Studio: pick the voice in the bed's Voiceover carve control;
turning it on adds the modules and strength adjusts what is there. Headless —
which is the path when you are authoring a composition rather than editing one:
```bash
node <SKILL_DIR>/scripts/carve.mjs --comp index.html
```
That is the whole command. It finds the voice and the bed itself, carves
dynamically at the default strength, and prints what it decided:
```
bed music-bed (name looks like music)
voice narration (only track left)
carve strength 0.25 dynamic
bands 400Hz -6dB q1.4, 1000Hz -3dB q1.4, 1600Hz -3.17dB q1.4
level 216-point envelope, floor -6 dB
```
Name the pair with `--bed` / `--voice` when the composition has several plausible
tracks, `--strength` to push it, `--static` to hold one depth, `--dry-run` to see
that report and write nothing.
**How it picks the pair.** Names first, because that is what you already told it
and the answer is explainable: a track whose id or filename looks like music
(`music`, `bgm`, `bed`, `score`…) is the bed, one that looks like a voice
(`voice`, `vo`, `narration`, `speech`…) is the voice, and SFX-shaped names are not
candidates for either. If one role is filled and a single track is left, that
track takes the other role. Only when names decide nothing does it listen: it
measures how much of each track is quiet, and the one that stops between phrases
is the voice. **When two tracks are too close to call it refuses and asks you to
name them** rather than carving the wrong one — a bed carved against a bed is
silent and confusing, and typing two ids is cheap.
Same analysis functions as the panel, so the result is identical. Needs `ffmpeg`
on PATH and `@hyperframes/core` installed in the project (`npm i -D
@hyperframes/core`) — the CLI inlines core rather than shipping it, so it cannot
be borrowed from there.
**What it writes** is an ordinary chain of peaking filters plus a gain stage,
tagged `fromCarve`. That tagging is the whole trick: a re-run replaces the
previous carve and leaves every effect you built by hand — and every lane you
drew by hand — exactly where it was. So re-carving at a new strength is safe and
repeatable, and `data-fx-carve` exists so the settings can be read back rather
than guessed from the filters.
## Automation
A lane is a set of breakpoints on one parameter: `{t, v}` in clip-local seconds
and the parameter's own units. Targets are `volume` for the track's level, or
`fx.<nodeId>.<param>` for an effect's knob.
**Only some parameters can be automated, and a lane on the others is silently
inert.** A knob is automatable when a Web Audio `AudioParam` backs it. The four
worklet-based effects — `compressor`, `limiter`, `gate`, `bitcrush` — expose
none at all, so no lane on any of their parameters will ever move: to make a
compressor's behaviour change over time, automate a `gain` stage before it
instead. `references/fx-registry.md` marks every parameter.
## Verify
Almost no static gate covers the mix. The linter reads `data-automation` for
exactly one conflict — `audio_volume_double_automation`, a volume lane on a track
that also has a GSAP tween on `volume`, where the lane wins and the tween is
ignored — and nothing validates the chain or the effect lanes at all. What
enforces those is the render: a chain it cannot parse fails the whole mix rather
than quietly writing the dry signal, because a mix that sounds plausible and is
wrong is worse than a refusal. Preview is the opposite by design: an unreadable
chain plays dry so the composition stays workable.
A lane pointing at a node the chain does not have is pruned on read, not an
error — so a typo'd `nodeId` costs you the envelope silently. Read the ids back
out of the chain rather than assuming what was minted.
Effects with a tail (`reverb`, `delay`) make the rendered track **longer** than
its source, and the mix is told how much by the chain. So a bed with reverb no
longer ends exactly at its `data-duration`; that is expected, not a bug.
Beyond that, a mix is verified by rendering and listening. For a carve: the voice
should be legible without the bed sounding hollowed, and with `dynamic` the bed
should come back up between phrases rather than staying flat. If the bed sounds
notched rather than simply quieter under the voice, the strength is too high —
that is the one failure mode with an obvious sound.
@@ -0,0 +1,110 @@
# The three audio attributes
All three go on the `<audio>` / `<video>` element itself, JSON-encoded, so a
composition carries its whole mix in the HTML with nothing to load beside it.
Nothing static validates them: preview plays an unreadable chain dry to stay
workable, and the render refuses the whole mix rather than shipping a dry track
that sounds plausible and is wrong.
## `data-fx-chain` — the effects
```json
{
"version": 1,
"nodes": [
{ "type": "highpass", "id": "n1", "params": { "frequency": 120, "q": 0.707, "poles": "2" } },
{
"type": "peaking",
"id": "n2",
"fromCarve": true,
"params": { "frequency": 1600, "gain": -6, "q": 1.4 }
},
{
"type": "limiter",
"id": "n3",
"enabled": false,
"params": { "limit": -1, "attack": 5, "release": 50, "level_out": 0 }
}
]
}
```
- **Order is signal order.** Each node processes what the one before produced.
- `type` is an effect id from the registry. `params` are in the units a person
thinks in — dB, ms, Hz — and out-of-range values are clamped on read, so a
chain that parses is always safe to realise.
- `id` is a stable handle. Automation addresses nodes by id, never by position,
so reordering the chain cannot re-point a lane at a different effect. A node
with no id loads fine but cannot be automated. Writing a chain by hand, any
unique string works; Studio hands out the first free `n1`, `n2`, … so matching
that convention keeps a hand-written chain and an edited one looking alike.
- `enabled: false` is bypass — the node stays in the chain, out of the signal
path. Absent means enabled.
- `fromCarve: true` marks a node the carve analysis generated. Re-running the
carve replaces exactly these and leaves hand-built effects alone. **Do not set
it by hand**: a node tagged this way will be deleted by the next carve.
## `data-automation` — the envelopes
```json
{
"version": 1,
"lanes": [
{
"target": "volume",
"points": [
{ "t": 0, "v": 1 },
{ "t": 2.5, "v": 0.4 }
]
},
{
"target": "fx.n2.gain",
"points": [
{ "t": 0, "v": 0 },
{ "t": 1, "v": -6, "curve": 0.4 }
]
}
]
}
```
- `target` is `volume` for the track's own level, or `fx.<nodeId>.<param>`.
- `t` is **seconds from the start of the clip**, not of the composition. A bed
starting at `data-start="8"` has `t: 0` at composition time 8.
- `v` is in the parameter's own unit: dB for a gain, Hz for a frequency, 0..1 for
volume.
- A lane holds its first value backwards to the start of its clip and its last
value forward to the end. So a bed that begins before the voice needs an
explicit "no cut" point at `t: 0`, or it starts out already ducked.
- `curve` (-1..1) bends the segment _leaving_ a point: positive holds low then
rises late. `viaX`/`viaY` name an interior point the segment passes through
(progress 0..1, value travelled 0..1) and supersede `curve` when both are
present — that is what the timeline writes when a bend is dragged.
- 512 points per lane, maximum.
- A lane whose node is gone is pruned on read rather than erroring.
**A lane on a non-automatable parameter is silently inert.** Automation is
delivered as native `AudioParam` scheduling, so a knob that no `AudioParam` backs
cannot move: worklet processor options, a WaveShaper curve and a convolution
impulse are all set wholesale. `fx-registry.md` marks each parameter; the four
worklet effects (`compressor`, `limiter`, `gate`, `bitcrush`) have none at all.
## `data-fx-carve` — the carve's settings
```json
{ "source": "narration", "strength": 0.35, "dynamic": true }
```
- `source` is the **element id of the voice track to listen to**. It lives on the
bed being processed, not on the voice.
- `strength` 0..1 derives the whole mechanism (see `carveProfile`).
- `dynamic` follows the voice moment to moment instead of holding one depth.
This attribute is not read at playback — the chain and lanes it produced are what
play. It exists so the settings can be read back and re-derived rather than
guessed from the filters, which is what makes changing strength on an existing
carve possible.
Older projects may carry the six mechanism numbers (`maxCutDb`, `bands`, `q`,
`intelligibilityBias`, `duckDb`, `headroomDb`) instead of `strength`. They still
load: the depth maps back onto a strength and everything else is re-derived.
@@ -0,0 +1,84 @@
# Effect registry
Every effect, its parameters and the usable range of each. Values outside a range
are clamped on read, so anything that parses is safe to realise. **AUTO** marks a
parameter an automation lane can drive; anything unmarked cannot move over time
(see the note at the bottom).
Generated from `HF_AUDIO_FX` in `@hyperframes/core/audio-fx`, which is the source
of truth — if this table and the code disagree, the code is right.
## Filter — which frequencies a track may occupy
| Effect | Parameter |
| ----------- | ----------------------------------------------------------------------------------------------------------- |
| `highpass` | `frequency` 2020000 Hz (300, log) **AUTO** · `q` 0.120 (0.707, log) **AUTO** · `poles` `1`\|`2` (2) |
| `lowpass` | `frequency` 10020000 Hz (8000, log) **AUTO** · `q` 0.120 (0.707, log) **AUTO** · `poles` `1`\|`2` (2) |
| `peaking` | `frequency` 2020000 Hz (1000, log) **AUTO** · `gain` 4040 dB (0) **AUTO** · `q` 0.120 (1, log) **AUTO** |
| `lowshelf` | `frequency` 202000 Hz (200, log) **AUTO** · `gain` 4040 dB (0) **AUTO** |
| `highshelf` | `frequency` 50020000 Hz (4000, log) **AUTO** · `gain` 4040 dB (0) **AUTO** |
`q` is bandwidth — higher is narrower. `poles` is the slope: `2` is the usual
biquad (12 dB/oct), `1` is gentler (6 dB/oct). Shelving filters have no `q`: the
Web Audio spec leaves it unused for them, so a control would have moved nothing.
## Dynamics — how level behaves over time
| Effect | Parameter |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `gain` | `gain` 6012 dB (0) **AUTO** |
| `compressor` | `threshold` 600 dB (24) · `ratio` 120 (4) · `attack` 0.012000 ms (20, log) · `release` 0.019000 ms (250, log) · `knee` 18 (2.83) · `makeup` 036 dB (0) · `mix` 01 (1) |
| `limiter` | `limit` 240 dB (1) · `attack` 0.180 ms (5) · `release` 18000 ms (50, log) · `level_out` 2424 dB (0) |
| `gate` | `threshold` 800 dB (35) · `range` 800 dB (24) · `ratio` 120 (10) · `attack` 0.019000 ms (1, log) · `release` 0.019000 ms (100, log) · `knee` 18 (2.83) |
Cuts on `gain` go to 60 dB, boosts stop at +12: it is a level stage for making
room, and a chain that could add 40 dB would clip long before that was useful.
`knee` of 1 is a hard corner, higher eases into it. `mix` below 1 blends the dry
signal back in (parallel compression). `range` is how far down the gate pulls
when closed — a gate that pulls all the way to silence sounds like a switch.
## Nonlinear — changes the waveform's shape
| Effect | Parameter |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `saturate` | `type` `tanh`\|`atan`\|`cubic`\|`exp`\|`alg`\|`quintic`\|`sin`\|`erf`\|`hard` (tanh) · `threshold` 400 dB (6) · `output` 2424 dB (0) **AUTO** · `oversample` 18× (4) |
| `bitcrush` | `bits` 132 (8) · `samples` 1250× (1) · `mix` 01 (1) |
`tanh` is the gentlest curve and `hard` is outright clipping. Higher `oversample`
costs more CPU and keeps aliasing down. `samples` repeats each sample N times — a
crude downsample, which is where the lo-fi character comes from.
## Time — space and width
| Effect | Parameter |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `delay` | `time` 15000 ms (250, log) **AUTO** · `feedback` 0.010.95 (0.35) **AUTO** · `mix` 01 (0.4) **AUTO** |
| `reverb` | `size` 0.051 (0.7) · `damping` 01 (0.5) · `wet` 01 (0.35) **AUTO** · `dry` 01 (0.7) **AUTO** |
| `chorus` | `delay` 1100 ms (7) **AUTO** · `depth` 010 ms (2) **AUTO** · `speed` 0.0110 Hz (1) **AUTO** · `mix` 01 (0.5) **AUTO** |
| `phaser` | `in_gain` 01 (0.4) **AUTO** · `out_gain` 02 (0.74) **AUTO** · `delay` 0.15 ms (3) · `decay` 00.99 (0.4) · `speed` 0.12 Hz (0.5) **AUTO** · `type` `0`\|`1` (0) |
Reverb convolves a _generated_ impulse, and both preview and render generate the
same one — so a room is reproducible without shipping an impulse file. Higher
`damping` rolls the top off the tail faster, which is what makes a large room
sound like a soft one. `feedback` near the top of its range is a very long tail;
it is bounded below 1 because at 1 it never decays.
## Why some parameters cannot be automated
Automation is handed to the audio thread once, as native `AudioParam` ramps and
curves, which is what keeps it sample-accurate and identical between preview and
render. A parameter can therefore only be automated if an `AudioParam` backs it.
Three kinds do not:
- **worklet processor options**`compressor`, `limiter`, `gate` and `bitcrush`
are AudioWorklets configured wholesale, so **none of their parameters are
automatable at all**.
- **a WaveShaper curve**`saturate`'s `type`, `threshold` and `oversample`
rebuild the curve; only its `output` stage is a real param.
- **a convolution impulse**`reverb`'s `size` and `damping` regenerate the
impulse; `wet`/`dry` are gain stages and automate fine.
To make one of those behave differently over time, automate a `gain` stage
around it instead: a lane on a `gain` before a compressor changes how hard the
compressor is driven, which is most of what automating its threshold would have
done.
+418
View File
@@ -0,0 +1,418 @@
#!/usr/bin/env node
/**
* Apply a voiceover carve to a composition, from the command line.
*
* The carve is an analysis: it listens to a voice track, finds the bands it
* occupies, and writes a chain of dips into the music bed plus a level match. In
* Studio a panel runs it. This is the same analysis for an agent that has no
* panel to click identical functions from `@hyperframes/core`, identical
* output, so a composition carved here and one carved in Studio are the same
* three attributes.
*
* node carve.mjs --comp index.html
* node carve.mjs --comp index.html --bed music-bed --voice narration \
* --strength 0.45
*
* With no --bed/--voice it works out the pair itself, and refuses rather than
* guessing when it cannot tell them apart. Dynamic by default, because a bed
* thinned through every pause is worse than one that follows the voice.
*
* Needs `ffmpeg` on PATH (to decode the audio) and `@hyperframes/core` resolvable
* from the composition's project (`npm i -D @hyperframes/core`) the CLI bundles
* core inline rather than shipping it as a package, so it cannot be borrowed from
* there.
*/
import { execFileSync } from "node:child_process";
import { createRequire } from "node:module";
import { readFileSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { pathToFileURL } from "node:url";
/** Sample rate the analysis runs at. Matches Studio's own decode rate, so the
* bands and envelopes come out the same either way. */
const SAMPLE_RATE = 48000;
const usage = `carve.mjs --comp <file.html> [--bed <elementId>] [--voice <elementId>]
[--strength 0..1] [--static] [--dry-run] [--core <dir>]
--bed id of the music track that gets carved (detected if omitted)
--voice id of the voice track to listen to (detected if omitted)
--strength how hard to carve, 0..1 (default 0.25)
--static hold one depth instead of following the voice (default: follow)
--dry-run report what it would write, touch nothing
--core directory to resolve @hyperframes/core from (default: the comp's)`;
function parseArgs(argv) {
// Dynamic unless told otherwise: a static carve thins the bed through every
// pause, which is the wrong default for anything with gaps in the narration.
const args = { strength: 0.25, dynamic: true, dryRun: false };
for (let i = 0; i < argv.length; i += 1) {
const flag = argv[i];
const next = () => {
const value = argv[i + 1];
if (value === undefined) fail(`${flag} needs a value`);
i += 1;
return value;
};
if (flag === "--comp") args.comp = next();
else if (flag === "--bed") args.bed = next();
else if (flag === "--voice") args.voice = next();
else if (flag === "--strength") args.strength = Number(next());
else if (flag === "--core") args.core = next();
else if (flag === "--dynamic") args.dynamic = true;
else if (flag === "--static") args.dynamic = false;
else if (flag === "--dry-run") args.dryRun = true;
else if (flag === "-h" || flag === "--help") fail(usage, 0);
else fail(`unknown flag: ${flag}\n\n${usage}`);
}
if (!args.comp) fail(`--comp is required\n\n${usage}`);
if (!Number.isFinite(args.strength) || args.strength < 0 || args.strength > 1) {
fail("--strength must be a number from 0 to 1");
}
return args;
}
function fail(message, code = 1) {
process.stderr.write(`${message}\n`);
process.exit(code);
}
/**
* Load the carve analysis out of `@hyperframes/core`.
*
* Resolved from the project rather than from this script, which lives wherever
* the skill was installed a sibling of the composition is what has the
* dependency.
*/
async function loadCore(fromDir) {
const require = createRequire(pathToFileURL(resolve(fromDir, "package.json")));
const load = (subpath) => {
const file = require.resolve(`@hyperframes/core/${subpath}`);
return import(pathToFileURL(file).href);
};
try {
return {
carve: await load("audio-carve"),
fx: await load("audio-fx"),
};
} catch (error) {
fail(
`cannot resolve @hyperframes/core from ${fromDir}\n` +
` install or update it: npm i -D @hyperframes/core\n` +
` (the audio-carve export needs a version that ships the carve analysis)\n` +
` or point at one: --core <dir containing node_modules/@hyperframes/core>\n` +
` (${error.message})`,
);
}
}
/** Mono float PCM for one media file, via ffmpeg. */
function decode(path) {
let raw;
try {
raw = execFileSync(
"ffmpeg",
[
"-v",
"error",
"-i",
path,
"-vn",
"-ac",
"1",
"-ar",
String(SAMPLE_RATE),
"-f",
"f32le",
"-",
],
{ maxBuffer: 1 << 30 },
);
} catch (error) {
fail(`could not decode ${path}\n ${error.message.split("\n")[0]}`);
}
if (raw.length === 0) fail(`no audio in ${path}`);
return new Float32Array(raw.buffer, raw.byteOffset, raw.length / 4);
}
const attrOf = (tag, name) => tag.match(new RegExp(`\\s${name}="([^"]*)"`, "i"))?.[1] ?? null;
const unescapeAttr = (value) =>
value
.replace(/&quot;/g, '"')
.replace(/&#39;/g, "'")
.replace(/&amp;/g, "&");
const escapeAttr = (value) => value.replace(/&/g, "&amp;").replace(/"/g, "&quot;");
/** Every media element with a src, as {id, tag, kind}. */
function mediaElements(html) {
const found = [];
for (const match of html.matchAll(/<(audio|video)\b[^>]*>/gi)) {
const tag = match[0];
// `\sid=` and not `id=`: `data-hf-id` would match first.
const id = tag.match(/\sid="([^"]+)"/)?.[1];
if (id && attrOf(tag, "src")) found.push({ id, tag, kind: match[1].toLowerCase() });
}
return found;
}
const MUSIC_NAME = /music|bgm|\bbed\b|soundtrack|score|song|theme/i;
const VOICE_NAME = /voice|\bvo\b|narrat|speech|dialog|vocal|\btts\b|announce/i;
const NOT_A_BED = /sfx|foley|whoosh|impact|riser|stinger|swoosh|click|ding|boom/i;
/**
* Fraction of the track that is more than 25 dB below its own peak.
*
* The one cheap thing that tells speech from music without recognising either:
* a voice stops between phrases and a music bed does not. Measured in 50 ms
* frames, which is short enough to see the gap between words and long enough not
* to see the gap between two cycles of a low note.
*/
function silenceFraction(samples) {
const frame = Math.floor(SAMPLE_RATE * 0.05);
const levels = [];
for (let i = 0; i + frame <= samples.length; i += frame) {
let sum = 0;
for (let k = i; k < i + frame; k += 1) sum += samples[k] * samples[k];
levels.push(Math.sqrt(sum / frame));
}
if (levels.length === 0) return 0;
const floor = Math.max(...levels) * 10 ** (-25 / 20);
return levels.filter((level) => level < floor).length / levels.length;
}
/**
* Work out which track is the bed and which is the voice.
*
* Names first, because they are what the author already told us and the answer is
* explainable. Only when a name does not decide does it listen to the audio, and
* it refuses rather than picking when the measurement is close carving the wrong
* track is a silent, confusing failure, and asking is cheap.
*/
function detectPair(html, compDir, given) {
const all = mediaElements(html);
const pick = (id, what) => {
const found = all.find((el) => el.id === id);
if (!found) fail(`no <audio>/<video> with id="${id}" in the composition`);
return { ...found, why: `--${what}` };
};
let bed = given.bed ? pick(given.bed, "bed") : null;
let voice = given.voice ? pick(given.voice, "voice") : null;
if (bed && voice) return { bed, voice };
// A voiceover and a bed are both normally <audio>; a <video> is only considered
// when there is no audio element left to be the voice, which is what a talking
// head recut looks like.
const free = all.filter((el) => el.id !== bed?.id && el.id !== voice?.id);
const candidates = free.filter((el) => !NOT_A_BED.test(`${el.id} ${attrOf(el.tag, "src")}`));
const named = (re) => candidates.filter((el) => re.test(`${el.id} ${attrOf(el.tag, "src")}`));
if (!bed) {
const byName = named(MUSIC_NAME).filter((el) => el.id !== voice?.id);
if (byName.length === 1) bed = { ...byName[0], why: "name looks like music" };
else if (byName.length > 1) {
fail(
`several tracks look like music (${byName.map((el) => el.id).join(", ")}) — name one with --bed`,
);
}
}
if (!voice) {
const byName = named(VOICE_NAME).filter((el) => el.id !== bed?.id);
if (byName.length === 1) voice = { ...byName[0], why: "name looks like a voice" };
else if (byName.length > 1) {
fail(
`several tracks look like a voice (${byName.map((el) => el.id).join(", ")}) — name one with --voice`,
);
}
}
const left = candidates.filter((el) => el.id !== bed?.id && el.id !== voice?.id);
const audioLeft = left.filter((el) => el.kind === "audio");
const pool = audioLeft.length > 0 ? audioLeft : left;
// One track left and one role open: no measurement can be more certain than
// that, and decoding to confirm it would only cost time.
if (!voice && bed && pool.length === 1) voice = { ...pool[0], why: "only track left" };
if (!bed && voice && pool.length === 1) bed = { ...pool[0], why: "only track left" };
if (!voice || !bed) {
if (pool.length < 2) {
fail(
`cannot find a voice and a music track to carve\n` +
` media in the composition: ${all.map((el) => el.id).join(", ") || "none"}\n` +
` name them with --bed and --voice`,
);
}
// Listen: the one that stops between phrases is the voice.
const scored = pool
.map((el) => ({
el,
pauses: silenceFraction(decode(resolve(compDir, unescapeAttr(attrOf(el.tag, "src"))))),
}))
.sort((a, b) => b.pauses - a.pauses);
const [first, second] = scored;
if (first.pauses - second.pauses < 0.08) {
fail(
`cannot tell the voice from the music by ear either — ` +
`${scored.map((s) => `${s.el.id} ${(s.pauses * 100).toFixed(0)}% quiet`).join(", ")}\n` +
` name them with --bed and --voice`,
);
}
const why = (s) => `${(s.pauses * 100).toFixed(0)}% of it is quiet`;
if (!voice) voice = { ...first.el, why: why(first) };
if (!bed) bed = { ...second.el, why: why(second) };
}
if (bed.id === voice.id) fail(`--bed and --voice are the same track ("${bed.id}")`);
return { bed, voice };
}
const startOf = (tag) => {
const raw = Number(attrOf(tag, "data-start"));
return Number.isFinite(raw) ? raw : 0;
};
async function main() {
const args = parseArgs(process.argv.slice(2));
const compPath = resolve(args.comp);
const compDir = dirname(compPath);
const { carve: carveApi, fx: fxApi } = await loadCore(args.core ? resolve(args.core) : compDir);
const html = readFileSync(compPath, "utf-8");
const { bed: bedEl, voice: voiceEl } = detectPair(html, compDir, args);
const bedTag = bedEl.tag;
const voiceTag = voiceEl.tag;
const bedSrc = attrOf(bedTag, "src");
const voiceSrc = attrOf(voiceTag, "src");
process.stdout.write(
`bed ${bedEl.id} (${bedEl.why})\nvoice ${voiceEl.id} (${voiceEl.why})\n`,
);
const profile = carveApi.carveProfile(args.strength);
const voice = decode(resolve(compDir, unescapeAttr(voiceSrc)));
const bands = carveApi.analyseCarveBands(voice, SAMPLE_RATE, profile);
// The level half of the carve needs both tracks: "how far over the voice is
// this bed" cannot be answered by listening to one of them. Times come back on
// the voice's clock, so the gap between the two clips' starts aligns them.
const offset = startOf(voiceTag) - startOf(bedTag);
const bed = profile.duckDb > 0 ? decode(resolve(compDir, unescapeAttr(bedSrc))) : null;
const duck = bed ? carveApi.analyseCarveDuck(voice, bed, SAMPLE_RATE, profile, offset) : [];
// Anything the author built by hand survives a carve; only the previous
// carve's own nodes are replaced. That is what `fromCarve` is for.
const existingChain = attrOf(bedTag, "data-fx-chain");
const existingNodes = existingChain
? fxApi.parseAudioFxChain(unescapeAttr(existingChain)).nodes
: [];
const kept = existingNodes.filter((n) => !n.fromCarve);
// Lanes belonging to the carve being replaced, addressed by the ids the OLD
// nodes had. Taken before anything is minted: those ids are freed by the
// replacement and a new node can be handed one of them, so reading them off the
// new chain would keep exactly the stale lanes it is supposed to drop.
const stalePrefixes = existingNodes.filter((n) => n.fromCarve && n.id).map((n) => `fx.${n.id}.`);
let claimed = { version: 1, nodes: kept };
const mint = (node) => {
const withId = { ...node, id: fxApi.mintAudioFxNodeId(claimed), fromCarve: true };
claimed = { version: 1, nodes: [...claimed.nodes, withId] };
return withId;
};
const bandNodes = bands.map((band) => mint(carveApi.carveBandsToChain([band]).nodes[0]));
// A static carve holds one value, so its level match is the duck the voice
// needs while it is actually speaking — the median, which ignores both the
// pauses and the single loudest bar.
const speaking = duck.filter((p) => p.v < 0).map((p) => p.v);
const staticDuckDb = speaking.length
? (speaking.sort((a, b) => a - b)[Math.floor(speaking.length / 2)] ?? 0)
: 0;
const duckNode =
duck.length > 0
? mint({
type: "gain",
enabled: true,
params: {
...fxApi.defaultAudioFxParams("gain"),
gain: args.dynamic ? 0 : staticDuckDb,
},
})
: null;
const chain = {
version: 1,
nodes: [...bandNodes, ...(duckNode ? [duckNode] : []), ...kept],
};
/**
* One carve envelope as a lane on the BED's clock.
*
* A lane holds its first value backwards to the start of its own clip, so a bed
* that begins before the voice needs an explicit "no cut" at zero or it starts
* out ducked.
*/
const laneFor = (id, points) => {
const shifted = points
.map((p) => ({ t: Number((p.t + offset).toFixed(3)), v: p.v }))
.filter((p) => p.t >= 0);
if ((shifted[0]?.t ?? 0) > 0) shifted.unshift({ t: 0, v: 0 });
return shifted.length > 1 ? [{ target: `fx.${id}.gain`, points: shifted }] : [];
};
const carvedLanes = args.dynamic
? [
...carveApi
.analyseCarveDynamics(voice, SAMPLE_RATE, bands)
.flatMap((dyn, i) => (bandNodes[i]?.id ? laneFor(bandNodes[i].id, dyn.points) : [])),
...(duckNode?.id && duck.length > 0 ? laneFor(duckNode.id, duck) : []),
]
: [];
// Hand-drawn lanes are kept the same way hand-built nodes are: by dropping only
// the ones that addressed the previous carve's nodes.
const existingAutomation = attrOf(bedTag, "data-automation");
const carriedLanes = existingAutomation
? (JSON.parse(unescapeAttr(existingAutomation)).lanes ?? []).filter(
(lane) => !stalePrefixes.some((prefix) => String(lane.target).startsWith(prefix)),
)
: [];
const lanes = [...carriedLanes, ...carvedLanes];
const settings = { source: voiceEl.id, strength: args.strength, dynamic: args.dynamic };
const written =
` data-fx-carve="${escapeAttr(JSON.stringify(settings))}"` +
` data-fx-chain="${escapeAttr(fxApi.serializeAudioFxChain(chain))}"` +
(lanes.length > 0
? ` data-automation="${escapeAttr(JSON.stringify({ version: 1, lanes }))}"`
: "");
process.stdout.write(
`carve strength ${args.strength}${args.dynamic ? " dynamic" : " static"}\n` +
`bands ${bands.map((b) => `${b.freq}Hz ${b.gainDb}dB q${b.q}`).join(", ")}\n` +
`level ${
duckNode
? args.dynamic
? `${duck.length}-point envelope, floor ${Math.min(...duck.map((p) => p.v))} dB`
: `${staticDuckDb.toFixed(1)} dB held`
: "no level match at this strength"
}\n` +
`lanes ${carvedLanes.length} carve${carriedLanes.length ? ` + ${carriedLanes.length} kept` : ""}\n`,
);
if (args.dryRun) {
process.stdout.write("dry run: nothing written\n");
return;
}
let stripped = bedTag;
for (const attr of ["data-fx-carve", "data-fx-chain", "data-automation"]) {
stripped = stripped.replace(new RegExp(`\\s${attr}="[^"]*"`, "i"), "");
}
// Inserted before the tag's own closing ">", which is the only place they can
// go: `stripped` is the opening tag alone, so appending would land outside it.
const nextTag = stripped.replace(/\/?>$/, (close) => `${written}${close}`);
if (nextTag === stripped) fail("attribute write produced no change — refusing to save");
writeFileSync(compPath, html.replace(bedTag, nextTag));
process.stdout.write(`wrote ${args.comp} (id="${bedEl.id}")\n`);
}
await main();
+1
View File
@@ -92,6 +92,7 @@ Use the bare name without `/`. If the command fails, surface the error; do not r
| Seek-safe GSAP, CSS, Anime.js, WAAPI, FLIP, paths, masks, SVG, 3D keyframes, or `hyperframes keyframes` diagnostics | `/hyperframes-keyframes` |
| Design specs, concept, palette, typography, narration, beat planning | `/hyperframes-creative` |
| Images, icons, logos, audio, captions, grades, LUTs, reusable media | `/media-use` |
| Voiceover carve, audio effect chains, or automation envelopes on a track | `/hyperframes-audio` |
| Init, lint, check, snapshots, compare, batch render, Studio, render, publish, or diagnostics | `/hyperframes-cli` |
| Registry blocks and components | `/hyperframes-registry` |
| Figma assets, tokens, components, or storyboard frames as reconstructed motion | `/figma` |