ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-11-14 21:29:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(skills): carve every voice from the command line too

Browse Source 
`--voice` repeats, and with none given the script takes the bed by name and every
non-SFX track playing over it. Audio elements are preferred: video counts only when
no audio track is left to be the voice, or every B-roll clip in a composition reads
as somebody talking — which is exactly what happened on the first run against a real
project.

The by-ear tie-break is gone with the single-voice model that needed it: it existed
to choose one voice, and there is nothing to choose now. `--static` is gone too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vance Ingalls
2026-08-07 16:56:58 -07:00
co-authored by Claude Opus 5
parent 56d6e1fc79
commit 45eb650d13
3 changed files with 157 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/hyperframes-audio/SKILL.md
+32 -25
View File
@@ -150,10 +150,17 @@ takes only those, and the bed keeps its low end and its top, so it is still musi
while the voice is still intelligible.
**It is a relationship, not an effect.** The settings live on the *bed* — the
track that gets processed — and they name the voice to listen to, exactly as a
track that gets processed — and they name the voices to listen to, exactly as a
sidechain compressor does: you select the track that gets quieter and pick what
makes it quieter. **Never put a carve on the voice track.** A voice carved
against itself is a bug, not a subtle mix choice.
makes it quieter. **Never put a carve on a voice track.** A voice carved against
itself is a bug, not a subtle mix choice.
**Every voice, not one of them.** `sources` is a list, because a bed usually runs
under a whole sequence — a narrator, an interview answer, a second presenter. They
are summed onto the bed's own clock before anything is measured (`mixCarveSources`),
so one analysis covers all of them: the bands come from all the speech there is, and
the envelopes rise wherever any of it is happening. Voices that never play while the
bed does are left out; they cannot mask it.
**One knob.** `strength` is 0..1 and derives everything: how deep to cut, how
many bands, how wide, how far to favour intelligibility over raw voice energy,
@@ -172,13 +179,11 @@ polish step to get to if there is time. Place both tracks, run the command below
listen. Skip it only when there is no narration for the music to sit under — a
music video, a title card, a montage cut to the track.
**Static or dynamic — dynamic unless you know otherwise.** A static carve holds
its cuts for the whole clip, including every pause, so the bed is thinned where
there is nothing to make room for. Dynamic turns every value into an envelope of
the voice's own level: silence leaves the bed alone, a loud passage pushes the
carve to full depth. That is what almost every voiceover wants, so it is the
default. Reach for `--static` only for wall-to-wall narration with no real gaps,
where an envelope is hundreds of breakpoints describing a constant.
**It always follows the voice.** There is no static mode: a fixed depth thins the
bed through every pause, and once you have heard both there is no reason to want it.
Every value becomes an envelope of the speech's own level — silence leaves the bed
alone, a loud passage pushes the carve to full depth — written as ordinary automation,
which is why the lanes show up in the timeline and can be edited afterwards.
**Level matching is part of it.** Spectral carving cannot fix a bed that is
simply louder than the voice. So the carve also measures how far over the voice
@@ -187,8 +192,13 @@ driven by an envelope for a dynamic one. That envelope releases slowly on
purpose — music that snaps back to full the instant a word ends sounds like a
machine doing it.
**Running it.** In Studio: pick the voice in the bed's Voiceover carve control;
turning it on adds the modules and strength adjusts what is there. Headless —
**Running it.** In Studio the carve is one module at the top of a track's effect
rack — voice, strength, dynamic, and the analysis it produced, in one card. It is
there whenever another track could be the voice, and a bed with exactly **one**
candidate above it is carved by default, dynamically, at the default strength:
that is what a bed under narration wants, and the module is where you change or
switch it off. Several candidates leaves the picker waiting rather than guessing.
Headless —
which is the path when you are authoring a composition rather than editing one:
```bash
@@ -206,20 +216,17 @@ bands 400Hz -6dB q1.4, 1000Hz -3dB q1.4, 1600Hz -3.17dB q1.4
level 216-point envelope, floor -6 dB
```
Name the pair with `--bed` / `--voice` when the composition has several plausible
tracks, `--strength` to push it, `--static` to hold one depth, `--dry-run` to see
that report and write nothing.
Name the tracks with `--bed` / `--voice` (repeatable) when the automatic choice is
wrong, `--strength` to push it, `--dry-run` to see that report and write nothing.
**How it picks the pair.** Names first, because that is what you already told it
and the answer is explainable: a track whose id or filename looks like music
(`music`, `bgm`, `bed`, `score`…) is the bed, one that looks like a voice
(`voice`, `vo`, `narration`, `speech`…) is the voice, and SFX-shaped names are not
candidates for either. If one role is filled and a single track is left, that
track takes the other role. Only when names decide nothing does it listen: it
measures how much of each track is quiet, and the one that stops between phrases
is the voice. **When two tracks are too close to call it refuses and asks you to
name them** rather than carving the wrong one — a bed carved against a bed is
silent and confusing, and typing two ids is cheap.
**How it picks the tracks.** Names first, because that is what you already told it
and the answer is explainable — `classifyAudioName` in core, the same classifier
Studio's own picker uses, so the two cannot disagree. A track whose id or filename
looks like music (`music`, `bgm`, `bed`, `score`…) is the bed; everything else that
plays over it and is not SFX-shaped is a voice. Audio elements are preferred: video
counts only when no audio track is left to be the voice, or every B-roll clip in the
composition would read as somebody talking. **It refuses when it cannot tell which
track is the bed** rather than carving the wrong one — typing one id is cheap.
Same analysis functions as the panel, so the result is identical. Needs `ffmpeg`
on PATH and `@hyperframes/core` installed in the project (`npm i -D
skills/hyperframes-audio/references/attributes.md
+12 -5
View File
@@ -70,13 +70,18 @@ worklet effects (`compressor`, `limiter`, `gate`, `bitcrush`) have none at all.
## `data-fx-carve` — the carve's settings
```json
{ "source": "narration", "strength": 0.35, "dynamic": true }
{ "enabled": true, "sources": ["narration", "interview-guest"], "strength": 0.35 }
```
- `source` is the **element id of the voice track to listen to**. It lives on the
bed being processed, not on the voice.
- `sources` are the **element ids of every voice this bed makes room for**. They live
on the bed being processed, not on the voices. Summed onto the bed's clock before
the analysis, so one set of filters and envelopes covers all of them.
- `strength` 0..1 derives the whole mechanism (see `carveProfile`).
- `dynamic` follows the voice moment to moment instead of holding one depth.
- There is no `dynamic`: a carve always follows the speech.
- `enabled` is whether the carve applies. It exists because a bed with exactly one
candidate voice is carved by default: with "off" represented by an absent
attribute, switching it off would read as never-configured and the default would
put it back. `enabled: false` keeps the settings and stops the carve.
This attribute is not read at playback — the chain and lanes it produced are what
play. It exists so the settings can be read back and re-derived rather than
@@ -85,4 +90,6 @@ carve possible.
Older projects may carry the six mechanism numbers (`maxCutDb`, `bands`, `q`,
`intelligibilityBias`, `duckDb`, `headroomDb`) instead of `strength`. They still
load: the depth maps back onto a strength and everything else is re-derived.
load: the depth maps back onto a strength and everything else is re-derived. A
stored carve with no `enabled` reads as on, and a single `source` reads as a one-voice
`sources` list. A stored `dynamic` is ignored — every carve follows the speech now.
skills/hyperframes-audio/scripts/carve.mjs
+113 -139
View File
@@ -11,11 +11,12 @@
*
* node carve.mjs --comp index.html
* node carve.mjs --comp index.html --bed music-bed --voice narration \
* --strength 0.45
* --voice interview-guest --strength 0.45
*
* With no --bed/--voice it works out the pair itself, and refuses rather than
* guessing when it cannot tell them apart. Dynamic by default, because a bed
* thinned through every pause is worse than one that follows the voice.
* With no --bed/--voice it works out the tracks itself: the bed, and every voice
* playing over it. `--voice` may be repeated to name them instead. Every named
* voice is analysed together, so a bed running under a narrator and an answer makes
* room for both.
*
* Needs `ffmpeg` on PATH (to decode the audio) and `@hyperframes/core` resolvable
* from the composition's project (`npm i -D @hyperframes/core`) — the CLI bundles
@@ -33,20 +34,17 @@ import { pathToFileURL } from "node:url";
* bands and envelopes come out the same either way. */
const SAMPLE_RATE = 48000;
const usage = `carve.mjs --comp <file.html> [--bed <elementId>] [--voice <elementId>]
[--strength 0..1] [--static] [--dry-run] [--core <dir>]
const usage = `carve.mjs --comp <file.html> [--bed <elementId>] [--voice <elementId> ...]
[--strength 0..1] [--dry-run] [--core <dir>]
--bed id of the music track that gets carved (detected if omitted)
--voice id of the voice track to listen to (detected if omitted)
--voice id of a voice to make room for; repeatable (detected if omitted)
--strength how hard to carve, 0..1 (default 0.25)
--static hold one depth instead of following the voice (default: follow)
--dry-run report what it would write, touch nothing
--core directory to resolve @hyperframes/core from (default: the comp's)`;
function parseArgs(argv) {
// Dynamic unless told otherwise: a static carve thins the bed through every
// pause, which is the wrong default for anything with gaps in the narration.
const args = { strength: 0.25, dynamic: true, dryRun: false };
const args = { strength: 0.25, dryRun: false, voices: [] };
for (let i = 0; i < argv.length; i += 1) {
const flag = argv[i];
const next = () => {
@@ -57,11 +55,9 @@ function parseArgs(argv) {
};
if (flag === "--comp") args.comp = next();
else if (flag === "--bed") args.bed = next();
else if (flag === "--voice") args.voice = next();
else if (flag === "--voice") args.voices.push(next());
else if (flag === "--strength") args.strength = Number(next());
else if (flag === "--core") args.core = next();
else if (flag === "--dynamic") args.dynamic = true;
else if (flag === "--static") args.dynamic = false;
else if (flag === "--dry-run") args.dryRun = true;
else if (flag === "-h" || flag === "--help") fail(usage, 0);
else fail(`unknown flag: ${flag}\n\n${usage}`);
@@ -158,114 +154,87 @@ function mediaElements(html) {
return found;
}
const MUSIC_NAME = /music|bgm|\bbed\b|soundtrack|score|song|theme/i;
const VOICE_NAME = /voice|\bvo\b|narrat|speech|dialog|vocal|\btts\b|announce/i;
const NOT_A_BED = /sfx|foley|whoosh|impact|riser|stinger|swoosh|click|ding|boom/i;
/**
* Fraction of the track that is more than 25 dB below its own peak.
*
* The one cheap thing that tells speech from music without recognising either:
* a voice stops between phrases and a music bed does not. Measured in 50 ms
* frames, which is short enough to see the gap between words and long enough not
* to see the gap between two cycles of a low note.
*/
function silenceFraction(samples) {
const frame = Math.floor(SAMPLE_RATE * 0.05);
const levels = [];
for (let i = 0; i + frame <= samples.length; i += frame) {
let sum = 0;
for (let k = i; k < i + frame; k += 1) sum += samples[k] * samples[k];
levels.push(Math.sqrt(sum / frame));
}
if (levels.length === 0) return 0;
const floor = Math.max(...levels) * 10 ** (-25 / 20);
return levels.filter((level) => level < floor).length / levels.length;
}
/**
* Work out which track is the bed and which is the voice.
* Work out which track is the bed and which tracks are its voices.
*
* Names first, because they are what the author already told us and the answer is
* explainable. Only when a name does not decide does it listen to the audio, and
* it refuses rather than picking when the measurement is close — carving the wrong
* track is a silent, confusing failure, and asking is cheap.
* explainable: a track whose id or filename looks like music is the bed, ones that
* look like speech are voices, SFX-shaped names are neither. `classifyAudioName`
* comes from core so Studio's own picker and this cannot disagree.
*
* EVERY voice over the bed, not one of them. A bed usually runs under a whole
* sequence, and they are analysed together — so there is nothing to disambiguate,
* which is why this no longer refuses when several tracks look like speech.
*
* Only tracks that actually play while the bed does: one somewhere else on the
* timeline cannot mask it. It still refuses when it cannot find a bed at all, or
* finds no voice to make room for.
*/
function detectPair(html, compDir, given) {
function detectTracks(html, given, classify, overlaps) {
const all = mediaElements(html);
const kindOf = (el) => classify(el.id, unescapeAttr(attrOf(el.tag, "src") ?? ""));
const spanOf = (el) => {
const raw = attrOf(el.tag, "data-duration");
const n = raw === null ? Number.NaN : Number(raw);
return {
start: startOf(el.tag),
duration: Number.isFinite(n) ? n : null,
};
};
const pick = (id, what) => {
const found = all.find((el) => el.id === id);
if (!found) fail(`no <audio>/<video> with id="${id}" in the composition`);
return { ...found, why: `--${what}` };
};
let bed = given.bed ? pick(given.bed, "bed") : null;
let voice = given.voice ? pick(given.voice, "voice") : null;
if (bed && voice) return { bed, voice };
// A voiceover and a bed are both normally <audio>; a <video> is only considered
// when there is no audio element left to be the voice, which is what a talking
// head recut looks like.
const free = all.filter((el) => el.id !== bed?.id && el.id !== voice?.id);
const candidates = free.filter((el) => !NOT_A_BED.test(`${el.id} ${attrOf(el.tag, "src")}`));
const named = (re) => candidates.filter((el) => re.test(`${el.id} ${attrOf(el.tag, "src")}`));
let bed = given.bed ? pick(given.bed, "bed") : null;
const named = given.voices.map((id) => pick(id, "voice"));
if (!bed) {
const byName = named(MUSIC_NAME).filter((el) => el.id !== voice?.id);
if (byName.length === 1) bed = { ...byName[0], why: "name looks like music" };
else if (byName.length > 1) {
const others = all.filter((el) => !named.some((v) => v.id === el.id));
const music = others.filter((el) => kindOf(el) === "music");
if (music.length === 1) bed = { ...music[0], why: "name looks like music" };
else if (music.length > 1) {
fail(
`several tracks look like music (${byName.map((el) => el.id).join(", ")}) — name one with --bed`,
`several tracks look like music (${music.map((el) => el.id).join(", ")}) — name one with --bed`,
);
}
}
if (!voice) {
const byName = named(VOICE_NAME).filter((el) => el.id !== bed?.id);
if (byName.length === 1) voice = { ...byName[0], why: "name looks like a voice" };
else if (byName.length > 1) {
} else if (others.length === 1) {
bed = { ...others[0], why: "only track left" };
} else {
fail(
`several tracks look like a voice (${byName.map((el) => el.id).join(", ")}) — name one with --voice`,
);
}
}
const left = candidates.filter((el) => el.id !== bed?.id && el.id !== voice?.id);
const audioLeft = left.filter((el) => el.kind === "audio");
const pool = audioLeft.length > 0 ? audioLeft : left;
// One track left and one role open: no measurement can be more certain than
// that, and decoding to confirm it would only cost time.
if (!voice && bed && pool.length === 1) voice = { ...pool[0], why: "only track left" };
if (!bed && voice && pool.length === 1) bed = { ...pool[0], why: "only track left" };
if (!voice || !bed) {
if (pool.length < 2) {
fail(
`cannot find a voice and a music track to carve\n` +
`cannot tell which track is the music bed\n` +
` media in the composition: ${all.map((el) => el.id).join(", ") || "none"}\n` +
` name them with --bed and --voice`,
` name it with --bed`,
);
}
// Listen: the one that stops between phrases is the voice.
const scored = pool
.map((el) => ({
el,
pauses: silenceFraction(decode(resolve(compDir, unescapeAttr(attrOf(el.tag, "src"))))),
}))
.sort((a, b) => b.pauses - a.pauses);
const [first, second] = scored;
if (first.pauses - second.pauses < 0.08) {
fail(
`cannot tell the voice from the music by ear either — ` +
`${scored.map((s) => `${s.el.id} ${(s.pauses * 100).toFixed(0)}% quiet`).join(", ")}\n` +
` name them with --bed and --voice`,
);
}
const why = (s) => `${(s.pauses * 100).toFixed(0)}% of it is quiet`;
if (!voice) voice = { ...first.el, why: why(first) };
if (!bed) bed = { ...second.el, why: why(second) };
}
if (bed.id === voice.id) fail(`--bed and --voice are the same track ("${bed.id}")`);
return { bed, voice };
const bedSpan = spanOf(bed);
const overlapping = (el) => overlaps(bedSpan, spanOf(el));
const plausible = all
.filter((el) => el.id !== bed.id && kindOf(el) !== "music" && kindOf(el) !== "sfx")
.filter(overlapping);
// A voiceover is normally its own <audio>. Video counts only when no audio track
// is left to be the voice — a talking-head recut — because otherwise every B-roll
// clip in the composition reads as somebody talking.
const spoken = plausible.filter((el) => el.kind === "audio");
const pool = spoken.length > 0 ? spoken : plausible;
const voices = named.length
? named
: pool.map((el) => ({
...el,
why: kindOf(el) === "voice" ? "name looks like a voice" : "plays over the bed",
}));
const usable = voices.filter((el) => attrOf(el.tag, "src"));
if (usable.length === 0) {
fail(
`no voice to make room for on ${bed.id}\n` +
` media in the composition: ${all.map((el) => el.id).join(", ") || "none"}\n` +
` name one with --voice`,
);
}
return { bed, voices: usable };
}
const startOf = (tag) => {
@@ -280,25 +249,39 @@ async function main() {
const { carve: carveApi, fx: fxApi } = await loadCore(args.core ? resolve(args.core) : compDir);
const html = readFileSync(compPath, "utf-8");
const { bed: bedEl, voice: voiceEl } = detectPair(html, compDir, args);
const { bed: bedEl, voices } = detectTracks(
html,
args,
carveApi.classifyAudioName,
carveApi.clipsOverlap,
);
const bedTag = bedEl.tag;
const voiceTag = voiceEl.tag;
const bedSrc = attrOf(bedTag, "src");
const voiceSrc = attrOf(voiceTag, "src");
process.stdout.write(
`bed ${bedEl.id} (${bedEl.why})\nvoice ${voiceEl.id} (${voiceEl.why})\n`,
`bed ${bedEl.id} (${bedEl.why})\n` +
voices.map((v) => `voice ${v.id} (${v.why})`).join("\n") +
"\n",
);
const profile = carveApi.carveProfile(args.strength);
const voice = decode(resolve(compDir, unescapeAttr(voiceSrc)));
// Every voice summed onto the BED's clock before anything is measured. One
// question — where and when is speech masking this bed — with one answer, even
// when the answer comes from several people at different times.
const voice = carveApi.mixCarveSources(
voices.map((v) => ({
samples: decode(resolve(compDir, unescapeAttr(attrOf(v.tag, "src")))),
offsetSeconds: startOf(v.tag) - startOf(bedTag),
})),
SAMPLE_RATE,
);
if (voice.length === 0) fail("the voices do not overlap the bed, so there is nothing to carve");
const bands = carveApi.analyseCarveBands(voice, SAMPLE_RATE, profile);
// The level half of the carve needs both tracks: "how far over the voice is
// this bed" cannot be answered by listening to one of them. Times come back on
// the voice's clock, so the gap between the two clips' starts aligns them.
const offset = startOf(voiceTag) - startOf(bedTag);
// The level half of the carve needs both sides: "how far over the speech is this
// bed" cannot be answered by listening to one of them. No offset — the mix is
// already on the bed's clock.
const bed = profile.duckDb > 0 ? decode(resolve(compDir, unescapeAttr(bedSrc))) : null;
const duck = bed ? carveApi.analyseCarveDuck(voice, bed, SAMPLE_RATE, profile, offset) : [];
const duck = bed ? carveApi.analyseCarveDuck(voice, bed, SAMPLE_RATE, profile, 0) : [];
// Anything the author built by hand survives a carve; only the previous
// carve's own nodes are replaced. That is what `fromCarve` is for.
@@ -321,13 +304,6 @@ async function main() {
};
const bandNodes = bands.map((band) => mint(carveApi.carveBandsToChain([band]).nodes[0]));
// A static carve holds one value, so its level match is the duck the voice
// needs while it is actually speaking — the median, which ignores both the
// pauses and the single loudest bar.
const speaking = duck.filter((p) => p.v < 0).map((p) => p.v);
const staticDuckDb = speaking.length
? (speaking.sort((a, b) => a - b)[Math.floor(speaking.length / 2)] ?? 0)
: 0;
const duckNode =
duck.length > 0
? mint({
@@ -335,7 +311,7 @@ async function main() {
enabled: true,
params: {
...fxApi.defaultAudioFxParams("gain"),
gain: args.dynamic ? 0 : staticDuckDb,
gain: 0,
},
})
: null;
@@ -347,26 +323,26 @@ async function main() {
/**
* One carve envelope as a lane on the BED's clock.
*
* A lane holds its first value backwards to the start of its own clip, so a bed
* that begins before the voice needs an explicit "no cut" at zero or it starts
* Nothing to shift: the voices were summed onto that clock before the analysis
* ran. A lane does hold its first value backwards to the start of its clip, so an
* envelope that begins later needs an explicit "no cut" at zero or the bed starts
* out ducked.
*/
const laneFor = (id, points) => {
const shifted = points
.map((p) => ({ t: Number((p.t + offset).toFixed(3)), v: p.v }))
const timed = points
.map((p) => ({ t: Number(p.t.toFixed(3)), v: p.v }))
.filter((p) => p.t >= 0);
if ((shifted[0]?.t ?? 0) > 0) shifted.unshift({ t: 0, v: 0 });
return shifted.length > 1 ? [{ target: `fx.${id}.gain`, points: shifted }] : [];
if ((timed[0]?.t ?? 0) > 0) timed.unshift({ t: 0, v: 0 });
return timed.length > 1 ? [{ target: `fx.${id}.gain`, points: timed }] : [];
};
const carvedLanes = args.dynamic
? [
...carveApi
.analyseCarveDynamics(voice, SAMPLE_RATE, bands)
.flatMap((dyn, i) => (bandNodes[i]?.id ? laneFor(bandNodes[i].id, dyn.points) : [])),
...(duckNode?.id && duck.length > 0 ? laneFor(duckNode.id, duck) : []),
]
: [];
// Every carve follows the speech: a fixed depth thins the bed through every pause.
const carvedLanes = [
...carveApi
.analyseCarveDynamics(voice, SAMPLE_RATE, bands)
.flatMap((dyn, i) => (bandNodes[i]?.id ? laneFor(bandNodes[i].id, dyn.points) : [])),
...(duckNode?.id && duck.length > 0 ? laneFor(duckNode.id, duck) : []),
];
// Hand-drawn lanes are kept the same way hand-built nodes are: by dropping only
// the ones that addressed the previous carve's nodes.
@@ -378,7 +354,7 @@ async function main() {
: [];
const lanes = [...carriedLanes, ...carvedLanes];
const settings = { source: voiceEl.id, strength: args.strength, dynamic: args.dynamic };
const settings = { enabled: true, sources: voices.map((v) => v.id), strength: args.strength };
const written =
` data-fx-carve="${escapeAttr(JSON.stringify(settings))}"` +
` data-fx-chain="${escapeAttr(fxApi.serializeAudioFxChain(chain))}"` +
@@ -387,13 +363,11 @@ async function main() {
: "");
process.stdout.write(
`carve strength ${args.strength}${args.dynamic ? " dynamic" : " static"}\n` +
`carve strength ${args.strength}, ${voices.length} voice${voices.length === 1 ? "" : "s"}\n` +
`bands ${bands.map((b) => `${b.freq}Hz ${b.gainDb}dB q${b.q}`).join(", ")}\n` +
`level ${
duckNode
? args.dynamic
? `${duck.length}-point envelope, floor ${Math.min(...duck.map((p) => p.v))} dB`
: `${staticDuckDb.toFixed(1)} dB held`
? `${duck.length}-point envelope, floor ${Math.min(...duck.map((p) => p.v))} dB`
: "no level match at this strength"
}\n` +
`lanes ${carvedLanes.length} carve${carriedLanes.length ? ` + ${carriedLanes.length} kept` : ""}\n`,