diff --git a/skills/hyperframes-audio/SKILL.md b/skills/hyperframes-audio/SKILL.md index 9bae288c13..f7c205e70d 100644 --- a/skills/hyperframes-audio/SKILL.md +++ b/skills/hyperframes-audio/SKILL.md @@ -150,10 +150,17 @@ takes only those, and the bed keeps its low end and its top, so it is still musi while the voice is still intelligible. **It is a relationship, not an effect.** The settings live on the *bed* — the -track that gets processed — and they name the voice to listen to, exactly as a +track that gets processed — and they name the voices to listen to, exactly as a sidechain compressor does: you select the track that gets quieter and pick what -makes it quieter. **Never put a carve on the voice track.** A voice carved -against itself is a bug, not a subtle mix choice. +makes it quieter. **Never put a carve on a voice track.** A voice carved against +itself is a bug, not a subtle mix choice. + +**Every voice, not one of them.** `sources` is a list, because a bed usually runs +under a whole sequence — a narrator, an interview answer, a second presenter. They +are summed onto the bed's own clock before anything is measured (`mixCarveSources`), +so one analysis covers all of them: the bands come from all the speech there is, and +the envelopes rise wherever any of it is happening. Voices that never play while the +bed does are left out; they cannot mask it. **One knob.** `strength` is 0..1 and derives everything: how deep to cut, how many bands, how wide, how far to favour intelligibility over raw voice energy, @@ -172,13 +179,11 @@ polish step to get to if there is time. Place both tracks, run the command below listen. Skip it only when there is no narration for the music to sit under — a music video, a title card, a montage cut to the track. -**Static or dynamic — dynamic unless you know otherwise.** A static carve holds -its cuts for the whole clip, including every pause, so the bed is thinned where -there is nothing to make room for. Dynamic turns every value into an envelope of -the voice's own level: silence leaves the bed alone, a loud passage pushes the -carve to full depth. That is what almost every voiceover wants, so it is the -default. Reach for `--static` only for wall-to-wall narration with no real gaps, -where an envelope is hundreds of breakpoints describing a constant. +**It always follows the voice.** There is no static mode: a fixed depth thins the +bed through every pause, and once you have heard both there is no reason to want it. +Every value becomes an envelope of the speech's own level — silence leaves the bed +alone, a loud passage pushes the carve to full depth — written as ordinary automation, +which is why the lanes show up in the timeline and can be edited afterwards. **Level matching is part of it.** Spectral carving cannot fix a bed that is simply louder than the voice. So the carve also measures how far over the voice @@ -187,8 +192,13 @@ driven by an envelope for a dynamic one. That envelope releases slowly on purpose — music that snaps back to full the instant a word ends sounds like a machine doing it. -**Running it.** In Studio: pick the voice in the bed's Voiceover carve control; -turning it on adds the modules and strength adjusts what is there. Headless — +**Running it.** In Studio the carve is one module at the top of a track's effect +rack — voice, strength, dynamic, and the analysis it produced, in one card. It is +there whenever another track could be the voice, and a bed with exactly **one** +candidate above it is carved by default, dynamically, at the default strength: +that is what a bed under narration wants, and the module is where you change or +switch it off. Several candidates leaves the picker waiting rather than guessing. +Headless — which is the path when you are authoring a composition rather than editing one: ```bash @@ -206,20 +216,17 @@ bands 400Hz -6dB q1.4, 1000Hz -3dB q1.4, 1600Hz -3.17dB q1.4 level 216-point envelope, floor -6 dB ``` -Name the pair with `--bed` / `--voice` when the composition has several plausible -tracks, `--strength` to push it, `--static` to hold one depth, `--dry-run` to see -that report and write nothing. +Name the tracks with `--bed` / `--voice` (repeatable) when the automatic choice is +wrong, `--strength` to push it, `--dry-run` to see that report and write nothing. -**How it picks the pair.** Names first, because that is what you already told it -and the answer is explainable: a track whose id or filename looks like music -(`music`, `bgm`, `bed`, `score`…) is the bed, one that looks like a voice -(`voice`, `vo`, `narration`, `speech`…) is the voice, and SFX-shaped names are not -candidates for either. If one role is filled and a single track is left, that -track takes the other role. Only when names decide nothing does it listen: it -measures how much of each track is quiet, and the one that stops between phrases -is the voice. **When two tracks are too close to call it refuses and asks you to -name them** rather than carving the wrong one — a bed carved against a bed is -silent and confusing, and typing two ids is cheap. +**How it picks the tracks.** Names first, because that is what you already told it +and the answer is explainable — `classifyAudioName` in core, the same classifier +Studio's own picker uses, so the two cannot disagree. A track whose id or filename +looks like music (`music`, `bgm`, `bed`, `score`…) is the bed; everything else that +plays over it and is not SFX-shaped is a voice. Audio elements are preferred: video +counts only when no audio track is left to be the voice, or every B-roll clip in the +composition would read as somebody talking. **It refuses when it cannot tell which +track is the bed** rather than carving the wrong one — typing one id is cheap. Same analysis functions as the panel, so the result is identical. Needs `ffmpeg` on PATH and `@hyperframes/core` installed in the project (`npm i -D diff --git a/skills/hyperframes-audio/references/attributes.md b/skills/hyperframes-audio/references/attributes.md index cb44a1bfd3..beae711e40 100644 --- a/skills/hyperframes-audio/references/attributes.md +++ b/skills/hyperframes-audio/references/attributes.md @@ -70,13 +70,18 @@ worklet effects (`compressor`, `limiter`, `gate`, `bitcrush`) have none at all. ## `data-fx-carve` — the carve's settings ```json -{ "source": "narration", "strength": 0.35, "dynamic": true } +{ "enabled": true, "sources": ["narration", "interview-guest"], "strength": 0.35 } ``` -- `source` is the **element id of the voice track to listen to**. It lives on the - bed being processed, not on the voice. +- `sources` are the **element ids of every voice this bed makes room for**. They live + on the bed being processed, not on the voices. Summed onto the bed's clock before + the analysis, so one set of filters and envelopes covers all of them. - `strength` 0..1 derives the whole mechanism (see `carveProfile`). -- `dynamic` follows the voice moment to moment instead of holding one depth. +- There is no `dynamic`: a carve always follows the speech. +- `enabled` is whether the carve applies. It exists because a bed with exactly one + candidate voice is carved by default: with "off" represented by an absent + attribute, switching it off would read as never-configured and the default would + put it back. `enabled: false` keeps the settings and stops the carve. This attribute is not read at playback — the chain and lanes it produced are what play. It exists so the settings can be read back and re-derived rather than @@ -85,4 +90,6 @@ carve possible. Older projects may carry the six mechanism numbers (`maxCutDb`, `bands`, `q`, `intelligibilityBias`, `duckDb`, `headroomDb`) instead of `strength`. They still -load: the depth maps back onto a strength and everything else is re-derived. +load: the depth maps back onto a strength and everything else is re-derived. A +stored carve with no `enabled` reads as on, and a single `source` reads as a one-voice +`sources` list. A stored `dynamic` is ignored — every carve follows the speech now. diff --git a/skills/hyperframes-audio/scripts/carve.mjs b/skills/hyperframes-audio/scripts/carve.mjs index e198a05a7d..271d4c0c91 100644 --- a/skills/hyperframes-audio/scripts/carve.mjs +++ b/skills/hyperframes-audio/scripts/carve.mjs @@ -11,11 +11,12 @@ * * node carve.mjs --comp index.html * node carve.mjs --comp index.html --bed music-bed --voice narration \ - * --strength 0.45 + * --voice interview-guest --strength 0.45 * - * With no --bed/--voice it works out the pair itself, and refuses rather than - * guessing when it cannot tell them apart. Dynamic by default, because a bed - * thinned through every pause is worse than one that follows the voice. + * With no --bed/--voice it works out the tracks itself: the bed, and every voice + * playing over it. `--voice` may be repeated to name them instead. Every named + * voice is analysed together, so a bed running under a narrator and an answer makes + * room for both. * * Needs `ffmpeg` on PATH (to decode the audio) and `@hyperframes/core` resolvable * from the composition's project (`npm i -D @hyperframes/core`) — the CLI bundles @@ -33,20 +34,17 @@ import { pathToFileURL } from "node:url"; * bands and envelopes come out the same either way. */ const SAMPLE_RATE = 48000; -const usage = `carve.mjs --comp [--bed ] [--voice ] - [--strength 0..1] [--static] [--dry-run] [--core ] +const usage = `carve.mjs --comp [--bed ] [--voice ...] + [--strength 0..1] [--dry-run] [--core ] --bed id of the music track that gets carved (detected if omitted) - --voice id of the voice track to listen to (detected if omitted) + --voice id of a voice to make room for; repeatable (detected if omitted) --strength how hard to carve, 0..1 (default 0.25) - --static hold one depth instead of following the voice (default: follow) --dry-run report what it would write, touch nothing --core directory to resolve @hyperframes/core from (default: the comp's)`; function parseArgs(argv) { - // Dynamic unless told otherwise: a static carve thins the bed through every - // pause, which is the wrong default for anything with gaps in the narration. - const args = { strength: 0.25, dynamic: true, dryRun: false }; + const args = { strength: 0.25, dryRun: false, voices: [] }; for (let i = 0; i < argv.length; i += 1) { const flag = argv[i]; const next = () => { @@ -57,11 +55,9 @@ function parseArgs(argv) { }; if (flag === "--comp") args.comp = next(); else if (flag === "--bed") args.bed = next(); - else if (flag === "--voice") args.voice = next(); + else if (flag === "--voice") args.voices.push(next()); else if (flag === "--strength") args.strength = Number(next()); else if (flag === "--core") args.core = next(); - else if (flag === "--dynamic") args.dynamic = true; - else if (flag === "--static") args.dynamic = false; else if (flag === "--dry-run") args.dryRun = true; else if (flag === "-h" || flag === "--help") fail(usage, 0); else fail(`unknown flag: ${flag}\n\n${usage}`); @@ -158,114 +154,87 @@ function mediaElements(html) { return found; } -const MUSIC_NAME = /music|bgm|\bbed\b|soundtrack|score|song|theme/i; -const VOICE_NAME = /voice|\bvo\b|narrat|speech|dialog|vocal|\btts\b|announce/i; -const NOT_A_BED = /sfx|foley|whoosh|impact|riser|stinger|swoosh|click|ding|boom/i; - /** - * Fraction of the track that is more than 25 dB below its own peak. - * - * The one cheap thing that tells speech from music without recognising either: - * a voice stops between phrases and a music bed does not. Measured in 50 ms - * frames, which is short enough to see the gap between words and long enough not - * to see the gap between two cycles of a low note. - */ -function silenceFraction(samples) { - const frame = Math.floor(SAMPLE_RATE * 0.05); - const levels = []; - for (let i = 0; i + frame <= samples.length; i += frame) { - let sum = 0; - for (let k = i; k < i + frame; k += 1) sum += samples[k] * samples[k]; - levels.push(Math.sqrt(sum / frame)); - } - if (levels.length === 0) return 0; - const floor = Math.max(...levels) * 10 ** (-25 / 20); - return levels.filter((level) => level < floor).length / levels.length; -} - -/** - * Work out which track is the bed and which is the voice. + * Work out which track is the bed and which tracks are its voices. * * Names first, because they are what the author already told us and the answer is - * explainable. Only when a name does not decide does it listen to the audio, and - * it refuses rather than picking when the measurement is close — carving the wrong - * track is a silent, confusing failure, and asking is cheap. + * explainable: a track whose id or filename looks like music is the bed, ones that + * look like speech are voices, SFX-shaped names are neither. `classifyAudioName` + * comes from core so Studio's own picker and this cannot disagree. + * + * EVERY voice over the bed, not one of them. A bed usually runs under a whole + * sequence, and they are analysed together — so there is nothing to disambiguate, + * which is why this no longer refuses when several tracks look like speech. + * + * Only tracks that actually play while the bed does: one somewhere else on the + * timeline cannot mask it. It still refuses when it cannot find a bed at all, or + * finds no voice to make room for. */ -function detectPair(html, compDir, given) { +function detectTracks(html, given, classify, overlaps) { const all = mediaElements(html); + const kindOf = (el) => classify(el.id, unescapeAttr(attrOf(el.tag, "src") ?? "")); + const spanOf = (el) => { + const raw = attrOf(el.tag, "data-duration"); + const n = raw === null ? Number.NaN : Number(raw); + return { + start: startOf(el.tag), + duration: Number.isFinite(n) ? n : null, + }; + }; const pick = (id, what) => { const found = all.find((el) => el.id === id); if (!found) fail(`no