mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-10-30 09:16:55 +00:00
Merge pull request #3878 from heygen-com/docs/require-voiceover-carve
feat(audio): require the voiceover carve, default strength 0.8, slower release
This commit is contained in:
Vance Ingalls
@@ -348,8 +348,9 @@ describe("analyseCarveDuck", () => {
|
||||
// of that rather than right against it.
|
||||
expect(at(duck, 2)).toBe(0);
|
||||
// Back to inaudible by the end of a long tail rather than held down for the
|
||||
// clip; the envelope's last point is pinned at exactly no cut.
|
||||
expect(at(duck, 15)).toBeCloseTo(0, 1);
|
||||
// clip; the envelope's last point is pinned at exactly no cut. Ten seconds
|
||||
// after the last word is four release constants, so the residue has snapped.
|
||||
expect(at(duck, 15.9)).toBeCloseTo(0, 1);
|
||||
expect(duck.at(-1)?.v).toBe(0);
|
||||
});
|
||||
|
||||
@@ -383,10 +384,14 @@ describe("analyseCarveDuck", () => {
|
||||
|
||||
// Still most of the way down a third of a second after the last word...
|
||||
expect(at(duck, 6.3)).toBeLessThan(atSpeech * 0.6);
|
||||
// ...noticeably recovered by a second and a half...
|
||||
expect(at(duck, 7.5)).toBeGreaterThan(atSpeech * 0.5);
|
||||
// ...and within a hair of flat a few seconds later, rather than held down.
|
||||
expect(at(duck, 11)).toBeGreaterThan(-0.6);
|
||||
// Three seconds past the last word the bed is still audibly down — at a 1.6s
|
||||
// release it is all but back (−1.5 dB), which is the sentence-break flip this
|
||||
// constant exists to stop. This is the point that reds on the old value.
|
||||
expect(at(duck, 9)).toBeLessThan(-2);
|
||||
// ...noticeably recovered two and a half seconds on...
|
||||
expect(at(duck, 8.5)).toBeGreaterThan(atSpeech * 0.5);
|
||||
// ...and within a hair of flat several seconds later, rather than held down.
|
||||
expect(at(duck, 13.5)).toBeGreaterThan(-0.6);
|
||||
});
|
||||
|
||||
it("finishes the release past the last word instead of snapping at the end", () => {
|
||||
|
||||
@@ -552,13 +552,14 @@ const RELEASE_S = 0.25;
|
||||
* jumped to full the instant a word ended, and what you hear then is the effect
|
||||
* switching off rather than a mix breathing. Slower than any gap inside a
|
||||
* sentence, so it rides through the pauses between words and only recovers
|
||||
* between sentences.
|
||||
* between sentences. 1.6s still read as the bed "coming back" at every sentence
|
||||
* break on narrated builds; 2.4s lets it swell back over a breath instead.
|
||||
*
|
||||
* It also has to be well clear of the envelope's own step. A long voice is
|
||||
* measured about every 0.3s, so a release anywhere near that puts the entire
|
||||
* recovery inside one step — a jump, whatever the constant claims.
|
||||
*/
|
||||
const DUCK_RELEASE_S = 1.6;
|
||||
const DUCK_RELEASE_S = 2.4;
|
||||
|
||||
/**
|
||||
* Windows an envelope may spend. Held under `MAX_AUTOMATION_POINTS` with room
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"files": 2
|
||||
},
|
||||
"general-video": {
|
||||
"hash": "ced729740550f414",
|
||||
"hash": "68910f1093ad15c2",
|
||||
"files": 4
|
||||
},
|
||||
"hyperframes": {
|
||||
@@ -26,7 +26,7 @@
|
||||
"files": 121
|
||||
},
|
||||
"hyperframes-audio": {
|
||||
"hash": "84035d8ce2a61fbc",
|
||||
"hash": "b39bac771e873eae",
|
||||
"files": 7
|
||||
},
|
||||
"hyperframes-cli": {
|
||||
@@ -34,7 +34,7 @@
|
||||
"files": 11
|
||||
},
|
||||
"hyperframes-core": {
|
||||
"hash": "6921c3b1dbeefb30",
|
||||
"hash": "e5b2b62d60995dbb",
|
||||
"files": 20
|
||||
},
|
||||
"hyperframes-creative": {
|
||||
|
||||
@@ -113,7 +113,7 @@ Use this dependency order. Skip a stage only when its input is absent.
|
||||
The builder writes one bounded packet per scene under `.hyperframes/frame-packets/` (the scene's exact storyboard block + the blueprint body + every cited rule recipe, inlined) and `_role.md` (`../hyperframes-core/references/frame-worker-core.md` + this skill's `sub-agents/frame-worker.md`, concatenated verbatim — the complete worker role). Dispatch the workers — 2–3 scene packets each, all in one wave (`../hyperframes-core/references/subagent-dispatch.md`); each worker's prompt carries `_role.md` and its packets — paste them in full, or hand the file paths for the worker to read first (equivalent either way) — plus a dispatch context with `PROJECT_DIR`, its `frame_id`s, and canvas size. WAIT on every scene's `compositions/<frame_id>.html` + `compositions/<frame_id>.motion.json`. Workers read only their packets and the design truth file; they never open `STORYBOARD.md` or the skill documents. With no delegation channel, fall back serially: process one packet at a time in this context, still working from the packet alone.
|
||||
|
||||
5. **Merge motion sidecars.** Collect the workers' `compositions/<frame_id>.motion.json` files and carry their durations and exit/entry vectors into assembly; where the doctrine chain (`/motion-doctrine`) is installed, translate them into the project ledger before stamping seams.
|
||||
6. **Assemble.** Mount scenes, media, transitions, captions, and audio using the production loop. Real voice duration overrides estimates.
|
||||
6. **Assemble.** Mount scenes, media, transitions, captions, and audio using the production loop. Real voice duration overrides estimates. When a music bed plays under any voice track, carve the bed before verifying: `/hyperframes-audio` → `scripts/carve.mjs --comp index.html`. A volume duck alone does not finish the mix.
|
||||
7. **Verify.** Use `npx hyperframes lint` for fast feedback after the first HTML pass and structural changes. For the final gate, run `npx hyperframes check`; it reruns lint internally, so do not run a redundant standalone lint immediately before it. For sub-compositions, inspect midpoint snapshots. For multi-scene work, review the animation map.
|
||||
8. **Final approval.** Open the final Studio preview only after checks pass. Ask whether to render or revise. Render only after approval.
|
||||
|
||||
|
||||
@@ -252,11 +252,7 @@ analysis time, so a clip added to the group later is covered without touching
|
||||
<audio id="vo-middle" data-audio-group="voiceover" …></audio>
|
||||
<audio id="vo-outro" data-audio-group="voiceover" …></audio>
|
||||
|
||||
<audio
|
||||
id="music"
|
||||
data-fx-carve='{"enabled":true,"sources":["voiceover"],"strength":0.25}'
|
||||
…
|
||||
></audio>
|
||||
<audio id="music" data-fx-carve='{"enabled":true,"sources":["voiceover"],"strength":0.8}' …></audio>
|
||||
```
|
||||
|
||||
A `sources` list naming two or more plain clip ids instead of a group is caught
|
||||
@@ -355,17 +351,26 @@ many bands, how wide, how far to favour intelligibility over raw voice energy,
|
||||
how far the level may drop, how far under the voice to aim. Those six move
|
||||
together in any real mix — a gentle carve is a shallow cut in few bands with
|
||||
little ducking, a hard one is deeper in more bands with more — so they are one
|
||||
relationship written once, in `carveProfile`. Default is `0.25` — a 6 dB dip in
|
||||
three bands with 6 dB of level room, audible without sounding like a hole. At
|
||||
`0.5` the dip reaches 10 dB, which is where a carve starts being heard as an
|
||||
effect rather than as room for the voice; above that is deliberate territory for
|
||||
a loud bed under a quiet voice. `0` is spectral only — one band, no level match
|
||||
at all.
|
||||
relationship written once, in `carveProfile`. `carve.mjs` defaults to `0.8` —
|
||||
six bands from 250 Hz to 2.5 kHz cut about 7 dB each and 15 dB at 1.6 kHz, with
|
||||
19 dB of level room — because a bed under narration has to get out of the way
|
||||
first and be music second; `0.25` (a 6 dB dip in three bands, 6 dB of room) kept
|
||||
the bed present but still let it fight the voice, and was judged too weak in
|
||||
practice. At `0.5` the dip reaches 10 dB, which is where a carve starts being
|
||||
heard as an effect rather than as room for the voice. Drop the strength when the
|
||||
bed is the point and the voice is sparse. `0` is spectral only — one band, no
|
||||
level match at all.
|
||||
|
||||
**Carve by default.** A bed playing under narration wants a carve; it is not a
|
||||
polish step to get to if there is time. Place both tracks, run the command below,
|
||||
listen. Skip it only when there is no narration for the music to sit under — a
|
||||
music video, a title card, a montage cut to the track.
|
||||
**Carve by default — required whenever music plays under a voice.** A bed
|
||||
under any voice track (narration, avatar speech, interview, voiceover) gets a
|
||||
carve as part of finishing the mix, not as a polish step to get to if there is
|
||||
time. Place both tracks, run the command below (default strength `0.8`; add
|
||||
`--bed` / `--voice` when detection picks wrong), confirm the written
|
||||
`data-fx-carve`, `data-fx-chain` and `data-automation` with `npx hyperframes check`,
|
||||
and only then render. A volume duck on its own is not a finished mix: it leaves
|
||||
the voice and the bed fighting in the 1–3 kHz band and costs the bed all of its
|
||||
presence for the whole voiceover. Skip the carve only when there is no voice for
|
||||
the music to sit under — a music video, a title card, a montage cut to the track.
|
||||
|
||||
**It always follows the voice.** There is no static mode: a fixed depth thins the
|
||||
bed through every pause, and once you have heard both there is no reason to want it.
|
||||
@@ -399,9 +404,9 @@ dynamically at the default strength, and prints what it decided:
|
||||
```
|
||||
bed music-bed (name looks like music)
|
||||
voice narration (only track left)
|
||||
carve strength 0.25 dynamic
|
||||
bands 400Hz -6dB q1.4, 1000Hz -3dB q1.4, 1600Hz -3.17dB q1.4
|
||||
level 216-point envelope, floor -6 dB
|
||||
carve strength 0.8 dynamic
|
||||
bands 250Hz -7.4dB q2.06, 400Hz -7.4dB q2.06, 630Hz -7.4dB q2.06, 1000Hz -7.4dB q2.06, 1600Hz -14.8dB q2.06, 2500Hz -7.4dB q2.06
|
||||
level 273-point envelope, floor -19.2 dB
|
||||
```
|
||||
|
||||
Name the tracks with `--bed` / `--voice` (repeatable) when the automatic choice is
|
||||
|
||||
@@ -34,17 +34,20 @@ import { pathToFileURL } from "node:url";
|
||||
* bands and envelopes come out the same either way. */
|
||||
const SAMPLE_RATE = 48000;
|
||||
|
||||
/** Default carve strength. 0.25 kept the bed present but still fighting the voice. */
|
||||
export const DEFAULT_STRENGTH = 0.8;
|
||||
|
||||
const usage = `carve.mjs --comp <file.html> [--bed <elementId>] [--voice <elementId> ...]
|
||||
[--strength 0..1] [--dry-run] [--core <dir>]
|
||||
|
||||
--bed id of the music track that gets carved (detected if omitted)
|
||||
--voice id of a voice to make room for; repeatable (detected if omitted)
|
||||
--strength how hard to carve, 0..1 (default 0.25)
|
||||
--strength how hard to carve, 0..1 (default ${DEFAULT_STRENGTH})
|
||||
--dry-run report what it would write, touch nothing
|
||||
--core directory to resolve @hyperframes/core from (default: the comp's)`;
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = { strength: 0.25, dryRun: false, voices: [] };
|
||||
export function parseArgs(argv) {
|
||||
const args = { strength: DEFAULT_STRENGTH, dryRun: false, voices: [] };
|
||||
for (let i = 0; i < argv.length; i += 1) {
|
||||
const flag = argv[i];
|
||||
const next = () => {
|
||||
|
||||
@@ -5,7 +5,13 @@ import { mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { carveSources, groupSourceRefusal, loadCore } from "./carve.mjs";
|
||||
import {
|
||||
DEFAULT_STRENGTH,
|
||||
carveSources,
|
||||
groupSourceRefusal,
|
||||
loadCore,
|
||||
parseArgs,
|
||||
} from "./carve.mjs";
|
||||
|
||||
const SCRIPTS_DIR = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
@@ -237,3 +243,12 @@ test("members is required, so dropping it cannot silently restore the group form
|
||||
assert.throws(() => carveSources(voices, bed("bgm", "music")), TypeError);
|
||||
assert.throws(() => groupSourceRefusal(voices, bed("bgm", "music")), TypeError);
|
||||
});
|
||||
|
||||
test("the default strength is 0.8, and --strength still overrides it", () => {
|
||||
// 0.25 left the bed audibly fighting the voice on narrated builds; a carve at
|
||||
// the default has to already be a finished mix. Pinned so it cannot drift back
|
||||
// unnoticed — core pins its own DEFAULT_CARVE the same way.
|
||||
assert.equal(DEFAULT_STRENGTH, 0.8);
|
||||
assert.equal(parseArgs(["--comp", "x.html"]).strength, 0.8);
|
||||
assert.equal(parseArgs(["--comp", "x.html", "--strength", "0.25"]).strength, 0.25);
|
||||
});
|
||||
|
||||
@@ -4,17 +4,17 @@ The stages between a plan the user has agreed to and a video in their hands, wri
|
||||
|
||||
The shipped narrative workflows implement these stages with their own scripts; a freeform build follows this file directly, borrowing tools where the capability menu says they live (`hyperframes/references/capability-menu.md`).
|
||||
|
||||
| Stage | Needs | Produces | Where the capability lives |
|
||||
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Blocks & assets** | the approved plan | registry blocks installed **once, before any parallel work** (parallel workers race the registry); user assets staged; logos / images / grades resolved | `npx hyperframes add <block>` per block the plan names; staging, adoption, and `resolve` via the menu's media rows |
|
||||
| **Audio** | narration text (when narrated); the storyboard's `music:` mood | voice files + **word timings** + BGM + SFX → `audio_meta.json` | the one engine — `media-use/audio/scripts/audio.mjs`, run in the background; `wait-bgm.mjs` before render when BGM generates |
|
||||
| **Frames** | design spec + the plan (+ a confirmed sketch when one exists — dress that layout, never redraw it: `review-loop.md` § 3) | each scene at `compositions/frames/NN-*.html`, marked `animated` in the storyboard as it lands | `frame.md` + `hyperframes-animation` blueprints / rules (+ the genre lens, menu § Genre lenses); parallel dispatch per `subagent-dispatch.md` |
|
||||
| **Duration sync** | word timings + frames | scene durations trued to real voice length — real duration wins, silent scenes keep estimates, synced values are never hand-edited | a mechanical rule; the narrative workflows' audio scripts apply it, a freeform build applies it by hand |
|
||||
| **Assembly** | frames | the index composition — scenes as sub-compositions on tracks | `sub-compositions.md` + `tracks-and-clips.md`; borrowable `assemble-index.mjs` (menu) |
|
||||
| **Transitions** | the assembled index | scene handoffs injected | `hyperframes-animation/transitions/overview.md` → `catalog.md`; borrowable `transitions.mjs` (menu) |
|
||||
| **Captions** | word timings + the index | the caption track | borrowable `captions.mjs` (menu); no script to time against → `media-use` `scripts/transcribe.mjs` first |
|
||||
| **Verify** | the index (+ captions / transitions when present) | `npx hyperframes lint` and `npx hyperframes check` **passing**; a contact-sheet glance (`snapshot --at <frame-midpoints>`) | `hyperframes-cli` |
|
||||
| **Deliver** | verify passing | the final-look pause → on approval `render` → optionally `publish` (a stable hosted link, private by default) → the recipe offer | final approval and recipe offer: `review-loop.md` § 4; render / publish: `hyperframes-cli` |
|
||||
| Stage | Needs | Produces | Where the capability lives |
|
||||
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Blocks & assets** | the approved plan | registry blocks installed **once, before any parallel work** (parallel workers race the registry); user assets staged; logos / images / grades resolved | `npx hyperframes add <block>` per block the plan names; staging, adoption, and `resolve` via the menu's media rows |
|
||||
| **Audio** | narration text (when narrated); the storyboard's `music:` mood | voice files + **word timings** + BGM + SFX → `audio_meta.json`; **when BGM plays under a voice, the bed is carved** (`hyperframes-audio/scripts/carve.mjs`) before Verify — a duck alone is not a finished mix | the one engine — `media-use/audio/scripts/audio.mjs`, run in the background; `wait-bgm.mjs` before render when BGM generates |
|
||||
| **Frames** | design spec + the plan (+ a confirmed sketch when one exists — dress that layout, never redraw it: `review-loop.md` § 3) | each scene at `compositions/frames/NN-*.html`, marked `animated` in the storyboard as it lands | `frame.md` + `hyperframes-animation` blueprints / rules (+ the genre lens, menu § Genre lenses); parallel dispatch per `subagent-dispatch.md` |
|
||||
| **Duration sync** | word timings + frames | scene durations trued to real voice length — real duration wins, silent scenes keep estimates, synced values are never hand-edited | a mechanical rule; the narrative workflows' audio scripts apply it, a freeform build applies it by hand |
|
||||
| **Assembly** | frames | the index composition — scenes as sub-compositions on tracks | `sub-compositions.md` + `tracks-and-clips.md`; borrowable `assemble-index.mjs` (menu) |
|
||||
| **Transitions** | the assembled index | scene handoffs injected | `hyperframes-animation/transitions/overview.md` → `catalog.md`; borrowable `transitions.mjs` (menu) |
|
||||
| **Captions** | word timings + the index | the caption track | borrowable `captions.mjs` (menu); no script to time against → `media-use` `scripts/transcribe.mjs` first |
|
||||
| **Verify** | the index (+ captions / transitions when present) | `npx hyperframes lint` and `npx hyperframes check` **passing**; a contact-sheet glance (`snapshot --at <frame-midpoints>`) | `hyperframes-cli` |
|
||||
| **Deliver** | verify passing | the final-look pause → on approval `render` → optionally `publish` (a stable hosted link, private by default) → the recipe offer | final approval and recipe offer: `review-loop.md` § 4; render / publish: `hyperframes-cli` |
|
||||
|
||||
The Frames stage follows the plan's citations: a scene planned on a blueprint or on named rules is built by reading that recipe's body (`hyperframes-animation/blueprints/<id>.md`, `rules/<id>.md`) before its motion is written — names come from the indexes, never invented, and a scene the plan left uncited gets its citation at build time, not improvised motion.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user