fix(skill): SFX default volume = natural; preserve timing-alignment knobs

User instruction: "tweak the sfx applying in the way so that it wouldn't
tweak anything for the volume....let them just be as they are...(but it
still need to account for when in the audio actual sound effect starts
and end in order to be able to align, even if colliding into other sfx
or smt...)"

Stance change:
- Volume default → natural catalog-normalized level (no data-volume
  attribute emitted, = 1.0). Catalog is loudness-normalized at the file
  level, so emitting clips bare lets them ring at the level the catalog
  committed to.
- Timing alignment knobs preserved + elevated: data-media-start (skip
  dead air at head, = `onsetSec` from `sfx add` analysis) is now the
  first-class control. data-start / data-duration / data-media-start
  together align the audible attack to the beat frame.
- Multiple SFX may overlap; SFX may stack on VO. The engine has no
  auto-ducking — that's by design now, not a hazard to defend against.
- The lone common exception is a "felt, not heard" ambient bed (~0.1
  volume) — deliberate texture, not a duck.
- Per-clip nudge (when measured LUFS shows real mismatch) is the only
  remaining reason to set data-volume — per-clip override, not per-project
  policy.

Files changed:

1. scripts/assemble-index.mjs SFX emit (lines 343-371):
   - Was: `const vol = cue.volume != null ? cue.volume : 0.35;`
     followed by unconditional `data-volume="${vol}"` emit.
   - Now: emit data-volume only when cue.volume is explicitly set; emit
     data-media-start when cue.media_start is set. Block comment above
     the loop documents the natural-level rationale + the timing-alignment
     stance.

2. skills/hyperframes/references/sound-effects.md:
   - "The audio contract" section: marked data-volume as Optional;
     reordered to put data-media-start before data-volume (timing knob
     first, level knob second).
   - "Volume hierarchy" → renamed "Volume — default is natural, no
     auto-duck." Table reframed as "reference nudge ranges (only if
     needed)" instead of defaults. SFX row now reads "0.7-1.0 (default =
     natural)."
   - "Reading the numbers correctly" item 3: updated to "default is to
     omit data-volume" instead of "the volume hierarchy is your baseline."
   - Worked examples: removed `data-volume="0.3"` from the basic embed
     example; the bed-only example explicitly calls out 0.12 as an
     exception that proves the natural-level rule.

3. skills/website-to-hyperframes/references/step-3-storyboard.md SFX
   assignment section (lines 286-301):
   - Dropped `volume 0.3` / `volume 0.5` from the example list.
   - "Volume under narration: keep SFX at 0.2-0.3 under VO" replaced
     with "Volume — default natural" + the trust-the-mix prose.
   - Added a line emphasizing that timing precision (`data-media-start`)
     matters more now that volume defaults to natural.

Sections deliberately kept:
- The "Combining sounds" rule about `data-volume` for stack-clipping
  control (genuine audio engineering, not a duck heuristic).
- The accent-the-downbeat data-volume nudge in the "Sequence" pattern
  (variation, not ducking).
- All onset/peak/tail/peak-time mechanics — these are the timing
  alignment knobs the user explicitly asked to preserve.

Backward-compatible behavior:
- Cues that explicitly set `volume` still get `data-volume` emitted.
- Existing project HTMLs render unchanged — the change affects only how
  new assemblies emit SFX.
This commit is contained in:
ukimsanov
2026-06-12 00:42:42 -07:00
parent f293d9daf3
commit cd5f6bcc7e
3 changed files with 42 additions and 27 deletions
+22 -17
View File
@@ -70,7 +70,7 @@ All times in the analysis are **clip-relative** (0 = the clip's own start), _not
1. **Trim dead air first.** A catalog clip can carry silence padding — a "5s riser" may be 2s silence + 5s rise + 1s silence — and if you place the raw file it fires late and wastes its tail. Set `data-media-start = onset` and `data-duration = tail − onset` so only the real content plays. (`sfx add`/`inspect` print this as a ready `trim` line.) After trimming, the audible sound begins **exactly at `data-start`**.
2. **Anchor to the beat.** A **hit** (click, impact, whoosh) aligns by its attack — now at the start — so `data-start = beat`. A **build** (riser) aligns by its peak, which sits `peak − onset` into the trimmed clip, so `data-start = beat − (peak − onset)`. Example: a riser with onset 2.0s, peak 6.0s landing on a beat at 10.0s → `data-media-start="2.0"`, `data-start="6.0"` (= 10.0 − 4.0); fire the drop at 10.0s. They overlap — that's the chain.
3. **Level** — the catalog is loudness-normalized, so the volume hierarchy is your baseline; you usually don't touch it. Use measured LUFS only to _nudge_: if two layered clips clash, drop `data-volume` on the one with the higher (louder, less-negative) LUFS.
3. **Level** — the catalog is loudness-normalized at the file level, so the default is to omit `data-volume` and let the clip play natural. Only nudge when measured LUFS shows a real mismatch — if two layered clips clash, drop `data-volume` on the one with the higher (louder, less-negative) LUFS.
4. **Shape** — the sparkline spans the whole clip; block `i` sits at `t ≈ (i + 0.5) / length × duration`. Read it for _where_ the energy builds, drops, or breaks — then place your trigger there.
(There's no need to _cut_ the audio into a new file — `data-media-start` + `data-duration` trim at render time, losslessly and reversibly. Only mid-clip gaps, which a single window can't remove, would call for a different clip.)
@@ -103,7 +103,10 @@ hyperframes sfx inspect <id>
Search hits return `name`, `description`, `duration`, and `score`. Use the **duration to tell the mode apart** — a 0.5s result is a hit, a 10s result is a bed. Always `add` (download) and embed the **local file**; never embed the search result's `audio_url` (it's a presigned URL that expires before render):
```html
<audio src="assets/sfx/<id>.mp3" data-start="3.2" data-duration="1.5" data-volume="0.3"></audio>
<!-- Default: omit data-volume — clip plays at natural catalog-normalized level.
data-media-start trims dead air at the head (set to the analysis `onsetSec`)
so the audible attack lines up with data-start. -->
<audio src="assets/sfx/<id>.mp3" data-start="3.2" data-duration="1.5" data-media-start="0.12"></audio>
```
### Access (free, but needs a key)
@@ -129,23 +132,25 @@ SFX are plain audio clips — no special engine handling. The mixer reads exactl
- **`data-start`** — when the clip fires (seconds): the trigger frame (or the scene start, for a bed).
- **`data-duration`** — how long it plays; use the catalog duration. (`data-end` also works; duration is end minus start.)
- **`data-volume`** — 0 to 1. This is the **only** knob that sets level under voiceover. There is **no pan** — volume and timing are the only audio controls.
- **`data-media-start`** — trim into the source file (skip the first N seconds), for clips with a slow attack.
- **`data-media-start`** — trim into the source file (skip the first N seconds), for clips with a slow attack. Set from `sfx add`'s `onsetSec` so the audible attack lines up with `data-start`.
- **`data-volume`** — 0 to 1. **Optional.** Omit to play at natural catalog level (= 1.0). Use only as a per-clip nudge, not as a default duck.
`data-track-index` does **not** affect audio mixing — every clip mixes together regardless of its index. It only organizes the visual timeline. So `data-volume`, not the track index, is what keeps SFX under the VO.
`data-track-index` does **not** affect audio mixing — every clip mixes together regardless of its index. It only organizes the visual timeline.
## Volume hierarchy
## Volume — default is natural, no auto-duck
Layer levels so nothing fights the voice:
**SFX play at their natural catalog-normalized level.** The catalog is loudness-normalized at the file level, so embedding a clip without `data-volume` lets it ring at the level the catalog committed to. Multiple SFX can overlap, and SFX can stack on top of voiceover — the engine never auto-ducks, by design. If the natural mix sounds wrong, fix it by trimming dead air, shifting timing, or swapping clips, not by reflexively scaling volume down.
| Layer | data-volume |
| -------------------- | ----------- |
| Narration / VO | 1.0 |
| Background music | 0.4 – 0.6 |
| Sound effects (hits) | 0.2 – 0.35 |
| Ambient bed | 0.08 – 0.2 |
The columns below are **reference nudge ranges** for the rare per-clip override (a single clip measurably louder than its neighbors, a bed designed to be felt-not-heard). They are NOT defaults the agent applies on every clip.
A bed sits at the very bottom — it should be _felt_, not heard. A whoosh at 1.0 over narration is jarring; at 0.3 it punctuates without masking the words. When two SFX should match in level, compare their measured LUFS (from `sfx add`/`inspect`) rather than assuming equal `data-volume` sounds equal.
| Layer | nudge range (only if needed) |
| ------------------------------ | ---------------------------- |
| Narration / VO | 1.0 (always) |
| Background music under VO | 0.4 – 0.6 |
| Sound effects (hits) — nudge | 0.7 – 1.0 (default = natural)|
| Ambient bed — felt-not-heard | 0.08 – 0.2 |
When two SFX should match in perceived level, compare their measured LUFS (from `sfx add` / `sfx inspect`) — that's more reliable than guessing-and-tweaking `data-volume`.
## Timing wisdom
@@ -172,9 +177,9 @@ A bed sits at the very bottom — it should be _felt_, not heard. A whoosh at 1.
**An atmospheric scene** — a tense product teaser over a dark background:
- Lay **one bed** (a low tension drone, ~10s) under the whole scene at `data-volume="0.12"`. No beat needed — it sets mood.
- Punctuate only the real beats on top: a riser into the product reveal, one impact on the reveal frame.
- Result: a continuous floor + two punctuation sounds.
- Lay **one bed** (a low tension drone, ~10s) under the whole scene. A "felt, not heard" bed is one of the few cases where setting `data-volume="0.12"` is right — it's the exception that proves the natural-level rule.
- Punctuate only the real beats on top — at natural volume: a riser into the product reveal, one impact on the reveal frame. Don't reflexively scale them down.
- Result: a continuous floor + two punctuation sounds, ringing as the catalog committed to.
## Constraints
@@ -287,16 +287,16 @@ When planning beats, decide which ones deserve an HTML-in-Canvas treatment vs. a
SFX come from HeyGen's global catalog via the `hyperframes sfx` CLI — **not** a bundled file set. The full model (when to use them, the five families, the volume hierarchy, the trim/anchor recipe) lives in [`../../hyperframes/references/sound-effects.md`](../../hyperframes/references/sound-effects.md); read it. This step only decides _which moments get sound and what kind_ — Step 5 searches the catalog and wires the clips.
**Browse first, then assign by function — not by filename.** Run `hyperframes sfx list` to see what families exist (impacts, whooshes, risers, memes, stingers, ambiences, …). For each beat that earns a sound, specify it by _what it does + how it feels_ (the catalog is searched by meaning), the moment, and a volume:
**Browse first, then assign by function — not by filename.** Run `hyperframes sfx list` to see what families exist (impacts, whooshes, risers, memes, stingers, ambiences, …). For each beat that earns a sound, specify it by _what it does + how it feels_ (the catalog is searched by meaning) and the moment:
- a punchy transition whoosh at `0.2s`, volume `0.3` — on the hero image snapping in
- a soft success chime at `3.8s`, volume `0.5` — on the logo appearing
- a punchy transition whoosh at `0.2s` — on the hero image snapping in
- a soft success chime at `3.8s` — on the logo appearing
**Less is more.** Most beats need zero SFX; one per beat is typical (see the global doc's "count beats, not animations"). Never place SFX on shader transitions — they're already an audio-visual event.
**Placement, trimming, and chains are all in the global doc — don't restate per-file timing here.** Step 5 follows it: `sfx add` measures each clip (peak, onset/tail, loudness), then trims dead air and anchors hits/risers precisely. (This replaces the old "peak at start/end, never trim" rules — catalog clips aren't a fixed hand-tuned set, so trimming to the _measured_ onset/tail is how you avoid late or cut-off sounds.)
**Placement, trimming, and chains are all in the global doc — don't restate per-file timing here.** Step 5 follows it: `sfx add` measures each clip (peak, onset/tail, loudness), then trims dead air via `data-media-start` (= `onsetSec`) and anchors hits/risers precisely against `data-start`. The trim is what aligns the audible attack to the beat frame — that timing precision matters even more now that volume defaults to natural.
**Volume under narration:** HyperFrames has no auto-ducking — keep SFX at 0.2–0.3 under VO. Note the intended volume per entry so Step 5 wires it.
**Volume — default natural.** Don't pre-prescribe per-clip volumes here. SFX play at the catalog's loudness-normalized level (no `data-volume` emitted). The engine has no auto-ducking — that's by design. Multiple SFX may overlap, and SFX may stack on top of VO; if a specific clip measurably overpowers its neighbors, Step 5 can nudge that one clip's `data-volume`, but it's the exception, not the default. The "felt, not heard" ambient bed is the one common case for an explicit lower volume (~0.1).
**Access:** the catalog is free but needs a HeyGen API key. If it isn't set when Step 5 runs `sfx`, ask the user for one (free at https://app.heygen.com/developers/api) or build without SFX — never silently drop them.
@@ -354,9 +354,14 @@ sfx.forEach((cue, i) => {
);
return;
}
const vol = cue.volume != null ? cue.volume : 0.35;
// SFX default: play at natural catalog-normalized level. The catalog is
// loudness-normalized at the file level, so emitting no `data-volume`
// attribute (= 1.0) lets clips ring as-recorded. Agents can still set
// `cue.volume` explicitly to nudge a specific clip — that override is
// honored. Even when multiple SFX overlap or stack with VO, the resulting
// mix is the catalog's natural sum (no auto-duck, by design).
if (sfxEmitted === 0) body.push(` <!-- SFX -->`);
body.push(
const lines = [
` <audio`,
` id="el-sfx-${i}"`,
` class="clip"`,
@@ -364,9 +369,14 @@ sfx.forEach((cue, i) => {
` data-start="${cue.t}"`,
` data-duration="${cue.duration}"`,
` data-track-index="${20 + i}"`,
` data-volume="${vol}"`,
` ></audio>`,
);
];
// `cue.media_start` lets a clip skip dead air at the head (set from the
// `sfx add` analysis's `onsetSec`). Aligning audible-onset to `data-start`
// is the timing-alignment knob the volume change preserves.
if (cue.media_start != null) lines.push(` data-media-start="${cue.media_start}"`);
if (cue.volume != null) lines.push(` data-volume="${cue.volume}"`);
lines.push(` ></audio>`);
body.push(...lines);
sfxEmitted++;
});