mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-04 16:42:27 +00:00
Vance IngallsandClaude Sonnet 5
99f42be04c
* feat(core): route grouped audio through a group bus in preview An audio element carrying `data-audio-group` no longer lands its gain on the master bus directly — it feeds a per-group `GainNode` (built lazily on first use, one per group id) which itself feeds master, so members of the same group sum before the ear, ready for a group-level FX chain and volume/mute in later steps. An id with no matching `<hf-audio-group>` element still gets a plain, unprocessed bus rather than losing the track. The group's own chain and volume lane are wired through the same `attachElementFxChain`/`scheduleVolumeLane` every element already uses, against the group's clock — composition time (design doc §1.3), since a group has no `data-start` and a missing one parses as 0. The bus persists across `stopAll()` (mirroring `_masterGain`'s own lifecycle) so replaying a group does not rebuild its chain; only `destroy()` disposes it. Render is untouched — stays flat until B4; `audio-groups` is still a 0% canary so nothing ships this to a real composition without hand-authoring `data-audio-group`. Also: `audioGroupOf` (B1) crashed on any element lacking a real `tagName`/ `getAttribute` — exactly the shape of most `HTMLMediaElement` test doubles in this suite, including this file's own `mockEl`. Made it tolerant, same style as `readChain`'s existing guard in `runtime/audioFx.ts`. `schedulePlayback` was already 110 lines pre-existing before this diff; extracted `resolveDestination` and `handleSourceEnded` to shrink it to 92, then suppressed the remainder (inherently sequential graph wiring, not a decision tree) per the same precedent B2 used on `TimelineLogicalRow`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(studio,lint): carve targets voiceover groups — always, when plural Plural voiceover carve now targets a group instead of naming each clip: `resolveCarveSourceIds` (core `audioGroups.ts`) expands a group id to its current members at analysis time, so a clip added to the group later is covered without touching `sources`. The picker (`useFxCarve.ts`) offers a grouped voice as one option instead of one row per member, tests overlap as a union of member spans (a group overlaps the bed if ANY member does), and prefers a qualifying group over its individual members in `autoSourceIds`. Picking two or more ungrouped voice clips in the carve flow now mints a group behind them (`mintGroupId`, de-duped against every id in the document) and writes `data-audio-group` on each picked clip atomically, one undo entry — `createAudioGroupAndAssignMembers` in `timelineTrackVisibility.ts` copies `setElementsHidden`'s multi-target write shape. The DSP is untouched: `mixCarveSources` already sums multiple sources correctly (verified in the design doc's own investigation) — this only fixes the picker. New lint rule `audio_carve_ungrouped_sources` (`packages/lint/src/rules/ media.ts`, alongside `audio_volume_double_automation`) warns when a `data-fx-carve`'s `sources` names two or more plain clip ids instead of a group — the shape that silently rots when a clip is added. `/hyperframes- audio` states the same rule as an invariant, not a tip, with the grouped- narration HTML example from the design doc. The group-matching and auto-group logic (`withAutoGroupedSources`, `collectCarveCandidates`) is split into `useFxCarveGrouping.ts` — `useFxCarve.ts` was pushing past the 600-line cap. `resolveNextCarveSettings` is deliberately NOT an `async function`: wrapping it in one would force a microtask on every call, including the synchronous branch — the exact bug `withAutoGroupedSources`'s own sync-when-possible contract exists to avoid, and one caught via `propertyPanelAudioFxGroup.test.tsx` (10 failures) before fixing it back to a plain function the caller conditionally awaits. Also extracted `useEffectiveTimelineDuration` out of `App.tsx` and `useRemoveBackground` out of `StudioRightPanel.tsx` (both pushed past 600 lines from an added prop wire), and decomposed `useFxCarve.ts`'s picker IIFE to clear fallow's complexity gate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(engine): render grouped audio through a summed, FX-processed bus Renders what B3 already routes in preview: a group's members sub-mix into one PCM WAV at full composition length (adelay already places each member at its composition position, so the group WAV's t=0 IS composition time), run through the group's own FX chain and automation via the same applyAudioFxChain/envelope-bake path a member uses, then fold into the flat track list as one processed AudioTrack — the final mixAudioTracks call never has to know groups exist. Gain law verified against plans/spikes/amix-nesting-spike.sh (brought over from the plans branch, along with audioMixer.grouping.test.ts, since both were committed there and never merged to origin/main — every step branch in this stack descends from origin/main): the sub-mix's own amix prefers normalize=0 (nulls exactly against a flat mix), falling back to per-node compensation by the group's OWN member count only when this ffmpeg build's amix rejects the option. Carrying any other count into a nested amix node is the exact +2.499 dB silent failure the spike measured — confirmed by a manual mutation check (wrong-count compensation landed 3.5 dB hot, exactly 20*log10(3/2) for a 2-member group compensated as 3; reverted after confirming the level test catches it). A group element carrying data-hidden drops every member before the sub-mix ever runs (RULES: mute-by-drop, never mute-by-volume-0) — parseAudioElements now resolves groups once per parse and skips hidden-group members the same way it already skips data-hidden ancestors. HfAudioGroup (packages/core/src/audioGroups.ts, from B1) gains fxChain, automation, volume and hidden, read off the group element the same way resolveAudioGroups already reads data-label — audioGroups.test.ts updated for the wider shape plus new coverage for the added reads. it.todo("mixes a grouped composition at the same level as the ungrouped one") is now a real, passing test; two more added per the step doc (FX routing isolation, member-level envelope survives grouping). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
104 lines
4.6 KiB
Bash
Executable File
104 lines
4.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Spike: does a nested amix (group bus) preserve the level of a flat amix?
|
|
#
|
|
# Invariant under test: a group whose FX chain is EMPTY must be a no-op on the
|
|
# mix. Grouping is routing, not processing — if grouping alone changes the
|
|
# level, every grouped export is silently wrong.
|
|
set -euo pipefail
|
|
|
|
D="$(dirname "$0")/amix-work"
|
|
rm -rf "$D"; mkdir -p "$D"
|
|
cd "$D"
|
|
|
|
SR=48000
|
|
DUR=3
|
|
|
|
# Four tracks, distinct frequencies so nothing cancels, distinct amplitudes so a
|
|
# mis-weighted track shows up rather than averaging out.
|
|
ffmpeg -v error -f lavfi -i "sine=frequency=220:sample_rate=$SR:duration=$DUR" -af "volume=0.50" -c:a pcm_s16le t1.wav
|
|
ffmpeg -v error -f lavfi -i "sine=frequency=440:sample_rate=$SR:duration=$DUR" -af "volume=0.25" -c:a pcm_s16le t2.wav
|
|
ffmpeg -v error -f lavfi -i "sine=frequency=880:sample_rate=$SR:duration=$DUR" -af "volume=0.40" -c:a pcm_s16le t3.wav
|
|
ffmpeg -v error -f lavfi -i "sine=frequency=1760:sample_rate=$SR:duration=$DUR" -af "volume=0.15" -c:a pcm_s16le t4.wav
|
|
|
|
rms () { # $1 = wav -> RMS dB
|
|
ffmpeg -v info -i "$1" -af astats=metadata=1:reset=0 -f null - 2>&1 \
|
|
| awk -F'dB: ' '/Overall/{o=1} o&&/RMS level dB/{print $2; exit}'
|
|
}
|
|
|
|
echo "=== per-track RMS (dB) ==="
|
|
for f in t1 t2 t3 t4; do printf " %-4s %s\n" "$f" "$(rms $f.wav)"; done
|
|
echo
|
|
|
|
# --- ARM A: flat mix, today's shipped shape -------------------------------
|
|
# amix normalizes by input count; multiply back by the same count.
|
|
ffmpeg -v error -i t1.wav -i t2.wav -i t3.wav -i t4.wav -filter_complex \
|
|
"[0:a]apad,atrim=0:$DUR[a0];\
|
|
[1:a]apad,atrim=0:$DUR[a1];\
|
|
[2:a]apad,atrim=0:$DUR[a2];\
|
|
[3:a]apad,atrim=0:$DUR[a3];\
|
|
[a0][a1][a2][a3]amix=inputs=4:duration=longest:dropout_transition=0[mixed];\
|
|
[mixed]volume=4[out]" -map "[out]" -c:a pcm_s16le flat.wav
|
|
|
|
# --- ARM B: nested, compensated PER NODE ---------------------------------
|
|
# group A = t1+t2 (2 inputs -> x2). outer = groupA + t3 + t4 (3 inputs -> x3).
|
|
ffmpeg -v error -i t1.wav -i t2.wav -i t3.wav -i t4.wav -filter_complex \
|
|
"[0:a]apad,atrim=0:$DUR[a0];\
|
|
[1:a]apad,atrim=0:$DUR[a1];\
|
|
[2:a]apad,atrim=0:$DUR[a2];\
|
|
[3:a]apad,atrim=0:$DUR[a3];\
|
|
[a0][a1]amix=inputs=2:duration=longest:dropout_transition=0[gmix];\
|
|
[gmix]volume=2[gA];\
|
|
[gA][a2][a3]amix=inputs=3:duration=longest:dropout_transition=0[mixed];\
|
|
[mixed]volume=3[out]" -map "[out]" -c:a pcm_s16le nested_ok.wav
|
|
|
|
# --- ARM C: nested, but the outer node keeps the GLOBAL track count -------
|
|
# The plausible mistake: tracks.length is 4, the outer amix has 3 inputs.
|
|
ffmpeg -v error -i t1.wav -i t2.wav -i t3.wav -i t4.wav -filter_complex \
|
|
"[0:a]apad,atrim=0:$DUR[a0];\
|
|
[1:a]apad,atrim=0:$DUR[a1];\
|
|
[2:a]apad,atrim=0:$DUR[a2];\
|
|
[3:a]apad,atrim=0:$DUR[a3];\
|
|
[a0][a1]amix=inputs=2:duration=longest:dropout_transition=0[gmix];\
|
|
[gmix]volume=2[gA];\
|
|
[gA][a2][a3]amix=inputs=3:duration=longest:dropout_transition=0[mixed];\
|
|
[mixed]volume=4[out]" -map "[out]" -c:a pcm_s16le nested_bug.wav
|
|
|
|
# --- ARM D: nested with normalize=0, no compensation anywhere ------------
|
|
ffmpeg -v error -i t1.wav -i t2.wav -i t3.wav -i t4.wav -filter_complex \
|
|
"[0:a]apad,atrim=0:$DUR[a0];\
|
|
[1:a]apad,atrim=0:$DUR[a1];\
|
|
[2:a]apad,atrim=0:$DUR[a2];\
|
|
[3:a]apad,atrim=0:$DUR[a3];\
|
|
[a0][a1]amix=inputs=2:normalize=0:duration=longest:dropout_transition=0[gA];\
|
|
[gA][a2][a3]amix=inputs=3:normalize=0:duration=longest:dropout_transition=0[out]" \
|
|
-map "[out]" -c:a pcm_s16le nested_norm0.wav
|
|
|
|
# --- ARM E: FLAT with normalize=0 ----------------------------------------
|
|
ffmpeg -v error -i t1.wav -i t2.wav -i t3.wav -i t4.wav -filter_complex \
|
|
"[0:a]apad,atrim=0:$DUR[a0];\
|
|
[1:a]apad,atrim=0:$DUR[a1];\
|
|
[2:a]apad,atrim=0:$DUR[a2];\
|
|
[3:a]apad,atrim=0:$DUR[a3];\
|
|
[a0][a1][a2][a3]amix=inputs=4:normalize=0:duration=longest:dropout_transition=0[out]" \
|
|
-map "[out]" -c:a pcm_s16le flat_norm0.wav
|
|
|
|
echo "=== mix RMS (dB) ==="
|
|
for f in flat nested_ok nested_bug nested_norm0 flat_norm0; do
|
|
printf " %-14s %s\n" "$f" "$(rms $f.wav)"
|
|
done
|
|
echo
|
|
|
|
# --- sample-exactness: null test (A inverted + B must be silence) ---------
|
|
null_test () { # $1 $2 -> peak dB of the difference
|
|
ffmpeg -v info -i "$1" -i "$2" -filter_complex \
|
|
"[1:a]volume=-1[inv];[0:a][inv]amix=inputs=2:normalize=0,astats=metadata=1:reset=0[d]" \
|
|
-map "[d]" -f null - 2>&1 \
|
|
| awk -F'dB: ' '/Overall/{o=1} o&&/Peak level dB/{print $2; exit}'
|
|
}
|
|
|
|
echo "=== null tests (peak dB of difference; -inf or < -90 = identical) ==="
|
|
printf " flat vs nested_ok %s\n" "$(null_test flat.wav nested_ok.wav)"
|
|
printf " flat vs nested_bug %s\n" "$(null_test flat.wav nested_bug.wav)"
|
|
printf " flat vs nested_norm0 %s\n" "$(null_test flat.wav nested_norm0.wav)"
|
|
printf " flat vs flat_norm0 %s\n" "$(null_test flat.wav flat_norm0.wav)"
|