mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-01 19:42:03 +00:00
fix(audio): raise the authoring gain ceiling and carry it through the probes (#3333)
* fix(audio): raise the authoring gain ceiling and carry it through the probes Builds on #3328, which made the preview graph apply author gain and user volume exactly once each. That ownership is now correct but everything is still clamped to 1.0, so a clip authored above unity cannot be heard or rendered. `HTMLMediaElement.volume` is spec-clamped to [0,1], so both timeline probes lost a clip's authored gain the moment it also carried a fade: the probe seeded the element at the clamped value and every sample read back at or below 0 dB, and the mixer prefers probed keyframes over the static volume. Both probes now shadow the accessor for their own duration and forward the clamped value to the native setter, so the authored gain survives while nothing outside the probe ever sees an illegal volume. Measured on one 6 s composition, first 4 s: unity -32.8 LUFS, boosted-with-fade -32.8 before and -27.0 after — +5.8 dB, exactly the gain the clip was authored at. One ceiling, defined once in `audioGain.ts` and reachable from both sides: the render mixer imports it, and the page-serialized probe takes it as a parameter rather than re-literalling it. User volume stays spec-clamped — it is a fader, not a gain. Also holds the percent volume slider above unity in both property panels. That control tops out at 100%, so one touch would cap a boosted clip and drop up to 12 dB that now genuinely renders; the dB fader that can represent these levels replaces it in the next PR. * fix(audio): carry a static above-unity gain onto the preview gain node Review follow-up. `setElementVolume` receives the clip's author gain and clamped it to [0,1], so a static `data-volume` above unity was capped on the WebAudio preview path while the render honoured it — the exact preview/render divergence this ceiling exists to close. Automation lanes hid it: they schedule ramps onto the param directly and never pass through here. The master volume beside it stays spec-clamped, because a user fader is not a gain. Verified by mutation: restoring the [0,1] clamp reds the new case. Also scope the leveller's rationale to this rung — `VOLUME_RANGE` still stops at unity until the dB fader lands, so "both now span the same range" was premature — and say why the GSAP-tracking fallback is unity-capped: it reads back through `el.volume`, which the spec pins to [0,1], so it cannot observe an above-unity value however wide the clamp gets. * fix(audio): restore the live test files this branch overwrote, and uncap preview Review blocker: three files were wholesale copies from the abandoned #3304 branch laid over a two-day-newer base, so they silently reverted work that had landed in between. CI could not see it — deleted tests do not fail. - `audioMixer.test.ts` was byte-identical to #3304's head: 1186 lines against a base of 1353. Gone with it were the `data-playback-start` fallthrough cases from #3322 — merged 54 minutes before this branch's own merge base — and all retiming coverage (`playbackRate` 7 to 0, `atempo` 5 to 0), the strict literal-timing table, and the zero-window cases. - `mediaVolumeEnvelope.test.ts` dropped the trailing-garbage duration case and the plateau-retention case. - `packages/core/package.json` rolled the package version back 0.8.3 to 0.7.109. All three are restored from `main` with only this PR's additions re-applied on top, and the subpath export is regenerated by the repo's own script rather than hand-edited. Also closes the preview/render split the same review raised. Two clamps had to go, not one: `setElementVolume` capped the author gain at the transport, and the first-tick branch in `syncRuntimeMedia` trusted `el.volume` — which is spec-bound to [0,1] and so cannot represent a boost, opening a boosted clip at 0 dB for one tick before the steady-state branch took over. Both pinned by tests, both verified by mutation.
This commit is contained in:
Miguel Ángel
@@ -158,6 +158,12 @@
|
||||
"types": "./dist/audioAutomation.d.ts",
|
||||
"environments": ["browser", "bun", "node"]
|
||||
},
|
||||
"./audio-gain": {
|
||||
"source": "./src/audioGain.ts",
|
||||
"runtime": "./dist/audioGain.js",
|
||||
"types": "./dist/audioGain.d.ts",
|
||||
"environments": ["browser", "bun", "node"]
|
||||
},
|
||||
"./color-grading": {
|
||||
"source": "./src/colorGrading.ts",
|
||||
"runtime": "./dist/colorGrading.js",
|
||||
|
||||
@@ -172,6 +172,12 @@
|
||||
"import": "./src/audioAutomation.ts",
|
||||
"types": "./src/audioAutomation.ts"
|
||||
},
|
||||
"./audio-gain": {
|
||||
"bun": "./src/audioGain.ts",
|
||||
"node": "./dist/audioGain.js",
|
||||
"import": "./src/audioGain.ts",
|
||||
"types": "./src/audioGain.ts"
|
||||
},
|
||||
"./color-grading": {
|
||||
"bun": "./src/colorGrading.ts",
|
||||
"node": "./dist/colorGrading.js",
|
||||
@@ -478,6 +484,10 @@
|
||||
"import": "./dist/audioAutomation.js",
|
||||
"types": "./dist/audioAutomation.d.ts"
|
||||
},
|
||||
"./audio-gain": {
|
||||
"import": "./dist/audioGain.js",
|
||||
"types": "./dist/audioGain.d.ts"
|
||||
},
|
||||
"./color-grading": {
|
||||
"import": "./dist/colorGrading.js",
|
||||
"types": "./dist/colorGrading.d.ts"
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
AUDIO_GAIN_FADER_MAX,
|
||||
formatAudioGain,
|
||||
AUDIO_GAIN_FADER_MIN,
|
||||
MAX_AUDIO_GAIN,
|
||||
audioGainToFaderPosition,
|
||||
audioGainToText,
|
||||
audioFaderPositionToGain,
|
||||
} from "./audioGain";
|
||||
|
||||
describe("audio gain fader", () => {
|
||||
it("puts unity gain at the physical midpoint", () => {
|
||||
expect(audioGainToFaderPosition(1)).toBe(0);
|
||||
expect(audioFaderPositionToGain(0)).toBe(1);
|
||||
});
|
||||
|
||||
it("provides +12 dB of boost above unity", () => {
|
||||
expect(audioFaderPositionToGain(AUDIO_GAIN_FADER_MAX)).toBeCloseTo(MAX_AUDIO_GAIN, 6);
|
||||
expect(audioGainToText(MAX_AUDIO_GAIN)).toBe("+12.0 dB");
|
||||
});
|
||||
|
||||
it("preserves a true silence endpoint below unity", () => {
|
||||
expect(audioFaderPositionToGain(AUDIO_GAIN_FADER_MIN)).toBe(0);
|
||||
expect(audioGainToText(0)).toBe("-∞ dB");
|
||||
});
|
||||
|
||||
it("pins sub-floor gain to the fader's silence endpoint", () => {
|
||||
expect(audioGainToFaderPosition(0.00001)).toBe(AUDIO_GAIN_FADER_MIN);
|
||||
});
|
||||
|
||||
it("round-trips representative attenuation and boost values", () => {
|
||||
for (const gain of [0.1, 0.5, 1, 2, MAX_AUDIO_GAIN]) {
|
||||
expect(audioFaderPositionToGain(audioGainToFaderPosition(gain))).toBeCloseTo(gain, 6);
|
||||
}
|
||||
});
|
||||
|
||||
describe("formatAudioGain", () => {
|
||||
it("never collapses an audible fader stop onto silence", () => {
|
||||
for (let position = AUDIO_GAIN_FADER_MIN + 1; position <= AUDIO_GAIN_FADER_MAX; position++) {
|
||||
const serialized = formatAudioGain(audioFaderPositionToGain(position));
|
||||
expect(Number(serialized)).toBeGreaterThan(0);
|
||||
}
|
||||
// Only the very bottom of the travel is a real mute.
|
||||
expect(formatAudioGain(audioFaderPositionToGain(AUDIO_GAIN_FADER_MIN))).toBe("0");
|
||||
});
|
||||
|
||||
it("puts the knob back where the user let go of it", () => {
|
||||
for (let position = AUDIO_GAIN_FADER_MIN; position <= AUDIO_GAIN_FADER_MAX; position++) {
|
||||
const written = Number(formatAudioGain(audioFaderPositionToGain(position)));
|
||||
expect(Math.round(audioGainToFaderPosition(written))).toBe(position);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a serialized gain short and inside the ceiling", () => {
|
||||
expect(formatAudioGain(1)).toBe("1");
|
||||
expect(formatAudioGain(0.5)).toBe("0.5");
|
||||
expect(formatAudioGain(99)).toBe(formatAudioGain(MAX_AUDIO_GAIN));
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* Authoring gain for a media clip.
|
||||
*
|
||||
* HTMLMediaElement.volume is limited to 0..1, but HyperFrames' Web Audio
|
||||
* preview and FFmpeg render paths both support gain above unity. Keep the
|
||||
* shared ceiling here so Studio, preview, and render cannot drift.
|
||||
*/
|
||||
export const MAX_AUDIO_GAIN_DB = 12;
|
||||
export const MAX_AUDIO_GAIN = 10 ** (MAX_AUDIO_GAIN_DB / 20);
|
||||
|
||||
/** Studio fader coordinates. Unity is deliberately the physical midpoint. */
|
||||
export const AUDIO_GAIN_FADER_MIN = -100;
|
||||
export const AUDIO_GAIN_FADER_MAX = 100;
|
||||
|
||||
const MIN_AUDIO_GAIN_DB = -60;
|
||||
|
||||
export function clampAudioGain(value: number): number {
|
||||
if (!Number.isFinite(value)) return 1;
|
||||
return Math.max(0, Math.min(MAX_AUDIO_GAIN, value));
|
||||
}
|
||||
|
||||
export function clampNativeMediaVolume(value: number): number {
|
||||
if (!Number.isFinite(value)) return 1;
|
||||
return Math.max(0, Math.min(1, value));
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize an authored gain for `data-volume`.
|
||||
*
|
||||
* The fader travels in dB, so its stops are irrational (position -70 is
|
||||
* 10 ** (-42/20)). Rounding to two decimals — what the generic numeric
|
||||
* attribute formatter does — collapses the whole bottom of the fader onto
|
||||
* `"0"` (a hard mute) and makes the knob jump on release everywhere below
|
||||
* unity. Six decimals round-trip every integer fader stop back to itself.
|
||||
*/
|
||||
export function formatAudioGain(gain: number): string {
|
||||
return clampAudioGain(gain)
|
||||
.toFixed(6)
|
||||
.replace(/\.?0+$/, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `probe` with `el.volume` shadowed by an accessor that keeps the authored
|
||||
* value instead of the spec's [0,1] clamp.
|
||||
*
|
||||
* `HTMLMediaElement.volume` cannot hold gain above unity, so a clip authored
|
||||
* at `data-volume="1.95"` reads back as 1 the moment the probe seeds it — and
|
||||
* a GSAP tween started from that seed fades from 0 dB rather than from the
|
||||
* authored boost. Both the FFmpeg mixer and the Web Audio transport carry gain
|
||||
* up to MAX_AUDIO_GAIN, so the clamp is a probe artefact, not a real ceiling.
|
||||
* The native setter still receives the clamped value, so nothing outside the
|
||||
* probe observes an out-of-range volume, and the shadow is removed afterwards.
|
||||
*/
|
||||
export function withUnclampedVolume<T>(el: HTMLMediaElement, probe: () => T): T {
|
||||
// Guarded for non-DOM runtimes: the probe that calls this is also reachable
|
||||
// from tests and tools that run outside a browser, where the clamped path is
|
||||
// the right (and only) answer.
|
||||
const descriptor =
|
||||
typeof HTMLMediaElement === "undefined"
|
||||
? undefined
|
||||
: Object.getOwnPropertyDescriptor(HTMLMediaElement.prototype, "volume");
|
||||
const nativeGet = descriptor?.get;
|
||||
const nativeSet = descriptor?.set;
|
||||
if (!nativeGet || !nativeSet) return probe();
|
||||
|
||||
let authored = Number(nativeGet.call(el));
|
||||
Object.defineProperty(el, "volume", {
|
||||
configurable: true,
|
||||
get: () => authored,
|
||||
set: (value: number) => {
|
||||
authored = Number(value);
|
||||
nativeSet.call(el, clampNativeMediaVolume(authored));
|
||||
},
|
||||
});
|
||||
try {
|
||||
return probe();
|
||||
} finally {
|
||||
delete (el as unknown as Record<"volume", unknown>).volume;
|
||||
nativeSet.call(el, clampNativeMediaVolume(authored));
|
||||
}
|
||||
}
|
||||
|
||||
export function audioFaderPositionToGain(position: number): number {
|
||||
const safe = Math.max(AUDIO_GAIN_FADER_MIN, Math.min(AUDIO_GAIN_FADER_MAX, position));
|
||||
if (safe === AUDIO_GAIN_FADER_MIN) return 0;
|
||||
const db =
|
||||
safe < 0
|
||||
? (safe / Math.abs(AUDIO_GAIN_FADER_MIN)) * Math.abs(MIN_AUDIO_GAIN_DB)
|
||||
: (safe / AUDIO_GAIN_FADER_MAX) * MAX_AUDIO_GAIN_DB;
|
||||
return 10 ** (db / 20);
|
||||
}
|
||||
|
||||
export function audioGainToFaderPosition(gain: number): number {
|
||||
const safe = clampAudioGain(gain);
|
||||
if (safe === 0) return AUDIO_GAIN_FADER_MIN;
|
||||
const db = 20 * Math.log10(safe);
|
||||
const position =
|
||||
db < 0
|
||||
? (db / Math.abs(MIN_AUDIO_GAIN_DB)) * Math.abs(AUDIO_GAIN_FADER_MIN)
|
||||
: (db / MAX_AUDIO_GAIN_DB) * AUDIO_GAIN_FADER_MAX;
|
||||
return Math.max(AUDIO_GAIN_FADER_MIN, Math.min(AUDIO_GAIN_FADER_MAX, position));
|
||||
}
|
||||
|
||||
export function audioGainToText(gain: number): string {
|
||||
const safe = clampAudioGain(gain);
|
||||
if (safe === 0) return "-∞ dB";
|
||||
const db = 20 * Math.log10(safe);
|
||||
const rounded = Math.abs(db) < 0.05 ? 0 : db;
|
||||
return (rounded > 0 ? "+" : "") + rounded.toFixed(1) + " dB";
|
||||
}
|
||||
@@ -13,11 +13,16 @@
|
||||
*
|
||||
* ## Why the lane rides a `gain` node
|
||||
*
|
||||
* The obvious home is the track's volume lane, and that cannot work: volume is
|
||||
* 0..1 and `normaliseEnvelope` clamps every keyframe into it, so a volume lane
|
||||
* can only ever attenuate. Lifting a quiet passage needs a `gain` node, which
|
||||
* spans -60..+12 dB — which is what the audio skill means when it calls `gain`
|
||||
* "what an automation lane rides when a track has to move".
|
||||
* The obvious home is the track's volume lane, and the old reason not to use it
|
||||
* — "volume is 0..1, so a lane can only ever attenuate" — is being retired in
|
||||
* stages: `normaliseEnvelope` now clamps to 0..+12 dB, while `VOLUME_RANGE`,
|
||||
* which bounds the lane itself, still stops at unity until the dB fader lands.
|
||||
*
|
||||
* The reason that survives either way is ownership: the volume lane is the
|
||||
* fader the author draws, and a leveller that wrote into it would silently
|
||||
* redraw their envelope. A `gain` node is a separate stage the leveller owns
|
||||
* outright, which is what the audio skill means when it calls `gain` "what an
|
||||
* automation lane rides when a track has to move".
|
||||
*/
|
||||
|
||||
import {
|
||||
|
||||
@@ -490,6 +490,27 @@ describe("syncRuntimeMedia", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("hands the transport an above-unity author gain, uncapped", () => {
|
||||
// The preview terminus. `el.volume` is spec-bound to [0,1] and always will
|
||||
// be, so the boost can only reach the ear through the Web Audio gain node —
|
||||
// which means the author gain handed to the transport must NOT be capped on
|
||||
// the way out, even though the native write beside it is.
|
||||
const clip = createMockClip({ start: 0, end: 10, volume: 1.949845 });
|
||||
const onElementVolume = vi.fn();
|
||||
|
||||
syncRuntimeMedia({
|
||||
clips: [clip],
|
||||
timeSeconds: 1,
|
||||
playing: false,
|
||||
playbackRate: 1,
|
||||
onElementVolume,
|
||||
});
|
||||
|
||||
const [, , authorVolume] = onElementVolume.mock.calls.at(-1) as [unknown, number, number];
|
||||
expect(authorVolume).toBeCloseTo(1.949845, 6);
|
||||
expect(clip.el.volume).toBe(1);
|
||||
});
|
||||
|
||||
it("plays active clip when playing and buffered", () => {
|
||||
const clip = createMockClip({ start: 0, end: 10 });
|
||||
Object.defineProperty(clip.el, "readyState", { value: 4, writable: true });
|
||||
|
||||
@@ -2,6 +2,7 @@ import { swallow } from "./diagnostics";
|
||||
import { interpolateVolumeGain, type VolumeKeyframe } from "./mediaVolumeEnvelope.js";
|
||||
import { elementVolumeLaneGain } from "./audioAutomationVolume.js";
|
||||
import { readElementPlaybackRate, readMediaStart } from "./playbackRate.js";
|
||||
import { clampAudioGain } from "../audioGain.js";
|
||||
export { readElementPlaybackRate, resolveNaturalMediaTimelineDuration } from "./playbackRate.js";
|
||||
|
||||
export function readElementPlaybackStart(el: Element): number {
|
||||
@@ -255,7 +256,7 @@ export function syncRuntimeMedia(params: {
|
||||
}
|
||||
}
|
||||
const userVol = clampVolume(params.userVolume ?? 1);
|
||||
const fallbackAuthorVolume = clampVolume(clip.volume ?? 1);
|
||||
const fallbackAuthorVolume = clampAudioGain(clip.volume ?? 1);
|
||||
const previousRuntimeVolume = lastRuntimeAppliedVolume.get(el);
|
||||
const currentElementVolume = clampVolume(el.volume);
|
||||
|
||||
@@ -273,7 +274,7 @@ export function syncRuntimeMedia(params: {
|
||||
// there is one time base, and this is it.
|
||||
const laneGain = elementVolumeLaneGain(el, params.timeSeconds - clip.start);
|
||||
if (laneGain !== null) {
|
||||
authorVolume = clampVolume(laneGain);
|
||||
authorVolume = clampAudioGain(laneGain);
|
||||
} else if (clip.volumeKeyframes && clip.volumeKeyframes.length > 0) {
|
||||
// Keyframes probed from the GSAP timeline — same source as the renderer.
|
||||
// Use the interpolated envelope value directly; no need to track GSAP changes.
|
||||
@@ -283,7 +284,7 @@ export function syncRuntimeMedia(params: {
|
||||
// and the playback rate — so it only coincides with the envelope's time base
|
||||
// for an untrimmed clip playing at 1x from t=0.
|
||||
const elapsedInClip = params.timeSeconds - clip.start;
|
||||
authorVolume = clampVolume(interpolateVolumeGain(clip.volumeKeyframes, elapsedInClip));
|
||||
authorVolume = clampAudioGain(interpolateVolumeGain(clip.volumeKeyframes, elapsedInClip));
|
||||
} else if (params.isWebAudioRouted?.(el)) {
|
||||
authorVolume = fallbackAuthorVolume;
|
||||
} else if (previousRuntimeVolume === undefined) {
|
||||
@@ -291,9 +292,22 @@ export function syncRuntimeMedia(params: {
|
||||
// to the current time (seekTimelineAndAdapters runs before syncRuntimeMedia),
|
||||
// so el.volume reflects the animated value — trust it rather than falling
|
||||
// back to data-volume, which would clobber the GSAP-seeked position.
|
||||
authorVolume = currentElementVolume;
|
||||
//
|
||||
// Except above unity. `el.volume` is spec-bound to [0,1], so it cannot
|
||||
// represent an authored boost, and reading it back can only lose the
|
||||
// gain. Without this, a boosted clip opened at 0 dB for one tick and
|
||||
// then jumped once the unchanged-since-last-tick branch below took over
|
||||
// — audible, and invisible to any test that ticks more than once.
|
||||
authorVolume = fallbackAuthorVolume > 1 ? fallbackAuthorVolume : currentElementVolume;
|
||||
} else if (Math.abs(currentElementVolume - previousRuntimeVolume) > 0.0001) {
|
||||
// GSAP (or user code) changed el.volume between ticks — track it.
|
||||
//
|
||||
// Unity-capped on purpose, and it is not a hole in the ceiling: this
|
||||
// reads back through `el.volume`, which the spec pins to [0,1], so it
|
||||
// cannot observe an above-unity value however wide the clamp gets. A
|
||||
// clip whose volume is actually animated takes the probed-keyframes
|
||||
// branch above, which carries the authored gain unclamped; this branch
|
||||
// is the fallback for elements no probe ran on.
|
||||
authorVolume = currentElementVolume;
|
||||
} else {
|
||||
// Volume unchanged since last tick — use data-volume as the baseline.
|
||||
|
||||
@@ -201,4 +201,60 @@ describe("probeAndCacheElementVolume", () => {
|
||||
expect(interpolateVolumeGain(envelope, 0.5)).toBeCloseTo(1, 5);
|
||||
expect(interpolateVolumeGain(envelope, 1)).toBeCloseTo(1, 5);
|
||||
});
|
||||
it("keeps a fade that starts from an above-unity authored gain", () => {
|
||||
const audio = document.createElement("audio");
|
||||
audio.dataset.start = "0";
|
||||
audio.dataset.duration = "2";
|
||||
audio.dataset.volume = "1.949845"; // +5.8 dB
|
||||
|
||||
// A GSAP tween reads the seeded value as its FROM. Through the spec's
|
||||
// [0,1] clamp on `HTMLMediaElement.volume` that read back as 1, so the
|
||||
// whole authored boost was thrown away by the mere presence of a fade.
|
||||
const keyframes = probeElementVolumeKeyframes(
|
||||
audio,
|
||||
(time) => {
|
||||
audio.volume = 1.949845 * Math.max(0, 1 - time / 2);
|
||||
},
|
||||
2,
|
||||
10,
|
||||
);
|
||||
|
||||
expect(keyframes?.[0]?.volume).toBeCloseTo(1.949845, 5);
|
||||
expect(audio.volume).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
it("carries an above-unity tween target through to the envelope", () => {
|
||||
const audio = document.createElement("audio");
|
||||
audio.dataset.start = "0";
|
||||
audio.dataset.duration = "1";
|
||||
audio.dataset.volume = "1";
|
||||
|
||||
const keyframes = probeElementVolumeKeyframes(
|
||||
audio,
|
||||
(time) => {
|
||||
audio.volume = 1 + time;
|
||||
},
|
||||
1,
|
||||
10,
|
||||
);
|
||||
|
||||
expect(keyframes?.at(-1)?.volume).toBeCloseTo(2, 5);
|
||||
});
|
||||
|
||||
it("restores the native accessor once the probe is done", () => {
|
||||
const audio = document.createElement("audio");
|
||||
audio.dataset.start = "0";
|
||||
audio.dataset.duration = "1";
|
||||
audio.dataset.volume = "2";
|
||||
|
||||
probeElementVolumeKeyframes(audio, () => {}, 1, 10);
|
||||
|
||||
// The own accessor is gone and the spec setter is back in charge: it
|
||||
// rejects an out-of-range volume rather than silently taking it.
|
||||
expect(Object.getOwnPropertyDescriptor(audio, "volume")).toBeUndefined();
|
||||
expect(audio.volume).toBe(1);
|
||||
expect(() => {
|
||||
audio.volume = 5;
|
||||
}).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { RuntimeTimelineLike } from "./types";
|
||||
import { clampAudioGain, withUnclampedVolume } from "../audioGain.js";
|
||||
import { parseStrictFiniteTimingNumber } from "./playbackRate";
|
||||
|
||||
/**
|
||||
@@ -32,7 +33,7 @@ export function normaliseEnvelope(
|
||||
.filter((k) => Number.isFinite(k.time) && Number.isFinite(k.volume))
|
||||
.map((k) => ({
|
||||
time: Math.max(0, k.time - trackStart),
|
||||
volume: Math.max(0, Math.min(1, k.volume)),
|
||||
volume: clampAudioGain(k.volume),
|
||||
}))
|
||||
.sort((a, b) => a.time - b.time);
|
||||
|
||||
@@ -48,7 +49,7 @@ export function normaliseEnvelope(
|
||||
|
||||
if (deduped.length === 0) return deduped;
|
||||
if (deduped[0]!.time > 0) {
|
||||
deduped.unshift({ time: 0, volume: Math.max(0, Math.min(1, baseVolume)) });
|
||||
deduped.unshift({ time: 0, volume: clampAudioGain(baseVolume) });
|
||||
}
|
||||
return deduped;
|
||||
}
|
||||
@@ -115,8 +116,7 @@ function resolveVolumeProbeWindow(
|
||||
end = endAttr;
|
||||
}
|
||||
const staticAttr = parseVolumeNumber(el.dataset.volume) ?? 1;
|
||||
const staticVolume = Math.max(0, Math.min(1, staticAttr));
|
||||
return { start, end, staticVolume };
|
||||
return { start, end, staticVolume: clampAudioGain(staticAttr) };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -137,29 +137,32 @@ export function probeElementVolumeKeyframes(
|
||||
): VolumeKeyframe[] | null {
|
||||
const { start, end, staticVolume } = resolveVolumeProbeWindow(el, compositionDuration);
|
||||
|
||||
// Reset to data-volume so GSAP captures the correct FROM value.
|
||||
el.volume = staticVolume;
|
||||
|
||||
const step = 1 / Math.min(60, Math.max(1, sampleFps));
|
||||
const sampleStart = Math.max(0, start);
|
||||
const sampleEnd = Math.min(compositionDuration, end);
|
||||
|
||||
const keyframes: VolumeKeyframe[] = [];
|
||||
let previousSample: VolumeKeyframe | undefined;
|
||||
for (let t = sampleStart; t <= sampleEnd + 1e-6; t = Math.min(sampleEnd, t + step)) {
|
||||
seekTimeline(t);
|
||||
const raw = Number(el.volume);
|
||||
if (Number.isFinite(raw)) {
|
||||
const volume = Math.max(0, Math.min(1, raw));
|
||||
const sample = {
|
||||
time: Number(t.toFixed(6)),
|
||||
volume: Number(volume.toFixed(6)),
|
||||
};
|
||||
recordVolumeSample(keyframes, previousSample, sample, t === sampleEnd);
|
||||
previousSample = sample;
|
||||
const keyframes: VolumeKeyframe[] = withUnclampedVolume(el, () => {
|
||||
// Reset to data-volume so GSAP captures the correct FROM value. Above
|
||||
// unity that only survives because the shadow accessor is installed.
|
||||
el.volume = staticVolume;
|
||||
|
||||
const samples: VolumeKeyframe[] = [];
|
||||
let previousSample: VolumeKeyframe | undefined;
|
||||
for (let t = sampleStart; t <= sampleEnd + 1e-6; t = Math.min(sampleEnd, t + step)) {
|
||||
seekTimeline(t);
|
||||
const raw = Number(el.volume);
|
||||
if (Number.isFinite(raw)) {
|
||||
const sample = {
|
||||
time: Number(t.toFixed(6)),
|
||||
volume: Number(clampAudioGain(raw).toFixed(6)),
|
||||
};
|
||||
recordVolumeSample(samples, previousSample, sample, t === sampleEnd);
|
||||
previousSample = sample;
|
||||
}
|
||||
if (t === sampleEnd) break;
|
||||
}
|
||||
if (t === sampleEnd) break;
|
||||
}
|
||||
return samples;
|
||||
});
|
||||
|
||||
const hasAutomation = keyframes.some((kf) => Math.abs(kf.volume - staticVolume) > 0.0001);
|
||||
return hasAutomation ? keyframes : null;
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
// fallow-ignore-file code-duplication complexity
|
||||
import { beforeEach, describe, it, expect, vi } from "vitest";
|
||||
import { MAX_AUDIO_GAIN } from "../audioGain.js";
|
||||
import { WebAudioTransport } from "./webAudioTransport";
|
||||
|
||||
function createMockAudioContext(currentTime = 100) {
|
||||
@@ -59,6 +60,42 @@ const mockEl = {
|
||||
getAttribute: (name: string) => (name === "data-playback-rate" ? "1" : null),
|
||||
} as unknown as HTMLMediaElement;
|
||||
|
||||
describe("WebAudioTransport author gain vs user volume", () => {
|
||||
it("carries a static above-unity author gain onto the element gain node", () => {
|
||||
// The regression this ceiling exists to prevent: a static `data-volume`
|
||||
// above unity was capped at 1 here while the render honoured it, so preview
|
||||
// and render disagreed on every boosted clip. Automation lanes hid it —
|
||||
// they schedule ramps onto the param directly and never pass through here.
|
||||
const { transport, mock } = setupTransport();
|
||||
const el = { muted: false } as HTMLMediaElement;
|
||||
transport["_activeSources"] = [{ el, gainNode: mock.gainNode, sourceKind: "buffer" }] as never;
|
||||
|
||||
transport.setElementVolume(el, 1.949845);
|
||||
|
||||
expect(mock.gainNode.gain.value).toBeCloseTo(1.949845, 6);
|
||||
});
|
||||
|
||||
it("still refuses a gain beyond the shared ceiling", () => {
|
||||
const { transport, mock } = setupTransport();
|
||||
const el = { muted: false } as HTMLMediaElement;
|
||||
transport["_activeSources"] = [{ el, gainNode: mock.gainNode, sourceKind: "buffer" }] as never;
|
||||
|
||||
transport.setElementVolume(el, 99);
|
||||
|
||||
expect(mock.gainNode.gain.value).toBeCloseTo(MAX_AUDIO_GAIN, 6);
|
||||
});
|
||||
|
||||
it("keeps the user's master volume spec-clamped — it is a fader, not a gain", () => {
|
||||
const transport = new WebAudioTransport();
|
||||
const master = { gain: { value: 1 }, connect: vi.fn() };
|
||||
(transport as unknown as { _masterGain: unknown })._masterGain = master;
|
||||
|
||||
transport.setVolume(99);
|
||||
|
||||
expect(master.gain.value).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("WebAudioTransport", () => {
|
||||
beforeEach(() => {
|
||||
mockEl.muted = false;
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
} from "../audio/audioFxAutomation.js";
|
||||
import { VOLUME_RANGE } from "../audioAutomation.js";
|
||||
import { swallow } from "./diagnostics";
|
||||
import { clampAudioGain } from "../audioGain.js";
|
||||
import { getDebugSurface } from "./globals.js";
|
||||
import { readElementPlaybackRate } from "./media.js";
|
||||
|
||||
@@ -449,8 +450,19 @@ export class WebAudioTransport {
|
||||
this.applyMasterGain();
|
||||
}
|
||||
|
||||
/**
|
||||
* The per-element gain carries the clip's AUTHOR gain, which reaches
|
||||
* MAX_AUDIO_GAIN — so it is clamped against that ceiling, not the spec's
|
||||
* [0,1]. `setVolume` above is the opposite case and stays spec-clamped: the
|
||||
* user's master volume is a fader, not a gain.
|
||||
*
|
||||
* Clamping this one at unity capped every static above-unity `data-volume` on
|
||||
* the preview path while the render honoured it — the exact preview/render
|
||||
* divergence this ceiling exists to close. Automation lanes hid it, because
|
||||
* they schedule ramps onto the param directly and never pass through here.
|
||||
*/
|
||||
setElementVolume(el: HTMLMediaElement, volume: number): void {
|
||||
const safeVolume = Math.max(0, Math.min(1, volume));
|
||||
const safeVolume = clampAudioGain(volume);
|
||||
for (const source of this._activeSources) {
|
||||
if (source.el !== el) continue;
|
||||
try {
|
||||
|
||||
@@ -1198,6 +1198,67 @@ describe("processCompositionAudio", () => {
|
||||
expect(result.error).toBeUndefined();
|
||||
expect(runFfmpegMock.mock.calls[0]?.[0]).toContain(join(baseDir, ".media", "tone.wav"));
|
||||
});
|
||||
|
||||
it("preserves authored clip gain above unity for quiet-source boosting", async () => {
|
||||
const baseDir = mkdtempSync(join(tmpdir(), "hf-audio-base-"));
|
||||
const workDir = mkdtempSync(join(tmpdir(), "hf-audio-work-"));
|
||||
tempDirs.push(baseDir, workDir);
|
||||
writeFileSync(join(baseDir, "quiet.wav"), "stub");
|
||||
|
||||
const result = await processCompositionAudio(
|
||||
[
|
||||
{
|
||||
id: "quiet",
|
||||
src: "quiet.wav",
|
||||
start: 0,
|
||||
end: 2,
|
||||
mediaStart: 0,
|
||||
layer: 0,
|
||||
volume: 3.98,
|
||||
type: "audio",
|
||||
},
|
||||
],
|
||||
baseDir,
|
||||
workDir,
|
||||
join(baseDir, "out.m4a"),
|
||||
2,
|
||||
);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(capturedFilterScripts[1]).toContain("volume=3.98");
|
||||
});
|
||||
|
||||
it("clamps an out-of-range gain to the shared authoring ceiling", async () => {
|
||||
const baseDir = mkdtempSync(join(tmpdir(), "hf-audio-base-"));
|
||||
const workDir = mkdtempSync(join(tmpdir(), "hf-audio-work-"));
|
||||
tempDirs.push(baseDir, workDir);
|
||||
writeFileSync(join(baseDir, "quiet.wav"), "stub");
|
||||
|
||||
const result = await processCompositionAudio(
|
||||
[
|
||||
{
|
||||
id: "quiet",
|
||||
src: "quiet.wav",
|
||||
start: 0,
|
||||
end: 2,
|
||||
mediaStart: 0,
|
||||
layer: 0,
|
||||
volume: 99,
|
||||
type: "audio",
|
||||
},
|
||||
],
|
||||
baseDir,
|
||||
workDir,
|
||||
join(baseDir, "out.m4a"),
|
||||
2,
|
||||
);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
// Pins the UPPER bound: the 3.98 case above only proves the clamp is not
|
||||
// min(1, ...). Without this, changing MAX_AUDIO_GAIN's effect in the mixer
|
||||
// leaves this suite green.
|
||||
expect(capturedFilterScripts[1]).toContain("volume=3.981072");
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseAudioElements — relative data-start resolution", () => {
|
||||
|
||||
@@ -10,6 +10,7 @@ import { join, dirname } from "path";
|
||||
import { parseHTML } from "linkedom";
|
||||
import { extractAudioMetadata } from "../utils/ffprobe.js";
|
||||
import { isNotMediaPayload } from "../utils/notMediaPayload.js";
|
||||
import { clampAudioGain } from "@hyperframes/core/audio-gain";
|
||||
import {
|
||||
downloadToTemp,
|
||||
isHttpUrl,
|
||||
@@ -66,8 +67,7 @@ export type { AudioElement, MixResult } from "./audioMixer.types.js";
|
||||
export const MIXED_AUDIO_FILENAME = "audio.m4a";
|
||||
|
||||
function clampVolume(volume: number): number {
|
||||
if (!Number.isFinite(volume)) return 1;
|
||||
return Math.max(0, Math.min(1, volume));
|
||||
return clampAudioGain(volume);
|
||||
}
|
||||
|
||||
function formatFilterNumber(value: number): string {
|
||||
|
||||
@@ -160,7 +160,7 @@ export interface TimelineMediaElement extends TimelineElementBase {
|
||||
isAroll?: boolean;
|
||||
sourceWidth?: number;
|
||||
sourceHeight?: number;
|
||||
volume?: number; // 0-1 (0% to 100%), default 1.0
|
||||
volume?: number; // linear gain; 0 is silent, 1 is 0 dB, values above 1 boost
|
||||
hasAudio?: boolean; // For videos - indicates if video has audio track
|
||||
}
|
||||
|
||||
|
||||
@@ -27,6 +27,7 @@ import {
|
||||
type ResolvedDuration,
|
||||
type UnresolvedElement,
|
||||
} from "@hyperframes/core";
|
||||
import { MAX_AUDIO_GAIN } from "@hyperframes/core/audio-gain";
|
||||
import {
|
||||
assignBundledRuntimeCompositionIds,
|
||||
type BundledHostCompositionIdentity,
|
||||
@@ -2256,8 +2257,44 @@ export async function discoverAudioVolumeAutomationFromTimeline(
|
||||
return { id, start, end };
|
||||
});
|
||||
return page.evaluate(
|
||||
({ clips, duration, step }) => {
|
||||
({ clips, duration, step, maxGain }) => {
|
||||
const results: { id: string; keyframes: { time: number; volume: number }[] }[] = [];
|
||||
const clampGain = (value: number) =>
|
||||
Number.isFinite(value) ? Math.max(0, Math.min(maxGain, value)) : 1;
|
||||
// `HTMLMediaElement.volume` is spec-clamped to [0,1], so a clip authored
|
||||
// above unity — or a GSAP tween seeded from one — reads back as 0 dB and
|
||||
// the whole authored boost is lost from the mix. Shadow the accessor for
|
||||
// the probe so the authored value survives; the native setter still gets
|
||||
// the clamped value. Mirrors `withUnclampedVolume` in
|
||||
// packages/core/src/audioGain.ts, which the preview probe uses; this copy
|
||||
// exists only because the probe body is serialized into the page.
|
||||
//
|
||||
// Guarded because this body is also executed directly by tests that stand
|
||||
// in for a Page, where there is no DOM and no HTMLMediaElement.
|
||||
const volumeDescriptor =
|
||||
typeof HTMLMediaElement === "undefined"
|
||||
? undefined
|
||||
: Object.getOwnPropertyDescriptor(HTMLMediaElement.prototype, "volume");
|
||||
const nativeVolumeGet = volumeDescriptor?.get;
|
||||
const nativeVolumeSet = volumeDescriptor?.set;
|
||||
const withUnclampedVolume = <T>(el: HTMLMediaElement, probe: () => T): T => {
|
||||
if (!nativeVolumeGet || !nativeVolumeSet) return probe();
|
||||
let authored = Number(nativeVolumeGet.call(el));
|
||||
Object.defineProperty(el, "volume", {
|
||||
configurable: true,
|
||||
get: () => authored,
|
||||
set: (value: number) => {
|
||||
authored = Number(value);
|
||||
nativeVolumeSet.call(el, Math.max(0, Math.min(1, authored)));
|
||||
},
|
||||
});
|
||||
try {
|
||||
return probe();
|
||||
} finally {
|
||||
delete (el as unknown as Record<"volume", unknown>).volume;
|
||||
nativeVolumeSet.call(el, Math.max(0, Math.min(1, authored)));
|
||||
}
|
||||
};
|
||||
const timelines = (window as unknown as { __timelines?: Record<string, unknown> })
|
||||
.__timelines;
|
||||
if (!timelines) return results;
|
||||
@@ -2290,42 +2327,46 @@ export async function discoverAudioVolumeAutomationFromTimeline(
|
||||
const sampleStart = Math.max(0, start);
|
||||
const sampleEnd = Math.min(duration, end);
|
||||
const initialVolumeAttr = Number.parseFloat(el.dataset.volume ?? "");
|
||||
if (Number.isFinite(initialVolumeAttr)) {
|
||||
el.volume = Math.max(0, Math.min(1, initialVolumeAttr));
|
||||
}
|
||||
|
||||
const keyframes: { time: number; volume: number }[] = [];
|
||||
let previousSample: { time: number; volume: number } | undefined;
|
||||
for (let t = sampleStart; t <= sampleEnd + 0.000001; t = Math.min(sampleEnd, t + step)) {
|
||||
seekTl(t);
|
||||
const rawVolume = Number(el.volume);
|
||||
if (!Number.isFinite(rawVolume)) {
|
||||
if (t === sampleEnd) break;
|
||||
continue;
|
||||
const keyframes = withUnclampedVolume(el, () => {
|
||||
if (Number.isFinite(initialVolumeAttr)) {
|
||||
el.volume = clampGain(initialVolumeAttr);
|
||||
}
|
||||
const volume = Math.max(0, Math.min(1, rawVolume));
|
||||
const sample = {
|
||||
time: Number(t.toFixed(6)),
|
||||
volume: Number(volume.toFixed(6)),
|
||||
};
|
||||
const last = keyframes.at(-1);
|
||||
if (!last || Math.abs(last.volume - volume) > 0.0001) {
|
||||
// Retain the preceding real sample when compression omitted a flat
|
||||
// run. Continuous ramps already have that sample as their last
|
||||
// keyframe, so their interpolation remains unchanged.
|
||||
if (last && previousSample && previousSample.time > last.time) {
|
||||
keyframes.push(previousSample);
|
||||
|
||||
const keyframes: { time: number; volume: number }[] = [];
|
||||
let previousSample: { time: number; volume: number } | undefined;
|
||||
for (let t = sampleStart; t <= sampleEnd + 0.000001; t = Math.min(sampleEnd, t + step)) {
|
||||
seekTl(t);
|
||||
const rawVolume = Number(el.volume);
|
||||
if (!Number.isFinite(rawVolume)) {
|
||||
if (t === sampleEnd) break;
|
||||
continue;
|
||||
}
|
||||
keyframes.push(sample);
|
||||
} else if (t === sampleEnd && sample.time > last.time) {
|
||||
keyframes.push(sample);
|
||||
const volume = clampGain(rawVolume);
|
||||
const sample = {
|
||||
time: Number(t.toFixed(6)),
|
||||
volume: Number(volume.toFixed(6)),
|
||||
};
|
||||
const last = keyframes.at(-1);
|
||||
if (!last || Math.abs(last.volume - volume) > 0.0001) {
|
||||
// Retain the preceding real sample when compression omitted a flat
|
||||
// run. Continuous ramps already have that sample as their last
|
||||
// keyframe, so their interpolation remains unchanged.
|
||||
if (last && previousSample && previousSample.time > last.time) {
|
||||
keyframes.push(previousSample);
|
||||
}
|
||||
keyframes.push(sample);
|
||||
} else if (t === sampleEnd && sample.time > last.time) {
|
||||
keyframes.push(sample);
|
||||
}
|
||||
previousSample = sample;
|
||||
if (t === sampleEnd) break;
|
||||
}
|
||||
previousSample = sample;
|
||||
if (t === sampleEnd) break;
|
||||
}
|
||||
return keyframes;
|
||||
});
|
||||
|
||||
const staticAttr = Number.parseFloat(el.dataset.volume ?? "");
|
||||
const staticVolume = Number.isFinite(staticAttr) ? Math.max(0, Math.min(1, staticAttr)) : 1;
|
||||
const staticVolume = Number.isFinite(staticAttr) ? clampGain(staticAttr) : 1;
|
||||
const hasAutomation = keyframes.some(
|
||||
(keyframe) => Math.abs(keyframe.volume - staticVolume) > 0.0001,
|
||||
);
|
||||
@@ -2337,7 +2378,7 @@ export async function discoverAudioVolumeAutomationFromTimeline(
|
||||
seekTl(0);
|
||||
return results;
|
||||
},
|
||||
{ clips, duration: compositionDuration, step: sampleStep },
|
||||
{ clips, duration: compositionDuration, step: sampleStep, maxGain: MAX_AUDIO_GAIN },
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -153,6 +153,41 @@ describe("FlatMediaSection — volume/rate/media-start", () => {
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("refuses to commit from the percent slider on a clip authored above unity", () => {
|
||||
// The control tops out at 100%, so any commit from it would cap a boosted
|
||||
// clip and silently drop up to 12 dB that now genuinely renders. Held until
|
||||
// the dB fader that can represent these levels replaces it.
|
||||
const onSetAttribute = vi.fn();
|
||||
const element = makeVideoElement({ dataAttributes: { volume: "1.949845" } });
|
||||
const host = document.createElement("div");
|
||||
document.body.append(host);
|
||||
const root = createRoot(host);
|
||||
act(() => {
|
||||
root.render(
|
||||
<FlatMediaSection
|
||||
projectDir={null}
|
||||
element={element}
|
||||
styles={{}}
|
||||
onSetStyle={vi.fn()}
|
||||
onSetAttribute={onSetAttribute}
|
||||
onSetHtmlAttribute={vi.fn()}
|
||||
/>,
|
||||
);
|
||||
});
|
||||
|
||||
const volumeTrack = host.querySelectorAll('[data-flat-slider-track="true"]')[0];
|
||||
Object.defineProperty(volumeTrack, "getBoundingClientRect", {
|
||||
value: () => ({ left: 0, width: 100, top: 0, height: 2, right: 100, bottom: 2 }),
|
||||
});
|
||||
act(() => {
|
||||
volumeTrack.dispatchEvent(new MouseEvent("pointerdown", { bubbles: true, clientX: 50 }));
|
||||
volumeTrack.dispatchEvent(new MouseEvent("pointerup", { bubbles: true, clientX: 50 }));
|
||||
});
|
||||
|
||||
expect(onSetAttribute).not.toHaveBeenCalled();
|
||||
act(() => root.unmount());
|
||||
});
|
||||
|
||||
it("commits a new volume value on slider track pointerdown", () => {
|
||||
const onSetAttribute = vi.fn();
|
||||
const element = makeVideoElement({ dataAttributes: { volume: "0.2" } });
|
||||
|
||||
@@ -207,7 +207,13 @@ export function FlatMediaSection({
|
||||
<>
|
||||
{/* The slider is disabled while a lane owns the level: a value set
|
||||
here would be overwritten by the envelope on the next tick. The
|
||||
toggle beside it carries the tooltip. */}
|
||||
toggle beside it carries the tooltip.
|
||||
|
||||
It is also disabled above unity, for the same reason in a
|
||||
different guise — this control tops out at 100%, so committing
|
||||
from it would silently cap a boosted clip and drop up to 12 dB
|
||||
that now genuinely renders. A hold, not a fix: the dB fader that
|
||||
can represent these levels replaces this control outright. */}
|
||||
<div
|
||||
className="hf-volume-row flex items-center gap-1"
|
||||
data-volume-automated={volumeAutomated ? "" : undefined}
|
||||
@@ -220,7 +226,7 @@ export function FlatMediaSection({
|
||||
max={100}
|
||||
tier={volumePercent === 100 ? "default" : "explicitCustom"}
|
||||
displayValue={`${volumePercent}%`}
|
||||
disabled={volumeAutomated}
|
||||
disabled={volumeAutomated || volume > 1}
|
||||
onCommit={(next) => void onSetAttribute("volume", formatNumericValue(next / 100))}
|
||||
/>
|
||||
</div>
|
||||
|
||||
@@ -246,6 +246,10 @@ export function MediaSection({
|
||||
|
||||
{(isVideo || isAudio) && (
|
||||
<>
|
||||
{/* Held above unity: this control tops out at 100%, so committing
|
||||
from it would silently cap a boosted clip and drop up to 12 dB
|
||||
that now genuinely renders. The dB fader that can represent
|
||||
these levels replaces this control outright. */}
|
||||
<div className="grid min-w-0 gap-1.5">
|
||||
<span className={LABEL}>Volume</span>
|
||||
<SliderControl
|
||||
@@ -254,6 +258,7 @@ export function MediaSection({
|
||||
min={0}
|
||||
max={100}
|
||||
step={1}
|
||||
disabled={volume > 1}
|
||||
displayValue={`${volumePercent}%`}
|
||||
formatDisplayValue={(next) => `${Math.round(next)}%`}
|
||||
onCommit={(next) => {
|
||||
|
||||
Reference in New Issue
Block a user