ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-01 19:42:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: remove hidden audio gain in renders (#362)

Browse Source 
## Summary

This fixes a render-time audio correctness bug where Hyperframes applied a hidden post-mix gain to every rendered output, boosting audio by about +2.6 dB and causing clipping on normally leveled sources.

It also fixes a related mute bug where `data-volume="0"` was treated as falsy and silently converted back to full volume during audio track preparation.

Additionally, this PR fixes the Studio workspace typecheck path for `@hyperframes/player`, so local pre-commit/typecheck flows no longer depend on the Player package having been built first.

## Root Cause

The issue report measured a near-constant gain increase and suspected a hidden normalization step. After tracing the engine audio path, the root cause turned out to be explicit code, not FFmpeg behavior:

- `packages/engine/src/config.ts` defaulted `audioGain` to `1.35`
- `packages/engine/src/services/audioMixer.ts` always appended a post-mix FFmpeg filter:
  - `[mixed]volume=${masterOutputGain}[out]`
- with the default config, that meant every render got multiplied by `1.35`

That exactly matches the issue reporter's measured scalar boost.

While investigating the workaround, I also found a second correctness bug:

- `processCompositionAudio()` used `element.volume || 1.0`
- that coerced `0` to `1.0`
- so `data-volume="0"` did not actually mute the track in rendered output

Separately, the repo-level Studio typecheck could fail before any build step because:

- `packages/studio/src/player/components/Player.tsx` imports `@hyperframes/player`
- `packages/player/package.json` points TypeScript at built `dist/*` outputs
- in a fresh workspace, those built outputs may not exist yet
- Studio therefore failed type resolution for `@hyperframes/player` during pre-commit/typecheck

## What Changed

1. Set the engine default `audioGain` back to unity (`1`)
2. Preserve explicit zero volumes by changing `element.volume || 1.0` to `element.volume ?? 1.0`
3. Added regression coverage for both behaviors
4. Updated the producer-side config fixture to reflect the corrected default
5. Added a Studio tsconfig path mapping for `@hyperframes/player` to the local workspace source and widened `rootDir` so workspace typecheck succeeds without requiring a prior Player build

## Why These Changes Are Needed

This is not a UX preference issue; it is a correctness and API contract issue.

- The docs describe `data-volume` as a direct 0-1 control.
- Rendered output should preserve source levels unless the author explicitly changes them.
- Hidden global gain makes output non-deterministic from the author's perspective.
- `data-volume="0"` must mean silence, not full-volume playback.
- Local workspace typecheck should not require unrelated package build artifacts to exist first.

Leaving the current behavior in place means:

- voice recordings near normal peak levels can clip during render
- authors need undocumented manual compensation (`0.75`-ish scaling) to get unity output
- mute semantics in docs and code diverge
- local pre-commit/typecheck can fail for reasons unrelated to the actual diff being committed

## Testing

### Focused regression tests

Ran:

- `packages/engine/node_modules/.bin/vitest run packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.test.ts`

Result:

- `10 passed`

These tests specifically verify:

- default resolved `audioGain` is `1`
- a track with `volume: 0` stays `volume=0` in the FFmpeg filter graph
- the post-mix output filter stays at unity gain (`[mixed]volume=1[out]`)

### Broader package verification

Ran:

- `bun run --filter @hyperframes/engine test`
- `bun run --filter @hyperframes/engine build`
- `packages/engine/node_modules/.bin/vitest run packages/producer/src/services/renderOrchestrator.test.ts`
- `bun run --filter @hyperframes/producer typecheck`
- `bun run --filter @hyperframes/studio typecheck`
- `bunx oxlint packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.ts packages/engine/src/services/audioMixer.test.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt packages/engine/src/config.ts packages/engine/src/config.test.ts packages/engine/src/services/audioMixer.ts packages/engine/src/services/audioMixer.test.ts packages/producer/src/services/renderOrchestrator.test.ts packages/studio/tsconfig.json`
- `bunx lefthook run pre-commit`

Results:

- full engine test suite passed (`309 passed`)
- engine build passed
- touched producer test file passed (`7 passed`)
- producer typecheck passed
- studio typecheck passed
- oxlint passed with `0 warnings, 0 errors`
- formatting passed
- pre-commit hook no longer hits the prior `@hyperframes/player` module-resolution blocker

## Known Verification Limitation

There is no meaningful browser UI flow for this bug: the defect is in the engine/CLI audio render pipeline rather than an interactive browser surface. Because of that, verification was done at the renderer and test level rather than through an agent-browser flow.

## User Impact

After this change:

- rendered audio matches source level by default
- authors no longer need to compensate for a hidden +2.6 dB boost
- `data-volume="0"` correctly mutes rendered audio
- the documented volume contract matches engine behavior again
- local workspace typecheck no longer depends on prebuilt `@hyperframes/player` artifacts

Closes #361.
This commit is contained in:
Miguel Ángel
2026-04-21 20:33:25 +02:00
committed by GitHub
parent e72bcfaed3
commit b98093aa1c
13 changed files with 307 additions and 249 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/engine/src/config.test.ts
+1
View File
@@ -29,6 +29,7 @@ describe("resolveConfig", () => {
expect(config.quality).toBe("standard");
expect(config.format).toBe("jpeg");
expect(config.jpegQuality).toBe(80);
expect(config.audioGain).toBe(1);
expect(config.debug).toBe(false);
});
packages/engine/src/config.ts
+1 -1
View File
@@ -105,7 +105,7 @@ export const DEFAULT_CONFIG: EngineConfig = {
hdr: false,
hdrAutoDetect: true,
audioGain: 1.35,
audioGain: 1,
frameDataUriCacheLimit: 256,
playerReadyTimeout: 45_000,
packages/engine/src/services/audioMixer.test.ts
+67
View File
@@ -0,0 +1,67 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
const { runFfmpegMock } = vi.hoisted(() => ({
runFfmpegMock: vi.fn(async () => ({
success: true,
durationMs: 1,
stderr: "",
exitCode: 0,
})),
}));
vi.mock("../utils/runFfmpeg.js", () => ({
runFfmpeg: runFfmpegMock,
}));
import { processCompositionAudio } from "./audioMixer.js";
describe("processCompositionAudio", () => {
const tempDirs: string[] = [];
afterEach(() => {
runFfmpegMock.mockClear();
for (const dir of tempDirs.splice(0)) {
rmSync(dir, { recursive: true, force: true });
}
});
it("preserves muted tracks and uses unity master gain by default", async () => {
const baseDir = mkdtempSync(join(tmpdir(), "hf-audio-base-"));
const workDir = mkdtempSync(join(tmpdir(), "hf-audio-work-"));
tempDirs.push(baseDir, workDir);
writeFileSync(join(baseDir, "voice.wav"), "stub");
const result = await processCompositionAudio(
[
{
id: "voice",
src: "voice.wav",
start: 0,
end: 2,
mediaStart: 0,
layer: 0,
volume: 0,
type: "audio",
},
],
baseDir,
workDir,
join(baseDir, "out.m4a"),
2,
);
expect(result.success).toBe(true);
expect(runFfmpegMock).toHaveBeenCalledTimes(2);
const mixArgs = runFfmpegMock.mock.calls[1]?.[0];
const filterIndex = mixArgs.indexOf("-filter_complex");
const filter = mixArgs[filterIndex + 1];
expect(filter).toContain("volume=0");
expect(filter).toContain("[mixed]volume=1[out]");
});
});
packages/engine/src/services/audioMixer.ts
+1 -1
View File
@@ -396,7 +396,7 @@ export async function processCompositionAudio(
end: element.end,
mediaStart: element.mediaStart,
duration: element.end - element.start,
volume: element.volume || 1.0,
volume: element.volume ?? 1.0,
});
} catch (err: unknown) {
errors.push(`Error: ${element.id} — ${err instanceof Error ? err.message : String(err)}`);
packages/producer/src/regression-harness.ts
+2 -61
View File
@@ -18,6 +18,7 @@ import { createRenderJob, executeRenderJob } from "./services/renderOrchestrator
import { compileForRender } from "./services/htmlCompiler.js";
import { validateCompilation } from "./services/compilationTester.js";
import { extractVideoMetadata } from "./utils/ffprobe.js";
import { buildRmsEnvelope, compareAudioEnvelopes } from "./utils/audioRegression.js";
// ── Types ────────────────────────────────────────────────────────────────────
@@ -348,66 +349,6 @@ function extractMonoPcm16(videoPath: string): Int16Array {
}
}
function buildRmsEnvelope(samples: Int16Array, windowSize = 2048, hopSize = 1024): number[] {
if (samples.length < windowSize) return [];
const envelope: number[] = [];
for (let start = 0; start + windowSize <= samples.length; start += hopSize) {
let energy = 0;
for (let i = 0; i < windowSize; i += 1) {
const normalized = (samples[start + i] ?? 0) / 32768;
energy += normalized * normalized;
}
envelope.push(Math.sqrt(energy / windowSize));
}
return envelope;
}
function correlationAtLag(a: number[], b: number[], lag: number): number {
const startA = Math.max(0, lag);
const startB = Math.max(0, -lag);
const length = Math.min(a.length - startA, b.length - startB);
if (length <= 32) return -1;
let meanA = 0;
let meanB = 0;
for (let i = 0; i < length; i += 1) {
meanA += a[startA + i] ?? 0;
meanB += b[startB + i] ?? 0;
}
meanA /= length;
meanB /= length;
let numerator = 0;
let denA = 0;
let denB = 0;
for (let i = 0; i < length; i += 1) {
const da = (a[startA + i] ?? 0) - meanA;
const db = (b[startB + i] ?? 0) - meanB;
numerator += da * db;
denA += da * da;
denB += db * db;
}
if (denA <= 1e-12 || denB <= 1e-12) return -1;
return numerator / Math.sqrt(denA * denB);
}
function bestEnvelopeCorrelation(
rendered: number[],
snapshot: number[],
maxLagWindows: number,
): { correlation: number; lagWindows: number } {
let best = -1;
let bestLag = 0;
for (let lag = -maxLagWindows; lag <= maxLagWindows; lag += 1) {
const corr = correlationAtLag(rendered, snapshot, lag);
if (corr > best) {
best = corr;
bestLag = lag;
}
}
return { correlation: best, lagWindows: bestLag };
}
// ── Failure Reporting ────────────────────────────────────────────────────────
function saveFailureDetails(
@@ -751,7 +692,7 @@ async function runTestSuite(
if (renderedAudio.length > 0 && snapshotAudio.length > 0) {
const renderedEnvelope = buildRmsEnvelope(renderedAudio);
const snapshotEnvelope = buildRmsEnvelope(snapshotAudio);
const audio = bestEnvelopeCorrelation(
const audio = compareAudioEnvelopes(
renderedEnvelope,
snapshotEnvelope,
suite.meta.maxAudioLagWindows,
packages/producer/src/services/renderOrchestrator.test.ts
+1 -1
View File
@@ -203,7 +203,7 @@ describe("applyRenderModeHints", () => {
ffmpegEncodeTimeout: 600000,
ffmpegProcessTimeout: 300000,
ffmpegStreamingTimeout: 600000,
audioGain: 1.35,
audioGain: 1,
frameDataUriCacheLimit: 256,
playerReadyTimeout: 45000,
renderReadyTimeout: 15000,
packages/producer/src/utils/audioRegression.test.ts
+16
View File
@@ -0,0 +1,16 @@
import { describe, expect, it } from "vitest";
import { buildRmsEnvelope, compareAudioEnvelopes } from "./audioRegression.js";
describe("compareAudioEnvelopes", () => {
it("treats silent-vs-silent audio as a perfect match", () => {
const silentSamples = new Int16Array(4096);
const rendered = buildRmsEnvelope(silentSamples);
const snapshot = buildRmsEnvelope(silentSamples);
expect(compareAudioEnvelopes(rendered, snapshot, 120)).toEqual({
correlation: 1,
lagWindows: 0,
});
});
});
packages/producer/src/utils/audioRegression.ts
+79
View File
@@ -0,0 +1,79 @@
export function buildRmsEnvelope(samples: Int16Array, windowSize = 2048, hopSize = 1024): number[] {
if (samples.length < windowSize) return [];
const envelope: number[] = [];
for (let start = 0; start + windowSize <= samples.length; start += hopSize) {
let energy = 0;
for (let i = 0; i < windowSize; i += 1) {
const normalized = (samples[start + i] ?? 0) / 32768;
energy += normalized * normalized;
}
envelope.push(Math.sqrt(energy / windowSize));
}
return envelope;
}
function correlationAtLag(a: number[], b: number[], lag: number): number {
const startA = Math.max(0, lag);
const startB = Math.max(0, -lag);
const length = Math.min(a.length - startA, b.length - startB);
if (length <= 32) return -1;
let meanA = 0;
let meanB = 0;
for (let i = 0; i < length; i += 1) {
meanA += a[startA + i] ?? 0;
meanB += b[startB + i] ?? 0;
}
meanA /= length;
meanB /= length;
let numerator = 0;
let denA = 0;
let denB = 0;
for (let i = 0; i < length; i += 1) {
const da = (a[startA + i] ?? 0) - meanA;
const db = (b[startB + i] ?? 0) - meanB;
numerator += da * db;
denA += da * da;
denB += db * db;
}
if (denA <= 1e-12 || denB <= 1e-12) return -1;
return numerator / Math.sqrt(denA * denB);
}
function bestEnvelopeCorrelation(
rendered: number[],
snapshot: number[],
maxLagWindows: number,
): { correlation: number; lagWindows: number } {
let best = -1;
let bestLag = 0;
for (let lag = -maxLagWindows; lag <= maxLagWindows; lag += 1) {
const corr = correlationAtLag(rendered, snapshot, lag);
if (corr > best) {
best = corr;
bestLag = lag;
}
}
return { correlation: best, lagWindows: bestLag };
}
function isSilentEnvelope(envelope: number[]): boolean {
return envelope.length > 0 && envelope.every((sample) => Math.abs(sample) <= 1e-9);
}
export function compareAudioEnvelopes(
rendered: number[],
snapshot: number[],
maxLagWindows: number,
): { correlation: number; lagWindows: number } {
if (rendered.length === 0 || snapshot.length === 0) {
return { correlation: 1, lagWindows: 0 };
}
if (isSilentEnvelope(rendered) && isSilentEnvelope(snapshot)) {
return { correlation: 1, lagWindows: 0 };
}
return bestEnvelopeCorrelation(rendered, snapshot, maxLagWindows);
}
packages/producer/tests/font-variant-numeric/output/compiled.html
+39 -55
View File
File diff suppressed because one or more lines are too long
packages/producer/tests/font-variant-numeric/output/output.mp4
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d2e098b9bd60796b15b6945a75bf4b7e9bd0939f806e0501ef59922d9522d0a6
size 142963
oid sha256:506ae124493ffc9bfc543eae709106249646069937014ded2d6edbe661ce832a
size 151013
packages/producer/tests/missing-host-comp-id/output/compiled.html
+95 -126
View File
File diff suppressed because one or more lines are too long
packages/producer/tests/missing-host-comp-id/output/output.mp4
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7a2e53926e44d66a469e74eee88035987e309786eb44fef17de9f34114de0813
size 105638
oid sha256:a84e50ad31694432affb1bffd42bde653514662ccb1dbb8172b0eab2d80147d5
size 129943
packages/studio/tsconfig.json
+1
View File
@@ -16,6 +16,7 @@
"declarationMap": true,
"sourceMap": true,
"outDir": "dist",
"rootDir": "..",
"types": ["vite/client"],
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,