mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-08-31 02:41:44 +00:00
fix(producer): mix audio into a container that can record encoder delay (#3200)
* fix(producer): mix audio into a container that can record encoder delay Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz) after its authored `data-start`, against a frame-accurate video track. The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix was written to a raw ADTS `.aac` file, which has nowhere to record that delay, so it decoded as real leading silence and every stage downstream preserved it faithfully. Measuring each intermediate localises it precisely: the source WAV is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and mux stages inherit it unchanged. The filter graph itself is correct - run by hand to PCM it lands on the authored start. Switch the artifact to an MP4-family container, which stores the delay as an edit list that decoders strip. Same codec, same bitrate, so no size or quality change. The filename is a contract shared by three consumers - the mux input, the distributed plan artifact, and the PNG-sequence sidecar handed to users for NLE ingest - and its extension is what selects the muxer. Give it one owner in the engine rather than five literals, so those consumers cannot drift onto different containers. Note for reviewers: this renames the distributed plan's audio artifact, which is an on-disk contract between the plan writer and the assembler. Both move together here, but a plan written by an older build would not be found by a newer assembler. Flagging in case that mixed-version window matters for how these are deployed. * fix(cloud): read the plan audio artifact name from the producer contract The aws-lambda and gcp-cloud-run adapters each restated the plan's audio filename in five places, so renaming it in the producer left them looking for a file that is no longer written. CI caught it: the gcp dispatch test asserting a plan has no audio artifact started seeing one. Export the name from `@hyperframes/producer/distributed` and consume it in both adapters. This is the same failure the constant exists to prevent, one package boundary further out: a literal that drifts from the writer's is a silently missing audio track rather than a loud error, because both call sites only ever ask whether the file exists. * fix(cloud): accept a legacy plan's audio artifact name for one release Review raised a rolling-deploy window I had flagged but left undecided: `plan` and `assemble` are separate invocations bridged by object storage, so a pre-rollout planner can be paired with a post-rollout assembler. Both readers locate the artifact by existence alone, which makes that pairing a silently muted video rather than an error. That is reachable enough to be worth two lines, so reads now accept the old name while writes only ever emit the new one. Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`) rather than four call sites, marked for deletion one release out. Also fixes a hole in the first pass of this: the plan-v2 materializer matched either name but then joined the CURRENT one, so a legacy plan resolved to a path that was never written. It now joins the artifact's own name. Review nits in the same pass: correct the pad-branch docstring, which still described a concat-copy shape the pad branch stopped using when it moved to apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so it cannot model a shape that reintroduces the priming delay. * test(producer): rebake the missing-host-comp-id golden without the audio delay The pinned reference was rendered before this branch, so it carries the 1024 sample encoder-priming delay in its audio. With the delay gone the correct audio now sits ahead of the reference and the harness's envelope correlation drops below its floor. Cross-correlating the old and new references at native 48 kHz gives a lag of exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved by exactly the amount this branch removes. Regenerated inside the CI container (Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches the encoder CI will compare against - the container reproduced CI's failure to the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and passes at correlation 1.0 after it. Note for archaeology: the new reference is also 3 dB louder than the old one. That gap is not from this branch - `main` and this branch render the fixture at the same level - it is pre-existing drift the reference had accumulated, which a scale-invariant correlator could never see. The rebake absorbs it. Only output.mp4 is updated. `--update` also rewrites compiled.html, but that diff is embedded-font churn with no bearing on the comparison, which reports "Failed at compilation: 0" either way. * test(producer): rebake the variables-prod golden without the audio delay Same cause as the missing-host-comp-id rebake, caught by shard-8 once the earlier shard stopped failing and the rest of the matrix could run: this reference also carries the encoder-priming delay this branch removes. Reproduced in the CI container to the digit (correlation 0.42704173048439215, lagWindows -12), rebaked there, and it now passes at correlation 1.0. Worth recording: the shift here is 2048 samples (42.67 ms) at correlation 0.99983, exactly twice the 1024 of the other fixture. The delay compounds once per un-compensated AAC generation, and this fixture's audio needs its duration normalized, so it takes the pad/trim branch's re-encode and picks up a second frame of priming on top of the mixer's. So the pre-fix error was not a fixed 21 ms - it grew with the number of times the audio was re-encoded. All nine shards ran in that CI round with only this one failing, so the matrix has now covered every fixture against this change.
This commit is contained in:
Miguel Ángel
@@ -117,7 +117,7 @@ interface AssembleEventBase {
|
||||
Action: "assemble";
|
||||
/** S3 URIs of every chunk, ordered by chunk index. Length must equal `chunkCount`. */
|
||||
ChunkS3Uris: string[];
|
||||
/** S3 URI of the planDir's `audio.aac` if the composition has audio; `null` otherwise. */
|
||||
/** S3 URI of the planDir's audio artifact if the composition has audio; `null` otherwise. */
|
||||
AudioS3Uri: string | null;
|
||||
/** Final output S3 URI (`s3://bucket/key.mp4`). */
|
||||
OutputS3Uri: string;
|
||||
|
||||
@@ -563,7 +563,7 @@ describe("handler dispatch", () => {
|
||||
);
|
||||
const renderChunkMock = mock(
|
||||
async (planDir: string, _chunkIndex: number, outputPath: string): Promise<ChunkResult> => {
|
||||
expect(existsSync(join(planDir, "audio.aac"))).toBe(false);
|
||||
expect(existsSync(join(planDir, "audio.m4a"))).toBe(false);
|
||||
writeFileSync(outputPath, "V2-CHUNK");
|
||||
return {
|
||||
outputPath,
|
||||
@@ -818,7 +818,7 @@ function makeMinimalV1PlanDir(dir: string, withAudio: boolean): void {
|
||||
JSON.stringify([{ index: 0, startFrame: 0, endFrame: 30 }]),
|
||||
);
|
||||
writeFileSync(join(dir, "meta", "encoder.json"), "{}");
|
||||
if (withAudio) writeFileSync(join(dir, "audio.aac"), "AAC");
|
||||
if (withAudio) writeFileSync(join(dir, "audio.m4a"), "AAC");
|
||||
planJson.planHash = recomputePlanHashFromPlanDir(dir);
|
||||
writeFileSync(join(dir, "plan.json"), JSON.stringify(planJson));
|
||||
}
|
||||
|
||||
@@ -24,6 +24,9 @@ import {
|
||||
listPlanV2ArtifactsForTarget,
|
||||
materializePlanV2Target,
|
||||
plan,
|
||||
isPlanAudioArtifactPath,
|
||||
PLAN_AUDIO_RELATIVE_PATH,
|
||||
resolvePlanAudioPath,
|
||||
planV2WithPublisher,
|
||||
type PlanResult,
|
||||
type PlanV2Artifact,
|
||||
@@ -318,9 +321,11 @@ async function handlePlan(event: PlanEvent, deps?: HandlerDeps): Promise<PlanLam
|
||||
const planTar = join(work, "plan.tar.gz");
|
||||
await tarDirectory(planDir, planTar);
|
||||
const planTarUri = `${trimTrailingSlash(event.PlanOutputS3Prefix)}/plan.tar.gz`;
|
||||
const audioPath = join(planDir, "audio.aac");
|
||||
const audioPath = join(planDir, PLAN_AUDIO_RELATIVE_PATH);
|
||||
const hasAudio = existsSync(audioPath) && statSync(audioPath).size > 0;
|
||||
const audioUri = hasAudio ? `${trimTrailingSlash(event.PlanOutputS3Prefix)}/audio.aac` : null;
|
||||
const audioUri = hasAudio
|
||||
? `${trimTrailingSlash(event.PlanOutputS3Prefix)}/${PLAN_AUDIO_RELATIVE_PATH}`
|
||||
: null;
|
||||
// Plan and audio are independent S3 PUTs; run them in parallel so
|
||||
// the response returns as soon as the slower of the two completes.
|
||||
await Promise.all([
|
||||
@@ -390,7 +395,7 @@ async function handlePlanV2(
|
||||
Width: manifest.width,
|
||||
Height: manifest.height,
|
||||
Format: manifest.format,
|
||||
HasAudio: manifest.artifacts.some((artifact) => artifact.path === "audio.aac"),
|
||||
HasAudio: manifest.artifacts.some((artifact) => isPlanAudioArtifactPath(artifact.path)),
|
||||
AudioS3Uri: null,
|
||||
FfmpegVersion: manifest.ffmpegVersion,
|
||||
ProducerVersion: manifest.producerVersion,
|
||||
@@ -568,7 +573,7 @@ async function handleAssemble(
|
||||
|
||||
let audioPath: string | null = null;
|
||||
if (event.AudioS3Uri) {
|
||||
audioPath = join(planDir, "audio.aac");
|
||||
audioPath = resolvePlanAudioPath(planDir) ?? join(planDir, PLAN_AUDIO_RELATIVE_PATH);
|
||||
await downloadS3ObjectToFile(s3, event.AudioS3Uri, audioPath);
|
||||
}
|
||||
|
||||
@@ -616,7 +621,7 @@ async function handleAssembleV2(
|
||||
const planDir = await downloadAndMaterializePlanV2(s3, event, { role: "assembler" }, work);
|
||||
// `downloadAndMaterializePlanV2` materializes atomically. Audio is
|
||||
// assembler-only and lives at the familiar v1-compatible location.
|
||||
const audioPath = existsSync(join(planDir, "audio.aac")) ? join(planDir, "audio.aac") : null;
|
||||
const audioPath = resolvePlanAudioPath(planDir);
|
||||
const chunkPaths = await downloadChunkObjects(s3, event.ChunkS3Uris, work, event.Format);
|
||||
const finalOutput =
|
||||
event.Format === "png-sequence"
|
||||
|
||||
@@ -213,7 +213,11 @@ export {
|
||||
|
||||
export { createVideoFrameInjector } from "./services/videoFrameInjector.js";
|
||||
|
||||
export { parseAudioElements, processCompositionAudio } from "./services/audioMixer.js";
|
||||
export {
|
||||
MIXED_AUDIO_FILENAME,
|
||||
parseAudioElements,
|
||||
processCompositionAudio,
|
||||
} from "./services/audioMixer.js";
|
||||
export { cloneCaptureWarning, cloneCaptureWarnings } from "./services/captureWarning.js";
|
||||
export type {
|
||||
AudioElement,
|
||||
|
||||
@@ -4,7 +4,7 @@ import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { getFfmpegBinary } from "../utils/ffmpegBinaries.js";
|
||||
import { processCompositionAudio } from "./audioMixer.js";
|
||||
import { MIXED_AUDIO_FILENAME, processCompositionAudio } from "./audioMixer.js";
|
||||
|
||||
const HAS_FFMPEG = spawnSync(getFfmpegBinary(), ["-version"], { encoding: "utf-8" }).status === 0;
|
||||
const tempDirs: string[] = [];
|
||||
@@ -22,6 +22,39 @@ function meanVolumeDb(path: string): number {
|
||||
return Number(match[1]);
|
||||
}
|
||||
|
||||
/** Seconds until the first sample loud enough to be signal rather than codec noise. */
|
||||
function firstAudibleSeconds(path: string): number {
|
||||
const sampleRate = 48_000;
|
||||
const result = spawnSync(
|
||||
getFfmpegBinary(),
|
||||
[
|
||||
"-nostdin",
|
||||
"-v",
|
||||
"error",
|
||||
"-i",
|
||||
path,
|
||||
"-map",
|
||||
"0:a",
|
||||
"-ac",
|
||||
"1",
|
||||
"-ar",
|
||||
String(sampleRate),
|
||||
"-f",
|
||||
"s16le",
|
||||
"-",
|
||||
],
|
||||
{ maxBuffer: 1 << 28 },
|
||||
);
|
||||
if (result.status !== 0) {
|
||||
throw new Error(`Could not decode ${path}: ${result.stderr?.toString()}`);
|
||||
}
|
||||
const pcm = result.stdout;
|
||||
for (let i = 0; i < pcm.length / 2; i += 1) {
|
||||
if (Math.abs(pcm.readInt16LE(i * 2)) > 512) return i / sampleRate;
|
||||
}
|
||||
throw new Error(`No audible sample found in ${path}`);
|
||||
}
|
||||
|
||||
describe.skipIf(!HAS_FFMPEG)("processCompositionAudio levels", () => {
|
||||
afterEach(() => {
|
||||
for (const dir of tempDirs.splice(0)) rmSync(dir, { recursive: true, force: true });
|
||||
@@ -75,4 +108,57 @@ describe.skipIf(!HAS_FFMPEG)("processCompositionAudio levels", () => {
|
||||
expect(result.success).toBe(true);
|
||||
expect(meanVolumeDb(outputPath) - meanVolumeDb(sourcePath)).toBeGreaterThan(-0.3);
|
||||
});
|
||||
|
||||
it("places a delayed track on its authored start, not one AAC frame later", async () => {
|
||||
// The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. A
|
||||
// raw ADTS container has nowhere to record that delay, so it decodes as
|
||||
// real leading silence and drags the whole track 21.33 ms late against a
|
||||
// frame-accurate video. MIXED_AUDIO_FILENAME picks a container that stores
|
||||
// the delay as an edit list instead; this asserts the artifact we actually
|
||||
// ship lands on time.
|
||||
const projectDir = mkdtempSync(join(tmpdir(), "hf-onset-"));
|
||||
const workDir = mkdtempSync(join(tmpdir(), "hf-onset-work-"));
|
||||
tempDirs.push(projectDir, workDir);
|
||||
const sourcePath = join(projectDir, "tone.wav");
|
||||
const outputPath = join(projectDir, MIXED_AUDIO_FILENAME);
|
||||
const setup = spawnSync(
|
||||
getFfmpegBinary(),
|
||||
[
|
||||
"-nostdin",
|
||||
"-v",
|
||||
"error",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"sine=frequency=1000:duration=1:sample_rate=48000",
|
||||
"-c:a",
|
||||
"pcm_s16le",
|
||||
sourcePath,
|
||||
],
|
||||
{ encoding: "utf-8" },
|
||||
);
|
||||
expect(setup.status, setup.stderr).toBe(0);
|
||||
|
||||
const result = await processCompositionAudio(
|
||||
[
|
||||
{
|
||||
id: "tone",
|
||||
src: "tone.wav",
|
||||
start: 2,
|
||||
end: 3,
|
||||
mediaStart: 0,
|
||||
layer: 0,
|
||||
volume: 1,
|
||||
type: "audio",
|
||||
},
|
||||
],
|
||||
projectDir,
|
||||
workDir,
|
||||
outputPath,
|
||||
4,
|
||||
);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(firstAudibleSeconds(outputPath)).toBeCloseTo(2, 2);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -32,6 +32,21 @@ import { applyVolumeEnvelopeToWav } from "./audioVolumeEnvelope.js";
|
||||
|
||||
export type { AudioElement, MixResult } from "./audioMixer.types.js";
|
||||
|
||||
/**
|
||||
* Filename every caller must use for the mixed-audio artifact.
|
||||
*
|
||||
* The extension is load-bearing, not cosmetic: FFmpeg picks the muxer from it,
|
||||
* and the mix is AAC-encoded. A raw ADTS `.aac` stream has nowhere to record
|
||||
* the encoder's priming delay, so those leading samples decode as real silence
|
||||
* and shift the whole track ~1024 samples (21.33 ms at 48 kHz) late against a
|
||||
* frame-accurate video track. An MP4-family container carries the delay as an
|
||||
* edit list, which every decoder then strips, so the mix lands on its authored
|
||||
* start. Keep the choice here rather than at each call site: the same file is
|
||||
* muxed into the video, shipped in a distributed plan, and handed to users as
|
||||
* the PNG-sequence sidecar, and all three have to agree.
|
||||
*/
|
||||
export const MIXED_AUDIO_FILENAME = "audio.m4a";
|
||||
|
||||
function clampVolume(volume: number): number {
|
||||
if (!Number.isFinite(volume)) return 1;
|
||||
return Math.max(0, Math.min(1, volume));
|
||||
|
||||
@@ -87,7 +87,7 @@ function makeMinimalV1PlanDir(dir: string, withAudio: boolean): void {
|
||||
JSON.stringify([{ index: 0, startFrame: 0, endFrame: 30 }]),
|
||||
);
|
||||
writeFileSync(join(dir, "meta", "encoder.json"), "{}");
|
||||
if (withAudio) writeFileSync(join(dir, "audio.aac"), "AAC");
|
||||
if (withAudio) writeFileSync(join(dir, "audio.m4a"), "AAC");
|
||||
planJson.planHash = recomputePlanHashFromPlanDir(dir);
|
||||
writeFileSync(join(dir, "plan.json"), JSON.stringify(planJson));
|
||||
}
|
||||
@@ -273,7 +273,7 @@ describe("dispatch", () => {
|
||||
chunkIndex: number,
|
||||
outputBase: string,
|
||||
): Promise<ChunkResult> => {
|
||||
expect(existsSync(join(planDir, "audio.aac"))).toBe(false);
|
||||
expect(existsSync(join(planDir, "audio.m4a"))).toBe(false);
|
||||
writeFileSync(outputBase, `chunk-${chunkIndex}`);
|
||||
return {
|
||||
outputPath: outputBase,
|
||||
|
||||
@@ -32,6 +32,9 @@ import {
|
||||
listPlanV2ArtifactsForTarget,
|
||||
materializePlanV2Target,
|
||||
plan,
|
||||
isPlanAudioArtifactPath,
|
||||
PLAN_AUDIO_RELATIVE_PATH,
|
||||
resolvePlanAudioPath,
|
||||
planV2WithPublisher,
|
||||
type PlanResult,
|
||||
type PlanV2Artifact,
|
||||
@@ -322,7 +325,7 @@ async function handlePlan(event: PlanEvent, deps?: HandlerDeps): Promise<PlanRes
|
||||
|
||||
// Upload the planDir as a single tarball. The workflow cannot pass a
|
||||
// directory-shaped artifact between steps; we serialize and rely on the
|
||||
// consumer (renderChunk / assemble) to untar. `audio.aac` lives inside
|
||||
// consumer (renderChunk / assemble) to untar. The audio artifact lives inside
|
||||
// planDir, so it already rides along in this tarball — every consumer
|
||||
// (including assemble) gets it from the untar. We deliberately do NOT
|
||||
// upload a separate audio object: it would duplicate the bytes on every
|
||||
@@ -331,7 +334,7 @@ async function handlePlan(event: PlanEvent, deps?: HandlerDeps): Promise<PlanRes
|
||||
const planTar = join(work, "plan.tar.gz");
|
||||
await tarDirectory(planDir, planTar);
|
||||
const planTarUri = `${trimTrailingSlash(event.PlanOutputGcsPrefix)}/plan.tar.gz`;
|
||||
const audioPath = join(planDir, "audio.aac");
|
||||
const audioPath = join(planDir, PLAN_AUDIO_RELATIVE_PATH);
|
||||
const hasAudio = existsSync(audioPath) && statSync(audioPath).size > 0;
|
||||
await uploadFileToGcs(storage, planTar, planTarUri, "application/gzip");
|
||||
|
||||
@@ -397,7 +400,7 @@ async function handlePlanV2(
|
||||
Width: manifest.width,
|
||||
Height: manifest.height,
|
||||
Format: manifest.format,
|
||||
HasAudio: manifest.artifacts.some((artifact) => artifact.path === "audio.aac"),
|
||||
HasAudio: manifest.artifacts.some((artifact) => isPlanAudioArtifactPath(artifact.path)),
|
||||
AudioGcsUri: null,
|
||||
FfmpegVersion: manifest.ffmpegVersion,
|
||||
ProducerVersion: manifest.producerVersion,
|
||||
@@ -567,7 +570,7 @@ async function handleAssemble(
|
||||
// only for backward compatibility with an older Plan that uploaded it
|
||||
// standalone.
|
||||
let audioPath: string | null = null;
|
||||
const planAudio = join(planDir, "audio.aac");
|
||||
const planAudio = resolvePlanAudioPath(planDir) ?? join(planDir, PLAN_AUDIO_RELATIVE_PATH);
|
||||
if (existsSync(planAudio) && statSync(planAudio).size > 0) {
|
||||
audioPath = planAudio;
|
||||
} else if (event.AudioGcsUri) {
|
||||
@@ -619,7 +622,7 @@ async function handleAssembleV2(
|
||||
const work = mkdtempSync(join(deps?.tmpRoot ?? tmpdir(), "hf-cr-assemble-v2-"));
|
||||
try {
|
||||
const planDir = await downloadAndMaterializePlanV2(storage, event, { role: "assembler" }, work);
|
||||
const audioPath = existsSync(join(planDir, "audio.aac")) ? join(planDir, "audio.aac") : null;
|
||||
const audioPath = resolvePlanAudioPath(planDir);
|
||||
const chunkPaths = await downloadChunkObjects(storage, event.ChunkGcsUris, work, event.Format);
|
||||
const finalOutput =
|
||||
event.Format === "png-sequence"
|
||||
|
||||
@@ -149,6 +149,18 @@ export {
|
||||
// CLI / adopter SDKs can derive runtime allowlists from one source.
|
||||
export { PlanVideosMetadataError, type DistributedFormat } from "./services/distributed/shared.js";
|
||||
|
||||
// ── Plan artifact names ─────────────────────────────────────────────────────
|
||||
// The cloud adapters locate and publish the plan's audio artifact by name. Its
|
||||
// extension selects the container, so they must read it from here rather than
|
||||
// restate it: a literal that drifts from the writer's is a silently missing
|
||||
// audio track, not a loud failure.
|
||||
export {
|
||||
isPlanAudioArtifactPath,
|
||||
PLAN_AUDIO_LEGACY_RELATIVE_PATH,
|
||||
PLAN_AUDIO_RELATIVE_PATH,
|
||||
resolvePlanAudioPath,
|
||||
} from "./services/distributed/shared.js";
|
||||
|
||||
// ── Plan-time shared types from `freezePlan` ───────────────────────────────
|
||||
// Re-exported so adopters that deserialize a planDir's `meta/encoder.json`
|
||||
// or `meta/chunks.json` see the same shapes the producer wrote them as.
|
||||
|
||||
@@ -35,7 +35,7 @@ import { existsSync, mkdirSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import type { Fps } from "@hyperframes/core";
|
||||
import { assemble, plan, renderChunk } from "./distributed.js";
|
||||
import type { DistributedFormat } from "./services/distributed/shared.js";
|
||||
import { PLAN_AUDIO_RELATIVE_PATH, type DistributedFormat } from "./services/distributed/shared.js";
|
||||
|
||||
/**
|
||||
* Three-mode contract that backs `--mode=<value>` on the regression
|
||||
@@ -199,9 +199,9 @@ export async function runDistributedSimulatedRender(
|
||||
chunkPaths.push(chunkPath);
|
||||
}
|
||||
|
||||
// Step C: assemble. `audio.aac` only exists when the composition has
|
||||
// Step C: assemble. The audio artifact only exists when the composition has
|
||||
// audio — pass null otherwise so `assemble()` doesn't try to mux silence.
|
||||
const audioPath = join(planDir, "audio.aac");
|
||||
const audioPath = join(planDir, PLAN_AUDIO_RELATIVE_PATH);
|
||||
const audioForAssemble = existsSync(audioPath) ? audioPath : null;
|
||||
await assemble(planDir, chunkPaths, audioForAssemble, input.renderedOutputPath);
|
||||
}
|
||||
|
||||
@@ -37,7 +37,7 @@ afterAll(() => {
|
||||
/**
|
||||
* Build a synthetic planDir whose `meta/chunks.json` declares N chunks of
|
||||
* `framesPerChunk` frames each. Does NOT materialize compiled/, video-frames/,
|
||||
* audio.aac — assemble only reads `plan.json` + `meta/chunks.json`, and we
|
||||
* audio.m4a — assemble only reads `plan.json` + `meta/chunks.json`, and we
|
||||
* pass chunk paths explicitly. Keeping the dir lean speeds up the test
|
||||
* loop.
|
||||
*/
|
||||
@@ -287,7 +287,7 @@ describe("assemble()", () => {
|
||||
);
|
||||
|
||||
it(
|
||||
"muxes audio with frame-count-derived duration when audio.aac is present",
|
||||
"muxes audio with frame-count-derived duration when audio.m4a is present",
|
||||
async () => {
|
||||
if (!hasFfmpeg) return;
|
||||
|
||||
@@ -301,7 +301,7 @@ describe("assemble()", () => {
|
||||
|
||||
const chunkAPath = join(planDir, "chunk-0.mp4");
|
||||
const chunkBPath = join(planDir, "chunk-1.mp4");
|
||||
const audioPath = join(planDir, "audio.aac");
|
||||
const audioPath = join(planDir, "audio.m4a");
|
||||
makeMp4Chunk(chunkAPath, 6);
|
||||
makeMp4Chunk(chunkBPath, 6);
|
||||
// Audio is half a second longer than the video — `padOrTrimAudioToVideoFrameCount`
|
||||
@@ -346,7 +346,7 @@ describe("assemble()", () => {
|
||||
|
||||
const chunkAPath = join(planDir, "chunk-0.mp4");
|
||||
const chunkBPath = join(planDir, "chunk-1.mp4");
|
||||
const audioPath = join(planDir, "audio.aac");
|
||||
const audioPath = join(planDir, "audio.m4a");
|
||||
makeMp4Chunk(chunkAPath, 6);
|
||||
makeMp4Chunk(chunkBPath, 6);
|
||||
// Audio is shorter than the video, forcing the distributed pad branch.
|
||||
|
||||
@@ -34,7 +34,12 @@ import {
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { applyFaststart, muxVideoWithAudio, runFfmpeg } from "@hyperframes/engine";
|
||||
import {
|
||||
applyFaststart,
|
||||
MIXED_AUDIO_FILENAME,
|
||||
muxVideoWithAudio,
|
||||
runFfmpeg,
|
||||
} from "@hyperframes/engine";
|
||||
import { fpsToFfmpegArg } from "@hyperframes/core";
|
||||
import { defaultLogger, type ProducerLogger } from "../../logger.js";
|
||||
import { formatExportFrameName } from "../../utils/paths.js";
|
||||
@@ -78,7 +83,7 @@ interface PlanJsonForAssemble {
|
||||
* @param chunkPaths — ordered chunk outputs, length === `chunks.json` length.
|
||||
* For mp4/mov each entry is a path to an encoded chunk file; for
|
||||
* png-sequence each entry is a path to a directory of frames.
|
||||
* @param audioPath — `<planDir>/audio.aac` for mux'd formats. Pass `null`
|
||||
* @param audioPath — `<planDir>/<MIXED_AUDIO_FILENAME>` for mux'd formats. Pass `null`
|
||||
* when the composition has no audio (or `assemble` is being called for a
|
||||
* format whose audio is muxed elsewhere). `assemble` always normalizes
|
||||
* audio length against the assembled video's frame count when
|
||||
@@ -385,9 +390,9 @@ export async function assemble(
|
||||
* into the merged output so consumers see one continuous numbered sequence.
|
||||
*
|
||||
* Audio is intentionally NOT muxed here — png-sequence has no container.
|
||||
* If `audioPath` is non-null we copy it alongside as `audio.aac` so callers
|
||||
* who need to re-mux later (After Effects, Nuke, ffmpeg image2 + audio) can
|
||||
* find it.
|
||||
* If `audioPath` is non-null we copy it alongside under the engine's mixed-audio
|
||||
* filename so callers who need to re-mux later (After Effects, Nuke, ffmpeg
|
||||
* image2 + audio) can find it.
|
||||
*/
|
||||
function mergePngFrameDirs(
|
||||
chunkPaths: readonly string[],
|
||||
@@ -435,7 +440,7 @@ function mergePngFrameDirs(
|
||||
// containers); png-sequence has no encoder, so we copy the audio
|
||||
// verbatim. The sidecar matches the in-process png-sequence convention.
|
||||
if (audioPath !== null && existsSync(audioPath)) {
|
||||
const sidecar = join(outputPath, "audio.aac");
|
||||
const sidecar = join(outputPath, MIXED_AUDIO_FILENAME);
|
||||
cpSync(audioPath, sidecar);
|
||||
}
|
||||
|
||||
|
||||
@@ -96,7 +96,7 @@ async function planAndAssemble(input: {
|
||||
chunkPaths.push(chunkPath);
|
||||
}
|
||||
|
||||
const audioPath = join(planDir, "audio.aac");
|
||||
const audioPath = join(planDir, "audio.m4a");
|
||||
const audioForAssemble = existsSync(audioPath) ? audioPath : null;
|
||||
await assemble(planDir, chunkPaths, audioForAssemble, outputPath);
|
||||
return outputPath;
|
||||
|
||||
@@ -492,8 +492,8 @@ describe("plan() — golden planDir + planHash determinism", () => {
|
||||
expect(existsSync(join(planDir, "plan.json"))).toBe(true);
|
||||
expect(existsSync(join(planDir, "compiled", "index.html"))).toBe(true);
|
||||
expect(existsSync(join(planDir, "video-frames"))).toBe(true);
|
||||
// No audio in the fixture — audio.aac must NOT exist.
|
||||
expect(existsSync(join(planDir, "audio.aac"))).toBe(false);
|
||||
// No audio in the fixture — audio.m4a must NOT exist.
|
||||
expect(existsSync(join(planDir, "audio.m4a"))).toBe(false);
|
||||
expect(existsSync(join(planDir, "meta", "composition.json"))).toBe(true);
|
||||
expect(existsSync(join(planDir, "meta", "encoder.json"))).toBe(true);
|
||||
expect(existsSync(join(planDir, "meta", "chunks.json"))).toBe(true);
|
||||
@@ -718,7 +718,7 @@ describe("plan() — golden planDir + planHash determinism", () => {
|
||||
// Verify the audio stage actually fired (otherwise the test
|
||||
// pins the wrong path — the same false-pass mode as the
|
||||
// no-audio variant above).
|
||||
expect(existsSync(join(planDir, "audio.aac"))).toBe(true);
|
||||
expect(existsSync(join(planDir, "audio.m4a"))).toBe(true);
|
||||
},
|
||||
TIMEOUT_MS,
|
||||
);
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
* ├── plan.json
|
||||
* ├── compiled/ # compileForRender output (self-contained)
|
||||
* ├── video-frames/ # per-video JPEG sequences (dereferenced)
|
||||
* ├── audio.aac # only when composition has audio
|
||||
* ├── audio.m4a # only when composition has audio
|
||||
* └── meta/
|
||||
* ├── composition.json
|
||||
* ├── encoder.json # LockedRenderConfig
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
isPlanAudioArtifactPath,
|
||||
PLAN_AUDIO_LEGACY_RELATIVE_PATH,
|
||||
PLAN_AUDIO_RELATIVE_PATH,
|
||||
resolvePlanAudioPath,
|
||||
} from "./shared.js";
|
||||
|
||||
/**
|
||||
* `plan` and `assemble` are separate invocations bridged by object storage, so a
|
||||
* rolling deploy can pair a pre-rollout planner with a post-rollout assembler.
|
||||
* Both readers locate the artifact by existence alone, which would make that
|
||||
* pairing a silently muted video rather than an error.
|
||||
*
|
||||
* Delete this file, the legacy constant, and the fallback branch one release
|
||||
* after the container change ships.
|
||||
*/
|
||||
describe("plan audio artifact compatibility", () => {
|
||||
const dirs: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const planDir = (files: string[]): string => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-plan-audio-"));
|
||||
dirs.push(dir);
|
||||
for (const name of files) writeFileSync(join(dir, name), "audio");
|
||||
return dir;
|
||||
};
|
||||
|
||||
it("prefers the current name", () => {
|
||||
const dir = planDir([PLAN_AUDIO_RELATIVE_PATH, PLAN_AUDIO_LEGACY_RELATIVE_PATH]);
|
||||
expect(resolvePlanAudioPath(dir)).toBe(join(dir, PLAN_AUDIO_RELATIVE_PATH));
|
||||
});
|
||||
|
||||
it("falls back to a legacy plan's name so a rolling deploy keeps its audio", () => {
|
||||
const dir = planDir([PLAN_AUDIO_LEGACY_RELATIVE_PATH]);
|
||||
expect(resolvePlanAudioPath(dir)).toBe(join(dir, PLAN_AUDIO_LEGACY_RELATIVE_PATH));
|
||||
});
|
||||
|
||||
it("reports no audio when the plan carries neither name", () => {
|
||||
expect(resolvePlanAudioPath(planDir([]))).toBeNull();
|
||||
});
|
||||
|
||||
it("recognises both names as the audio artifact, and nothing else", () => {
|
||||
expect(isPlanAudioArtifactPath(PLAN_AUDIO_RELATIVE_PATH)).toBe(true);
|
||||
expect(isPlanAudioArtifactPath(PLAN_AUDIO_LEGACY_RELATIVE_PATH)).toBe(true);
|
||||
expect(isPlanAudioArtifactPath("plan.json")).toBe(false);
|
||||
expect(isPlanAudioArtifactPath("meta/chunks.json")).toBe(false);
|
||||
});
|
||||
|
||||
it("writes only the current name", () => {
|
||||
expect(PLAN_AUDIO_RELATIVE_PATH).not.toBe(PLAN_AUDIO_LEGACY_RELATIVE_PATH);
|
||||
expect(PLAN_AUDIO_RELATIVE_PATH.endsWith(".m4a")).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstatSync, readdirSync } from "node:fs";
|
||||
import { extname, join, relative } from "node:path";
|
||||
import { PLAN_AUDIO_RELATIVE_PATH } from "./shared.js";
|
||||
|
||||
const MAX_TOP_COMPONENTS = 10;
|
||||
const SOURCE_MEDIA_EXTENSIONS = new Set([
|
||||
@@ -56,7 +57,7 @@ interface ClassifiedFile {
|
||||
|
||||
const PLAN_ROOT_CATEGORY: Readonly<Record<string, PlanSizeCategory>> = {
|
||||
compiled: "compiled",
|
||||
"audio.aac": "audio",
|
||||
[PLAN_AUDIO_RELATIVE_PATH]: "audio",
|
||||
meta: "metadata",
|
||||
"plan.json": "metadata",
|
||||
};
|
||||
|
||||
@@ -87,7 +87,7 @@ describe("measurePlanSizeBreakdown", () => {
|
||||
join(dir, "video-frames", "customer-video-name", "frame_000001.jpg"),
|
||||
Buffer.alloc(300),
|
||||
);
|
||||
writeFileSync(join(dir, "audio.aac"), Buffer.alloc(40));
|
||||
writeFileSync(join(dir, "audio.m4a"), Buffer.alloc(40));
|
||||
writeFileSync(join(dir, "meta", "encoder.json"), Buffer.alloc(20));
|
||||
writeFileSync(join(dir, "plan.json"), Buffer.alloc(10));
|
||||
writeFileSync(join(dir, "other.bin"), Buffer.alloc(5));
|
||||
|
||||
@@ -141,7 +141,7 @@ function createV1Plan(
|
||||
rmSync(join(planDir, "meta", "videos.json"));
|
||||
}
|
||||
}
|
||||
if (audio) writeFileSync(join(planDir, "audio.aac"), "assemble-only-audio");
|
||||
if (audio) writeFileSync(join(planDir, "audio.m4a"), "assemble-only-audio");
|
||||
writeFileSync(
|
||||
join(planDir, "plan.json"),
|
||||
JSON.stringify({
|
||||
@@ -387,9 +387,9 @@ describe("Plan v2 manifest", () => {
|
||||
const chunk = listPlanV2ArtifactsForTarget(manifest, { role: "chunk", chunkIndex: 0 });
|
||||
const assembler = listPlanV2ArtifactsForTarget(manifest, { role: "assembler" });
|
||||
|
||||
expect(chunk.some((artifact) => artifact.path === "audio.aac")).toBe(false);
|
||||
expect(chunk.some((artifact) => artifact.path === "audio.m4a")).toBe(false);
|
||||
expect(chunk.some((artifact) => artifact.path === "compiled/index.html")).toBe(true);
|
||||
expect(assembler.some((artifact) => artifact.path === "audio.aac")).toBe(true);
|
||||
expect(assembler.some((artifact) => artifact.path === "audio.m4a")).toBe(true);
|
||||
expect(assembler.some((artifact) => artifact.path === "compiled/index.html")).toBe(false);
|
||||
});
|
||||
|
||||
@@ -626,11 +626,11 @@ describe("Plan v2 manifest", () => {
|
||||
);
|
||||
const assembler = materializePlanV2Target(result.planDir, { role: "assembler" }, assemblerDir);
|
||||
|
||||
expect(existsSync(join(chunkDir, "audio.aac"))).toBe(false);
|
||||
expect(existsSync(join(chunkDir, "audio.m4a"))).toBe(false);
|
||||
expect(
|
||||
validatePlanV2MaterializedTarget(chunkDir, { role: "chunk", chunkIndex: 1 })?.planHash,
|
||||
).toBe(result.planHash);
|
||||
expect(assembler.audioPath).toBe(join(assemblerDir, "audio.aac"));
|
||||
expect(assembler.audioPath).toBe(join(assemblerDir, "audio.m4a"));
|
||||
expect(validatePlanV2MaterializedTarget(assemblerDir, { role: "assembler" })?.planHash).toBe(
|
||||
result.planHash,
|
||||
);
|
||||
|
||||
@@ -47,7 +47,7 @@ import {
|
||||
import { PLAN_V2_INTEGRITY_UNRECOVERABLE, PlanV2IntegrityError } from "./planV2Errors.js";
|
||||
import { planV2BlobPath } from "./planV2Layout.js";
|
||||
import {
|
||||
PLAN_AUDIO_RELATIVE_PATH,
|
||||
isPlanAudioArtifactPath,
|
||||
PLAN_VIDEOS_META_RELATIVE_PATH,
|
||||
type DistributedFormat,
|
||||
parsePlanVideosJson as parseSharedPlanVideosJson,
|
||||
@@ -283,7 +283,7 @@ function artifactTargets(
|
||||
path: string,
|
||||
videoDependencies: ReadonlyMap<string, readonly number[]> | null,
|
||||
): Pick<PlanV2Artifact, "chunks" | "assembler"> {
|
||||
if (path === PLAN_AUDIO_RELATIVE_PATH) return { chunks: [], assembler: true };
|
||||
if (isPlanAudioArtifactPath(path)) return { chunks: [], assembler: true };
|
||||
if (path === "plan.json" || path === "meta/chunks.json" || path === "meta/encoder.json") {
|
||||
return { chunks: "all", assembler: true };
|
||||
}
|
||||
@@ -898,6 +898,7 @@ export function materializePlanV2Target(
|
||||
}
|
||||
const manifest = readPlanV2Manifest(planV2Dir);
|
||||
const artifacts = listPlanV2ArtifactsForTarget(manifest, target);
|
||||
const audioArtifact = artifacts.find((artifact) => isPlanAudioArtifactPath(artifact.path));
|
||||
const verified = artifacts.map((artifact) => ({
|
||||
artifact,
|
||||
sourcePath: verifyBlob(planV2Dir, artifact),
|
||||
@@ -933,10 +934,14 @@ export function materializePlanV2Target(
|
||||
sourcePlanV1Hash: getPlanV2ExecutionPlanHash(manifest),
|
||||
artifactCount: artifacts.length,
|
||||
sizeBytes: artifacts.reduce((sum, artifact) => sum + artifact.sizeBytes, 0),
|
||||
// Join the artifact's OWN name, not the current constant: a plan written
|
||||
// before the container change carries the legacy name, and materializing it
|
||||
// under the new one would point at a file that was never written.
|
||||
audioPath:
|
||||
target.role === "assembler" &&
|
||||
artifacts.some((artifact) => artifact.path === PLAN_AUDIO_RELATIVE_PATH)
|
||||
? join(destinationDir, PLAN_AUDIO_RELATIVE_PATH)
|
||||
target.role === "assembler"
|
||||
? audioArtifact
|
||||
? join(destinationDir, audioArtifact.path)
|
||||
: null
|
||||
: null,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -10,7 +10,12 @@ import { existsSync, readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
import { type Fps } from "@hyperframes/core";
|
||||
import { type VideoElement, type VideoFrameFormat, type VideoMetadata } from "@hyperframes/engine";
|
||||
import {
|
||||
MIXED_AUDIO_FILENAME,
|
||||
type VideoElement,
|
||||
type VideoFrameFormat,
|
||||
type VideoMetadata,
|
||||
} from "@hyperframes/engine";
|
||||
import { type RenderConfig, type RenderJob, createRenderJob } from "../renderOrchestrator.js";
|
||||
import { defaultLogger, type ProducerLogger } from "../../logger.js";
|
||||
|
||||
@@ -35,8 +40,42 @@ export const PLAN_VIDEOS_META_RELATIVE_PATH = "meta/videos.json";
|
||||
* Relative path of the normalized audio artifact written into a distributed
|
||||
* plan. Keep writers and transport readers coupled through this contract
|
||||
* rather than duplicating a filename literal.
|
||||
*
|
||||
* Derived from the engine's filename rather than restated, because the
|
||||
* extension selects the muxer: the plan artifact is the mix moved into place,
|
||||
* so if the two ever disagreed the plan would claim a container the file
|
||||
* doesn't have.
|
||||
*/
|
||||
export const PLAN_AUDIO_RELATIVE_PATH = "audio.aac";
|
||||
export const PLAN_AUDIO_RELATIVE_PATH = MIXED_AUDIO_FILENAME;
|
||||
|
||||
/**
|
||||
* Name the audio artifact carried before it moved to an MP4-family container.
|
||||
*
|
||||
* COMPATIBILITY SHIM, remove one release after the container change ships.
|
||||
* `plan` and `assemble` are separate invocations bridged by object storage, so
|
||||
* a rolling deploy can pair a pre-rollout planner with a post-rollout
|
||||
* assembler. Both readers locate the artifact by existence alone, which makes
|
||||
* that pairing a silently muted video rather than an error, so reads accept the
|
||||
* old name for one release while writes only ever emit the new one.
|
||||
*/
|
||||
export const PLAN_AUDIO_LEGACY_RELATIVE_PATH = "audio.aac";
|
||||
|
||||
/** True for either the current or the legacy plan-audio artifact name. */
|
||||
export function isPlanAudioArtifactPath(path: string): boolean {
|
||||
return path === PLAN_AUDIO_RELATIVE_PATH || path === PLAN_AUDIO_LEGACY_RELATIVE_PATH;
|
||||
}
|
||||
|
||||
/**
|
||||
* Locate a plan's audio artifact on disk, preferring the current name and
|
||||
* falling back to the legacy one. Returns `null` when the plan has no audio.
|
||||
*/
|
||||
export function resolvePlanAudioPath(planDir: string): string | null {
|
||||
for (const name of [PLAN_AUDIO_RELATIVE_PATH, PLAN_AUDIO_LEGACY_RELATIVE_PATH]) {
|
||||
const candidate = join(planDir, name);
|
||||
if (existsSync(candidate)) return candidate;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* On-disk shape of `<planDir>/meta/videos.json`. The engine's
|
||||
|
||||
@@ -97,8 +97,8 @@ describe("buildPadTrimAudioArgs", () => {
|
||||
// Bundled Windows FFmpeg builds reject `apad=whole_dur`. Match the
|
||||
// portable finite-padding shape used by the main audio mixer.
|
||||
const winPlan = buildPadTrimAudioPlan(
|
||||
"C:\\Users\\alice\\AppData\\Local\\Temp\\hf-render-abc\\audio.aac",
|
||||
"C:\\Users\\alice\\AppData\\Local\\Temp\\hf-render-abc\\audio-padded.aac",
|
||||
"C:\\Users\\alice\\AppData\\Local\\Temp\\hf-render-abc\\audio.m4a",
|
||||
"C:\\Users\\alice\\AppData\\Local\\Temp\\hf-render-abc\\audio-padded.m4a",
|
||||
4.0,
|
||||
5.0,
|
||||
);
|
||||
@@ -301,7 +301,7 @@ describe("PadTrimAudioResult.error never carries the input path", () => {
|
||||
it(`redacts ${name} raised by the video probe`, async () => {
|
||||
const result = await padOrTrimAudioToVideoFrameCount({
|
||||
videoPath,
|
||||
audioPath: "/tmp/audio.aac",
|
||||
audioPath: "/tmp/audio.m4a",
|
||||
outputPath: "/tmp/out.aac",
|
||||
// Reproduces the real thrower: defaultProbeVideoFrameInfo raises
|
||||
// `ffprobe found no video stream in ${videoPath}` with the raw path.
|
||||
@@ -323,12 +323,12 @@ describe("PadTrimAudioResult.error never carries the input path", () => {
|
||||
it("redacts raw ffprobe stderr surfaced through the audio probe", async () => {
|
||||
const result = await padOrTrimAudioToVideoFrameCount({
|
||||
videoPath: "/tmp/v.mp4",
|
||||
audioPath: "/data/acme-secret/audio.aac",
|
||||
audioPath: "/data/acme-secret/audio.m4a",
|
||||
outputPath: "/tmp/out.aac",
|
||||
probeVideoFrameInfo: () => Promise.resolve({ frameCount: 30, fpsNum: 30, fpsDen: 1 }),
|
||||
probeAudioInfo: () =>
|
||||
Promise.reject(
|
||||
new Error("/data/acme-secret/audio.aac: Invalid data found when processing input"),
|
||||
new Error("/data/acme-secret/audio.m4a: Invalid data found when processing input"),
|
||||
),
|
||||
runFfmpeg: () => Promise.resolve({ success: true }),
|
||||
});
|
||||
@@ -354,7 +354,7 @@ describe("PadTrimAudioResult.error never carries the input path", () => {
|
||||
it(`still returns a failed result when the video probe rejects with ${label}`, async () => {
|
||||
const result = await padOrTrimAudioToVideoFrameCount({
|
||||
videoPath: "/data/acme-secret/video.mp4",
|
||||
audioPath: "/tmp/audio.aac",
|
||||
audioPath: "/tmp/audio.m4a",
|
||||
outputPath: "/tmp/out.aac",
|
||||
probeVideoFrameInfo: () => Promise.reject(reason),
|
||||
probeAudioInfo: () => Promise.resolve({ durationSeconds: 1 }),
|
||||
@@ -367,7 +367,7 @@ describe("PadTrimAudioResult.error never carries the input path", () => {
|
||||
it(`still returns a failed result when the audio probe rejects with ${label}`, async () => {
|
||||
const result = await padOrTrimAudioToVideoFrameCount({
|
||||
videoPath: "/tmp/v.mp4",
|
||||
audioPath: "/data/acme-secret/audio.aac",
|
||||
audioPath: "/data/acme-secret/audio.m4a",
|
||||
outputPath: "/tmp/out.aac",
|
||||
probeVideoFrameInfo: () => Promise.resolve({ frameCount: 30, fpsNum: 30, fpsDen: 1 }),
|
||||
probeAudioInfo: () => Promise.reject(reason),
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// fallow-ignore-file complexity
|
||||
/**
|
||||
* audioPadTrim — pad-or-trim an `audio.aac` file so its exact duration
|
||||
* audioPadTrim — pad-or-trim the mixed-audio file so its exact duration
|
||||
* matches the assembled video's frame count divided by fps.
|
||||
*
|
||||
* Distributed render assemble step needs this because:
|
||||
@@ -62,7 +62,7 @@ export interface AudioProbeInfo {
|
||||
export interface PadTrimAudioInput {
|
||||
/** Path to the assembled video. Used to derive `frameCount / fps`. */
|
||||
videoPath: string;
|
||||
/** Path to the pre-mixed audio (typically `<planDir>/audio.aac`). */
|
||||
/** Path to the pre-mixed audio (typically `<planDir>/audio.m4a`). */
|
||||
audioPath: string;
|
||||
/** Path the helper writes the duration-corrected audio to. */
|
||||
outputPath: string;
|
||||
@@ -109,10 +109,10 @@ export interface PadTrimAudioPlan {
|
||||
* sequence that materializes it. Exported separately so unit tests can pin
|
||||
* every branch without spawning ffmpeg.
|
||||
*
|
||||
* - `sourceDuration < targetDuration` → generate only the missing silence
|
||||
* tail, then concat-copy the source AAC plus that tail. This avoids
|
||||
* re-encoding the already mixed `audio.aac`; the pad branch remains the
|
||||
* inverse of trim instead of becoming a second full-source AAC encode.
|
||||
* - `sourceDuration < targetDuration` → pad with `apad` to the exact target
|
||||
* and re-encode AAC. This decodes and re-encodes the already mixed audio;
|
||||
* an earlier concat-copy shape avoided that but could not produce a
|
||||
* portable result on the bundled Windows FFmpeg builds.
|
||||
* - `sourceDuration > targetDuration` → filter to the exact target and
|
||||
* re-encode AAC so packet padding cannot outlast the video.
|
||||
* - `|Δ| < AUDIO_DURATION_TOLERANCE_SECONDS` → no-op `copy`, but we still
|
||||
@@ -241,7 +241,7 @@ function sanitizeProbeFailure(reason: unknown, paths: readonly string[]): string
|
||||
}
|
||||
|
||||
/**
|
||||
* Pad or trim `audio.aac` so its exact duration matches `frameCount / fps`
|
||||
* Pad or trim the mixed audio so its exact duration matches `frameCount / fps`
|
||||
* for the assembled video.
|
||||
*/
|
||||
export async function padOrTrimAudioToVideoFrameCount(
|
||||
|
||||
@@ -33,7 +33,7 @@ function makeInput(overrides: Partial<AssembleStageInput> = {}): AssembleStageIn
|
||||
duration: 1,
|
||||
},
|
||||
videoOnlyPath: "/tmp/video-only.mp4",
|
||||
audioOutputPath: "/tmp/audio.aac",
|
||||
audioOutputPath: "/tmp/audio.m4a",
|
||||
outputPath: "/tmp/output.mp4",
|
||||
hasAudio: true,
|
||||
abortSignal: undefined,
|
||||
@@ -61,7 +61,7 @@ describe("runAssembleStage audio duration parity", () => {
|
||||
|
||||
expect(padOrTrimAudioMock).toHaveBeenCalledWith({
|
||||
videoPath: "/tmp/video-only.mp4",
|
||||
audioPath: "/tmp/audio.aac",
|
||||
audioPath: "/tmp/audio.m4a",
|
||||
outputPath: "/tmp/audio.duration-normalized.m4a",
|
||||
});
|
||||
expect(muxVideoWithAudioMock).toHaveBeenCalledWith(
|
||||
|
||||
@@ -49,7 +49,7 @@ describe("runAudioStage", () => {
|
||||
it("surfaces the mixer's error as audioError when the mix fails", async () => {
|
||||
processCompositionAudioMock.mockResolvedValue({
|
||||
success: false,
|
||||
outputPath: "audio.aac",
|
||||
outputPath: "audio.m4a",
|
||||
durationMs: 1,
|
||||
tracksProcessed: 0,
|
||||
error: "Source not found: a1 (narration.wav)",
|
||||
@@ -77,7 +77,7 @@ describe("runAudioStage", () => {
|
||||
it("falls back to a generic message when the mixer fails without an error string", async () => {
|
||||
processCompositionAudioMock.mockResolvedValue({
|
||||
success: false,
|
||||
outputPath: "audio.aac",
|
||||
outputPath: "audio.m4a",
|
||||
durationMs: 1,
|
||||
tracksProcessed: 0,
|
||||
});
|
||||
@@ -91,7 +91,7 @@ describe("runAudioStage", () => {
|
||||
it("does not set audioError when the mix succeeds", async () => {
|
||||
processCompositionAudioMock.mockResolvedValue({
|
||||
success: true,
|
||||
outputPath: "audio.aac",
|
||||
outputPath: "audio.m4a",
|
||||
durationMs: 1,
|
||||
tracksProcessed: 1,
|
||||
});
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
/**
|
||||
* audioStage — mix the composition's audio tracks into `workDir/audio.aac`.
|
||||
* audioStage — mix the composition's audio tracks into
|
||||
* `workDir/<MIXED_AUDIO_FILENAME>`.
|
||||
*
|
||||
* Trivial wrapper around `processCompositionAudio`. The stage is skipped
|
||||
* (no ffmpeg invocation) when the composition has no audio elements; the
|
||||
* timer is still set so the perf summary stays consistent across renders.
|
||||
*
|
||||
* Hard constraints preserved verbatim:
|
||||
* - `audioOutputPath` is always `join(workDir, "audio.aac")`, regardless
|
||||
* of whether any audio was actually produced.
|
||||
* - `audioOutputPath` is always `join(workDir, MIXED_AUDIO_FILENAME)`,
|
||||
* regardless of whether any audio was actually produced. The engine owns
|
||||
* that filename because its extension selects the muxer (see the constant).
|
||||
* - `hasAudio` reflects `audioResult.success` from
|
||||
* `processCompositionAudio`; it is `false` when there are no audio
|
||||
* elements (skips the call entirely) and also when the call returns
|
||||
@@ -16,7 +18,11 @@
|
||||
*/
|
||||
|
||||
import { join } from "node:path";
|
||||
import { processCompositionAudio, type AudioProcessingFailure } from "@hyperframes/engine";
|
||||
import {
|
||||
MIXED_AUDIO_FILENAME,
|
||||
processCompositionAudio,
|
||||
type AudioProcessingFailure,
|
||||
} from "@hyperframes/engine";
|
||||
import type { CompositionMetadata } from "../shared.js";
|
||||
|
||||
export interface AudioStageInput {
|
||||
@@ -33,7 +39,7 @@ export interface AudioStageInput {
|
||||
}
|
||||
|
||||
export interface AudioStageResult {
|
||||
/** Always `join(workDir, "audio.aac")`. */
|
||||
/** Always `join(workDir, MIXED_AUDIO_FILENAME)`. */
|
||||
audioOutputPath: string;
|
||||
/** True iff the audio mix actually produced a file. False when there are no audio elements. */
|
||||
hasAudio: boolean;
|
||||
@@ -54,7 +60,7 @@ export async function runAudioStage(input: AudioStageInput): Promise<AudioStageR
|
||||
input;
|
||||
|
||||
const stage3Start = Date.now();
|
||||
const audioOutputPath = join(workDir, "audio.aac");
|
||||
const audioOutputPath = join(workDir, MIXED_AUDIO_FILENAME);
|
||||
let hasAudio = false;
|
||||
let audioError: string | undefined;
|
||||
let audioFailures: AudioProcessingFailure[] | undefined;
|
||||
|
||||
@@ -32,6 +32,7 @@ const runFfmpegMock = mock(async () => ({
|
||||
}));
|
||||
|
||||
mock.module("@hyperframes/engine", () => ({
|
||||
MIXED_AUDIO_FILENAME: "audio.m4a",
|
||||
DEFAULT_CONFIG: { ffmpegEncodeTimeout: 600_000 },
|
||||
encodeFramesChunkedConcat: encodeFramesChunkedConcatMock,
|
||||
encodeFramesFromDir: encodeFramesFromDirMock,
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
*
|
||||
* 1. png-sequence: no encoder. Captured PNGs are renamed to
|
||||
* `frame_NNNNNN.png` and copied to `outputPath`. Audio (if any) is
|
||||
* written as an `audio.aac` sidecar.
|
||||
* written as a `MIXED_AUDIO_FILENAME` sidecar.
|
||||
* 2. gif: runs a two-pass FFmpeg palette encode and writes directly to
|
||||
* `outputPath`. GIF has no mux/faststart stage and ignores audio.
|
||||
* 3. mp4 / webm / mov: invokes `encodeFramesFromDir` (or the chunked-
|
||||
@@ -35,6 +35,7 @@ import {
|
||||
encodeFramesFromDir,
|
||||
formatFfmpegError,
|
||||
getEncoderPreset,
|
||||
MIXED_AUDIO_FILENAME,
|
||||
resolveConfig,
|
||||
runFfmpeg,
|
||||
type EngineConfig,
|
||||
@@ -246,8 +247,10 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
|
||||
// Sidecar audio for callers that need to re-mux later. png-sequence
|
||||
// has no container of its own, so this is the only place audio
|
||||
// can land alongside the frames.
|
||||
copyFileSync(audioOutputPath, join(outputPath, "audio.aac"));
|
||||
log.info(`[Render] png-sequence: audio.aac sidecar written to ${outputPath}/audio.aac`);
|
||||
copyFileSync(audioOutputPath, join(outputPath, MIXED_AUDIO_FILENAME));
|
||||
log.info(
|
||||
`[Render] png-sequence: ${MIXED_AUDIO_FILENAME} sidecar written to ${outputPath}/${MIXED_AUDIO_FILENAME}`,
|
||||
);
|
||||
}
|
||||
return { encodeMs: Date.now() - stage5Start };
|
||||
}
|
||||
|
||||
@@ -114,7 +114,7 @@ export interface ChunkSliceJson {
|
||||
|
||||
/**
|
||||
* Inputs to `freezePlan`. `planDir` already contains `compiled/`,
|
||||
* `video-frames/`, and (optionally) `audio.aac` by the time freezePlan
|
||||
* `video-frames/`, and (optionally) the mixed audio by the time freezePlan
|
||||
* runs — those are materialized by the upstream compile/probe/extract/audio
|
||||
* stages composed in `services/distributed/plan.ts`.
|
||||
*/
|
||||
@@ -132,7 +132,7 @@ export interface FreezePlanInput {
|
||||
durationSeconds: number;
|
||||
/** Total frame count, separately materialized for callers that read `plan.json` without parsing chunks.json. */
|
||||
totalFrames: number;
|
||||
/** Whether `<planDir>/audio.aac` was produced. */
|
||||
/** Whether the plan's mixed-audio artifact was produced. */
|
||||
hasAudio: boolean;
|
||||
}
|
||||
|
||||
|
||||
@@ -296,7 +296,7 @@ export interface RenderConfig {
|
||||
* stream; transparency is binary because GIF has no partial alpha.
|
||||
* - `"png-sequence"`: a directory of zero-padded RGBA PNGs
|
||||
* (`frame_000001.png` …). Lossless alpha, largest on disk, no muxed
|
||||
* audio (an `audio.aac` sidecar is written alongside the PNGs when
|
||||
* audio (an `audio.m4a` sidecar is written alongside the PNGs when
|
||||
* the composition has audio elements). Use for After Effects / Nuke
|
||||
* / Fusion ingest, or when frames need post-processing before
|
||||
* encoding. `outputPath` is treated as a directory; it is created if
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a84e50ad31694432affb1bffd42bde653514662ccb1dbb8172b0eab2d80147d5
|
||||
size 129943
|
||||
oid sha256:61cc9a67d53cfe17fd701f716347567be875f53a71fd3a6df9bef9c306937ace
|
||||
size 130189
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:694ae6f83914f3c51161804ffdb51acddc02bb523a0fa877f8bff667bfa0b5e9
|
||||
size 117758
|
||||
oid sha256:8661c4f42b2ec4315180d51a498b199f90e8234983e27b4c572f93e12ca44992
|
||||
size 116892
|
||||
|
||||
Reference in New Issue
Block a user