mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-10-30 09:16:55 +00:00
fix(engine): recover audio duration by decoding the final frame (#3986)
* fix(engine): recover audio duration from packet timestamps before clamping An authored `data-duration` on an <audio> element could be silently shortened when a source file's container-summary duration (mvhd/tkhd/mdhd) undercounts the real audio length while the packet stream itself is intact. Add `probeAudioDurationFromPackets()`, which re-derives the true duration from the last packet's own pts_time + duration_time — figures ffprobe computes from the stream's own timestamps, not the container's summary header — and wire it into the compiler's existing clamp check as a rescue attempt before accepting a shorter value. Works regardless of codec/profile; verified against real HE-AAC/SBR fixtures and a binary-patched "lying container" MP4 built for the regression test. The rescue only fires for candidates the existing clamp predicate already flagged (no added cost on the common path), runs in parallel across candidates rather than serially, and requires the recovered duration to exceed the existing clamp epsilon (absorbing the few milliseconds of priming-delay drift raw packet timestamps carry relative to an edit-list-corrected container duration) before overriding the shorter value. Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com> * fix(engine): move audio duration correction into extractAudioMetadata The previous approach compared the compiler's own packet-scan probe against the already-corrected duration returned by extractAudioMetadata, which meant the correction only applied to one caller and the regression fixture happened to exercise a pre-existing AAC-LC-specific refinement instead of the new code path. Move the fix to the single canonical place duration is derived: replace the old AAC-LC-only packet-count refinement (a profile allowlist assuming a fixed 1024-samples-per-frame constant) with a codec-agnostic mechanism that decodes the final audio frame and reads its own timestamp plus real sample count. This has no per-codec assumption, so it applies uniformly regardless of codec or profile, and every caller of extractAudioMetadata benefits automatically. The compiler's own clamp logic is now unchanged from its pre-existing form. Verified against a real, previously-untouched case: a FLAC-in-MP4 file (a codec the old AAC-LC-only path never corrected) with its mvhd/tkhd/mdhd duration atoms binary-patched to half their true value while every real frame stays intact. Confirmed this reproduces on the prior code (by literally substituting its actual content) and is fixed by the new code. The decode probe reads CSV output (each line stands alone) rather than JSON (a single top-level array), and retains only the output tail under a small cap — a lying container's true duration can be arbitrarily larger than its declared one, and JSON truncated from the front isn't valid JSON, so an unbounded scan of a long file silently fell back to the wrong duration once ffprobe's stdout cap was hit. Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com> * test(producer): update style-7-prod golden for corrected AAC duration probe The deleted AAC-LC packet-count refinement assumed every demuxed packet decodes to exactly 1024 samples. For this suite's real audio fixture (a 48kHz stereo AAC-LC MP4), that assumption overcounts: ffprobe reports 753 demuxed packets, but only 752 of them actually decode into a real audio frame (verified via raw PCM sample count: 770048 samples / 48000Hz = 16.0426666...s). The old mechanism's 753 * 1024 / 48000 = 16.064s was therefore wrong; the new decode-based probe correctly derives ~16.043s from the real final frame. Regression-shards caught this as a compilation mismatch on the previous commit. Verified the corrected golden against the real source: full compilation, visual (0 failed frames), and audio (correlation 0.966) checks all pass locally with the actual LFS-hosted golden video, and a standalone compileForRender() check confirms all other audio-bearing fixture suites in the repo are unaffected. Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com> * perf(engine): seek the audio decode probe instead of scanning the whole file probeDecodedAudioDuration ran a full, single-threaded decode of the entire audio stream on every extractAudioMetadata call (once per audio element in the HTML compiler, plus audio mixing/padding/encoding steps). On real 30-minute files this took over a second, and could approach ffprobe's 30s deadline on multi-hour files while holding a shared probe slot the whole time. Add -read_intervals so the probe seeks to ~1 second before the container's already-known claimed duration and decodes only that tail. This is safe because the correction only ever raises the duration: an honest or over-claiming container's seek lands at or past the real end and finds nothing extra, while an under-claiming container (the only case this exists to fix) still has real frames past the seek point. If the tail read finds nothing at all, fall back to a full unseeked decode before giving up, protecting files whose seek index is unreliable (an index-less or non-seekably-sourced container) from silently missing a real correction — mirroring the existing seek-then-fallback pattern already used by the sibling final-video-frame probe in this file. Also replaces the real-file regression test's runtime ffmpeg-spawning fixture generation with two small, committed binary fixtures (an honest FLAC-in-MP4 file and a copy with its container duration atoms halved) plus the ffprobe-static package, so this test's real-file proof actually runs in CI instead of being silently skipped when no system ffmpeg is on PATH. Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com> * refactor(engine): trim the audio duration probe comments to the invariant * test(engine): cache tests queue the decode probe and count metadata probes only * test(engine): probe for ffprobe with a literal argv so the argv contract classifies it * test(producer): regenerate the speed-ramp golden for the single-encode mix The audio probe now reads the mixed AAC as exactly 4.000 s instead of the packet-count 4.032 s (which counted the priming frame), so the final mix is stream-copied instead of re-encoded once more. Same waveform (sample correlation 0.9996 vs the old golden), 3.1 dB less true-peak attenuation; video frames are byte-identical. * fix(engine): measure decoded audio from the stream start and trust it for raw ADTS The decoded last-frame end is an absolute timestamp, so subtract the audio stream's start time (format start as fallback) before comparing it with the start-relative container duration, and start the tail read at start + duration - 1 s. MPEG-TS audio (1.4 s default start) and offset MP4s were over-extended by their start time. Raw ADTS (format aac) has no duration field; ffprobe estimates it from the opening bitrate and overclaims after a quiet opening. Its decoded timestamps are sample-counted, so for raw ADTS the decoded end now wins in both directions beyond the margin. Other containers stay upward-only so an MP4 edit list is never extended. --------- Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
This commit is contained in:
miga-heygen
co-authored by
Miguel Ángel
Miguel Ángel
parent
1de8b5ae33
commit
9460c983e0
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -552,6 +552,9 @@ type SpawnOutcome =
|
||||
stdoutChunks?: Buffer[];
|
||||
};
|
||||
|
||||
const NO_DECODED_FRAMES: SpawnOutcome = { kind: "exit", code: 0, stdout: "" };
|
||||
const metadataProbes = (calls: SpawnCall[]) => calls.filter((c) => c.args.includes("-show_format"));
|
||||
|
||||
function createSpawnSpy(outcomes: SpawnOutcome[]): {
|
||||
spawn: (command: string, args: readonly string[]) => FakeProc;
|
||||
calls: SpawnCall[];
|
||||
@@ -656,6 +659,8 @@ describe("media metadata cache invalidation", () => {
|
||||
format: { duration: "1" },
|
||||
}),
|
||||
},
|
||||
NO_DECODED_FRAMES,
|
||||
NO_DECODED_FRAMES,
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
@@ -666,6 +671,7 @@ describe("media metadata cache invalidation", () => {
|
||||
format: { duration: "2" },
|
||||
}),
|
||||
},
|
||||
NO_DECODED_FRAMES,
|
||||
]);
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
@@ -680,7 +686,7 @@ describe("media metadata cache invalidation", () => {
|
||||
const refreshed = await extractAudioMetadata(file);
|
||||
expect(refreshed).toMatchObject({ durationSeconds: 2, sampleRate: 44100, channels: 2 });
|
||||
expect(await extractAudioMetadata(file)).toBe(refreshed);
|
||||
expect(calls).toHaveLength(2);
|
||||
expect(metadataProbes(calls)).toHaveLength(2);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
@@ -753,6 +759,8 @@ describe("media metadata cache invalidation", () => {
|
||||
format: { duration: "1" },
|
||||
}),
|
||||
},
|
||||
NO_DECODED_FRAMES,
|
||||
NO_DECODED_FRAMES,
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
@@ -761,6 +769,7 @@ describe("media metadata cache invalidation", () => {
|
||||
format: { duration: "2" },
|
||||
}),
|
||||
},
|
||||
NO_DECODED_FRAMES,
|
||||
]);
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
@@ -768,7 +777,7 @@ describe("media metadata cache invalidation", () => {
|
||||
try {
|
||||
expect(await extractAudioMetadata(file)).toMatchObject({ durationSeconds: 1 });
|
||||
expect(await extractAudioMetadata(file)).toMatchObject({ durationSeconds: 2 });
|
||||
expect(calls).toHaveLength(2);
|
||||
expect(metadataProbes(calls)).toHaveLength(2);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
@@ -905,99 +914,136 @@ describe("ffprobe missing-binary fallback", () => {
|
||||
}
|
||||
});
|
||||
|
||||
// `profile` matters now: the packet refinement is an allowlist on AAC-LC,
|
||||
// because the 1024-sample formula is wrong for LD/ELD/HE and unverified for
|
||||
// the rest. An unprofiled "aac" stream deliberately keeps its container
|
||||
// duration rather than being refined on an assumption.
|
||||
// The container always reports 1.25s; each case varies what the decode probe finds.
|
||||
// `spawns` says whether the probe ran (no-frames also runs the full-scan fallback).
|
||||
it.each([
|
||||
{
|
||||
name: "non-AAC metadata",
|
||||
codec: "mp3",
|
||||
profile: undefined,
|
||||
packets: undefined,
|
||||
expected: 1.25,
|
||||
calls: 1,
|
||||
name: "non-AAC codec, decoded duration well beyond the container summary",
|
||||
codec: "flac",
|
||||
sampleRate: "48000",
|
||||
lastFrame: { best_effort_timestamp_time: "3.9", nb_samples: 4800 },
|
||||
expected: 4.0, // 3.9 + 4800/48000
|
||||
spawns: 2,
|
||||
},
|
||||
{
|
||||
name: "unprofiled AAC",
|
||||
name: "decoded duration within probe-precision margin of the summary — kept as-is",
|
||||
codec: "aac",
|
||||
profile: undefined,
|
||||
packets: "783",
|
||||
sampleRate: "48000",
|
||||
// 1.2 + 3840/48000 = 1.28, only 0.03s over the 1.25s summary — under
|
||||
// the margin, so the summary (edit-list-corrected) duration wins.
|
||||
lastFrame: { best_effort_timestamp_time: "1.2", nb_samples: 3840 },
|
||||
expected: 1.25,
|
||||
calls: 1,
|
||||
spawns: 2,
|
||||
},
|
||||
{
|
||||
name: "valid AAC-LC packet count",
|
||||
codec: "aac",
|
||||
profile: "LC",
|
||||
packets: "783",
|
||||
expected: 16.704,
|
||||
calls: 2,
|
||||
name: "no decodable frames — keeps the container duration",
|
||||
codec: "opus",
|
||||
sampleRate: "48000",
|
||||
lastFrame: undefined,
|
||||
expected: 1.25,
|
||||
// An empty tail read falls back to a full scan, which also finds
|
||||
// nothing (the mock's last outcome — the empty stdout — repeats).
|
||||
spawns: 3,
|
||||
},
|
||||
{
|
||||
name: "missing AAC packet count",
|
||||
name: "zero sample rate — skips the decode probe entirely",
|
||||
codec: "aac",
|
||||
profile: "LC",
|
||||
packets: undefined,
|
||||
sampleRate: "0",
|
||||
// Would have won on any other row; the probe is never spawned to see it.
|
||||
lastFrame: { best_effort_timestamp_time: "10", nb_samples: 999_999 },
|
||||
expected: 1.25,
|
||||
calls: 2,
|
||||
},
|
||||
{
|
||||
name: "zero AAC packet count",
|
||||
codec: "aac",
|
||||
profile: "LC",
|
||||
packets: "0",
|
||||
expected: 1.25,
|
||||
calls: 2,
|
||||
},
|
||||
{
|
||||
name: "invalid AAC packet count",
|
||||
codec: "aac",
|
||||
profile: "LC",
|
||||
packets: "invalid",
|
||||
expected: 1.25,
|
||||
calls: 2,
|
||||
spawns: 1,
|
||||
},
|
||||
])(
|
||||
"derives audio duration for $name",
|
||||
async ({ codec, profile, packets, expected, calls: expectedCalls }) => {
|
||||
const outcomes: SpawnOutcome[] = [
|
||||
async ({ codec, sampleRate, lastFrame, expected, spawns }) => {
|
||||
const { spawn, calls } = createSpawnSpy([
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({
|
||||
streams: [
|
||||
{
|
||||
codec_type: "audio",
|
||||
codec_name: codec,
|
||||
sample_rate: "48000",
|
||||
channels: 2,
|
||||
profile,
|
||||
},
|
||||
{ codec_type: "audio", codec_name: codec, sample_rate: sampleRate, channels: 2 },
|
||||
],
|
||||
format: { duration: "1.25", bit_rate: "128000" },
|
||||
}),
|
||||
},
|
||||
];
|
||||
if (codec === "aac" && profile === "LC") {
|
||||
outcomes.push({
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({ streams: [{ nb_read_packets: packets }], format: {} }),
|
||||
});
|
||||
}
|
||||
const { spawn, calls } = createSpawnSpy(outcomes);
|
||||
stdout: lastFrame
|
||||
? `${lastFrame.best_effort_timestamp_time},${lastFrame.nb_samples}\n`
|
||||
: "",
|
||||
},
|
||||
]);
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
|
||||
const { extractAudioMetadata } = await import("./ffprobe.js");
|
||||
const meta = await extractAudioMetadata(`/tmp/${codec}-${packets ?? "none"}.audio`);
|
||||
const meta = await extractAudioMetadata(`/tmp/${codec}-${sampleRate}.audio`);
|
||||
|
||||
expect(meta.durationSeconds).toBeCloseTo(expected, 6);
|
||||
expect(calls).toHaveLength(expectedCalls);
|
||||
expect(calls).toHaveLength(spawns);
|
||||
},
|
||||
);
|
||||
|
||||
// A bad seek index can make the tail read find nothing past a real final frame.
|
||||
it("falls back to a full decode when the tail-seeked read finds no frames", async () => {
|
||||
const { spawn, calls } = createSpawnSpy([
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({
|
||||
streams: [{ codec_type: "audio", codec_name: "opus", sample_rate: "48000", channels: 2 }],
|
||||
format: { duration: "1.25" },
|
||||
}),
|
||||
},
|
||||
{ kind: "exit", code: 0, stdout: "" }, // tail-seeked read: nothing found
|
||||
{ kind: "exit", code: 0, stdout: "1.4,4800\n" }, // full-scan fallback: real frame
|
||||
]);
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
|
||||
const { extractAudioMetadata } = await import("./ffprobe.js");
|
||||
const meta = await extractAudioMetadata("/tmp/tail-seek-miss.opus");
|
||||
|
||||
expect(meta.durationSeconds).toBeCloseTo(1.5, 6); // 1.4 + 4800/48000, well past the margin
|
||||
expect(calls).toHaveLength(3);
|
||||
expect(calls[1]?.args).toContain("-read_intervals");
|
||||
expect(calls[2]?.args).not.toContain("-read_intervals");
|
||||
});
|
||||
|
||||
// An abort is the caller's own intent, so the decode probe must surface it
|
||||
// rather than swallow it the way it swallows a genuine probe failure. The
|
||||
// signal is aborted only once the decode probe has spawned — aborting up
|
||||
// front trips the container probe instead, leaving this path untested.
|
||||
it("propagates an abort raised during the decode probe", async () => {
|
||||
const controller = new AbortController();
|
||||
const { spawn: baseSpawn, calls } = createSpawnSpy([
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({
|
||||
streams: [{ codec_type: "audio", codec_name: "opus", sample_rate: "48000", channels: 2 }],
|
||||
format: { duration: "1.25" },
|
||||
}),
|
||||
},
|
||||
{ kind: "exit", code: 0, stdout: "9,960\n" },
|
||||
]);
|
||||
const spawn = (command: string, args: readonly string[]) => {
|
||||
if (calls.length === 1) controller.abort();
|
||||
return baseSpawn(command, args);
|
||||
};
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
|
||||
const { extractAudioMetadata } = await import("./ffprobe.js");
|
||||
await expect(
|
||||
extractAudioMetadata("/tmp/aborted.audio", { signal: controller.signal }),
|
||||
).rejects.toThrow();
|
||||
expect(calls).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("extractMediaMetadata falls back to PNG cICP metadata when ffprobe is missing", async () => {
|
||||
const { spawn, calls } = createSpawnSpy([{ kind: "missing" }]);
|
||||
hidePathBinaries();
|
||||
@@ -1427,9 +1473,6 @@ describe("ffprobe option separator", () => {
|
||||
{
|
||||
codec_type: "audio",
|
||||
codec_name: "aac",
|
||||
// LC, so the packet-count refinement actually runs and its
|
||||
// argv is covered here too.
|
||||
profile: "LC",
|
||||
sample_rate: "48000",
|
||||
channels: 2,
|
||||
},
|
||||
@@ -1437,14 +1480,9 @@ describe("ffprobe option separator", () => {
|
||||
format: { duration: "1.25" },
|
||||
}),
|
||||
},
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({
|
||||
streams: [{ nb_read_packets: "783" }],
|
||||
format: {},
|
||||
}),
|
||||
},
|
||||
// The decode-based duration probe also runs, so its argv (including its
|
||||
// own `--`) is covered here too.
|
||||
{ kind: "exit", code: 0, stdout: "1.2,3840\n" },
|
||||
{ kind: "exit", code: 0, stdout: "0.000\n1.000\n" },
|
||||
]);
|
||||
vi.resetModules();
|
||||
@@ -1688,7 +1726,7 @@ describe("pix_fmt alpha detection", () => {
|
||||
it.each(OPAQUE)("reports %s as opaque", (fmt) => expect(pixelFormatHasAlpha(fmt)).toBe(false));
|
||||
});
|
||||
|
||||
describe("AAC duration refinement must never fail or distort the call", () => {
|
||||
describe("audio duration decode probe never fails the call and needs no profile allowlist", () => {
|
||||
afterEach(() => {
|
||||
vi.resetModules();
|
||||
vi.doUnmock("child_process");
|
||||
@@ -1710,91 +1748,142 @@ describe("AAC duration refinement must never fail or distort the call", () => {
|
||||
return { meta: await extractAudioMetadata(file), calls };
|
||||
}
|
||||
|
||||
// Regression: the refinement had no try/catch, so its failure rejected a
|
||||
// Regression: the old refinement had no try/catch, so its failure rejected a
|
||||
// call whose duration was already correct. htmlCompiler catches that as
|
||||
// "no audio stream", returns 0, and the render ships silent.
|
||||
it("keeps the container duration when the packet probe fails", async () => {
|
||||
const { meta } = await probe(
|
||||
it("keeps the container duration when the decode probe fails", async () => {
|
||||
const { meta, calls } = await probe(
|
||||
[
|
||||
{ kind: "exit", code: 0, stdout: aacStream("LC") },
|
||||
{ kind: "exit", code: 0, stdout: aacStream() },
|
||||
{ kind: "exit", code: 1, stdout: "", stderr: "ffprobe exploded" },
|
||||
],
|
||||
"/tmp/aac-packet-probe-fails.m4a",
|
||||
"/tmp/decode-probe-fails.m4a",
|
||||
);
|
||||
expect(meta.durationSeconds).toBe(600);
|
||||
// The tail-seeked attempt fails, so the full-scan fallback also runs (and
|
||||
// also fails, via the mock's repeated last outcome) before giving up.
|
||||
expect(calls).toHaveLength(3);
|
||||
});
|
||||
|
||||
it("keeps the container duration when the packet probe returns junk", async () => {
|
||||
const { meta } = await probe(
|
||||
it("keeps the container duration when the decode probe returns junk", async () => {
|
||||
const { meta, calls } = await probe(
|
||||
[
|
||||
{ kind: "exit", code: 0, stdout: aacStream("LC") },
|
||||
{ kind: "exit", code: 0, stdout: aacStream() },
|
||||
{ kind: "exit", code: 0, stdout: "not json at all" },
|
||||
],
|
||||
"/tmp/aac-packet-probe-junk.m4a",
|
||||
"/tmp/decode-probe-junk.m4a",
|
||||
);
|
||||
expect(meta.durationSeconds).toBe(600);
|
||||
expect(calls).toHaveLength(3);
|
||||
});
|
||||
|
||||
// Regression: codec_name is "aac" for HE-AAC too, but its packets carry
|
||||
// 2048 output samples — assuming 1024 halved a 10:00 podcast to 5:00.
|
||||
// The old refinement needed a profile allowlist because it guessed a fixed
|
||||
// samples-per-frame value (right for AAC-LC, wrong for HE-AAC/LD/ELD/etc).
|
||||
// Decoding the real last frame's own sample count has no such assumption,
|
||||
// so every profile goes through the same path with no special-casing.
|
||||
it.each(["LC", "HE-AAC", "HE-AACv2", "LD", "ELD", "Main", "", "SomethingNew"])(
|
||||
"corrects duration for AAC profile %s, with no allowlist",
|
||||
async (profile) => {
|
||||
const { meta, calls } = await probe(
|
||||
[
|
||||
{ kind: "exit", code: 0, stdout: aacStream(profile) },
|
||||
{ kind: "exit", code: 0, stdout: "603.9,4410\n" },
|
||||
],
|
||||
`/tmp/aac-${profile || "none"}.m4a`,
|
||||
);
|
||||
expect(meta.durationSeconds).toBeCloseTo(604, 5); // 603.9 + 4410/44100
|
||||
expect(calls).toHaveLength(2);
|
||||
},
|
||||
);
|
||||
|
||||
const audioProbe = (stream: object, format: object) =>
|
||||
JSON.stringify({
|
||||
streams: [{ codec_type: "audio", codec_name: "aac", sample_rate: "44100", ...stream }],
|
||||
format: { duration: "600", ...format },
|
||||
});
|
||||
const readInterval = (calls: SpawnCall[]) =>
|
||||
calls[1]?.args[calls[1].args.indexOf("-read_intervals") + 1];
|
||||
|
||||
it.each([
|
||||
"HE-AAC",
|
||||
"HE-AACv2",
|
||||
"he-aac",
|
||||
// 512- and 480-sample framing: the 1024 multiplier overstates these by
|
||||
// 2x and ~2.13x, overwriting an already-correct container duration.
|
||||
"LD",
|
||||
"ELD",
|
||||
// Not verified for this maths, so not allowlisted.
|
||||
"Main",
|
||||
"SSR",
|
||||
"LTP",
|
||||
"xHE-AAC",
|
||||
// Missing or unrecognised profile must NOT fall through to the formula —
|
||||
// that is how an unknown HE spelling kept the truncation bug.
|
||||
"",
|
||||
"SomethingNew",
|
||||
])("does not apply the LC packet maths to profile %s", async (profile) => {
|
||||
["the stream", { start_time: "1.4" }, {}],
|
||||
["the format, when the stream has none", {}, { start_time: "1.4" }],
|
||||
])("measures the decoded end from the start time of %s", async (_, stream, format) => {
|
||||
const { meta, calls } = await probe(
|
||||
[{ kind: "exit", code: 0, stdout: aacStream(profile) }],
|
||||
`/tmp/heaac-${profile}.m4a`,
|
||||
[
|
||||
{ kind: "exit", code: 0, stdout: audioProbe(stream, format) },
|
||||
{ kind: "exit", code: 0, stdout: "601.3,4410\n" },
|
||||
],
|
||||
`/tmp/start-offset-${Object.keys(format).length}.ts`,
|
||||
);
|
||||
expect(meta.durationSeconds).toBe(600);
|
||||
// The second probe is not even attempted.
|
||||
expect(calls).toHaveLength(1);
|
||||
expect(meta.durationSeconds).toBe(600); // 601.3 + 0.1 - 1.4 is within the margin
|
||||
expect(readInterval(calls)).toBe("600.4%");
|
||||
});
|
||||
|
||||
it.each(["LC", " lc "])("still refines AAC profile %s", async (profile) => {
|
||||
it.each([
|
||||
["trusts a shorter decode for raw ADTS", "aac", 10],
|
||||
["keeps the container for an MP4, whose edit list may trim", "mov,mp4,m4a,3gp,3g2,mj2", 600],
|
||||
])("%s", async (_, formatName, expected) => {
|
||||
const { meta } = await probe(
|
||||
[
|
||||
{ kind: "exit", code: 0, stdout: aacStream(profile) },
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({ streams: [{ nb_read_packets: "861" }], format: {} }),
|
||||
},
|
||||
{ kind: "exit", code: 0, stdout: audioProbe({}, { format_name: formatName }) },
|
||||
{ kind: "exit", code: 0, stdout: "9.9,4410\n" },
|
||||
],
|
||||
`/tmp/aac-lc-${profile.trim()}.m4a`,
|
||||
`/tmp/shorter-decode-${formatName.length}.aac`,
|
||||
);
|
||||
expect(meta.durationSeconds).toBeCloseTo((861 * 1024) / 44100, 5);
|
||||
expect(meta.durationSeconds).toBeCloseTo(expected, 5);
|
||||
});
|
||||
|
||||
it("still refines a plain AAC-LC stream", async () => {
|
||||
it("reads the last frame of a decode listing longer than the stdout cap", async () => {
|
||||
const rows = Array.from({ length: 10000 }, (_, i) => `${(i / 100).toFixed(2)},4410`);
|
||||
const { meta } = await probe(
|
||||
[
|
||||
{ kind: "exit", code: 0, stdout: aacStream("LC") },
|
||||
{
|
||||
kind: "exit",
|
||||
code: 0,
|
||||
stdout: JSON.stringify({ streams: [{ nb_read_packets: "861" }], format: {} }),
|
||||
},
|
||||
{ kind: "exit", code: 0, stdout: aacStream() },
|
||||
{ kind: "exit", code: 0, stdout: `${rows.join("\n")}\n603.9,4410\n` },
|
||||
],
|
||||
"/tmp/aac-lc-refined.m4a",
|
||||
"/tmp/long-decode-listing.m4a",
|
||||
);
|
||||
expect(meta.durationSeconds).toBeCloseTo((861 * 1024) / 44100, 5);
|
||||
expect(meta.durationSeconds).toBeCloseTo(604, 5);
|
||||
});
|
||||
});
|
||||
|
||||
// honest.mp4 is FLAC-in-MP4; lying.mp4 is the same bytes with the mvhd/tkhd/mdhd
|
||||
// durations halved, every audio frame intact.
|
||||
const HAS_FFPROBE = spawnSync("ffprobe", ["-version"]).status === 0;
|
||||
|
||||
describe.skipIf(!HAS_FFPROBE)(
|
||||
"extractAudioMetadata recovers a real lying container's true duration",
|
||||
() => {
|
||||
const honestPath = resolve(__dirname, "__fixtures__/lying-container/honest.mp4");
|
||||
const lyingPath = resolve(__dirname, "__fixtures__/lying-container/lying.mp4");
|
||||
|
||||
it("recovers the true duration from a FLAC-in-MP4 file whose container atoms were patched to lie", async () => {
|
||||
const { extractAudioMetadata } = await import("./ffprobe.js");
|
||||
const trueMeta = await extractAudioMetadata(honestPath);
|
||||
expect(trueMeta.audioCodec).toBe("flac");
|
||||
expect(trueMeta.durationSeconds).toBeGreaterThan(3.9);
|
||||
|
||||
const lyingMeta = await extractAudioMetadata(lyingPath);
|
||||
|
||||
// Recovers the real duration — not the halved container summary.
|
||||
expect(lyingMeta.durationSeconds).toBeCloseTo(trueMeta.durationSeconds, 1);
|
||||
});
|
||||
|
||||
// 2 s of AAC in MPEG-TS starting at 1.4 s: last frame 3.448 + 1024/16000 - 1.4.
|
||||
it("does not add an MPEG-TS start time to the duration", async () => {
|
||||
const { extractAudioMetadata } = await import("./ffprobe.js");
|
||||
const path = resolve(__dirname, "__fixtures__/lying-container/start-offset.mpegts");
|
||||
expect((await extractAudioMetadata(path)).durationSeconds).toBeCloseTo(2.112, 3);
|
||||
});
|
||||
|
||||
// 2 s silence + 1 s noise as raw ADTS; ffprobe 8.1 estimates about 9.1 s from the quiet opening.
|
||||
it("corrects a raw ADTS duration that the container overclaims", async () => {
|
||||
const { extractAudioMetadata } = await import("./ffprobe.js");
|
||||
const path = resolve(__dirname, "__fixtures__/lying-container/overclaim.aac");
|
||||
expect((await extractAudioMetadata(path)).durationSeconds).toBeCloseTo(3.072, 3);
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
describe("final video frame timestamp probes", () => {
|
||||
afterEach(() => {
|
||||
vi.resetModules();
|
||||
|
||||
@@ -157,8 +157,14 @@ interface MediaProbeCacheEntry<T = FFProbeOutput> {
|
||||
const mediaProbeOutputCache = new Map<string, MediaProbeCacheEntry>();
|
||||
const mediaProbeOutputSignalCaches = new WeakMap<AbortSignal, Map<string, MediaProbeCacheEntry>>();
|
||||
const MEDIA_PROBE_OUTPUT_CACHE_MAX_ENTRIES = 128;
|
||||
// FFmpeg's built-in AAC encoder emits AAC-LC, which has 1024 samples per packet.
|
||||
const AAC_LC_SAMPLES_PER_PACKET = 1024;
|
||||
// Ignore decoded-vs-container duration differences below this: ffprobe's
|
||||
// container summary is edit-list-corrected (encoder priming delay trimmed) while
|
||||
// a decoded final frame's own timestamp + sample count is not, so the two differ
|
||||
// by a few milliseconds even on an honest file.
|
||||
const AUDIO_DURATION_PROBE_MARGIN_SECONDS = 0.05;
|
||||
// How far before the container's claimed end the decode probe starts reading,
|
||||
// so it decodes only the tail instead of the whole stream.
|
||||
const AUDIO_DURATION_PROBE_TAIL_SECONDS = 1;
|
||||
|
||||
export interface VideoColorSpace {
|
||||
/** Color transfer characteristics, e.g. "bt709", "smpte2084", "arib-std-b67" */
|
||||
@@ -236,6 +242,7 @@ interface FFProbeStream {
|
||||
|
||||
interface FFProbeFormat {
|
||||
duration?: string;
|
||||
start_time?: string;
|
||||
bit_rate?: string;
|
||||
format_name?: string;
|
||||
}
|
||||
@@ -959,59 +966,18 @@ export async function extractAudioMetadata(
|
||||
const streamDuration = audioStream.duration ? parseFloat(audioStream.duration) : undefined;
|
||||
const sampleRate = audioStream.sample_rate ? parseInt(audioStream.sample_rate) : 44100;
|
||||
const audioCodec = audioStream.codec_name || "unknown";
|
||||
// AAC-LC container durations are often slightly wrong, so the packet
|
||||
// count gives a better one. Three constraints on that refinement:
|
||||
//
|
||||
// 1. It must never fail the call. durationSeconds is ALREADY correct from
|
||||
// format.duration at this point. `-count_packets` demuxes the whole
|
||||
// container against runFfprobe's fixed 30s deadline, so a long file on
|
||||
// slow or network storage times out — and the caller in htmlCompiler
|
||||
// catches that under "Source file has no audio stream", returns
|
||||
// duration 0, drops the audio element and ships a silent render.
|
||||
// 2. It must honour the caller's AbortSignal. Only the first probe
|
||||
// received it, so aborting during this one was ignored and the call
|
||||
// resolved with full metadata long after cancellation.
|
||||
// 3. It must apply ONLY to profiles whose 1024-sample framing is
|
||||
// established. ffprobe reports codec_name "aac" for every AAC
|
||||
// variant — the framing lives in the profile:
|
||||
//
|
||||
// LC 1024 samples/frame <- the only one this maths fits
|
||||
// HE-AAC v1/v2 2048 output samples against a doubled sample_rate
|
||||
// LD 512
|
||||
// ELD 480
|
||||
// Main/SSR/LTP 1024 nominally, but not verified here
|
||||
// xHE-AAC (USAC) variable
|
||||
//
|
||||
// An ALLOWLIST, not a HE-AAC denylist. The denylist form let LD/ELD
|
||||
// through (halving to a third of the true duration), and let an
|
||||
// unknown or missing profile through too — so an unrecognised HE
|
||||
// spelling preserved the exact truncation this is meant to close.
|
||||
// A skipped refinement is harmless: format.duration is already correct.
|
||||
const isAacLc = /^\s*LC\s*$/i.test(audioStream.profile ?? "");
|
||||
if (audioCodec === "aac" && isAacLc && sampleRate > 0) {
|
||||
try {
|
||||
const packetStdout = await runFfprobe(
|
||||
filePath,
|
||||
[
|
||||
"-select_streams",
|
||||
"a:0",
|
||||
"-count_packets",
|
||||
"-show_entries",
|
||||
"stream=nb_read_packets",
|
||||
"-print_format",
|
||||
"json",
|
||||
],
|
||||
options?.signal,
|
||||
);
|
||||
const packetOutput = parseProbeJson(packetStdout);
|
||||
const packetCount = Number(packetOutput.streams[0]?.nb_read_packets);
|
||||
if (Number.isFinite(packetCount) && packetCount > 0) {
|
||||
durationSeconds = (packetCount * AAC_LC_SAMPLES_PER_PACKET) / sampleRate;
|
||||
}
|
||||
} catch (error) {
|
||||
// An abort is the caller's intent, not a refinement failure — let it
|
||||
// through. Anything else keeps the container duration we already have.
|
||||
if (options?.signal?.aborted) throw error;
|
||||
// Container summaries can undercount intact audio, so the decoded last frame wins.
|
||||
// The probe returns null instead of throwing: a throw here drops the audio element.
|
||||
if (sampleRate > 0) {
|
||||
const decoded = await probeDecodedAudioDuration(
|
||||
filePath,
|
||||
sampleRate,
|
||||
audioStartSeconds(audioStream, output.format),
|
||||
durationSeconds,
|
||||
options?.signal,
|
||||
);
|
||||
if (decoded !== null) {
|
||||
durationSeconds = reconcileAudioDuration(durationSeconds, decoded, output.format);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1030,6 +996,83 @@ export async function extractAudioMetadata(
|
||||
return probePromise;
|
||||
}
|
||||
|
||||
function audioStartSeconds(stream: FFProbeStream, format: FFProbeFormat): number {
|
||||
const start = parseFloat(stream.start_time ?? format.start_time ?? "");
|
||||
return Number.isFinite(start) ? start : 0;
|
||||
}
|
||||
|
||||
// Raw ADTS durations are bitrate estimates that overclaim after a quiet opening, so a shorter
|
||||
// decode wins there. Elsewhere a shorter decode can be an edit-list trim, so only longer wins.
|
||||
function reconcileAudioDuration(
|
||||
containerSeconds: number,
|
||||
decodedSeconds: number,
|
||||
format: FFProbeFormat,
|
||||
): number {
|
||||
const difference = decodedSeconds - containerSeconds;
|
||||
const trustShorter = format.format_name === "aac";
|
||||
if (difference > AUDIO_DURATION_PROBE_MARGIN_SECONDS) return decodedSeconds;
|
||||
if (trustShorter && difference < -AUDIO_DURATION_PROBE_MARGIN_SECONDS) return decodedSeconds;
|
||||
return containerSeconds;
|
||||
}
|
||||
|
||||
/**
|
||||
* True audio duration from the last decoded frame (timestamp + nb_samples - stream start): tail
|
||||
* first, then the whole file if a bad seek finds nothing. Null, never a throw, except on abort.
|
||||
*/
|
||||
async function probeDecodedAudioDuration(
|
||||
filePath: string,
|
||||
sampleRate: number,
|
||||
startSeconds: number,
|
||||
containerDurationSeconds: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<number | null> {
|
||||
const probe = async (
|
||||
readInterval?: string,
|
||||
): Promise<{ timestamp: number; nbSamples: number } | undefined> => {
|
||||
const args = [
|
||||
"-select_streams",
|
||||
"a:0",
|
||||
"-show_entries",
|
||||
"frame=best_effort_timestamp_time,nb_samples",
|
||||
"-of",
|
||||
"csv=p=0",
|
||||
];
|
||||
if (readInterval) args.unshift("-read_intervals", readInterval);
|
||||
try {
|
||||
const stdout = await runFfprobe(
|
||||
filePath,
|
||||
args,
|
||||
signal,
|
||||
// CSV, not JSON: a tail truncated under the size cap still parses.
|
||||
{ retainTail: true, maxChars: 64 * 1024 },
|
||||
);
|
||||
return stdout
|
||||
.split("\n")
|
||||
.map((line) => {
|
||||
const [timestampText, samplesText] = line.trim().split(",");
|
||||
const timestamp = Number(timestampText);
|
||||
const nbSamples = Number(samplesText);
|
||||
return Number.isFinite(timestamp) && Number.isFinite(nbSamples)
|
||||
? { timestamp, nbSamples }
|
||||
: undefined;
|
||||
})
|
||||
.filter((frame) => frame !== undefined)
|
||||
.at(-1);
|
||||
} catch (error) {
|
||||
if (signal?.aborted) throw signal.reason ?? error;
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
const tailStart = Math.max(
|
||||
startSeconds,
|
||||
startSeconds + containerDurationSeconds - AUDIO_DURATION_PROBE_TAIL_SECONDS,
|
||||
);
|
||||
const lastFrame = (await probe(`${tailStart}%`)) ?? (await probe());
|
||||
if (!lastFrame) return null;
|
||||
return lastFrame.timestamp + lastFrame.nbSamples / sampleRate - startSeconds;
|
||||
}
|
||||
|
||||
export interface KeyframeAnalysis {
|
||||
avgIntervalSeconds: number;
|
||||
maxIntervalSeconds: number;
|
||||
|
||||
@@ -594,7 +594,7 @@ async function defaultProbeAudioInfo(
|
||||
audioPath: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<AudioProbeInfo> {
|
||||
// The shared ffprobe wrapper derives AAC-LC duration from packet count so
|
||||
// The shared ffprobe wrapper corrects for a lying container duration so
|
||||
// every consumer sees the same VBR-safe metadata while preserving cancellation.
|
||||
const metadata: AudioMetadata = await extractAudioMetadata(audioPath, { signal });
|
||||
return {
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a0aa5f4a6c032286630b386ea53feea7739a8deef5792a4c770bff54ebf171ae
|
||||
size 277335
|
||||
oid sha256:45a3e8e979aed5ca14e6896366231bc7767a8d4526ffaeed0282c31c31a10dd8
|
||||
size 277990
|
||||
|
||||
@@ -1048,7 +1048,7 @@
|
||||
</div>
|
||||
|
||||
<!-- Background Audio (A-roll Audio) -->
|
||||
<audio id="aroll-audio" src="_remote_media/download_054a544691a0.mp4" data-start="0" data-duration="16.064" data-track-index="1" data-end="16.064"></audio>
|
||||
<audio id="aroll-audio" src="_remote_media/download_054a544691a0.mp4" data-start="0" data-duration="16.042" data-track-index="1" data-end="16.042"></audio>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user