mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-07 18:26:17 +00:00
fix(engine): treat ffmpegStreamingTimeout as per-frame inactivity, not total render time (#901)
## Summary - Convert `streamingEncoder.ts`'s safety timer from a total-render hard cap to a per-frame inactivity timeout - Reset the timer only on `accepted === true` writes — buffered writes don't count as consumer progress - Update the `ffmpegStreamingTimeout` config doc to reflect the new semantics ## The bug The timer was set once at spawn and fired SIGTERM unconditionally at `ffmpegStreamingTimeout` ms — turning a "FFmpeg is hung" guard into a hard cap on total render duration. Slow-but-progressing captures (CI runner under load, large compositions, slower compositor paths after [#838](https://github.com/heygen-com/hyperframes/pull/838)'s always-clip change) regularly exceeded the 600s default and were killed mid-encode. The symptom surfaced as: ``` Streaming encode failed: FFmpeg exited with code 255 video:NNNkB audio:0kB ... [libx264 @ ...] frame I:3 Avg QP:12.91 size: 73263 [libx264 @ ...] frame P:431 Avg QP:14.72 size: 31633 ... [libx264 @ ...] kb/s:7661.05 Exiting normally, received signal 15. ``` libx264 had encoded most frames cleanly; SIGTERM arrived during the encode, libx264 printed its end-of-encode stats, and Node observed a non-zero exit. The `audio:0kB` in stderr is incidental — `streamingEncoder` is video-only; audio is muxed later in `assembleStage`. Downstream reproduction: `style-13-prod` fails deterministically in `heygen-com/hyperframes-internal` CI after bumping `@hyperframes/producer` from 0.6.7 → 0.6.10. Bisects to #838 widening the SDR capture path at dpr=1 — same composition shape, slower per-frame, total render now crosses 600s. ## The fix Convert the timer to a heartbeat: each `writeFrame` that goes through to the kernel pipe (i.e. `stdin.write` returns `true`) resets it. Only true hangs (no successful frame write for the timeout window) trip SIGTERM now; "slow but progressing" renders are unbounded. Crucially, the heartbeat does **not** reset on `accepted === false`. A `false` return means Node had to buffer the write because FFmpeg hasn't drained the pipe yet — that's not proof of consumer progress, just proof we produced. Without this distinction, a hung FFmpeg with a live Chrome would queue frames into Node's writable buffer indefinitely (no backpressure path back to the capture loop) and grow until OOM. In steady state with a slow-but-alive FFmpeg, writes alternate between `true` and `false` as the buffer drains and refills; the `true`s are enough to keep the heartbeat ticking. Renames are intentionally avoided — `ffmpegStreamingTimeout` keeps its name and `600_000` default; only the semantics changed. The config doc spells out the new behavior so downstream consumers know what 600s now means. ## Test plan - [x] **Slow-but-progressing capture** (`accepted=true`): 9× `writeFrame` at 900ms intervals (under the 1000ms threshold) — encoder stays alive through 8.1s. Stall past the threshold — SIGTERM fires. - [x] **Stalled FFmpeg with live producer** (`accepted=false`): override `stdin.write` to return false; pump 9× `writeFrame` at 900ms intervals. SIGTERM still fires inside the 1000ms window — buffered writes don't keep the heartbeat alive. - [x] Existing 33 tests in `streamingEncoder.test.ts` still pass - [x] Lint (`oxlint`) + format (`oxfmt --check`) clean - [ ] CI regression suite 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This commit is contained in:
@@ -572,4 +572,71 @@ describe("spawnStreamingEncoder lifecycle and cleanup", () => {
|
||||
controller.abort();
|
||||
expect(proc.kill).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("inactivity timeout fires only after a no-frame gap exceeds ffmpegStreamingTimeout", async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const { spawn, calls } = createSpawnSpy();
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
|
||||
const { spawnStreamingEncoder } = await import("./streamingEncoder.js");
|
||||
const dir = mkdtempSync(join(tmpdir(), "se-heartbeat-"));
|
||||
const encoder = await spawnStreamingEncoder(join(dir, "out.mp4"), baseOptions, undefined, {
|
||||
ffmpegStreamingTimeout: 1000,
|
||||
});
|
||||
|
||||
const proc = calls[0]!.proc;
|
||||
|
||||
// Frames every 900ms — under the 1000ms inactivity threshold — should
|
||||
// keep resetting the timer. After 9× 900ms = 8.1s of "slow but
|
||||
// progressing" capture the encoder must still be alive. The old total-
|
||||
// render timeout would have fired SIGTERM at ~1000ms.
|
||||
for (let i = 0; i < 9; i++) {
|
||||
encoder.writeFrame(Buffer.from([i]));
|
||||
vi.advanceTimersByTime(900);
|
||||
}
|
||||
expect(proc.kill).not.toHaveBeenCalled();
|
||||
|
||||
// Now stall — no writeFrame for longer than the threshold. SIGTERM fires.
|
||||
vi.advanceTimersByTime(1100);
|
||||
expect(proc.kill).toHaveBeenCalledWith("SIGTERM");
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("inactivity timeout still fires when stdin is backpressured (stalled ffmpeg, live producer)", async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
// Simulate the FFmpeg-hangs-but-Chrome-keeps-producing case: stdin.write
|
||||
// always returns false (Node has to buffer because ffmpeg isn't draining
|
||||
// the pipe). The heartbeat must NOT reset on those buffered writes —
|
||||
// otherwise a hung ffmpeg with a steady frame producer would never
|
||||
// SIGTERM and we'd grow Node's stdin buffer until OOM.
|
||||
const { spawn, calls } = createSpawnSpy();
|
||||
vi.resetModules();
|
||||
vi.doMock("child_process", () => ({ spawn }));
|
||||
|
||||
const { spawnStreamingEncoder } = await import("./streamingEncoder.js");
|
||||
const dir = mkdtempSync(join(tmpdir(), "se-backpressure-"));
|
||||
const encoder = await spawnStreamingEncoder(join(dir, "out.mp4"), baseOptions, undefined, {
|
||||
ffmpegStreamingTimeout: 1000,
|
||||
});
|
||||
|
||||
const proc = calls[0]!.proc;
|
||||
proc.stdin.write = (_chunk: Buffer) => false;
|
||||
|
||||
// Pump 9 frames at 900ms intervals — all returning false. The reset
|
||||
// should NOT fire (every write was buffered, not accepted), so the
|
||||
// 1000ms timer (last reset on spawn) elapses near the start.
|
||||
for (let i = 0; i < 9; i++) {
|
||||
encoder.writeFrame(Buffer.from([i]));
|
||||
vi.advanceTimersByTime(900);
|
||||
}
|
||||
expect(proc.kill).toHaveBeenCalledWith("SIGTERM");
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user