feat(producer): make sub-frame motion blur reachable from a render (#4020)

* feat(producer): make sub-frame motion blur reachable from a render

Second deliverable for issue #4010. The engine gained the accumulation path, but
nothing set CaptureOptions.motionBlur, so the feature could not be switched on
from a real render. RenderConfig now carries it and the producer threads it into
the capture options.

Doing that surfaced the defect that would have made it fail silently. "Does this
render capture PNG frames" was spelled four times, every one of them re-derived
from needsAlpha, which describes the OUTPUT container rather than the capture:

  probeStage        picked the probe capture format
  renderOrchestrator picked the render capture format
  encodeStage       built the encoder's frame_%06d input pattern, twice
  captureStage      built the disk worker's frame extension

Motion blur is a second, independent reason to capture PNG: the sub-frame samples
are averaged pixel by pixel, so JPEG samples would be averaged after a lossy
quantization. Add that reason to only one of those four and the encoder goes
looking for frame_%06d.jpg against files written as .png, and the render finds no
frames at all rather than reporting an error.

So the rule now has one owner, resolveCaptureImageFormat, which names both
reasons. Everyone downstream reads the resolved capture format instead of
re-deriving it: the encoder and the disk worker share frameFileExtension with
writeCapturedFrame, which is what actually names the files. needsAlpha keeps only
the question it really answers, whether the output preserves alpha.

captureStage no longer needs needsAlpha at all, which is the tell that the
re-derivation was its only reason to know.

No JPEG encoder is involved. The PNG capture path is already load-bearing, since
every transparent-output render uses it and the streaming encoder already selects
png as its input codec.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* fix(engine): resolve the motion-blur plan on every initialization exit

A real render produced no blur at all. The plan was correct, the producer
wiring was correct, and the sub-frame sample times were genuinely distinct, but
K=16 and K=64 rendered identically to no blur because neither ever ran.

initializeSession has two exits: screenshot mode finishes its own block and
returns, every other capture mode falls through to the end. The plan was
resolved at the fall-through exit only, and screenshot is the one capture mode
motion blur supports, so on exactly the path the feature runs on
session.motionBlur was never assigned. captureFrameCore then read it as
undefined and took the single-capture branch: one capture per output frame, no
accumulation, no error.

finalizeSessionInit is now the single owner of what finishing initialization
means. It sets the plan and isInitialized together and is called at both exits,
so a third exit cannot mark a session ready while forgetting the plan.

Why the existing tests passed, which is the part worth keeping: the accumulation
test set session.motionBlur directly on a stub session. It exercised the
mechanism while bypassing the only code that decides whether the mechanism runs
at all. A test that proves something works and never asks whether it is
reachable is not end to end, however much behaviour it drives.

Two tests. A structural guard that session.isInitialized = true appears exactly
once and that the same place resolves the plan, which fails if the defect is
reintroduced at either exit. And a sample-distinctness assertion that hashes the
K buffers captured within one output frame and requires K distinct ones, which
covers the different failure of every sub-frame seek landing on one instant.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

---------

Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>

packages/engine/src/index.ts

@@ -273,6 +273,8 @@ export {
 
 // ── Utilities ──────────────────────────────────────────────────────────────────
 export { quantizeTimeToFrame, MEDIA_VISUAL_STYLE_PROPERTIES } from "@hyperframes/core";
+export { frameFileExtension } from "./services/frameCapture.js";
+export type { MotionBlurOptions, MotionBlurBlendSpace } from "./services/motionBlur.js";
 
 export {
   assertSwiftShader,

packages/engine/src/services/frameCapture-motionBlur.test.ts

@@ -5,7 +5,7 @@ import {
   resolveSessionMotionBlur,
   type CaptureSession,
 } from "./frameCapture.js";
-import { mkdtempSync } from "fs";
+import { mkdtempSync, readFileSync } from "fs";
 import { tmpdir } from "os";
 import { join } from "path";
 import { resolveMotionBlurPlan } from "./motionBlur.js";
@@ -129,6 +129,23 @@ describe("sub-frame accumulation reaches the page with distinct sample times", (
     expect(seeks).toHaveLength(18);
   });
 
+  it("captures K distinct samples rather than K copies of one instant", async () => {
+    // Guards the failure mode where every sub-frame seek lands on the same time: the
+    // average of K identical frames is the unblurred frame, which looks like the feature
+    // doing nothing rather than like an error.
+    const captured: string[] = [];
+    vi.mocked(pageScreenshotCapture).mockImplementation(async () => {
+      const png = solidPng(16 * captured.length);
+      captured.push(png.toString("base64"));
+      return png;
+    });
+
+    await captureFrameToBuffer(makeSession(), 10, 10 / 30);
+
+    expect(captured).toHaveLength(16);
+    expect(new Set(captured).size).toBe(16);
+  });
+
   it("averages the captured samples into the output frame", async () => {
     const result = await captureFrameToBuffer(makeSession(), 10, 10 / 30);
 
@@ -184,6 +201,27 @@ describe("accumulation composes with static-frame dedup", () => {
   });
 });
 
+describe("every path that marks a session ready also resolves the plan", () => {
+  // The defect this locks: `initializeSession` has two exits, and screenshot mode (the only
+  // mode motion blur supports) returns from the first one. Resolving the plan at the other
+  // exit left `session.motionBlur` undefined on exactly the path the feature runs on, so a
+  // real render captured one frame per output frame and produced no blur at all, while the
+  // plan, the producer wiring and every unit test stayed correct.
+  const source = readFileSync(new URL("./frameCapture.ts", import.meta.url), "utf8");
+
+  it("assigns isInitialized in exactly one place", () => {
+    const assignments = source.match(/session\.isInitialized = true/g) ?? [];
+    expect(assignments).toHaveLength(1);
+  });
+
+  it("resolves the motion-blur plan in that same place", () => {
+    const start = source.indexOf("function finalizeSessionInit");
+    const finalize = source.slice(start, start + 400);
+    expect(finalize).toContain("resolveSessionMotionBlur");
+    expect(finalize).toContain("session.isInitialized = true");
+  });
+});
+
 describe("resolveSessionMotionBlur rejects what accumulation cannot render", () => {
   const withOptions = (overrides: Partial<CaptureSession>): CaptureSession =>
     makeSession({

packages/engine/src/services/frameCapture.ts

@@ -2311,7 +2311,7 @@ export async function initializeSession(session: CaptureSession): Promise<void>
 
     await armStaticDedup(session, session.page, logInitPhase);
     await ensureRenderFrameSiblings(session.page);
-    session.isInitialized = true;
+    finalizeSessionInit(session);
     return;
   }
 
@@ -2512,9 +2512,7 @@ export async function initializeSession(session: CaptureSession): Promise<void>
   const commitCdp = await getCdpSession(page);
   await commitCdp.send("HeadlessExperimental.beginFrame", preparedBeginFrameTimeline.commitParams);
 
-  session.motionBlur = resolveSessionMotionBlur(session);
-
-  session.isInitialized = true;
+  finalizeSessionInit(session);
 }
 
 async function captureFrameErrorDiagnostics(
@@ -3748,6 +3746,20 @@ async function captureFrameSurface(
   return { buffer: screenshotBuffer, quantizedTime, seekMs, beforeCaptureMs, screenshotMs };
 }
 
+/**
+ * Mark the session ready to capture.
+ *
+ * The single owner of what finishing initialization means, because `initializeSession`
+ * has two exits: screenshot mode returns early, and every other mode falls through the
+ * end. Resolving motion blur at only one of them marks the session ready with no plan,
+ * which silently renders unblurred rather than failing. Both fields are set here so a
+ * third exit cannot forget one.
+ */
+function finalizeSessionInit(session: CaptureSession): void {
+  session.motionBlur = resolveSessionMotionBlur(session);
+  session.isInitialized = true;
+}
+
 /**
  * Capture one output frame as the average of `plan.samplesPerFrame` sub-frame captures.
  *
@@ -3881,6 +3893,16 @@ export async function captureFrame(
   return { frameIndex, time: quantizedTime, path: framePath, captureTimeMs };
 }
 
+/**
+ * File extension for a captured frame, keyed on the format the frames were actually
+ * captured in. The encoder's input pattern and the writer must agree, so both read this
+ * rather than re-deriving the answer from whether the OUTPUT needs alpha, which is a
+ * different question and stops being equivalent as soon as anything else forces PNG.
+ */
+export function frameFileExtension(format: "jpeg" | "png" | undefined): "png" | "jpg" {
+  return format === "png" ? "png" : "jpg";
+}
+
 /**
  * Write an already-captured frame buffer to the session's output dir using the
  * canonical `frame_NNNNNN.{jpg,png}` naming. `fileIndex` is the ENCODER-facing
@@ -3894,7 +3916,7 @@ export function writeCapturedFrame(
   fileIndex: number,
   buffer: Buffer,
 ): string {
-  const ext = session.options.format === "png" ? "png" : "jpg";
+  const ext = frameFileExtension(session.options.format);
   const framePath = join(session.outputDir, `frame_${String(fileIndex).padStart(6, "0")}.${ext}`);
   writeFileSync(framePath, buffer);
   return framePath;

packages/producer/src/services/distributed/renderChunk.ts

@@ -957,6 +957,7 @@ export async function renderChunk(
         width: plan.dimensions.width * encoder.deviceScaleFactor,
         height: plan.dimensions.height * encoder.deviceScaleFactor,
         needsAlpha: plan.dimensions.format !== "mp4",
+        captureImageFormat: captureOptions.format ?? "jpeg",
         // Each chunk produces video only — audio is muxed once at assemble
         // time. Suppressing `hasAudio` skips the png-sequence audio sidecar
         // AND the mp4 audio mux.

packages/producer/src/services/render/captureImageFormat.test.ts

@@ -0,0 +1,29 @@
+import { describe, expect, it } from "bun:test";
+import { frameFileExtension } from "@hyperframes/engine";
+import { resolveCaptureImageFormat } from "./captureImageFormat.js";
+
+describe("resolveCaptureImageFormat", () => {
+  it("captures JPEG for an ordinary opaque render", () => {
+    expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: undefined })).toBe("jpeg");
+  });
+
+  it("captures PNG when the output container keeps alpha", () => {
+    expect(resolveCaptureImageFormat({ needsAlpha: true, motionBlur: undefined })).toBe("png");
+  });
+
+  it("captures PNG for motion blur even when the output needs no alpha", () => {
+    // The sub-frame samples are averaged pixel by pixel, so they cannot be JPEG.
+    expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: {} })).toBe("png");
+  });
+
+  it("agrees with the name the capture writer gives a frame", () => {
+    // The encoder's input pattern is built from this extension, so a disagreement here is
+    // a render that captures frames and then finds none.
+    for (const needsAlpha of [false, true]) {
+      for (const motionBlur of [undefined, {}]) {
+        const format = resolveCaptureImageFormat({ needsAlpha, motionBlur });
+        expect(frameFileExtension(format)).toBe(format === "png" ? "png" : "jpg");
+      }
+    }
+  });
+});

packages/producer/src/services/render/captureImageFormat.ts

@@ -0,0 +1,21 @@
+import type { MotionBlurOptions } from "@hyperframes/engine";
+
+/**
+ * The format a render captures its frames in.
+ *
+ * Two independent reasons force PNG, and conflating them is how the encoder ends up
+ * hunting for `frame_%06d.jpg` against files written as `.png`:
+ *
+ * - the output container keeps alpha, which JPEG cannot carry;
+ * - motion blur is on, because the sub-frame samples are averaged pixel by pixel and
+ *   JPEG samples would be averaged after a lossy quantization.
+ *
+ * Every consumer of "is this render PNG" reads the resolved capture format rather than
+ * re-deriving it from one of the reasons.
+ */
+export function resolveCaptureImageFormat(input: {
+  needsAlpha: boolean;
+  motionBlur: MotionBlurOptions | undefined;
+}): "jpeg" | "png" {
+  return input.needsAlpha || input.motionBlur ? "png" : "jpeg";
+}

packages/producer/src/services/render/stages/captureStage.ts

@@ -38,6 +38,7 @@
 
 import { statfsSync } from "node:fs";
 import {
+  frameFileExtension,
   type BeforeCaptureHook,
   type CaptureOptions,
   type CapturePerfSummary,
@@ -220,7 +221,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
   } = input;
   let { probeSession } = input;
   let { workerCount } = plan;
-  const { forceScreenshot, needsAlpha } = plan;
+  const { forceScreenshot } = plan;
   let lastBrowserConsole: string[] = [];
   let captureBeyondViewport: boolean | undefined = probeSession?.options.captureBeyondViewport;
 
@@ -271,7 +272,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
       totalFrames,
       initialWorkerCount: workerCount,
       allowRetry: shouldAllowAdaptiveCaptureRetry(workerCount, job.config.workers !== undefined),
-      frameExt: needsAlpha ? "png" : "jpg",
+      frameExt: frameFileExtension(captureOptions.format),
       captureOptions,
       createBeforeCaptureHook: createRenderVideoFrameInjector,
       abortSignal,

packages/producer/src/services/render/stages/encodeStage.test.ts

@@ -36,6 +36,7 @@ mock.module("@hyperframes/engine", () => ({
   DEFAULT_CONFIG: { ffmpegEncodeTimeout: 600_000 },
   encodeFramesChunkedConcat: encodeFramesChunkedConcatMock,
   encodeFramesFromDir: encodeFramesFromDirMock,
+  frameFileExtension: (format: string | undefined) => (format === "png" ? "png" : "jpg"),
   formatFfmpegError: (code: number | null, stderr: string) => `${String(code)} ${stderr}`,
   getEncoderPreset: () => ({
     codec: "h264",
@@ -94,6 +95,7 @@ function makeInput(overrides: Partial<EncodeStageInput> = {}): EncodeStageInput
     width: 2,
     height: 2,
     needsAlpha: false,
+    captureImageFormat: "jpeg" as const,
     hasAudio: false,
     isPngSequence: false,
     isGif: false,
@@ -170,6 +172,38 @@ describe("gif encode args", () => {
   });
 });
 
+describe("frame pattern follows the capture format, not the output's alpha need", () => {
+  it("encodes frame_%06d.png for a motion-blur render whose output is opaque", async () => {
+    const { runEncodeStage } = await import("./encodeStage.js");
+    const paths = createFramesDir("png");
+    encodeFramesFromDirMock.mockClear();
+
+    await runEncodeStage(
+      makeInput({
+        framesDir: paths.framesDir,
+        outputPath: join(paths.root, "out.mp4"),
+        videoOnlyPath: join(paths.root, "video-only.mp4"),
+        // Motion blur forces PNG capture even though an mp4 output needs no alpha. Before
+        // the capture format owned this, the encoder looked for frame_%06d.jpg against
+        // files written as .png and the render found no frames at all.
+        needsAlpha: false,
+        captureImageFormat: "png",
+      }),
+    );
+
+    expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.png");
+  });
+
+  it("still encodes frame_%06d.jpg for an ordinary opaque render", async () => {
+    const { runEncodeStage } = await import("./encodeStage.js");
+    encodeFramesFromDirMock.mockClear();
+
+    await runEncodeStage(makeInput());
+
+    expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.jpg");
+  });
+});
+
 describe("runEncodeStage config plumbing", () => {
   it("throws a typed retryable error when the GIF encoder is externally interrupted", async () => {
     const { EncoderInterruptedError } = await import("../encoderInterruption.js");
@@ -379,6 +413,7 @@ describe("runEncodeStage config plumbing", () => {
         videoOnlyPath: join(paths.root, "video-only.mp4"),
         isGif: true,
         needsAlpha: true,
+        captureImageFormat: "png",
       }),
     );
 

packages/producer/src/services/render/stages/encodeStage.ts

@@ -52,6 +52,7 @@ import {
 } from "./gifEncodeArgs.js";
 import { updateJobStatus } from "../shared.js";
 import { encoderFailureError } from "../encoderInterruption.js";
+import { frameFileExtension } from "@hyperframes/engine";
 
 export interface EncodeStageInput {
   job: RenderJob;
@@ -67,6 +68,12 @@ export interface EncodeStageInput {
   height: number;
   /** True when the output format requires an alpha channel; selects frame extension. */
   needsAlpha: boolean;
+  /**
+   * Format the frames on disk were captured in. Drives the encoder's input pattern, which
+   * must match what `writeCapturedFrame` named them. Not derivable from `needsAlpha`:
+   * motion blur also forces PNG capture on an opaque output.
+   */
+  captureImageFormat: "jpeg" | "png";
   /** True iff the composition has audio. Drives the sidecar copy. */
   hasAudio: boolean;
   /**
@@ -273,7 +280,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
     if (hasAudio) {
       log.warn("[Render] GIF output does not support audio; audio tracks will be ignored.");
     }
-    const frameExt = needsAlpha ? "png" : "jpg";
+    const frameExt = frameFileExtension(input.captureImageFormat);
     const framePattern = `frame_%06d.${frameExt}`;
     const loop = resolveGifLoop(job.config.gifLoop);
     const encodeResult = await encodeGifFromDir(framesDir, framePattern, outputPath, {
@@ -306,7 +313,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
       ? { ...engineCfg, ffmpegEncodeTimeout: scaledEncodeTimeout }
       : engineCfg;
 
-  const frameExt = needsAlpha ? "png" : "jpg";
+  const frameExt = frameFileExtension(input.captureImageFormat);
   const framePattern = `frame_%06d.${frameExt}`;
   const encoderOpts = {
     fps: job.config.fps,

packages/producer/src/services/render/stages/probeStage.ts

@@ -69,6 +69,7 @@ import {
 import type { RenderJob } from "../../renderOrchestrator.js";
 import { isActionableProbeFailure } from "./probeFailures.js";
 import { preflightCompositionAssetMediaTypes } from "../../assetMediaType.js";
+import { resolveCaptureImageFormat } from "../captureImageFormat.js";
 
 export interface ProbeStageInput {
   projectDir: string;
@@ -328,14 +329,21 @@ export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageR
     });
     assertNotAborted();
 
+    // Motion blur averages the sub-frame samples pixel by pixel, so the frames have to be
+    // captured losslessly even when the output container wants no alpha. This is the one
+    // place a render picks its capture format; the encoder reads the result rather than
+    // re-deriving it, so the two cannot disagree about what is on disk.
+    const motionBlur = job.config.motionBlur;
+    const captureFormat = resolveCaptureImageFormat({ needsAlpha, motionBlur });
     const captureOpts: CaptureOptions = {
       width,
       height,
       fps: job.config.fps,
-      format: needsAlpha ? "png" : "jpeg",
-      quality: needsAlpha ? undefined : 80,
+      format: captureFormat,
+      quality: captureFormat === "png" ? undefined : 80,
       variables: job.config.variables,
       deviceScaleFactor,
+      motionBlur,
     };
 
     const PROBE_MAX_ATTEMPTS = 2;

packages/producer/src/services/renderOrchestrator.ts

@@ -93,6 +93,7 @@ import {
   getDrawElementVerificationDetails,
   augmentProtocolTimeoutError,
   augmentPageNavigationTimeoutError,
+  type MotionBlurOptions,
 } from "@hyperframes/engine";
 import { join, dirname, resolve } from "path";
 import { totalmem } from "node:os";
@@ -185,6 +186,7 @@ import { runCaptureHdrStage } from "./render/stages/captureHdrStage.js";
 import { runEncodeStage } from "./render/stages/encodeStage.js";
 import { runAssembleStage } from "./render/stages/assembleStage.js";
 import { shouldUseLayeredComposite } from "./hdrCompositor.js";
+import { resolveCaptureImageFormat } from "./render/captureImageFormat.js";
 
 function sampleDirectoryBytes(dir: string): number {
   let total = 0;
@@ -346,6 +348,12 @@ export interface RenderConfig {
    * A non-integer value throws at the start of `executeRenderJob`.
    */
   hlsSegmentSeconds?: number;
+  /**
+   * Opt into sub-frame multi-sample motion blur. Absent means off and the capture path is
+   * unchanged. Forces PNG frame capture, because the samples are averaged pixel by pixel
+   * and JPEG samples would be averaged after a lossy quantization.
+   */
+  motionBlur?: MotionBlurOptions;
   workers?: number;
   useGpu?: boolean;
   debug?: boolean;
@@ -2925,14 +2933,19 @@ async function executeRenderPipeline(input: {
     });
     const videoCaptureBeyondViewport = resolveVideoCaptureBeyondViewport(composition.videos.length);
 
+    const captureImageFormat = resolveCaptureImageFormat({
+      needsAlpha,
+      motionBlur: job.config.motionBlur,
+    });
     const captureOptions: CaptureOptions = {
       width,
       height,
       fps: job.config.fps,
-      format: needsAlpha ? "png" : "jpeg",
-      quality: needsAlpha ? undefined : job.config.quality === "draft" ? 80 : 95,
+      format: captureImageFormat,
+      quality: captureImageFormat === "png" ? undefined : job.config.quality === "draft" ? 80 : 95,
       variables: job.config.variables,
       deviceScaleFactor,
+      motionBlur: job.config.motionBlur,
       ...(videoCaptureBeyondViewport !== undefined
         ? { captureBeyondViewport: videoCaptureBeyondViewport }
         : {}),
@@ -4193,6 +4206,7 @@ async function executeRenderPipeline(input: {
               width,
               height,
               needsAlpha,
+              captureImageFormat: captureOptions.format ?? "jpeg",
               hasAudio,
               audioOutputPath,
               isPngSequence,