fix: skip metadata waits for injected video frames (#575)

## Problem Closes #574. On Windows with cached headless-shell Chrome, a composition that reuses the same video file in three timeline clips can fail before frame capture starts: ```html <video id="video1" src="1.mp4" data-start="0" muted data-duration="4" data-track-index="0" data-media-start="0"></video> <video id="video2" src="1.mp4" data-start="4" muted data-duration="4" data-track-index="0" data-media-start="4"></video> <video id="video3" src="1.mp4" data-start="8" muted data-duration="4" data-track-index="0" data-media-start="8"></video> ``` The reported render reaches video frame extraction, then dies at frame-capture initialization with: ```text [FrameCapture] video metadata not ready after 45000ms. Video elements must load metadata before capture starts. ``` The important detail is that by this stage HyperFrames has already extracted video pixels through FFmpeg. Native Chromium video metadata is only being waited on for DOM layout stability, not because Chromium is the source of rendered pixels. ## Root Cause The render pipeline has two separate media responsibilities: - FFmpeg extracts video frames and audio from declared media. - Chromium owns DOM layout and capture, while injected FFmpeg frames supply the video pixels before each captured frame. Before this PR, every capture session still waited for every DOM `<video>` to reach `readyState >= 1` unless the element was a native HDR exception. That made native browser media metadata a hard render prerequisite even when the browser would not decode or provide the final video pixels. That is why the issue fails at `25% Starting frame capture`: FFmpeg extraction has already succeeded, but capture initialization blocks on repeated native `<video src="1.mp4">` metadata loading in cached Windows headless-shell Chrome. There was a second constraint: the readiness wait also prevents first-frame layout bugs. If a skipped `<video>` has no native metadata, Chromium can use the default `300x150` intrinsic video size, which breaks layouts such as `width: 100%; height: auto` before the first injected frame. The fix therefore must not simply skip all video readiness waits; it must provide dimensions for any skipped videos. ## What This Fixes - Treats videos with successfully extracted FFmpeg frames and usable dimensions as out-of-band rendered video sources. - Skips native browser metadata readiness waits for those extracted videos because Chromium is not responsible for their pixels. - Passes FFmpeg-probed dimensions into capture as `videoMetadataHints`. - Applies those hints before the readiness wait in both screenshot and BeginFrame initialization paths. - Sets missing `width` / `height` attributes and an explicit `aspect-ratio` only when the element does not already provide one, preserving author styles where present. - Keeps native HDR video IDs in the skip list, preserving the existing HEVC/HDR behavior where Chrome may not decode the source but FFmpeg/native HDR compositing can still render it. - Uses one `buildCaptureOptions()` helper so calibration, HDR DOM capture, streaming capture, parallel capture, and sequential capture receive the same skip IDs and metadata hints. - Adds tests for the skip-list and metadata-hint contract. - Adds a Windows CI regression that reproduces the issue shape after the canary render warms the cached-browser path. ## Reviewer Map Primary files: - `packages/producer/src/services/renderOrchestrator.ts` - `collectVideoReadinessSkipIds()` includes native HDR IDs plus extracted videos that have finite positive FFmpeg dimensions. - `collectVideoMetadataHints()` converts extracted FFmpeg metadata into capture hints. - `buildCaptureOptions()` threads `skipReadinessVideoIds` and `videoMetadataHints` into every capture path. - `packages/engine/src/services/frameCapture.ts` - `applyVideoMetadataHints()` runs in the page before video readiness polling. - Both screenshot and BeginFrame initialization call it before checking non-skipped videos for `readyState >= 1`. - `packages/engine/src/types.ts` - Adds `CaptureVideoMetadataHint` and documents that readiness skips should be paired with metadata hints when layout may depend on intrinsic dimensions. - `packages/producer/src/services/renderOrchestrator.test.ts` - Covers that extracted videos with dimensions are skipped, invalid dimensions are not, native HDR IDs are preserved, and hints are stable/sorted. - `.github/workflows/windows-render.yml` - Adds the issue #574 Windows regression with the exact three-clip markup and a generated deterministic `1.mp4`. ## Why This Is Safe The skip is intentionally gated: - A standard video is skipped only after `extractAllVideoFrames()` succeeded for that video and returned usable dimensions. - Videos with invalid dimensions are not skipped, so the old browser readiness guard still applies. - DOM videos are still present for layout and element bounds; only the native metadata wait is skipped for sources whose pixels come from FFmpeg injection. - Metadata hints are applied conservatively: existing `width`, `height`, and explicit `aspect-ratio` are not overwritten. - Non-extracted videos, images, fonts, page readiness, and `window.__hf` readiness keep the existing waits. - The fix is not limited to the sequential path from the issue; it is threaded through calibration, HDR DOM capture, streaming encode, parallel capture, and sequential capture. A first local revision skipped readiness too broadly and caused `overlay-montage-prod` first-frame layout shrinkage. The current version fixes that by pairing skips with FFmpeg metadata hints; `overlay-montage-prod` now passes and is listed in verification below. ## Verification ### Root-Cause Reproduction Before Fix The reporter did not attach the actual `1.mp4`, so the regression uses the exact issue markup and a deterministic generated 12s H.264 file named `1.mp4`. I reproduced the failure in GitHub Actions by running this branch's new Windows workflow against unpatched `main`: ```bash gh workflow run windows-render.yml --repo heygen-com/hyperframes --ref fix/reused-video-metadata -f ref=main ``` That means the workflow contains the new issue #574 regression, but the code under test is `main` without this fix. Baseline failure: - Run: https://github.com/heygen-com/hyperframes/actions/runs/25174603730 - Failed job: https://github.com/heygen-com/hyperframes/actions/runs/25174603730/job/73803179086 - Checkout proof: `ref: main`, `origin/main`, commit `8662598a3ac64018a2999d189ffb369e6d46b53a`. - Failure proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, then `25% Starting frame capture` -> `[FrameCapture] video metadata not ready after 15000ms`. This is the same failure class as the issue, on Windows, in cached-browser mode, before the fix. ### Fixed Windows Regression The same regression passes on this PR branch: - Run: https://github.com/heygen-com/hyperframes/actions/runs/25175048215 - Passing job: https://github.com/heygen-com/hyperframes/actions/runs/25175048215/job/73804781017 - Checkout proof: PR merge contains `79d6b41c9f2ba137cbfb9301678e0815b16c4f5a` merged into `8662598a3ac64018a2999d189ffb369e6d46b53a`. - Passing proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, `25% Starting frame capture`, captures `360/360` frames, renders `issue-574.mp4`, and `ffprobe` verifies `1920x1080 @ 30/1, 12s`. ### Local Checks - `bun run build:hyperframes-runtime` - `bunx vitest run packages/producer/src/services/renderOrchestrator.test.ts` - `bun run --filter @hyperframes/producer typecheck` - `bun run --filter @hyperframes/engine typecheck` - `bunx oxlint packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts` - `bunx oxfmt --check .github/workflows/windows-render.yml packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts` - `git diff --check` - Lefthook pre-commit: lint, format, typecheck where applicable - Lefthook commit-msg: commitlint ### Local Render Checks - Created `/tmp/hf-issue-574-repro` with the issue shape: three clips using the same `1.mp4`, `data-media-start=0/4/8`, 12s total. - `PRODUCER_PLAYER_READY_TIMEOUT_MS=5000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-repro --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-h264-fixed-v2.mp4` -> completed. - Created `/tmp/hf-issue-574-prores` with the same three-clip shape using one FFmpeg-readable ProRes `.mov`, which exercises the browser-metadata failure class because Chromium should not be needed to decode the source. - `PRODUCER_PLAYER_READY_TIMEOUT_MS=3000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-prores --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-prores-fixed-v2.mp4` -> completed. - `bun run --filter @hyperframes/producer test --sequential --keep-temp overlay-montage-prod` -> passed; this guards against skipped metadata shrinking `height:auto` video layout before the first injected frame. - `ffmpeg -v error -i /tmp/hf-issue-574-prores-fixed-v2.mp4 -f null -` - `ffmpeg -v error -i /tmp/hf-issue-574-h264-fixed-v2.mp4 -f null -` - `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-issue-574-h264-fixed-v2.mp4` -> H.264, 320x180, 30fps, 12.0s. ### Current PR Checks - Windows render verification: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215. - Windows tests: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215. - Main CI build/lint/typecheck/test/smoke jobs: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048175. - Regression shards observed passing include HDR, render-compat, styles A-G, and `overlay-montage-prod`. At the time this body was updated, the `fast` regression shard was still in progress in run https://github.com/heygen-com/hyperframes/actions/runs/25174515546. ### Browser Verification - Used `agent-browser` to open `file:///tmp/hf-issue-574-h264-fixed-v2.mp4` and verify the rendered output displays in Chromium. - Screenshot: `.debug/issue-574/h264-output-page.png` - Agent-browser recording: `.debug/issue-574/h264-output-playback.webm` ## Notes / Caveats - The reporter's exact `1.mp4` was not attached to #574. The committed Windows regression uses a generated deterministic H.264 file with the same filename and exact markup from the issue. - The exact H.264 issue shape did not reproduce the timeout on this macOS/system-Chrome machine before the fix; it rendered successfully locally. The GitHub Actions baseline above reproduces it on Windows/cache without the fix. - The Windows fixture intentionally runs after the existing canary render so the browser path is `Browser: cache`, matching the reporter's environment. - The generated fixture emits sparse-keyframe warnings. Those warnings are expected and are not the failure being fixed; the baseline failure occurs before any frame capture because native browser video metadata never becomes ready. - Browser proof artifacts are local-only under `.debug/issue-574/` and intentionally not committed.
2026-09-03 04:38:33 +00:00 · 2026-04-30 18:50:57 +02:00
parent 8662598a3a
commit 6a59ef6106
6 changed files with 276 additions and 26 deletions
@@ -36,6 +36,7 @@ export type {
  HfMediaElement,
  HfTransitionMeta,
  CaptureOptions,
+  CaptureVideoMetadataHint,
  CaptureResult,
  CaptureBufferResult,
  CapturePerfSummary,
@@ -29,6 +29,7 @@ import {
 import { DEFAULT_CONFIG, type EngineConfig } from "../config.js";
 import type {
  CaptureOptions,
+  CaptureVideoMetadataHint,
  CaptureResult,
  CaptureBufferResult,
  CapturePerfSummary,
@@ -221,6 +222,44 @@ async function pollPageExpression(
  return Boolean(await page.evaluate(expression));
 }

+async function applyVideoMetadataHints(
+  page: Page,
+  hints: readonly CaptureVideoMetadataHint[] | undefined,
+): Promise<void> {
+  if (!hints || hints.length === 0) return;
+
+  await page.evaluate(
+    (metadataHints: CaptureVideoMetadataHint[]) => {
+      for (const hint of metadataHints) {
+        if (
+          !hint.id ||
+          !Number.isFinite(hint.width) ||
+          !Number.isFinite(hint.height) ||
+          hint.width <= 0 ||
+          hint.height <= 0
+        ) {
+          continue;
+        }
+
+        const video = document.getElementById(hint.id) as HTMLVideoElement | null;
+        if (!video) continue;
+
+        if (!video.hasAttribute("width")) video.setAttribute("width", String(hint.width));
+        if (!video.hasAttribute("height")) video.setAttribute("height", String(hint.height));
+
+        const computed = window.getComputedStyle(video);
+        if (
+          !video.style.aspectRatio &&
+          (!computed.aspectRatio || computed.aspectRatio === "auto")
+        ) {
+          video.style.aspectRatio = `${hint.width} / ${hint.height}`;
+        }
+      }
+    },
+    [...hints],
+  );
+}
+
 export async function initializeSession(session: CaptureSession): Promise<void> {
  const { page, serverUrl } = session;

@@ -290,11 +329,14 @@ export async function initializeSession(session: CaptureSession): Promise<void>
      );
    }

+    await applyVideoMetadataHints(page, session.options.videoMetadataHints);
+
    // Wait for all video elements to have loaded metadata (dimensions + duration)
    // Without this, frame 0 captures videos at their 300x150 default size.
    // skipReadinessVideoIds excludes natively-extracted videos (e.g. HDR HEVC
-    // sources) whose frames come from ffmpeg out-of-band — Chromium may not be
-    // able to decode them at all (e.g. HEVC on Linux headless-shell).
+    // sources) whose frames come from ffmpeg out-of-band. videoMetadataHints
+    // supply intrinsic dimensions for skipped videos whose layout depends on
+    // aspect ratio, while Chromium may still fail to decode/load metadata.
    const skipIdsLiteral = JSON.stringify(session.options.skipReadinessVideoIds ?? []);
    const videosReady = await pollPageExpression(
      page,
@@ -382,9 +424,12 @@ export async function initializeSession(session: CaptureSession): Promise<void>
    );
  }

+  await applyVideoMetadataHints(page, session.options.videoMetadataHints);
+
  // Wait for all video elements to have loaded metadata (dimensions + duration).
  // Without this, frame 0 captures videos at their 300x150 default size.
-  // See screenshot-mode comment above for why skipReadinessVideoIds exists.
+  // See screenshot-mode comment above for why skipReadinessVideoIds and
+  // videoMetadataHints are paired.
  const beginframeSkipIdsLiteral = JSON.stringify(session.options.skipReadinessVideoIds ?? []);
  const videoDeadline =
    Date.now() + (session.config?.playerReadyTimeout ?? DEFAULT_CONFIG.playerReadyTimeout);
@@ -85,20 +85,31 @@ export interface CaptureOptions {
  format?: "jpeg" | "png";
  quality?: number;
  deviceScaleFactor?: number;
+  /**
+   * FFmpeg-probed intrinsic dimensions for videos whose frames are injected
+   * out-of-band. Applied before the readiness wait so layout that depends on
+   * video aspect ratio (e.g. `height:auto`) stays stable even if Chromium never
+   * loads native metadata.
+   */
+  videoMetadataHints?: readonly CaptureVideoMetadataHint[];
  /**
   * Video element IDs to exclude from the in-page readiness check that waits
   * for `video.readyState >= 1` before capture starts.
   *
-   * Use for videos whose frames are supplied out-of-band (e.g. native HDR
-   * frame extraction via ffmpeg). The DOM `<video>` element is then only
-   * needed for layout (`getBoundingClientRect` / `offsetWidth`), which works
-   * at `readyState=0`. Without this, codecs that headless Chromium can't
-   * decode (HEVC on Linux `headless-shell`) cause a fatal timeout even
-   * though we never asked the browser to play the video.
+   * Use for videos whose frames are supplied out-of-band, including standard
+   * FFmpeg frame injection and native HDR extraction. Pair with
+   * `videoMetadataHints` for any skipped video whose CSS layout may depend on
+   * intrinsic media dimensions.
   */
  skipReadinessVideoIds?: readonly string[];
 }

+export interface CaptureVideoMetadataHint {
+  id: string;
+  width: number;
+  height: number;
+}
+
 export interface CaptureResult {
  frameIndex: number;
  time: number;
@@ -8,6 +8,8 @@ import type { CompiledComposition } from "./htmlCompiler.js";
 import {
  applyRenderModeHints,
  buildMissingFrameRetryBatches,
+  collectVideoMetadataHints,
+  collectVideoReadinessSkipIds,
  createCaptureCalibrationConfig,
  estimateMeasuredCaptureCostMultiplier,
  estimateCaptureCostMultiplier,
@@ -260,6 +262,35 @@ describe("applyRenderModeHints", () => {
  });
 });

+describe("collectVideoReadinessSkipIds", () => {
+  it("skips native metadata waits for every injected video with dimensions", () => {
+    expect(
+      collectVideoReadinessSkipIds(new Set(["hdr-video"]), [
+        { videoId: "video1", metadata: { width: 1920, height: 1080 } },
+        { videoId: "video2", metadata: { width: 1920, height: 1080 } },
+        { videoId: "video3", metadata: { width: 1920, height: 1080 } },
+        { videoId: "hdr-video", metadata: { width: 1920, height: 1080 } },
+        { videoId: "bad-metadata", metadata: { width: 0, height: 0 } },
+      ]),
+    ).toEqual(["hdr-video", "video1", "video2", "video3"]);
+  });
+});
+
+describe("collectVideoMetadataHints", () => {
+  it("passes extracted video dimensions to capture sessions", () => {
+    expect(
+      collectVideoMetadataHints([
+        { videoId: "video2", metadata: { width: 1080, height: 1920, durationSeconds: 4 } },
+        { videoId: "video1", metadata: { width: 1920, height: 1080, durationSeconds: 12 } },
+        { videoId: "bad-metadata", metadata: { width: 0, height: 1080, durationSeconds: 1 } },
+      ]),
+    ).toEqual([
+      { id: "video1", width: 1920, height: 1080 },
+      { id: "video2", width: 1080, height: 1920 },
+    ]);
+  });
+});
+
 describe("resolveRenderWorkerCount", () => {
  const cfg = { ...createConfig(), coresPerWorker: 100 };
  const audio = {
@@ -46,6 +46,7 @@ import {
  getCompositionDuration,
  prepareCaptureSessionForReuse,
  type CaptureOptions,
+  type CaptureVideoMetadataHint,
  type CaptureSession,
  type BeforeCaptureHook,
  createVideoFrameInjector,
@@ -678,6 +679,50 @@ export function applyRenderModeHints(
  });
 }

+export function collectVideoReadinessSkipIds(
+  nativeHdrVideoIds: ReadonlySet<string>,
+  extractedVideos: readonly ExtractedVideoReadinessInput[],
+): string[] {
+  return Array.from(
+    new Set([
+      ...nativeHdrVideoIds,
+      ...extractedVideos
+        .filter((video) => hasUsableVideoDimensions(video.metadata))
+        .map((video) => video.videoId),
+    ]),
+  ).sort();
+}
+
+interface ExtractedVideoReadinessInput {
+  videoId: string;
+  metadata: {
+    width: number;
+    height: number;
+  };
+}
+
+function hasUsableVideoDimensions(metadata: ExtractedVideoReadinessInput["metadata"]) {
+  return (
+    Number.isFinite(metadata.width) &&
+    Number.isFinite(metadata.height) &&
+    metadata.width > 0 &&
+    metadata.height > 0
+  );
+}
+
+export function collectVideoMetadataHints(
+  extractedVideos: readonly ExtractedVideoReadinessInput[],
+): CaptureVideoMetadataHint[] {
+  return extractedVideos
+    .filter((video) => hasUsableVideoDimensions(video.metadata))
+    .map((video) => ({
+      id: video.videoId,
+      width: video.metadata.width,
+      height: video.metadata.height,
+    }))
+    .sort((a, b) => a.id.localeCompare(b.id));
+}
+
 export function resolveRenderWorkerCount(
  totalFrames: number,
  requestedWorkers: number | undefined,
@@ -2123,6 +2168,8 @@ export async function executeRenderJob(
    let frameLookup: FrameLookupTable | null = null;
    const compiledDir = join(workDir, "compiled");
    let extractionResult: Awaited<ReturnType<typeof extractAllVideoFrames>> | null = null;
+    let videoReadinessSkipIds: string[] = [];
+    let videoMetadataHints: CaptureVideoMetadataHint[] = [];

    // Probe ORIGINAL color spaces before extraction (which may convert SDR→HDR).
    // This is needed to identify which videos are natively HDR vs converted-SDR
@@ -2196,6 +2243,11 @@ export async function executeRenderJob(
      if (extractionResult.extracted.length > 0) {
        frameLookup = createFrameLookupTable(composition.videos, extractionResult.extracted);
      }
+      videoReadinessSkipIds = collectVideoReadinessSkipIds(
+        nativeHdrVideoIds,
+        extractionResult.extracted,
+      );
+      videoMetadataHints = collectVideoMetadataHints(extractionResult.extracted);
      perfStages.videoExtractMs = Date.now() - stage2Start;

      // Auto-detect audio from video files via ffprobe metadata
@@ -2338,17 +2390,18 @@ export async function executeRenderJob(
      quality: needsAlpha ? undefined : job.config.quality === "draft" ? 80 : 95,
    };

-    // Native HDR videos (e.g. HEVC) may be undecodable by Chrome on the current
-    // platform — Linux headless-shell ships without HEVC support. Their pixels
-    // come from out-of-band ffmpeg extraction, so the DOM `<video>` element is
-    // only kept around for layout. Skip the per-page readiness wait for these
-    // IDs in every capture session we open during HDR rendering; otherwise the
-    // render hangs 45s and throws "video metadata not ready" even though we
-    // never asked the browser to decode the video. Encapsulating the spread
-    // here avoids drifting copies across the five capture call sites below.
-    const buildHdrCaptureOptions = (): CaptureOptions => ({
+    // Capture sessions do not need native browser metadata for videos whose
+    // pixels come from out-of-band FFmpeg frame extraction. Waiting on those
+    // `<video>` elements lets browser decode/cache quirks block renders even
+    // though the browser never supplies their pixels. We still pass FFmpeg
+    // dimensions as metadata hints so CSS layouts that depend on intrinsic
+    // aspect ratio stay stable before the first injected frame. Native HDR
+    // videos are included for the same reason: Chrome may not decode them at
+    // all, while the renderer composites their extracted frames separately.
+    const buildCaptureOptions = (): CaptureOptions => ({
      ...captureOptions,
-      skipReadinessVideoIds: Array.from(nativeHdrVideoIds),
+      videoMetadataHints,
+      skipReadinessVideoIds: videoReadinessSkipIds,
    });

    let captureCalibration:
@@ -2368,7 +2421,7 @@ export async function executeRenderJob(
        calibrationSession = await createCaptureSession(
          fileServer.url,
          calibrationDir,
-          buildHdrCaptureOptions(),
+          buildCaptureOptions(),
          videoInjector,
          calibrationCfg,
        );
@@ -2562,7 +2615,7 @@ export async function executeRenderJob(
      const domSession = await createCaptureSession(
        fileServer.url,
        framesDir,
-        buildHdrCaptureOptions(),
+        buildCaptureOptions(),
        createVideoFrameInjector(frameLookup),
        cfg,
      );
@@ -3307,7 +3360,7 @@ export async function executeRenderJob(
              fileServer.url,
              workDir,
              tasks,
-              buildHdrCaptureOptions(),
+              buildCaptureOptions(),
              () => createVideoFrameInjector(frameLookup),
              abortSignal,
              (progress) => {
@@ -3346,7 +3399,7 @@ export async function executeRenderJob(
              (await createCaptureSession(
                fileServer.url,
                framesDir,
-                buildHdrCaptureOptions(),
+                buildCaptureOptions(),
                videoInjector,
                cfg,
              ));
@@ -3411,7 +3464,7 @@ export async function executeRenderJob(
              initialWorkerCount: workerCount,
              allowRetry: job.config.workers === undefined,
              frameExt: needsAlpha ? "png" : "jpg",
-              captureOptions: buildHdrCaptureOptions(),
+              captureOptions: buildCaptureOptions(),
              createBeforeCaptureHook: () => createVideoFrameInjector(frameLookup),
              abortSignal,
              onProgress: (progress) => {
@@ -3454,7 +3507,7 @@ export async function executeRenderJob(
              (await createCaptureSession(
                fileServer.url,
                framesDir,
-                buildHdrCaptureOptions(),
+                buildCaptureOptions(),
                videoInjector,
                cfg,
              ));