fix(engine): auto-normalize VFR video inputs to CFR before frame extraction (#360)

* fix(engine): auto-normalize VFR video inputs to CFR before frame extraction Screen recordings (macOS ScreenCaptureKit, QuickTime, phone videos) are commonly variable-frame-rate. When such inputs hit the extractor's `-ss <start> -i <video> -t <dur> -vf fps=N` pipeline, the fps filter can emit fewer frames than requested — for a 4-second 30fps segment starting mid-file, the output was ~90 frames instead of 120. `FrameLookupTable.getFrameAtTime` returns null for out-of-range indices, so the compositor held the last valid frame and the user perceived the video as freezing. This matches the bug report from an X community post where a user said "all of them freezes" on their screen recording scenes. The engine already detects VFR via `metadata.isVFR` in ffprobe.ts but never acted on it — the compiler only logged a warning. This change mirrors the existing SDR→HDR normalization pattern: when a source is detected as VFR, re-encode only the used segment with `-fps_mode cfr -r <fps> -preset fast -crf 18` before extraction. Scoping the re-encode to `[mediaStart, mediaStart+duration]` means a 30-second clip cut from a 60-minute screen recording pays ~1s of transcode cost, not 18s. Benchmarked locally: Baseline (current): 32-39% duplicate frames, 25% frame-count shortfall on mid-file segments. Tier 1 (flag changes only): ~same — fps filter issue is not flag-fixable. Tier 2 (CFR preflight): 1.7-6% duplicate frames, correct frame count in every scenario tested. The compiler warning that previously told users to manually re-encode is downgraded to `console.info` since the engine now handles it. — Rames Jusso * refactor(engine): clean up VFR normalization loop after review - Drop the `vfrNormDirCreated` flag; `mkdirSync({recursive:true})` is idempotent and cheap. - Don't re-wrap the `VFR→CFR conversion failed` prefix — `convertVfrToCfr` already throws a message with that label; adding it again in the catch produced "VFR→CFR conversion failed: VFR→CFR conversion failed (exit 1)". - Shorten the Phase 2b header comment; the function docstring above `convertVfrToCfr` already explains the failure modes and rationale. - Note which frame windows the VFR fixture's select filter drops so the magic numbers are scannable. No behavior change; 311/311 engine tests still pass. — Rames Jusso * test(engine): add VFR regression unit tests Adds a describe block that synthesizes a VFR fixture via ffmpeg and asserts the extractor produces the expected frame count (no shortfall) and no long runs of duplicate frames — the user-visible "frozen screen recording" symptom. Covers both a mid-file segment and the full-file case. Guarded with describe.skipIf(!HAS_FFMPEG) because the CI Test job on ubuntu-24.04 and the Windows test-windows job don't install ffmpeg. The producer-level regression test in packages/producer/tests/vfr-screen-recording/ runs inside Dockerfile.test (which has ffmpeg) and is the primary CI signal for this bug; these unit tests are supplementary coverage for local and any ffmpeg-equipped CI environment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(producer): add vfr-screen-recording regression test End-to-end CI regression coverage for PR #360 via the existing regression-harness: renders a 3s composition containing a real macOS ScreenCaptureKit clip (r_frame_rate=120, avg≈36fps) seeked to mediaStart=1, then PSNR-compares against a committed output.mp4. Fixture src/clip.mp4 (108 KB) is a 5-second excerpt downscaled to 480×332 with -fps_mode passthrough to preserve the VFR timestamps. Content is the public hyperframes OSS repo root page — see NOTICE.md for provenance. With the fix applied, all 100 PSNR checkpoints pass. With the fix reverted, 66 of 100 fail (PSNR drops from ~43 dB to ~20 dB in the duplicate-frame windows). Tagged "regression,video,vfr" so it runs in the fast shard of .github/workflows/regression.yml automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(producer): regenerate vfr-screen-recording baseline in Docker The committed golden output.mp4 was initially rendered on the host machine; CI runs the renderer inside Dockerfile.test with a different Chrome + ffmpeg build, producing pixel-level drift that failed PSNR at 54/100 checkpoints (~20 dB vs 41 dB in the VFR sparse-content windows). Both renders are valid — the VFR source has inherent sampling ambiguity in static segments, and different Chrome/ffmpeg builds make different valid choices. Regenerated the baseline via `bun run docker:test:update vfr-screen-recording` so it matches the Docker environment CI actually uses. Matches the flow the existing sub-composition-video, hdr-pq, etc. baselines were captured with. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: document that producer test baselines must be captured in Docker Hit this 2026-04-21 with the vfr-screen-recording regression test: host-generated output.mp4 baseline tripped 54/100 PSNR checkpoints in CI because Chrome + ffmpeg drift between the host and Dockerfile.test. Document the `bun run --cwd packages/producer docker:test:update <name>` flow so future contributors don't repeat the mistake. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-09-08 10:46:06 +00:00 · 2026-04-21 12:14:23 -07:00
parent b98093aa1c
commit ffc06827c4
10 changed files with 461 additions and 5 deletions
@@ -296,6 +296,62 @@ async function convertSdrToHdr(
  }
 }

+/**
+ * Re-encode a VFR (variable frame rate) video segment to CFR so the downstream
+ * fps filter can extract frames reliably. Screen recordings, phone videos, and
+ * some webcams emit irregular timestamps that cause two failure modes:
+ *   1. Output has fewer frames than expected (e.g. -ss 3 -t 4 produces 90
+ *      frames instead of 120 @ 30fps). FrameLookupTable.getFrameAtTime then
+ *      returns null for late timestamps and the caller freezes on the last
+ *      valid frame.
+ *   2. Large duplicate-frame runs where source PTS don't land on target
+ *      timestamps.
+ *
+ * Only the [startTime, startTime+duration] window is re-encoded, so long
+ * recordings aren't fully transcoded when only a short clip is used.
+ */
+async function convertVfrToCfr(
+  inputPath: string,
+  outputPath: string,
+  targetFps: number,
+  startTime: number,
+  duration: number,
+  signal?: AbortSignal,
+  config?: Partial<Pick<EngineConfig, "ffmpegProcessTimeout">>,
+): Promise<void> {
+  const timeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout;
+
+  const args = [
+    "-ss",
+    String(startTime),
+    "-i",
+    inputPath,
+    "-t",
+    String(duration),
+    "-fps_mode",
+    "cfr",
+    "-r",
+    String(targetFps),
+    "-c:v",
+    "libx264",
+    "-preset",
+    "fast",
+    "-crf",
+    "18",
+    "-c:a",
+    "copy",
+    "-y",
+    outputPath,
+  ];
+
+  const result = await runFfmpeg(args, { signal, timeout });
+  if (!result.success) {
+    throw new Error(
+      `VFR→CFR conversion failed (exit ${result.exitCode}): ${result.stderr.slice(-300)}`,
+    );
+  }
+}
+
 export async function extractAllVideoFrames(
  videos: VideoElement[],
  baseDir: string,
@@ -371,6 +427,47 @@ export async function extractAllVideoFrames(
    }
  }

+  // Phase 2b: Re-encode VFR inputs to CFR so the fps filter in Phase 3 produces
+  // the expected frame count. Only the used segment is transcoded.
+  const vfrNormDir = join(options.outputDir, "_vfr_normalized");
+  for (let i = 0; i < resolvedVideos.length; i++) {
+    if (signal?.aborted) break;
+    const entry = resolvedVideos[i];
+    if (!entry) continue;
+    const metadata = await extractVideoMetadata(entry.videoPath);
+    if (!metadata.isVFR) continue;
+
+    let segDuration = entry.video.end - entry.video.start;
+    if (!Number.isFinite(segDuration) || segDuration <= 0) {
+      const sourceRemaining = metadata.durationSeconds - entry.video.mediaStart;
+      segDuration = sourceRemaining > 0 ? sourceRemaining : metadata.durationSeconds;
+    }
+
+    mkdirSync(vfrNormDir, { recursive: true });
+    const normalizedPath = join(vfrNormDir, `${entry.video.id}_cfr.mp4`);
+    try {
+      await convertVfrToCfr(
+        entry.videoPath,
+        normalizedPath,
+        options.fps,
+        entry.video.mediaStart,
+        segDuration,
+        signal,
+        config,
+      );
+      entry.videoPath = normalizedPath;
+      // Segment-scoped re-encode starts the new file at t=0, so downstream
+      // extraction must seek from 0, not the original mediaStart. Shallow-copy
+      // to avoid mutating the caller's VideoElement.
+      entry.video = { ...entry.video, mediaStart: 0 };
+    } catch (err) {
+      errors.push({
+        videoId: entry.video.id,
+        error: err instanceof Error ? err.message : String(err),
+      });
+    }
+  }
+
  // Phase 3: Extract frames (parallel)
  const results = await Promise.all(
    resolvedVideos.map(async ({ video, videoPath }) => {