perf(engine): content-addressed extraction cache for video frames (#446)

## What

Adds a content-addressed cache for extracted video frames, keyed on the tuple `(path, mtime, size, mediaStart, duration, fps, format)`. Repeat renders of the same composition (studio edit → re-render, preview → final) skip the ffmpeg extraction entirely.

## Why

Video frame extraction is the dominant non-capture phase for video-heavy compositions. Studio iteration workflows extract the same frames over and over — each render burns ffmpeg time that adds no value.

Validated on `/tmp/hf-fixtures/cfr-sdr-cache`:
```
Cold (miss): extractMs=69,  videoExtractMs=70,  totalElapsedMs=2052
Warm (hit):  extractMs=1,   videoExtractMs=2,   totalElapsedMs=1964
cacheHits: 0→1, cacheMisses: 1→0
```
The fixture is tiny (3s CFR SDR @ 30fps), so the wall-clock delta is small; the extraction-time delta (69→1ms, 98%) scales linearly with source length. For heavy-iteration workflows (a user rendering the same composition while tuning encoding params), extraction time goes to zero on every repeat render.

Depends on #444 (instrumentation surface) and #445 (segment-scope HDR preflight — otherwise cache keys would be unstable across renders on mixed-HDR compositions).

## How

- New `packages/engine/src/services/extractionCache.ts`:
  - SHA-256 key over a stable JSON encoding of `(path, mtime_ms, size, mediaStart, duration, fps, format)`. Infinity duration is normalized to `-1` so unresolved natural-duration sources still produce stable keys.
  - Truncates to 16 hex chars in the entry directory name — 64 bits of entropy is plenty at cache scale and keeps `ls` output short.
  - `hfcache-v2-` schema prefix — bumping it invalidates old entries (callers own gc policy; the cache owns keys).
  - `.hf-complete` dotfile sentinel. An entry dir without the sentinel is treated as a miss (covers crash-mid-extract and abandoned writes); the next render re-extracts over the partial frames with `-y`.
  - `FRAME_FILENAME_PREFIX = "frame_"` shared with the extractor — future refactors only need to touch one place to rename frames.
- `EngineConfig.extractCacheDir` (env: `HYPERFRAMES_EXTRACT_CACHE_DIR`) gates the feature. Undefined disables caching — extraction runs into the render's workDir and cleanup removes it on render end, preserving the prior behaviour exactly. No default root is chosen by the engine; the caller (CLI, app, studio) owns the location policy.
- `ExtractedFrames.ownedByLookup` flag prevents `FrameLookupTable.cleanup` from rm'ing a shared cache dir at render end. Set to `true` on both hits and misses (misses own the directory they wrote into, but hand it over to the cache rather than deleting it).
- Phase 3 extractor flow:
  1. Snapshot `(videoPath, mediaStart, start, end)` per resolved video BEFORE Phase 2a/2b preflight mutates them — so cache keys are stable across renders that use workDir-local normalized files (those files have fresh mtimes every render).
  2. Compute key, `lookupCacheEntry`.
  3. On hit: rebuild `ExtractedFrames` from the cache dir plus the Phase 2-probed `VideoMetadata` — no re-ffprobe.
  4. On miss: `ensureCacheEntryDir`, extract with `extractVideoFramesRange(..., outputDirOverride)`, then `markCacheEntryComplete` (the sentinel write is the last step so a crash leaves the dir un-sentineled).
- `extractVideoFramesRange` gains an `outputDirOverride` parameter so cache-miss writes land directly in the keyed dir (no `join(outputDir, videoId)` wrapping).

## Test plan

- [x] 19 unit tests in `extractionCache.test.ts` covering key determinism, mtime/size invalidation, format/fps/mediaStart/duration invalidation, Infinity normalization, sentinel semantics, missing-file tolerance
- [x] 2 integration tests in `videoFrameExtractor.test.ts`:
  - "reuses extracted frames on a warm cache hit" — asserts `cacheHits=1`, `extractMs<50ms` on second call against a CFR SDR fixture
  - "invalidates the cache when fps changes" — different fps on second call forces a new miss
- [x] End-to-end validation with `HYPERFRAMES_EXTRACT_CACHE_DIR` set, two runs of the same fixture
- [x] Lint + format (oxlint + oxfmt)
- [x] Typecheck (engine + producer)
This commit is contained in:
James Russo
2026-04-24 00:35:31 -04:00
committed by GitHub
parent 9912d3730a
commit 57cdf7d80e
6 changed files with 715 additions and 15 deletions
+26
View File
@@ -72,6 +72,30 @@ export interface EngineConfig {
/** Custom manifest path for Hyperframe runtime. */
runtimeManifestPath?: string;
// ── Cache ────────────────────────────────────────────────────────────
/**
* Directory where the content-addressed extraction cache persists frame
* bundles keyed on (path, mtime, size, mediaStart, duration, fps, format).
* Undefined disables caching — extraction runs into the render's workDir
* and cleanup removes it when the render ends, preserving the pre-cache
* behaviour.
*
* **Single-writer.** The cache is not safe for concurrent renders pointing
* at the same directory. A `.hf-complete` sentinel prevents another render
* from serving an entry that hasn't finished extracting, but individual
* frame files are written non-atomically — a second render reading during
* the write window can observe a truncated frame. Give each concurrent
* render pipeline its own `extractCacheDir`, or gate with an external mutex.
*
* **Network filesystems.** `mtime` resolution on NFS/SMB mounts can be
* coarser than expected (seconds rather than nanoseconds), which may
* produce spurious cache hits if a source file is overwritten within the
* same mtime tick. Local filesystems are the intended deployment target.
*
* Env fallback: `HYPERFRAMES_EXTRACT_CACHE_DIR`.
*/
extractCacheDir?: string;
// ── Debug ────────────────────────────────────────────────────────────
debug: boolean;
}
@@ -203,6 +227,8 @@ export function resolveConfig(overrides?: Partial<EngineConfig>): EngineConfig {
verifyRuntime: env("PRODUCER_VERIFY_HYPERFRAME_RUNTIME") !== "false",
runtimeManifestPath: env("PRODUCER_HYPERFRAME_MANIFEST_PATH"),
extractCacheDir: env("HYPERFRAMES_EXTRACT_CACHE_DIR"),
};
// Remove undefined values so they don't override defaults
@@ -0,0 +1,199 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import {
COMPLETE_SENTINEL,
FRAME_FILENAME_PREFIX,
SCHEMA_PREFIX,
cacheEntryDirName,
computeCacheKey,
ensureCacheEntryDir,
lookupCacheEntry,
markCacheEntryComplete,
readKeyStat,
type CacheKeyInput,
} from "./extractionCache.js";
const keyFor = (videoPath: string, overrides: Partial<CacheKeyInput> = {}): CacheKeyInput => {
const stat = readKeyStat(videoPath);
if (!stat) throw new Error(`keyFor fixture missing on disk: ${videoPath}`);
return {
videoPath,
mtimeMs: stat.mtimeMs,
size: stat.size,
mediaStart: 0,
duration: 3,
fps: 30,
format: "jpg",
...overrides,
};
};
describe("extractionCache constants", () => {
it("exposes the v2 schema prefix", () => {
expect(SCHEMA_PREFIX).toBe("hfcache-v2-");
});
it("exposes the frame filename prefix shared with the extractor", () => {
expect(FRAME_FILENAME_PREFIX).toBe("frame_");
});
it("uses a dotfile sentinel so ls-without-A hides it", () => {
expect(COMPLETE_SENTINEL.startsWith(".")).toBe(true);
});
});
describe("computeCacheKey", () => {
let tmpRoot: string;
let sourceFile: string;
beforeEach(() => {
tmpRoot = mkdtempSync(join(tmpdir(), "hf-extract-cache-test-"));
sourceFile = join(tmpRoot, "clip.mp4");
writeFileSync(sourceFile, "fake-video-bytes", "utf-8");
});
afterEach(() => {
if (existsSync(tmpRoot)) rmSync(tmpRoot, { recursive: true, force: true });
});
const base = (videoPath: string): CacheKeyInput => keyFor(videoPath);
it("returns the same key for identical inputs", () => {
const a = computeCacheKey(base(sourceFile));
const b = computeCacheKey(base(sourceFile));
expect(a).toBe(b);
});
it("produces a 64-char hex SHA-256 digest", () => {
const key = computeCacheKey(base(sourceFile));
expect(key).toMatch(/^[0-9a-f]{64}$/);
});
it("changes when path changes (moved files re-extract)", () => {
const other = join(tmpRoot, "other.mp4");
writeFileSync(other, "fake-video-bytes", "utf-8");
const a = computeCacheKey(base(sourceFile));
const b = computeCacheKey(base(other));
expect(a).not.toBe(b);
});
it("changes when mediaStart changes", () => {
const a = computeCacheKey(base(sourceFile));
const b = computeCacheKey({ ...base(sourceFile), mediaStart: 1 });
expect(a).not.toBe(b);
});
it("changes when duration changes", () => {
const a = computeCacheKey(base(sourceFile));
const b = computeCacheKey({ ...base(sourceFile), duration: 5 });
expect(a).not.toBe(b);
});
it("changes when fps changes (different frame count invalidates key)", () => {
const a = computeCacheKey(base(sourceFile));
const b = computeCacheKey({ ...base(sourceFile), fps: 60 });
expect(a).not.toBe(b);
});
it("changes when format changes", () => {
const a = computeCacheKey(base(sourceFile));
const b = computeCacheKey({ ...base(sourceFile), format: "png" });
expect(a).not.toBe(b);
});
it("normalizes non-finite duration so Infinity doesn't produce unstable keys", () => {
const a = computeCacheKey({ ...base(sourceFile), duration: Infinity });
const b = computeCacheKey({ ...base(sourceFile), duration: Infinity });
expect(a).toBe(b);
});
it("changes when file content changes (mtime+size bump)", () => {
const before = computeCacheKey(base(sourceFile));
// Force an mtime change by waiting 5ms then overwriting with different bytes.
// 5ms is well above the Linux mtime resolution (typically nanoseconds) and
// below any Windows cache coherency window. Using a longer sleep pads against
// coarse filesystem mtime granularity without slowing the suite.
const start = Date.now();
while (Date.now() - start < 5) {
/* spin */
}
writeFileSync(sourceFile, "different-bytes-longer-than-before", "utf-8");
const after = computeCacheKey(base(sourceFile));
expect(after).not.toBe(before);
});
it("readKeyStat returns null for a missing source (callers skip the cache)", () => {
// Previously readKeyStat returned a `{mtimeMs: 0, size: 0}` sentinel for
// missing files; two unrelated missing paths then shared the same cache
// key tuple and polluted the cache. The contract now returns null so
// callers can explicitly skip the cache path and let the extractor
// surface the real file-not-found error.
const missing = join(tmpRoot, "does-not-exist.mp4");
expect(readKeyStat(missing)).toBeNull();
});
});
describe("cacheEntryDirName", () => {
it("prefixes with the schema and truncates to 16 hex chars", () => {
const full = "a".repeat(64);
expect(cacheEntryDirName(full)).toBe(`${SCHEMA_PREFIX}${"a".repeat(16)}`);
});
});
describe("lookupCacheEntry / markCacheEntryComplete", () => {
let tmpRoot: string;
let sourceFile: string;
beforeEach(() => {
tmpRoot = mkdtempSync(join(tmpdir(), "hf-extract-cache-test-"));
sourceFile = join(tmpRoot, "clip.mp4");
writeFileSync(sourceFile, "fake-video-bytes", "utf-8");
});
afterEach(() => {
if (existsSync(tmpRoot)) rmSync(tmpRoot, { recursive: true, force: true });
});
const base = (videoPath: string): CacheKeyInput => keyFor(videoPath);
it("misses on an empty cache root", () => {
const lookup = lookupCacheEntry(tmpRoot, base(sourceFile));
expect(lookup.hit).toBe(false);
expect(lookup.entry.dir.startsWith(tmpRoot)).toBe(true);
});
it("hits after ensureCacheEntryDir + markCacheEntryComplete", () => {
const first = lookupCacheEntry(tmpRoot, base(sourceFile));
ensureCacheEntryDir(first.entry);
markCacheEntryComplete(first.entry);
const second = lookupCacheEntry(tmpRoot, base(sourceFile));
expect(second.hit).toBe(true);
expect(second.entry.dir).toBe(first.entry.dir);
});
it("treats an in-progress dir without the sentinel as a miss", () => {
const lookup = lookupCacheEntry(tmpRoot, base(sourceFile));
ensureCacheEntryDir(lookup.entry);
// Simulate abandoned extraction — frames written but sentinel never marked.
writeFileSync(join(lookup.entry.dir, "frame_00001.jpg"), "x", "utf-8");
const again = lookupCacheEntry(tmpRoot, base(sourceFile));
expect(again.hit).toBe(false);
});
it("places entries under the cache root, not the source parent", () => {
const subroot = join(tmpRoot, "cache-root");
mkdirSync(subroot, { recursive: true });
const lookup = lookupCacheEntry(subroot, base(sourceFile));
expect(lookup.entry.dir.startsWith(subroot)).toBe(true);
});
it("uses the same directory for identical inputs across lookups", () => {
const a = lookupCacheEntry(tmpRoot, base(sourceFile));
const b = lookupCacheEntry(tmpRoot, base(sourceFile));
expect(a.entry.dir).toBe(b.entry.dir);
});
});
@@ -0,0 +1,216 @@
/**
* Content-Addressed Extraction Cache
*
* Video frame extraction is the single most expensive phase of a render
* after capture. Repeat renders of the same composition (preview → final,
* studio iteration) re-extract identical frames from the same source file,
* burning ffmpeg time that adds no value. This module keys extracted frame
* bundles on the (path, mtime, size, mediaStart, duration, fps, format)
* tuple so re-renders resolve to a pre-extracted directory instead of
* re-invoking ffmpeg.
*
* ### Scheme
*
* - The key is the SHA-256 of a stable JSON encoding of the tuple above.
* - Cache entries live under `<rootDir>/<SCHEMA_PREFIX><key[0..16]>/` so
* `ls` output and tracing logs stay short. Truncation to 16 hex chars
* leaves 64 bits of entropy — collision risk at cache scale is negligible.
* - A completed entry is marked by writing the `.hf-complete` sentinel file
* after all frames are on disk. A dir without the sentinel is treated as
* absent (stale/abandoned) and re-extracted into a fresh key (the old dir
* is left for external gc — the cache owns keys, not deletion policy).
*
* ### Versioning
*
* `SCHEMA_PREFIX` bumps when the cache-contents invariant changes (e.g.
* extraction format, frame layout). Old entries under the previous prefix
* become inert and can be gc'd by the caller.
*/
import { createHash } from "node:crypto";
import { mkdirSync, readdirSync, statSync, writeFileSync } from "node:fs";
import { existsSync } from "node:fs";
import { join } from "node:path";
import type { VideoMetadata } from "../utils/ffprobe.js";
/** Filename prefix for extracted frames. Shared with the extractor. */
export const FRAME_FILENAME_PREFIX = "frame_";
/** Sentinel filename written after a cache entry is fully populated. */
export const COMPLETE_SENTINEL = ".hf-complete";
/** Current schema version. Bump when cache-entry layout changes. */
export const SCHEMA_PREFIX = "hfcache-v2-";
/** Truncated hex chars of SHA-256 used for the entry directory name. */
const KEY_HEX_CHARS = 16;
export type CacheFrameFormat = "jpg" | "png";
export interface CacheKeyInput {
/** Absolute path to the source video file. Part of the key so moved files
* re-extract rather than match by (size, mtime) alone. */
videoPath: string;
/** Source file modification time in ms (floored). Invalidates the key on edit. */
mtimeMs: number;
/** Source file size in bytes. Invalidates the key on content change. */
size: number;
/** Seconds into source the composition starts reading (video.mediaStart). */
mediaStart: number;
/** Seconds of source the composition uses. Infinity is normalized to -1
* so callers that pass an unresolved "natural duration" still produce a
* stable key across invocations. */
duration: number;
/** Target output frames-per-second. */
fps: number;
/** Output image format. */
format: CacheFrameFormat;
}
export interface CacheEntry {
/** Absolute path to the cache entry directory. */
dir: string;
/** Full 64-char SHA-256 hex digest (parent of the truncated key). */
keyHash: string;
}
export interface CacheLookup {
/** Cache entry information — always returned even on a miss so the caller
* can extract directly into `dir` then call `markCacheEntryComplete`. */
entry: CacheEntry;
/** True when the entry exists AND carries the completion sentinel. */
hit: boolean;
}
/**
* Read `(mtimeMs, size)` for a path. Returns `null` if the file is missing —
* callers should skip the cache path for that entry so the extractor surfaces
* the real file-not-found error. Returning a zero-stat sentinel would let two
* missing files share the same `(0, 0)` tuple and pollute the cache with an
* orphaned entry.
*/
export function readKeyStat(videoPath: string): { mtimeMs: number; size: number } | null {
try {
const stat = statSync(videoPath);
return { mtimeMs: Math.floor(stat.mtimeMs), size: stat.size };
} catch {
return null;
}
}
function canonicalKeyBlob(input: CacheKeyInput): string {
const durationForKey = Number.isFinite(input.duration) ? input.duration : -1;
return JSON.stringify({
p: input.videoPath,
m: input.mtimeMs,
s: input.size,
ms: input.mediaStart,
d: durationForKey,
f: input.fps,
fmt: input.format,
});
}
/**
* Compute the SHA-256 hex digest for a cache key input.
*/
export function computeCacheKey(input: CacheKeyInput): string {
return createHash("sha256").update(canonicalKeyBlob(input)).digest("hex");
}
/**
* Derive the truncated cache-entry directory name from a full key hash.
* Exposed so tests and the entry dir resolver share one truncation rule.
*/
export function cacheEntryDirName(keyHash: string): string {
return SCHEMA_PREFIX + keyHash.slice(0, KEY_HEX_CHARS);
}
/**
* Look up a cache entry by key input. Returns the resolved entry path plus a
* `hit` flag. On miss, callers should extract frames into `entry.dir`
* (after calling `ensureCacheEntryDir`) and then call `markCacheEntryComplete`
* once the extraction succeeds.
*/
export function lookupCacheEntry(rootDir: string, input: CacheKeyInput): CacheLookup {
const keyHash = computeCacheKey(input);
const dir = join(rootDir, cacheEntryDirName(keyHash));
const complete = existsSync(join(dir, COMPLETE_SENTINEL));
return { entry: { dir, keyHash }, hit: complete };
}
/**
* Ensure a cache entry's directory exists so the extractor can write into it.
* Idempotent: `mkdirSync({recursive:true})` is a no-op when the dir exists.
*/
export function ensureCacheEntryDir(entry: CacheEntry): void {
mkdirSync(entry.dir, { recursive: true });
}
/**
* Write the completion sentinel so subsequent lookups treat this entry as a
* hit. Must be called only after every frame has been written.
*
* Concurrency: lookup→populate→mark is non-atomic. Two concurrent renders of
* the same key may both miss, both extract into the same dir, and the later
* writer's frames win. The result is correct (identical inputs yield identical
* frames) but wasteful. Acceptable for a single-process render pipeline;
* anyone running concurrent renders against a shared cache root should front
* it with an external lock.
*/
export function markCacheEntryComplete(entry: CacheEntry): void {
writeFileSync(join(entry.dir, COMPLETE_SENTINEL), "", "utf-8");
}
/**
* Rebuild the in-memory frame index for a cached entry. Called on cache hits
* so the extractor's caller receives the same `ExtractedFrames` shape it
* would get from a fresh extraction — without re-running ffmpeg or ffprobe.
*
* The `metadata` argument is the `VideoMetadata` probed in the extractor's
* Phase 2 (pre-preflight). Passing it here avoids an extra ffprobe on the
* hit path.
*/
export interface RehydrateOptions {
videoId: string;
srcPath: string;
fps: number;
format: CacheFrameFormat;
metadata: VideoMetadata;
}
export interface RehydratedFrames {
videoId: string;
srcPath: string;
outputDir: string;
framePattern: string;
fps: number;
totalFrames: number;
metadata: VideoMetadata;
framePaths: Map<number, string>;
}
export function rehydrateCacheEntry(
entry: CacheEntry,
options: RehydrateOptions,
): RehydratedFrames {
const framePattern = `${FRAME_FILENAME_PREFIX}%05d.${options.format}`;
const framePaths = new Map<number, string>();
const suffix = `.${options.format}`;
const files = readdirSync(entry.dir)
.filter((f) => f.startsWith(FRAME_FILENAME_PREFIX) && f.endsWith(suffix))
.sort();
files.forEach((file, idx) => {
framePaths.set(idx, join(entry.dir, file));
});
return {
videoId: options.videoId,
srcPath: options.srcPath,
outputDir: entry.dir,
framePattern,
fps: options.fps,
totalFrames: framePaths.size,
metadata: options.metadata,
framePaths,
};
}
@@ -196,6 +196,137 @@ describe.skipIf(!HAS_FFMPEG)("extractAllVideoFrames on a VFR source", () => {
expect(result.phaseBreakdown.vfrPreflightMs).toBeGreaterThan(0);
}, 60_000);
it("reuses extracted frames on a warm cache hit", async () => {
const CACHE_DIR = mkdtempSync(join(tmpdir(), "hf-extract-cache-test-"));
const SRC = join(FIXTURE_DIR, "cache-src.mp4");
// Synthesize a clean CFR SDR clip — bypasses VFR preflight so the cache
// key is stable across the two runs.
const synth = await runFfmpeg([
"-y",
"-hide_banner",
"-loglevel",
"error",
"-f",
"lavfi",
"-i",
"testsrc2=s=320x180:d=2:rate=30",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-pix_fmt",
"yuv420p",
SRC,
]);
if (!synth.success) {
throw new Error(`Cache fixture synthesis failed: ${synth.stderr.slice(-400)}`);
}
const video: VideoElement = {
id: "cv1",
src: SRC,
start: 0,
end: 2,
mediaStart: 0,
hasAudio: false,
};
const outDirA = join(FIXTURE_DIR, "out-cache-miss");
mkdirSync(outDirA, { recursive: true });
const miss = await extractAllVideoFrames(
[video],
FIXTURE_DIR,
{ fps: 30, outputDir: outDirA },
undefined,
{ extractCacheDir: CACHE_DIR },
);
expect(miss.errors).toEqual([]);
expect(miss.phaseBreakdown.cacheHits).toBe(0);
expect(miss.phaseBreakdown.cacheMisses).toBe(1);
const outDirB = join(FIXTURE_DIR, "out-cache-hit");
mkdirSync(outDirB, { recursive: true });
const hit = await extractAllVideoFrames(
[video],
FIXTURE_DIR,
{ fps: 30, outputDir: outDirB },
undefined,
{ extractCacheDir: CACHE_DIR },
);
expect(hit.errors).toEqual([]);
expect(hit.phaseBreakdown.cacheHits).toBe(1);
expect(hit.phaseBreakdown.cacheMisses).toBe(0);
// extractMs on a hit is only the cache-lookup bookkeeping; asserting <50ms
// is loose enough to survive CI jitter but tight enough to catch a
// regression that accidentally triggered ffmpeg again.
expect(hit.phaseBreakdown.extractMs).toBeLessThan(50);
expect(hit.extracted).toHaveLength(1);
expect(hit.extracted[0]!.totalFrames).toBe(miss.extracted[0]!.totalFrames);
rmSync(CACHE_DIR, { recursive: true, force: true });
}, 60_000);
it("invalidates the cache when fps changes", async () => {
const CACHE_DIR = mkdtempSync(join(tmpdir(), "hf-extract-cache-test-"));
const SRC = join(FIXTURE_DIR, "cache-fps-src.mp4");
const synth = await runFfmpeg([
"-y",
"-hide_banner",
"-loglevel",
"error",
"-f",
"lavfi",
"-i",
"testsrc2=s=320x180:d=1:rate=30",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-pix_fmt",
"yuv420p",
SRC,
]);
if (!synth.success) {
throw new Error(`Cache-fps fixture synthesis failed: ${synth.stderr.slice(-400)}`);
}
const video: VideoElement = {
id: "cv2",
src: SRC,
start: 0,
end: 1,
mediaStart: 0,
hasAudio: false,
};
const outA = join(FIXTURE_DIR, "out-cache-fps-30");
mkdirSync(outA, { recursive: true });
const first = await extractAllVideoFrames(
[video],
FIXTURE_DIR,
{ fps: 30, outputDir: outA },
undefined,
{ extractCacheDir: CACHE_DIR },
);
expect(first.phaseBreakdown.cacheMisses).toBe(1);
const outB = join(FIXTURE_DIR, "out-cache-fps-60");
mkdirSync(outB, { recursive: true });
const second = await extractAllVideoFrames(
[video],
FIXTURE_DIR,
{ fps: 60, outputDir: outB },
undefined,
{ extractCacheDir: CACHE_DIR },
);
expect(second.phaseBreakdown.cacheMisses).toBe(1);
expect(second.phaseBreakdown.cacheHits).toBe(0);
rmSync(CACHE_DIR, { recursive: true, force: true });
}, 60_000);
// Regression test for the segment-scope HDR preflight fix: pre-fix,
// convertSdrToHdr re-encoded the entire source, so a 30-minute SDR source
// contributing a 2-second clip took ~200× longer than needed. Post-fix the
@@ -18,6 +18,15 @@ import {
import { downloadToTemp, isHttpUrl } from "../utils/urlDownloader.js";
import { runFfmpeg } from "../utils/runFfmpeg.js";
import { DEFAULT_CONFIG, type EngineConfig } from "../config.js";
import {
FRAME_FILENAME_PREFIX,
ensureCacheEntryDir,
lookupCacheEntry,
markCacheEntryComplete,
readKeyStat,
rehydrateCacheEntry,
type CacheFrameFormat,
} from "./extractionCache.js";
export interface VideoElement {
id: string;
@@ -37,6 +46,13 @@ export interface ExtractedFrames {
totalFrames: number;
metadata: VideoMetadata;
framePaths: Map<number, string>;
/**
* True when the extractor owns `outputDir` and cleanup should rm it when
* the render ends. Cache hits set this to false so the shared entry isn't
* deleted by a single render's cleanup — the cache dir is owned by the
* caller's gc policy, not any one render.
*/
ownedByLookup?: boolean;
}
export interface ExtractionOptions {
@@ -181,15 +197,22 @@ export async function extractVideoFramesRange(
options: ExtractionOptions,
signal?: AbortSignal,
config?: Partial<Pick<EngineConfig, "ffmpegProcessTimeout">>,
/**
* Override the output directory for this extraction. When provided, frames
* are written directly into `outputDirOverride` (no per-videoId subdir).
* Used by the cache layer to materialize frames straight into the keyed
* cache entry directory.
*/
outputDirOverride?: string,
): Promise<ExtractedFrames> {
const ffmpegProcessTimeout = config?.ffmpegProcessTimeout ?? DEFAULT_CONFIG.ffmpegProcessTimeout;
const { fps, outputDir, quality = 95, format = "jpg" } = options;
const videoOutputDir = join(outputDir, videoId);
const videoOutputDir = outputDirOverride ?? join(outputDir, videoId);
if (!existsSync(videoOutputDir)) mkdirSync(videoOutputDir, { recursive: true });
const metadata = await extractMediaMetadata(videoPath);
const framePattern = `frame_%05d.${format}`;
const framePattern = `${FRAME_FILENAME_PREFIX}%05d.${format}`;
const outputPattern = join(videoOutputDir, framePattern);
// When extracting from HDR source, tone-map to SDR in FFmpeg rather than
@@ -253,7 +276,7 @@ export async function extractVideoFramesRange(
const framePaths = new Map<number, string>();
const files = readdirSync(videoOutputDir)
.filter((f) => f.startsWith("frame_") && f.endsWith(`.${format}`))
.filter((f) => f.startsWith(FRAME_FILENAME_PREFIX) && f.endsWith(`.${format}`))
.sort();
files.forEach((file, index) => {
framePaths.set(index, join(videoOutputDir, file));
@@ -353,6 +376,21 @@ async function convertSdrToHdr(
}
}
/**
* Resolve the used-segment duration for a video, falling back to the source's
* natural duration when the caller hasn't specified bounds (end=Infinity) or
* the bounds are nonsensical (end<=start).
*/
function resolveSegmentDuration(
requested: number,
mediaStart: number,
metadata: VideoMetadata,
): number {
if (Number.isFinite(requested) && requested > 0) return requested;
const sourceRemaining = metadata.durationSeconds - mediaStart;
return sourceRemaining > 0 ? sourceRemaining : metadata.durationSeconds;
}
/**
* Re-encode a VFR (variable frame rate) video segment to CFR so the downstream
* fps filter can extract frames reliably. Screen recordings, phone videos, and
@@ -414,7 +452,7 @@ export async function extractAllVideoFrames(
baseDir: string,
options: ExtractionOptions,
signal?: AbortSignal,
config?: Partial<Pick<EngineConfig, "ffmpegProcessTimeout">>,
config?: Partial<Pick<EngineConfig, "ffmpegProcessTimeout" | "extractCacheDir">>,
compiledDir?: string,
): Promise<ExtractionResult> {
const startTime = Date.now();
@@ -469,6 +507,28 @@ export async function extractAllVideoFrames(
breakdown.resolveMs = Date.now() - phase1Start;
// Snapshot the pre-preflight key inputs so the extraction cache keys on the
// user-visible source (original path, original mediaStart, original segment
// bounds) rather than the workDir-local normalized file produced by
// Phase 2a/2b preflight. Without this, every render would write a new
// normalized file with a fresh mtime → fresh cache key → perpetual misses.
const cacheKeyInputs = resolvedVideos.map(({ video, videoPath }) => {
const stat = readKeyStat(videoPath);
// Missing files return null — skip the cache path for that entry. The
// extractor will surface the real file-not-found error downstream, and we
// avoid polluting the cache with a `(mtimeMs: 0, size: 0)` tuple that two
// unrelated missing paths would otherwise share.
if (!stat) return null;
return {
videoPath,
mtimeMs: stat.mtimeMs,
size: stat.size,
mediaStart: video.mediaStart,
start: video.start,
end: video.end,
};
});
// Phase 2: Probe color spaces and normalize if mixed HDR/SDR
const phase2ProbeStart = Date.now();
const videoMetadata = await Promise.all(
@@ -565,6 +625,10 @@ export async function extractAllVideoFrames(
resolvedVideos.splice(i, 1);
videoMetadata.splice(i, 1);
videoColorSpaces.splice(i, 1);
// Added by the extraction-cache commit: keep cacheKeyInputs aligned
// with the other parallel arrays so Phase 3's `cacheKeyInputs[i]`
// lookup doesn't point at a stale slot after the splice.
cacheKeyInputs.splice(i, 1);
}
}
}
@@ -615,25 +679,85 @@ export async function extractAllVideoFrames(
}
breakdown.vfrPreflightMs = Date.now() - vfrPreflightStart;
// Phase 3: Extract frames (parallel)
const phase3Start = Date.now();
const cacheRootDir = config?.extractCacheDir;
const cacheFormat: CacheFrameFormat = options.format ?? "jpg";
async function tryCachedExtract(
video: VideoElement,
videoPath: string,
videoDuration: number,
i: number,
): Promise<ExtractedFrames | null> {
if (!cacheRootDir) return null;
const keyInput = cacheKeyInputs[i];
const probedMeta = videoMetadata[i];
if (!keyInput || !probedMeta) return null;
const keyDuration = resolveSegmentDuration(
keyInput.end - keyInput.start,
keyInput.mediaStart,
probedMeta,
);
const lookup = lookupCacheEntry(cacheRootDir, {
videoPath: keyInput.videoPath,
mtimeMs: keyInput.mtimeMs,
size: keyInput.size,
mediaStart: keyInput.mediaStart,
duration: keyDuration,
fps: options.fps,
format: cacheFormat,
});
if (lookup.hit) {
breakdown.cacheHits += 1;
const rehydrated = rehydrateCacheEntry(lookup.entry, {
videoId: video.id,
srcPath: keyInput.videoPath,
fps: options.fps,
format: cacheFormat,
metadata: probedMeta,
});
return { ...rehydrated, ownedByLookup: true };
}
breakdown.cacheMisses += 1;
ensureCacheEntryDir(lookup.entry);
const result = await extractVideoFramesRange(
videoPath,
video.id,
video.mediaStart,
videoDuration,
options,
signal,
config,
lookup.entry.dir,
);
// Mark complete only AFTER frames are on disk — a crash mid-extract
// leaves the entry un-sentineled so the next lookup re-extracts over it.
markCacheEntryComplete(lookup.entry);
return { ...result, ownedByLookup: true };
}
const results = await Promise.all(
resolvedVideos.map(async ({ video, videoPath }) => {
resolvedVideos.map(async ({ video, videoPath }, i) => {
if (signal?.aborted) {
throw new Error("Video frame extraction cancelled");
}
try {
let videoDuration = video.end - video.start;
// Fallback: if no data-duration/data-end was specified (end is Infinity or 0),
// probe the actual video file to get its natural duration.
if (!Number.isFinite(videoDuration) || videoDuration <= 0) {
const metadata = await extractMediaMetadata(videoPath);
const sourceDuration = metadata.durationSeconds - video.mediaStart;
videoDuration = sourceDuration > 0 ? sourceDuration : metadata.durationSeconds;
const probedMeta = videoMetadata[i] ?? (await extractMediaMetadata(videoPath));
const videoDuration = resolveSegmentDuration(
video.end - video.start,
video.mediaStart,
probedMeta,
);
if (video.end - video.start !== videoDuration) {
video.end = video.start + videoDuration;
}
const cached = await tryCachedExtract(video, videoPath, videoDuration, i);
if (cached) return { result: cached };
const result = await extractVideoFramesRange(
videoPath,
video.id,
@@ -800,6 +924,10 @@ export class FrameLookupTable {
cleanup(): void {
for (const video of this.videos.values()) {
// Cache-hit / cache-write entries are owned by the extraction cache —
// a single render must not delete them, or the next render's lookup
// would miss and re-extract unnecessarily.
if (video.extracted.ownedByLookup) continue;
if (existsSync(video.extracted.outputDir)) {
rmSync(video.extracted.outputDir, { recursive: true, force: true });
}
@@ -1427,7 +1427,7 @@ export async function executeRenderJob(
projectDir,
{ fps: job.config.fps, outputDir: join(workDir, "video-frames") },
abortSignal,
undefined,
{ extractCacheDir: cfg.extractCacheDir },
compiledDir,
);
assertNotAborted();