perf(engine): superset extraction for overlapping trims of one source (#1885)

* perf(engine): superset extraction for overlapping trims of one source

Cache-missing trims of the same source that are frame-aligned and
overlapping decode their union window in ONE ffmpeg pass; each trim's
frames are materialized by hardlinking the superset frames with
renumbered names (copy fallback on EXDEV). Byte-identical to per-trim
extraction on CFR sources (verified by content hash in the A/B run),
~2x less decode+encode work for typical overlapping trims, and
sparse-keyframe sources pay the keyframe seek once instead of once per
trim. Disjoint or misaligned trims keep the direct path; any union
failure falls back to per-trim extraction.

Also: warm renders (zero cache misses) skip the extraction-cache GC
sweep instead of paying a full cache size scan.

* fix(engine): superset review hardening - clustering, abort, cache-fs temp, gc staleness

- Partition each source's trims into overlap-connected components before
  the union check, so one disjoint outlier no longer collapses the whole
  bucket to direct extraction (pinned by a 3-of-4-overlap test).
- On abort, the superset fallback no longer re-runs every member through
  direct extraction (N doomed ffmpeg spawns); the cancellation surfaces
  per member instead.
- The superset temp dir moves onto the cache filesystem when the cache
  is active so member hardlinks into partial dirs cannot EXDEV-copy and
  silently multiply disk usage; its .partial- name puts crashed
  leftovers under the GC's aged-partial sweep.
- GC staleness fallback: a .hf-last-gc marker is stamped per sweep and
  all-hit renders sweep anyway once it is older than 24h, so 100%-warm
  workloads still reclaim space (pinned by a stale-marker test).
This commit is contained in:
Miguel Ángel
2026-07-03 15:09:39 -07:00
committed by GitHub
parent 48f158a0c2
commit 1a7002f208
3 changed files with 689 additions and 138 deletions
@@ -53,6 +53,9 @@ export const FRAME_FILENAME_PREFIX = "frame_";
/** Sentinel filename written after a cache entry is fully populated. */
export const COMPLETE_SENTINEL = ".hf-complete";
/** Marker file stamped after each GC sweep; drives the staleness fallback. */
export const GC_MARKER = ".hf-last-gc";
/**
* Current schema version. Bump when the cache-contents invariant changes.
* v2 -> v3: one-pass VFR extraction (-fps_mode cfr) replaces the two-pass
@@ -390,11 +393,29 @@ export interface GcStats {
* a liveness heuristic, not a lock. Returns counts so the caller can surface
* eviction pressure in render observability.
*/
/**
* Whether the staleness fallback should force a sweep: true when no sweep
* marker exists or the last sweep is older than `maxAgeMs`. Lets 100%-warm
* workloads (which skip the per-miss sweep) still reclaim space eventually.
*/
export function gcSweepDue(rootDir: string, maxAgeMs: number): boolean {
try {
return Date.now() - statSync(join(rootDir, GC_MARKER)).mtimeMs > maxAgeMs;
} catch {
return true;
}
}
export function gcExtractionCache(
rootDir: string,
opts: { maxBytes: number; minAgeMs: number },
): GcStats {
const stats: GcStats = { evictedEntries: 0, evictedBytes: 0, agedPartialsRemoved: 0 };
try {
writeFileSync(join(rootDir, GC_MARKER), "", "utf-8");
} catch {
// Unwritable root: the sweep below will no-op on the same root anyway.
}
try {
const now = Date.now();
const entries: GcEntry[] = [];