mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-05 00:56:23 +00:00
perf(engine): superset extraction for overlapping trims of one source (#1885)
* perf(engine): superset extraction for overlapping trims of one source Cache-missing trims of the same source that are frame-aligned and overlapping decode their union window in ONE ffmpeg pass; each trim's frames are materialized by hardlinking the superset frames with renumbered names (copy fallback on EXDEV). Byte-identical to per-trim extraction on CFR sources (verified by content hash in the A/B run), ~2x less decode+encode work for typical overlapping trims, and sparse-keyframe sources pay the keyframe seek once instead of once per trim. Disjoint or misaligned trims keep the direct path; any union failure falls back to per-trim extraction. Also: warm renders (zero cache misses) skip the extraction-cache GC sweep instead of paying a full cache size scan. * fix(engine): superset review hardening - clustering, abort, cache-fs temp, gc staleness - Partition each source's trims into overlap-connected components before the union check, so one disjoint outlier no longer collapses the whole bucket to direct extraction (pinned by a 3-of-4-overlap test). - On abort, the superset fallback no longer re-runs every member through direct extraction (N doomed ffmpeg spawns); the cancellation surfaces per member instead. - The superset temp dir moves onto the cache filesystem when the cache is active so member hardlinks into partial dirs cannot EXDEV-copy and silently multiply disk usage; its .partial- name puts crashed leftovers under the GC's aged-partial sweep. - GC staleness fallback: a .hf-last-gc marker is stamped per sweep and all-hit renders sweep anyway once it is older than 24h, so 100%-warm workloads still reclaim space (pinned by a stale-marker test).
This commit is contained in:
@@ -53,6 +53,9 @@ export const FRAME_FILENAME_PREFIX = "frame_";
|
||||
/** Sentinel filename written after a cache entry is fully populated. */
|
||||
export const COMPLETE_SENTINEL = ".hf-complete";
|
||||
|
||||
/** Marker file stamped after each GC sweep; drives the staleness fallback. */
|
||||
export const GC_MARKER = ".hf-last-gc";
|
||||
|
||||
/**
|
||||
* Current schema version. Bump when the cache-contents invariant changes.
|
||||
* v2 -> v3: one-pass VFR extraction (-fps_mode cfr) replaces the two-pass
|
||||
@@ -390,11 +393,29 @@ export interface GcStats {
|
||||
* a liveness heuristic, not a lock. Returns counts so the caller can surface
|
||||
* eviction pressure in render observability.
|
||||
*/
|
||||
/**
|
||||
* Whether the staleness fallback should force a sweep: true when no sweep
|
||||
* marker exists or the last sweep is older than `maxAgeMs`. Lets 100%-warm
|
||||
* workloads (which skip the per-miss sweep) still reclaim space eventually.
|
||||
*/
|
||||
export function gcSweepDue(rootDir: string, maxAgeMs: number): boolean {
|
||||
try {
|
||||
return Date.now() - statSync(join(rootDir, GC_MARKER)).mtimeMs > maxAgeMs;
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
export function gcExtractionCache(
|
||||
rootDir: string,
|
||||
opts: { maxBytes: number; minAgeMs: number },
|
||||
): GcStats {
|
||||
const stats: GcStats = { evictedEntries: 0, evictedBytes: 0, agedPartialsRemoved: 0 };
|
||||
try {
|
||||
writeFileSync(join(rootDir, GC_MARKER), "", "utf-8");
|
||||
} catch {
|
||||
// Unwritable root: the sweep below will no-op on the same root anyway.
|
||||
}
|
||||
try {
|
||||
const now = Date.now();
|
||||
const entries: GcEntry[] = [];
|
||||
|
||||
Reference in New Issue
Block a user