mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-05 00:56:23 +00:00
060b6f8ae53e8ae243d8e2c2c3afd690dbe7f99c
27
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Miguel Ángel
|
2e8f871bc8 | fix(video): hold final frame through composition | ||
|
Miguel Ángel
|
6fc92308d6 | fix(engine): resolve root-absolute media from project (#2399) | ||
|
Miguel Ángel
|
9ac4ab8dee |
test(engine): write HDR fixture color tags into the H.264 VUI (#2389)
The SDR-to-HDR extraction tests synthesize their HDR fixture with -color_trc/-color_primaries flags and rely on the encoder propagating them into the bitstream. The pinned Windows CI ffmpeg build drops the transfer on that path, so after #2377 narrowed HDR detection to the transfer function, the fixture probes as SDR on Windows and both tests fail (hdrPreflightCount 0). Write the VUI directly with the h264_metadata bitstream filter so the tag survives on every build. |
||
|
Miguel Ángel
|
4a36655b2b |
fix(engine): resolve relative data-start references in video-frame extraction
* fix(engine): resolve relative data-start references in video-frame extraction <video data-start="intro"> (a relative reference to another clip's end) is resolved by the browser runtime but parseVideoElements/parseImageElements did a raw parseFloat, yielding NaN start/end. The FrameLookupTable active-window checks (start <= t <= end) are then always false, so the clip is never injected and composites BLANK in the final render — while lint/validate/inspect/snapshot and the live preview all look fine. The docs' Relative Timing section teaches exactly this pattern on <video>. Share the pure reference-syntax parser (parseStartExpression) out of the runtime resolver into @hyperframes/core, and resolve references in the extractor against the linkedom document it already holds: a reference resolves to the target clip's resolved start + its duration (data-duration or data-end) + offset, mirroring the runtime. Cycle-guarded; an unknown target or unknown duration falls back to the target's start / 0 (never NaN), matching runtime semantics. Natural-media-duration-only targets aren't known at parse time (same limit as the runtime's fallback). parseImageElements gets the same fix. Runtime resolver behavior is unchanged (its 25-case suite still passes). * chore: re-trigger CI to refresh a stuck CodeQL aggregate check |
||
|
Miguel Ángel
|
1a7002f208 |
perf(engine): superset extraction for overlapping trims of one source (#1885)
* perf(engine): superset extraction for overlapping trims of one source Cache-missing trims of the same source that are frame-aligned and overlapping decode their union window in ONE ffmpeg pass; each trim's frames are materialized by hardlinking the superset frames with renumbered names (copy fallback on EXDEV). Byte-identical to per-trim extraction on CFR sources (verified by content hash in the A/B run), ~2x less decode+encode work for typical overlapping trims, and sparse-keyframe sources pay the keyframe seek once instead of once per trim. Disjoint or misaligned trims keep the direct path; any union failure falls back to per-trim extraction. Also: warm renders (zero cache misses) skip the extraction-cache GC sweep instead of paying a full cache size scan. * fix(engine): superset review hardening - clustering, abort, cache-fs temp, gc staleness - Partition each source's trims into overlap-connected components before the union check, so one disjoint outlier no longer collapses the whole bucket to direct extraction (pinned by a 3-of-4-overlap test). - On abort, the superset fallback no longer re-runs every member through direct extraction (N doomed ffmpeg spawns); the cancellation surfaces per member instead. - The superset temp dir moves onto the cache filesystem when the cache is active so member hardlinks into partial dirs cannot EXDEV-copy and silently multiply disk usage; its .partial- name puts crashed leftovers under the GC's aged-partial sweep. - GC staleness fallback: a .hf-last-gc marker is stamped per sweep and all-hit renders sweep anyway once it is older than 24h, so 100%-warm workloads still reclaim space (pinned by a stale-marker test). |
||
|
Miguel Ángel
|
48f158a0c2 |
perf(engine): one-pass SDR-to-HDR extraction with cache-key transform (#1902)
* perf(engine): one-pass SDR-to-HDR extraction with cache-key transform Mixed-HDR compositions converted each SDR source with a full libx264 re-encode (convertSdrToHdr) before extraction. The BT.709 to BT.2020 colorspace remap now runs as a filter inside the extraction pass itself; convertSdrToHdr and the _hdr_normalized intermediate are deleted. Same shape as the earlier one-pass VFR change. Also fixes a cache-poisoning bug this exposed: the HDR preflight rewrote entry.videoPath AFTER the cache-key snapshot, so a mixed-HDR render cached converted frames under the plain source key and a later SDR render of the same trim would have served HDR-tinted frames. The cache key now carries an optional transform discriminator; keys without a transform stay byte-compatible with existing entries. * fix(engine): attribute SDR-to-HDR extract failures, pin filter-order intent Review hardening for one-pass SDR-to-HDR: - ffmpeg failures now carry an 'SDR→HDR conversion failed (colorspace filter in extract pass)' prefix when the remap is in the chain, so a filter-less ffmpeg build fails loudly with attribution instead of a generic extract error. - Comments pin the fps-before-colorspace ordering intent and mark sdrToHdrTransfers as the canonical read for both the cache key and extraction options. - Cross-render cache-poisoning regression test now compares frame BYTES across the cache boundary: mixed-HDR render then plain-SDR render of the same trim must produce different pixels, and a repeat plain render must hit the plain entry with byte-identical frames. |
||
|
Miguel Ángel
|
34590649a0 |
perf(engine): extraction cache on by default with atomic publish and LRU gc (#1901)
* perf(engine): extraction cache on by default with atomic publish and LRU gc Warm re-renders now skip source-video frame extraction entirely (video_extract 400ms -> 13ms on a 4-video composition; outputs are pixel-identical, PSNR inf). What made default-on safe: - Atomic entry publish: frames extract into a unique .partial-<pid>-<uuid> dir, the completion sentinel is written there, and the dir is renamed into the final key atomically. Concurrent renders sharing a cache can duplicate work but can never serve a torn entry (previously documented as single-writer only). - Size-capped LRU gc: best-effort sweep after extraction evicts oldest-used entries past a 2 GiB default budget (HYPERFRAMES_EXTRACT_CACHE_MAX_MB) and clears crashed writers' partials. Entries younger than 60 min are never evicted so live renders keep their frames. - Default cache dir: <tmpdir>/hyperframes-extract-cache-<uid>. Opt out with HYPERFRAMES_EXTRACT_CACHE_DIR=off (or none/false/0); a non-writable dir degrades to uncached with a single warning instead of failing the render. * fix(engine): harden extraction cache publish and surface cache ops signals Review hardening for the default-on extraction cache: - Bypass the cache for HDR-converted intermediates: the key snapshot describes the original source, so publishing converted frames under it would poison later plain-SDR renders of the same trim. (The follow-up transform-keyed change re-enables caching for these.) - publishCacheEntry TOCTOU: adopt a concurrent writer's completed entry both before removing an apparently-stale dir and after a failed retry rename, so a winner's publish is never destroyed or reported as a failure. - Observability for the failure paths: cachePublishFailures, cacheGcEvictions, cacheGcBytesFreed, and cacheAgedPartialsCleared on ExtractionPhaseBreakdown; gcExtractionCache now returns sweep stats. * fix(engine): sweep superseded cache generations in gc After a SCHEMA_PREFIX bump, old-generation entries (hfcache-v2-*) no longer matched the sweep's prefix filter and would orphan their disk forever. The gc now matches any hfcache-v* generation; superseded entries never receive sentinel touches, so the LRU evicts them first. |
||
|
Miguel Ángel
|
8d64d48e4a |
perf(engine): dedupe identical extractions within one render (#1900)
* perf(engine): write PNG frames at compression_level 1 Extracted video frames are render-scoped temp files read once during capture, so zlib effort above level 1 buys nothing. Measured 3.3x faster on 60s of 1080p H.264 to PNG (11.4s to 3.5s) and 5.4x on a 20s vp9-alpha webm (4.3s to 0.79s), for ~14% larger temp files. * perf(engine): one-pass VFR extraction with -fps_mode cfr VFR sources (screen recordings, phone videos) were re-encoded to CFR with libx264 and then extracted in a second ffmpeg pass. Extraction now runs a single pass with -fps_mode cfr -r <fps>. Same frame counts on the VFR regression fixtures (120/120 mid-seek, 297-303 full file), one less x264 generation of quality loss, ~3.4x faster on VFR inputs. convertVfrToCfr and the _vfr_normalized intermediate are deleted. The full-VFR test's byte-identical duplicate-frame cap is retired with cause: the fixture has no source frames for 40% of its timeline, so held frames are correct; the two-pass path only scored under it because x264 encoder noise made frozen frames hash differently. The freeze regression (missing frames) stays pinned by the frame-count windows. * docs(engine): pin vfrPreflightMs definition change after one-pass VFR vfrPreflightMs used to time a per-source VFR-to-CFR re-encode; it now times only the cached classification probe and collapses to ~0. Call that out on ExtractionPhaseBreakdown so dashboards keyed on the old threshold semantics migrate to vfrPreflightCount / extractMs. * fix(engine): bump extraction cache schema to v3 for one-pass VFR frames One-pass VFR extraction changes frame CONTENTS for VFR sources while the cache key tuple (path, mtime, size, trim, fps, format) is unchanged, so warm v2 entries holding two-pass frames would keep being served across the deploy boundary. Bumping the schema prefix makes v2 entries inert; affected sources re-extract once. * perf(engine): dedupe identical extractions within one render N <video> elements sharing (resolved path, mediaStart, duration, fps, format) extracted N times; they now share one extraction via an in-flight promise map keyed on that tuple. Duplicate elements receive the shared frame set under their own videoId. This also removes a race where two identical clips on a cache miss wrote the same extraction-cache entry dir concurrently. 3x duplicated 60s 1080p video: 4426ms to 1521ms in the A/B benchmark, one frame set on disk. * fix(engine): attribute shared-extraction failures to the dedupe leader When a deduped extraction fails, every follower reported the leader's error verbatim under its own videoId, reading as N independent failures in traces. Follower errors now carry a '[shared extraction, leader <id>]' prefix so the fan-out is traceable to one root failure. |
||
|
Miguel Ángel
|
7860583341 |
perf(engine): one-pass VFR extraction with -fps_mode cfr (#1899)
* perf(engine): write PNG frames at compression_level 1 Extracted video frames are render-scoped temp files read once during capture, so zlib effort above level 1 buys nothing. Measured 3.3x faster on 60s of 1080p H.264 to PNG (11.4s to 3.5s) and 5.4x on a 20s vp9-alpha webm (4.3s to 0.79s), for ~14% larger temp files. * perf(engine): one-pass VFR extraction with -fps_mode cfr VFR sources (screen recordings, phone videos) were re-encoded to CFR with libx264 and then extracted in a second ffmpeg pass. Extraction now runs a single pass with -fps_mode cfr -r <fps>. Same frame counts on the VFR regression fixtures (120/120 mid-seek, 297-303 full file), one less x264 generation of quality loss, ~3.4x faster on VFR inputs. convertVfrToCfr and the _vfr_normalized intermediate are deleted. The full-VFR test's byte-identical duplicate-frame cap is retired with cause: the fixture has no source frames for 40% of its timeline, so held frames are correct; the two-pass path only scored under it because x264 encoder noise made frozen frames hash differently. The freeze regression (missing frames) stays pinned by the frame-count windows. * docs(engine): pin vfrPreflightMs definition change after one-pass VFR vfrPreflightMs used to time a per-source VFR-to-CFR re-encode; it now times only the cached classification probe and collapses to ~0. Call that out on ExtractionPhaseBreakdown so dashboards keyed on the old threshold semantics migrate to vfrPreflightCount / extractMs. * fix(engine): bump extraction cache schema to v3 for one-pass VFR frames One-pass VFR extraction changes frame CONTENTS for VFR sources while the cache key tuple (path, mtime, size, trim, fps, format) is unchanged, so warm v2 entries holding two-pass frames would keep being served across the deploy boundary. Bumping the schema prefix makes v2 entries inert; affected sources re-extract once. |
||
|
Miguel Ángel
|
92385711dc |
fix(engine): hold last frame when a clip's media is shorter than its slot (#1726)
Renders showed the page background (a one-frame black flash) right before a cut when a video clip's source media was a hair shorter than its data-duration slot — the common case, since `ffmpeg -t 1.45` emits 43 frames = 1.433s at 30fps. The frame lookup only held the last frame at the exact clip end, so the sub-frame remainder rendered blank. - Hold the last extracted frame for the rest of the slot once the source is exhausted, within a tolerance floored at the compiler's 0.05s clamp epsilon so the seam is covered at any fps (2 frames alone is < 0.05s above 40fps). Clips deliberately much shorter than their slot still blank for the tail (unchanged). - Warn when the compiler clamps a video's data-duration down to its media length (slot longer than source by more than the clamp epsilon): a render-time `[compile]` warning in the producer, plus a matching `validate` warning that reads each <video>'s live duration in headless Chrome (static HTML lint can't see media durations). A shared `analyzeClipMediaFit` keeps both on one threshold. Adds engine unit tests for the hold behavior and the analyzer. |
||
|
|
fdb8f33fc0 |
fix(engine): hold the last video frame at the inclusive clip end (#1564)
* fix(engine): hold the last video frame at the inclusive clip end The frame-lookup active set deactivated a video on an exclusive end-bound (globalTime < end), while the runtime keeps an element visible through currentTime <= end (core/runtime init.ts). The rendered frame landing exactly on a clip's end went blank even though the runtime still showed the element on its final frame: one blank frame at the end of every clip whose end lands on a frame boundary. Make the active window inclusive of the end to match the runtime, and at t === end serve the last extracted frame (the runtime holds the element's final frame there too). Mid-clip source exhaustion (t < end) stays blank, unchanged. * fix(engine): align getFrame boundary to match refreshActiveSet Make getFrame's end-bound inclusive (> instead of >=) for consistency with the refreshActiveSet changes. getFrame is currently unused externally but should match the same contract. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Carlos Alcaraz <193642530+calcarazgre646@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
36b24acf20 |
feat: add video frame format render option (#1481)
* feat: add video frame format render option * refactor: single source of truth for video-frame-format allow-list Addresses PR review (Via) on #1481: the ["auto","jpg","png"] set was declared three times — render.ts (VIDEO_FRAME_FORMATS), server.ts (inline includes), and renderConfigValidation.ts (ALLOWED_VIDEO_FRAME_FORMATS) — three boundaries to update when a new extraction format lands. Hoist the constant + a reusable `isVideoFrameFormat` type guard into @hyperframes/engine (where VideoFrameFormat is defined) and route all three call sites through them. Behavior unchanged; also drops two `as RenderConfig[...]` casts in favor of the guard (narrowing over assertion, per repo TS conventions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Xuelong Mu <xuelongmu@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Miguel Ángel
|
3a72aa528d |
fix(engine): add epsilon to frame index floor to prevent IEEE 754 boundary duplicates (#1318)
## Summary Fixes #1317 — systematic duplicate+skip video frames when clip `data-start` is aligned to the output frame grid. ### Root cause `Math.floor(localTime * fps)` in `getFrameAtTime` produces off-by-one errors when the product lands exactly on an integer boundary due to IEEE 754 float noise. For example, `0.28 * 25 === 6.999999999999999` instead of `7`, causing `Math.floor` to return 6 (duplicate of previous frame) instead of 7. ### Fix 1. Add `1e-9` epsilon before flooring: `Math.floor(localTime * fps + 1e-9)` — nudges boundary values like `6.999999` to `7.000000` without affecting mid-frame values. 2. Include `mediaStart` in the frame index computation so trimmed clips (`data-media-start`) map to the correct extracted frames. Both call sites fixed: `getFrameAtTime()` (public API) and the `FrameLookupTable.getFramesAtTime()` bulk lookup. ### Reporter's measurements (before fix) | Case | Duplicates (of 351 frames) | |---|---| | Source file | 1 | | data-start="0" | 14 | | data-start="230.44" (production) | 127 | | data-start="0.02" (half-frame offset workaround) | 1 | ## Test plan - [x] 4 new regression tests for IEEE 754 boundary precision - [x] No duplicate frames when data-start is grid-aligned (25fps) - [x] Monotonically increasing frame indices across 100 frames - [x] Correct frame at the `0.28 * 25` boundary (frame 7, not 6) - [x] `mediaStart` correctly offsets frame index - [x] Typecheck clean |
||
|
Miguel Ángel
|
526709cad2 | fix(cli): handle encoded lint asset paths | ||
|
Miguel Ángel
|
7ad10a2dff | fix(engine): resolve encoded media src paths | ||
|
Miguel Ángel
|
e87f5bb769 |
fix(engine): widen VFR test frame count tolerance for cross-platform FFmpeg
FFmpeg's VFR-to-CFR normalization produces slightly different frame counts across versions due to timestamp rounding in the fps filter. The ±1 tolerance was too tight for Linux FFmpeg builds. Widen to ±3 frames — still catches the 25% shortfall regression these tests guard against. |
||
|
JamesandClaude Opus 4.7
|
f4ecf96918 |
fix(engine,cli,producer): address PR #627 review feedback
- engine/chunkEncoder, engine/streamingEncoder: extend `-bf 0` to GPU h264
paths (nvenc, qsv, vaapi) and `-b_strategy 0` for qsv so GPU-encoded
outputs avoid negative-DTS freezes too — not just SW libx264.
- engine/videoFrameExtractor: detect mid-path traversal (e.g.
`assets/../../foo.mp4`) by normalizing first and re-anchoring at the
project root. Adds a regression test.
- engine/videoFrameExtractor: dedupe stderr "src not resolvable" warnings
by `video.src` so a comp with N broken sources logs once, not N times.
- engine/videoFrameExtractor.test: drop dynamic `require("node:fs")`,
use ES `import { writeFileSync } from "node:fs"`.
- engine/ffprobe: extract `readTagCI` helper for case-insensitive ffprobe
tag reads (will recur for other libavformat-versioned sidecar tags).
- cli/background-removal/pipeline: collapse Quality / QUALITIES /
QUALITY_CRF / DEFAULT_QUALITY / isQuality surface using
`Quality = keyof typeof QUALITY_CRF`.
- producer/renderOrchestrator: replace `v.src.startsWith("/")` with
`isAbsolute(v.src)` in the HDR probe path so Windows absolute paths
(`C:\...`) aren't treated as relative — matches the audioMixer guard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
JamesandClaude Opus 4.7
|
39bc3749b4 |
fix(engine): default to codec-based alpha capability instead of relying on tags
Tag-based alpha detection (alpha_mode / ALPHA_MODE / pix_fmt yuva*) is fundamentally brittle. Failure modes seen in the wild: - case-sensitivity across ffmpeg versions (alpha_mode vs ALPHA_MODE) - older muxers that omit the sidecar tag entirely - mp4-as-webm rewraps that drop the tag - ffprobe reporting yuv420p for VP9-with-alpha because the alpha plane lives in a Matroska BlockAdditional sidecar, not the main pix_fmt Each of those silently strips alpha at extraction time. The bug doesn't surface until the rendered output is missing layers — frustrating to debug, silent in stdout. The previous case-insensitive fix patched one of the failure modes; this commit removes the class. The robust alternative is codec-based: any bitstream that CAN carry alpha (VP9, VP8, ProRes 4444) gets the alpha-aware decoder and PNG output by default, regardless of what the tag says. The cost is a small file-size increase on opaque VP9/VP8 sources (cached PNGs vs JPGs); the benefit is no class of silent alpha loss from tag misdetection. - Adds codecMayHaveAlpha() + decoderForCodec() helpers and exports them. - Updates extractVideoFramesRange to force libvpx-vp9 / libvpx for VP9 / VP8 unconditionally (was: only when metadata.hasAlpha). - Updates resolveFrameFormat to default to PNG for any alpha-capable codec (was: only when metadata.hasAlpha). - +4 unit tests covering the codec table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
2f96d5c7ab |
fix(engine,producer): URL-clamp sub-comp src paths and warn on silent extraction misses
A <video src='../assets/foo.mp4'> inside a sub-composition silently dropped from extraction; the rendered output froze on the first decoded frame for the entire clip, with no error in stdout. Root cause: browser URL resolver clamps '..' at origin root (studio preview loads fine), but path.join(projectDir, '../assets/foo.mp4') normalizes to parent-of-project/assets/foo.mp4, which usually doesn't exist. existsSync returns false, extraction is skipped, no frame lookup is built, the per-frame injector has nothing to swap, and the <video> element's first decoded frame paints every screenshot. - Adds resolveProjectRelativeSrc in videoFrameExtractor that mirrors browser clamping (literal join first, then leading '..' stripped). - Surfaces a loud stderr warning when the resolver misses. - Mirrors fix in audioMixer.ts (same bug for <audio src='../'>) and renderOrchestrator HDR probe loop. - +6 regression tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Miguel Ángel
|
31e8144304 |
fix: render parity for transparent looped videos (#478)
## Summary - preserve alpha for render-injected video frames by detecting alpha streams with ffprobe and extracting alpha video frames as PNG - keep `<video loop>` semantics through static parsing, compiler duration resolution, browser media discovery, and render frame lookup - fail embedded preview startup before opening a broken browser page when the Studio bundle is missing - align snapshot frame injection with looped media timing and VP9 alpha extraction ## Why The Studio preview and rendered MP4 could disagree for timed transparent looped videos. The Comfy funding composition exposed two separate parity bugs: render-injected frames needed alpha-preserving PNG extraction, and the compiler was clamping a looped `data-duration="4"` video down to the 3.125s source duration. After the first source cycle, render lookup treated the video as inactive, hid the native video, and produced the blank polygon/glow the user saw around the rounded `0:03` mark. `hyperframes lint` and `hyperframes validate` did not catch this because they check syntax/load/console/accessibility, not preview-vs-render visual parity. This PR adds regression coverage for the compiler loop-duration path and frame lookup path. ## Verification - `bun run --filter @hyperframes/core test -- src/compiler/timingCompiler.test.ts src/compiler/htmlCompiler.test.ts` - `bun test packages/producer/src/services/htmlCompiler.test.ts` - `bun run --filter @hyperframes/engine test -- videoFrameExtractor ffprobe` - `bun run --filter @hyperframes/core typecheck` - `bun run --filter @hyperframes/engine typecheck` - `bun run --filter @hyperframes/producer typecheck` - `bun run --filter @hyperframes/cli typecheck` - `bun run lint` - `bun run format:check ...` on touched files - Comfy project: `node packages/cli/dist/cli.js validate` -> no console errors, 44 text elements pass WCAG AA - Comfy project patched render from source: `/tmp/comfy-render-compare/fixed6-comfy.mp4`, 1920x1080, 30fps, 21.8s, 654 frames - 3.00s-3.97s render contact sheet: `/tmp/comfy-render-compare/fixed6-window-contact.png` - targeted fixed render capture at 3.733s: `/tmp/comfy-render-compare/probe-capture-fixed/captured/frame_000112.jpg` - agent-browser Studio proof screenshot at 3.7s: `/tmp/comfy-render-compare/agent-browser-studio-3_7-fixed.png` - agent-browser-driven recording of 3s seek pass: `/tmp/comfy-render-compare/agent-browser-wysiwyg-3s-fixed.webm` Note: `bun run --filter @hyperframes/cli dev -- validate` is blocked in source mode by the existing `contrast-audit.browser.js` default-export loader issue; packaged `node packages/cli/dist/cli.js validate` passes for this project. |
||
|
James Russo
|
57cdf7d80e |
perf(engine): content-addressed extraction cache for video frames (#446)
## What Adds a content-addressed cache for extracted video frames, keyed on the tuple `(path, mtime, size, mediaStart, duration, fps, format)`. Repeat renders of the same composition (studio edit → re-render, preview → final) skip the ffmpeg extraction entirely. ## Why Video frame extraction is the dominant non-capture phase for video-heavy compositions. Studio iteration workflows extract the same frames over and over — each render burns ffmpeg time that adds no value. Validated on `/tmp/hf-fixtures/cfr-sdr-cache`: ``` Cold (miss): extractMs=69, videoExtractMs=70, totalElapsedMs=2052 Warm (hit): extractMs=1, videoExtractMs=2, totalElapsedMs=1964 cacheHits: 0→1, cacheMisses: 1→0 ``` The fixture is tiny (3s CFR SDR @ 30fps), so the wall-clock delta is small; the extraction-time delta (69→1ms, 98%) scales linearly with source length. For heavy-iteration workflows (a user rendering the same composition while tuning encoding params), extraction time goes to zero on every repeat render. Depends on #444 (instrumentation surface) and #445 (segment-scope HDR preflight — otherwise cache keys would be unstable across renders on mixed-HDR compositions). ## How - New `packages/engine/src/services/extractionCache.ts`: - SHA-256 key over a stable JSON encoding of `(path, mtime_ms, size, mediaStart, duration, fps, format)`. Infinity duration is normalized to `-1` so unresolved natural-duration sources still produce stable keys. - Truncates to 16 hex chars in the entry directory name — 64 bits of entropy is plenty at cache scale and keeps `ls` output short. - `hfcache-v2-` schema prefix — bumping it invalidates old entries (callers own gc policy; the cache owns keys). - `.hf-complete` dotfile sentinel. An entry dir without the sentinel is treated as a miss (covers crash-mid-extract and abandoned writes); the next render re-extracts over the partial frames with `-y`. - `FRAME_FILENAME_PREFIX = "frame_"` shared with the extractor — future refactors only need to touch one place to rename frames. - `EngineConfig.extractCacheDir` (env: `HYPERFRAMES_EXTRACT_CACHE_DIR`) gates the feature. Undefined disables caching — extraction runs into the render's workDir and cleanup removes it on render end, preserving the prior behaviour exactly. No default root is chosen by the engine; the caller (CLI, app, studio) owns the location policy. - `ExtractedFrames.ownedByLookup` flag prevents `FrameLookupTable.cleanup` from rm'ing a shared cache dir at render end. Set to `true` on both hits and misses (misses own the directory they wrote into, but hand it over to the cache rather than deleting it). - Phase 3 extractor flow: 1. Snapshot `(videoPath, mediaStart, start, end)` per resolved video BEFORE Phase 2a/2b preflight mutates them — so cache keys are stable across renders that use workDir-local normalized files (those files have fresh mtimes every render). 2. Compute key, `lookupCacheEntry`. 3. On hit: rebuild `ExtractedFrames` from the cache dir plus the Phase 2-probed `VideoMetadata` — no re-ffprobe. 4. On miss: `ensureCacheEntryDir`, extract with `extractVideoFramesRange(..., outputDirOverride)`, then `markCacheEntryComplete` (the sentinel write is the last step so a crash leaves the dir un-sentineled). - `extractVideoFramesRange` gains an `outputDirOverride` parameter so cache-miss writes land directly in the keyed dir (no `join(outputDir, videoId)` wrapping). ## Test plan - [x] 19 unit tests in `extractionCache.test.ts` covering key determinism, mtime/size invalidation, format/fps/mediaStart/duration invalidation, Infinity normalization, sentinel semantics, missing-file tolerance - [x] 2 integration tests in `videoFrameExtractor.test.ts`: - "reuses extracted frames on a warm cache hit" — asserts `cacheHits=1`, `extractMs<50ms` on second call against a CFR SDR fixture - "invalidates the cache when fps changes" — different fps on second call forces a new miss - [x] End-to-end validation with `HYPERFRAMES_EXTRACT_CACHE_DIR` set, two runs of the same fixture - [x] Lint + format (oxlint + oxfmt) - [x] Typecheck (engine + producer) |
||
|
James Russo
|
9912d3730a |
perf(engine): segment-scope SDR→HDR preflight (#445)
## What Scopes the SDR→HDR preflight re-encode to the segment the composition actually uses, mirroring the existing VFR→CFR segment-scope fix. ## Why `convertSdrToHdr` was re-encoding entire source files, so a 30-minute SDR screen recording contributing a 2-second clip in a mixed HDR/SDR composition ate multi-second preflight time that produced frames no one would ever read. Validated on a mixed 30s-SDR + 2s-HDR fixture: `hdrPreflightMs` drops **87%** (1162→148ms), `videoExtractMs` drops **82%** (1272→231ms), `tmpPeakBytes` drops **45%** (8.2MB→4.5MB). Depends on #444 (phase-level instrumentation) for the measurement surface. ## How - `convertSdrToHdr` gains `startTime` and `duration` parameters ahead of the upstream `targetTransfer` arg added by #370. New signature: `convertSdrToHdr(input, output, startTime, duration, targetTransfer, signal, config)`. `-ss $start -t $duration` is added to the ffmpeg args. - Phase 2 now captures the full `VideoMetadata` per `resolvedVideos` entry (previously just `colorSpace`) so the caller can compute `segDuration` from `video.end - video.start` with a fallback to `metadata.durationSeconds - video.mediaStart` for unbounded (Infinity) clips — without firing another ffprobe. - After a successful convert, `entry.video.mediaStart` is zeroed out via shallow-copy (doesn't mutate the caller's `VideoElement`) so downstream extraction seeks from 0 instead of the original offset. Mirrors what the VFR→CFR path already does. ## Test plan Validation on `/tmp/hf-fixtures/hdr-sdr-mixed-scope`: ``` hdrPreflightMs: >1000 → 150 (gate: <300) ✓ videoExtractMs: 1272 → 237 (-82%) tmpPeakBytes: 8.2MB → 4.5MB (-45%) ``` - [x] Unit test: new regression test synthesizes 10s SDR + 2s HDR fixture inline and asserts the converted file's duration matches the 2s used segment (pre-fix matched the 10s source) - [x] Lint + format - [x] Typecheck - [x] Manual perf validation against synthesized fixture |
||
|
James Russo
|
31354d52da |
perf(engine): extraction-phase instrumentation (#444)
## What Adds per-phase timings and counters to `extractAllVideoFrames` and surfaces them on the producer's `RenderPerfSummary` as `videoExtractBreakdown` alongside a new `tmpPeakBytes` workDir size sample. ## Why Phase 2 video extraction has five distinct sub-phases (resolve, HDR probe, HDR preflight, VFR probe, VFR preflight, per-video extract) and today they collapse into a single `videoExtractMs` stage timing. That makes every subsequent perf PR in this stack immeasurable — you can't tell whether a win came from cache hits, preflight scope reduction, or pure extraction speed. This PR is foundational for PR #445 (segment-scope HDR preflight) and PR #446 (content-addressed extraction cache). ## How - New `ExtractionPhaseBreakdown` type with `resolveMs`, `hdrProbeMs`, `hdrPreflightMs/Count`, `vfrProbeMs`, `vfrPreflightMs/Count`, `extractMs`, `cacheHits`, `cacheMisses`. Populated inline with `Date.now()` wrappers — overhead is sub-millisecond on every phase. - Returned on `ExtractionResult.phaseBreakdown`. - Producer extends `RenderPerfSummary` with `videoExtractBreakdown?: ExtractionPhaseBreakdown` and `tmpPeakBytes?: number`. `tmpPeakBytes` is sampled from the workDir right before cleanup via a new recursive-size helper that swallows errors (purely observational — a missing workDir must never fail the render). No changes to the capture-lifecycle resource tracking — earlier versions of this instrumentation plumbed injector LRU stats through `RenderOrchestrator`, which conflicted hard with upstream #371 (`buildHdrCaptureOptions` refactor). Dropped that piece for a marginal observability loss. ## Test plan Validation on `packages/producer/tests/vfr-screen-recording`: ```json "videoExtractBreakdown": { "resolveMs": 0, "hdrProbeMs": 0, "hdrPreflightMs": 0, "hdrPreflightCount": 0, "vfrProbeMs": 0, "vfrPreflightMs": 166, "vfrPreflightCount": 1, "extractMs": 97, "cacheHits": 0, "cacheMisses": 0 }, "tmpPeakBytes": 4578598 ``` Total elapsed within noise of pre-PR baseline (2665 → 2673 → 3228ms across hosts). - [x] Unit test: phase-breakdown assertion added to `videoFrameExtractor.test.ts` - [x] Lint + format (oxlint + oxfmt) - [x] Typecheck (engine + producer) - [x] Manual perf validation against VFR fixture |
||
|
Vance Ingalls
|
cc9403b6bd |
test(producer): extract frameDirMaxIndexCache to its own module and pin cross-job isolation (#381)
## Summary Extract the `frameDirMaxIndexCache` from a private module-scoped Map inside `renderOrchestrator.ts` into its own `frameDirCache.ts` module, then add a 11-test bun:test suite that pins the cross-job isolation contract added in Chunk 5B. ## Why `Chunk 9E` of `plans/hdr-followups.md`. The cache lived as a private Map inside `renderOrchestrator.ts`, which made the cross-job isolation contract from Chunk 5B impossible to unit-test directly. Extracting it both makes the contract testable and reduces orchestrator complexity slightly. ## What changed - New `packages/producer/src/services/frameDirCache.ts` exposes `getMaxFrameIndex` / `clearMaxFrameIndex` / `getMaxFrameIndexCacheSize` (plus a test-only `__resetMaxFrameIndexCacheForTests` helper). Behavior is unchanged: callers still get the same module-scoped sharing inside a job, and `renderOrchestrator`'s outer `finally` still clears every entry it registered so the cache cannot grow monotonically across renders. - `renderOrchestrator.ts`: imports the new helpers, drops the unused `readdirSync` import, updates inline comments, and replaces two `frameDirMaxIndexCache.delete` sites with `clearMaxFrameIndex`. - New `frameDirCache.test.ts` (bun:test, 11 tests) covering: - Reading the max index from a populated directory. - Ignoring filenames that don't match `frame_NNNN.png` (wrong ext, wrong prefix, wrong case, double extension, empty index group, same-named subdirectory). - Empty- and missing-directory paths returning `0` and being cached. - Intra-job invariant: subsequent readdir mutations not observed once cached. - `clearMaxFrameIndex` forcing a re-read; returns `false` for paths that were never cached. - Per-directory isolation when multiple directories are registered. - The cross-job contract from Chunk 5B: cache empty between well-behaved jobs, doesn't grow monotonically across 20 simulated renders with 3 HDR videos each (steady-state cache size stays at 3), and a buggy job that forgets to clear leaks exactly its own entries rather than affecting unrelated jobs. ## Test plan - [x] `frameDirCache.test.ts` 11/11 pass. - [x] Existing producer tests unchanged. - [x] Behavior preserved: same module-scoped sharing inside a job, same outer-`finally` eviction. ## Stack Chunk 9E of `plans/hdr-followups.md`. Test-driven extraction; complements Chunk 5B. |
||
|
James RussoandClaude Opus 4.7
|
ffc06827c4 |
fix(engine): auto-normalize VFR video inputs to CFR before frame extraction (#360)
* fix(engine): auto-normalize VFR video inputs to CFR before frame extraction
Screen recordings (macOS ScreenCaptureKit, QuickTime, phone videos) are
commonly variable-frame-rate. When such inputs hit the extractor's
`-ss <start> -i <video> -t <dur> -vf fps=N` pipeline, the fps filter
can emit fewer frames than requested — for a 4-second 30fps segment
starting mid-file, the output was ~90 frames instead of 120.
`FrameLookupTable.getFrameAtTime` returns null for out-of-range indices,
so the compositor held the last valid frame and the user perceived the
video as freezing. This matches the bug report from an X community post
where a user said "all of them freezes" on their screen recording scenes.
The engine already detects VFR via `metadata.isVFR` in ffprobe.ts but
never acted on it — the compiler only logged a warning. This change
mirrors the existing SDR→HDR normalization pattern: when a source is
detected as VFR, re-encode only the used segment with
`-fps_mode cfr -r <fps> -preset fast -crf 18` before extraction.
Scoping the re-encode to `[mediaStart, mediaStart+duration]` means a
30-second clip cut from a 60-minute screen recording pays ~1s of
transcode cost, not 18s. Benchmarked locally:
Baseline (current): 32-39% duplicate frames, 25% frame-count
shortfall on mid-file segments.
Tier 1 (flag changes only): ~same — fps filter issue is not flag-fixable.
Tier 2 (CFR preflight): 1.7-6% duplicate frames, correct frame
count in every scenario tested.
The compiler warning that previously told users to manually re-encode
is downgraded to `console.info` since the engine now handles it.
— Rames Jusso
* refactor(engine): clean up VFR normalization loop after review
- Drop the `vfrNormDirCreated` flag; `mkdirSync({recursive:true})` is
idempotent and cheap.
- Don't re-wrap the `VFR→CFR conversion failed` prefix — `convertVfrToCfr`
already throws a message with that label; adding it again in the catch
produced "VFR→CFR conversion failed: VFR→CFR conversion failed (exit 1)".
- Shorten the Phase 2b header comment; the function docstring above
`convertVfrToCfr` already explains the failure modes and rationale.
- Note which frame windows the VFR fixture's select filter drops so the
magic numbers are scannable.
No behavior change; 311/311 engine tests still pass.
— Rames Jusso
* test(engine): add VFR regression unit tests
Adds a describe block that synthesizes a VFR fixture via ffmpeg and asserts
the extractor produces the expected frame count (no shortfall) and no long
runs of duplicate frames — the user-visible "frozen screen recording"
symptom. Covers both a mid-file segment and the full-file case.
Guarded with describe.skipIf(!HAS_FFMPEG) because the CI Test job on
ubuntu-24.04 and the Windows test-windows job don't install ffmpeg. The
producer-level regression test in packages/producer/tests/vfr-screen-recording/
runs inside Dockerfile.test (which has ffmpeg) and is the primary CI signal
for this bug; these unit tests are supplementary coverage for local and
any ffmpeg-equipped CI environment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(producer): add vfr-screen-recording regression test
End-to-end CI regression coverage for PR #360 via the existing
regression-harness: renders a 3s composition containing a real macOS
ScreenCaptureKit clip (r_frame_rate=120, avg≈36fps) seeked to
mediaStart=1, then PSNR-compares against a committed output.mp4.
Fixture src/clip.mp4 (108 KB) is a 5-second excerpt downscaled to 480×332
with -fps_mode passthrough to preserve the VFR timestamps. Content is the
public hyperframes OSS repo root page — see NOTICE.md for provenance.
With the fix applied, all 100 PSNR checkpoints pass. With the fix reverted,
66 of 100 fail (PSNR drops from ~43 dB to ~20 dB in the duplicate-frame
windows). Tagged "regression,video,vfr" so it runs in the fast shard
of .github/workflows/regression.yml automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(producer): regenerate vfr-screen-recording baseline in Docker
The committed golden output.mp4 was initially rendered on the host machine;
CI runs the renderer inside Dockerfile.test with a different Chrome +
ffmpeg build, producing pixel-level drift that failed PSNR at 54/100
checkpoints (~20 dB vs 41 dB in the VFR sparse-content windows). Both
renders are valid — the VFR source has inherent sampling ambiguity in
static segments, and different Chrome/ffmpeg builds make different valid
choices.
Regenerated the baseline via `bun run docker:test:update vfr-screen-recording`
so it matches the Docker environment CI actually uses. Matches the flow
the existing sub-composition-video, hdr-pq, etc. baselines were captured
with.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: document that producer test baselines must be captured in Docker
Hit this 2026-04-21 with the vfr-screen-recording regression test:
host-generated output.mp4 baseline tripped 54/100 PSNR checkpoints in CI
because Chrome + ffmpeg drift between the host and Dockerfile.test.
Document the `bun run --cwd packages/producer docker:test:update <name>`
flow so future contributors don't repeat the mistake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Vance Ingalls
|
00af29c169 |
fix(cli): forward --hdr through Docker render + HDR docs (#346)
## Summary This PR ended up covering the full HDR Docker/docs follow-through plus the producer/engine work needed to make HDR still images render and regress correctly in CI. The branch now does four things: - forwards `--hdr` through the Docker render path in the CLI - adds and expands HDR documentation across the docs site - adds first-class HDR still-image support to the engine/producer pipeline - adds targeted HDR regression coverage, including a CI-safe fallback for PNG HDR metadata detection when `ffprobe` does not expose PNG color tags ## What changed ### CLI and docs - `hyperframes render --docker --hdr` now preserves `--hdr` when invoking the in-container CLI - added a dedicated HDR guide and linked it from CLI, producer, engine, rendering, and common-mistakes docs - documented HDR constraints and verification flow: HDR source requirements, MP4/H.265 Main10 output, PQ/HLG handling, Docker usage, and common SDR fallback causes ### Engine and producer HDR image support - added `ImageElement` support to the engine composition model and parsing path - threaded image elements through producer compilation and orchestration - probed image sources for HDR color spaces so image-only compositions can trigger HDR output without requiring an HDR video source - included HDR image start times in stacking queries so the layered compositor can place images correctly in z-order - integrated HDR image compositing into the layered HDR render loop alongside native HDR video layers and SDR DOM overlays - forced screenshot mode for HDR layered compositing where required to keep DOM/HDR layer composition deterministic - skipped readiness waiting for natively extracted HDR videos in the engine path where it was unnecessary and could block layered HDR flows ### HDR metadata robustness - added a fallback in `extractVideoMetadata()` to read PNG `cICP` metadata directly when `ffprobe` omits color-space fields for PNGs - this specifically fixes CI/Docker detection for the `hdr-image-only` fixture, where the render was falling back to SDR because the PNG was not being recognized as BT.2020 PQ ### Regression coverage and fixture cleanup - added `hdr-image-only`, a regression fixture that validates HDR still-image rendering end to end - added `hdr-pq`, a focused HDR PQ regression fixture for the video path - updated regression CI to run an `hdr` shard with `--sequential hdr-pq hdr-image-only` - removed the older larger `hdr-regression/*` fixture set in favor of the smaller targeted regressions used by CI - added the necessary fixture generation/readme material and checked-in golden outputs for the new HDR tests ## Why The original PR description only covered the CLI flag forwarding and docs work. Since then, the branch also picked up the missing runtime support needed for HDR still images and the regression coverage to keep that path from breaking. The practical issue this closes is: - local host runs could pass while CI failed `hdr-image-only` - the failure was a full-frame visual mismatch caused by SDR fallback, not unstable rendering - root cause was PNG HDR metadata not being surfaced by `ffprobe` in the CI Docker environment - parsing the PNG `cICP` chunk directly makes HDR detection deterministic across environments ## Test plan ### Local targeted checks ```bash bunx oxlint packages/engine/src/utils/ffprobe.ts packages/engine/src/utils/ffprobe.test.ts bunx oxfmt packages/engine/src/utils/ffprobe.ts packages/engine/src/utils/ffprobe.test.ts bun --cwd packages/engine test src/utils/ffprobe.test.ts src/utils/hdr.test.ts ``` ### Producer regression runs on host ```bash bun run --cwd packages/core build:hyperframes-runtime:modular bun --cwd packages/producer test -- --sequential --exclude-tags slow,render-compat,hdr bun --cwd packages/producer test -- --sequential hdr-pq hdr-image-only ``` Observed result: - `fast` shard: 7 passed, 0 failed - `hdr` shard: 2 passed, 0 failed ### CI-equivalent Docker verification ```bash docker build -f Dockerfile.test -t hyperframes-producer:test . docker run --rm \ --security-opt seccomp=unconfined \ --shm-size=4g \ -v "$PWD/packages/producer/tests:/app/packages/producer/tests" \ hyperframes-producer:test \ --sequential hdr-pq hdr-image-only ``` Observed result: - `hdr-image-only`: passed - `hdr-pq`: passed - shard summary: 2 passed, 0 failed ### Specific regression fixed Before the PNG `cICP` fallback, the Docker/CI run failed `hdr-image-only` with: - missing `"[Render] HDR source detected — output: PQ ..."` log line - full-frame visual mismatch across all 100 checkpoints - PSNR ~17 on every frame, indicating a consistent SDR-vs-HDR pipeline mismatch After the fallback, the same Docker path recognizes the PNG as HDR and the shard passes. |
||
|
Miguel Ángel
|
110ea12597 |
fix(core,engine,producer): handle id-less media in sub-composition renders (#96)
## What Move the id-less media fix into the shared timing compiler so producer can resolve durations for sub-composition videos before inlining, then carry the merged result through engine parsing, regression coverage, and the regression Docker image used in CI. This PR now does five concrete things: - assigns stable ids to id-less media in core `compileTimingAttrs()` so unresolved duration injection can target them - keeps the engine-side `parseVideoElements()` support for `video[src]` plus the newer `data-duration` / natural-duration fallback from `main` - makes producer prefer sub-composition media metadata over the later inlined-document parse when the same media id appears in both places - makes `sub-composition-video` a runnable regression test by fixing its metadata and checking in the missing `output/compiled.html` snapshot - removes the stale `pnpm-workspace.yaml` copy step from `Dockerfile.test`, so regression CI builds the Bun-based test image from the current workspace layout ## Why - media without an explicit `id` could not participate in unresolved-duration resolution early enough - producer could lose the resolved sub-composition timing by overwriting it with the later inlined parse - the regression fixture intended to cover this case was not actually running in CI because its `meta.json` was incomplete and the required compiled snapshot was missing - the regression image definition still expected a deleted `pnpm-workspace.yaml`, so GitHub Actions failed before the test shard could start Putting the id-generation step in core makes the behavior reusable instead of relying on producer-only HTML patching. ## How ### Shared compiler - core `compileTimingAttrs()` now auto-assigns stable ids to id-less `video` / `audio` tags - those generated ids are returned in `unresolved`, so `injectDurations()` can add `data-duration` and `data-end` to the same media element later in the pipeline - added core tests that cover auto-id assignment and duration injection for generated ids ### Producer - when producer combines `subVideos` / `subAudios` with the media re-parsed from the final inlined HTML, it now lets the sub-composition metadata win - this preserves the resolved/clamped timing already computed for nested media instead of overwriting it with the later parse - `sub-composition-video` now has valid regression metadata and a checked-in `output/compiled.html` snapshot so CI actually executes it ### Engine - resolved the merge conflict in `videoFrameExtractor` by keeping the broader `video[src]` parsing from this branch and the `data-duration` / natural-duration fallback that landed on `main` - added a focused engine unit test for videos without ids ### CI image - `Dockerfile.test` now copies only `package.json` and `bun.lock` at the workspace root before `bun install --frozen-lockfile` - this matches the current monorepo layout and removes the obsolete pnpm-era dependency on `pnpm-workspace.yaml` ## Test plan - [x] `bun run --filter @hyperframes/core test` - [x] `bun run --filter @hyperframes/engine test` - [x] `bun run --filter @hyperframes/producer test --update --sequential sub-composition-video` - [x] `bun run --filter @hyperframes/producer test --sequential sub-composition-video` - [x] Browser check with `agent-browser` against the compiled fixture page (`http://127.0.0.1:8123/compiled.html`) - [x] Clean tracked-only Docker build of `Dockerfile.test` with the PR version of the file applied ## Notes - Latest regression workflow is green on `main`, but before this PR the `sub-composition-video` fixture was being skipped by the harness rather than exercised end to end. - The CI Docker fix was validated from a tracked-only export to avoid local untracked worktree artifacts affecting the result. |