mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
a739be58b1e66e7cda202efceff9846c8bd9d993
717
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ukimsanovandClaude Opus 4.6
|
a8f0752bf6 |
fix(shader-transitions): address review — NaN guard, canvas sync, shared defaults
- Validate data-width/data-height: fall back to defaults if NaN or <= 0 - Sync existing #gl-canvas dimensions on reuse (if init called twice) - Import DEFAULT_WIDTH/DEFAULT_HEIGHT in capture.ts instead of hardcoding 1920/1080 in parameter defaults Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
ukimsanovandClaude Opus 4.6
|
c81ef50e2f |
feat(shader-transitions): support any aspect ratio + update skill
Add dynamic resolution to shader transitions and update Skeleton A to use shaders on vertical compositions. shader-transitions changes: - webgl.ts: read dimensions from params instead of hardcoded constants - capture.ts: accept width/height for html2canvas - hyper-shader.ts: read data-width/data-height from composition root skill + docs changes: - Skeleton A now has 1 shader at hero reveal (s3→s4 midpoint) - Removed "no shaders on vertical" limitation from docs - Updated claude-design.mdx known limitations section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
ukimsanovandClaude Opus 4.6
|
6b84971f6b |
feat(shader-transitions): support any aspect ratio (vertical, square)
Read data-width/data-height from the composition root instead of hardcoding 1920x1080. Enables shader transitions on vertical (1080x1920) and square (1080x1080) compositions. Changes across 3 files: - webgl.ts: WIDTH/HEIGHT constants → DEFAULT_WIDTH/DEFAULT_HEIGHT, createContext and renderShader accept width/height params - capture.ts: captureScene and captureIncomingScene accept width/height params for html2canvas - hyper-shader.ts: reads data-width/data-height from root element, passes dimensions to all webgl and capture calls GLSL shaders unchanged — they already use u_resolution uniform for all coordinate math and work at any aspect ratio. Backwards compatible: all params default to 1920x1080 when not provided or when data-width/data-height are missing from the DOM. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Miguel Ángel
|
31cd0ea6e7 | chore: release v0.4.19 v0.4.19 | ||
|
Ular Kimsanov
|
60d1494279 |
Merge pull request #468 from heygen-com/fix/claude-design-docs-download-ux
docs: fix SKILL.md download UX across all references |
||
|
Miguel Ángel
|
f8cb2b17f0 | chore: release v0.4.18 v0.4.18 | ||
|
ukimsanovandClaude Opus 4.6
|
c6da00d9c8 |
docs: fix SKILL.md download UX across all references
Raw GitHub URLs serve as text/plain — clicking opens a text tab instead of downloading. Updated all 5 references across 4 files: - claude-design.mdx (2 refs): "right-click → Save Link As" - prompting.mdx (1 ref): "right-click → Save Link As" - README.md (1 ref): "click download button on GitHub" - quickstart.mdx (1 ref): "click download button on GitHub" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Ular Kimsanov
|
95cc264fe7 |
Merge pull request #467 from heygen-com/feat/claude-design-skill-template-first
refactor(claude-design-skill): template-first rewrite with bug fixes |
||
|
ukimsanovandClaude Opus 4.6
|
95ca0200d4 |
docs: update all Claude Design references for template-first skill
Update docs, quickstart, prompting guide, and README to reflect: - Template-first approach (attach file, not paste URL) - Claude Design produces drafts, refine in any AI coding agent - Known limitations (vertical shaders, seeking, no linting) - Practical example prompts (feature announcement, founder pitch) - Removed outdated references (invisible bridges, fetch-the-skills-tree) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Miguel Ángel
|
970b446c49 |
feat(studio): drag assets from the sidebar onto the timeline (#464)
## Problem Studio still broke down in three concrete authoring flows around timeline assets: - you could import media into Assets, but not drag an already-imported asset from the Assets tab onto the timeline and persist it into source - dragging a file from outside the app onto the timeline only uploaded it into Assets instead of placing it at the dropped time/track - once a clip was on the timeline, there was no reliable keyboard delete flow for removing it safely from source While implementing direct external drops, another real bug showed up: - valid binary uploads like `raycast.mp4` from `Downloads` were being rejected as unsupported media in Studio dev because the Vite API bridge was corrupting multipart request bodies before they reached the upload route ## What this fixes ### Timeline asset placement from inside Studio - asset cards in the Assets tab are draggable - the timeline accepts asset drops even when it already has clips - dropping an asset onto the timeline inserts a new clip into the active composition source at the dropped time / track - asset paths are rewritten relative to the target composition file so drops into sub-compositions resolve correctly - the new clip is persisted immediately and the preview refreshes ### Direct external file drops onto the timeline - dropping a file from outside the app onto the timeline now uploads it and places it onto the dropped track/time in one shot - it no longer stops halfway by only adding the file into Assets - multiple dropped files are placed using the same drop start and successive tracks ### Delete key support - selected timeline clips can now be deleted with `Delete` / `Backspace` - deletion is persisted back to source, not just removed from local state - the delete path now uses a server-side DOM mutation helper with LinkeDOM for structural safety instead of client-side string surgery ### Binary upload fix for media files - the Studio Vite API bridge now forwards non-GET request bodies as raw bytes instead of decoding them as UTF-8 text - that preserves multipart uploads for binary media like MP4s - valid local videos from `Downloads` no longer get rejected as `Unsupported media skipped` just because the dev bridge corrupted the request body - upload validation now probes buffered media through a temp file path that preserves the file extension before saving into the project ## Root cause There were really two separate gaps: ### 1. Asset placement / deletion workflow gaps The timeline and asset systems already existed, but they were disconnected: - `AssetsTab` only supported copy/import flows - `Timeline` only handled raw file import, not positioned placement for existing assets - there was no utility layer for converting a dropped asset into persisted timeline HTML - there was no structurally safe deletion path for arbitrary selected timeline clips ### 2. Binary upload corruption in Studio dev The Studio Vite API bridge rebuilt non-GET request bodies like this: - read each request chunk - call `chunk.toString()` - concatenate into a string - construct the Fetch `Request` from that string body That works for text, but it corrupts multipart binary uploads. By the time the upload route wrote the received file and ran `ffprobe`, otherwise valid MP4s had already been mangled in-flight. ## Behavior - dropping on `index.html` inserts the asset into the root composition - dropping while drilled into a composition inserts into that composition file instead - drop X position maps to `data-start` - drop Y position maps to the current visible track row, with a new bottom track created if the drop lands below existing rows - images default to a short finite duration - audio/video default to their metadata duration when available, with a fallback duration if metadata cannot be read quickly - pressing `Delete` on a selected clip removes that clip from the underlying HTML source and clears selection in Studio - valid uploaded MP4s now survive the Studio dev API bridge intact instead of being rejected during upload validation ## Verification ### Local checks - `bunx oxlint packages/core/src/studio-api/helpers/sourceMutation.ts packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/core/src/studio-api/routes/files.ts packages/studio/src/App.tsx packages/studio/src/components/nle/NLELayout.tsx packages/studio/src/components/sidebar/AssetsTab.tsx packages/studio/src/player/components/Timeline.tsx packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.config.ts packages/studio/vite.request-body.ts packages/studio/vite.request-body.test.ts` - `bunx oxfmt --check` on the touched files - `bun run --filter @hyperframes/core typecheck` - `bun run --filter @hyperframes/studio typecheck` - `bun test packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.request-body.test.ts` ### Browser / live verification Verified against a live local Studio fixture: - dragging an existing asset from the Assets tab onto the timeline creates a persisted clip at the dropped position - dropping a file from outside the app directly onto the timeline uploads it and creates a persisted clip at the dropped position - selecting a dropped clip and pressing `Delete` removes it from both the live timeline and the saved source HTML - valid MP4 uploads like `raycast.mp4` now succeed through the live Studio upload route instead of being rejected as unsupported media ## Notes - the local `timeline-trio-verify` and `timeline-overlap-debug` projects used for verification are local-only and are not part of this PR - this PR is about asset placement, upload correctness, and deletion safety; it does not broaden into richer editing workflows beyond placing/removing clips from the timeline |
||
|
Ular Kimsanov
|
078a8b349b |
Merge pull request #456 from heygen-com/fix/shader-transitions-capture-robustness
fix(shader-transitions): harden capture against visibility, scrub, Safari taint |
||
|
ukimsanovandClaude Opus 4.6
|
18136d4eb5 |
fix(shader-transitions): add inWindow guard to .catch fallback handler
Address Vance's review on #456: the .catch handler was missing the same tl.time() window check that .then has. Late-rejecting captures (Safari + SVG-filter compositions) could fire gsap.to/fromTo on scenes the playhead already left, causing flash-to-black mid-scene. Wraps the CSS crossfade fallback in the same inWindow guard so stale catch handlers are no-ops, matching the .then behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
ukimsanov
|
ee7c6c3c24 |
refactor(claude-design-skill): template-first rewrite with bug fixes
Replace the rule-heavy Claude Design skill with a template-first approach. Instead of teaching all rules from scratch (863 lines), provide pre-valid skeletons where structural rules are embedded. Claude Design fills in palette, content, and animations. Key changes: - Pre-valid HTML skeletons for social reel (vertical), launch teaser, product explainer, and cinematic title - Mixed transitions: 2-3 shader transitions at key moments, rest hard cuts (matches professional video practice) - autoAlpha toggles for non-anchor scenes (fixes invisible middle scenes caused by HyperShader's blanket opacity reset) - Explicit first-anchor opacity fix (HyperShader browser mode never auto-shows the first anchor scene) - No shaders on vertical (1080x1920) — WebGL canvas is hardcoded to 1920x1080 in webgl.ts - Inline animation patterns (counter, stroke draw, stagger, float, bar chart, orbit, highlight sweep, safe CSS grain) - Claude Design → Claude Code handoff workflow documented in README template and delivery step - Troubleshooting table for black preview, invisible scenes, seeking - Self-review checklist split: structural validity, brand accuracy, animation baseline - Skill description updated per review feedback on PR #353 to disambiguate from the hyperframes skill on cross-surface routing Bugs fixed by this rewrite (confirmed across demov4-1 through demov4-6 and demov5-1 through demov5-5): - First shader anchor invisible in every composition - Non-anchor scenes killed by blanket querySelectorAll(".scene") opacity reset - visibility toggles insufficient (opacity:0 persists after reset) - Shaders on vertical compositions produce distorted transitions |
||
|
Miguel Ángel
|
d3899b16ff | chore: release v0.4.17 v0.4.17 | ||
|
Miguel Ángel
|
6610b8ad00 |
fix: harden studio timeline editing and local renders (#463)
* fix: harden studio timeline editing and local renders * test: cover studio local render fallback * fix(studio): scale composition hover previews to stage size * test: normalize studio producer fallback paths * fix(studio): preserve move surface and retry render fallback |
||
|
Vance Ingalls
|
21063c66d9 |
perf(producer): gate per-frame debug meta via optional isLevelEnabled (#383)
## Summary
Add an optional `isLevelEnabled(level)` method to `ProducerLogger` and use it to short-circuit per-frame HDR composite metadata construction in `renderOrchestrator` when the log level is above debug.
Closes Chunks 8C and 8D from `plans/hdr-followups.md`.
## Why
`Chunk 8C` of `plans/hdr-followups.md`. The per-frame HDR composite snapshot (every 30 frames) was building an `Array.find` + `toFixed` + struct allocation unconditionally and handing it to a debug logger that immediately discarded it at `level="info"`. On long renders, this is allocation pressure and CPU time wasted on log meta nobody reads.
`Chunk 8D` was investigated in the same pass and found to already be guarded — see below.
## What changed
- New optional `isLevelEnabled(level: ProducerLogLevel): boolean` on `ProducerLogger`.
- `createConsoleLogger` implements it.
- `renderOrchestrator.ts` per-frame HDR composite snapshot is now gated on `i % 30 === 0 && (log.isLevelEnabled?.("debug") ?? true)` — production runs at `level="info"` skip the meta-object construction entirely; custom loggers without the new method keep their existing behavior thanks to the `?? true` fallback.
- New `packages/producer/src/logger.test.ts` (17 tests) covering level filtering, meta formatting, the `isLevelEnabled` path, a hot-loop call-site simulation that asserts zero builder invocations at info level, and the `?? true` fallback for loggers that omit the method.
- `docs/packages/producer.mdx` gains a new "Logging" section documenting `ProducerLogger`, `createConsoleLogger`, `defaultLogger`, and the `isLevelEnabled` gating pattern.
**8D resolution (no code change).** `countNonZeroAlpha` / `countNonZeroRgb48` calls live behind `shouldLog = debugDumpEnabled && debugFrameIndex >= 0`, where `debugDumpEnabled` is itself driven by `KEEP_TEMP=1`. The pixel iteration is fully skipped on production runs already, so 8D needed no fix — verified during the 8C work.
## Test plan
- [x] `bun test` in producer — 17/17 logger tests pass; existing service tests unchanged.
- [x] Hot-loop call-site simulation asserts the meta builder is invoked **zero times** at `level="info"`.
- [x] `?? true` fallback preserves prior behavior for custom logger implementations that don't define the method.
- [x] Re-ran the HDR benchmark from Chunk 8A — no regression on wall-clock, peak heap unchanged at info level.
## Stack
Chunks 8C + 8D of `plans/hdr-followups.md`. Sits on top of the benchmark harness PR (Chunk 8A) so the optimization is measurable.
|
||
|
Vance Ingalls
|
3da8c2e969 |
perf(producer): hdr benchmark harness — --tags filter, peak heap/RSS tracking, bench:hdr script (#382)
## Summary Make the existing benchmark harness genuinely useful for HDR perf work: positive `--tags` filter, peak heap/RSS sampling, a `bench:hdr` script, and a perf README documenting the captured April-2026 baseline. Lands first in the Chunk 8 sub-stack so subsequent perf PRs can be measured against a known starting point. ## Why `Chunk 8A` of `plans/hdr-followups.md`. Wall-clock timing alone can't catch slow memory regressions like an unbounded image cache — peak RSS does. And the existing harness only had `--exclude-tags`, so HDR runs had to wait for unrelated SDR fixtures. ## What changed **1. Positive `--tags` filter** in `benchmark.ts`. Adds `--tags hdr` so HDR runs don't have to wait for unrelated fixtures. Filters compose: a fixture must match `--tags` (if provided) AND must not match `--exclude-tags`. **2. Peak heap + RSS tracking** in `executeRenderJob`. A 250 ms periodic `process.memoryUsage()` sampler runs alongside every render and reports `peakRssMb` / `peakHeapUsedMb` in `RenderPerfSummary`. Sampler is `unref`'d and always cleared in `finally` so it never keeps the event loop alive or leaks across jobs. Both fields are optional on the interface for back-compat with serialized older summaries. **3. `bench:hdr` convenience script** plus a perf README at `tests/perf/README.md` documenting the harness, the new flags, and the captured April-2026 HDR baseline (PQ regression: 34.5 s / 272 MiB RSS, HLG regression: 11.5 s / 227 MiB RSS, both 1080p / 1 worker / 1 run). The benchmark output table is widened and gains `PeakRSS` / `PeakHeap` columns. A new `avgOrNull` helper preserves `null` in the JSON when no run reported memory (avoids silently coercing missing data to 0 in older snapshots). No behavior change for non-benchmark renders — the sampler runs in every `executeRenderJob` but its overhead is a single `process.memoryUsage()` call every 250 ms, well below noise. ## Test plan - [x] `bunx tsc --noEmit -p packages/producer` — clean. - [x] `bunx oxlint` / `bunx oxfmt --check` on changed files — clean. - [x] `bun test src/services/` — 60/60 pass (frameDirCache, orchestrator, etc.). - [x] `bunx tsx src/benchmark.ts --tags hdr --runs 1` — both HDR fixtures render successfully, summary table prints `PeakRSS`/`PeakHeap` columns, per-run output shows new memory line. - [x] `bunx tsx src/benchmark.ts --tags nonexistent` — exits 1 with a helpful message naming the active filters. ## Stack Chunk 8A of `plans/hdr-followups.md`. First PR in the Chunk 8 perf sub-stack; subsequent PRs (image cache, logger gating) measured against this baseline. |
||
|
Vance Ingalls
|
cc9403b6bd |
test(producer): extract frameDirMaxIndexCache to its own module and pin cross-job isolation (#381)
## Summary Extract the `frameDirMaxIndexCache` from a private module-scoped Map inside `renderOrchestrator.ts` into its own `frameDirCache.ts` module, then add a 11-test bun:test suite that pins the cross-job isolation contract added in Chunk 5B. ## Why `Chunk 9E` of `plans/hdr-followups.md`. The cache lived as a private Map inside `renderOrchestrator.ts`, which made the cross-job isolation contract from Chunk 5B impossible to unit-test directly. Extracting it both makes the contract testable and reduces orchestrator complexity slightly. ## What changed - New `packages/producer/src/services/frameDirCache.ts` exposes `getMaxFrameIndex` / `clearMaxFrameIndex` / `getMaxFrameIndexCacheSize` (plus a test-only `__resetMaxFrameIndexCacheForTests` helper). Behavior is unchanged: callers still get the same module-scoped sharing inside a job, and `renderOrchestrator`'s outer `finally` still clears every entry it registered so the cache cannot grow monotonically across renders. - `renderOrchestrator.ts`: imports the new helpers, drops the unused `readdirSync` import, updates inline comments, and replaces two `frameDirMaxIndexCache.delete` sites with `clearMaxFrameIndex`. - New `frameDirCache.test.ts` (bun:test, 11 tests) covering: - Reading the max index from a populated directory. - Ignoring filenames that don't match `frame_NNNN.png` (wrong ext, wrong prefix, wrong case, double extension, empty index group, same-named subdirectory). - Empty- and missing-directory paths returning `0` and being cached. - Intra-job invariant: subsequent readdir mutations not observed once cached. - `clearMaxFrameIndex` forcing a re-read; returns `false` for paths that were never cached. - Per-directory isolation when multiple directories are registered. - The cross-job contract from Chunk 5B: cache empty between well-behaved jobs, doesn't grow monotonically across 20 simulated renders with 3 HDR videos each (steady-state cache size stays at 3), and a buggy job that forgets to clear leaks exactly its own entries rather than affecting unrelated jobs. ## Test plan - [x] `frameDirCache.test.ts` 11/11 pass. - [x] Existing producer tests unchanged. - [x] Behavior preserved: same module-scoped sharing inside a job, same outer-`finally` eviction. ## Stack Chunk 9E of `plans/hdr-followups.md`. Test-driven extraction; complements Chunk 5B. |
||
|
Miguel Ángel
|
154511247a |
feat: add Claude Code plugin manifest (#462)
* feat(claude-code-plugin): add Claude Code plugin manifest Adds .claude-plugin/plugin.json so the repo can be loaded as a Claude Code plugin. Reuses the existing skills/ directory — all 5 skills (hyperframes, gsap, hyperframes-cli, hyperframes-registry, website-to-hyperframes) are auto-discovered under the hyperframes: namespace. * docs(readme): document Claude Code plugin usage * chore(claude-code-plugin): add submission branding assets * docs(claude-code-plugin): address review notes |
||
|
Miguel Ángel
|
25d7a54330 |
docs: add Claude Design HyperFrames entry point (#353)
## Summary - add a GitHub-hosted `claude-design-hyperframes` skill entry point that tells Claude Design to fetch the upstream HyperFrames skills tree - add a dedicated Claude Design docs guide and link it from quickstart, prompting, and the README - fix `@hyperframes/player` CDN docs to show a working ESM include and the explicit global-build fallback ## Verification - `bunx oxfmt --check README.md docs/docs.json docs/guides/prompting.mdx docs/packages/player.mdx docs/quickstart.mdx packages/player/README.md docs/guides/claude-design.mdx skills/claude-design-hyperframes/SKILL.md` - `bun run lint:skills` - `bunx mintlify broken-links` - browser-engine screenshots captured with Playwright CLI for the changed docs/source surfaces: - `/tmp/hyperframes-pr-artifacts/claude-design-guide-source.png` - `/tmp/hyperframes-pr-artifacts/player-docs-source.png` ## Notes - `mintlify dev`, `mintlify validate`, and `mintlify export` stalled in this environment during preview/bootstrap, so I used the broken-links check plus screenshot-based browser fallback instead of claiming a full rendered-site pass. - The GitHub entry-point setup reflects current Claude Design behavior discussed in the task: point Claude Design at the repo-hosted skill URL rather than a ZIP upload flow. |
||
|
James Russo
|
d125d65164 |
feat(plugins): add Cursor plugin manifest + refresh marketplace branding (#461)
Adds .cursor-plugin/plugin.json at the repo root alongside the existing .codex-plugin/, so this repo is the single source of truth for the Codex plugin AND the new Cursor Marketplace submission. Refreshes the shared assets/logo.png + assets/icon.png to a 1024x1024 / 512x512 symbol-only mark rasterized from docs/favicon.svg (white background), and renames the marketplace display title from "HyperFrames" to "HyperFrames by HeyGen" in both Codex and Cursor manifests. No skill content changes; purely marketplace-visible branding and the new Cursor manifest file. |
||
|
ukimsanov
|
41b4017273 |
docs(shader-transitions): clarify allowTaint caveat in capture comment
Address Copilot review comment on #456: the old `allowTaint` doc comment said the resulting canvas is "still usable as a WebGL texture via gl.texImage2D (no pixel read-back required)", which is wrong. A tainted canvas CANNOT be uploaded to WebGL — the spec requires SecurityError on non-origin-clean sources with no opt-out. That's exactly what we observe in Safari + SVG-filter compositions, and what hyper-shader.ts's catch handler now handles via CSS crossfade. Update the comment to correctly describe the flag's effect: it only moves the failure point from html2canvas to texImage2D; the end-user UX is the same (smooth CSS fade in either case). The flag remains defensively correct for the non-taint branches where it genuinely helps (cross-origin images with `Access-Control-Allow-Origin`). No code change — comment only. Made-with: Cursor |
||
|
Vance Ingalls
|
2b25023e10 |
test(engine): cover spawnStreamingEncoder lifecycle and cleanup paths (#380)
## Summary Add unit tests that mock `child_process.spawn` to drive an in-memory "ffmpeg" through the success/failure paths used by the producer's HDR encoder and by Chunk 5A's defensive `close()` in `renderOrchestrator`. ## Why `Chunk 9D` of `plans/hdr-followups.md`. The contracts the orchestrator's try/finally cleanup (Chunk 5A) and the abort path rely on were entirely uncovered. A regression in `spawnStreamingEncoder`'s lifecycle handling would surface as a leaked ffmpeg process or a hung render, both of which are hard to diagnose after the fact. ## What changed `packages/engine/src/services/streamingEncoder.test.ts`: 7 new specs covering: - Successful exit after explicit `close()`. - Non-zero exit before `close()` returns a failure result (no throw). - `ENOENT` on spawn returns a failure result (no throw). - Abort signal triggers `SIGTERM` and a `"cancelled"` result. - `close()` is idempotent and never throws on a second call. - `writeFrame` returns `false` after the encoder has exited. - `close()` detaches the abort listener so post-close aborts don't re-kill ffmpeg. These contracts are what the `renderOrchestrator` try/finally cleanup added in Chunk 5A relies on, and what the ffprobe-unavailable test (Chunk 9B) hinted at for the encoder side. ## Test plan - [x] All new specs pass. - [x] No production code changes — pure regression coverage of existing lifecycle behavior. ## Stack Chunk 9D of `plans/hdr-followups.md`. Test-only change, complements Chunk 5A. |
||
|
Miguel Ángel
|
072814e65c |
fix(core,cli,ci): harden runtime resolution + inline constant + smoke test (#458)
Guard buildHyperframesRuntimeScript() against missing entry.ts so it returns null instead of crashing with esbuild stderr output. Add getHyperframeRuntimeScript() that returns the pre-built IIFE as a baked-in string constant — no esbuild, no file I/O, no import.meta.url. Consolidate CLI runtime source resolution into a single module with a clear priority chain: esbuild from source (dev) → inlined constant (production) → pre-built artifact file (fallback). Add CI smoke test that npm-packs the CLI, installs globally, runs hyperframes preview, and asserts no stderr errors + runtime endpoint returns JS. Bump version to 0.4.16.v0.4.16 |
||
|
ukimsanov
|
b3b458fda4 |
fix(shader-transitions): harden capture against visibility, scrub, Safari taint
Three interrelated fixes for the live-player path in @hyperframes/shader-transitions (Studio preview, <hyperframes-player> embeds, Claude Design in-pane iframe). Zero changes to engine mode — initEngineMode is byte-identical; producer render pipeline and CLI hyperframes render produce byte-identical output. 1. captureIncomingScene now forces visibility:visible during capture. The HF runtime sets visibility:hidden on [data-start] elements outside their playback window. With centered shader timing (transition.time = boundary - duration/2), html2canvas captures the incoming scene while it's still hidden → blank texture → visible blink mid-transition. Fix saves, overrides, captures, and restores visibility only for the capture window. Empirically validated via a direct html2canvas probe: captures of visibility:hidden elements return blank; with override, they return real content. 2. post-capture.dom guards on tl.time() window before mutating DOM. On scrub across multiple shader transitions, tl.call() fires several transitions' callbacks in rapid succession; each launches async html2canvas; each .then() unconditionally set all .scene opacities to 0, enabled shader canvas, and pointed state at that transition. The last to resolve won — often for a transition the playhead had left. Result: scenes stuck opacity:0 mid-scene; blank screen until the next transition's end.call ran. Fix: check tl.time() is still inside [T, T+dur] before applying state; otherwise skip. 3. .catch fallback does CSS crossfade instead of hard cut. When capture fails (Safari canvas taint from SVG data URLs, CORS errors, extreme DOM complexity) the old catch snapped all scenes to opacity:0 then set incoming to opacity:1 — jarring instant jump. Fix uses gsap.to/fromTo on opacity over the intended transition duration; smooth 0.5s fade is strictly better UX. Hard cut preserved as last-resort if elements are missing. Also adds defensive useCORS: true and allowTaint: true to the html2canvas call. No behavior change in Chrome (capture normally succeeds); adds resilience for cross-origin images with CORS headers and SVG-tainted canvases respectively. Known limitations (out of scope, follow-up tracked): - Safari + cross-origin iframe: html2canvas is 10-12x slower than Chrome due to WebKit's DocumentCloner.cloneNode perf (html2canvas#3108), causing perceptible per-transition freezes (1.5-2s each) in Claude Design's in-pane preview. Needs pre-capture architecture (cache incoming-scene textures at init) to eliminate per-transition cost. - SVG filter data URLs fundamentally taint html2canvas output in Safari; WebGL's texImage2D has no framework opt-out (WebGL spec). Addressed at the composition level via the Claude Design skill's anti-pattern 4 in a parallel PR. Made-with: Cursor |
||
|
Miguel Ángel
|
34db66ef0a |
fix(cli): prevent esbuild runtime error in global/npx installs (#452)
* fix(cli): resolve runtime fallback for globally-installed hyperframes When hyperframes is installed globally via npm, the `loadRuntimeSourceFallback()` path that dynamically imports @hyperframes/core and runs esbuild fails because @hyperframes/core is inlined into cli.js and import.meta.url resolves to the wrong location for the entry.ts source file. Add a disk-based fallback that searches for the pre-built IIFE runtime artifact in multiple locations: - Alongside the bundled CLI (dist/hyperframe-runtime.js, dist/hyperframe.runtime.iife.js) - Walking up from __dirname through node_modules The esbuild path is tried first to preserve live-rebuild behavior in dev, with the pre-built artifact search as a safety net for the bundled context. Also adds the IIFE artifact name variant to resolveRuntimePath() in the studio server so it checks both naming conventions. * fix(cli): gate esbuild fallback on source availability The previous fix still triggered esbuild's stderr output before the catch could suppress it. Now check whether the runtime entry.ts source file actually exists before attempting the on-the-fly build, avoiding the noisy error in global installs entirely. * fix(cli): remove noisy console.warn from runtime fallback The caller already handles a null return — no need to warn about something the user can't act on. If both paths fail, the /api/runtime.js route returns a 404 which the studio handles gracefully. * style(engine): fix oxfmt trailing blank line in chunkEncoder test * fix(cli): guard against null/undefined from loadHyperframeRuntimeSource Fall through to the pre-built artifact if the function returns a falsy value without throwing. * refactor(cli): consolidate runtime source resolution into single module Replace the scattered path-probing logic with a single loadRuntimeSource() that encodes the full priority chain: esbuild from source (dev only, gated on entry.ts existence) → pre-built artifact alongside cli.js → core/dist artifact → node_modules walk. Rename loadRuntimeSourceFallback → loadRuntimeSource since it's now the primary resolution function, not a fallback. |
||
|
Vance Ingalls
|
147bb737c9 |
test(engine): add ffprobe-unavailable fallback regression tests (#379)
## Summary Mock `node:child_process.spawn` to surface `ENOENT` and verify ffprobe's three callers behave correctly when ffprobe is missing. ## Why `Chunk 9B` of `plans/hdr-followups.md`. The PNG cICP fallback in `extractMediaMetadata` was added to support environments without ffprobe, but no test pinned the behavior — silently regressing it would break HDR image support on any system without ffprobe installed. ## What changed `packages/engine/src/utils/ffprobe.test.ts`: mocks `child_process.spawn` to surface `ENOENT` and asserts: - `extractMediaMetadata` falls back to PNG cICP metadata for image inputs. - `extractMediaMetadata` rethrows for non-image inputs lacking a still-image fallback. - `extractAudioMetadata` + `analyzeKeyframeIntervals` propagate the install-hint error verbatim. ## Test plan - [x] All new tests pass. - [x] No production code changes — pure regression coverage of existing fallback behavior. ## Stack Chunk 9B of `plans/hdr-followups.md`. Test-only change, independent of all other chunks. |
||
|
Vance Ingalls
|
4101cb721a |
test(shader-transitions): add midpoint (p=0.5) regression invariants for all shaders (#378)
## Summary Add four midpoint (`p=0.5`) regression invariants applied via a `describe` loop over `ALL_SHADERS`, so every existing and future shader transition automatically gets coverage at the most viewer-visible point in the animation. ## Why `Chunk 9G` of `plans/hdr-followups.md`. Existing smoke tests cover only the endpoints (`p=0 ≈ from`, `p=1 ≈ to`), which miss a class of regressions that surface specifically at the midpoint and let shaders silently rot in CI: - A shader becomes a no-op (returns input as-is) - A shader prematurely completes (returns target at midpoint) - A shader doesn't write to the output buffer at all - A shader loses determinism (`Math.random` / `Date.now` / leaked state) ## What changed `packages/engine/src/utils/shaderTransitions.test.ts`: a single `describe` loop over `ALL_SHADERS` that asserts at `p=0.5`: 1. `output ≠ from` — catches no-ops 2. `output ≠ to` — catches premature completion 3. `output` is non-zero — catches blank output 4. `output` is deterministic — catches accidental non-determinism Uses two distinct uniform input colors (40000/30000/20000 vs 10000/10000/10000) so equality checks have distinct byte patterns to compare against. Even shaders that warp UVs (which would be no-ops on uniform input alone) produce `mix16(from, to, 0.5)` at every pixel, distinct from both inputs. ## Test plan - [x] 60 new tests (4 invariants × 15 shaders), all passing. - [x] Any new transition added to the registry automatically picks up the same coverage. ## Stack Chunk 9G of `plans/hdr-followups.md`. Test-only change, independent of all other chunks. |
||
|
Vance Ingalls
|
ba6197a2e4 |
ci(windows-ffmpeg): pin BtbN release to specific autobuild tag (#447)
## What Pin the BtbN/FFmpeg-Builds Windows download to a specific `autobuild-2026-04-23-13-16` release tag instead of the rolling `latest` nightly. ## Why Follow-up to #436. The `release-url` input description claimed "pinned by default" but the actual default pointed at `releases/latest/download/…` — a nightly rolling build. A new upstream build could silently change encoder behavior or ABI between CI runs. The feature-inventory check catches codec removals but not within-codec behavioral shifts across nightlies. ## How Replaced the `latest` URL with a specific dated autobuild tag + its git-hash-stamped asset filename. Updated the input description to note that both the tag and filename must be bumped together when upgrading (the asset filename embeds the git hash). ## Test plan - [x] Verified pinned URL resolves (302 → 200) via `curl -sI -L` - [x] Confirmed feature-inventory step uses `ffmpeg -encoders`/`-decoders` output, not the asset filename — no assumptions broken by the rename - [ ] **Note:** Windows install/render/test jobs were skipped on this PR (YAML-only change didn't trigger Windows paths). The pinned download will be exercised by the next Windows-touching PR. |
||
|
Vance Ingalls
|
bb9e6bdf05 |
test(engine): lock down sRGB→BT.2020 LUT with byte-exact reference values (#377)
## Summary Add a 12-row reference table covering the full sRGB range with byte-exact 16-bit HLG and PQ signal values, plus three guard tests, locking down the `buildSrgbToHdrLut()` math. ## Why `Chunk 9F` of `plans/hdr-followups.md`. The matrix-free fast path through `blitRgba8OverRgb48le` runs every DOM pixel through `buildSrgbToHdrLut()` (sRGB EOTF → linear → HDR OETF → 16-bit). Any drift in the EOTF/OETF math — constant changes, branch swaps, rounding-mode regressions — would silently corrupt every text / UI / overlay pixel composited onto an HDR frame. Existing tests covered structural invariants (transparent passthrough, opaque overwrite, alpha blending, channel symmetry, HLG ≠ PQ) but no byte-exact reference values, so a uniform scale or constant tweak could pass everything. ## What changed - 12-row reference table in `alphaBlit.test.ts` covering black, shadow, mid-grays, highlight, near-white, and white with exact 16-bit HLG and PQ signal values. - Three guard tests: - **Asymmetric R/G/B (HLG):** each channel hits the LUT independently. - **Asymmetric R/G/B (PQ):** same, on the PQ path. - **BT.2408 SDR-white invariant:** PQ caps sRGB 255 at 38055 (~203 nits), well below HLG's 65535. This is the load-bearing detail that makes PQ headroom work — locking the exact value prevents a future "fix" that would re-scale PQ to peak-at-SDR-white and clip every real HDR pixel. Reference values mirror `buildSrgbToHdrLut()` exactly and were verified against the existing HLG mid-gray comment in the file. ## Test plan - [x] All new tests pass against the current LUT. - [x] Existing `alphaBlit.test.ts` invariants unchanged. ## Stack Chunk 9F of `plans/hdr-followups.md`. Test-only change, independent of all other chunks. |
||
|
Vance Ingalls
|
3089c8ee3a |
build(lfs): track tests/*/src/*.png via Git LFS (#376)
## Summary Track `tests/*/src/*.png` via Git LFS to mirror the existing policy for golden videos and `.mp4` fixtures. ## Why `Chunk 11C` of `plans/hdr-followups.md`. Without this rule, regression suites that grow PNG fixtures over time would bloat the working-tree history and slow shallow clones. ## What changed - `.gitattributes`: add `tests/*/src/*.png` to the LFS-tracked patterns. - Migrates the six existing PNG fixtures (1.6 MB combined: `hdr-photo-pq.png` plus `heygen-promo-preview-assets/` screenshots) onto LFS in the same commit so the rule applies retroactively. ## Test plan - [x] `git lfs ls-files` includes the HDR PNG fixtures after commit. - [x] Working tree size for these files goes from 1.6 MB to 6 × ~130 B LFS pointers. ## Stack Chunk 11C of `plans/hdr-followups.md`. Independent of all code changes. |
||
|
roiizchakandroi32
|
c11a332ef8 |
test(engine): cover h265 NVENC in preset-mapping regression tests (#443)
hevc_nvenc uses the same p1..p7 preset vocabulary as h264_nvenc, so the mapping in `mapPresetForGpuEncoder` applies to both codecs. The initial regression suite only covered `codec: "h264"`, which left a gap: a future refactor that split the H.264 and H.265 NVENC paths could silently regress one codec without any test catching it. Add three-case loops (ultrafast → p1, medium → p4, veryslow → p7) under `codec: "h265"` to both `buildEncoderArgs` and `buildStreamingArgs` test blocks. Each case also asserts that `-c:v hevc_nvenc` is selected so the test fails loudly if the codec plumbing is broken, not just the preset translation. Follow-up to #442 per review comment from @jrusso1020. Co-authored-by: roi32 <75878108+roi32@users.noreply.github.com> |
||
|
roiizchakandroi32
|
3b8de7a5eb |
fix(engine): accept libx264 preset names with NVENC and QSV (#442)
NVENC rejects the libx264 preset vocabulary (ultrafast / medium / slow /
...) with AVERROR(EINVAL) ("Error applying encoder options: Invalid
argument"), which surfaces as a bare `FFmpeg exited with code -22` from
spawn(). Because ENCODER_PRESETS passes these names straight through to
h264_nvenc / hevc_nvenc, every `--gpu` render using the `draft` tier
failed; `standard` (medium) and `high` (slow) only worked coincidentally
on ffmpeg builds that happened to accept those aliases. QSV has the same
problem on a narrower set (ultrafast / superfast / placebo).
Add `mapPresetForGpuEncoder` in utils/gpuEncoder.ts that translates the
libx264 vocabulary to each encoder's native names:
- nvenc: libx264 -> p1..p7 (already-native pN values pass through);
unknown values fall back to p4 (medium)
- qsv: ultrafast / superfast -> veryfast; placebo -> veryslow;
everything else passes through
- videotoolbox / vaapi / null: unchanged
Both buildEncoderArgs (chunkEncoder.ts) and buildStreamingArgs
(streamingEncoder.ts) now route through the helper before pushing
`-preset` to the ffmpeg arg vector.
To make the next encoder-options failure diagnosable without re-running
ffmpeg by hand, \`formatFfmpegError\` in utils/runFfmpeg.ts now appends
the last 15 non-empty stderr lines to the error string. The four call
sites that previously swallowed stderr (encodeFramesFromDir,
muxVideoWithAudio, applyFaststart, and the streaming encoder exit
handler) have been updated.
Tested end-to-end on an RTX 4080 with ffmpeg 8.1 NVENC across
\`--quality draft|standard|high\` plus \`--video-bitrate\` and \`--crf\`
overrides; the 6 renders were visually equivalent to the CPU baseline.
Co-authored-by: roi32 <75878108+roi32@users.noreply.github.com>
|
||
|
Vance Ingalls
|
8ffd007716 |
test(hdr-regression): tighten Window F maxFrameFailures budget after Chunk 4 fix (#375)
## Summary Tighten `hdr-regression` Window F `maxFrameFailures` from 5 → 0 now that Chunk 4 (matrix3d support + scene initial-state) has landed. ## Why Window F (transform + scale + border-radius on the video itself) was the remaining known-fail in the `hdr-regression` suite, baked into the golden so the suite stayed green while Chunk 4 was outstanding. After Chunk 4 fixed `parseTransformMatrix` (matrix3d support) and the shader-transitions initial-state, re-running the suite shows **0 failed frames** against the existing golden — the encoder is byte-deterministic, and Window F's GSAP rotation/scale happens to emit 2D `matrix()` rather than `matrix3d()`, so the same golden is still correct after the fix. Tightening the budget catches any drift in the layered HDR compositor immediately. ## What changed - `tests/hdr-regression/meta.json`: `maxFrameFailures` 5 → 0 (matches `hdr-hlg-regression`). - `tests/hdr-regression/README.md`: Window F row + Fix history section updated to reflect the new state. ## Test plan - [x] `bun run test --filter hdr-regression` — passes with 0 failed frames at the new budget. ## Stack Follow-up to Chunk 4 (transform & clipping). Reviewable separately so the budget tightening is decoupled from the code fix. |
||
|
Vance Ingalls
|
2e1a1d91a2 |
fix(engine,shader): handle matrix3d transforms and hide non-first scenes (#374)
## Summary
Two correctness fixes in the HDR transform & clipping pipeline: `parseTransformMatrix` now handles `matrix3d(...)` (GSAP's default `force3D: true`), and shader-transitions sets every non-first scene to `opacity: 0` at `t=0` so the engine doesn't over-composite at the start.
## Why
`Chunk 4` of `plans/hdr-followups.md`. Transform extraction and border-radius computation existed but were dead — an HDR video with `rotation: 45` rendered un-rotated, and 3-scene compositions ghosted at `t=0` because every scene defaulted to CSS `opacity: 1` and contributed to the first frame.
## What changed
**Matrix3d support in `parseTransformMatrix`.** `DOMMatrix.toString()` emits `matrix3d` whenever any ancestor in the chain has used a 3D transform — most importantly GSAP's default `force3D: true`, which converts `translate(...)` into `translate3d(..., 0)`. Without this, every GSAP-driven transform was silently dropped during HDR compositing because `videoFrameInjector.getViewportMatrix()` would return `matrix3d(...)` and the blit path would parse it as `null` and fall back to identity. The 16-value column-major form is converted to its 2D affine projection (indices 0, 1, 4, 5, 12, 13 → m11, m12, m21, m22, m41, m42); Z, perspective, and out-of-plane rotation components are dropped.
**Initial-state opacity in `initEngineMode`.** The browser preview branch uses a GL canvas overlay during transitions, so scene opacity at `t=0` doesn't matter visually. The engine branch reads scene opacity directly via `queryElementStacking()` to decide which layers to composite. Without an explicit initial-state tween, every scene defaulted to CSS `opacity: 1` and contributed to the very first frame, causing ghosting/overlap until the first transition fired. `tl.set()` at position 0 anchors the initial state in the timeline graph so reverse seeks from inside a later transition restore it correctly.
These two fixes together make `el.transform` and `el.borderRadius` (already wired in Chunk 7A's `compositeHdrFrame`) actually flow through the GSAP-animated case, and keep the engine's per-frame compositing aligned with what the user sees in browser preview.
## Test plan
- [x] 6 new `alphaBlit.test.ts` cases (identity matrix3d, translate3d, scale + translate3d, rotateZ, malformed arg count, non-finite values).
- [x] Existing `hdr-regression` Window H already CSS-sets `#scene-b { opacity: 0 }` as a fallback; the new `tl.set` is redundant for that case but harmless and removes the need for compositions to remember the CSS workaround.
- [x] Manual: rotated HDR video (`rotation: 45`) appears rotated; `border-radius: 50%` clips to circle; 3-scene composition has no overlap at `t=0`.
## Stack
Chunk 4 of `plans/hdr-followups.md`. Window F of the regression suite documents the bug; the next PR in the stack tightens the `maxFrameFailures` budget to 0.
|
||
|
Vance Ingalls
|
a3d7cc1c95 |
refactor(producer): extract HDR compositing helpers and rename media metadata (#373)
## Summary
Four behavior-preserving refactors that reduce complexity in `renderOrchestrator.ts` and clarify the engine ffprobe utility surface. Lands after the correctness fixes (Chunks 1–5) so the refactored code is already correct.
## Why
`Chunk 7` of `plans/hdr-followups.md`. The HDR composite block had grown a ~200 LOC inline closure with 14 captured deps, a repeated capture-options spread, a `extractVideoMetadata` name that now also handles still images, and per-frame re-creation of debug helpers.
## What changed
**7A — Hoist `compositeToBuffer` into a module-scoped helper.** Extract the inline HDR closure into a top-level `compositeHdrFrame()` that takes an `HdrCompositeContext` struct. Construct the context once at the top of the HDR render block and pass it through. Removes a deeply-nested closure from the middle of the orchestrator.
**7B — `buildHdrCaptureOptions()` helper.** Factor the repeated `{ ...captureOptions, skipReadinessVideoIds: ... }` spread into a named helper at the call site.
**7C — Rename `extractVideoMetadata` → `extractMediaMetadata`.** Reflects that the helper handles still images (PNG/JPEG/WebP) in addition to video. Update all callers in engine + producer (`videoFrameExtractor`, `htmlCompiler`, regression-harness, producer ffprobe re-export, tests). Re-export the old name as a deprecated alias from `@hyperframes/engine` for backward compatibility, plus the producer re-export shim.
**7D — Hoist debug counters to module scope.** `countNonZeroAlpha` and `countNonZeroRgb48` are now module-scoped so they aren't re-created per frame and so the closure has fewer captures.
Also touches the `hdr-regression` and `hdr-hlg-regression` README + `meta.json` files reviewed during this refactor.
## Test plan
- [x] `bunx tsc --noEmit -p packages/producer && bunx tsc --noEmit -p packages/engine` clean.
- [x] Engine tests: 313 pass, 0 fail (1218 expect calls).
- [x] `bunx oxlint` + `bunx oxfmt --check` clean on 8 changed source files.
- [x] Diff is structural only — no behavioral changes.
## Stack
Chunk 7 of `plans/hdr-followups.md`. Lands after the correctness fixes (Chunks 1–5) per the suggested merge order.
|
||
|
Vance Ingalls
|
2f58e9d188 |
ci(windows-render): bypass Chocolatey, fetch ffmpeg from BtbN/GitHub (#436)
## What Replace the `choco install ffmpeg` step in `windows-render.yml` with a direct download of the upstream Windows GPL build from [`BtbN/FFmpeg-Builds`](https://github.com/BtbN/FFmpeg-Builds/releases/latest) on GitHub Releases. ## Why The `Render on windows-latest` canary started failing on every PR with: ``` [NuGet] Response status code does not indicate success: 504 (Gateway Timeout). [NuGet] Response status code does not indicate success: 503 (Service Unavailable). ``` The Chocolatey community feed (`community.chocolatey.org/api/v2/package/ffmpeg/8.1.0`) is degraded for the `ffmpeg` package right now. The earlier 3-attempt retry I added wasn't enough — every attempt across multiple runs failed with 503/504, so retrying does nothing. The Chocolatey path is also a bit indirect for what this job actually validates. The real point of the canary is the [PR #336](https://github.com/heygen-com/hyperframes/pull/336) fix where `findFFmpeg()` / `where ffmpeg` discovery has to work on a fresh Windows runner. As long as `ffmpeg.exe` ends up on `PATH`, the underlying thing under test (the harness can find ffmpeg, capture frames, mux to MP4) is exercised exactly the same. BtbN/FFmpeg-Builds is the canonical upstream nightly Windows GPL build (Chocolatey itself rebundles essentially the same artifact), so this is closer to the source, not further from it. ## How - Download `ffmpeg-master-latest-win64-gpl.zip` from the BtbN release with `Invoke-WebRequest` (3-attempt retry with backoff). - Extract to `$env:RUNNER_TEMP/ffmpeg` and locate `ffmpeg.exe` recursively. - Add the bin directory to `$env:GITHUB_PATH` so all subsequent steps in the job (the Bun-driven harness, `findFFmpeg()`, etc.) see ffmpeg on `PATH` exactly the same way as before. - Print `ffmpeg -version` as a sanity check. ## Test plan - [ ] CI: `Render on windows-latest` job goes green on this PR. - [ ] Subsequent PRs no longer get blocked on `choco install ffmpeg` 503s. |
||
|
Miguel Ángel
|
e853dd7a1d | chore: release v0.4.15 v0.4.15 | ||
|
Vance Ingalls
|
53e1aeaadc |
fix(producer): wire --crf and --video-bitrate CLI overrides into encoders (#372)
## Summary Re-wire the `--crf` and `--video-bitrate` CLI flags through the three encoder spawn sites in `renderOrchestrator.ts`. They were defined and parsed in the CLI but silently dropped before reaching ffmpeg. ## Why `Chunk 10` of `plans/hdr-followups.md`. PR #292 originally wired these through with a `baseEncoderOpts` object using `effectiveQuality`/`effectiveBitrate`; PR #268 rewrote the encode paths and reverted to `preset.quality` only, accidentally dropping the override. This is a user-facing regression — `hyperframes render --crf 18` was being silently ignored. ## What changed - At the three encoder spawn sites (HDR streaming, SDR streaming, disk-based encode), `quality` defaults to `preset.quality` but is overridden by `job.config.crf` when set, and `bitrate` is set from `job.config.videoBitrate`. Mutual exclusivity is enforced upstream in the CLI, so we don't re-check it here. - Fix the contradictory note in `docs/packages/cli.mdx` that claimed CRF/bitrate were now driven only by `--quality`. The flags table now lists `--crf` and `--video-bitrate` consistent with `docs/guides/rendering.mdx`. ## Test plan - [x] `hyperframes render --crf 18 ...` now respects the CRF override (verified via ffprobe of the encoded output). - [x] `hyperframes render --hdr ...` still works (no behavior change at the default path). - [x] `hyperframes render --help` shows all flags consistent with the docs. ## Stack Chunk 10 of `plans/hdr-followups.md`. Independent of all other chunks. |
||
|
Vance Ingalls
|
6fd99109c9 |
fix(producer): tighten resource lifecycle and harden file server (#371)
## Summary Five resource-management fixes in `renderOrchestrator.ts` and `fileServer.ts`: HDR encoder cleanup on non-abort errors, `frameDirMaxIndexCache` eviction, mid-transition abort responsiveness, pre-allocated transition buffers, and a path-traversal guard for the local file server. ## Why `Chunk 5` of `plans/hdr-followups.md`. These are independent leaks/hangs/security issues that had each been called out in prior PR reviews and never landed. ## What changed **5A — HDR encoder + `domSession` cleanup.** The HDR streaming encoder and `domSession` were spawned outside any outer `try/finally`, so a non-abort error between encoder spawn and the inner cleanup leaked the FFmpeg process and held the browser page open. Wrapped the entire HDR (and SDR streaming) capture path in a `try/finally` with explicit `*Closed` flags, and defensively close both in the outer `finally` if they haven't been closed already. `StreamingEncoder.close()` and `closeCaptureSession()` are both idempotent, so double-close is safe. **5B — `frameDirMaxIndexCache` + `hdrFrameDirs` eviction.** `frameDirMaxIndexCache` is module-scoped and grew monotonically: every render added entries that were never removed. Lifted `hdrFrameDirs` to the outer scope, drop the matching cache entry in the per-video `rmSync` block, and sweep any survivors in the outer `finally`. The on-disk frames themselves were already torn down with `workDir`; this just stops the in-process Map from leaking entries across renders. **5C — Abort signal between scene A and scene B.** During a shader transition the orchestrator captures scene A and scene B back-to-back inside a single outer frame iteration. An abort that arrived while scene A was capturing wouldn't be noticed until the next outer frame — after scene B had already been fully composited and discarded. Added `assertNotAborted()` at the top of the inner `[transBufferA, transBufferB]` loop so abort is observed before the second scene's DOM seek + screenshot. **5D — Pre-allocated transition buffers (already addressed).** The transition buffers (`transBufferA`, `transBufferB`, `transOutput`, `normalCanvas`) are pre-allocated outside the per-frame loop. The remaining `Buffer.from` copies sit in HDR transfer conversion (Chunk 8B territory) and image preload, neither of which is the per-frame hot path. **5E — `fileServer` path-traversal guard.** `fileServer.ts` joined `compiledDir` / `projectDir` with the request path and only checked `existsSync` + `isFile`. `path.join` normalizes `..` segments, so `GET /../etc/passwd` would resolve to `/etc/passwd` and be served straight off disk if the file existed. Added an `isPathInside(child, parent)` helper that resolves both sides and compares prefixes with the platform separator appended (so `/foo` doesn't match `/foobar`), and rejects any candidate that lands outside its intended root. ## Test plan - [x] `bun run --filter @hyperframes/producer typecheck` passes. - [x] `fileServer.test.ts` 13/13 pass (4 existing + 9 new `isPathInside` cases covering same-path, nested, prefix-only siblings, escaping traversal, traversal that resolves back inside, trailing-slash handling, and relative-path resolution). - [x] Manual: kill a render mid-flight with a non-abort error; no orphaned `ffmpeg` processes (5A). - [x] Manual: two render jobs back-to-back; cache cleared between jobs (5B). - [x] Manual: abort during a transition frame; stops promptly, not after scene B (5C). - [x] Manual: `GET /../../../etc/passwd` against the local file server returns 403/404 (5E). ## Stack Chunk 5 of `plans/hdr-followups.md`. |
||
|
Miguel Ángel
|
aea85af044 | fix: improve studio timeline discoverability (#431) | ||
|
Vance Ingalls
|
5256a93b2d |
feat(engine): wire options.hdr through chunkEncoder + dynamic SDR→HDR transfer (#370)
## Summary
Three independent fixes that share a common thread: HDR config flowing correctly from `EngineConfig` down through every encoder. The headline fix: disk-based HDR encodes via `chunkEncoder` were silently producing BT.709-tagged output despite `options.hdr` being set.
## Why
`Chunk 3` of `plans/hdr-followups.md`. The streaming encoder was correct but `chunkEncoder.buildEncoderArgs` hard-coded BT.709 color tags and the `bt709` VUI block in `-x265-params`, even when callers passed an HDR `EncoderOptions`. Today this is harmless because `renderOrchestrator` routes native-HDR content to `streamingEncoder` and only feeds `chunkEncoder` sRGB Chrome screenshots — but the contract was a lie, and any future caller that wired HDR through `chunkEncoder` would silently get SDR output.
## What changed
**3A — `chunkEncoder` respects `options.hdr` (BT.2020 + mastering metadata).** When `options.hdr` is set, the libx265 software path emits `bt2020nc` plus the matching transfer (`smpte2084` for PQ, `arib-std-b67` for HLG) at the codec level *and* embeds master-display + max-cll SEI in `-x265-params` via `getHdrEncoderColorParams`. libx264 still tags BT.709 inside `-x264-params` (libx264 has no HDR support) but the codec-level color flags flip so the container describes pixels truthfully. GPU H.265 (nvenc/videotoolbox/qsv/vaapi) gets the BT.2020 tags but no `-x265-params` block, so static mastering metadata is omitted — acceptable for previews, not HDR-aware delivery.
**3B — `convertSdrToHdr` accepts a target transfer.** `videoFrameExtractor.convertSdrToHdr` was hard-coded to `transfer=arib-std-b67` (HLG) regardless of the surrounding composition's dominant transfer. `extractAllVideoFrames` now calls `analyzeCompositionHdr` first, then passes the dominant transfer (`"pq"` or `"hlg"`) into `convertSdrToHdr` so an SDR clip mixed into a PQ timeline gets converted with `smpte2084`, not `arib-std-b67`.
**3C — `EngineConfig.hdr` type matches its declared shape.** The IIFE for the `hdr` field returned `undefined` when `PRODUCER_HDR_TRANSFER` wasn't `"hlg"` or `"pq"`, but the field is typed as `{ transfer: HdrTransfer } | false`. Returning `false` matches the type and avoids a downstream `undefined` check.
## Test plan
- [x] `chunkEncoder.test.ts`: replaced the previous "HDR options ignored" assertions with 8 new specs covering BT.2020 + transfer tagging, master-display/max-cll embedding, libx264 fallback behavior, GPU H.265 + HDR (tags but no x265-params), and range conversion for both SDR and HDR CPU paths.
- [x] All 313 engine unit tests pass (5 new HDR specs).
- [x] `ffprobe` an HDR composition rendered through the chunk encoder path: shows `bt2020nc` color matrix, `smpte2084` transfer, and mastering display metadata.
## Stack
Chunk 3 of `plans/hdr-followups.md`. Independent of Chunks 1/4 (touches separate code paths).
|
||
|
Vance Ingalls
|
60f4ebbf13 |
test(hdr-regression): tighten Window C maxFrameFailures budget after Chunk 1 fix (#369)
## Summary Tighten `hdr-regression` Window C `maxFrameFailures` from 30 → 5 now that Chunk 1 (opacity pipeline) has landed. ## Why Window C (direct `<video>` opacity tween) was previously listed as a known failure with a `maxFrameFailures` budget of 30 to absorb expected drift until Chunk 1 landed. After the Chunk 1 fix, the regression test passes against the existing golden with **0 failed frames**. Tightening the budget catches any future drift in the opacity path immediately rather than letting up to 30 broken frames slip through. ## What changed - `tests/hdr-regression/meta.json`: `maxFrameFailures` 30 → 5 (small budget remains for HEVC encoder noise). - `tests/hdr-regression/README.md`: updated to mark Window C as fixed and note the tightened budget. The HEVC encoder is byte-deterministic and the opacity fix doesn't perturb pixels at the PSNR ≥ 28 checkpoint threshold, so regenerating the golden produces byte-identical output. The golden is therefore unchanged. Window F (transform + border-radius) remains pending Chunk 4; its broken state is currently baked into the golden, so the suite is green and Chunk 4's regen will catch any drift. ## Test plan - [x] `bun run test --filter hdr-regression` — passes with 0 failed frames at the new budget. ## Stack Follow-up to Chunk 1 (opacity pipeline). Reviewable separately so the golden churn (none in this case) is decoupled from the code fix. |
||
|
Vance Ingalls
|
2d57918f64 |
fix(engine): stop clobbering native <video> opacity in HDR pipeline (#368)
## Summary Fix four interrelated bugs in the opacity pipeline. The headline fix: the HDR compositor was effectively ignoring direct-on-`<video>` opacity animation because the engine itself was clobbering inline opacity with `opacity: 0 !important` — switching to `visibility: hidden` resolves the bug at the root. ## Why `Chunk 1` of `plans/hdr-followups.md`. This was the most user-visible bug in the entire follow-ups list: a GSAP-controlled opacity tween directly on a `<video>` element under HDR rendered at full brightness instead of fading. ## What changed **1A — Stop clobbering native `<video>` opacity.** `screenshotService.injectVideoFramesBatch` and `syncVideoFrameVisibility` were applying `opacity: 0 !important` to native `<video>` elements to hide them under the injected `<img>`. That stomp clobbered any GSAP-controlled inline opacity, so the next seek read 0 from computed style and the comp went black. Switched to `visibility: hidden !important` only. Visibility hides the element from rendering without changing its opacity, so subsequent reads (and `queryElementStacking`) see the real GSAP value on every frame. The `parseFloat(...) || 1` recovery hack at `injectVideoFramesBatch` was specifically there to compensate for this stomp; it's now replaced with a `Number.isNaN` guard that defaults to 1 only when parsing actually fails. **1B — `Number.isNaN` guards in `queryVideoElementBounds`.** `parseFloat(style.opacity) || 1` silently coerced a real opacity of 0 into 1. Switched to explicit `Number.isNaN` checks so opacity 0 stays 0. Same fix for `parseFloat(style.zIndex)`. **1C — `instanceof HTMLElement` instead of cast.** `resolveRadius` cast `el as HTMLElement` to read `offsetWidth`/`Height`. SVG and other non-HTML elements would have crashed at runtime. Replaced the cast with an `instanceof HTMLElement` guard, and made the numeric fallback `Number.isNaN`-safe. **1D — Opacity walk starts from the element itself.** The walk in `queryVideoElementBounds` started from `el.parentElement` for HDR videos to skip past the engine's forced `opacity: 0` on the element itself. Now that the engine never sets opacity, the special case is unnecessary — always walk from `el`. Kept the `isHdrEl` lookup because transform/border-radius logic further down still branches on it. ## Test plan - [x] `bun run --filter @hyperframes/engine typecheck` clean. - [x] `bun run --filter @hyperframes/engine test` — 308/308 passing. - [x] `bun run --filter @hyperframes/producer typecheck` clean. - [x] `oxlint` + `oxfmt --check` on both touched files. - [x] `hdr-regression` Window C (the direct-opacity window) now passes against the regenerated golden — see follow-up PR in this stack which tightens the budget. ## Stack Chunk 1 of `plans/hdr-followups.md`. Window C of the regression suite documents the bug; the next PR in the stack regenerates the golden and tightens its `maxFrameFailures` budget. |
||
|
Miguel Ángel
|
293d92af05 | chore: release v0.4.15-alpha.1 v0.4.15-alpha.1 | ||
|
Miguel Ángel
|
b4e9d64e29 |
feat(cli): hyperframes publish — share projects via a public URL (#312)
## Summary This PR adds `hyperframes publish` as the OSS handoff into the persisted HyperFrames publish flow. Instead of opening a local tunnel, the CLI now: 1. zips the local project 2. uploads it to the HeyGen publish backend 3. gets back a stable `hyperframes.dev` project URL plus claim token 4. prints a claimable URL for the user Example output: ```bash $ hyperframes publish Project my-video Files 12 Public https://hyperframes.dev/p/hfp_123?claim_token=... Open the URL on hyperframes.dev to claim the project and continue editing. ``` ## User Flow The intended user flow is: 1. Run `hyperframes publish` from a local HyperFrames project. 2. The CLI uploads the project as a zip to the publish API. 3. The CLI prints a stable `hyperframes.dev` URL with the claim token attached. 4. The user opens that URL in the browser. 5. `hyperframes.dev` uses that URL to claim the published project and import it into the web app. 6. The user continues editing from a normal web session. So the CLI is only responsible for packaging, upload, and printing the URL. The browser-side claim/import flow lives in the backend and web app stack. ## Routing This PR does not expose a separate user-facing canary mode. The CLI posts to the normal publish API host: - `https://api2.heygen.com/v1/hyperframes/projects/publish` Backend routing behavior is handled server-side. If the default path routes through canary, it does so without a dedicated CLI flag; if that path is unavailable, traffic falls back to prod behavior on the backend side. ## What Changed | File | Role | |---|---| | `packages/cli/src/commands/publish.ts` | Adds the `hyperframes publish` command, confirmation prompt, lint-before-upload behavior, and user-facing output. | | `packages/cli/src/utils/publishProject.ts` | Zips the local project, filters ignored files/directories, posts the archive to the publish API, and returns the published project metadata. | | `packages/cli/src/utils/publishProject.test.ts` | Covers archive creation and successful upload response parsing. | | `packages/cli/src/cli.ts` | Registers the new `publish` command. | | `packages/cli/src/help.ts` | Adds `publish` to root help and examples. | | `docs/packages/cli.mdx` | Documents the persisted publish flow. | ## Important Behavior - Requires `index.html` at the project root. - Ignores hidden files and common non-project directories like `.git`, `node_modules`, `dist`, `.next`, and `coverage`. - Lints the project before upload and prints findings, but does not block publish on warnings. - Does **not** keep a local process alive after upload. - Does **not** open a public tunnel. - Does **not** require HeyGen OAuth inside the CLI. ## Why This Shape This keeps the OSS CLI simple and matches the current product direction: - project persistence lives in HeyGen's backend - the public URL comes from the persisted project row - claiming/importing happens on `hyperframes.dev` - the CLI should not own browser auth or long-lived sharing infrastructure ## Verification In the earlier PR worktree, this flow was verified locally with the CLI build/test path and with real backend integration. In this cleanup worktree, the narrow code/doc change was verified by inspection, but the repo-level commands are currently blocked here by missing local tool binaries and typings in the worktree environment: - `bun run --filter @hyperframes/cli test` -> `vitest: command not found` - `bun run --filter @hyperframes/cli typecheck` -> local dependency/type resolution failures outside this diff - `bun run --filter @hyperframes/cli build` -> `tsx: command not found` ## Notes This PR only covers the OSS CLI side of the flow. The full end-to-end experience depends on the corresponding backend and `hyperframes.dev` changes that store published projects, return the stable URL, and support claim/import in the web app. |
||
|
Miguel Ángel
|
64779a0c16 | chore: release v0.4.14 v0.4.14 | ||
|
Miguel Ángel
|
95bf333895 |
fix: stabilize apple master timeline and playback (#419)
## Summary - preserve authored non-root composition timing before runtime sanitization so Studio can build the correct master timeline for chained subcompositions - prefer the fresh runtime source in Studio dev so local preview does not serve a stale `/api/runtime.js` - restrict preserved authored timing inference to the Studio timeline payload instead of the general runtime resolver ## What this fixes This PR fixes the Apple presentation class of failures where the root `index.html` / `Master` view looked correct at first and then collapsed into an incorrect short timeline. Before this change: - the master transport could report a short duration like `0:12` instead of the real deck length (`2:21` in the Apple project) - composition clips bunched near the start instead of laying out sequentially across the deck - seeking into later parts of the deck would land in the wrong place or show the wrong active composition - local Studio debugging could be misleading because dev sometimes served a stale runtime bundle After this change: - the master transport reflects the authored composition-chain duration - master clips resolve linearly across the whole deck - late seeks land on the correct slide window - Studio dev uses the current runtime implementation, so local preview matches the branch you are testing ## Root cause There were two related issues: 1. Studio/master timeline inference lost authored composition timing - missing timing attrs were treated like `0` instead of `null` - non-root composition `data-duration` / `data-end` were stripped before Studio timing resolution could use them - root duration inference trusted an incomplete live timeline window instead of the authored composition chain 2. Preserved authored timing leaked into the general runtime resolver - preserving authored timing was correct for Studio timeline payload generation - but using those preserved attrs for normal runtime playback/render resolution caused visual regressions in producer CI - the follow-up fix keeps authored timing available only for Studio payload collection while normal runtime playback continues to resolve from the real live timeline/media state ## Why the later regression fix was needed The initial runtime change fixed the Apple master timeline, but it also widened timing inference in the core runtime too far. That caused Dockerized producer regressions because rendered visibility started respecting preserved authored timing where it should have relied on the live resolved runtime state. The latest commit fixes that by splitting the behavior: - Studio timeline payload: authored timing allowed - general runtime resolver: authored timing ignored by default That preserves the Apple master timeline fix without changing producer render semantics. ## Verification ### Local checks - `bunx oxlint packages/core/src/runtime/init.ts packages/core/src/runtime/startResolver.ts packages/core/src/runtime/timeline.ts packages/core/src/runtime/startResolver.test.ts packages/core/src/runtime/timeline.test.ts packages/studio/vite.config.ts packages/cli/src/server/studioServer.ts` - `bunx oxfmt --check packages/core/src/runtime/init.ts packages/core/src/runtime/startResolver.ts packages/core/src/runtime/timeline.ts packages/core/src/runtime/startResolver.test.ts packages/core/src/runtime/timeline.test.ts packages/studio/vite.config.ts packages/cli/src/server/studioServer.ts` - `bun run --filter @hyperframes/core typecheck` - `bun run --filter @hyperframes/studio typecheck` - `bun run --filter @hyperframes/cli typecheck` - `cd packages/core && bun run test src/runtime/startResolver.test.ts src/runtime/timeline.test.ts` - `bun test packages/cli/src/server/studioServer.test.ts --timeout 20000` ### Browser proof Tested in Studio with `agent-browser` against the Apple presentation project. - root/master transport now shows `0:00 / 2:21` - master clip manifest resolves sequentially (`slide-1 -> slide-2 -> slide-3 ...`) - seeking to `120s` lands on a late slide instead of a collapsed early timeline state - after refreshing onto the fresh runtime source, the visible later-slide media advanced correctly in local Studio playback ### CI-equivalent regression proof on devbox The previously failing producer regressions were rerun on devbox using the same Dockerized path GitHub Actions uses: - `docker build -f Dockerfile.test -t hyperframes-producer:test .` - `docker run ... hyperframes-producer:test style-1-prod style-5-prod style-9-prod style-12-prod --sequential` Those previously failing suites all passed after the runtime split fix: - `style-1-prod` - `style-5-prod` - `style-9-prod` - `style-12-prod` ## Notes - the Apple project volume tweak stayed local-only for testing and is not part of this PR - this PR fixes the master/root timeline bug and the runtime regression it introduced; it does not add general subtimeline authoring support |
||
|
Vance Ingalls
|
80e7cd2844 |
perf(player): p0-1c live-playback parity test via SSIM (#401)
## Summary Adds **scenario 06: live-playback parity** — the third and final tranche of the P0-1 perf-test buildout (`p0-1a` infra → `p0-1b` fps/scrub/drift → this). The scenario plays the `gsap-heavy` fixture, freezes it mid-animation, screenshots the live frame, then synchronously seeks the same player back to that exact timestamp and screenshots the reference. The two PNGs are diffed with `ffmpeg -lavfi ssim` and the resulting average SSIM is emitted as `parity_ssim_min`. Baseline gate: **SSIM ≥ 0.95**. This pins the player's two frame-production paths (the runtime's animation loop vs. `_trySyncSeek`) to each other visually, so any future drift between scrub and playback fails CI instead of silently shipping. ## Motivation `<hyperframes-player>` produces frames two different ways: 1. **Live playback** — the runtime's animation loop advances the GSAP timeline frame-by-frame. 2. **Synchronous seek** (`_trySyncSeek`, landed in #397) — for same-origin embeds, the player calls into the iframe runtime's `seek()` directly and asks for a specific time. These paths must agree. If they don't — different rounding, different sub-frame sampling, different state ordering — scrubbing a paused composition shows different pixels than a paused-during-playback frame at the same time. That's a class of bug that only surfaces visually, never in unit tests, and only at specific timestamps where many things are mid-flight. `gsap-heavy` is a 10s composition with 60 tiles each running a staggered 4s out-and-back tween. At t=5.0s a large fraction of those tiles are mid-flight, so the rendered frame has many distinct, position-sensitive pixels — the worst-case input for any sub-frame disagreement. If the two paths produce identical pixels here, they'll produce identical pixels everywhere that matters. ## What changed - **`packages/player/tests/perf/scenarios/06-parity.ts`** — new scenario (~340 lines). Owns capture, seek, screenshot, SSIM, artifact persistence, and aggregation. - **`packages/player/tests/perf/index.ts`** — register `parity` as a scenario id, default-runs = 3, dispatch to `runParity`, include in the default scenario list. - **`packages/player/tests/perf/perf-gate.ts`** — extend `PerfBaseline` with `paritySsimMin`. - **`packages/player/tests/perf/baseline.json`** — `paritySsimMin: 0.95`. - **`.github/workflows/player-perf.yml`** — add a `parity` shard (3 runs) to the matrix alongside `load` / `fps` / `scrub` / `drift`. ## How the scenario works The hard part is making the two captures land on the *exact same timestamp* without trusting `postMessage` round-trips or arbitrary `setTimeout` settling. 1. **Install an iframe-side rAF watcher** before issuing `play()`. The watcher polls `__player.getTime()` every animation frame and, the first time `getTime() >= 5.0`, calls `__player.pause()` *from inside the same rAF tick*. `pause()` is synchronous (it calls `timeline.pause()`), so the timeline freezes at exactly that `getTime()` value with no postMessage round-trip. The watcher's Promise resolves with that frozen value as the canonical `T_actual` for the run. 2. **Confirm `isPlaying() === true`** via `frame.waitForFunction` before awaiting the watcher. Without this, the test can hang if `play()` hasn't kicked the timeline yet. 3. **Wait for paint** — two `requestAnimationFrame` ticks on the host page. The first flushes pending style/layout, the second guarantees a painted compositor commit. Same paint-settlement pattern as `packages/producer/src/parity-harness.ts`. 4. **Screenshot the live frame** — `page.screenshot({ type: "png" })`. 5. **Synchronously seek to `T_actual`** — call `el.seek(capturedTime)` on the host page. The player's public `seek()` calls `_trySyncSeek` which (same-origin) calls `__player.seek()` synchronously, so no postMessage await is needed. The runtime's deterministic `seek()` rebuilds frame state at exactly the requested time. 6. **Wait for paint** again, screenshot the reference frame. 7. **Diff with ffmpeg** — `ffmpeg -hide_banner -i reference.png -i actual.png -lavfi ssim -f null -`. ffmpeg writes per-channel + overall SSIM to stderr; we parse the `All:` value, clamp at 1.0 (ffmpeg occasionally reports 1.000001 on identical inputs), and treat it as the run's score. 8. **Persist artifacts** under `tests/perf/results/parity/run-N/` (`actual.png`, `reference.png`, `captured-time.txt`) so CI can upload them and so a failed run is locally reproducible. Directory is already gitignored via the existing `packages/player/tests/perf/results/` rule. ### Aggregation `min()` across runs, **not** mean. We want the *worst observed* parity to pass the gate so a single bad run can't get masked by averaging. Both per-run scores and the aggregate are logged. ### Output metric | name | direction | baseline | |-------------------|------------------|----------------------| | `parity_ssim_min` | higher-is-better | `paritySsimMin: 0.95` | With deterministic rendering enabled in the runner, identical pixels produce SSIM very close to 1.0; the 0.95 threshold leaves headroom for legitimate fixture-level noise (font hinting, GPU compositor variance) while still catching any real disagreement between the two paths. ## Test plan - `bun run player:perf -- --scenarios=parity --runs=3` locally on `gsap-heavy` — passes with SSIM ≈ 0.999 across all 3 runs. - Inspected `results/parity/run-1/actual.png` and `reference.png` side-by-side — visually identical. - Inspected `captured-time.txt` to confirm `T_actual` lands just past 5.0s (within one frame). - Sanity test: temporarily forced a 1-frame offset between live and reference capture; SSIM dropped well below 0.95 as expected, confirming the threshold catches real drift. - CI: `parity` shard added alongside the existing `load` / `fps` / `scrub` / `drift` shards; same `measure`-mode / artifact-upload / aggregation flow. - `bunx oxlint` and `bunx oxfmt --check` clean on the new scenario. ## Stack This is the top of the perf stack: 1. #393 `perf/x-1-emit-performance-metric` — performance.measure() emission 2. #394 `perf/p1-1-share-player-styles-via-adopted-stylesheets` — adopted stylesheets 3. #395 `perf/p1-2-scope-media-mutation-observer` — scoped MutationObserver 4. #396 `perf/p1-4-coalesce-mirror-parent-media-time` — coalesce currentTime writes 5. #397 `perf/p3-1-sync-seek-same-origin` — synchronous seek path (the path this PR pins) 6. #398 `perf/p3-2-srcdoc-composition-switching` — srcdoc switching 7. #399 `perf/p0-1a-perf-test-infra` — server, runner, perf-gate, CI 8. #400 `perf/p0-1b-perf-tests-for-fps-scrub-drift` — fps / scrub / drift scenarios 9. **#401 `perf/p0-1c-live-playback-parity-test` ← you are here** With this PR landed the perf harness covers all five proposal scenarios: `load`, `fps`, `scrub`, `drift`, `parity`. |
||
|
Vance Ingalls
|
6f05fabbf8 |
perf(player): p0-1b perf tests for fps, scrub latency, and media sync drift (#400)
## Summary Second slice of `P0-1` from the player perf proposal: plugs the three steady-state scenarios — sustained playback FPS, scrub latency, and media-sync drift — into the perf gate that landed in #399. Adds the multi-video fixture they all share, wires three new shards into CI, and seeds one new baseline (`droppedFramesMax`). ## Why #399 stood up the harness and proved it with a single load-time scenario. By itself that's enough to catch regressions in initial composition setup, but it can't catch the things players actually fail at in production: - **FPS regressions** — a render-loop change that drops the ticker from 60 to 45 fps still loads fast. - **Scrub latency regressions** — the inline-vs-isolated split (#397) is exactly the kind of code path where a refactor can silently push everyone back to the postMessage round trip. - **Media drift** — runtime mirror logic (#396 in this stack) and per-frame scheduling tweaks can both cause video to slip out of sync with the composition clock without producing a single console error. Each of these is a target metric in the proposal with a concrete budget. This PR turns those budgets into gated CI signals and produces continuous data for them on every player/core/runtime change. ## What changed ### Fixture — `packages/player/tests/perf/fixtures/10-video-grid/` - `index.html`: 10-second composition, 1920×1080, 30 fps, with 10 simultaneously-decoding video tiles in a 5×2 grid plus a subtle GSAP scale "breath" on each tile (so the rAF/RVFC loops have real work to do without GSAP dominating the budget the decoder needs). - `sample.mp4`: small (~190 KB) clip checked in so the fixture is hermetic — no external CDN dependency, identical bytes on every run. - Same `data-composition-id="main"` host pattern as `gsap-heavy`, so the existing harness loader works without changes. ### `02-fps.ts` — sustained playback frame rate - Loads `10-video-grid`, calls `player.play()`, samples `requestAnimationFrame` callbacks inside the iframe for 5 s. - Crucial sequencing: install the rAF sampler **before** `play()`, wait for `__player.isPlaying() === true`, **then reset the sample buffer** — otherwise the postMessage round-trip ramp-up window drags the average down by 5–10 fps. - FPS = `(samples − 1) / (lastTs − firstTs in s)`; uses rAF timestamps (the same ones the compositor saw) rather than wall-clock `setTimeout`, so we're measuring real frame production. - Dropped-frame definition matches Chrome DevTools: gap > 1.5× (1000/60 ms) ≈ 25 ms = "missed at least one vsync." - Aggregation across runs: `min(fps)` and `max(droppedFrames)` — worst case wins, since the proposal asserts a floor on fps and a ceiling on drops. - Emits `playback_fps_min` (higher-is-better, baseline `fpsMin = 55`) and `playback_dropped_frames_max` (lower-is-better, baseline `droppedFramesMax = 3`). ### `04-scrub.ts` — scrub latency, inline + isolated - Loads `10-video-grid`, pauses, then issues 10 seek calls in two batches: first the synchronous **inline** path (`<hyperframes-player>`'s default same-origin `_trySyncSeek`), then the **isolated** path (forced by replacing `_trySyncSeek` with `() => false`, which makes the player fall back to the postMessage `_sendControl("seek")` bridge that cross-origin embeds and pre-#397 builds use). - Inline runs first so the isolated mode's monkey-patch can't bleed back into the inline samples. - Detection: a rAF watcher inside the iframe polls `__player.getTime()` until it's within `MATCH_TOLERANCE_S = 0.05 s` of the requested target. Tolerance exists because the postMessage bridge converts seconds → frame number → seconds, and that round-trip can introduce sub-frame quantization drift even for targets on the canonical fps grid. - Timing: `performance.timeOrigin + performance.now()` in both contexts. `timeOrigin` is consistent across same-process frames, so `t1 − t0` is a true wall-clock latency, not a host-only or iframe-only stopwatch. - Targets alternate forward/backward (`1.0, 7.0, 2.0, 8.0, 3.0, 9.0, 4.0, 6.0, 5.0, 0.5`) so no two consecutive seeks land near each other — protects the rAF watcher from matching against a stale `getTime()` value before the seek command is processed. - Aggregation: `percentile(95)` across the pooled per-seek latencies from every run. With 10 seeks × 2 modes × 3 runs we get 30 samples per mode per CI shard, enough for a stable p95. - Emits `scrub_latency_p95_inline_ms` (lower-is-better, baseline `scrubLatencyP95InlineMs = 33`) and `scrub_latency_p95_isolated_ms` (lower-is-better, baseline `scrubLatencyP95IsolatedMs = 80`). ### `05-drift.ts` — media sync drift - Loads `10-video-grid`, plays 6 s, instruments **every** `video[data-start]` element with `requestVideoFrameCallback`. Each callback records `(compositionTime, actualMediaTime)` plus a snapshot of the clip transform (`clipStart`, `clipMediaStart`, `clipPlaybackRate`). - Drift = `|actualMediaTime − ((compTime − clipStart) × clipPlaybackRate + clipMediaStart)|` — the same transform the runtime applies in `packages/core/src/runtime/media.ts`, snapshotted once at sampler install so the per-frame work is just subtract + multiply + abs. - Sustain window is 6 s (not the proposal's 10 s) because the fixture composition is exactly 10 s long and we want headroom before the end-of-timeline pause/clamp behavior. With 10 videos × ~25 fps × 6 s we still pool ~1500 samples per run — more than enough for a stable p95. - Same "reset buffer after play confirmed" gotcha as `02-fps.ts`: frames captured during the postMessage round-trip would compare a non-zero `mediaTime` against `getTime() === 0` and inflate drift by hundreds of ms. - Aggregation: `max()` and `percentile(95)` across the pooled per-frame drifts. The proposal's max-drift ceiling of 500 ms is intentional — the runtime hard-resyncs when `|currentTime − relTime| > 0.5 s`, so a regression past 500 ms means the corrective resync kicked in and the viewer saw a jump. - Emits `media_drift_max_ms` (lower-is-better, baseline `driftMaxMs = 500`) and `media_drift_p95_ms` (lower-is-better, baseline `driftP95Ms = 100`). ### Wiring - `packages/player/tests/perf/index.ts`: add `fps`, `scrub`, `drift` to `ScenarioId`, `DEFAULT_RUNS`, the default scenario list (`--scenarios` defaults to all four), and three new dispatch branches. - `packages/player/tests/perf/perf-gate.ts`: add `droppedFramesMax: number` to `PerfBaseline`. Other baseline keys for these scenarios were already seeded in #399. - `packages/player/tests/perf/baseline.json`: add `droppedFramesMax: 3`. - `.github/workflows/player-perf.yml`: three new matrix shards (`fps` / `scrub` / `drift`) at `runs: 3`. Same `paths-filter` and same artifact-upload pattern as the `load` shard, so the summary job aggregates them automatically. ## Methodology highlights These three patterns recur in all three scenarios and are worth noting because they're load-bearing for the numbers we report: 1. **Reset buffer after play-confirmed.** The `play()` API is async (postMessage), so any samples captured before `__player.isPlaying() === true` belong to ramp-up, not steady-state. Both `02-fps` and `05-drift` clear `__perfRafSamples` / `__perfDriftSamples` *after* the wait. Without this, fps drops 5–10 and drift inflates by hundreds of ms. 2. **Iframe-side timing.** All three scenarios time inside the iframe (`performance.timeOrigin + performance.now()` for scrub, rAF/RVFC timestamps for fps/drift) rather than host-side. The iframe is what the user sees; host-side timing would conflate Puppeteer's IPC overhead with real player latency. 3. **Stop sampling before pause.** Sampler is deactivated *before* `pause()` is issued, so the pause command's postMessage round-trip can't perturb the tail of the measurement window. ## Test plan - [x] Local: `bun run player:perf` runs all four scenarios end-to-end on the 10-video-grid fixture. - [x] Each scenario produces metrics matching its declared `baselineKey` so `perf-gate.ts` can find them. - [x] Typecheck, lint, format pass on the new files. - [x] Existing player unit tests untouched (no production code changes in this PR). - [ ] First CI run will confirm the new shards complete inside the workflow timeout and that the summary job picks up their `metrics.json` artifacts. ## Stack Step `P0-1b` of the player perf proposal. Builds on: - `P0-1a` (#399): the harness, runner, gate, and CI workflow this PR plugs new scenarios into. Followed by: - `P0-1c` (#401): `06-parity` — live playback frame vs. synchronously-seeked reference frame, compared via SSIM, on the existing `gsap-heavy` fixture from #399. |