- extract embedded @font-face base64 fonts from Chrome, write to disk
- pass font file paths in scene JSON for Rust FontRegistry
- extract padding-left/top and text-align for text positioning
- apply padding offsets and text alignment in Skia text painter
Video compositing:
- Style gains data_start, data_end, video_frames_dir, video_fps, video_media_start
- Painter skips elements outside their active time range
- Video elements load pre-extracted frames from disk by timestamp
- Scene extractor captures video metadata from DOM attributes
Font registry:
- FontRegistry loads custom fonts from disk (TTF/OTF) via Skia FontMgr
- paint_element_at_time_with_fonts uses registry for font-family matching
- Pipeline builds font registry from scene.fonts descriptors
All 49 tests pass.
Wire pre-extracted video frames into the Rust paint pipeline:
- Add data_start, data_end, video_frames_dir, video_fps, and
video_media_start fields to the Style struct so the scene JSON can
carry element timing and video frame directory metadata.
- Update paint_element_inner to skip elements outside their
data_start..data_end range, and prefer loading pre-extracted JPEG
frames from a directory (frame_00001.jpg, ...) over the FFmpeg
single-frame fallback path.
- Update the TypeScript CDP scene extractor to capture data-start,
data-end, and video metadata attributes from DOM elements, and
expose an injectVideoFramesMeta() helper the producer can call
after frame extraction to populate frame directory paths.
- Fix FontRegistry to use FontMgr::new_from_data() instead of the
removed Typeface::from_data() in skia-safe 0.93.
- Sync extract.ts types and font collection with Scene/Style changes.
- Clean up unused import and variable warnings in pipeline.rs.
The native render path requires a Chrome session for scene extraction +
timeline baking. When the composition has a known static duration, the
probe session is null (Chrome was never launched). Now creates a
dedicated session when HYPERFRAMES_NATIVE_RENDER=1 is set.
Tested: missing-host-comp-id fixture renders natively at 1.29ms/frame.
Full pipeline: compile → extract scene → bake timeline → Skia paint →
openh264 encode → audio mux → final MP4.
- Remove unstable ID check (extractor generates positional IDs)
- Allow multiple box-shadows (Rust painter loops)
- Allow video elements with resolved sources (frame extraction path)
- Allow wrapped text and flex/grid text (Skia paragraph API)
- Allow transparent composition roots (renderer clears to black)
- Add __name polyfill for tsx ESM compat in CLI entry + nativeRender
Tested: Apple presentation passes support detection and renders through
the native pipeline. Output is a valid MP4 (sub-composition resolution
needs producer compilation for full multi-slide rendering).
Two optimizations to the render-only path:
1. Write BGRA directly (Skia's native format) instead of converting
to I420 during render. FFmpeg batch-converts in the encode step.
2. Pre-allocate BGRA buffer once, reuse across frames (zero allocation
per frame via read_pixels_bgra_into).
macOS benchmark: 116.6ms/30frames (3.89ms/frame), down from 153ms (5.1ms).
Expected Linux CI: ~1.8ms/frame (paint) + ~0.1ms (readback) = ~1.9ms/frame.
That's 7.4-21x faster than Chrome CDP.
Separate rendering from encoding for maximum render throughput:
- render_animated_raw_then_encode: paint→I420→file, then FFmpeg batch
- render_only benchmark: pure paint+convert speed without any encode
Benchmark (macOS, 30 frames, 1080p, CPU raster):
render only (paint+I420): 153ms (5.1ms/frame)
raw render + FFmpeg batch: 308ms (10.3ms/frame)
native openh264 in-process: 331ms (11.0ms/frame)
FFmpeg VideoToolbox pipe: 348ms (11.6ms/frame)
Render-only is 2.7-7.8x faster than Chrome CDP (14-40ms).
Rust-native encoding: Skia paint → dcv BGRA→I420 → openh264 H.264 → minimp4 MP4.
No subprocess, no pipe, no FFmpeg for encoding. Inspired by MediaBunny's
browser-side approach (WebCodecs + MP4 muxer) but in Rust.
Benchmark (macOS, 30 frames, 1080p):
native (openh264+minimp4): 331ms (11.0ms/frame)
ffmpeg (VideoToolbox): 348ms (11.6ms/frame)
ffmpeg (JPEG pipe): 555ms (18.5ms/frame)
The native path is ~5% faster than FFmpeg VideoToolbox on macOS.
On Linux without hardware encoders, the gap should be larger since
we skip FFmpeg subprocess startup + pipe serialization entirely.
Cargo feature split: metal-gpu (macOS), vulkan-gpu (Linux), no-default (CPU).
Docker builds with --no-default-features for pure CPU raster path.
All GPU code gated behind feature flags — no compilation errors on Linux.
Encoder detection probes actual encoder functionality (not just ffmpeg list).
Tested: docker run hyperframes-native:test → 51/51 pass on Linux ARM64.
Dockerfile.test for CI: builds Skia from source on Linux ARM64/x86,
runs all 49 Rust tests with CPU raster + FFmpeg libx264 fallback.
Includes ninja, python3, clang for Skia source build, plus fonts
and FFmpeg for rendering tests.
Supports: `docker run hyperframes-native:test` (tests)
`docker run hyperframes-native:test bench` (benchmarks)
- add Skia Vulkan backend for Linux GPU instances (A10G, T4, L4)
- new_gpu_or_raster() auto-detects best GPU backend, falls back to CPU
- remove macOS-only guards from pipeline, benchmarks, and tests
- render_animated_gpu works on all platforms (Vulkan/Metal/raster fallback)
- flush_and_submit is now cross-platform
- all 49 tests pass
Production path: Linux NVIDIA → Skia Vulkan paint → NVENC hw encode
Docker CI path: Linux no GPU → Skia CPU raster → FFmpeg libx264
Local dev path: macOS → Skia Metal paint → VideoToolbox hw encode
- box-shadow with offset, spread, blur radius, border-radius aware
- filter:blur via Skia ImageFilter + SaveLayerRec
- linear-gradient and radial-gradient via Skia gradient shaders
- gradient takes priority over solid background when both present
- 7 new tests (26 total), all passing
Add timeline baking module that uses Chrome CDP to evaluate a GSAP
timeline at every frame timestamp, extracting per-frame transform,
opacity, and visibility for all animated elements. The output JSON
uses snake_case field names compatible with Rust serde deserialization,
bridging GSAP animations to the native renderer without embedding V8.
Rust-based video composition renderer using Skia (Chrome's 2D engine):
- scene graph types with serde JSON parsing
- Skia raster surface with JPEG/PNG encoding
- element painter: backgrounds, border-radius, transforms, opacity, text
- static render pipeline: scene → Skia paint → FFmpeg pipe → MP4
- CDP scene extraction bridge (TypeScript)
- criterion benchmark: 1080p, 20 elements with text
Benchmark results (CPU raster, Apple Silicon):
paint_1080p_20_elements: 29.8ms/frame
paint_and_encode_jpeg_1080p: 35.7ms/frame
CPU raster is comparable to Chrome CDP (14-40ms). The GPU backend
(Metal/Vulkan, Phase 3) is where the 10-50x speedup materializes —
the infrastructure is proven, the bottleneck is CPU vs GPU painting.
Criterion benchmark with a realistic 1080p scene: 20 overlapping
card containers with text children, rounded corners, partial opacity,
and overflow clipping.
Two bench functions:
- paint_1080p_20_elements: clear + recursive paint
- paint_and_encode_jpeg_1080p: clear + paint + JPEG encode
Baseline numbers (Apple Silicon, raster CPU backend):
paint only: ~177 ms
paint + JPEG encode: ~184 ms
The high paint time is dominated by FontMgr::new() being called per
text element per frame (20x per iteration). Font/typeface caching
in the paint path is the obvious next optimization target.
Walks a Chrome page's DOM via Puppeteer and extracts a scene graph as
JSON matching the Rust parse_scene_json() format. Uses kind nested-object
structure with serde-compatible internally-tagged enum discriminator for
Container/Text/Image/Video.
Extracts bounds, computed styles (background, opacity, border-radius,
overflow, transform, visibility, font properties, color), and recurses
through children. Text detection based on text-only child nodes, image
and video detection based on element tag.
Tests verify JSON round-trip compatibility with the Rust scene types
including all element kinds, nested children, and transform serialization.
Wire Scene -> Skia paint -> JPEG encode -> FFmpeg pipe into an
end-to-end render_static function. The scene is painted once and the
JPEG frame is repeated N times via image2pipe, producing an H.264 MP4.
Includes RenderConfig/RenderResult types and two integration tests
(1s@30fps, 0.5s@24fps) that assert frame count, file existence, and
non-trivial output size.
Adds paint_element() that recursively renders a scene graph Element onto
a Skia Canvas. Supports:
- Visibility culling (skip invisible elements)
- Position translation + CSS-like transform (rotate, scale around center)
- Layer-based opacity blending (save_layer_alpha)
- Overflow clipping with rounded rects
- Background fill (rect or rrect depending on border_radius)
- Text rendering via FontMgr default typeface
- Recursive child painting
Helper functions: to_color4f, to_sk_rect, make_rrect.
Four new tests covering background+text, border-radius+opacity pixel
verification, transforms, and invisible element skipping.
Streaming encode is a regression on Linux/headless-shell (confirmed by
both local benchmarks and Rames' Linux VM A/B). The pipe becomes a
bottleneck when capture is already fast — Chrome and FFmpeg compete for
CPU instead of overlapping.
Reverts:
- enableStreamingEncode: true → false
- useMultiPageCapture: true → false (shared WebSocket serializes CDP calls)
- gpuBackend: "hardware" → "swiftshader" (needs real GPU to validate)
All three features remain available as opt-in via config/env vars.
The remaining 6 always-on optimizations (string-eval seek, FFmpeg
threading, parallel audio, hwaccel extraction, worker tuning, lower
PNG compression) are net-positive on all tested fixtures.
9 optimizations targeting the frame capture hot path (86% of total render time):
1. Enable streaming encode by default — overlaps capture with FFmpeg encoding,
eliminating the encode stage as a separate step. Saves ~10% wall time.
2. Multi-page shared browser — screenshot-mode workers share a single Chrome
process instead of launching N separate browsers. Eliminates N-1 Chrome
startup costs (~2-3s each) and shares GPU/cache across pages.
3. Lower streaming JPEG quality (55 vs 80/95) — intermediate frames piped to
FFmpeg are re-encoded, so quality loss is invisible in the final output.
Smaller buffers transfer faster over CDP, cutting per-frame capture time.
4. Hardware GPU acceleration — replace forced SwiftShader (software renderer)
with native GPU (Metal on macOS, Vulkan on Linux). CSS effects like
box-shadow, blur, and transforms render 10-100x faster. Falls back to
SwiftShader automatically in Docker/CI where no GPU exists.
5. Optimized seek evaluation — string eval replaces function serialization
in the per-frame page.evaluate() call, skipping Puppeteer's serialize +
Runtime.callFunctionOn overhead. Removes redundant __hf.seek guard that
was already validated during session initialization.
6. FFmpeg multi-threading — adds -threads 0 across all encoding paths
(streaming, chunk, extraction) to let FFmpeg auto-parallelize.
7. Parallel audio processing — audio extraction now runs concurrently with
frame capture instead of blocking it. Audio uses separate FFmpeg processes
that don't compete with Chrome for CPU.
8. Hardware-accelerated video extraction — adds -hwaccel auto for H.264
sources and reduces PNG compression from level 6 to 1 (temp files).
9. Tuned parallelism defaults — more aggressive worker allocation (8 max,
2 cores/worker, 24 min frames/worker) suited to modern hardware.
All 494 engine tests pass. No regressions.
* fix(lint): remove root_composition_missing_data_duration
Lint cannot statically observe the runtime's true Infinity-emission
condition: it requires a finite GSAP timeline duration AND a finite
media/sub-comp window AND timeline > floor + 1, none of which are
visible to the static linter. The looping shapes that drive the
condition are already covered by `gsap_infinite_repeat` and
`gsap_repeat_ceil_overshoot` (both from #243), which point at the
real authoring mistake — flagging the missing duration separately
was a noisy proxy for the same signal.
Per #490 review discussion, deprecate the static rule and let those
two GSAP rules carry the pre-render coverage. If perfect precision
on `durationInFrames = Infinity` is needed, that belongs on the
runtime/render path where `shouldEmitNonDeterministicInf` is
actually known.
Add regression tests pinning the removal: a docs-compliant root
without `data-duration` no longer warns, and the canonical
loop-inflated shape now surfaces only via `gsap_infinite_repeat`
instead of two duplicate findings.
* docs(skills): update step-6 build checklist after rule removal
Drops the `root_composition_missing_data_duration` reference now that
the rule is gone. Keeps the authoring recommendation (and explains the
runtime Infinity case) but points authors at the GSAP rules that
actually carry the lint signal: `gsap_infinite_repeat` and
`gsap_repeat_ceil_overshoot`.
`lintAudioSrcNotFound` resolves every `<audio src>` against the project
root, which is correct for index.html but wrong for sub-compositions —
their srcs are written relative to the sub-composition file (e.g.
`../assets/foo.mp3`), and the bundler rewrites them at compile time.
Carry the sub-composition path alongside each html source and run
`<audio src>` strings starting with `../` through the existing
`rewriteAssetPath` helper before checking existence on disk. Mirrors the
runtime/bundler behaviour so the lint check sees the same path the
renderer will fetch.
Original (un-rewritten) src is still surfaced in the finding message so
authors can grep for it in their HTML.
The studio shell paints a #0a0a0a body but never declares `color-scheme`,
so browsers render native UA chrome (scrollbars, form controls, focus
rings) in the light palette and the mismatch is obvious — especially
scrollbars, which appear as light tracks sitting on top of a near-black
panel.
Studio doesn't expose a theme toggle; the UI is dark-only. Declaring
`color-scheme: dark` on `:root` lines up the browser-native surfaces
with the rest of the chrome.
* fix(studio): restore flex layout inside the NLE timeline panel
Without `flex flex-col`, the toolbar and Timeline children render as
block elements and Timeline's `h-full` takes the panel's entire height.
The toolbar still occupies its own ~45px of flow, pushing the scroll
area below the viewport and clipping the scrollbar at the bottom of the
right-side panel.
* fix(studio): pin toolbar height with flex-shrink-0 and trim comment
Wrap the toolbar slot in a flex-shrink-0 div so Timeline back-pressure
can't squeeze the toolbar below its natural height. The visible bug
doesn't manifest at current sizes, but in a flex-col container both
children are flex items and the toolbar has no shrink guard of its own.
Trim the inline comment to a single line — the rationale belongs in
the commit message, not at the call site.
* fix(cli): shut down preview embedded-mode server on Ctrl+C
runEmbeddedMode awaited a promise that never resolved, relying on Node
to exit on SIGINT. Two things kept that from working in practice:
1. @hono/node-server's listening handle keeps the event loop alive after
the signal fires, so the process hangs even when SIGINT does arrive.
2. On Windows, some terminals (Git Bash / MSYS) don't deliver Ctrl+C to
the Node process as a SIGINT at all — the keystroke is eaten at the
TTY layer.
Register a SIGINT/SIGTERM handler that closes the server explicitly and
resolves the promise, with a 2s force-exit fallback. On Windows, run a
readline interface on stdin to catch Ctrl+C at the TTY and re-emit it
as SIGINT so the same handler fires.
Print "Shutting down studio..." as soon as the signal is received —
server.close() can take a second or two to drain keep-alive connections
and an unmarked pause reads as "stuck".
Exit code 0 because a user-initiated Ctrl+C isn't an error; non-zero
codes make pnpm print ELIFECYCLE right where the user just asked the
process to stop.
* fix(cli): close readline interface in shutdown to honour exit-code intent
After the SIGINT handler removes itself, the Windows readline interface
is still alive and listening. A second Ctrl+C during the 2s grace
period would re-emit SIGINT with no registered handler, triggering
Node's default exit-130 behaviour and contradicting the explicit
exit(0) we chose for clean teardown.
Hoist the readline handle out of the win32 branch so shutdown can
close it before invoking server.close(). Also pass the signal name
to process.emit("SIGINT", "SIGINT") to match Node's ProcessEvents
overload — runtime behaviour is unchanged.
* fix(engine): suppress benign AbortError spam from frame-capture pageerror
Frame capture pauses → seeks → screenshots → plays audio/video many times
per second. HTMLMediaElement.play() returns a promise that rejects with
AbortError whenever another pause() lands before it resolves — which it
does, every frame. The rejection is benign (output frames and mixed
audio are unaffected) but the frameCapture pageerror handler was logging
it to stderr, producing dozens of identical lines per render:
[Browser:PAGEERROR] AbortError: The play() request was interrupted by
a call to pause(). https://goo.gl/LdLk22
Filter out exactly this pattern before console.error — still pushed to
browserConsoleBuffer so it's available in the failure-diagnostic dump.
* fix(engine): trim play-abort filter comment and drop unnecessary regex flags
The why-it-exists explanation belongs in the commit message and PR
description, not as a 12-line comment block at the call site.
Chrome's play()/pause() AbortError message is always lowercase, so
the case-insensitive flag implies uncertainty that doesn't exist.
Replace the two /play\(\)/i and /pause\(\)/i regexes with plain
String.prototype.includes — same outcome, less ceremony.