HYPERFRAMES_NATIVE_RENDER=pixel-perfect captures full-frame Chrome
screenshots at 30fps during video-active periods. Non-video periods
use sparse states (every 2.5s). Videos are shown/hidden dynamically
at video/non-video boundaries.
Each video-period frame is Chrome's exact compositor output —
pixel-identical to what a CDP screenshot produces at that timestamp.
No video overlay compositing needed; Rust just encodes the frames.
Fast mode (=1) unchanged: sparse states + baked overlay at 13x.
Pixel-perfect mode (=pixel-perfect): ~4-8x depending on video %.
Reverts to sparse states (57) + video overlay architecture (12.5x
speed). Adds video-only timeline baking (405ms) for per-frame
bounds, opacity, and transform on video elements. De-duplicates
overlays by ID. Carries object_fit and other rendering styles.
PSNR improvement: 20.1 → 23.2 dB overall (+3.1 dB).
Video frame PSNR: +5.6 dB on t=30s (10.7 → 16.3 dB).
Speed: 32.4s = 12.5x (141s composition, 4240 frames).
Adds bakeVideoTimeline, which evaluates GSAP animation state for a
specific set of video element IDs (rather than all DOM elements),
using batches of 50 frames per CDP call for efficiency.
Transform fields are forced to identity values (translate=0, scale=1,
rotate=0) because getBoundingClientRect returns screen-space coordinates
that already include CSS transforms — applying both would double-apply
the transform in the Rust painter.
- Hide video elements during Chrome state capture for clean backgrounds
- Collect video element bounds/timing before scene replacement
- Overlay extracted video frames (from FFmpeg) on top of state frames
- State capture uses ~29 evenly-spaced timestamps (not per-frame)
Results (Apple presentation 141s, 7 slides, 5 videos):
Speed: 38s (10.7x faster than Chrome 405s)
PSNR non-video slides: 35-39dB (pixel-perfect)
PSNR video slides: 14-20dB (video frame overlays working but timing gap)
PSNR overall: 5/8 above 30dB, improving
For compositions with video elements, capture EVERY frame from Chrome
(pixel-perfect by definition) and encode natively with openh264+minimp4.
For compositions without video, use state-based capture (29 states).
Apple presentation PSNR: 6/8 checkpoints above 30dB (near pixel-perfect)
t=15s: 32.5dB (was 20dB)
t=50-135s: 35-39dB (pixel-perfect)
t=2s, t=30s: 12-13dB (video element first-frame timing)
Speed: 3:37 (217s) vs Chrome 6:45 (405s) = 1.86x faster
Instead of capturing elements at one timestamp, capture the FULL FRAME
from Chrome at N evenly-spaced timestamps (10 states for 141s video).
The Rust binary displays the correct state frame for each render frame
based on data_start/data_end time ranges.
Results on Apple presentation (141s, 7 slides):
Speed: 35s total (vs Chrome 405s = 11.6x faster)
PSNR: 5 of 8 checkpoints above 35dB (near pixel-perfect)
Before: only 2 of 8 above 30dB
Remaining gap: slides with video at timestamps between state captures
show the wrong video frame. Increasing state count will close this.
Video frame compositing is confirmed working:
- frame paths resolve correctly (exists=true)
- frames load per-timestamp (frame_00001, frame_00101, frame_00201)
- video_frames_dir is parsed by serde
Remaining issue: identical frames at t=30 and t=70 (same MD5 hash)
indicates the baked timeline deltas aren't animating elements across
slide transitions. The scene composition appears frozen for some slides.
This is a timeline baking fidelity issue, not a video compositing issue.
The element capture converts Video elements to Image type (for Chrome PNG
capture). The Rust painter now checks style.video_frames_dir on Image
elements and uses per-frame extracted JPEGs when present, falling back to
the static Chrome PNG for that element.
PSNR results on Apple presentation (141s, 7 slides, 5 videos):
Slides WITHOUT video: 35.7dB (above 30dB imperceptible threshold)
Slides WITH video: 11-15dB (video frames partially mapped — 3/5 videos)
The remaining gap is from cinematic-vid and avatar-vid which are in
sub-compositions and may not have frame directories at the expected paths.
Three fixes for the Apple presentation render:
1. Skip HTTP/HTTPS URLs in Rust ImageCache — the file server is closed
by the time the binary runs. curl hangs waiting for localhost.
2. Reuse single FontMgr in FontRegistry (was creating 88 instances).
3. Skip fonts in scene JSON since text is pre-rendered as Chrome PNGs.
4. Add --cpu flag to orchestrator binary invocation.
5. Add debug logging to render_native binary for hang diagnosis.
Root cause found: Image elements with `src: "http://localhost:PORT/..."`
(file server URLs) cause the binary to hang when file server is closed.
Tests for video, unstable IDs, wrapped text, grid text, multiple shadows,
and transparent roots were expecting rejection but these features are now
accepted. Updated test expectations to match relaxed support detector.
49 Rust tests + 26 TypeScript tests pass.
Capture Chrome's pixel-perfect text rendering as transparent PNGs during
scene extraction. Text elements are converted to Image elements in the
scene JSON, so the Rust painter draws Chrome's actual glyphs instead of
re-rendering with Skia (which produces different anti-aliasing).
Also extends BakedElementState with per-frame bounds, colors, and
border-radius for higher fidelity on animated properties.
PSNR improvement on missing-host-comp-id fixture:
Before: 19.1dB (visible differences)
After: 44.2dB (near pixel-perfect)
30dB = imperceptible. 44dB = nearly identical.
- extract embedded @font-face base64 fonts from Chrome, write to disk
- pass font file paths in scene JSON for Rust FontRegistry
- extract padding-left/top and text-align for text positioning
- apply padding offsets and text alignment in Skia text painter
Video compositing:
- Style gains data_start, data_end, video_frames_dir, video_fps, video_media_start
- Painter skips elements outside their active time range
- Video elements load pre-extracted frames from disk by timestamp
- Scene extractor captures video metadata from DOM attributes
Font registry:
- FontRegistry loads custom fonts from disk (TTF/OTF) via Skia FontMgr
- paint_element_at_time_with_fonts uses registry for font-family matching
- Pipeline builds font registry from scene.fonts descriptors
All 49 tests pass.
Wire pre-extracted video frames into the Rust paint pipeline:
- Add data_start, data_end, video_frames_dir, video_fps, and
video_media_start fields to the Style struct so the scene JSON can
carry element timing and video frame directory metadata.
- Update paint_element_inner to skip elements outside their
data_start..data_end range, and prefer loading pre-extracted JPEG
frames from a directory (frame_00001.jpg, ...) over the FFmpeg
single-frame fallback path.
- Update the TypeScript CDP scene extractor to capture data-start,
data-end, and video metadata attributes from DOM elements, and
expose an injectVideoFramesMeta() helper the producer can call
after frame extraction to populate frame directory paths.
- Fix FontRegistry to use FontMgr::new_from_data() instead of the
removed Typeface::from_data() in skia-safe 0.93.
- Sync extract.ts types and font collection with Scene/Style changes.
- Clean up unused import and variable warnings in pipeline.rs.
The native render path requires a Chrome session for scene extraction +
timeline baking. When the composition has a known static duration, the
probe session is null (Chrome was never launched). Now creates a
dedicated session when HYPERFRAMES_NATIVE_RENDER=1 is set.
Tested: missing-host-comp-id fixture renders natively at 1.29ms/frame.
Full pipeline: compile → extract scene → bake timeline → Skia paint →
openh264 encode → audio mux → final MP4.
- Remove unstable ID check (extractor generates positional IDs)
- Allow multiple box-shadows (Rust painter loops)
- Allow video elements with resolved sources (frame extraction path)
- Allow wrapped text and flex/grid text (Skia paragraph API)
- Allow transparent composition roots (renderer clears to black)
- Add __name polyfill for tsx ESM compat in CLI entry + nativeRender
Tested: Apple presentation passes support detection and renders through
the native pipeline. Output is a valid MP4 (sub-composition resolution
needs producer compilation for full multi-slide rendering).
Two optimizations to the render-only path:
1. Write BGRA directly (Skia's native format) instead of converting
to I420 during render. FFmpeg batch-converts in the encode step.
2. Pre-allocate BGRA buffer once, reuse across frames (zero allocation
per frame via read_pixels_bgra_into).
macOS benchmark: 116.6ms/30frames (3.89ms/frame), down from 153ms (5.1ms).
Expected Linux CI: ~1.8ms/frame (paint) + ~0.1ms (readback) = ~1.9ms/frame.
That's 7.4-21x faster than Chrome CDP.
Separate rendering from encoding for maximum render throughput:
- render_animated_raw_then_encode: paint→I420→file, then FFmpeg batch
- render_only benchmark: pure paint+convert speed without any encode
Benchmark (macOS, 30 frames, 1080p, CPU raster):
render only (paint+I420): 153ms (5.1ms/frame)
raw render + FFmpeg batch: 308ms (10.3ms/frame)
native openh264 in-process: 331ms (11.0ms/frame)
FFmpeg VideoToolbox pipe: 348ms (11.6ms/frame)
Render-only is 2.7-7.8x faster than Chrome CDP (14-40ms).
Rust-native encoding: Skia paint → dcv BGRA→I420 → openh264 H.264 → minimp4 MP4.
No subprocess, no pipe, no FFmpeg for encoding. Inspired by MediaBunny's
browser-side approach (WebCodecs + MP4 muxer) but in Rust.
Benchmark (macOS, 30 frames, 1080p):
native (openh264+minimp4): 331ms (11.0ms/frame)
ffmpeg (VideoToolbox): 348ms (11.6ms/frame)
ffmpeg (JPEG pipe): 555ms (18.5ms/frame)
The native path is ~5% faster than FFmpeg VideoToolbox on macOS.
On Linux without hardware encoders, the gap should be larger since
we skip FFmpeg subprocess startup + pipe serialization entirely.
Cargo feature split: metal-gpu (macOS), vulkan-gpu (Linux), no-default (CPU).
Docker builds with --no-default-features for pure CPU raster path.
All GPU code gated behind feature flags — no compilation errors on Linux.
Encoder detection probes actual encoder functionality (not just ffmpeg list).
Tested: docker run hyperframes-native:test → 51/51 pass on Linux ARM64.
Dockerfile.test for CI: builds Skia from source on Linux ARM64/x86,
runs all 49 Rust tests with CPU raster + FFmpeg libx264 fallback.
Includes ninja, python3, clang for Skia source build, plus fonts
and FFmpeg for rendering tests.
Supports: `docker run hyperframes-native:test` (tests)
`docker run hyperframes-native:test bench` (benchmarks)
- add Skia Vulkan backend for Linux GPU instances (A10G, T4, L4)
- new_gpu_or_raster() auto-detects best GPU backend, falls back to CPU
- remove macOS-only guards from pipeline, benchmarks, and tests
- render_animated_gpu works on all platforms (Vulkan/Metal/raster fallback)
- flush_and_submit is now cross-platform
- all 49 tests pass
Production path: Linux NVIDIA → Skia Vulkan paint → NVENC hw encode
Docker CI path: Linux no GPU → Skia CPU raster → FFmpeg libx264
Local dev path: macOS → Skia Metal paint → VideoToolbox hw encode
- box-shadow with offset, spread, blur radius, border-radius aware
- filter:blur via Skia ImageFilter + SaveLayerRec
- linear-gradient and radial-gradient via Skia gradient shaders
- gradient takes priority over solid background when both present
- 7 new tests (26 total), all passing
Add timeline baking module that uses Chrome CDP to evaluate a GSAP
timeline at every frame timestamp, extracting per-frame transform,
opacity, and visibility for all animated elements. The output JSON
uses snake_case field names compatible with Rust serde deserialization,
bridging GSAP animations to the native renderer without embedding V8.