mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-05 05:37:09 +00:00
Miguel ÁngelandKenneth Shortrede
c9b3d9c962
* fix(engine): declare SDR extracted frames as sRGB so Chrome does not darken footage * fix(engine): encode SDR mp4 frames with the BT.709 matrix they are tagged with Chrome hands the SDR encoder BT.601 full-range JPEG captures (or RGB PNGs). The encoders only converted the range, so swscale kept the BT.601 matrix under the BT.709 tag (ffmpeg 5.1 to 7.0: CSS #C83C28 delivered as #D44824) or converted YUV to YUV through a low-precision step (ffmpeg 8+: #101010 delivered as #0B0D0D). Decode to RGB with the capture matrix, then encode with BT.709. Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * fix(engine): tag SDR frames plain sRGB so every ffmpeg writes the sRGB chunk ffmpeg writes a PNG sRGB chunk only for BT.709 primaries and writes cICP only from 6.1. On ffmpeg 5.1 (Debian bookworm) a BT.601-tagged source therefore got cHRM plus gAMA 1/2.2, which Chrome decodes as a pure 2.2 power: 0x10 shadows rendered as 0x07. Declaring BT.709 primaries too gives the sRGB chunk on every version, so Chrome shows the frame code values for every SDR source. The test now reads the PNG chunks instead of ffprobe, which reports no transfer for an sRGB chunk on ffmpeg 5.1. Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * fix(engine): extract SDR jpg frames as the BT.601 full range JPEG readers assume The default frame format for opaque SDR sources is JPEG. ffmpeg converted the BT.709 source to full-range YUV without changing the matrix, and every JPEG reader (Chrome included) decodes BT.601, so footage shifted: 0xC83C28 showed as 0xBB2D29. The old mp4 encode made the inverse mistake and hid it; with the encode fixed, it would reach the output. Decode through RGB with the source matrix, then encode BT.601 full range, keeping 4:2:0. Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * fix(engine): round the SDR mp4 colour conversion accurately swscale's default rounding moved saturated colours up to 3 codes (pure red 254 delivered as 251 on ffmpeg 5.1 to 7.0); accurate_rnd brings that to 1. The chunk encoder now decides the HDR-or-SDR filter once, and the 10-bit GPU pad test sets hdr, which its name always claimed. * style(engine): keep the colour filter reasons in names and tests, not source comments The comment ratchet caps a file's comment share: the new filter module and the extractor carry their intent in constant names, and the encode test states why the conversion goes through RGB. * fix(engine): keep colour edges sharp through the SDR colour conversions Going through RGB upsampled chroma with swscale's default filter, so ffmpeg 4.4 to 7.x smeared colour across edges by about 2 px (red 254 next to blue came out 8 inside the blue). Nearest-neighbour resampling maps the half-resolution chroma back one to one, keeps the flat-colour results, and makes accurate_rnd redundant. The pixel tests now also check a red to blue edge. * test(producer): regenerate goldens whose colours the old mp4 matrix bent These four fixtures compare at 40 to 45 dB, and their goldens were rendered with the BT.601-under-BT.709 encode, so the corrected colours score 34 to 37 dB. Regenerated in the Dockerfile.test image. Each new golden is closer to the colour its page declares, e.g. #00d4aa was 00bcab and is now 00d3aa, and #087443 was 006643 and is now 087241. * fix(engine): keep VFR frame timing when extraction applies colour filters ffmpeg before 6.1 ignores frame durations in CFR resampling once any -vf is set, so a VFR window ending on a still lost frames. Filtered VFR extraction now resamples in one process exactly as before and filters in a second. Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * fix(engine): encode raw SDR layered frames in BT.709 Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * test(engine): keep the new colour tests inside the format, comment and argv checks * fix(engine): keep HDR and unusual pixel formats out of the two-process VFR path The raw handoff dropped HDR10 light levels and misread semi-planar formats, and decoder options rejected some ffprobe colour names. Only SDR sources in formats nut carries unchanged now take it, their colour tags come back through setparams, and a process that cannot start releases the other. Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * fix(engine): read untagged footage with the matrix Chrome plays it with Chrome decodes an untagged matrix as BT.709 from 720 coded lines up and as BT.601 below; extraction assumed BT.601 at every size, so HD footage without colour tags shifted hue against Studio's own preview. Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * fix(engine): untagged VP9 and AV1 footage keeps BT.601, as Chrome plays it Chrome applies the 720-line rule only to codecs it decodes through ffmpeg (H.264, VP8); untagged VP9, VP9 alpha and AV1 play as BT.601 at every size. Also widens the raw-pipe pixel formats to all verified ones, and reports the pipeline process that failed first by settle order, not by per-process clocks. Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com> * test(producer): regenerate goldens for corrected page colours and untagged SD footage * docs(engine): say which untagged codecs the matrix rule was measured on --------- Co-authored-by: Kenneth Shortrede <95665480+sontiac@users.noreply.github.com>
@hyperframes/engine
Seekable web-page-to-video rendering engine built on Puppeteer and FFmpeg.
Framework-agnostic: works with GSAP, Lottie, Three.js, CSS animations, or any web content that implements the window.__hf seek protocol.
Install
npm install @hyperframes/engine
Requirements: Node.js >= 22, Chrome/Chromium (auto-downloaded by Puppeteer), FFmpeg
What it does
The engine opens your HTML composition in a headless Chrome instance, seeks frame-by-frame using Chrome's HeadlessExperimental.beginFrame API, captures screenshots, and encodes them into video with FFmpeg.
Key services
| Service | Description |
|---|---|
| browserManager | Launches and pools headless Chrome instances (chrome-headless-shell) |
| frameCapture | Manages capture sessions — seek, screenshot, buffer lifecycle |
| screenshotService | BeginFrame-based capture with CDP (Chrome DevTools Protocol) |
| chunkEncoder | FFmpeg encoding with chunked concat, GPU detection, faststart |
| streamingEncoder | Pipe frames to FFmpeg in real time (no intermediate PNGs on disk) |
| audioMixer | Parse <audio> elements and mix audio tracks via FFmpeg |
| videoFrameExtractor | Extract frames from <video> elements for compositing |
| parallelCoordinator | Split frame ranges across worker processes |
| fileServer | Serve local HTML files to the browser via Hono |
Usage
import {
acquireBrowser,
createCaptureSession,
initializeSession,
captureFrame,
closeCaptureSession,
} from "@hyperframes/engine";
// 1. Launch browser
const browserLease = await acquireBrowser({ captureMode: "beginFrame" });
// 2. Open a capture session
const session = createCaptureSession({
browser: browserLease.browser,
url: "http://localhost:3000/my-composition.html",
width: 1920,
height: 1080,
fps: 30,
});
await initializeSession(session);
// 3. Capture frames
for (let i = 0; i < totalFrames; i++) {
await captureFrame(session, i, `/tmp/frames/frame-${i}.png`);
}
// 4. Clean up
await closeCaptureSession(session);
await browserLease.release();
Most users should use @hyperframes/producer or the hyperframes CLI instead of calling the engine directly.
Documentation
Full documentation: hyperframes.heygen.com/packages/engine
Related packages
@hyperframes/core— types, parsers, frame adapters@hyperframes/producer— high-level render pipeline built on this enginehyperframes— CLI