Files
hyperframes/packages/producer
Miguel Ángel dc4383113c fix(producer): mix audio into a container that can record encoder delay (#3200)
* fix(producer): mix audio into a container that can record encoder delay

Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz)
after its authored `data-start`, against a frame-accurate video track.

The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix
was written to a raw ADTS `.aac` file, which has nowhere to record that delay,
so it decoded as real leading silence and every stage downstream preserved it
faithfully. Measuring each intermediate localises it precisely: the source WAV
is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and
mux stages inherit it unchanged. The filter graph itself is correct - run by
hand to PCM it lands on the authored start.

Switch the artifact to an MP4-family container, which stores the delay as an
edit list that decoders strip. Same codec, same bitrate, so no size or quality
change.

The filename is a contract shared by three consumers - the mux input, the
distributed plan artifact, and the PNG-sequence sidecar handed to users for
NLE ingest - and its extension is what selects the muxer. Give it one owner in
the engine rather than five literals, so those consumers cannot drift onto
different containers.

Note for reviewers: this renames the distributed plan's audio artifact, which
is an on-disk contract between the plan writer and the assembler. Both move
together here, but a plan written by an older build would not be found by a
newer assembler. Flagging in case that mixed-version window matters for how
these are deployed.

* fix(cloud): read the plan audio artifact name from the producer contract

The aws-lambda and gcp-cloud-run adapters each restated the plan's audio
filename in five places, so renaming it in the producer left them looking for a
file that is no longer written. CI caught it: the gcp dispatch test asserting a
plan has no audio artifact started seeing one.

Export the name from `@hyperframes/producer/distributed` and consume it in both
adapters. This is the same failure the constant exists to prevent, one package
boundary further out: a literal that drifts from the writer's is a silently
missing audio track rather than a loud error, because both call sites only ever
ask whether the file exists.

* fix(cloud): accept a legacy plan's audio artifact name for one release

Review raised a rolling-deploy window I had flagged but left undecided: `plan`
and `assemble` are separate invocations bridged by object storage, so a
pre-rollout planner can be paired with a post-rollout assembler. Both readers
locate the artifact by existence alone, which makes that pairing a silently
muted video rather than an error. That is reachable enough to be worth two
lines, so reads now accept the old name while writes only ever emit the new one.

Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`)
rather than four call sites, marked for deletion one release out.

Also fixes a hole in the first pass of this: the plan-v2 materializer matched
either name but then joined the CURRENT one, so a legacy plan resolved to a path
that was never written. It now joins the artifact's own name.

Review nits in the same pass: correct the pad-branch docstring, which still
described a concat-copy shape the pad branch stopped using when it moved to
apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so
it cannot model a shape that reintroduces the priming delay.

* test(producer): rebake the missing-host-comp-id golden without the audio delay

The pinned reference was rendered before this branch, so it carries the 1024
sample encoder-priming delay in its audio. With the delay gone the correct audio
now sits ahead of the reference and the harness's envelope correlation drops
below its floor.

Cross-correlating the old and new references at native 48 kHz gives a lag of
exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved
by exactly the amount this branch removes. Regenerated inside the CI container
(Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches
the encoder CI will compare against - the container reproduced CI's failure to
the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and
passes at correlation 1.0 after it.

Note for archaeology: the new reference is also 3 dB louder than the old one.
That gap is not from this branch - `main` and this branch render the fixture at
the same level - it is pre-existing drift the reference had accumulated, which a
scale-invariant correlator could never see. The rebake absorbs it.

Only output.mp4 is updated. `--update` also rewrites compiled.html, but that
diff is embedded-font churn with no bearing on the comparison, which reports
"Failed at compilation: 0" either way.

* test(producer): rebake the variables-prod golden without the audio delay

Same cause as the missing-host-comp-id rebake, caught by shard-8 once the
earlier shard stopped failing and the rest of the matrix could run: this
reference also carries the encoder-priming delay this branch removes.

Reproduced in the CI container to the digit (correlation 0.42704173048439215,
lagWindows -12), rebaked there, and it now passes at correlation 1.0.

Worth recording: the shift here is 2048 samples (42.67 ms) at correlation
0.99983, exactly twice the 1024 of the other fixture. The delay compounds once
per un-compensated AAC generation, and this fixture's audio needs its duration
normalized, so it takes the pad/trim branch's re-encode and picks up a second
frame of priming on top of the mixer's. So the pre-fix error was not a fixed
21 ms - it grew with the number of times the audio was re-encoded.

All nine shards ran in that CI round with only this one failing, so the matrix
has now covered every fixture against this change.
2026-08-11 00:12:43 -04:00
..
2026-08-10 22:53:34 -04:00

@hyperframes/producer

Full HTML-to-video rendering pipeline: capture frames with Chrome's BeginFrame API, encode with FFmpeg, mix audio — all in one call.

Install

npm install @hyperframes/producer

Requirements: Node.js >= 22, Chrome/Chromium (auto-downloaded), FFmpeg

Usage

Render a video

import { createRenderJob, executeRenderJob } from "@hyperframes/producer";

const job = createRenderJob({
  inputPath: "./my-composition.html",
  outputPath: "./output.mp4",
  width: 1920,
  height: 1080,
  fps: 30,
});

const result = await executeRenderJob(job, (progress) => {
  console.log(`${Math.round(progress.percent * 100)}%`);
});

console.log(result.outputPath); // ./output.mp4

Run as an HTTP server

The producer can also run as a render server, accepting render requests over HTTP:

import { startServer } from "@hyperframes/producer";

await startServer({ port: 8080 });
// POST /render with a RenderConfig body

Configuration

RenderConfig controls the render pipeline:

Option Default Description
inputPath Path to the HTML composition
outputPath Output video file path (or directory, for format: "png-sequence")
width 1920 Frame width in pixels
height 1080 Frame height in pixels
fps 30 Frames per second (24, 30, or 60)
quality "standard" Encoder preset ("draft", "standard", "high")
format "mp4" Output container — "mp4", "webm", "mov", or "png-sequence". See Transparent Video Output below.
videoFrameFormat "auto" Source video frame extraction format — "auto", "jpg", or "png". Use "png" for UI recordings, screen captures, and color-sensitive source videos.

Transparent Video Output

The producer can render HTML compositions to formats that carry a true alpha channel — not chroma key. The same composition that renders an opaque MP4 renders a layerable overlay when you set format.

format Codec / pixel format Alpha Audio Use case
"mp4" (default) H.264 (yuv420p) or H.265 + HDR10 No AAC Streaming, social, default deliverable
"webm" VP9 + yuva420p True alpha Opus Web playback as overlay (<video> over background); supported in Chrome, Edge, Firefox
"mov" ProRes 4444 + yuva444p10le True alpha + 10-bit AAC Editor ingest (Premiere, Final Cut Pro, DaVinci Resolve)
"png-sequence" Numbered RGBA PNGs in a directory Lossless alpha Sidecar audio.aac After Effects / Nuke / Fusion, or pipelines that post-process frames before encoding

Example

import { createRenderJob, executeRenderJob } from "@hyperframes/producer";

const job = createRenderJob({
  inputPath: "./my-composition.html",
  outputPath: "./output.webm", // or a directory for "png-sequence"
  width: 1080,
  height: 1920,
  fps: 30,
  format: "webm", // "mp4" | "webm" | "mov" | "png-sequence"
});

await executeRenderJob(job);

What "transparent background" means here

The producer captures Chrome screenshots with the page background forced transparent (html, body, [data-composition-id] { background: transparent !important }) and the CDP default background override set to RGBA 0,0,0,0. The captured PNGs carry a real alpha channel and that channel is preserved end-to-end:

  • VP9 (webm) is encoded with -pix_fmt yuva420p, -auto-alt-ref 0, -cpu-used 4 by default, and alpha_mode=1 metadata. Tune the speed/quality tradeoff with PRODUCER_VP9_CPU_USED (-8 to 8) or local CLI --vp9-cpu-used.
  • ProRes 4444 (mov) is encoded with -pix_fmt yuva444p10le.
  • PNG sequences are written without re-encoding (zero-padded frame_NNNNNN.png).

This is not chroma keying. There is no green/blue background to remove and no "key" tolerance to tune — pixels that were transparent in the browser are transparent in the output.

Caveats

  • Linux + alpha forces screenshot capture. Chrome's BeginFrame compositor (the default deterministic capture path on Linux headless-shell) does not preserve alpha; the orchestrator falls back to Page.captureScreenshot, which is slower per frame. macOS and Windows already use screenshot mode by default, so they are unaffected.
  • HDR + alpha is not supported. Setting hdr: true together with an alpha-capable format logs a warning and falls back to SDR. Use format: "mp4" for HDR10 output.
  • png-sequence does not produce a single muxed file. When the composition contains audio elements, an audio.aac sidecar is written alongside the PNGs in outputPath.
  • Safari + WebM alpha is incomplete. For broad browser playback of an alpha video, ship format: "mov" to your editor and re-encode for the codec your distribution target supports.

Authoring transparent compositions

Don't paint a fullscreen background in your HTML. The default body background is overridden to transparent automatically — any body { background: ... }, #root { background: ... }, or [data-composition-id] { background: ... } rule is force-overridden during alpha rendering. Backgrounds on inner elements (cards, scenes, components) are kept.

Distributed rendering

For renders too large for a single machine, the producer ships a public set of distributed-render primitives. They are pure functions over local file paths — networking and orchestration live in adapter packages (Temporal, AWS Lambda + Step Functions, Cloud Run Jobs, K8s Jobs).

Plan v2 is recommended for new integrations. It publishes an immutable manifest plus content-addressed artifacts and materializes only each worker's declared dependencies:

import { planV2, renderChunkV2, assembleV2 } from "@hyperframes/producer/distributed";

// Controller-side: produce a v2 manifest + local content-addressed store.
const planResult = await planV2(
  projectDir,
  { fps: 30, width: 1920, height: 1080, format: "mp4" },
  "/tmp/plan-v2",
);

const chunk = await renderChunkV2("/tmp/plan-v2", 0, "/tmp/chunks/0.mp4");

// Controller-side: stitch chunks into the final deliverable.
await assembleV2("/tmp/plan-v2", ["/tmp/chunks/0.mp4", "/tmp/chunks/1.mp4"], "/tmp/output.mp4");

Cloud adapters should use planV2WithPublisher() so artifacts publish directly to object storage. The legacy plan() / renderChunk() / assemble() v1 layout remains supported, and cloud SDKs still interpret an omitted protocol as v1 for backwards compatibility.

The activity functions plus their result types are also re-exported from @hyperframes/producer so callers that pin the main package don't need a separate subpath import. Supported formats: mp4 SDR, mov ProRes 4444, and png-sequence. webm and HDR mp4 trip a typed FormatNotSupportedInDistributedError — use the in-process renderer (executeRenderJob) for those.

How it works

  1. Serve — spins up a local file server for the HTML composition
  2. Capture — opens the page in headless Chrome, seeks frame-by-frame via HeadlessExperimental.beginFrame (or Page.captureScreenshot for transparent / non-Linux renders), captures screenshots
  3. Encode — pipes frames through FFmpeg (with GPU encoder detection and chunked concat). Skipped for format: "png-sequence".
  4. Mix — extracts <audio> elements and mixes them into the final video. For png-sequence, audio is written as an audio.aac sidecar.
  5. Finalize — applies faststart for streaming-friendly MP4 (no-op for WebM, MOV, and png-sequence)

Documentation

Full documentation: hyperframes.heygen.com/packages/producer