ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-05 00:56:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,427 Commits 878 Branches 404 Tags
0971f3b2a703eb829c86990f7fd4b33dd8f60ed9
Commit Graph
27 Commits
Author SHA1 Message Date
Miguel Ángel eef4690752 fix(engine): name the fix in the ffmpeg encode-timeout error message (#1858) 
Two independent post-release feedback reports of hitting
ffmpegEncodeTimeout (600000ms default) on long or high-frame-count
renders, both resolved by setting FFMPEG_ENCODE_TIMEOUT_MS to a higher
value and/or PRODUCER_ENABLE_CHUNKED_ENCODE=true — env vars that already
exist and already solve this, but that neither user found from the error
message itself.

appendEncodeTimeoutMessage only stated what happened ("FFmpeg killed after
exceeding ffmpegEncodeTimeout"), not what to do about it. Name both
existing knobs in the message so the fix is immediately visible at the
point of failure instead of requiring a source dive.

One function, six call sites, all fixed at once. Existing tests assert
with toContain, so the appended text doesn't break them; added two
assertions confirming both env var names appear in the message.
 2026-07-02 17:45:16 -07:00
Miguel Ángel 3a2f052889 fix(engine): pad odd output dimensions up to even for H.264/H.265 encode (#1802) 
* fix(engine): pad odd output dimensions up to even for H.264/H.265 encode

A composition with an odd data-width or data-height (e.g. a custom 3:1
canvas at 1080x723) failed to encode to MP4. libx264/libx265 with 4:2:0
chroma subsampling (yuv420p, yuv420p10le) require both dimensions to be
even and abort before writing a packet:

  [libx264] height not divisible by 2 (1080x723)
  Error while opening encoder ... Invalid argument

Both the streaming encoder and the chunk encoder built the software
range-conversion filter ("scale=in_range=pc:out_range=tv") with no
even-dimension enforcement, so any odd-sized canvas reached libx264
unmodified and the whole render failed.

Add a shared withEvenDimensionPad helper that appends
pad=ceil(iw/2)*2:ceil(ih/2)*2 to the filter chain only for 4:2:0 pixel
formats. The pad rounds each odd dimension up by one pixel (a no-op when
already even) without scaling, so content is never resampled. Formats
that accept odd dimensions (ProRes 4444 yuva444p10le, VP9 yuva420p) are
excluded, so transparent/alpha output is untouched.

* fix(engine): extend even-dimension pad to GPU 4:2:0 encode paths

The odd-dimension pad added for libx264/libx265 only covered the software
encoder branches. nvenc, videotoolbox, qsv, and amf feed software frames
straight to the hardware encoder with no -vf chain, so an odd-sized 4:2:0
canvas on --gpu (or an auto-selected hardware encoder) reproduced the same
"height not divisible by 2" abort before any packet was written.

Add the even-dimension pad to the software-side -vf chain for those four
GPU paths in both the chunk and streaming encoders, reusing the shared
withEvenDimensionPad helper (the pad runs on CPU before the encode). vaapi
is left as-is: its existing format=nv12,hwupload conversion already aligns
odd dimensions before upload, so it is not double-padded. ProRes 4444 and
VP9 alpha stay untouched, exactly as the software fix excludes them.

nvenc/videotoolbox/qsv/amf arg construction is logic-tested (the pad filter
is asserted on the built arg list for 8-bit and 10-bit 4:2:0, with alpha
ProRes asserted padless); runtime hardware encode is not exercised here.
 2026-06-30 10:55:22 -07:00
Miguel Ángel e9957ecaf4 fix(engine): remove -shortest from muxVideoWithAudio (#1648) (#1650) 
FFmpeg 6.0 (bundled by ffmpeg-static) has a regression where -shortest
combined with -c:v copy over-truncates the video stream while leaving
audio untouched. The flag is also redundant — the audio mixer already
pads/caps all tracks to totalDuration via apad=whole_dur and -t.

Closes #1648
 2026-06-22 17:06:03 -04:00
Miguel Ángel 0473254bdd fix(engine): preserve AAC start time during MP4 mux (#1615) 
* fix(engine): copy mixed AAC during MP4 mux

* fix(producer): avoid AAC re-encode in distributed audio pad

* fix(engine): probe AAC sidecars before mux copy decision

* fix(producer): avoid temp concat file for audio padding
 2026-06-20 17:22:15 -04:00
Miguel Ángel 8408a44745 fix(engine): tune VP9 cpu-used across render paths (#1614) 
* fix(engine): tune VP9 cpu-used across render paths

* fix: address VP9 review feedback
 2026-06-20 16:36:59 -04:00
Miguel Ángel 5917c0382d fix(producer): pass resolved engine config through every encode path (#1371) 
encodeFramesFromDir was called with 5 of its 6 args, dropping the config
param — the encode timeout always fell back to the hardcoded 600s default
and FFMPEG_ENCODE_TIMEOUT_MS was silently ignored, so any encode over
600s wall time was deterministically SIGTERM-killed. Resolve the engine
config once in the encode stage and pass it to the non-chunked, chunked,
and GIF paths. The chunked per-chunk encodes previously had no timeout at
all; they now honor the same config value.

Review follow-ups: the chunked path's final concat spawn gains the same
config-driven timeout (it previously had none); every encode-timeout
kill now appends 'FFmpeg killed after exceeding ffmpegEncodeTimeout
(N ms)' to the failure instead of surfacing a bare exit-255; and the
orchestrator threads its already-resolved config into the encode stage
via an optional EncodeStageInput.engineConfig field (direct callers and
distributed chunks keep the producerConfig ?? resolveConfig() fallback).
The encode-timeout tests run against a mocked child_process spawn with
fake timers, removing the real-ffmpeg dependency that made the previous
default-timeout test environment-fragile in CI.

Fixes #1348
 2026-06-12 12:21:35 -04:00
Miguel Ángel cee6fd02d6 fix(cli): verify browser/ffmpeg binaries exist before render starts (#1365) 
## Problem

Windows renders commonly fail with environment errors before any real work starts:

- `Browser was not found at the configured executablePath (...chrome-headless-shell.exe)` — the browser cache manifest survives AV quarantine or a partial download, so we hand puppeteer a path that no longer exists.
- `[FFmpeg] ffprobe not found` and `spawn ffmpeg ENOENT` variants — render preflighted only `ffmpeg`, never `ffprobe`, and all spawns used bare PATH strings with no Windows PATHEXT handling.

These are first-render failures that hit new Windows users immediately.

## Fix

- Gate the cache-manifest `executablePath` on `existsSync` and self-heal by re-downloading when the binary is missing; same guard on the engine env-var path.
- New shared environment preflight (`packages/cli/src/browser/preflight.ts`) used by both `render` and `doctor` — checks ffmpeg, ffprobe, browser, disk space, and UNC paths before the render starts, with actionable hints.
- Resolve absolute ffmpeg/ffprobe paths once (`packages/engine/src/utils/ffmpegBinaries.ts`) and pass them to every engine spawn instead of relying on PATH.
- Map opaque Windows ffmpeg exit codes to actionable messages.

## Testing

- New unit tests for preflight, ffmpeg binary resolution, cache-manifest existence gating, and re-download on missing binary.
- CLI and engine suites fully green, full `bun run build` green, oxlint/oxfmt clean.
- Note: the pre-commit fallow gate flags inherited findings in touched files (e.g. `audioExtractor.ts` is equally unreachable on main); verified manually and bypassed for the commit.
 2026-06-12 01:36:28 -04:00
Miguel Ángel 0e052e42d2 fix(engine): support AMD AMF GPU encoding 2026-05-26 13:35:55 -04:00
James a4c4b2ff03 fix(distributed): enforce exact framerate at concat + mux boundaries 
When the distributed render path stitches chunks with `-c copy`,
ffmpeg averages the container framerate from PTS rather than
carrying the source's exact rational rate, producing values like
`360000/12001` instead of `30/1` and ~5ms duration drift over
60s.

This is a known ffmpeg behavior at the concat-demuxer-copy
boundary. The industry-standard fix is `-r <fps>` as an input
flag on the concat step plus an output flag on the subsequent
mux step — both with `-c copy` retained, no re-encode required.

Three sites updated:
- `assemble.ts` concat step: `-r <fps>` input flag.
- `chunkEncoder.muxVideoWithAudio`: `-r <fps>` output flag.
- `chunkEncoder.applyFaststart`: same, threaded from caller.

Adds `r_frame_rate` + duration-equivalence assertions to
`assemble.test.ts` to close the regression hole.
 2026-05-23 01:12:19 -04:00
Miguel Ángel a54953b936 fix: clean up orphaned Chrome and ffmpeg processes on preview exit 
The preview command's shutdown handler only closed the HTTP server,
leaving Chrome (browser pool) and ffmpeg processes alive. This caused
silent resource leaks — orphaned processes consuming CPU and RAM with
no parent.

Root cause: preview.ts never called drainBrowserPool() or killed
tracked ffmpeg processes. The thumbnail browser in studioServer.ts
registered its own competing signal handlers that raced with
preview's shutdown.

Fix:
- Add a central process tracker (processTracker.ts) that registers
  every spawned ffmpeg across engine and producer packages
- Centralize thumbnail browser cleanup via exported
  closeThumbnailBrowser() instead of scattered signal handlers
- Wire preview shutdown to call closeThumbnailBrowser(),
  drainBrowserPool(), and killTrackedProcesses() before closing the
  HTTP server (embedded mode)
- Add killProcessTree() for dev/local modes where Chrome runs in a
  child process tree
- Add startup orphan detection that finds and kills orphaned
  chrome-headless-shell/Puppeteer Chrome processes (PPID=1) from
  previously crashed sessions

Closes #1038
 2026-05-22 23:31:00 -04:00
James RussoandClaude Opus 4.7 5d264e146c docs(lambda): document webm support + simplify-review fixes (#953) 
* docs(lambda): document webm support in distributed mode

PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see
DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the
shipped capability.

Updates docs/deploy/migrating-to-hyperframes-lambda.mdx:

- "Output format" row in the migration table now lists `webm` alongside
  mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 +
  closed-GOP concat-copy. HDR mp4 remains the only refused format.

- "No webm distributed" caveat replaced with "webm uses closed-GOP VP9"
  explainer covering the encoder args (`-g <chunkSize>`,
  `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why
  alt-ref disable is load-bearing, and that the output preserves alpha
  via yuva420p with Opus audio.

- Migration checklist no longer asks adopters to filter out webm
  compositions; only HDR-dependent renders need to stay on the previous
  framework.

aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR
in the v1 surface list), so it gets no copy edits beyond the migration
guide.

The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8,
§12 — kept outside the repo) gets matching updates: format support
matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the
new top item, and the rev-12 → rev-13 status line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: address simplify-review findings on webm stack

Folds in cleanups identified by a multi-agent code-review pass over the
4-PR webm-distributed stack:

- plan.ts: `resolveEncoderTriple()` webm case now calls
  `getEncoderPreset(quality, "webm")` for its preset string instead of
  hardcoding "good". The hardcode was wrong for `quality: "draft"`
  (`getEncoderPreset` returns "realtime" for that tier) — would have
  silently overridden the draft → realtime mapping for distributed webm
  renders.
- chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18
  lines of WHY narration down to the 6 lines that actually explain why
  (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push
  comment to the same standard.
- chunkEncoder.test.ts: drop the duplicate WHY comment that restated
  the implementation comment in plain words.
- webm-concat-copy.test.ts: rewrite the file-header docstring to
  describe the contract being tested instead of the PR-8.1-gating
  history; strip "PR 8.2 / Path A / Path B" references from error
  messages (they belong in PR bodies, not in test output). Consolidate
  the yuva420p alpha smoke into a single `it()` block (was a full
  4-test describe with duplicated setup) — the yuv420p block already
  covers the probe/decode/frame-count contract; the alpha smoke only
  needs to prove the alpha args don't break concat-copy.
- plan.test.ts: drop the "PR 8.1 proved the contract" comment.
- webm-vp9 fixture: drop the aspirational "Other webm-with-audio
  fixtures cover the mux path separately when added" sentence (no
  other fixtures exist). Regenerated the baseline via
  `docker:test:update webm-vp9` to reflect the updated comment.
- migrating-to-hyperframes-lambda.mdx: add a paragraph about
  distributed webm's perf cost — ~10-25% larger files at constant CRF
  due to forced keyframes, and slower per-chunk encode due to
  `-cpu-used 2` being more conservative than the libvpx default.

All unit tests + the webm-vp9 distributed-simulated regression still
pass after these changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cli): accept --format=webm in `hyperframes lambda render`

The CLI's `lambda render` subcommand's FORMATS allowlist and the
`RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`,
so even though the producer + aws-lambda packages now support webm
end-to-end, the CLI surface rejected it with `--format must be mp4|mov|
png-sequence`. Add webm to both spots and update the --help description.

Surfaced during real-AWS deploy prep — the local lambda-local /
distributed-simulated tests didn't go through the CLI so the gap went
unnoticed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME)

The deterministic Google Fonts cache was rooted at
`\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the
runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's
read-only. `mkdirSync(..., { recursive: true })` can't create that
path and the plan stage trips with `ENOENT: no such file or directory,
mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on
every Lambda render that pulls a Google Font (i.e. every distributed
fixture using `@import url("https://fonts.googleapis.com/...")`).

Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to
`tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives
across invocations on a warm container, so cache hit rate is the same
as non-Lambda runs. Also honor an explicit
`\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a
different location regardless of the runtime.

Surfaced while verifying webm distributed end-to-end on real AWS — the
same bug affects mp4 fixtures using Google Fonts; webm just happened to
be the one I tried first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: extract DistributedFormat type + trim font-cache resolver

Second simplify-review pass on the webm stack flagged two cleanups:

1. **`DistributedFormat` type duplicated 10 times.** Every file in the
   distributed pipeline carried its own copy of
   `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format
   meant a 10-place edit with no compile-time guarantee they stayed in
   sync. Extract a single source of truth in
   `packages/producer/src/services/distributed/shared.ts`, re-export
   from `@hyperframes/producer/distributed` and
   `@hyperframes/aws-lambda/sdk`, and have all callers pull from
   there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's
   `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]`
   so the compiler enforces the runtime allowlist stays in sync with
   the type.

2. **`deterministicFonts.ts` font-cache resolver was over-commented.**
   Trim the 7-line block to 4 lines (drop the aspirational
   "and other read-only-FS execution environments" — only Lambda is
   detected — and the warm-container `/tmp` persistence narration —
   anyone reading already knows Lambda /tmp semantics). Collapse the
   two-step `if (explicit && explicit.length > 0)` into a single
   nullish-coalesce expression now that the empty-string defensive
   check is gone (`process.env.X` is `string | undefined`, no third
   shape to guard against).

Out-of-scope skips (called out by the agents, deferred):
- In-process `RenderConfig.format` and the in-process CLI's
  `render.ts` format union still carry their own inline copies. The
  union happens to coincide today but they're separate concerns —
  leaving them alone limits this PR's blast radius.
- `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry
  flagged as taste; skipping.
- Pre-existing redundant `existsSync` before `mkdirSync({ recursive:
  true })` in `fontCacheDir` — out of scope.

All tests + typecheck still pass. Lambda render still works
end-to-end (no functional changes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(lambda): drop plan-doc reference from migration checklist

PR review feedback: source/docs should not mention the
distributed-rendering planning doc. Tighten the migration checklist
sentence to describe the webm path directly rather than referencing
the doc's version label.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers

CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 —
the function interleaved (a) mp4 codec validation + dispatch, (b) the
non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting
into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the
top-level function's cyclomatic complexity below the threshold while
preserving every error message and code path. Behavior unchanged.

Also extracts an `EncoderTriple` type alias so the three functions
share the return shape declaratively rather than repeating it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 04:11:26 -04:00
James Russo 07de7e61ed feat(engine): closed-GOP VP9 encoder args + concat-copy smoke test (#950) 
## Description

PR 1 of 4 in the WebM (VP9) distributed-rendering series. A gating
experiment that proves closed-GOP libvpx-vp9 chunks survive
`ffmpeg -f concat -c copy` losslessly, so the rest of the stack can
ship Path A (concat-copy) rather than the slower
re-encode-in-assemble fallback.

Two changes:

1. **Closed-GOP VP9 encoder args.** `buildEncoderArgs` now lays
   `-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, and
   `-cpu-used 2` on libvpx-vp9 when `lockGopForChunkConcat=true`.
   Mirrors the existing libx264/libx265 branches. The alt-ref disable
   is load-bearing — libvpx-vp9's default non-displayable alt-ref
   frames can reach across chunk seams and break concat-copy.
   `-cpu-used 2` pins the speed/quality tradeoff so chunks encoded on
   workers with different libvpx-vp9 defaults produce visually
   consistent output across seams. Default (`lockGopForChunkConcat`
   unset) preserves the existing in-process VP9 path unchanged.

2. **Concat-copy smoke test** at
   `packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts`.
   Generates 60 PNGs via lavfi `testsrc2`, encodes them as 4 VP9 chunks
   of 15 frames using `buildEncoderArgs` with
   `lockGopForChunkConcat=true`, concat-copies via `ffmpeg -f concat -c
   copy`, then runs three independent verifications:
   `ffprobe -show_streams`, `ffmpeg -f null -` decode test, and
   `ffprobe -count_frames`. Each verification surfaces its failure
   fingerprint in the error message.

Smoke test passes 6/6 locally → Path A works; the rest of the stack
takes it.

Also exports `buildEncoderArgs` from `@hyperframes/engine` so
adapters / tests can construct args without re-implementing the
contract.

## Testing

- [x] `bunx vitest run --root packages/engine src/services/chunkEncoder.test.ts` — 62/62 pass (new VP9 closed-GOP tests included)
- [x] `bun test packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts` — passes
- [x] `bunx oxlint` + `bunx oxfmt --check` on all changed files — clean
- [x] `bunx tsc --noEmit -p packages/engine/tsconfig.json` — clean

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-05-19 01:53:08 -04:00
JamesandClaude Opus 4.7 2d6372ac2a feat(engine): add lockGopForChunkConcat option to buildEncoderArgs 
Part of Phase 2 of the distributed rendering plan (determinism hardening).
See DISTRIBUTED-RENDERING-PLAN.md §7.1 and §17.2 (gating table).

Adds two optional fields to EncoderOptions:

  lockGopForChunkConcat?: boolean  // default false
  gopSize?: number                 // required when lockGopForChunkConcat=true

When the flag is true on the SW libx264 / libx265 paths, buildEncoderArgs
emits closed-GOP / forced-keyframe args so the resulting chunk file can be
losslessly concatenated (`ffmpeg -f concat -c copy`) with sibling chunks:

  -g <gopSize>
  -keyint_min <gopSize>
  -sc_threshold 0
  -force_key_frames "expr:eq(mod(n,<gopSize>),0)"
  -x264-params "...:scenecut=0:open-gop=0:repeat-headers=1"
  -x265-params "keyint=<gopSize>:min-keyint=<gopSize>:scenecut=0:open-gop=0:repeat-headers=1"
  -bf 0   (added for h265 too when locked)

GPU encoders, vp9, and prores ignore the flag (their concat-copy story is
separate — see plan §7.2 / §8).

In-process behavior is unchanged: the default (false) path emits no new
args. New unit tests pin both branches in packages/engine/src/services/
chunkEncoder.test.ts.

This is part of a stack of 10 PRs; this is PR 1 of 10.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 04:36:11 +00:00
Theodor Kleynhans 5dcc89c930 feat(cli): accept ffmpeg-style rational fps (NTSC, PAL, slow-mo) 
Replaces the rigid `--fps 24|30|60` whitelist with a numeric range and
adds support for ffmpeg-style fractional framerates so NTSC stays exact
end-to-end.

- `--fps 30` keeps working (integer fps)
- `--fps 30000/1001` now means exact NTSC 29.97 (not the lossy decimal)
- `--fps 24000/1001`, `--fps 60000/1001`, `--fps 25/50/120/240` all work
- Decimals like `--fps 29.97` are rejected with a friendly error pointing
  the user at the rational form, since `29.97` and `30000/1001` round
  to different framerates inside ffmpeg

Carries an `Fps = { num: number; den: number }` rational end-to-end:
RenderConfig, EncoderOptions, StreamingEncoderOptions, CaptureOptions,
DockerRenderOptions, Studio API request body, regression-harness
meta.json. The `-r` and `-framerate` ffmpeg args emit the rational form
verbatim (`30000/1001`) so no decimal round-trip happens at the encoder
boundary. Frame-interval math uses `1000 * den / num` ms (33.366… for
NTSC, 33.333… for integer 30).

Helpers live in @hyperframes/core:
- `parseFps(input: string | number): FpsParseResult` — discriminated
  parser used by both the CLI and the Studio API route
- `fpsToFfmpegArg(fps: Fps): string` — emits "30" or "30000/1001"
- `fpsToNumber(fps: Fps): number` — for arithmetic (telemetry, frame
  count, frame-index → time)

Studio API wire format accepts polymorphic `fps: number | string`:
- number → integer fps (`30`)
- string → rational (`"30000/1001"`)
Decimals are rejected; matches the same rule as the CLI.

Existing meta.json fixtures with integer `"fps": 30` continue to load
unchanged — the regression-harness validator now normalizes both number
and string inputs through `parseFps`.
 2026-05-09 00:09:15 +02:00
JamesandClaude Opus 4.7 f4ecf96918 fix(engine,cli,producer): address PR #627 review feedback 
- engine/chunkEncoder, engine/streamingEncoder: extend `-bf 0` to GPU h264
  paths (nvenc, qsv, vaapi) and `-b_strategy 0` for qsv so GPU-encoded
  outputs avoid negative-DTS freezes too — not just SW libx264.
- engine/videoFrameExtractor: detect mid-path traversal (e.g.
  `assets/../../foo.mp4`) by normalizing first and re-anchoring at the
  project root. Adds a regression test.
- engine/videoFrameExtractor: dedupe stderr "src not resolvable" warnings
  by `video.src` so a comp with N broken sources logs once, not N times.
- engine/videoFrameExtractor.test: drop dynamic `require("node:fs")`,
  use ES `import { writeFileSync } from "node:fs"`.
- engine/ffprobe: extract `readTagCI` helper for case-insensitive ffprobe
  tag reads (will recur for other libavformat-versioned sidecar tags).
- cli/background-removal/pipeline: collapse Quality / QUALITIES /
  QUALITY_CRF / DEFAULT_QUALITY / isQuality surface using
  `Quality = keyof typeof QUALITY_CRF`.
- producer/renderOrchestrator: replace `v.src.startsWith("/")` with
  `isAbsolute(v.src)` in the HDR probe path so Windows absolute paths
  (`C:\...`) aren't treated as relative — matches the audioMixer guard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 20:29:34 -07:00
JamesandClaude Opus 4.7 0e541673e0 fix(engine): wait for first frame decode + drop B-frames so renders play in every player 
Three related render robustness fixes:

1. frameCapture.ts: bump videos-ready check from `readyState >= 1`
   (HAVE_METADATA — only dimensions known) to `>= 2` (HAVE_CURRENT_DATA —
   first frame is rasterized). Without this, when two `<video>` elements
   with different codecs (h264 mp4 + VP9 webm) decode at different rates,
   the faster one passes readiness while the slower one still hasn't
   painted, producing a black "first frame" for the slower clip.

2. chunkEncoder.ts (libx264 path) + streamingEncoder.ts: disable B-frames
   for h264 (`-bf 0`). Standard libx264 with B-frames produces negative
   DTS at stream start (the first B-frame's decode order is "before" the
   first I-frame's presentation time). VS Code preview, several browser
   <video> implementations, and some HW decoders freeze on the first
   frame and only audio plays. -bf 0 makes PTS == DTS at every frame,
   eliminating the issue at the source. Quality cost is ~5–10% larger
   files at the same CRF — worthwhile for "the file plays everywhere".

3. chunkEncoder.ts (encoder + mux paths): add `-avoid_negative_ts make_zero`
   as belt-and-suspenders against negative DTS sneaking back in via
   `-c:v copy` mux passes when audio/video PTS bases differ.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 20:27:56 -07:00
roiizchakandroi32 3b8de7a5eb fix(engine): accept libx264 preset names with NVENC and QSV (#442) 
NVENC rejects the libx264 preset vocabulary (ultrafast / medium / slow /
...) with AVERROR(EINVAL) ("Error applying encoder options: Invalid
argument"), which surfaces as a bare `FFmpeg exited with code -22` from
spawn(). Because ENCODER_PRESETS passes these names straight through to
h264_nvenc / hevc_nvenc, every `--gpu` render using the `draft` tier
failed; `standard` (medium) and `high` (slow) only worked coincidentally
on ffmpeg builds that happened to accept those aliases. QSV has the same
problem on a narrower set (ultrafast / superfast / placebo).

Add `mapPresetForGpuEncoder` in utils/gpuEncoder.ts that translates the
libx264 vocabulary to each encoder's native names:

- nvenc: libx264 -> p1..p7 (already-native pN values pass through);
  unknown values fall back to p4 (medium)
- qsv:   ultrafast / superfast -> veryfast; placebo -> veryslow;
  everything else passes through
- videotoolbox / vaapi / null: unchanged

Both buildEncoderArgs (chunkEncoder.ts) and buildStreamingArgs
(streamingEncoder.ts) now route through the helper before pushing
`-preset` to the ffmpeg arg vector.

To make the next encoder-options failure diagnosable without re-running
ffmpeg by hand, \`formatFfmpegError\` in utils/runFfmpeg.ts now appends
the last 15 non-empty stderr lines to the error string. The four call
sites that previously swallowed stderr (encodeFramesFromDir,
muxVideoWithAudio, applyFaststart, and the streaming encoder exit
handler) have been updated.

Tested end-to-end on an RTX 4080 with ffmpeg 8.1 NVENC across
\`--quality draft|standard|high\` plus \`--video-bitrate\` and \`--crf\`
overrides; the 6 renders were visually equivalent to the CPU baseline.

Co-authored-by: roi32 <75878108+roi32@users.noreply.github.com>
 2026-04-23 08:38:12 -07:00
Vance Ingalls 5256a93b2d feat(engine): wire options.hdr through chunkEncoder + dynamic SDR→HDR transfer (#370) 
## Summary

Three independent fixes that share a common thread: HDR config flowing correctly from `EngineConfig` down through every encoder. The headline fix: disk-based HDR encodes via `chunkEncoder` were silently producing BT.709-tagged output despite `options.hdr` being set.

## Why

`Chunk 3` of `plans/hdr-followups.md`. The streaming encoder was correct but `chunkEncoder.buildEncoderArgs` hard-coded BT.709 color tags and the `bt709` VUI block in `-x265-params`, even when callers passed an HDR `EncoderOptions`. Today this is harmless because `renderOrchestrator` routes native-HDR content to `streamingEncoder` and only feeds `chunkEncoder` sRGB Chrome screenshots — but the contract was a lie, and any future caller that wired HDR through `chunkEncoder` would silently get SDR output.

## What changed

**3A — `chunkEncoder` respects `options.hdr` (BT.2020 + mastering metadata).** When `options.hdr` is set, the libx265 software path emits `bt2020nc` plus the matching transfer (`smpte2084` for PQ, `arib-std-b67` for HLG) at the codec level *and* embeds master-display + max-cll SEI in `-x265-params` via `getHdrEncoderColorParams`. libx264 still tags BT.709 inside `-x264-params` (libx264 has no HDR support) but the codec-level color flags flip so the container describes pixels truthfully. GPU H.265 (nvenc/videotoolbox/qsv/vaapi) gets the BT.2020 tags but no `-x265-params` block, so static mastering metadata is omitted — acceptable for previews, not HDR-aware delivery.

**3B — `convertSdrToHdr` accepts a target transfer.** `videoFrameExtractor.convertSdrToHdr` was hard-coded to `transfer=arib-std-b67` (HLG) regardless of the surrounding composition's dominant transfer. `extractAllVideoFrames` now calls `analyzeCompositionHdr` first, then passes the dominant transfer (`"pq"` or `"hlg"`) into `convertSdrToHdr` so an SDR clip mixed into a PQ timeline gets converted with `smpte2084`, not `arib-std-b67`.

**3C — `EngineConfig.hdr` type matches its declared shape.** The IIFE for the `hdr` field returned `undefined` when `PRODUCER_HDR_TRANSFER` wasn't `"hlg"` or `"pq"`, but the field is typed as `{ transfer: HdrTransfer } | false`. Returning `false` matches the type and avoids a downstream `undefined` check.

## Test plan

- [x] `chunkEncoder.test.ts`: replaced the previous "HDR options ignored" assertions with 8 new specs covering BT.2020 + transfer tagging, master-display/max-cll embedding, libx264 fallback behavior, GPU H.265 + HDR (tags but no x265-params), and range conversion for both SDR and HDR CPU paths.
- [x] All 313 engine unit tests pass (5 new HDR specs).
- [x] `ffprobe` an HDR composition rendered through the chunk encoder path: shows `bt2020nc` color matrix, `smpte2084` transfer, and mastering display metadata.

## Stack

Chunk 3 of `plans/hdr-followups.md`. Independent of Chunks 1/4 (touches separate code paths).
 2026-04-22 20:36:29 -07:00
Vance Ingalls 5a3fde19d4 feat(engine): add HDR video output pipeline (#265) 
## Summary

Adds the ability to render HDR video output (H.265 10-bit, BT.2020) from HyperFrames compositions. When the renderer detects HDR source video, it automatically switches to the HDR output pipeline — no flags needed.

## What it does

- **Auto-detection** — Probes each video source with `ffprobe`. If any has bt2020/PQ/HLG color metadata, the output switches to H.265 10-bit with correct color tags. SDR-only compositions are unaffected (H.264, bt709).
- **HLG pass-through** — Native HLG pixels from FFmpeg extraction are piped directly to the encoder without conversion. This avoids brightness loss from HLG→linear→PQ conversion (which requires an OOTF system gamma we can't reliably apply).
- **Encoder HDR support** — Both chunk and streaming encoders accept HDR presets: `libx265`, `yuv420p10le`, BT.2020 color primaries, `hvc1` codec tag (required for Apple playback).
- **WebGPU HDR capture (gated)** — A complete WebGPU float16 readback pipeline is implemented and tested but gated behind headed Chrome (headless doesn't expose WebGPU). Ready for future use with WebGPU canvas content.
- **HDR utilities** — `detectTransfer()` (PQ vs HLG), `getHdrEncoderColorParams()`, `analyzeCompositionHdr()`. 15 unit tests.

## Key design decisions

| Decision | Why |
|----------|-----|
| No `--hdr` flag | SDR content encoded as HDR causes orange shift in browsers. Auto-detect eliminates this. |
| HLG pass-through (not HLG→PQ) | Conversion loses brightness without OOTF. Pass-through matches source exactly. |
| `hvc1` codec tag | Apple QuickTime requires `hvc1` (not `hev1`) for HEVC playback. |
| 1-hour streaming timeout | HDR capture at ~6fps needs more time than the default 10-minute FFmpeg timeout. |

## Files changed

| File | What changed |
|------|-------------|
| `packages/engine/src/utils/hdr.ts` | **NEW** — HDR detection, transfer types, encoder params (15 tests) |
| `packages/engine/src/services/hdrCapture.ts` | **NEW** — WebGPU readback, HLG conversion, PQ encode |
| `packages/engine/src/services/streamingEncoder.ts` | HDR presets, raw rgb48le input, color tags |
| `packages/engine/src/services/chunkEncoder.ts` | HDR presets, conditional color tags |
| `packages/producer/src/services/renderOrchestrator.ts` | Auto-detection loop, HDR pass-through capture path |

## How to test

Render a composition with an HDR video source. The output should be H.265 10-bit with HDR metadata visible in `ffprobe` (bt2020, arib-std-b67 or smpte2084). Plays correctly in QuickTime and on HDR displays.

## Stack position

**2 of 6** — Stacked on #258 (SDR/HDR normalization). Provides the encoder infrastructure that phases 1-5 build on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-19 15:10:59 -07:00
James RussoandClaude Opus 4.6 ebc12f7dc9 feat(render): add CRF/bitrate controls and improve default quality (#292) 
Raise default encoding quality to visually lossless at 1080p (CRF 18)
and expose fine-grained encoding controls for power users.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-16 12:02:24 -07:00
Miguel Ángel 43e9252065 feat: add MOV (ProRes 4444) as transparent video output format (#224) 
## Summary

- Adds `--format mov` to the render CLI for ProRes 4444 transparent video output
- ProRes 4444 with alpha is the industry standard for transparent video overlays, supported by CapCut, Final Cut, Premiere, DaVinci, and After Effects
- WebM VP9 alpha technically works but is ignored by all major video editors — only browsers decode it
- Adds MOV to the studio export dropdown alongside MP4 and WebM

## Transparency format comparison

| Format | Codec | Alpha | Video editors | Browsers | File size |
| --- | --- | --- | --- | --- | --- |
| **MOV** | ProRes 4444 | Yes | CapCut, Final Cut, Premiere, DaVinci, After Effects | No (won't play in browser) | Large (~5-40 MB) |
| **WebM** | VP9 | Yes | None (shows black) | Chrome, Firefox | Small (~200 KB) |
| **MP4** | H.264 | No | All | All | Small |

> **Note:** ProRes MOV files do not play in Chromium browsers — they are an intermediate/editing format, not a delivery format. Use [rotato.app/tools/transparent-video](https://rotato.app/tools/transparent-video) to verify transparency works correctly.

## Changes

- **CLI**: Add `mov` to `--format` validation, examples, and output path logic
- **Engine**: `getEncoderPreset()` returns ProRes 4444 (`yuva444p10le`) for `mov` format; handle `.mov` in `applyFaststart` and `muxVideoWithAudio`; add `pix_fmt` to streaming encoder ProRes path
- **Producer**: Treat `mov` like `webm` for alpha capture (PNG frames, screenshot mode, `forceScreenshot`)
- **Studio**: Add MOV option to export format dropdown and render queue hook
- **Core**: Add `mov` to studio API types, render route, and mime helpers
- **Tests**: Add encoder preset tests for mov format (42 total, all passing)

## Usage

```bash
hyperframes render --format mov --output overlay.mov
```

## Test plan

- [x] `pnpm build` passes
- [x] `pnpm --filter @hyperframes/engine test` — 42 tests pass (2 new for MOV)
- [x] `oxlint` and `oxfmt` clean on all 12 changed files
- [x] End-to-end local render produces ProRes 4444 (`yuva444p12le`) with working alpha
- [x] Docker render with `--format mov` — ProRes 4444 confirmed via ffprobe
- [x] Studio dropdown shows MOV option in built JS
- [x] Transparency verified with [rotato.app/tools/transparent-video](https://rotato.app/tools/transparent-video)
 2026-04-08 03:11:37 +02:00
James Russo b491679c71 fix(engine): add bt709 color space + range conversion to encoding (#223) 
## Description

Adds proper BT.709 color space metadata and full→limited range conversion to H.264/H.265 encoding. Chrome captures frames in full-range sRGB (BT.709 primaries), but without explicit color tagging, players guess the wrong color space and range — causing color shifts across iOS/Android/desktop and crushed dark values that compound the gradient banding issue fixed in #222.

**What changed:**

| Setting | Before | After |
|---------|--------|-------|
| `color_space` | `bt470bg` (guessed) | `bt709` (explicit) |
| `color_primaries` | `unknown` | `bt709` |
| `color_transfer` | `unknown` | `bt709` |
| `color_range` | `pc` (full, wrong for H.264) | `tv` (limited, correct) |
| `time_base` | `1/15360` (varies by platform) | `1/90000` (fixed) |

**Approach:**
- BT.709 VUI params embedded via x264-params/x265-params (`colorprim=bt709:transfer=bt709:colormatrix=bt709`) — ensures the bitstream itself carries color info
- FFmpeg-level metadata flags (`-colorspace:v bt709`, etc.) — belt-and-suspenders
- `scale=in_range=pc:out_range=tv` filter converts Chrome's full-range output to TV/limited range
- VAAPI path chains the range filter with existing `format=nv12,hwupload`
- `-video_track_timescale 90000` for consistent cross-platform A/V timing (same as Remotion)
- VP9 and ProRes encoding unaffected

## Testing

- Verified via ffprobe: all 5 color metadata fields now correct
- Directly tested FFmpeg args produce expected output
- 40 engine tests pass (8 new: color metadata h264/h265, range filter CPU, VAAPI filter chain, GPU skip, VP9 skip, timescale)
- Builds cleanly, lint + format pass
 2026-04-07 10:40:18 -07:00
James RussoandClaude Opus 4.6 7bd8939143 fix(engine): add anti-banding x264/x265 params for dark gradients (#222) 
Add aq-mode=3 (auto-variance adaptive quantization) to CPU H.264/H.265
encoding. This redistributes bits from bright/textured areas to dark flat
areas where color banding is most visible in 8-bit yuv420p output.

- standard/high presets: aq-mode=3 + aq-strength=0.8 + deblock=1,1
- draft (ultrafast): aq-mode=3 only (deblock too slow for ultrafast)
- GPU and VP9 encoders unaffected (have their own AQ implementations)

Adds 6 regression tests verifying the params are emitted correctly.

Fixes color banding on dark gradients (eval issue #3, prompts 3,5,10,14).

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-07 09:49:52 -07:00
JamesandClaude Opus 4.6 83d11ac595 refactor: simplify review fixes for WebM PR 
- Use static import for copyFileSync (was unnecessary dynamic import)
- Shallow-copy config before mutating forceScreenshot (prevents
  caller-provided config from being permanently modified)
- Consolidate isWebm/isWebmRender/outputFormat into single early
  declaration in renderOrchestrator
- Fix debug output extension for WebM (was hardcoded .mp4)
- Log unexpected audio extraction errors instead of silently swallowing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 03:40:35 +00:00
JamesandClaude Opus 4.6 15f6b86ac0 feat(render): add WebM output with VP9 alpha transparency 
Support rendering compositions with transparent backgrounds via
`--format webm`. VP9+alpha is the standard format for overlayable
video (captions, lower thirds, overlays).

Changes by layer:
- CLI: `--format mp4|webm` flag on render command
- Producer: threads format through RenderConfig, switches to PNG
  capture and VP9 encoding when webm
- Engine: getEncoderPreset() returns VP9 config with yuva420p;
  transparent page background via CDP when capturing PNG;
  mux uses Opus audio for WebM; VP9 flags from production:
  -row-mt 1, -auto-alt-ref 0, alpha_mode=1 metadata
- Frame capture: Emulation.setDefaultBackgroundColorOverride a=0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 02:12:39 +00:00
Vance Ingalls 20be2ea1c2 style: apply oxfmt baseline formatting across all source files (#25) 
## Summary
- Run `oxfmt .` across the entire codebase to establish formatted baseline
- 299 files changed — mechanical formatting only, no logic changes
- Double quotes, semicolons, 2-space indent, trailing commas, 100 print width

Part 3/4 of [VA-851](https://linear.app/heygen/issue/VA-851/pre-migration-configure-eslint-prettier-and-conventional-commits)

## Test plan
- [x] `pnpm format:check` — all 426 files pass
- [x] `pnpm -r typecheck` — all packages pass
- [x] `pnpm build` — all packages build
- [x] All 348 tests pass
 2026-03-23 17:15:14 -07:00
Vance IngallsandClaude Opus 4.6 9f8e5ba5a1 initial code (#2) 
* feat: initial code port from hyperframes-internal

Port all OSS-ready packages from the internal monorepo:
- @hyperframes/core — shared types, HTML generation, GSAP utilities, runtime
- @hyperframes/cli — CLI for creating, previewing, and rendering compositions
- @hyperframes/engine — framework-agnostic rendering engine (BeginFrame + FFmpeg)
- @hyperframes/producer — video rendering pipeline (Puppeteer + FFmpeg)
- @hyperframes/ui-player — browser-based video player component
- @hyperframes/studio — composition editor (React frontend + Hono backend)

Includes regression test suite with Docker-based test harness.

All HeyGen-internal references, deployment infrastructure, and
proprietary assets have been removed. Package names migrated
from @app/* to @hyperframes/*.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: scrub internal codenames and stale references from OSS port

- Replace static.heygen.ai runtime URLs in test fixtures
- Remove internal CDN publish script (publish-hyperframe-runtime.ts)
- Replace sandbox-studio, sandbox-interceptor, __magicEditRuntime
  with neutral names (studio, hyperframe-runtime, __hyperframeRuntime)
- Fix stale Vault API / localhost references in docs
- Remove broken deprecated_studio link

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove remaining internal codenames and stale references

- Delete stale producer README.md and PIPELINE.md (referenced nonexistent files)
- Replace "Cerberus" codename with "HyperFrames" in test design reviews
- Replace magic-edit postMessage identifiers with hf-preview/hf-parent
- Rename debug-magic-edit-timeline.ts to debug-timeline.ts
- Replace "Motion Cut" with "HyperFrames" in Timeline comments
- Fix studio/CLI references to nonexistent archive package
  (use local data/projects/ dir, stub render proxy)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-21 22:43:56 -07:00