mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 23:03:09 +00:00
1f5dbdbc4cc1d9faebe56c8ffd34a3a319b3a91b
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1f5dbdbc4c |
fix(studio,producer): post-merge regressions and open issues (#1643)
- fix gsapDragCommit.test.ts import for commitGsapPositionFromDrag - add generateId() helper with insecure-context fallback (fixes #1637) - enable STUDIO_KEYFRAMES_ENABLED by default - prevent clock duration from shrinking during playback (fixes #1636) - add maxRetries to rmSync cleanup calls to handle ENOTEMPTY race (fixes #1644) |
||
|
|
8408a44745 |
fix(engine): tune VP9 cpu-used across render paths (#1614)
* fix(engine): tune VP9 cpu-used across render paths * fix: address VP9 review feedback |
||
|
|
7310223b66 |
feat(engine): static-frame dedup default-on + render telemetry (#1549)
* feat(engine): static-frame dedup for screenshot capture (opt-in) Skip re-seeking + re-screenshotting frames byte-identical to their predecessor. A frame is dedupable iff no GSAP tween or clip cut is active in it or its predecessor (predicted from window.__timelines + clip schedule) AND an empirical anchor-compare confirms it. Opt-in HF_STATIC_DEDUP=true, default off. Correctness (designed for the multi-worker / distributed render paths): - Reuse is keyed by the ABSOLUTE composition frame (derived from the frame's time), NOT the captureFrameCore frameIndex arg — chunked/parallel callers pass a chunk- relative index. Validated lossless (PSNR=inf) on both single- and multi-worker renders of a static-hold comp. - verifyStaticFramesSafe checks EVERY run (no longest-first budget truncation that left runs armed-but-unverified), and samples each run's FIRST reused frame, its END, and interior points at a stride; a hard cap disables dedup rather than trust an unverified set. - Conservative arming: skipped when capture mode != screenshot (BeginFrame tick semantics + the verifier's screenshot path wouldn't transfer), when a before-capture hook is set (per-frame video injection), when page-side compositing is active (shader / drawElement composite the plain verification screenshot can't reproduce), and when any data-start is a non-numeric reference expression the clip-boundary parser can't protect, or duration is unknown/zero. - Session reuse (prepareCaptureSessionForReuse) resets lastFrameBuffer + dedup counter so a probe/prior-render buffer can't bleed into the first static frame; the armed set is kept (same-composition reuse). Cost calibration bypasses dedup for its sparse, non-contiguous sample sweep, then restores the armed set. - HF_STATIC_DEDUP_SAMPLES is NaN-guarded. Disqualifies on signals the GSAP predictor can't see: video, canvas/webgl, zero tweens, running CSS/WAAPI animation. Pays on static-hold content (title cards, slideshow/kiosk loops, data-viz pauses); no-op on continuously-animated comps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(engine): static-frame dedup default-on + render telemetry Flip dedup from opt-in (HF_STATIC_DEDUP=true) to default-on (opt-out HF_STATIC_DEDUP=false). Verification (verifyStaticFramesSafe) is the safety net that keeps reuse sound at scale. Add end-to-end dedup observability. The capture session records enabled / armed / skipReason / predicted; these surface via CapturePerfSummary -> a dedupPerfs accumulator (disk sequential + parallel AND streaming sequential + parallel) -> aggregated into RenderPerfSummary.staticDedup (OR armed, SUM frames across workers) -> render_complete props static_dedup_{enabled,armed,skip_reason, predicted_frames,reused_frames}. skip_reason is a low-cardinality code: capture_mode | video_injection | page_composite | ineligible | verification_failed. Distributed chunks run on Linux/beginframe where dedup never arms, so they pass a throwaway dedupPerfs sink (no per-chunk reporting). Tests: aggregation logic (OR/SUM/skip-reason) + opt-out passthrough. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(engine): address review on dedup default-on + telemetry Review feedback (miga-heygen) + self-review fixes: - Retry double-count: executeDiskCaptureWithAdaptiveRetry pushed worker dedup perf inside the retry loop, so an adaptive retry counted frames twice (reused/predicted could exceed totalFrames). Reset dedupPerfs at the start of each attempt — retry now REPLACES rather than accumulates; common no-retry path is unchanged. - Opt-out parsing: HF_STATIC_DEDUP now disables on {false,0,off} case/space-insensitive (was strict !== "false", so `False`/`0` silently kept dedup on — the kill-switch could no-op). - Verification budget vs drift: verifyStaticFramesSafe returns {badFrame, budgetExhausted}; armStaticDedup reports a distinct `verification_budget` skip reason so a telemetry spike means "raise HF_STATIC_DEDUP_SAMPLES", not "compositions are non-static". - Index idiom: captureFrameCore now uses Math.floor(time*fps + 1e-9) (matches quantizeTimeToFrame) so the dedup lookup agrees with the frame the seek lands on even for non-exact times. - Stale "opt-in HF_STATIC_DEDUP=true" comments -> "opt-out HF_STATIC_DEDUP=false" across frameCapture.ts + types.ts. - Extract pushWorkerDedupPerfs helper (perfSummary.ts), used by the disk and streaming parallel paths — removes the duplicated push loop and drops captureStreamingStage back under the complexity threshold. - dedupPerfs is now required (not optional) on executeDiskCaptureWithAdaptiveRetry — a missing arg silently dropped telemetry. - Test: captureStreamingStage createInput() now provides the required dedupPerfs field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(engine): address deferred dedup-review items - Derivable state: drop session.staticDedupArmed/staticDedupPredicted; derive both from session.staticFrames in getCapturePerfSummary (armed ⟺ non-empty set, predicted === size) so they can't desync. - Config altitude: HF_STATIC_DEDUP now resolves into EngineConfig.staticFrameDedup (resolveConfig, opt-out on {false,0,off}), alongside forceScreenshot/browserGpuMode — armStaticDedup reads config instead of process.env. Default-on preserved (missing config → enabled). - Lossy aggregation: aggregateDedup now reports DISTINCT skip reasons (sorted, `|`-joined) across diverging unarmed workers instead of just the first. - discardWarmupCapture: also snapshot/restore staticDedupCount and lastFrameBuffer so a warmup capture can't leak a phantom reuse or a stale buffer anchor into the real summary. - Convention: perfSummary-dedup.test builds its job via createRenderJob instead of `as unknown as RenderJob`. - Docs: verification_budget added to skip-reason lists. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
1f9c70cf15 |
fix(producer): guard fileServer.close in all render cleanup paths (#1406)
* fix(producer): guard fileServer.close in distributed cleanup paths `plan()` and `renderChunk()` both close the probe/chunk file server with a bare `fileServer.close()` in their cleanup sequence. `FileServerHandle.close` tears down the underlying http.Server, whose `close()` throws `ERR_SERVER_NOT_RUNNING` if the server was already torn down (for example a cancellation path that closed it once already). An unguarded throw there escapes the cleanup and masks the original plan/render result, exactly the failure the adjacent probe-session close already guards against with a try/catch (its comment even spells this out). Add `closeFileServerSafely`, which wraps the close in a try/catch and logs, and route both cleanup sites through it so the two stay consistent and a throwing close can never mask the real result. Covered by unit tests for both the throwing and happy paths. * fix(producer): extend fileServer.close guard to renderOrchestrator success path --------- Co-authored-by: Carlos Alcaraz <193642530+calcarazgre646@users.noreply.github.com> |
||
|
|
e6b8d66c2d |
feat(cli,producer): add gif output format with two-pass palette encode (#1333)
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
|
69eac249d5 | feat(producer): stage wall-clock split in chunk perf telemetry (#1327) | ||
|
|
852008bd44 |
feat(producer): thread variables through plan() + renderChunk() (#962)
Add `variables?: Record<string, unknown>` to DistributedRenderConfig (§4.4) and LockedRenderConfig (§4.3). plan() snapshots the value into meta/encoder.json so every chunk worker re-injects the same set via captureOptions.variables, mirroring the in-process renderer's path. The variables fold into planHash automatically because canonical encoder.json bytes feed the hash: two plans with different variables produce different hashes (chunked output depends on the injected values); two plans with the same variables produce identical hashes because canonical-JSON sorts keys. The regression harnesses (distributed-simulated, lambda-local) also forward the input's variables to plan() / Step Functions event so fixtures that declare `renderConfig.variables` produce the same pixels across modes. Previously the field was on the harness input shape but silently dropped at the call boundary. Phase 9 PR 9.1 of the distributed rendering plan. |
||
|
|
5d264e146c |
docs(lambda): document webm support + simplify-review fixes (#953)
* docs(lambda): document webm support in distributed mode PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the shipped capability. Updates docs/deploy/migrating-to-hyperframes-lambda.mdx: - "Output format" row in the migration table now lists `webm` alongside mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 + closed-GOP concat-copy. HDR mp4 remains the only refused format. - "No webm distributed" caveat replaced with "webm uses closed-GOP VP9" explainer covering the encoder args (`-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why alt-ref disable is load-bearing, and that the output preserves alpha via yuva420p with Opus audio. - Migration checklist no longer asks adopters to filter out webm compositions; only HDR-dependent renders need to stay on the previous framework. aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR in the v1 surface list), so it gets no copy edits beyond the migration guide. The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8, §12 — kept outside the repo) gets matching updates: format support matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the new top item, and the rev-12 → rev-13 status line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: address simplify-review findings on webm stack Folds in cleanups identified by a multi-agent code-review pass over the 4-PR webm-distributed stack: - plan.ts: `resolveEncoderTriple()` webm case now calls `getEncoderPreset(quality, "webm")` for its preset string instead of hardcoding "good". The hardcode was wrong for `quality: "draft"` (`getEncoderPreset` returns "realtime" for that tier) — would have silently overridden the draft → realtime mapping for distributed webm renders. - chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18 lines of WHY narration down to the 6 lines that actually explain why (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push comment to the same standard. - chunkEncoder.test.ts: drop the duplicate WHY comment that restated the implementation comment in plain words. - webm-concat-copy.test.ts: rewrite the file-header docstring to describe the contract being tested instead of the PR-8.1-gating history; strip "PR 8.2 / Path A / Path B" references from error messages (they belong in PR bodies, not in test output). Consolidate the yuva420p alpha smoke into a single `it()` block (was a full 4-test describe with duplicated setup) — the yuv420p block already covers the probe/decode/frame-count contract; the alpha smoke only needs to prove the alpha args don't break concat-copy. - plan.test.ts: drop the "PR 8.1 proved the contract" comment. - webm-vp9 fixture: drop the aspirational "Other webm-with-audio fixtures cover the mux path separately when added" sentence (no other fixtures exist). Regenerated the baseline via `docker:test:update webm-vp9` to reflect the updated comment. - migrating-to-hyperframes-lambda.mdx: add a paragraph about distributed webm's perf cost — ~10-25% larger files at constant CRF due to forced keyframes, and slower per-chunk encode due to `-cpu-used 2` being more conservative than the libvpx default. All unit tests + the webm-vp9 distributed-simulated regression still pass after these changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): accept --format=webm in `hyperframes lambda render` The CLI's `lambda render` subcommand's FORMATS allowlist and the `RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`, so even though the producer + aws-lambda packages now support webm end-to-end, the CLI surface rejected it with `--format must be mp4|mov| png-sequence`. Add webm to both spots and update the --help description. Surfaced during real-AWS deploy prep — the local lambda-local / distributed-simulated tests didn't go through the CLI so the gap went unnoticed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME) The deterministic Google Fonts cache was rooted at `\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's read-only. `mkdirSync(..., { recursive: true })` can't create that path and the plan stage trips with `ENOENT: no such file or directory, mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on every Lambda render that pulls a Google Font (i.e. every distributed fixture using `@import url("https://fonts.googleapis.com/...")`). Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to `tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives across invocations on a warm container, so cache hit rate is the same as non-Lambda runs. Also honor an explicit `\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a different location regardless of the runtime. Surfaced while verifying webm distributed end-to-end on real AWS — the same bug affects mp4 fixtures using Google Fonts; webm just happened to be the one I tried first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract DistributedFormat type + trim font-cache resolver Second simplify-review pass on the webm stack flagged two cleanups: 1. **`DistributedFormat` type duplicated 10 times.** Every file in the distributed pipeline carried its own copy of `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format meant a 10-place edit with no compile-time guarantee they stayed in sync. Extract a single source of truth in `packages/producer/src/services/distributed/shared.ts`, re-export from `@hyperframes/producer/distributed` and `@hyperframes/aws-lambda/sdk`, and have all callers pull from there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]` so the compiler enforces the runtime allowlist stays in sync with the type. 2. **`deterministicFonts.ts` font-cache resolver was over-commented.** Trim the 7-line block to 4 lines (drop the aspirational "and other read-only-FS execution environments" — only Lambda is detected — and the warm-container `/tmp` persistence narration — anyone reading already knows Lambda /tmp semantics). Collapse the two-step `if (explicit && explicit.length > 0)` into a single nullish-coalesce expression now that the empty-string defensive check is gone (`process.env.X` is `string | undefined`, no third shape to guard against). Out-of-scope skips (called out by the agents, deferred): - In-process `RenderConfig.format` and the in-process CLI's `render.ts` format union still carry their own inline copies. The union happens to coincide today but they're separate concerns — leaving them alone limits this PR's blast radius. - `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry flagged as taste; skipping. - Pre-existing redundant `existsSync` before `mkdirSync({ recursive: true })` in `fontCacheDir` — out of scope. All tests + typecheck still pass. Lambda render still works end-to-end (no functional changes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(lambda): drop plan-doc reference from migration checklist PR review feedback: source/docs should not mention the distributed-rendering planning doc. Tighten the migration checklist sentence to describe the webm path directly rather than referencing the doc's version label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 — the function interleaved (a) mp4 codec validation + dispatch, (b) the non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the top-level function's cyclomatic complexity below the threshold while preserving every error message and code path. Behavior unchanged. Also extracts an `EncoderTriple` type alias so the three functions share the return shape declaratively rather than repeating it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
21f5066832 |
feat(producer): enable webm in distributed mode via concat-copy (#951)
* feat(producer): enable webm in distributed mode via concat-copy PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the distributed pipeline now that PR 8.1 proved concat-copy works. Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams). The simpler architecture wins; no re-encode in assemble, no encode- parallelism loss. Changes: - plan.ts: - DistributedRenderConfig.format and PlanResult.format now include "webm" — type-level acceptance matches the runtime gate. - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR mp4 remains the only refused configuration. - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p + preset="good" for format="webm". yuva420p preserves alpha — the format's main reason for existing for web delivery. - codec= remains rejected for non-mp4 formats (mov is always ProRes 4444; webm is always libvpx-vp9). The error message lists all four distributed-supported formats. - FormatNotSupportedInDistributedError docstring updated to reflect the new reality (only HDR is unsupported). - freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software". Mirrors libx265-software / prores-software / png-sequence in shape; the chunk worker reads this discriminant to decide encode args. - renderChunk.ts: drops the now-incorrect cast that excluded webm from buildSyntheticRenderJob's format input; tightens the preset-format cast to include webm. - assemble.ts: docstring + comment updates. The mp4/mov concat-copy path is format-agnostic — webm uses the exact same code (applyFaststart is a no-op for webm via the existing chunkEncoder.ts gate; muxVideoWithAudio already routes webm to libopus audio). - planFormatBanlist.test.ts: webm-rejection tests removed; replaced with "accepts webm" tests + a HDR+webm combo test that verifies HDR is the trip regardless of format. - plan.test.ts: new describe block pins the webm wiring contract: format="webm" produces an encoder=libvpx-vp9-software / pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize. - webm-concat-copy.test.ts (smoke): extended with a yuva420p variant that proves the alpha pixel format the distributed pipeline actually emits also round-trips through concat-copy. 9/9 tests pass locally. §8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end fixture (PR 8.3) is green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke PR review feedback from Miguel and Vai on #951 caught a real bug: `plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan stage froze `forceScreenshot: false` into the `LockedRenderConfig` even though distributed webm uses `yuva420p`. Every chunk worker captured opaque RGB via BeginFrame (which doesn't preserve alpha on Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha that the encoder then dropped — producing un-keyable webm. Two changes: 1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the in-process renderer's logic at `renderOrchestrator.ts:1469` (`const needsAlpha = isWebm || isMov || isPngSequence`); the two sites must stay in sync since the distributed pipeline's PSNR regression compares against the in-process baseline. 2. **Smoke test (yuva420p describe)**: source frames now use a real alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of `testsrc2 + format=rgba` which was uniformly opaque. The decode- pix_fmt assertion is dropped (ffprobe reports `yuv420p` for VP9-with-alpha because the alpha lives in a Matroska `BlockAdditional` sidecar) and replaced with two stronger checks: - `TAG:ALPHA_MODE=1` is present on the stream — proves the encoder was actually configured for alpha - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba -vf extractplanes=a,signalstats` — proves the alpha sub-stream round-trips through concat-copy with spatially-varying content, not uniform/dropped alpha - decode-test gate is now exit-code-only (was `exitCode || stderr` which would flake on chatty ffmpeg `-v error` builds emitting non-fatal DTS/container notes) These checks would have caught the `needsAlpha` bug before review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(aws-lambda): widen narrow format types to include webm CI on PR #951 was failing at typecheck/build because the producer's `DistributedRenderConfig.format` widened to include webm in this PR but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"` type literals in `events.ts`, `handler.ts`, and `validateConfig.ts` hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now including webm) into a parameter typed against the narrow union, producing TS2345. This widening originally landed in PR #952 (test fixture PR) but needs to be atomic with the producer's widening here to keep each PR independently typecheck-clean. Also refactor `formatExtension` from a switch dispatch to a `Record<DistributedFormat, string>` lookup. Adding the webm case tipped the switch's CRAP to the 30.0 fallow threshold; the lookup table drops cyclomatic from 5 to 1 with the same compile-time exhaustiveness guarantee (TS errors on missing entries when `DistributedFormat` adds a new format). The runtime `_exhaustive: never` throw was only protecting against a string slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already gates untrusted input at the SDK boundary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
17f47f30dd |
fix(distributed): gate per-worker SwiftShader probe to worker 0 only (#956)
After #916 moved `assertSwiftShader` from `renderChunk()`'s eager probe session into `executeWorkerTask`, every parallel worker began running its own `chrome://gpu` / canvas-WebGL probe. At `chunkWorkerCount=6` (texture launch at chunks=3) that's 6 concurrent CDP page-loads per chunk × 3 chunks = 18 simultaneous probes. Bench data on dev (12 producer pods × 22 vCPU) showed c=3 worst-case wall-clock at 67.3s, 24.7s above c=6 worst (42.6s) — pod_total inflates 100s → 147s uniformly across all three chunks per slow iter, the signature of cluster-level CDP contention rather than within-pod contention. Workers within a chunk share the same Chrome binary, flags, and OS/driver state on a single pod, so worker 0's success is representative for the rest. Gate the probe via `shouldVerifyWorkerGpu(workerId, config)` so only worker 0 navigates to the probe page; workers 1..N-1 skip it. The fail-fast contract still holds at the chunk level (worker 0 still aborts the chunk if SwiftShader didn't load) — just without the concurrent CDP traffic. Expected wall-clock impact: c=3 worst drops from ~67s to in line with c=6 worst (~42-44s). c=6 (3 workers/pod) and c=8 (2 workers/pod) should see smaller wins; c=12 (1 worker/pod, sequential branch) is unaffected. Closes #955. |
||
|
|
f01fccb0ea | perf(distributed): skip eager probe session when chunkWorkerCount > 1 (#916) | ||
|
|
22363a11c9 |
perf(distributed): parallelize chunk capture across multiple workers (#906)
* perf(distributed): parallelize chunk capture across multiple workers
The distributed `renderChunk` primitive hardcoded `workerCount: 1` and
`captureStage` explicitly forbade `workerCount > 1` when `frameRange` was
set, with the comment:
"Distributed chunk workers fan out at the activity layer; reduce
workerCount to 1 when passing frameRange."
The assumption was that orchestration-layer fan-out (Temporal / Lambda /
K8s Jobs / SSH) saturates the available CPU on its own. In practice
adopters that deploy chunks onto multi-core hosts (8-24 vCPU is the
standard producer-worker pod sizing) end up pinning only ~3-4 cores per
chunk while the rest sit idle: chunk-level fan-out at the orchestration
layer gives each pod one chunk at a time, and the chunk render itself
was single-threaded.
Validated against a real 1080p / 30fps / 22-second shader-heavy
composition on a 22-vCPU Temporal pod: each chunk rendered at
165-273ms per frame (vs 94-98ms for the in-process streaming render
which runs `workerCount=2` by default). The slowest chunk gates total
wall-clock under parallel chunk fan-out, so the 2-3x per-frame gap
compounds and `distributed` was net-slower than `in-process` on every
composition smaller than ~5min of texture-class content. Lifting the
restriction is a measured ~2x per-chunk speedup with no contract
change at the framesDir or encoder layer.
Wire-up:
* `WorkerTask.outputFrameOffset` — optional offset subtracted from the
absolute frame index when computing the captured file's name.
Default 0 (the in-process contract; file name == absolute index).
Distributed chunks set this to the chunk's startFrame so file names
land 0-indexed within the chunk's range, matching the sequential
chunk-capture contract and the encoder's expectation that frames
are read sequentially without an `-start_number` override.
* `distributeFrames(totalFrames, workerCount, workDir, rangeStart=0)` —
offsets both `startFrame`/`endFrame` (used for per-frame time math
on the page's virtual clock) by `rangeStart`, and threads
`outputFrameOffset = rangeStart` onto each task it emits. With the
default `rangeStart=0` it is a no-op for in-process renders.
* `executeWorkerTask` — uses `i - (task.outputFrameOffset ?? 0)` for
the captured file name, leaving the per-frame TIME computation
`(i * fps.den) / fps.num` untouched so the page's virtual clock is
unchanged.
* `executeDiskCaptureWithAdaptiveRetry({ frameRangeStart? })` — accepts
the chunk's absolute startFrame and forwards it to `distributeFrames`
and `buildMissingFrameRetryBatches`. Default `undefined` preserves
the in-process contract.
* `buildMissingFrameRetryBatches(ranges, ..., rangeStart=0)` —
`findMissingFrameRanges` walks LOCAL 0-indexed file names; the retry
batch translates the local missing-range pair back to ABSOLUTE
composition indices for `WorkerTask.startFrame/endFrame` and sets
`outputFrameOffset = rangeStart` so the retried capture writes back
to the same local file name.
* `captureStage` — drops the assert; passes
`frameRangeStart: frameRange?.startFrame` to the parallel branch so
workers land on absolute composition frame indices for time math
while file names stay 0-indexed within the chunk range. Docstring
updated to reflect that the parallel branch is now supported.
* `renderChunk` — `workerCount: 1` → `workerCount: 2`. The pre-warmed
`probeSession` is consumed only by the sequential branch; the
parallel branch closes it during stage entry and creates its own
worker sessions. Documented as a follow-up: skip probeSession
creation when `workerCount > 1` to recover the ~3-5s warmup cost.
Backwards compatibility: every change is gated on a parameter that
defaults to the prior behavior. In-process callers (`executeRenderJob`)
pass no `frameRangeStart`, so `rangeStart === 0`, `outputFrameOffset`
defaults to 0, and the file-name math collapses to the prior `i` value.
The framesDir contract (`frame_0..frame_(totalFrames-1)`) and the
WorkerTask interface are extended, not replaced.
Tests: 24 pass / 0 fail across the distributed test suite (renderChunk,
plan, assemble, planFormatBanlist, planSizeCap, publicExports). 7 pass /
0 fail in `parallelCoordinator.test.ts`. The renderOrchestrator suite
has one pre-existing Windows-only failure
(`writeCompiledArtifacts — external assets on Windows drive-letter
paths`) unrelated to this change; the other 56 tests pass.
Refs: distributed-vs-inprocess benchmark thread at
heygen-com/experiment-framework#36950
* perf(distributed): auto-size chunk workerCount via calculateOptimalWorkers
Match the in-process renderer's worker selection instead of hardcoding 2.
`calculateOptimalWorkers(framesInChunk, undefined, cfg)` is the same call
`resolveRenderWorkerCount` makes under the hood, minus the capture-cost
calibration reduction (which would require plumbing the chunk's compiled
metadata through — left as a follow-up).
For a typical 22-vCPU producer-worker pod with `cfg.concurrency: "auto"`
this resolves to ~6 workers for a 240-frame chunk (capped by
`defaultSafeMaxWorkers() = max(6, min(16, floor(cpuCount/8)))`), matching
what `executeRenderJob` (the in-process path) already does. The prior
hardcoded `workerCount: 2` was a safe-minimum starting point that
undersized chunks vs prod's auto behavior.
Tests: 12/12 pass in `renderChunk.test.ts` (unchanged — the test suite
mocks the inner runCaptureStage call so workerCount selection is opaque
to it).
* refactor(distributed): /simplify pass on PR #906
Review pass on the parallel-capture frame-range change. Four targeted
cleanups identified by code-quality and efficiency review agents:
1. Add the missing `frameRange.endFrame - frameRange.startFrame === totalFrames`
assert. The parallel branch forwards `totalFrames` separately from
`frameRangeStart`; a caller passing mismatched values would have got a
silently wrong distribution. The sequential branch already implicitly
relied on this via its `rangeFrames = rangeEnd - rangeStart` arithmetic.
2. Collapse three near-duplicate docstrings (on `WorkerTask.outputFrameOffset`,
`executeDiskCaptureWithAdaptiveRetry.frameRangeStart`, and `runCaptureStage`'s
`frameRange`) so only the WorkerTask field carries the full contract. The
other two cross-reference it.
3. Drop the WHAT-narrating comments inside `executeWorkerTask`'s per-frame
loop. The variable names (`fileFrameIdx = i - outputOffset`) already say
what the line does; the only remaining comment flags the non-obvious
contract that the streaming callback gets the absolute index.
4. Trim the 30-line `chunkWorkerCount` block in `renderChunk` to one paragraph
explaining the one non-obvious thing (why we use `calculateOptimalWorkers`
directly instead of `resolveRenderWorkerCount`). The probeSession-wasted-on-
parallel acknowledgement stays as a 3-line follow-up flag — investigated
skipping it in this pass, but the SwiftShader probe is safety-critical and
has no per-worker equivalent, so deferred to a separate change with proper
per-worker assertion plumbing.
Tests + format + lint clean:
* `bun test parallelCoordinator.test.ts` — 7/7
* `bun test distributed/{renderChunk,plan}.test.ts` — 24/24
* `bunx oxfmt` + `bunx oxlint` — clean
|
||
|
|
38e08e81c7 |
style(producer): apply oxfmt to resolvePresetForLockedEncoder signature
The generic parameter constraint exceeded oxfmt's line width, so the formatter wraps the type-param list onto its own line. Applies the same formatting locally that CI's 'Format' job would have produced via 'bun run format:check' — no behavior change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
|
db62e31d39 |
fix(producer): reject unknown codec strings + extract testable preset-override helper
Address @vanceingalls review on #850: 1. Unknown codec strings (typos like 'H265', future additions like 'av1') silently fell through to libx264 in resolveEncoderTriple. Add an explicit throw symmetric to the non-mp4-format branch already there. A JS caller building config from JSON who passes 'codec: "h266"' now gets a clear error at plan time instead of unflagged h264 output. 2. The preset.codec override in renderChunk had no fast unit coverage — only the heavyweight Docker fixture in #851 would catch a regression if someone refactored the spread (e.g. moved it into getEncoderPreset itself). Extract resolvePresetForLockedEncoder() and add 4 fast unit tests pinning the four encoder shapes (libx265/libx264/prores/png-seq). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
|
eccea88daf | feat(producer): add codec knob to DistributedRenderConfig | ||
|
|
e50587496f |
fix(producer): address PR review feedback on harness mode + plan() copy filter
Miguel (approved) and Vai (commented) both flagged the same PSNR-threshold doc/code mismatch; Vai additionally flagged a path-anchoring bug in the projectDir-copy filter and a dishonest type cast. Addressed all five findings: PSNR threshold doc/code mismatch (important): - Module docstring, `resolveMinPsnrForMode` JSDoc, and tests/README.md all claimed distributed-simulated tightens to ≥50 dB. The actual code uses `max(fixture.minPsnr, 10)` — 10 dB is a pathology floor, the per-test gate is the fixture's authored `minPsnr`. Updated all three doc sites to describe what the code does. The 50 dB target in §5.1 is a per- render distributed-vs-in-process contract; against the frozen baseline it's unreachable for either mode (shared encoder/JPEG jitter), so it can't be a per-fixture gate. `PLAN_PROJECT_DIR_COPY_SKIP` regex matched absolute paths (important): - `cpSync` calls the filter with the absolute source path, so a `projectDir` whose absolute path happens to contain a blocklisted segment (`/home/user/work/output/comp/`, `~/projects/dist/foo/`, etc.) caused the filter to return false for every descendant — empty compiled directory, broken render. Now matches relative-to-projectDir segments via `path.relative()` + `split(sep)`. Switched from a regex to a Set for clarity. Harness fixtures don't hit this because they live under `tests/<name>/src/`, but adapters call `plan()` with caller-supplied paths. Dishonest type cast in regression-harness.ts (important): - `as "mp4" | "mov" | "png-sequence"` claimed reachability for formats that `validateMetadata` doesn't accept (the schema is `"mp4" | "webm"`, and webm is rejected by `checkDistributedSupport`). Narrowed to hardcoded `format: "mp4"` with a comment naming the metadata-schema invariant that lets us do that. Renamed `chunkVideoInjectorFactory` (nit): - The variable was invoked once and never used again — "factory" implied repeated calls. Inlined as a plain `videoInjector: BeforeCaptureHook | null` ternary. Replaced tautology test (nit): - `expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10)` was a value-pin over an exported constant. The invariant the JSDoc actually asserts is "10 dB is below any real fixture's authored minPsnr"; if someone lands a permissive fixture (minPsnr: 5), the value-pin doesn't catch it. Replaced with a test that walks `tests/*/meta.json` and asserts every authored `minPsnr` is ≥ the floor. Validated in `docker:test --mode=distributed-simulated`: font-variant-numeric, many-cuts, gsap-letters-render-compat, style-1-prod, sub-composition-video — all PASSED. Unit tests: 15/15 pass (new fixture-scan test included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
b8e8617f80 |
refactor(producer): apply /simplify cleanups across distributed harness stack
Code reuse:
- Move `PlanVideosJson` interface + `meta/videos.json` path constant into
`services/distributed/shared.ts`; plan.ts and renderChunk.ts import from
there instead of redeclaring the same shape with the "duplicated here so
renderChunk doesn't import from plan.ts" comment.
- Replace hand-rolled `framePattern.slice(lastIndexOf("."))` with
`extname()` from `node:path` in `rebuildExtractedFramesFromPlanDir`.
Efficiency:
- Hoist `rebuildExtractedFramesFromPlanDir` + `createFrameLookupTable` +
`createVideoFrameInjector` out of the per-chunk closure in renderChunk.
Computed once per chunk now, not once per `createRenderVideoFrameInjector`
callsite (which `runCaptureStage` may invoke multiple times).
- Add a regex filter to `cpSync(projectDir → planDir/compiled/)` so
`node_modules`, `.git`, `output/`, `failures/`, `dist/`, etc. are not
copied. Real projects can have hundreds of MB in those directories;
shipping them to S3/Lambda /tmp on every render bloats cost and time.
- Drop redundant `if (!existsSync(metaDir)) mkdirSync(metaDir, {recursive:true})`
guards; `mkdirSync({recursive:true})` is already idempotent.
Quality:
- Strip narrative comments that told the story of debugging:
- renderChunk's 30-line "Two failure modes made the call actively
harmful" block → 4-line invariant on why `discardWarmupCapture` is
omitted.
- plan.ts's "DO NOT call cleanup()" block → 3 lines naming the
invariant.
- plan.ts's pre-seed-projectDir block → 7 lines on the file-server
invariant.
- renderChunk.ts top docstring's discardWarmupCapture paragraph.
- regression-harness-distributed.ts's PSNR-drift table (belongs in
DISTRIBUTED-RENDERING-PLAN.md, not the source).
- test file's docstring about which tests live where.
- Drop the unreachable IIFE-throw on `format === "webm"` in the harness
(the support check above rejected webm); replace with a plain
`as` cast.
- Replace dynamic `await import("node:fs")` with a top-level import in
`regression-harness-distributed.ts`.
All 54 distributed unit tests still pass in Docker. Full fixture sweep
in `docker:test --mode=distributed-simulated` (font-variant-numeric,
many-cuts, gsap-letters-render-compat, style-1-prod, sub-composition-video)
all PASSED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
30f931b503 |
fix(producer): wire video frame injector into renderChunk
The chunk worker passed `createRenderVideoFrameInjector: () => null` to `runCaptureStage`, leaving the page's `<video>` elements to decode the source mp4 against the virtual clock. Chrome's native video pipeline seeks ±1 frame off what the in-process renderer captures (which uses pre-extracted frames injected as images via createVideoFrameInjector). That ±1 frame drift produced the PSNR gap on sub-composition-video and style-1-prod against the in-process baselines. Two pieces: 1. `plan()` now persists the engine's `VideoElement[]` (composition.videos) and a serialized form of `extractionResult.extracted` (videoId, srcPath, framePattern, fps, totalFrames, metadata — paths omitted) to `<planDir>/meta/videos.json`. This is the data renderChunk needs to reconstruct a `FrameLookupTable` without re-running the extract stage. 2. `plan()` no longer calls `frameLookup.cleanup()` after extraction. That cleanup was rm-rf-ing each video's outputDir, which for the in-process orchestrator is a scratch tree the renderer owns — but for plan() that "scratch" IS `compiledDir/__hyperframes_video_frames/<videoId>/`, the source material that the subsequent rename moves into `planDir/video-frames/`. Cleaning it up before the rename left planDir/video-frames/ with only the `_downloads/` subdirectory and no actual frame files. Both `style-1-prod` and `sub-composition-video` reproduced this on every distributed-simulated run; both pass after the cleanup is dropped. 3. `renderChunk` reads `meta/videos.json`, rebuilds `ExtractedFrames[]` by re-listing `planDir/video-frames/<videoId>/` for each video, calls `createFrameLookupTable(videos, extracted)`, and wraps the result in `createVideoFrameInjector` — the same hook the in-process renderer uses. The rebuilt entries set `ownedByLookup: false` so any later cleanup() call from the engine doesn't rm the planDir bytes another worker may still be reading. Validated in `docker:test --mode=distributed-simulated`: font-variant-numeric: PASSED many-cuts: PASSED gsap-letters-render-compat: PASSED style-1-prod: PASSED (was: 15 frames at 26-29 dB) sub-composition-video: PASSED (was: most frames at 21-25 dB) In-process unchanged; 54 distributed unit tests still pass in Docker. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
24f64f02c0 |
fix(producer): remove discardWarmupCapture call entirely
Validating the harness against a multi-chunk render (chunkSize=50 on many-cuts, N=4 chunks) revealed that the previous "discard at startFrame-1 for chunk N>0" fix had a second deadlock mode: the discard's frameTimeTicks (base + 49*interval) ended up LARGER than the captureStage first-call's frameTimeTicks (base + 0). Chrome's compositor wedges when asked to go backward in time as predictably as it wedges on a same-time duplicate. Both attempted fixes were trying to work around a problem that doesn't exist: lastFrameCache is only consulted when Chrome returns hasDamage=false, and every chunk frame seeks fresh DOM via __hf.seek() before the screenshot, so hasDamage is always true and the cache is never read. The priming step is unnecessary. Validated: - many-cuts at chunkSize=50 (N=4 chunks): distributed-simulated PASSED - many-cuts at default chunkSize (N=1): distributed-simulated PASSED - font-variant-numeric (N=1): distributed-simulated PASSED - 39 unit tests across distributed/ : PASSED in Docker - in-process mode unchanged: font-variant-numeric + many-cuts PASSED Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
9273eb2229 |
fix(producer): correct discardWarmupCapture chunk-0 deadlock and walk back probe overcorrection
Empirical investigation of --mode=distributed-simulated against many-cuts revealed that the BeginFrame "hang" attributed earlier to a Chrome 148 SwiftShader compositor wedge was actually a renderChunk bug: discardWarmupCapture was called with frameIndex=slice.startFrame, then captureStage immediately captured frame 0 (relative) of the chunk's range. For chunk 0 (slice.startFrame=0) these two calls produced the same frameTimeTicks. Chrome's HeadlessExperimental.beginFrame deadlocks when called twice in a row with the same frameTimeTicks — the compositor has no new damage to advance for, and the second call hangs until the Puppeteer protocolTimeout fires. Tracing the chunk worker confirmed: warmup call 1 t=0 -> ok warmup call 60 t=1947 -> ok (loop exited) beginFrame call #1 t=2333.33 -> returned, hasData=true, hasDamage=true beginFrame call #2 t=2333.33 -> HANG Fix: discardWarmupCapture skips chunk 0 (no prior frame to prime, and the in-process renderer also has an empty cache at frame 0) and uses slice.startFrame - 1 for chunk N>0 (the actual previous absolute frame, which more accurately matches what the in-process renderer's cache holds at the start of frame N). The engine probe complications I added earlier — multi-step screenshot test, inline data:URL pre-navigation, rastered-bytes assertion — were chasing a phantom and are reverted to the original simple form. chrome-headless-shell @stable on Linux with --use-angle=swiftshader renders BeginFrame screenshots correctly after the warmup loop; what looked like "wedged compositor" was the same frameTimeTicks deadlock masquerading as a Chrome regression. Also lowers the harness's distributed-simulated PSNR floor from 45 dB to 10 dB and switches to using the fixture's own minPsnr for both modes. The 45 dB floor was set against font-variant-numeric's static-content baseline drift (~48 dB), but dynamic compositions like many-cuts produce 34-44 dB baseline drift even in-process — both renderers share the same encoder/JPEG jitter floor, so requiring distributed to clear a tighter threshold than in-process catches no real regression. 10 dB remains as an absolute-pathology guard for fixtures with a permissive authored threshold. Validated end-to-end in `docker:test --mode=distributed-simulated`: font-variant-numeric: PASSED (PSNR ~48 dB, audio correlation 1.000) many-cuts: PASSED (PSNR 37-44 dB across rapid transitions) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
e80bf61d61 |
fix(producer,engine): make distributed renderChunk actually work end-to-end
Three Phase 3 regressions surfaced when validating --mode=distributed-simulated:
engine: probeBeginFrameSupport approved chrome-headless-shell 148 even when
its SwiftShader compositor was wedged. The existing noDisplayUpdates:true
probe returns instantly on 148 and the screenshot variant returned empty
data without erroring. The real capture loop then hung on first frame with
"HeadlessExperimental.beginFrame timed out". Probe now navigates to a small
inline page (matching the real capture's compositor state, not about:blank)
and asserts that 3 back-to-back beginFrame calls each return non-empty
screenshotData. Catches the 148 soft-failure mode; falls back to
Page.captureScreenshot.
producer/plan: plan() didn't copy local assets (style.css, script.js, etc.
referenced by relative URL) into planDir/compiled/. The in-process file
server serves these from projectDir, but the distributed chunk worker's
file server only sees compiledDir. Result: every composition with external
local files rendered as unstyled HTML. Now plan() pre-seeds compiledDir
with cpSync(projectDir, ..., {dereference:true}) before compileStage
overwrites the entry HTML, so the planDir is the self-contained bundle
the docstring claims.
producer/renderChunk: force forceScreenshot:true in the chunk worker's
EngineConfig. Chrome 148's BeginFrame screenshot wedge is content-dependent
— the engine probe (now improved) catches it for some pages but not all,
and the real capture loop hangs on composition-shaped content the probe
can't simulate. Page.captureScreenshot works on every chrome-headless-shell
build we've tested, and executeRenderJob already takes this path for
multi-worker mp4, so the distributed pipeline inherits the proven Linux
reliability profile.
Also lowers the harness's distributed-simulated PSNR floor to 45 dB.
The plan's 50 dB target was written for per-render comparison; against
the frozen baseline file, the in-process renderer itself drifts ~2 dB
due to libx264/JPEG-capture jitter, so 50 dB is empirically unreachable
for either mode. 45 dB tracks the observed ~47-48 dB floor and stays
well above the 30 dB fixture threshold.
Validated:
- font-variant-numeric in distributed-simulated: PASSED (PSNR ~48 dB
across 100 checkpoints, audio correlation 1.000).
- many-cuts surfaces a fourth Phase 3 issue: timing drift on compositions
with external script src= files. First ~5 frames render the
pre-script-execution state and later variants come in ~200 ms late vs
baseline. Tracking separately — the harness mode is correctly detecting
it as a regression, which is the point.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
e55a8d0a0c | feat(producer): add services/distributed/renderChunk.ts |