When the distributed render path stitches chunks with `-c copy`,
ffmpeg averages the container framerate from PTS rather than
carrying the source's exact rational rate, producing values like
`360000/12001` instead of `30/1` and ~5ms duration drift over
60s.
This is a known ffmpeg behavior at the concat-demuxer-copy
boundary. The industry-standard fix is `-r <fps>` as an input
flag on the concat step plus an output flag on the subsequent
mux step — both with `-c copy` retained, no re-encode required.
Three sites updated:
- `assemble.ts` concat step: `-r <fps>` input flag.
- `chunkEncoder.muxVideoWithAudio`: `-r <fps>` output flag.
- `chunkEncoder.applyFaststart`: same, threaded from caller.
Adds `r_frame_rate` + duration-equivalence assertions to
`assemble.test.ts` to close the regression hole.
Remaining review follow-ups:
- killProcessTree now escalates to SIGKILL after 500ms if SIGTERM
doesn't kill the process (same pattern as killTrackedProcesses).
Covers orphan cleanup and dev/local mode tree kill.
- Added unit tests for both new modules:
- processTracker.test.ts (6 tests): track/remove on exit/error,
kill running processes, SIGKILL escalation for SIGTERM-resistant
processes, idempotency.
- orphanCleanup.test.ts (5 tests): tree kill with children,
SIGKILL escalation, non-existent PID handling, orphan detection
returns 0 when clean.
- Blocker: arm 3s force-exit timer BEFORE awaiting cleanup, not
inside .finally(). Prevents hang if drainBrowserPool() blocks on
dead Chrome.
- Reorder cleanup: killTrackedProcesses() (sync, fast) runs first,
then async browser drain. Ffmpeg dies immediately instead of
surviving if the hard timer fires early.
- SIGKILL escalation: processTracker now SIGTERMs all tracked
processes, then SIGKILLs survivors after 500ms grace period.
- Scope pgrep to current user (pgrep -u $(id -u)) so orphan
detection doesn't touch other users' Chrome on shared machines.
- Add process.on('exit') handler for crash paths (unhandled
exceptions/rejections that bypass signal handlers).
- Document Windows no-op behavior on killProcessTree handlers.
FFmpeg's VFR-to-CFR normalization produces slightly different frame
counts across versions due to timestamp rounding in the fps filter.
The ±1 tolerance was too tight for Linux FFmpeg builds. Widen to ±3
frames — still catches the 25% shortfall regression these tests
guard against.
The preview command's shutdown handler only closed the HTTP server,
leaving Chrome (browser pool) and ffmpeg processes alive. This caused
silent resource leaks — orphaned processes consuming CPU and RAM with
no parent.
Root cause: preview.ts never called drainBrowserPool() or killed
tracked ffmpeg processes. The thumbnail browser in studioServer.ts
registered its own competing signal handlers that raced with
preview's shutdown.
Fix:
- Add a central process tracker (processTracker.ts) that registers
every spawned ffmpeg across engine and producer packages
- Centralize thumbnail browser cleanup via exported
closeThumbnailBrowser() instead of scattered signal handlers
- Wire preview shutdown to call closeThumbnailBrowser(),
drainBrowserPool(), and killTrackedProcesses() before closing the
HTTP server (embedded mode)
- Add killProcessTree() for dev/local modes where Chrome runs in a
child process tree
- Add startup orphan detection that finds and kills orphaned
chrome-headless-shell/Puppeteer Chrome processes (PPID=1) from
previously crashed sessions
Closes#1038
Allow omitting the shader field in TransitionConfig to get a smooth CSS
opacity crossfade instead of a WebGL effect. HyperShader manages all scene
visibility regardless of transition type, so shader and CSS crossfade
transitions can now be mixed freely in the same composition.
When shader is omitted:
- No WebGL program is compiled or cached for that transition
- The existing applyFallbackTransition() path handles the crossfade
- No texture prewarming needed — transition is marked ready immediately
Tested: verified with a 3-scene composition (sdf-iris + CSS crossfade)
rendered to MP4. Both transition types render correctly.
engine/src/types.ts: HfTransitionMeta.shader is now optional to match
The renderSeek override in init.ts called seekTimelineAndAdapters() which
only did rootTimeline.totalTime(t) without activating child timelines.
GSAP does not propagate totalTime() to internally paused children.
Also simplifies pollSubCompositionTimelines to always call rebind when
timelines are ready, removing the before/after count comparison that
could skip the rebind on fast page loads.
* docs(lambda): document webm support in distributed mode
PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see
DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the
shipped capability.
Updates docs/deploy/migrating-to-hyperframes-lambda.mdx:
- "Output format" row in the migration table now lists `webm` alongside
mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 +
closed-GOP concat-copy. HDR mp4 remains the only refused format.
- "No webm distributed" caveat replaced with "webm uses closed-GOP VP9"
explainer covering the encoder args (`-g <chunkSize>`,
`-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why
alt-ref disable is load-bearing, and that the output preserves alpha
via yuva420p with Opus audio.
- Migration checklist no longer asks adopters to filter out webm
compositions; only HDR-dependent renders need to stay on the previous
framework.
aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR
in the v1 surface list), so it gets no copy edits beyond the migration
guide.
The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8,
§12 — kept outside the repo) gets matching updates: format support
matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the
new top item, and the rev-12 → rev-13 status line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: address simplify-review findings on webm stack
Folds in cleanups identified by a multi-agent code-review pass over the
4-PR webm-distributed stack:
- plan.ts: `resolveEncoderTriple()` webm case now calls
`getEncoderPreset(quality, "webm")` for its preset string instead of
hardcoding "good". The hardcode was wrong for `quality: "draft"`
(`getEncoderPreset` returns "realtime" for that tier) — would have
silently overridden the draft → realtime mapping for distributed webm
renders.
- chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18
lines of WHY narration down to the 6 lines that actually explain why
(alt-ref + cpu-used drift). Match the alpha branch's idempotent-push
comment to the same standard.
- chunkEncoder.test.ts: drop the duplicate WHY comment that restated
the implementation comment in plain words.
- webm-concat-copy.test.ts: rewrite the file-header docstring to
describe the contract being tested instead of the PR-8.1-gating
history; strip "PR 8.2 / Path A / Path B" references from error
messages (they belong in PR bodies, not in test output). Consolidate
the yuva420p alpha smoke into a single `it()` block (was a full
4-test describe with duplicated setup) — the yuv420p block already
covers the probe/decode/frame-count contract; the alpha smoke only
needs to prove the alpha args don't break concat-copy.
- plan.test.ts: drop the "PR 8.1 proved the contract" comment.
- webm-vp9 fixture: drop the aspirational "Other webm-with-audio
fixtures cover the mux path separately when added" sentence (no
other fixtures exist). Regenerated the baseline via
`docker:test:update webm-vp9` to reflect the updated comment.
- migrating-to-hyperframes-lambda.mdx: add a paragraph about
distributed webm's perf cost — ~10-25% larger files at constant CRF
due to forced keyframes, and slower per-chunk encode due to
`-cpu-used 2` being more conservative than the libvpx default.
All unit tests + the webm-vp9 distributed-simulated regression still
pass after these changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): accept --format=webm in `hyperframes lambda render`
The CLI's `lambda render` subcommand's FORMATS allowlist and the
`RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`,
so even though the producer + aws-lambda packages now support webm
end-to-end, the CLI surface rejected it with `--format must be mp4|mov|
png-sequence`. Add webm to both spots and update the --help description.
Surfaced during real-AWS deploy prep — the local lambda-local /
distributed-simulated tests didn't go through the CLI so the gap went
unnoticed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME)
The deterministic Google Fonts cache was rooted at
`\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the
runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's
read-only. `mkdirSync(..., { recursive: true })` can't create that
path and the plan stage trips with `ENOENT: no such file or directory,
mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on
every Lambda render that pulls a Google Font (i.e. every distributed
fixture using `@import url("https://fonts.googleapis.com/...")`).
Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to
`tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives
across invocations on a warm container, so cache hit rate is the same
as non-Lambda runs. Also honor an explicit
`\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a
different location regardless of the runtime.
Surfaced while verifying webm distributed end-to-end on real AWS — the
same bug affects mp4 fixtures using Google Fonts; webm just happened to
be the one I tried first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: extract DistributedFormat type + trim font-cache resolver
Second simplify-review pass on the webm stack flagged two cleanups:
1. **`DistributedFormat` type duplicated 10 times.** Every file in the
distributed pipeline carried its own copy of
`"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format
meant a 10-place edit with no compile-time guarantee they stayed in
sync. Extract a single source of truth in
`packages/producer/src/services/distributed/shared.ts`, re-export
from `@hyperframes/producer/distributed` and
`@hyperframes/aws-lambda/sdk`, and have all callers pull from
there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's
`FORMATS` tuple now both use `satisfies readonly DistributedFormat[]`
so the compiler enforces the runtime allowlist stays in sync with
the type.
2. **`deterministicFonts.ts` font-cache resolver was over-commented.**
Trim the 7-line block to 4 lines (drop the aspirational
"and other read-only-FS execution environments" — only Lambda is
detected — and the warm-container `/tmp` persistence narration —
anyone reading already knows Lambda /tmp semantics). Collapse the
two-step `if (explicit && explicit.length > 0)` into a single
nullish-coalesce expression now that the empty-string defensive
check is gone (`process.env.X` is `string | undefined`, no third
shape to guard against).
Out-of-scope skips (called out by the agents, deferred):
- In-process `RenderConfig.format` and the in-process CLI's
`render.ts` format union still carry their own inline copies. The
union happens to coincide today but they're separate concerns —
leaving them alone limits this PR's blast radius.
- `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry
flagged as taste; skipping.
- Pre-existing redundant `existsSync` before `mkdirSync({ recursive:
true })` in `fontCacheDir` — out of scope.
All tests + typecheck still pass. Lambda render still works
end-to-end (no functional changes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(lambda): drop plan-doc reference from migration checklist
PR review feedback: source/docs should not mention the
distributed-rendering planning doc. Tighten the migration checklist
sentence to describe the webm path directly rather than referencing
the doc's version label.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers
CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 —
the function interleaved (a) mp4 codec validation + dispatch, (b) the
non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting
into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the
top-level function's cyclomatic complexity below the threshold while
preserving every error message and code path. Behavior unchanged.
Also extracts an `EncoderTriple` type alias so the three functions
share the return shape declaratively rather than repeating it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Description
PR 1 of 4 in the WebM (VP9) distributed-rendering series. A gating
experiment that proves closed-GOP libvpx-vp9 chunks survive
`ffmpeg -f concat -c copy` losslessly, so the rest of the stack can
ship Path A (concat-copy) rather than the slower
re-encode-in-assemble fallback.
Two changes:
1. **Closed-GOP VP9 encoder args.** `buildEncoderArgs` now lays
`-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, and
`-cpu-used 2` on libvpx-vp9 when `lockGopForChunkConcat=true`.
Mirrors the existing libx264/libx265 branches. The alt-ref disable
is load-bearing — libvpx-vp9's default non-displayable alt-ref
frames can reach across chunk seams and break concat-copy.
`-cpu-used 2` pins the speed/quality tradeoff so chunks encoded on
workers with different libvpx-vp9 defaults produce visually
consistent output across seams. Default (`lockGopForChunkConcat`
unset) preserves the existing in-process VP9 path unchanged.
2. **Concat-copy smoke test** at
`packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts`.
Generates 60 PNGs via lavfi `testsrc2`, encodes them as 4 VP9 chunks
of 15 frames using `buildEncoderArgs` with
`lockGopForChunkConcat=true`, concat-copies via `ffmpeg -f concat -c
copy`, then runs three independent verifications:
`ffprobe -show_streams`, `ffmpeg -f null -` decode test, and
`ffprobe -count_frames`. Each verification surfaces its failure
fingerprint in the error message.
Smoke test passes 6/6 locally → Path A works; the rest of the stack
takes it.
Also exports `buildEncoderArgs` from `@hyperframes/engine` so
adapters / tests can construct args without re-implementing the
contract.
## Testing
- [x] `bunx vitest run --root packages/engine src/services/chunkEncoder.test.ts` — 62/62 pass (new VP9 closed-GOP tests included)
- [x] `bun test packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts` — passes
- [x] `bunx oxlint` + `bunx oxfmt --check` on all changed files — clean
- [x] `bunx tsc --noEmit -p packages/engine/tsconfig.json` — clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* ci: run fallow audit in lefthook pre-commit
Mirrors the same `fallow audit --base ... --fail-on-issues` check that
runs in CI, but locally against HEAD so issues surface at commit time
instead of after the push round-trip.
Scoped to `packages/**` source files via the glob — non-code edits
(README, docs, top-level configs) skip the hook entirely.
Measured locally: ~5s in parallel with the existing lint/format/typecheck
checks. Doesn't extend wall-clock time because typecheck (~11s) is the
long pole, and lefthook runs commands in parallel.
The default `--gate new-only` means inherited findings don't block the
commit — same gate behavior as CI, so local pre-commit and PR audit
agree.
* refactor: delete orphan declarations flagged by fallow
After fallow's auto-fix de-exports unused symbols, oxlint surfaces them
as no-unused-vars. This PR deletes those orphan declarations outright.
Biggest cleanup: studio/src/icons/SystemIcons.tsx shrinks from 132 to 57
lines — 33 unused icon wrappers and their phosphor-icon imports deleted.
Other deletions across 14 more files covering paired getter/setters,
helper functions, dead env constants, internal components with no
callers, and cascading unused imports.
Cascade-causing files held back for follow-up PRs: renderOrchestrator
barrel of captureCost re-exports, telemetry/portUtils/remote barrels,
Button.tsx + ui/index.ts (would orphan whole file), studioMotion
type re-exports.
Test plan: typecheck clean across 8 packages, oxlint + oxfmt clean,
fallow audit exit 0 (remaining findings inherited), cli + studio
vitest suites pass.
After #916 moved `assertSwiftShader` from `renderChunk()`'s eager probe
session into `executeWorkerTask`, every parallel worker began running its
own `chrome://gpu` / canvas-WebGL probe. At `chunkWorkerCount=6` (texture
launch at chunks=3) that's 6 concurrent CDP page-loads per chunk × 3
chunks = 18 simultaneous probes. Bench data on dev (12 producer pods × 22
vCPU) showed c=3 worst-case wall-clock at 67.3s, 24.7s above c=6 worst
(42.6s) — pod_total inflates 100s → 147s uniformly across all three
chunks per slow iter, the signature of cluster-level CDP contention
rather than within-pod contention.
Workers within a chunk share the same Chrome binary, flags, and OS/driver
state on a single pod, so worker 0's success is representative for the
rest. Gate the probe via `shouldVerifyWorkerGpu(workerId, config)` so
only worker 0 navigates to the probe page; workers 1..N-1 skip it. The
fail-fast contract still holds at the chunk level (worker 0 still aborts
the chunk if SwiftShader didn't load) — just without the concurrent CDP
traffic.
Expected wall-clock impact: c=3 worst drops from ~67s to in line with
c=6 worst (~42-44s). c=6 (3 workers/pod) and c=8 (2 workers/pod) should
see smaller wins; c=12 (1 worker/pod, sequential branch) is unaffected.
Closes#955.
Adds a Blocks tab to the Studio left sidebar with the full 78-item registry
catalog (58 blocks + 20 components). Users can browse by category, search by
title/description, preview CDN-hosted poster thumbnails with video-on-hover,
and install items on-demand with one click or drag-to-timeline.
Core changes:
- BlockCategory type + resolveBlockCategory() for 7 categories (Captions, VFX,
Transitions, Effects, Social, Data, Scenes)
- Registry API routes: GET /api/registry/blocks (catalog) + POST install
- StudioApiAdapter extended with listRegistryCatalog + installRegistryBlock
- Vite adapter reads from disk; CLI adapter fetches from GitHub (24h cache)
- BlockParam interface + params on 6 blocks for future parameter controls
Studio UI:
- 4th sidebar tab "Blocks" with responsive grid, category pills, search bar
- BlockCard: CDN poster thumbnail, video autoplay on hover, duration + WebGL badges
- On-demand install: blocks append as sub-compositions on timeline; components
overlay at start=0 spanning full duration with transparent background patching
- TIMELINE_BLOCK_MIME drag-and-drop to timeline
- BlockParamsPanel (Phase 3 scaffold) auto-opens for parameterized blocks
Registry manifests:
- All 58 blocks backfilled with preview: { video, poster } CDN URLs
- All 20 components normalized to object format + poster URLs added
- 6 blocks annotated with params (Liquid Glass/Background, Portal, Chart,
Logo Outro, Magnetic)
- flowchart-vertical preview generated and uploaded to CDN
Run `fallow fix --auto-fixable` to remove `export` keywords from symbols
fallow's reachability analysis identifies as unused. Keeps only the cases
where the symbol is still referenced internally in its own file (so
removing `export` doesn't surface a new oxlint `no-unused-vars` error).
Result: fallow dead-code findings drop from 276 → 208 (68 fewer unused
exports), with no behavior change — each symbol is still defined and used
exactly the same way within its file.
Reverted ~20 files where fallow's auto-fix would have created cascading
"declared but never used" lint errors — those are cases where the symbol
isn't used at all, and properly cleaning them up means deleting the
declaration, not just dropping `export`. Better to land that as a
separate, narrower PR rather than mixing it into a mechanical de-export.
Also reverted four false positives where fallow missed real consumers:
- `captureCost.ts` (renderOrchestrator has two separate import blocks
from the same module; fallow only saw the first)
- `propertyPanelHelpers.ts`, `domEditingLayers.ts` (real internal uses
fallow's reachability missed)
- `render.ts` (functions imported via `await import()` dynamic import,
which fallow's static analysis doesn't follow)
Test plan: bun run --filter '*' typecheck (clean), oxlint + oxfmt clean,
cli/core/studio/engine vitest suites pass (335 + 917 + 576 + 605 tests).
Three changes to fix regression failures without breaking baselines:
1. Revert flattenInnerRoot in producer — use the original innerHTML
inlining that preserves the existing DOM structure. Instead, set
data-hf-authored-id on the HOST element so the scoped proxy can
still rewrite #id selectors for sub-composition scripts.
2. Revert compiled.html baselines to main (no DOM structure changes).
3. Use timeline-count comparison instead of poll duration to decide
whether to call __hfForceTimelineRebind. Compare timeline count
before vs after the poll — rebind only when new timelines appeared
during polling. This correctly identifies async compositions
regardless of fetch speed, while leaving sync compositions
untouched.
1. Only call __hfForceTimelineRebind() when the timeline poll actually
had to wait (pollDuration > 2 intervals). For compositions with
synchronous timeline registration, the rebind was unnecessary and
shifted render timing, causing PSNR regressions in chat and
gsap-letters-render-compat.
2. Regenerate compiled.html baselines for missing-host-comp-id and
overlay-montage-prod to match the new flattenInnerRoot behavior
(data-composition-id stripped from inlined inner roots, replaced
with data-hf-authored-id).
3. Add late-bind polling to runtime init.ts — after external
compositions load, poll for 5s to detect async timelines that
register after initial binding (e.g. from fetch callbacks).
Review items addressed:
1. Mirror video-failure warning in beginFrame path (was screenshot-only)
2. Fix resolveProjectRelativeSrc escape-fallback to use query-stripped
cleanSrc instead of raw src for the normalize/strip arm
3. Export prepareFlattenedInnerRoot from @hyperframes/core/compiler and
consume in the producer instead of duplicating the implementation
4. Use typed Window cast instead of (window as any) for __hfForceTimelineRebind
5. Regenerate docs/public/catalog-index.json with all 6 map blocks
6. Restore Maps nav group in docs.json (catalog generator had merged
them into Data)
resolveProjectRelativeSrc now strips query parameters (e.g. ?v=4)
before joining with the project directory. Browsers ignore query
strings when loading local files, but the filesystem resolver was
looking for the literal path including the query — causing video
extraction to silently skip the file and render frozen first frames.
Previously, a missing video file (404) caused the renderer to hard-fail
after a 45-second timeout waiting for readyState >= 2. Now:
1. pollVideosReady treats errored videos (v.error set or
NETWORK_NO_SOURCE) as ready, so 404'd sources don't block
2. Screenshot mode downgrades the video timeout from a throw to a
console.warn listing affected sources, then continues rendering
3. The composition renders with the missing video as a blank area
instead of failing entirely
Two fixes for compositions that register timelines after async data
loading (e.g. fetch for TopoJSON map data):
1. engine/frameCapture: remove the hosts.length <= 1 early return
so the timeline readiness poll runs for ALL compositions, not just
multi-composition galleries. Single-composition pages with async
setup were silently skipped.
2. core/runtime/init: expose window.__hfForceTimelineRebind() which
resets childrenBound and re-runs bindRootTimelineIfAvailable().
The renderer calls this after all timelines are confirmed present,
ensuring the root player discovers late-registered timelines from
fetch callbacks.
Without these fixes, compositions using fetch() to load data at
runtime would render blank frames because the root player bound
timelines before the async setup completed, and seek() never
reached the unbound composition timeline.
The renderer now waits for all sub-composition timelines to be
registered in window.__timelines before starting frame capture.
Previously only window.__hf root readiness was checked, causing
blank frames when sub-compositions use async data loading (fetch)
or when the headless renderer starts capturing before scripts
complete.
Adds pollSubCompositionTimelines() to both screenshot and
beginFrame render paths, with a diagnostic warning listing which
composition IDs are missing if the timeout expires.
* perf(distributed): parallelize chunk capture across multiple workers
The distributed `renderChunk` primitive hardcoded `workerCount: 1` and
`captureStage` explicitly forbade `workerCount > 1` when `frameRange` was
set, with the comment:
"Distributed chunk workers fan out at the activity layer; reduce
workerCount to 1 when passing frameRange."
The assumption was that orchestration-layer fan-out (Temporal / Lambda /
K8s Jobs / SSH) saturates the available CPU on its own. In practice
adopters that deploy chunks onto multi-core hosts (8-24 vCPU is the
standard producer-worker pod sizing) end up pinning only ~3-4 cores per
chunk while the rest sit idle: chunk-level fan-out at the orchestration
layer gives each pod one chunk at a time, and the chunk render itself
was single-threaded.
Validated against a real 1080p / 30fps / 22-second shader-heavy
composition on a 22-vCPU Temporal pod: each chunk rendered at
165-273ms per frame (vs 94-98ms for the in-process streaming render
which runs `workerCount=2` by default). The slowest chunk gates total
wall-clock under parallel chunk fan-out, so the 2-3x per-frame gap
compounds and `distributed` was net-slower than `in-process` on every
composition smaller than ~5min of texture-class content. Lifting the
restriction is a measured ~2x per-chunk speedup with no contract
change at the framesDir or encoder layer.
Wire-up:
* `WorkerTask.outputFrameOffset` — optional offset subtracted from the
absolute frame index when computing the captured file's name.
Default 0 (the in-process contract; file name == absolute index).
Distributed chunks set this to the chunk's startFrame so file names
land 0-indexed within the chunk's range, matching the sequential
chunk-capture contract and the encoder's expectation that frames
are read sequentially without an `-start_number` override.
* `distributeFrames(totalFrames, workerCount, workDir, rangeStart=0)` —
offsets both `startFrame`/`endFrame` (used for per-frame time math
on the page's virtual clock) by `rangeStart`, and threads
`outputFrameOffset = rangeStart` onto each task it emits. With the
default `rangeStart=0` it is a no-op for in-process renders.
* `executeWorkerTask` — uses `i - (task.outputFrameOffset ?? 0)` for
the captured file name, leaving the per-frame TIME computation
`(i * fps.den) / fps.num` untouched so the page's virtual clock is
unchanged.
* `executeDiskCaptureWithAdaptiveRetry({ frameRangeStart? })` — accepts
the chunk's absolute startFrame and forwards it to `distributeFrames`
and `buildMissingFrameRetryBatches`. Default `undefined` preserves
the in-process contract.
* `buildMissingFrameRetryBatches(ranges, ..., rangeStart=0)` —
`findMissingFrameRanges` walks LOCAL 0-indexed file names; the retry
batch translates the local missing-range pair back to ABSOLUTE
composition indices for `WorkerTask.startFrame/endFrame` and sets
`outputFrameOffset = rangeStart` so the retried capture writes back
to the same local file name.
* `captureStage` — drops the assert; passes
`frameRangeStart: frameRange?.startFrame` to the parallel branch so
workers land on absolute composition frame indices for time math
while file names stay 0-indexed within the chunk range. Docstring
updated to reflect that the parallel branch is now supported.
* `renderChunk` — `workerCount: 1` → `workerCount: 2`. The pre-warmed
`probeSession` is consumed only by the sequential branch; the
parallel branch closes it during stage entry and creates its own
worker sessions. Documented as a follow-up: skip probeSession
creation when `workerCount > 1` to recover the ~3-5s warmup cost.
Backwards compatibility: every change is gated on a parameter that
defaults to the prior behavior. In-process callers (`executeRenderJob`)
pass no `frameRangeStart`, so `rangeStart === 0`, `outputFrameOffset`
defaults to 0, and the file-name math collapses to the prior `i` value.
The framesDir contract (`frame_0..frame_(totalFrames-1)`) and the
WorkerTask interface are extended, not replaced.
Tests: 24 pass / 0 fail across the distributed test suite (renderChunk,
plan, assemble, planFormatBanlist, planSizeCap, publicExports). 7 pass /
0 fail in `parallelCoordinator.test.ts`. The renderOrchestrator suite
has one pre-existing Windows-only failure
(`writeCompiledArtifacts — external assets on Windows drive-letter
paths`) unrelated to this change; the other 56 tests pass.
Refs: distributed-vs-inprocess benchmark thread at
heygen-com/experiment-framework#36950
* perf(distributed): auto-size chunk workerCount via calculateOptimalWorkers
Match the in-process renderer's worker selection instead of hardcoding 2.
`calculateOptimalWorkers(framesInChunk, undefined, cfg)` is the same call
`resolveRenderWorkerCount` makes under the hood, minus the capture-cost
calibration reduction (which would require plumbing the chunk's compiled
metadata through — left as a follow-up).
For a typical 22-vCPU producer-worker pod with `cfg.concurrency: "auto"`
this resolves to ~6 workers for a 240-frame chunk (capped by
`defaultSafeMaxWorkers() = max(6, min(16, floor(cpuCount/8)))`), matching
what `executeRenderJob` (the in-process path) already does. The prior
hardcoded `workerCount: 2` was a safe-minimum starting point that
undersized chunks vs prod's auto behavior.
Tests: 12/12 pass in `renderChunk.test.ts` (unchanged — the test suite
mocks the inner runCaptureStage call so workerCount selection is opaque
to it).
* refactor(distributed): /simplify pass on PR #906
Review pass on the parallel-capture frame-range change. Four targeted
cleanups identified by code-quality and efficiency review agents:
1. Add the missing `frameRange.endFrame - frameRange.startFrame === totalFrames`
assert. The parallel branch forwards `totalFrames` separately from
`frameRangeStart`; a caller passing mismatched values would have got a
silently wrong distribution. The sequential branch already implicitly
relied on this via its `rangeFrames = rangeEnd - rangeStart` arithmetic.
2. Collapse three near-duplicate docstrings (on `WorkerTask.outputFrameOffset`,
`executeDiskCaptureWithAdaptiveRetry.frameRangeStart`, and `runCaptureStage`'s
`frameRange`) so only the WorkerTask field carries the full contract. The
other two cross-reference it.
3. Drop the WHAT-narrating comments inside `executeWorkerTask`'s per-frame
loop. The variable names (`fileFrameIdx = i - outputOffset`) already say
what the line does; the only remaining comment flags the non-obvious
contract that the streaming callback gets the absolute index.
4. Trim the 30-line `chunkWorkerCount` block in `renderChunk` to one paragraph
explaining the one non-obvious thing (why we use `calculateOptimalWorkers`
directly instead of `resolveRenderWorkerCount`). The probeSession-wasted-on-
parallel acknowledgement stays as a 3-line follow-up flag — investigated
skipping it in this pass, but the SwiftShader probe is safety-critical and
has no per-worker equivalent, so deferred to a separate change with proper
per-worker assertion plumbing.
Tests + format + lint clean:
* `bun test parallelCoordinator.test.ts` — 7/7
* `bun test distributed/{renderChunk,plan}.test.ts` — 24/24
* `bunx oxfmt` + `bunx oxlint` — clean
James