mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-04 07:19:52 +00:00
1f5dbdbc4cc1d9faebe56c8ffd34a3a319b3a91b
51
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Miguel Ángel
|
1f5dbdbc4c |
fix(studio,producer): post-merge regressions and open issues (#1643)
- fix gsapDragCommit.test.ts import for commitGsapPositionFromDrag - add generateId() helper with insecure-context fallback (fixes #1637) - enable STUDIO_KEYFRAMES_ENABLED by default - prevent clock duration from shrinking during playback (fixes #1636) - add maxRetries to rmSync cleanup calls to handle ENOTEMPTY race (fixes #1644) |
||
|
Miguel Ángel
|
0473254bdd |
fix(engine): preserve AAC start time during MP4 mux (#1615)
* fix(engine): copy mixed AAC during MP4 mux * fix(producer): avoid AAC re-encode in distributed audio pad * fix(engine): probe AAC sidecars before mux copy decision * fix(producer): avoid temp concat file for audio padding |
||
|
Miguel Ángel
|
8408a44745 |
fix(engine): tune VP9 cpu-used across render paths (#1614)
* fix(engine): tune VP9 cpu-used across render paths * fix: address VP9 review feedback |
||
|
Vance IngallsandClaude Opus 4.8
|
7310223b66 |
feat(engine): static-frame dedup default-on + render telemetry (#1549)
* feat(engine): static-frame dedup for screenshot capture (opt-in) Skip re-seeking + re-screenshotting frames byte-identical to their predecessor. A frame is dedupable iff no GSAP tween or clip cut is active in it or its predecessor (predicted from window.__timelines + clip schedule) AND an empirical anchor-compare confirms it. Opt-in HF_STATIC_DEDUP=true, default off. Correctness (designed for the multi-worker / distributed render paths): - Reuse is keyed by the ABSOLUTE composition frame (derived from the frame's time), NOT the captureFrameCore frameIndex arg — chunked/parallel callers pass a chunk- relative index. Validated lossless (PSNR=inf) on both single- and multi-worker renders of a static-hold comp. - verifyStaticFramesSafe checks EVERY run (no longest-first budget truncation that left runs armed-but-unverified), and samples each run's FIRST reused frame, its END, and interior points at a stride; a hard cap disables dedup rather than trust an unverified set. - Conservative arming: skipped when capture mode != screenshot (BeginFrame tick semantics + the verifier's screenshot path wouldn't transfer), when a before-capture hook is set (per-frame video injection), when page-side compositing is active (shader / drawElement composite the plain verification screenshot can't reproduce), and when any data-start is a non-numeric reference expression the clip-boundary parser can't protect, or duration is unknown/zero. - Session reuse (prepareCaptureSessionForReuse) resets lastFrameBuffer + dedup counter so a probe/prior-render buffer can't bleed into the first static frame; the armed set is kept (same-composition reuse). Cost calibration bypasses dedup for its sparse, non-contiguous sample sweep, then restores the armed set. - HF_STATIC_DEDUP_SAMPLES is NaN-guarded. Disqualifies on signals the GSAP predictor can't see: video, canvas/webgl, zero tweens, running CSS/WAAPI animation. Pays on static-hold content (title cards, slideshow/kiosk loops, data-viz pauses); no-op on continuously-animated comps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(engine): static-frame dedup default-on + render telemetry Flip dedup from opt-in (HF_STATIC_DEDUP=true) to default-on (opt-out HF_STATIC_DEDUP=false). Verification (verifyStaticFramesSafe) is the safety net that keeps reuse sound at scale. Add end-to-end dedup observability. The capture session records enabled / armed / skipReason / predicted; these surface via CapturePerfSummary -> a dedupPerfs accumulator (disk sequential + parallel AND streaming sequential + parallel) -> aggregated into RenderPerfSummary.staticDedup (OR armed, SUM frames across workers) -> render_complete props static_dedup_{enabled,armed,skip_reason, predicted_frames,reused_frames}. skip_reason is a low-cardinality code: capture_mode | video_injection | page_composite | ineligible | verification_failed. Distributed chunks run on Linux/beginframe where dedup never arms, so they pass a throwaway dedupPerfs sink (no per-chunk reporting). Tests: aggregation logic (OR/SUM/skip-reason) + opt-out passthrough. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(engine): address review on dedup default-on + telemetry Review feedback (miga-heygen) + self-review fixes: - Retry double-count: executeDiskCaptureWithAdaptiveRetry pushed worker dedup perf inside the retry loop, so an adaptive retry counted frames twice (reused/predicted could exceed totalFrames). Reset dedupPerfs at the start of each attempt — retry now REPLACES rather than accumulates; common no-retry path is unchanged. - Opt-out parsing: HF_STATIC_DEDUP now disables on {false,0,off} case/space-insensitive (was strict !== "false", so `False`/`0` silently kept dedup on — the kill-switch could no-op). - Verification budget vs drift: verifyStaticFramesSafe returns {badFrame, budgetExhausted}; armStaticDedup reports a distinct `verification_budget` skip reason so a telemetry spike means "raise HF_STATIC_DEDUP_SAMPLES", not "compositions are non-static". - Index idiom: captureFrameCore now uses Math.floor(time*fps + 1e-9) (matches quantizeTimeToFrame) so the dedup lookup agrees with the frame the seek lands on even for non-exact times. - Stale "opt-in HF_STATIC_DEDUP=true" comments -> "opt-out HF_STATIC_DEDUP=false" across frameCapture.ts + types.ts. - Extract pushWorkerDedupPerfs helper (perfSummary.ts), used by the disk and streaming parallel paths — removes the duplicated push loop and drops captureStreamingStage back under the complexity threshold. - dedupPerfs is now required (not optional) on executeDiskCaptureWithAdaptiveRetry — a missing arg silently dropped telemetry. - Test: captureStreamingStage createInput() now provides the required dedupPerfs field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(engine): address deferred dedup-review items - Derivable state: drop session.staticDedupArmed/staticDedupPredicted; derive both from session.staticFrames in getCapturePerfSummary (armed ⟺ non-empty set, predicted === size) so they can't desync. - Config altitude: HF_STATIC_DEDUP now resolves into EngineConfig.staticFrameDedup (resolveConfig, opt-out on {false,0,off}), alongside forceScreenshot/browserGpuMode — armStaticDedup reads config instead of process.env. Default-on preserved (missing config → enabled). - Lossy aggregation: aggregateDedup now reports DISTINCT skip reasons (sorted, `|`-joined) across diverging unarmed workers instead of just the first. - discardWarmupCapture: also snapshot/restore staticDedupCount and lastFrameBuffer so a warmup capture can't leak a phantom reuse or a stale buffer anchor into the real summary. - Convention: perfSummary-dedup.test builds its job via createRenderJob instead of `as unknown as RenderJob`. - Docs: verification_budget added to skip-reason lists. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
36b24acf20 |
feat: add video frame format render option (#1481)
* feat: add video frame format render option * refactor: single source of truth for video-frame-format allow-list Addresses PR review (Via) on #1481: the ["auto","jpg","png"] set was declared three times — render.ts (VIDEO_FRAME_FORMATS), server.ts (inline includes), and renderConfigValidation.ts (ALLOWED_VIDEO_FRAME_FORMATS) — three boundaries to update when a new extraction format lands. Hoist the constant + a reusable `isVideoFrameFormat` type guard into @hyperframes/engine (where VideoFrameFormat is defined) and route all three call sites through them. Behavior unchanged; also drops two `as RenderConfig[...]` casts in favor of the guard (narrowing over assertion, per repo TS conventions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Xuelong Mu <xuelongmu@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Carlos Alcaraz GregorandCarlos Alcaraz
|
1f9c70cf15 |
fix(producer): guard fileServer.close in all render cleanup paths (#1406)
* fix(producer): guard fileServer.close in distributed cleanup paths `plan()` and `renderChunk()` both close the probe/chunk file server with a bare `fileServer.close()` in their cleanup sequence. `FileServerHandle.close` tears down the underlying http.Server, whose `close()` throws `ERR_SERVER_NOT_RUNNING` if the server was already torn down (for example a cancellation path that closed it once already). An unguarded throw there escapes the cleanup and masks the original plan/render result, exactly the failure the adjacent probe-session close already guards against with a try/catch (its comment even spells this out). Add `closeFileServerSafely`, which wraps the close in a try/catch and logs, and route both cleanup sites through it so the two stay consistent and a throwing close can never mask the real result. Covered by unit tests for both the throwing and happy paths. * fix(producer): extend fileServer.close guard to renderOrchestrator success path --------- Co-authored-by: Carlos Alcaraz <193642530+calcarazgre646@users.noreply.github.com> |
||
|
Miguel Ángel
|
a0d7295367 |
refactor(producer): simplify — extract HDR compositor, delete dead code, consolidate patterns (#1414)
* refactor(producer): extract HDR compositor from renderOrchestrator Move ~700 LOC of HDR compositing primitives (countNonZeroAlpha, countNonZeroRgb48, cropRgb48le, HdrVideoFrameSource, closeHdrVideoFrameSource, blitHdrVideoLayer, HdrImageBuffer, blitHdrImageLayer, CompositeTransfer, shouldUseLayeredComposite, resolveCompositeTransfer, HdrCompositeContext, compositeHdrFrame, HdrTransitionMeta, TransitionRange) into a dedicated hdrCompositor.ts module. Remove backward-compat re-exports from renderOrchestrator (hdrPerf, captureCost, shared) and rewire all import sites to the authoritative source modules. * refactor(producer): delete 4 re-export shim files screenshotService.ts, videoFrameExtractor.ts, videoFrameInjector.ts, and streamingEncoder.ts existed solely to re-export symbols from @hyperframes/engine. No internal consumer imported from them except index.ts → videoFrameInjector, which now imports directly from engine. * refactor(producer): delete unused PNG decode/blit worker pool The pool (455 LOC) and worker (127 LOC) were built speculatively for pipelining Chrome screenshots with PNG decode/blit but were never wired into any capture path. Zero non-test source files imported them. Also removed the esbuild entry point from producer/build.mjs, the tsup entry point + alpha-blit alias from cli/tsup.config.ts, and the PNG worker bootstrap from cli/src/cli.ts. * refactor(producer): centralize frame filename construction Replace 4 inline padStart(6) template literals with shared helpers: - formatCaptureFrameName(index, ext): zero-based, for internal capture - formatExportFrameName(index, ext): zero-based input, one-based output for user-facing png-sequence export * perf(producer): hoist allElementIds out of compositing loop Move fullStacking.map() from inside the per-layer iteration to before the loop, computing the element ID list once per frame instead of once per DOM layer per frame. * refactor(producer): consolidate HDR timing instrumentation * refactor(producer): remove typecasts and deduplicate HDR capture patterns - Extract seekInjectAndQueryStacking() and seekAndInject() helpers to deduplicate the seek+inject+query pattern across sequential loop, hybrid loop, and per-scene transition capture (3 call sites → 1 helper) - Fix sceneBuf as Buffer casts by properly typing the scene-capture arrays as [Buffer, Set<string>][] instead of using as const + cast - Replace as NonNullable<> cast on outputFormat with as const fallback - Add explanatory comments on inherent linkedom DOM casts * refactor(producer): name constants, type matrix, extract opacity helper - Replace magic 0.001/0.999 with TRANSFORM_IDENTITY_EPSILON and OPAQUE_ALPHA_THRESHOLD; replace BPP=6 with RGB48_BYTES_PER_PIXEL - Add AffineMatrix tuple type + isAffineMatrix guard, eliminating all 4 non-null assertions on matrix indices - Extract resolveBlitOpacity() to replace 5 identical ternaries - Narrow fallow-ignore-file to line-level complexity suppressions |
||
|
Matt Van HornandMatt Van Horn
|
e6b8d66c2d |
feat(cli,producer): add gif output format with two-pass palette encode (#1333)
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
James Russo
|
9b18fadccd |
feat(producer): optional targetChunkFrames to bound per-chunk frames (#1332)
* feat(producer): optional targetChunkFrames to bound per-chunk frames * feat(cli): expose --target-chunk-frames on lambda + cloudrun render; document it |
||
|
James Russo
|
69eac249d5 | feat(producer): stage wall-clock split in chunk perf telemetry (#1327) | ||
|
Miguel Ángel
|
0bf15119f8 |
feat: font resolution pipeline — compositions capture and embed their own fonts (#1255)
Compositions are now self-contained: the compiler captures font files and embeds them as woff2 data URIs, eliminating silent render-time fallback when the render environment lacks the author's fonts. Resolution order (each tier falls through to the next): 1. Existing @font-face → use as-is 2. Bundled alias (38 cross-platform mappings) → embed data URI 3. Google Fonts → fetch, cache, embed 4. Local system font → locate on OS, compress to woff2, embed 5. Local @font-face paths → read file, compress, inline as data URI 6. External CDN stylesheets → fetch CSS, extract @font-face, inline 7. Alias map fallback → closest bundled equivalent 8. Actionable error with guidance Key changes: - System font locator (macOS/Windows/Linux) with path-bounding and symlink defense (realpathSync + O_NOFOLLOW) - woff2 compression via wawoff2 (WASM, cross-platform) - Multi-weight/style variant capture with length-sorted token matching - External stylesheet inlining with SSRF defense (assertPublicHttpsUrl, HTTPS-only, private-host blocking, 2MB cap, 4-concurrent limit) - Studio auto-import via GET /fonts/file API + renderAliasFor() derived from shared FONT_ALIAS_MAP (no more hand-curated drift) - failClosedFontFetch throws on unresolved fonts in distributed renders - Single source of truth: @hyperframes/core/fonts/aliases - system_font_will_alias lint rule (escalates to warning for distributed) - Default to Inter + JetBrains Mono in templates and CSS reset |
||
|
James RussoandClaude Opus 4.8
|
4da567df22 |
feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Miguel Ángel
|
ca3bdd149b |
fix(producer): thread forceScreenshot to probe stage for low-memory mode (#1237)
* fix(producer): thread forceScreenshot to probe stage for low-memory mode The render orchestrator sets captureForceScreenshot=true when low-memory mode is active and passes it to all three capture stages, but never passed it to the probe stage. The probe stage received the original cfg with forceScreenshot=false, so on Linux it launched the browser in beginframe mode — which hangs on memory-constrained hardware. Add a forceScreenshot parameter to ProbeStageInput (matching the pattern used by captureStage, captureStreamingStage, and captureHdrStage) and wire it through from both the render orchestrator and the distributed planner. Fixes heygen-com/hyperframes#1236 * fix(producer): add lowMemoryMode→forceScreenshot bump in distributed plan + regression test - plan.ts: mirror the renderOrchestrator's lowMemoryMode bump so that distributed runs on low-RAM hosts also force screenshot capture mode. Previously the bump was only applied in the in-process path (renderOrchestrator.ts:1598); plan.ts derived forceScreenshot from compileResult without the bump, leaving the distributed probe stage in beginframe mode on low-memory hosts (same shape of bug as #1236). Added TODO to unify the bump into compileStage so all paths share one source of truth. - probeStage.test.ts: add regression test pinning that createCaptureSession receives forceScreenshot:true when the stage input carries it but cfg.forceScreenshot is false (low-memory mode override). Mirrors worker-count test shape from captureStreamingStage.test.ts. Two cases: override active, override inactive. Addresses review feedback from #1237. |
||
|
Carlos Alcaraz GregorandCarlos Alcaraz
|
81521a9a97 |
fix(producer): drop empty trailing chunk slice in distributed render plan (#1133)
resolveChunkPlan caps chunkCount at maxParallelChunks from the naive count, then rounds effectiveChunkSize up to ceil(totalFrames / chunkCount). When that ceil rounds up, the first (chunkCount - 1) chunks can already cover every frame, so buildChunkSlices emits a final slice with startFrame >= totalFrames — an empty [n, n) or inverted range. renderChunk rejects it (framesInChunk <= 0) and, under Step Functions retries, fails the whole distributed render even though [0, totalFrames) is fully covered. This is reachable from the user-facing CLI: `hyperframes lambda render --chunk-size 10 --max-parallel-chunks 12` on a ~4s/30fps (121-frame) composition yields chunkCount=12, effectiveChunkSize=11, and a 12th slice of [121, 121). Tighten chunkCount to ceil(totalFrames / effectiveChunkSize) after the size is finalized, so the union stays exactly [0, totalFrames) with no empty tail. This only lowers chunkCount in the explicit-small-chunkSize case; the auto-sized and large-chunkSize paths already satisfy ceil(totalFrames / effectiveChunkSize) >= chunkCount, so it's a no-op there (existing tests' chunkCount values are unchanged). Adds a regression test for the 121/10/12 case plus a grid property test asserting contiguous, non-empty, exact coverage across explicit sizes. Co-authored-by: Carlos Alcaraz <193642530+calcarazgre646@users.noreply.github.com> |
||
|
James
|
90bf485db8 |
fix(distributed): reject cfr:true with h265 codec (per review)
The cfr re-encode pass hardcodes `-c:v libx264`. Pairing it with `codec: "h265"` would silently transcode the h265 chunks to h264. Detect the encoder discriminant in `meta/encoder.json` and throw a typed error parallel to the existing non-mp4 format guard, so callers surface the conflict instead of producing a wrong-codec deliverable. — Rames Jusso |
||
|
James
|
71d1889da6 |
feat(distributed): add optional cfr flag for exact constant frame rate
Distributed-render output today uses -c:v copy through concat → mux → faststart, which means PTS timestamps from each chunk pass through unchanged. Container r_frame_rate is exact (#1040 + this PR's parent), but stream-level avg_frame_rate stays PTS-derived and can land on fractional rationals like 27648000/921677 over a 60s render. Same for sub-ms duration drift. This is the achievable bar within -c copy stream-copy concat. For most consumers (browser playback, YouTube, etc.) the difference is invisible. For downstream tools that strict-check avg_frame_rate or ms-precision duration (broadcast workflows, frame-accurate compositors, some third-party transcoders), it matters. Adds an opt-in cfr config flag (default false). When true, the assemble step's final pass re-encodes with -fps_mode cfr -r <fps> instead of -c copy, producing exact CFR output. Trade-off: ~2-5x the stitch time for a 60s 1080p clip; second-generation H.264 quality loss is negligible at -crf 18 but is non-zero. |
||
|
James
|
49281a5c17 |
fix(distributed): apply -r <fps> to single-chunk pass-through path
The v0.6.39 fix added -r <fps> to the multi-chunk concat ffmpeg invocation but didn't reach the single-chunk pass-through path, which is taken when totalFrames * fpsDen / fpsNum fits in one chunk. Result: 1-chunk renders shipped with fractional r_frame_rate (e.g. 359/12) while multi-chunk renders shipped with exact 30/1. Single-chunk path now goes through the same -r <fps> + -c copy ffmpeg invocation as the concat path, ensuring uniform exact r_frame_rate metadata across all chunk-count configurations. Adds a regression test exercising the 1-chunk path and asserting r_frame_rate === "<fpsNum>/<fpsDen>" exact. |
||
|
James
|
a4c4b2ff03 |
fix(distributed): enforce exact framerate at concat + mux boundaries
When the distributed render path stitches chunks with `-c copy`, ffmpeg averages the container framerate from PTS rather than carrying the source's exact rational rate, producing values like `360000/12001` instead of `30/1` and ~5ms duration drift over 60s. This is a known ffmpeg behavior at the concat-demuxer-copy boundary. The industry-standard fix is `-r <fps>` as an input flag on the concat step plus an output flag on the subsequent mux step — both with `-c copy` retained, no re-encode required. Three sites updated: - `assemble.ts` concat step: `-r <fps>` input flag. - `chunkEncoder.muxVideoWithAudio`: `-r <fps>` output flag. - `chunkEncoder.applyFaststart`: same, threaded from caller. Adds `r_frame_rate` + duration-equivalence assertions to `assemble.test.ts` to close the regression hole. |
||
|
James Russo
|
852008bd44 |
feat(producer): thread variables through plan() + renderChunk() (#962)
Add `variables?: Record<string, unknown>` to DistributedRenderConfig (§4.4) and LockedRenderConfig (§4.3). plan() snapshots the value into meta/encoder.json so every chunk worker re-injects the same set via captureOptions.variables, mirroring the in-process renderer's path. The variables fold into planHash automatically because canonical encoder.json bytes feed the hash: two plans with different variables produce different hashes (chunked output depends on the injected values); two plans with the same variables produce identical hashes because canonical-JSON sorts keys. The regression harnesses (distributed-simulated, lambda-local) also forward the input's variables to plan() / Step Functions event so fixtures that declare `renderConfig.variables` produce the same pixels across modes. Previously the field was on the harness input shape but silently dropped at the call boundary. Phase 9 PR 9.1 of the distributed rendering plan. |
||
|
James RussoandClaude Opus 4.7
|
5d264e146c |
docs(lambda): document webm support + simplify-review fixes (#953)
* docs(lambda): document webm support in distributed mode PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the shipped capability. Updates docs/deploy/migrating-to-hyperframes-lambda.mdx: - "Output format" row in the migration table now lists `webm` alongside mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 + closed-GOP concat-copy. HDR mp4 remains the only refused format. - "No webm distributed" caveat replaced with "webm uses closed-GOP VP9" explainer covering the encoder args (`-g <chunkSize>`, `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why alt-ref disable is load-bearing, and that the output preserves alpha via yuva420p with Opus audio. - Migration checklist no longer asks adopters to filter out webm compositions; only HDR-dependent renders need to stay on the previous framework. aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR in the v1 surface list), so it gets no copy edits beyond the migration guide. The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8, §12 — kept outside the repo) gets matching updates: format support matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the new top item, and the rev-12 → rev-13 status line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: address simplify-review findings on webm stack Folds in cleanups identified by a multi-agent code-review pass over the 4-PR webm-distributed stack: - plan.ts: `resolveEncoderTriple()` webm case now calls `getEncoderPreset(quality, "webm")` for its preset string instead of hardcoding "good". The hardcode was wrong for `quality: "draft"` (`getEncoderPreset` returns "realtime" for that tier) — would have silently overridden the draft → realtime mapping for distributed webm renders. - chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18 lines of WHY narration down to the 6 lines that actually explain why (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push comment to the same standard. - chunkEncoder.test.ts: drop the duplicate WHY comment that restated the implementation comment in plain words. - webm-concat-copy.test.ts: rewrite the file-header docstring to describe the contract being tested instead of the PR-8.1-gating history; strip "PR 8.2 / Path A / Path B" references from error messages (they belong in PR bodies, not in test output). Consolidate the yuva420p alpha smoke into a single `it()` block (was a full 4-test describe with duplicated setup) — the yuv420p block already covers the probe/decode/frame-count contract; the alpha smoke only needs to prove the alpha args don't break concat-copy. - plan.test.ts: drop the "PR 8.1 proved the contract" comment. - webm-vp9 fixture: drop the aspirational "Other webm-with-audio fixtures cover the mux path separately when added" sentence (no other fixtures exist). Regenerated the baseline via `docker:test:update webm-vp9` to reflect the updated comment. - migrating-to-hyperframes-lambda.mdx: add a paragraph about distributed webm's perf cost — ~10-25% larger files at constant CRF due to forced keyframes, and slower per-chunk encode due to `-cpu-used 2` being more conservative than the libvpx default. All unit tests + the webm-vp9 distributed-simulated regression still pass after these changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): accept --format=webm in `hyperframes lambda render` The CLI's `lambda render` subcommand's FORMATS allowlist and the `RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`, so even though the producer + aws-lambda packages now support webm end-to-end, the CLI surface rejected it with `--format must be mp4|mov| png-sequence`. Add webm to both spots and update the --help description. Surfaced during real-AWS deploy prep — the local lambda-local / distributed-simulated tests didn't go through the CLI so the gap went unnoticed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME) The deterministic Google Fonts cache was rooted at `\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's read-only. `mkdirSync(..., { recursive: true })` can't create that path and the plan stage trips with `ENOENT: no such file or directory, mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on every Lambda render that pulls a Google Font (i.e. every distributed fixture using `@import url("https://fonts.googleapis.com/...")`). Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to `tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives across invocations on a warm container, so cache hit rate is the same as non-Lambda runs. Also honor an explicit `\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a different location regardless of the runtime. Surfaced while verifying webm distributed end-to-end on real AWS — the same bug affects mp4 fixtures using Google Fonts; webm just happened to be the one I tried first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract DistributedFormat type + trim font-cache resolver Second simplify-review pass on the webm stack flagged two cleanups: 1. **`DistributedFormat` type duplicated 10 times.** Every file in the distributed pipeline carried its own copy of `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format meant a 10-place edit with no compile-time guarantee they stayed in sync. Extract a single source of truth in `packages/producer/src/services/distributed/shared.ts`, re-export from `@hyperframes/producer/distributed` and `@hyperframes/aws-lambda/sdk`, and have all callers pull from there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]` so the compiler enforces the runtime allowlist stays in sync with the type. 2. **`deterministicFonts.ts` font-cache resolver was over-commented.** Trim the 7-line block to 4 lines (drop the aspirational "and other read-only-FS execution environments" — only Lambda is detected — and the warm-container `/tmp` persistence narration — anyone reading already knows Lambda /tmp semantics). Collapse the two-step `if (explicit && explicit.length > 0)` into a single nullish-coalesce expression now that the empty-string defensive check is gone (`process.env.X` is `string | undefined`, no third shape to guard against). Out-of-scope skips (called out by the agents, deferred): - In-process `RenderConfig.format` and the in-process CLI's `render.ts` format union still carry their own inline copies. The union happens to coincide today but they're separate concerns — leaving them alone limits this PR's blast radius. - `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry flagged as taste; skipping. - Pre-existing redundant `existsSync` before `mkdirSync({ recursive: true })` in `fontCacheDir` — out of scope. All tests + typecheck still pass. Lambda render still works end-to-end (no functional changes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(lambda): drop plan-doc reference from migration checklist PR review feedback: source/docs should not mention the distributed-rendering planning doc. Tighten the migration checklist sentence to describe the webm path directly rather than referencing the doc's version label. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 — the function interleaved (a) mp4 codec validation + dispatch, (b) the non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the top-level function's cyclomatic complexity below the threshold while preserving every error message and code path. Behavior unchanged. Also extracts an `EncoderTriple` type alias so the three functions share the return shape declaratively rather than repeating it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James RussoandClaude Opus 4.7
|
21f5066832 |
feat(producer): enable webm in distributed mode via concat-copy (#951)
* feat(producer): enable webm in distributed mode via concat-copy PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the distributed pipeline now that PR 8.1 proved concat-copy works. Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams). The simpler architecture wins; no re-encode in assemble, no encode- parallelism loss. Changes: - plan.ts: - DistributedRenderConfig.format and PlanResult.format now include "webm" — type-level acceptance matches the runtime gate. - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR mp4 remains the only refused configuration. - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p + preset="good" for format="webm". yuva420p preserves alpha — the format's main reason for existing for web delivery. - codec= remains rejected for non-mp4 formats (mov is always ProRes 4444; webm is always libvpx-vp9). The error message lists all four distributed-supported formats. - FormatNotSupportedInDistributedError docstring updated to reflect the new reality (only HDR is unsupported). - freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software". Mirrors libx265-software / prores-software / png-sequence in shape; the chunk worker reads this discriminant to decide encode args. - renderChunk.ts: drops the now-incorrect cast that excluded webm from buildSyntheticRenderJob's format input; tightens the preset-format cast to include webm. - assemble.ts: docstring + comment updates. The mp4/mov concat-copy path is format-agnostic — webm uses the exact same code (applyFaststart is a no-op for webm via the existing chunkEncoder.ts gate; muxVideoWithAudio already routes webm to libopus audio). - planFormatBanlist.test.ts: webm-rejection tests removed; replaced with "accepts webm" tests + a HDR+webm combo test that verifies HDR is the trip regardless of format. - plan.test.ts: new describe block pins the webm wiring contract: format="webm" produces an encoder=libvpx-vp9-software / pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize. - webm-concat-copy.test.ts (smoke): extended with a yuva420p variant that proves the alpha pixel format the distributed pipeline actually emits also round-trips through concat-copy. 9/9 tests pass locally. §8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end fixture (PR 8.3) is green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke PR review feedback from Miguel and Vai on #951 caught a real bug: `plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan stage froze `forceScreenshot: false` into the `LockedRenderConfig` even though distributed webm uses `yuva420p`. Every chunk worker captured opaque RGB via BeginFrame (which doesn't preserve alpha on Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha that the encoder then dropped — producing un-keyable webm. Two changes: 1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the in-process renderer's logic at `renderOrchestrator.ts:1469` (`const needsAlpha = isWebm || isMov || isPngSequence`); the two sites must stay in sync since the distributed pipeline's PSNR regression compares against the in-process baseline. 2. **Smoke test (yuva420p describe)**: source frames now use a real alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of `testsrc2 + format=rgba` which was uniformly opaque. The decode- pix_fmt assertion is dropped (ffprobe reports `yuv420p` for VP9-with-alpha because the alpha lives in a Matroska `BlockAdditional` sidecar) and replaced with two stronger checks: - `TAG:ALPHA_MODE=1` is present on the stream — proves the encoder was actually configured for alpha - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba -vf extractplanes=a,signalstats` — proves the alpha sub-stream round-trips through concat-copy with spatially-varying content, not uniform/dropped alpha - decode-test gate is now exit-code-only (was `exitCode || stderr` which would flake on chatty ffmpeg `-v error` builds emitting non-fatal DTS/container notes) These checks would have caught the `needsAlpha` bug before review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(aws-lambda): widen narrow format types to include webm CI on PR #951 was failing at typecheck/build because the producer's `DistributedRenderConfig.format` widened to include webm in this PR but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"` type literals in `events.ts`, `handler.ts`, and `validateConfig.ts` hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now including webm) into a parameter typed against the narrow union, producing TS2345. This widening originally landed in PR #952 (test fixture PR) but needs to be atomic with the producer's widening here to keep each PR independently typecheck-clean. Also refactor `formatExtension` from a switch dispatch to a `Record<DistributedFormat, string>` lookup. Adding the webm case tipped the switch's CRAP to the 30.0 fallow threshold; the lookup table drops cyclomatic from 5 to 1 with the same compile-time exhaustiveness guarantee (TS errors on missing entries when `DistributedFormat` adds a new format). The runtime `_exhaustive: never` throw was only protecting against a string slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already gates untrusted input at the SDK boundary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James Russo
|
17f47f30dd |
fix(distributed): gate per-worker SwiftShader probe to worker 0 only (#956)
After #916 moved `assertSwiftShader` from `renderChunk()`'s eager probe session into `executeWorkerTask`, every parallel worker began running its own `chrome://gpu` / canvas-WebGL probe. At `chunkWorkerCount=6` (texture launch at chunks=3) that's 6 concurrent CDP page-loads per chunk × 3 chunks = 18 simultaneous probes. Bench data on dev (12 producer pods × 22 vCPU) showed c=3 worst-case wall-clock at 67.3s, 24.7s above c=6 worst (42.6s) — pod_total inflates 100s → 147s uniformly across all three chunks per slow iter, the signature of cluster-level CDP contention rather than within-pod contention. Workers within a chunk share the same Chrome binary, flags, and OS/driver state on a single pod, so worker 0's success is representative for the rest. Gate the probe via `shouldVerifyWorkerGpu(workerId, config)` so only worker 0 navigates to the probe page; workers 1..N-1 skip it. The fail-fast contract still holds at the chunk level (worker 0 still aborts the chunk if SwiftShader didn't load) — just without the concurrent CDP traffic. Expected wall-clock impact: c=3 worst drops from ~67s to in line with c=6 worst (~42-44s). c=6 (3 workers/pod) and c=8 (2 workers/pod) should see smaller wins; c=12 (1 worker/pod, sequential branch) is unaffected. Closes #955. |
||
|
James
|
7e0a447325 |
refactor: drop unused exports detected by fallow auto-fix
Run `fallow fix --auto-fixable` to remove `export` keywords from symbols fallow's reachability analysis identifies as unused. Keeps only the cases where the symbol is still referenced internally in its own file (so removing `export` doesn't surface a new oxlint `no-unused-vars` error). Result: fallow dead-code findings drop from 276 → 208 (68 fewer unused exports), with no behavior change — each symbol is still defined and used exactly the same way within its file. Reverted ~20 files where fallow's auto-fix would have created cascading "declared but never used" lint errors — those are cases where the symbol isn't used at all, and properly cleaning them up means deleting the declaration, not just dropping `export`. Better to land that as a separate, narrower PR rather than mixing it into a mechanical de-export. Also reverted four false positives where fallow missed real consumers: - `captureCost.ts` (renderOrchestrator has two separate import blocks from the same module; fallow only saw the first) - `propertyPanelHelpers.ts`, `domEditingLayers.ts` (real internal uses fallow's reachability missed) - `render.ts` (functions imported via `await import()` dynamic import, which fallow's static analysis doesn't follow) Test plan: bun run --filter '*' typecheck (clean), oxlint + oxfmt clean, cli/core/studio/engine vitest suites pass (335 + 917 + 576 + 605 tests). |
||
|
JamesandClaude Opus 4.7
|
a8499632c6 |
test(producer): preserve single-chunk-path coverage + add auto-size integration
Address PR review feedback on #939: - Pin chunkSize=240 on the golden planDir layout test so the 1-chunk path through plan() stays exercised after the auto-sizer change. Assert chunkCount === 1 explicitly (previously just >= 1). - Add an integration test that runs plan() with chunkSize=undefined and asserts the auto-sizer produces multi-chunk output end-to-end (chunkCount=3, encoder.gopSize=10, encoder.chunkSize=10) for the same 30-frame fixture. - Document the GOP/file-size trade-off on the chunkSize docstring so adopters who optimize for output bytes know to pin chunkSize. - Update the resolveChunkPlan docstring formula to reference the operative variable (resolvedChunkSize) instead of the now-ambiguous chunkSize. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
fd3fce9955 |
refactor(producer): tighten resolveChunkPlan assertion + trim comments
Address self-review findings: - assertPositiveInteger now only runs on the caller-supplied path so the error message names `configChunkSize` only when the caller actually passed one. Previously, the assertion fired against `resolvedChunkSize` on both paths and would have lied about the offending input. - Drop the call-site comment that narrated the diff/history; the function docstring already covers the contract. - Drop the internal-track name and date from the MIN_CHUNK_SIZE rationale and the test block header. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
e79a8faa42 |
feat(producer): auto-size chunkSize from maxParallelChunks when undefined
Previously, plan() defaulted chunkSize to 240 on a `?? DEFAULT_CHUNK_SIZE` line, so a 660-frame composition with maxParallelChunks=16 ended up at 3 chunks (ceil(660/240)) regardless of the caller's fan-out intent. When config.chunkSize is undefined, auto-size from maxParallelChunks: effectiveChunkSize = max(MIN_CHUNK_SIZE, ceil(totalFrames / maxParallelChunks)) MIN_CHUNK_SIZE=10 keeps per-chunk fixed overhead from swamping the parallelism gain on tiny renders. Explicit numbers, including 240, take precedence over the auto-sizer — no behavior change for callers that set chunkSize explicitly. Surfaced by the lever-1 chunk-scaling benchmark on 2026-05-17. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
James Russo
|
f01fccb0ea | perf(distributed): skip eager probe session when chunkWorkerCount > 1 (#916) | ||
|
James Russo
|
34d1f0e1d0 |
feat(lambda): add TypeScript SDK and CDK construct (#909)
* feat(lambda): add TypeScript SDK and CDK construct
Adds the client-side surface on top of the Phase 6a Lambda handler so
adopters can drive a deployed stack from Node without writing AWS-SDK
boilerplate:
- renderToLambda(opts) starts a Step Functions execution and returns a
handle. Does NOT poll.
- getRenderProgress({ executionArn }) returns a snapshot of progress,
frames rendered, cost (Lambda GB-seconds + SFN transitions), errors,
and the final output object once Assemble completes.
- deploySite({ projectDir, bucketName }) content-addresses the project
tree, tar.gzs it, and uploads to S3 with a HeadObject short-circuit so
re-renders of the same tree skip the tar+PUT.
- validateDistributedRenderConfig throws a typed InvalidConfigError
before StartExecution, so shape errors surface synchronously.
- computeRenderCost is exposed for callers who want to format cost out
of band.
Also ships HyperframesRenderStack, an aws-cdk-lib L2 construct that
emits the same topology as examples/aws-lambda/template.yaml. Lives on
the ./cdk subpath export so SDK-only consumers don't pull aws-cdk-lib
into their runtime graph (declared as an optional peer dependency).
Tests: 24 new unit tests across the SDK plus 9 CDK synth / contract /
snapshot tests. All 83 tests in packages/aws-lambda/src pass.
* refactor(lambda): /simplify pass on the SDK + CDK PR
Pulls shared logic out so the SDK doesn't re-invent things the handler
and the producer already have:
- `formatExtension` extracted to packages/aws-lambda/src/formatExtension.ts.
handler.ts and renderToLambda.ts both used identical 12-line copies of
this switch.
- `PLAN_PROJECT_DIR_SKIP_SEGMENTS` is now exported from
@hyperframes/producer/distributed. deploySite consumes it instead of
its own duplicate SKIP_TOP_LEVEL set; the two lists were trivially
identical and would have drifted silently.
- `FakeS3` + `drainBody` factored out of the two SDK test files into
src/sdk/__fixtures__/fakeS3.ts. Drops ~110 lines of test-file
duplication and gives future SDK tests a one-line FakeS3 import.
- S3 URI building in deploySite and renderToLambda routes through the
existing `formatS3Uri` helper instead of inline `s3://...`
concatenation; matches the convention already in handler.ts.
Net -133 lines across the touched files. All 83 aws-lambda tests still
pass; all 60 producer distributed tests still pass.
* fix(lambda): bump CDK test timeouts for CI cold-start synth
The bun:test default 5s timeout tripped the first CDK snapshot test
in CI when the cold-start `Template.fromStack(stack)` synth took ~5-8s
on the slowest GitHub Actions runner. Locally on a warm shell the
synth measures <1s, so the failure didn't reproduce until PR #909 hit
CI.
Two changes:
- Both CDK test files cache one synth in `beforeAll(..., 30000)` and
reuse the result across every test that uses the default props.
Each individual test now runs in microseconds (pure assertions
against the already-synthed template), so the 5s timeout no longer
applies on the hot path.
- The two contract tests that exercise non-default props
(reservedConcurrency, projectName) still synth fresh per-test; they
get a per-test `it(..., 30000)` timeout.
No behavior changes.
* fix(lambda): address PR review on SDK + CDK construct
Three correctness + ergonomics fixes raised in Vai's review:
- getRenderProgress over-counted SFN transitions by 3-5×. Step
Functions Standard Workflows bill per state-entry, not per
history event. Each Task produces ~5-7 history events
(Scheduled / Started / Succeeded / TaskStateExited / …);
counting `events.length` reported the runaway. Switch to
counting `*StateEntered` events explicitly.
- assembleComplete + outputFile detection was coupled to the
Lambda payload's `Action` field. Move both signals onto the
enclosing state name (`StateExited.name === "Assemble"`), which
is the state-machine identity rather than the Lambda event
contract. framesRendered increment moves to the same boundary
(RenderChunk state).
- SiteHandle now carries `bucketName` directly so README + CLI
callers don't have to re-parse `projectS3Uri.split("/")[2]`.
Test updates: getRenderProgress tests wrap renderChunk/assemble
events in matching StateEntered + StateExited pairs so the new
state-name-driven dispatch is exercised end-to-end. SiteHandle
fixture in renderToLambda.test.ts gets the new bucketName field.
All 83 aws-lambda tests still pass.
|
||
|
James Russo
|
22363a11c9 |
perf(distributed): parallelize chunk capture across multiple workers (#906)
* perf(distributed): parallelize chunk capture across multiple workers
The distributed `renderChunk` primitive hardcoded `workerCount: 1` and
`captureStage` explicitly forbade `workerCount > 1` when `frameRange` was
set, with the comment:
"Distributed chunk workers fan out at the activity layer; reduce
workerCount to 1 when passing frameRange."
The assumption was that orchestration-layer fan-out (Temporal / Lambda /
K8s Jobs / SSH) saturates the available CPU on its own. In practice
adopters that deploy chunks onto multi-core hosts (8-24 vCPU is the
standard producer-worker pod sizing) end up pinning only ~3-4 cores per
chunk while the rest sit idle: chunk-level fan-out at the orchestration
layer gives each pod one chunk at a time, and the chunk render itself
was single-threaded.
Validated against a real 1080p / 30fps / 22-second shader-heavy
composition on a 22-vCPU Temporal pod: each chunk rendered at
165-273ms per frame (vs 94-98ms for the in-process streaming render
which runs `workerCount=2` by default). The slowest chunk gates total
wall-clock under parallel chunk fan-out, so the 2-3x per-frame gap
compounds and `distributed` was net-slower than `in-process` on every
composition smaller than ~5min of texture-class content. Lifting the
restriction is a measured ~2x per-chunk speedup with no contract
change at the framesDir or encoder layer.
Wire-up:
* `WorkerTask.outputFrameOffset` — optional offset subtracted from the
absolute frame index when computing the captured file's name.
Default 0 (the in-process contract; file name == absolute index).
Distributed chunks set this to the chunk's startFrame so file names
land 0-indexed within the chunk's range, matching the sequential
chunk-capture contract and the encoder's expectation that frames
are read sequentially without an `-start_number` override.
* `distributeFrames(totalFrames, workerCount, workDir, rangeStart=0)` —
offsets both `startFrame`/`endFrame` (used for per-frame time math
on the page's virtual clock) by `rangeStart`, and threads
`outputFrameOffset = rangeStart` onto each task it emits. With the
default `rangeStart=0` it is a no-op for in-process renders.
* `executeWorkerTask` — uses `i - (task.outputFrameOffset ?? 0)` for
the captured file name, leaving the per-frame TIME computation
`(i * fps.den) / fps.num` untouched so the page's virtual clock is
unchanged.
* `executeDiskCaptureWithAdaptiveRetry({ frameRangeStart? })` — accepts
the chunk's absolute startFrame and forwards it to `distributeFrames`
and `buildMissingFrameRetryBatches`. Default `undefined` preserves
the in-process contract.
* `buildMissingFrameRetryBatches(ranges, ..., rangeStart=0)` —
`findMissingFrameRanges` walks LOCAL 0-indexed file names; the retry
batch translates the local missing-range pair back to ABSOLUTE
composition indices for `WorkerTask.startFrame/endFrame` and sets
`outputFrameOffset = rangeStart` so the retried capture writes back
to the same local file name.
* `captureStage` — drops the assert; passes
`frameRangeStart: frameRange?.startFrame` to the parallel branch so
workers land on absolute composition frame indices for time math
while file names stay 0-indexed within the chunk range. Docstring
updated to reflect that the parallel branch is now supported.
* `renderChunk` — `workerCount: 1` → `workerCount: 2`. The pre-warmed
`probeSession` is consumed only by the sequential branch; the
parallel branch closes it during stage entry and creates its own
worker sessions. Documented as a follow-up: skip probeSession
creation when `workerCount > 1` to recover the ~3-5s warmup cost.
Backwards compatibility: every change is gated on a parameter that
defaults to the prior behavior. In-process callers (`executeRenderJob`)
pass no `frameRangeStart`, so `rangeStart === 0`, `outputFrameOffset`
defaults to 0, and the file-name math collapses to the prior `i` value.
The framesDir contract (`frame_0..frame_(totalFrames-1)`) and the
WorkerTask interface are extended, not replaced.
Tests: 24 pass / 0 fail across the distributed test suite (renderChunk,
plan, assemble, planFormatBanlist, planSizeCap, publicExports). 7 pass /
0 fail in `parallelCoordinator.test.ts`. The renderOrchestrator suite
has one pre-existing Windows-only failure
(`writeCompiledArtifacts — external assets on Windows drive-letter
paths`) unrelated to this change; the other 56 tests pass.
Refs: distributed-vs-inprocess benchmark thread at
heygen-com/experiment-framework#36950
* perf(distributed): auto-size chunk workerCount via calculateOptimalWorkers
Match the in-process renderer's worker selection instead of hardcoding 2.
`calculateOptimalWorkers(framesInChunk, undefined, cfg)` is the same call
`resolveRenderWorkerCount` makes under the hood, minus the capture-cost
calibration reduction (which would require plumbing the chunk's compiled
metadata through — left as a follow-up).
For a typical 22-vCPU producer-worker pod with `cfg.concurrency: "auto"`
this resolves to ~6 workers for a 240-frame chunk (capped by
`defaultSafeMaxWorkers() = max(6, min(16, floor(cpuCount/8)))`), matching
what `executeRenderJob` (the in-process path) already does. The prior
hardcoded `workerCount: 2` was a safe-minimum starting point that
undersized chunks vs prod's auto behavior.
Tests: 12/12 pass in `renderChunk.test.ts` (unchanged — the test suite
mocks the inner runCaptureStage call so workerCount selection is opaque
to it).
* refactor(distributed): /simplify pass on PR #906
Review pass on the parallel-capture frame-range change. Four targeted
cleanups identified by code-quality and efficiency review agents:
1. Add the missing `frameRange.endFrame - frameRange.startFrame === totalFrames`
assert. The parallel branch forwards `totalFrames` separately from
`frameRangeStart`; a caller passing mismatched values would have got a
silently wrong distribution. The sequential branch already implicitly
relied on this via its `rangeFrames = rangeEnd - rangeStart` arithmetic.
2. Collapse three near-duplicate docstrings (on `WorkerTask.outputFrameOffset`,
`executeDiskCaptureWithAdaptiveRetry.frameRangeStart`, and `runCaptureStage`'s
`frameRange`) so only the WorkerTask field carries the full contract. The
other two cross-reference it.
3. Drop the WHAT-narrating comments inside `executeWorkerTask`'s per-frame
loop. The variable names (`fileFrameIdx = i - outputOffset`) already say
what the line does; the only remaining comment flags the non-obvious
contract that the streaming callback gets the absolute index.
4. Trim the 30-line `chunkWorkerCount` block in `renderChunk` to one paragraph
explaining the one non-obvious thing (why we use `calculateOptimalWorkers`
directly instead of `resolveRenderWorkerCount`). The probeSession-wasted-on-
parallel acknowledgement stays as a 3-line follow-up flag — investigated
skipping it in this pass, but the SwiftShader probe is safety-critical and
has no per-worker equivalent, so deferred to a separate change with proper
per-worker assertion plumbing.
Tests + format + lint clean:
* `bun test parallelCoordinator.test.ts` — 7/7
* `bun test distributed/{renderChunk,plan}.test.ts` — 24/24
* `bunx oxfmt` + `bunx oxlint` — clean
|
||
|
James Russo
|
c50f59a53b |
feat(lambda): add Lambda handler, ZIP bundling, and BeginFrame probe (#878)
* feat(lambda): add Lambda handler, ZIP bundling, and BeginFrame probe
Phase 6 of the distributed rendering plan: AWS Lambda turnkey adoption
(see DISTRIBUTED-RENDERING-PLAN.md §11 Phase 6 + §15).
This PR adds the new packages/aws-lambda/ workspace package that wraps
the OSS plan/renderChunk/assemble primitives in an AWS Lambda handler,
plus a build pipeline that bundles the handler + Chromium runtime +
ffmpeg into a deployable ZIP.
Architecture: ZIP deploy (not Docker image), Chrome via @sparticuz/chromium
with chrome-headless-shell fallback, dispatch on event.Action ∈ {plan,
renderChunk, assemble}.
The load-bearing concern — does @sparticuz/chromium's chrome-headless-shell
build honour CDP HeadlessExperimental.beginFrame? — is pinned by the new
scripts/probe-beginframe.ts regression guard. Probe boots the runtime
inside public.ecr.aws/lambda/nodejs:22, navigates to a static page, and
asserts beginFrame returns a PNG buffer. Verified locally + inside the
Docker container; both pass with hasDamage=true.
Sizes (sparticuz source): unzipped 157 MiB, zipped 99 MiB. Well under
the 240 MiB / 150 MiB in-house gates and the Lambda 250 MiB hard ceiling.
This is part of a stack of 8 PRs (3 in Phase 6a, 5 in Phase 6b); this is
PR 6.1.
* fix(lambda): address PR 878 review feedback
- Verify event.PlanHash against the untarred plan.json at the handler
boundary before invoking the producer primitive. Throws typed
PLAN_HASH_MISMATCH on divergence so Step Functions routes it as
non-retryable; previously the field was schema bloat the handler
ignored, leaving enforcement entirely inside the producer.
- Standardize on MiB throughout build-zip.ts, verify-zip-size.ts, and
the README. Lambda's hard ceiling is 250 MiB (AWS docs label "250 MB"
but use binary mebibytes); previously mixed units made the 248 MiB
budget look like a ~5 MB margin instead of the 2 MiB it actually is.
- stageChromeHeadlessShell now picks Chrome versions via numeric semver
comparison instead of lexicographic sort+reverse — the latter would
silently pick "99.x" over "131.x" once Chrome cached three-digit
majors that aren't width-aligned.
- Drop _setSparticuzChromiumForTests from the public index barrel.
Test-only DI seam imported directly from ./chromium.js in tests.
- Replace require("node:fs") inside walkSize() with the top-level fs
imports — file is ESM and the same module is already imported.
* docs(lambda): drop internal plan-doc refs from package README
* ci(windows): fix bun filter UNION bug excluding producer from Windows tests
`bun run --filter "!a" --filter "!b" test` composes as a UNION (any
package matching either negation runs), not an intersection. Effect:
@hyperframes/producer was still being tested on Windows even though
it's explicitly excluded — its regression harness (Docker + LFS golden
mp4 baselines) is Linux-only and was driving the 32min timeout.
Enumerate the packages we DO want to test instead.
|
||
|
James
|
47fe69aff0 | test(producer): extend cross-worker idempotency to non-zero chunks + dedupe soft-skip regex | ||
|
James Russo
|
808b10cb0e |
Merge pull request #844 from heygen-com/05-14-test_producer_add_cross-worker_idempotency_unit_test
test(producer): add cross-worker idempotency unit test |
||
|
James
|
6b3ad09436 |
fix(producer): tighten chunk-boundary test gates + narrow VIDEO_EXT indexing
Address @vanceingalls and @miguel-heygen review findings on #852: 1. Asymmetric soft-skip — only the N=1 plan+render+assemble call was wrapped in the host-Chrome-failure catch; an SwiftShader / cold-Chrome flake on the N=4 call would hard-fail instead of soft-skip. Factor a local runRender() helper and wrap both calls. 2. Vacuously-passing length assertion — 'expect(framesOne.length).toBe( framesFour.length)' passes when both runs produce 0 frames. Pin the absolute count (EXPECTED_FRAME_COUNT = 60) so a regression that identically truncates both renders shows red. 3. CDN version drift — anime-boundary loaded gsap@3.14.2 from jsdelivr while every other boundary fixture loaded 3.12.2 from cdnjs. Unify on cdnjs@3.12.2 so the next reader doesn't have to wonder why one fixture diverges. (gsap is an empty duration-driver in all six fixtures so the version was never load-bearing — but the divergence reads as intentional and isn't.) 4. VIDEO_EXT type narrowing — the lookup is Record<"mp4"|"mov"|"webm"> but outputFormat includes "png-sequence". The isPngSequence ternary short-circuits before png-sequence can reach the indexing site, but TS can't narrow through that. Add an explicit cast at the indexing site (not the lookup definition — over-widening to include "png-sequence": undefined would defeat the existence guarantee). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
James
|
bd21d00b13 |
refactor(producer): /simplify Phase 4 distributed-rendering changes
Address findings from a three-agent code-review pass over the Phase 4 stack:
- regression-harness: hoist `readdirSync` out of the per-checkpoint
failure-extraction loop (was running 20 redundant syscalls on every
failing png-sequence test). Drop redundant `existsSync` guards before
`mkdirSync(recursive: true)` and `rmSync(force: true)`. Replace the
three-deep ternary that built the output filename suffix with a
single `Record<format, ext>` lookup.
- regression-harness-distributed: flatten the `format === "mp4" ? {...} : {...}`
branching in the `plan()` call into a single config object with a
conditional spread. `plan()` already accepts `codec: undefined` for
non-mp4 formats, so the duplicate object was unnecessary.
- chunkBoundary.test: rename the stale "byte-identical mp4" test title
to "byte-identical frames" (the test now uses png-sequence). Trim the
10-line comment justifying `rejectOnSystemFonts: false` to the
essential WHY.
- renderChunk / plan.test / regression-harness: drop trailing-edge
comment phrases that pinned the prose to the PR's calendar context
("today", "v1.5", "pre-codec-knob output", section-numbered cross-
references to the planning doc).
No behavior change. All 49 distributed unit tests pass. Smoke + four
distributed format fixtures pass in --mode=distributed-simulated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
James
|
0b31465b2e | test(producer): add chunk-boundary fixtures per first-party adapter | ||
|
James
|
38e08e81c7 |
style(producer): apply oxfmt to resolvePresetForLockedEncoder signature
The generic parameter constraint exceeded oxfmt's line width, so the formatter wraps the type-param list onto its own line. Applies the same formatting locally that CI's 'Format' job would have produced via 'bun run format:check' — no behavior change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
James
|
db62e31d39 |
fix(producer): reject unknown codec strings + extract testable preset-override helper
Address @vanceingalls review on #850: 1. Unknown codec strings (typos like 'H265', future additions like 'av1') silently fell through to libx264 in resolveEncoderTriple. Add an explicit throw symmetric to the non-mp4-format branch already there. A JS caller building config from JSON who passes 'codec: "h266"' now gets a clear error at plan time instead of unflagged h264 output. 2. The preset.codec override in renderChunk had no fast unit coverage — only the heavyweight Docker fixture in #851 would catch a regression if someone refactored the spread (e.g. moved it into getEncoderPreset itself). Extract resolvePresetForLockedEncoder() and add 4 fast unit tests pinning the four encoder shapes (libx265/libx264/prores/png-seq). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
James
|
eccea88daf | feat(producer): add codec knob to DistributedRenderConfig | ||
|
James
|
4b9a17d520 | test(producer): add cross-worker idempotency unit test | ||
|
JamesandClaude Opus 4.7
|
e50587496f |
fix(producer): address PR review feedback on harness mode + plan() copy filter
Miguel (approved) and Vai (commented) both flagged the same PSNR-threshold doc/code mismatch; Vai additionally flagged a path-anchoring bug in the projectDir-copy filter and a dishonest type cast. Addressed all five findings: PSNR threshold doc/code mismatch (important): - Module docstring, `resolveMinPsnrForMode` JSDoc, and tests/README.md all claimed distributed-simulated tightens to ≥50 dB. The actual code uses `max(fixture.minPsnr, 10)` — 10 dB is a pathology floor, the per-test gate is the fixture's authored `minPsnr`. Updated all three doc sites to describe what the code does. The 50 dB target in §5.1 is a per- render distributed-vs-in-process contract; against the frozen baseline it's unreachable for either mode (shared encoder/JPEG jitter), so it can't be a per-fixture gate. `PLAN_PROJECT_DIR_COPY_SKIP` regex matched absolute paths (important): - `cpSync` calls the filter with the absolute source path, so a `projectDir` whose absolute path happens to contain a blocklisted segment (`/home/user/work/output/comp/`, `~/projects/dist/foo/`, etc.) caused the filter to return false for every descendant — empty compiled directory, broken render. Now matches relative-to-projectDir segments via `path.relative()` + `split(sep)`. Switched from a regex to a Set for clarity. Harness fixtures don't hit this because they live under `tests/<name>/src/`, but adapters call `plan()` with caller-supplied paths. Dishonest type cast in regression-harness.ts (important): - `as "mp4" | "mov" | "png-sequence"` claimed reachability for formats that `validateMetadata` doesn't accept (the schema is `"mp4" | "webm"`, and webm is rejected by `checkDistributedSupport`). Narrowed to hardcoded `format: "mp4"` with a comment naming the metadata-schema invariant that lets us do that. Renamed `chunkVideoInjectorFactory` (nit): - The variable was invoked once and never used again — "factory" implied repeated calls. Inlined as a plain `videoInjector: BeforeCaptureHook | null` ternary. Replaced tautology test (nit): - `expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10)` was a value-pin over an exported constant. The invariant the JSDoc actually asserts is "10 dB is below any real fixture's authored minPsnr"; if someone lands a permissive fixture (minPsnr: 5), the value-pin doesn't catch it. Replaced with a test that walks `tests/*/meta.json` and asserts every authored `minPsnr` is ≥ the floor. Validated in `docker:test --mode=distributed-simulated`: font-variant-numeric, many-cuts, gsap-letters-render-compat, style-1-prod, sub-composition-video — all PASSED. Unit tests: 15/15 pass (new fixture-scan test included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
b8e8617f80 |
refactor(producer): apply /simplify cleanups across distributed harness stack
Code reuse:
- Move `PlanVideosJson` interface + `meta/videos.json` path constant into
`services/distributed/shared.ts`; plan.ts and renderChunk.ts import from
there instead of redeclaring the same shape with the "duplicated here so
renderChunk doesn't import from plan.ts" comment.
- Replace hand-rolled `framePattern.slice(lastIndexOf("."))` with
`extname()` from `node:path` in `rebuildExtractedFramesFromPlanDir`.
Efficiency:
- Hoist `rebuildExtractedFramesFromPlanDir` + `createFrameLookupTable` +
`createVideoFrameInjector` out of the per-chunk closure in renderChunk.
Computed once per chunk now, not once per `createRenderVideoFrameInjector`
callsite (which `runCaptureStage` may invoke multiple times).
- Add a regex filter to `cpSync(projectDir → planDir/compiled/)` so
`node_modules`, `.git`, `output/`, `failures/`, `dist/`, etc. are not
copied. Real projects can have hundreds of MB in those directories;
shipping them to S3/Lambda /tmp on every render bloats cost and time.
- Drop redundant `if (!existsSync(metaDir)) mkdirSync(metaDir, {recursive:true})`
guards; `mkdirSync({recursive:true})` is already idempotent.
Quality:
- Strip narrative comments that told the story of debugging:
- renderChunk's 30-line "Two failure modes made the call actively
harmful" block → 4-line invariant on why `discardWarmupCapture` is
omitted.
- plan.ts's "DO NOT call cleanup()" block → 3 lines naming the
invariant.
- plan.ts's pre-seed-projectDir block → 7 lines on the file-server
invariant.
- renderChunk.ts top docstring's discardWarmupCapture paragraph.
- regression-harness-distributed.ts's PSNR-drift table (belongs in
DISTRIBUTED-RENDERING-PLAN.md, not the source).
- test file's docstring about which tests live where.
- Drop the unreachable IIFE-throw on `format === "webm"` in the harness
(the support check above rejected webm); replace with a plain
`as` cast.
- Replace dynamic `await import("node:fs")` with a top-level import in
`regression-harness-distributed.ts`.
All 54 distributed unit tests still pass in Docker. Full fixture sweep
in `docker:test --mode=distributed-simulated` (font-variant-numeric,
many-cuts, gsap-letters-render-compat, style-1-prod, sub-composition-video)
all PASSED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
JamesandClaude Opus 4.7
|
30f931b503 |
fix(producer): wire video frame injector into renderChunk
The chunk worker passed `createRenderVideoFrameInjector: () => null` to `runCaptureStage`, leaving the page's `<video>` elements to decode the source mp4 against the virtual clock. Chrome's native video pipeline seeks ±1 frame off what the in-process renderer captures (which uses pre-extracted frames injected as images via createVideoFrameInjector). That ±1 frame drift produced the PSNR gap on sub-composition-video and style-1-prod against the in-process baselines. Two pieces: 1. `plan()` now persists the engine's `VideoElement[]` (composition.videos) and a serialized form of `extractionResult.extracted` (videoId, srcPath, framePattern, fps, totalFrames, metadata — paths omitted) to `<planDir>/meta/videos.json`. This is the data renderChunk needs to reconstruct a `FrameLookupTable` without re-running the extract stage. 2. `plan()` no longer calls `frameLookup.cleanup()` after extraction. That cleanup was rm-rf-ing each video's outputDir, which for the in-process orchestrator is a scratch tree the renderer owns — but for plan() that "scratch" IS `compiledDir/__hyperframes_video_frames/<videoId>/`, the source material that the subsequent rename moves into `planDir/video-frames/`. Cleaning it up before the rename left planDir/video-frames/ with only the `_downloads/` subdirectory and no actual frame files. Both `style-1-prod` and `sub-composition-video` reproduced this on every distributed-simulated run; both pass after the cleanup is dropped. 3. `renderChunk` reads `meta/videos.json`, rebuilds `ExtractedFrames[]` by re-listing `planDir/video-frames/<videoId>/` for each video, calls `createFrameLookupTable(videos, extracted)`, and wraps the result in `createVideoFrameInjector` — the same hook the in-process renderer uses. The rebuilt entries set `ownedByLookup: false` so any later cleanup() call from the engine doesn't rm the planDir bytes another worker may still be reading. Validated in `docker:test --mode=distributed-simulated`: font-variant-numeric: PASSED many-cuts: PASSED gsap-letters-render-compat: PASSED style-1-prod: PASSED (was: 15 frames at 26-29 dB) sub-composition-video: PASSED (was: most frames at 21-25 dB) In-process unchanged; 54 distributed unit tests still pass in Docker. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
24f64f02c0 |
fix(producer): remove discardWarmupCapture call entirely
Validating the harness against a multi-chunk render (chunkSize=50 on many-cuts, N=4 chunks) revealed that the previous "discard at startFrame-1 for chunk N>0" fix had a second deadlock mode: the discard's frameTimeTicks (base + 49*interval) ended up LARGER than the captureStage first-call's frameTimeTicks (base + 0). Chrome's compositor wedges when asked to go backward in time as predictably as it wedges on a same-time duplicate. Both attempted fixes were trying to work around a problem that doesn't exist: lastFrameCache is only consulted when Chrome returns hasDamage=false, and every chunk frame seeks fresh DOM via __hf.seek() before the screenshot, so hasDamage is always true and the cache is never read. The priming step is unnecessary. Validated: - many-cuts at chunkSize=50 (N=4 chunks): distributed-simulated PASSED - many-cuts at default chunkSize (N=1): distributed-simulated PASSED - font-variant-numeric (N=1): distributed-simulated PASSED - 39 unit tests across distributed/ : PASSED in Docker - in-process mode unchanged: font-variant-numeric + many-cuts PASSED Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
9273eb2229 |
fix(producer): correct discardWarmupCapture chunk-0 deadlock and walk back probe overcorrection
Empirical investigation of --mode=distributed-simulated against many-cuts revealed that the BeginFrame "hang" attributed earlier to a Chrome 148 SwiftShader compositor wedge was actually a renderChunk bug: discardWarmupCapture was called with frameIndex=slice.startFrame, then captureStage immediately captured frame 0 (relative) of the chunk's range. For chunk 0 (slice.startFrame=0) these two calls produced the same frameTimeTicks. Chrome's HeadlessExperimental.beginFrame deadlocks when called twice in a row with the same frameTimeTicks — the compositor has no new damage to advance for, and the second call hangs until the Puppeteer protocolTimeout fires. Tracing the chunk worker confirmed: warmup call 1 t=0 -> ok warmup call 60 t=1947 -> ok (loop exited) beginFrame call #1 t=2333.33 -> returned, hasData=true, hasDamage=true beginFrame call #2 t=2333.33 -> HANG Fix: discardWarmupCapture skips chunk 0 (no prior frame to prime, and the in-process renderer also has an empty cache at frame 0) and uses slice.startFrame - 1 for chunk N>0 (the actual previous absolute frame, which more accurately matches what the in-process renderer's cache holds at the start of frame N). The engine probe complications I added earlier — multi-step screenshot test, inline data:URL pre-navigation, rastered-bytes assertion — were chasing a phantom and are reverted to the original simple form. chrome-headless-shell @stable on Linux with --use-angle=swiftshader renders BeginFrame screenshots correctly after the warmup loop; what looked like "wedged compositor" was the same frameTimeTicks deadlock masquerading as a Chrome regression. Also lowers the harness's distributed-simulated PSNR floor from 45 dB to 10 dB and switches to using the fixture's own minPsnr for both modes. The 45 dB floor was set against font-variant-numeric's static-content baseline drift (~48 dB), but dynamic compositions like many-cuts produce 34-44 dB baseline drift even in-process — both renderers share the same encoder/JPEG jitter floor, so requiring distributed to clear a tighter threshold than in-process catches no real regression. 10 dB remains as an absolute-pathology guard for fixtures with a permissive authored threshold. Validated end-to-end in `docker:test --mode=distributed-simulated`: font-variant-numeric: PASSED (PSNR ~48 dB, audio correlation 1.000) many-cuts: PASSED (PSNR 37-44 dB across rapid transitions) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JamesandClaude Opus 4.7
|
e80bf61d61 |
fix(producer,engine): make distributed renderChunk actually work end-to-end
Three Phase 3 regressions surfaced when validating --mode=distributed-simulated:
engine: probeBeginFrameSupport approved chrome-headless-shell 148 even when
its SwiftShader compositor was wedged. The existing noDisplayUpdates:true
probe returns instantly on 148 and the screenshot variant returned empty
data without erroring. The real capture loop then hung on first frame with
"HeadlessExperimental.beginFrame timed out". Probe now navigates to a small
inline page (matching the real capture's compositor state, not about:blank)
and asserts that 3 back-to-back beginFrame calls each return non-empty
screenshotData. Catches the 148 soft-failure mode; falls back to
Page.captureScreenshot.
producer/plan: plan() didn't copy local assets (style.css, script.js, etc.
referenced by relative URL) into planDir/compiled/. The in-process file
server serves these from projectDir, but the distributed chunk worker's
file server only sees compiledDir. Result: every composition with external
local files rendered as unstyled HTML. Now plan() pre-seeds compiledDir
with cpSync(projectDir, ..., {dereference:true}) before compileStage
overwrites the entry HTML, so the planDir is the self-contained bundle
the docstring claims.
producer/renderChunk: force forceScreenshot:true in the chunk worker's
EngineConfig. Chrome 148's BeginFrame screenshot wedge is content-dependent
— the engine probe (now improved) catches it for some pages but not all,
and the real capture loop hangs on composition-shaped content the probe
can't simulate. Page.captureScreenshot works on every chrome-headless-shell
build we've tested, and executeRenderJob already takes this path for
multi-worker mp4, so the distributed pipeline inherits the proven Linux
reliability profile.
Also lowers the harness's distributed-simulated PSNR floor to 45 dB.
The plan's 50 dB target was written for per-render comparison; against
the frozen baseline file, the in-process renderer itself drifts ~2 dB
due to libx264/JPEG-capture jitter, so 50 dB is empirically unreachable
for either mode. 45 dB tracks the observed ~47-48 dB floor and stays
well above the 30 dB fixture threshold.
Validated:
- font-variant-numeric in distributed-simulated: PASSED (PSNR ~48 dB
across 100 checkpoints, audio correlation 1.000).
- many-cuts surfaces a fourth Phase 3 issue: timing drift on compositions
with external script src= files. First ~5 frames render the
pre-script-execution state and later variants come in ~200 ms late vs
baseline. Tracking separately — the harness mode is correctly detecting
it as a regression, which is the point.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
James
|
91a91f66ba | feat(producer): export distributed render primitives | ||
|
James
|
e5068487c1 | feat(producer): refuse distributed-unsupported formats early | ||
|
James
|
fbbd41a797 | feat(producer): enforce planDir size cap with PLAN_TOO_LARGE | ||
|
James
|
b606106ea5 | feat(producer): add services/distributed/assemble.ts | ||
|
James
|
e55a8d0a0c | feat(producer): add services/distributed/renderChunk.ts |