Commit Graph
22 Commits
Author SHA1 Message Date
Miguel Ángel 1f5dbdbc4c fix(studio,producer): post-merge regressions and open issues (#1643)
- fix gsapDragCommit.test.ts import for commitGsapPositionFromDrag
- add generateId() helper with insecure-context fallback (fixes #1637)
- enable STUDIO_KEYFRAMES_ENABLED by default
- prevent clock duration from shrinking during playback (fixes #1636)
- add maxRetries to rmSync cleanup calls to handle ENOTEMPTY race (fixes #1644)
2026-06-22 11:19:52 -04:00
Miguel Ángel 8408a44745 fix(engine): tune VP9 cpu-used across render paths (#1614)
* fix(engine): tune VP9 cpu-used across render paths

* fix: address VP9 review feedback
2026-06-20 16:36:59 -04:00
Vance IngallsandClaude Opus 4.8 7310223b66 feat(engine): static-frame dedup default-on + render telemetry (#1549)
* feat(engine): static-frame dedup for screenshot capture (opt-in)

Skip re-seeking + re-screenshotting frames byte-identical to their predecessor. A
frame is dedupable iff no GSAP tween or clip cut is active in it or its predecessor
(predicted from window.__timelines + clip schedule) AND an empirical anchor-compare
confirms it. Opt-in HF_STATIC_DEDUP=true, default off.

Correctness (designed for the multi-worker / distributed render paths):
- Reuse is keyed by the ABSOLUTE composition frame (derived from the frame's time),
  NOT the captureFrameCore frameIndex arg — chunked/parallel callers pass a chunk-
  relative index. Validated lossless (PSNR=inf) on both single- and multi-worker
  renders of a static-hold comp.
- verifyStaticFramesSafe checks EVERY run (no longest-first budget truncation that
  left runs armed-but-unverified), and samples each run's FIRST reused frame, its END,
  and interior points at a stride; a hard cap disables dedup rather than trust an
  unverified set.
- Conservative arming: skipped when capture mode != screenshot (BeginFrame tick
  semantics + the verifier's screenshot path wouldn't transfer), when a before-capture
  hook is set (per-frame video injection), when page-side compositing is active (shader
  / drawElement composite the plain verification screenshot can't reproduce), and when
  any data-start is a non-numeric reference expression the clip-boundary parser can't
  protect, or duration is unknown/zero.
- Session reuse (prepareCaptureSessionForReuse) resets lastFrameBuffer + dedup counter
  so a probe/prior-render buffer can't bleed into the first static frame; the armed set
  is kept (same-composition reuse). Cost calibration bypasses dedup for its sparse,
  non-contiguous sample sweep, then restores the armed set.
- HF_STATIC_DEDUP_SAMPLES is NaN-guarded.

Disqualifies on signals the GSAP predictor can't see: video, canvas/webgl, zero
tweens, running CSS/WAAPI animation. Pays on static-hold content (title cards,
slideshow/kiosk loops, data-viz pauses); no-op on continuously-animated comps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(engine): static-frame dedup default-on + render telemetry

Flip dedup from opt-in (HF_STATIC_DEDUP=true) to default-on (opt-out
HF_STATIC_DEDUP=false). Verification (verifyStaticFramesSafe) is the
safety net that keeps reuse sound at scale.

Add end-to-end dedup observability. The capture session records
enabled / armed / skipReason / predicted; these surface via
CapturePerfSummary -> a dedupPerfs accumulator (disk sequential +
parallel AND streaming sequential + parallel) -> aggregated into
RenderPerfSummary.staticDedup (OR armed, SUM frames across workers) ->
render_complete props static_dedup_{enabled,armed,skip_reason,
predicted_frames,reused_frames}. skip_reason is a low-cardinality code:
capture_mode | video_injection | page_composite | ineligible |
verification_failed.

Distributed chunks run on Linux/beginframe where dedup never arms, so
they pass a throwaway dedupPerfs sink (no per-chunk reporting).

Tests: aggregation logic (OR/SUM/skip-reason) + opt-out passthrough.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(engine): address review on dedup default-on + telemetry

Review feedback (miga-heygen) + self-review fixes:

- Retry double-count: executeDiskCaptureWithAdaptiveRetry pushed worker
  dedup perf inside the retry loop, so an adaptive retry counted frames
  twice (reused/predicted could exceed totalFrames). Reset dedupPerfs at
  the start of each attempt — retry now REPLACES rather than accumulates;
  common no-retry path is unchanged.
- Opt-out parsing: HF_STATIC_DEDUP now disables on {false,0,off}
  case/space-insensitive (was strict !== "false", so `False`/`0` silently
  kept dedup on — the kill-switch could no-op).
- Verification budget vs drift: verifyStaticFramesSafe returns
  {badFrame, budgetExhausted}; armStaticDedup reports a distinct
  `verification_budget` skip reason so a telemetry spike means "raise
  HF_STATIC_DEDUP_SAMPLES", not "compositions are non-static".
- Index idiom: captureFrameCore now uses Math.floor(time*fps + 1e-9)
  (matches quantizeTimeToFrame) so the dedup lookup agrees with the frame
  the seek lands on even for non-exact times.
- Stale "opt-in HF_STATIC_DEDUP=true" comments -> "opt-out
  HF_STATIC_DEDUP=false" across frameCapture.ts + types.ts.
- Extract pushWorkerDedupPerfs helper (perfSummary.ts), used by the disk
  and streaming parallel paths — removes the duplicated push loop and
  drops captureStreamingStage back under the complexity threshold.
- dedupPerfs is now required (not optional) on
  executeDiskCaptureWithAdaptiveRetry — a missing arg silently dropped
  telemetry.
- Test: captureStreamingStage createInput() now provides the required
  dedupPerfs field.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(engine): address deferred dedup-review items

- Derivable state: drop session.staticDedupArmed/staticDedupPredicted;
  derive both from session.staticFrames in getCapturePerfSummary
  (armed ⟺ non-empty set, predicted === size) so they can't desync.
- Config altitude: HF_STATIC_DEDUP now resolves into
  EngineConfig.staticFrameDedup (resolveConfig, opt-out on {false,0,off}),
  alongside forceScreenshot/browserGpuMode — armStaticDedup reads config
  instead of process.env. Default-on preserved (missing config → enabled).
- Lossy aggregation: aggregateDedup now reports DISTINCT skip reasons
  (sorted, `|`-joined) across diverging unarmed workers instead of just
  the first.
- discardWarmupCapture: also snapshot/restore staticDedupCount and
  lastFrameBuffer so a warmup capture can't leak a phantom reuse or a
  stale buffer anchor into the real summary.
- Convention: perfSummary-dedup.test builds its job via createRenderJob
  instead of `as unknown as RenderJob`.
- Docs: verification_budget added to skip-reason lists.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 01:12:05 -07:00
Carlos Alcaraz GregorandCarlos Alcaraz 1f9c70cf15 fix(producer): guard fileServer.close in all render cleanup paths (#1406)
* fix(producer): guard fileServer.close in distributed cleanup paths

`plan()` and `renderChunk()` both close the probe/chunk file server with a
bare `fileServer.close()` in their cleanup sequence. `FileServerHandle.close`
tears down the underlying http.Server, whose `close()` throws
`ERR_SERVER_NOT_RUNNING` if the server was already torn down (for example a
cancellation path that closed it once already). An unguarded throw there
escapes the cleanup and masks the original plan/render result, exactly the
failure the adjacent probe-session close already guards against with a
try/catch (its comment even spells this out).

Add `closeFileServerSafely`, which wraps the close in a try/catch and logs,
and route both cleanup sites through it so the two stay consistent and a
throwing close can never mask the real result. Covered by unit tests for
both the throwing and happy paths.

* fix(producer): extend fileServer.close guard to renderOrchestrator success path

---------

Co-authored-by: Carlos Alcaraz <193642530+calcarazgre646@users.noreply.github.com>
2026-06-15 17:51:32 -07:00
Matt Van HornandMatt Van Horn e6b8d66c2d feat(cli,producer): add gif output format with two-pass palette encode (#1333)
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-06-10 21:17:55 -04:00
James Russo 69eac249d5 feat(producer): stage wall-clock split in chunk perf telemetry (#1327) 2026-06-10 18:09:36 -07:00
James Russo 852008bd44 feat(producer): thread variables through plan() + renderChunk() (#962)
Add `variables?: Record<string, unknown>` to DistributedRenderConfig
(§4.4) and LockedRenderConfig (§4.3). plan() snapshots the value into
meta/encoder.json so every chunk worker re-injects the same set via
captureOptions.variables, mirroring the in-process renderer's path.

The variables fold into planHash automatically because canonical
encoder.json bytes feed the hash: two plans with different variables
produce different hashes (chunked output depends on the injected
values); two plans with the same variables produce identical hashes
because canonical-JSON sorts keys.

The regression harnesses (distributed-simulated, lambda-local) also
forward the input's variables to plan() / Step Functions event so
fixtures that declare `renderConfig.variables` produce the same pixels
across modes. Previously the field was on the harness input shape but
silently dropped at the call boundary.

Phase 9 PR 9.1 of the distributed rendering plan.
2026-05-19 18:47:53 -04:00
James RussoandClaude Opus 4.7 5d264e146c docs(lambda): document webm support + simplify-review fixes (#953)
* docs(lambda): document webm support in distributed mode

PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see
DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the
shipped capability.

Updates docs/deploy/migrating-to-hyperframes-lambda.mdx:

- "Output format" row in the migration table now lists `webm` alongside
  mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 +
  closed-GOP concat-copy. HDR mp4 remains the only refused format.

- "No webm distributed" caveat replaced with "webm uses closed-GOP VP9"
  explainer covering the encoder args (`-g <chunkSize>`,
  `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why
  alt-ref disable is load-bearing, and that the output preserves alpha
  via yuva420p with Opus audio.

- Migration checklist no longer asks adopters to filter out webm
  compositions; only HDR-dependent renders need to stay on the previous
  framework.

aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR
in the v1 surface list), so it gets no copy edits beyond the migration
guide.

The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8,
§12 — kept outside the repo) gets matching updates: format support
matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the
new top item, and the rev-12 → rev-13 status line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: address simplify-review findings on webm stack

Folds in cleanups identified by a multi-agent code-review pass over the
4-PR webm-distributed stack:

- plan.ts: `resolveEncoderTriple()` webm case now calls
  `getEncoderPreset(quality, "webm")` for its preset string instead of
  hardcoding "good". The hardcode was wrong for `quality: "draft"`
  (`getEncoderPreset` returns "realtime" for that tier) — would have
  silently overridden the draft → realtime mapping for distributed webm
  renders.
- chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18
  lines of WHY narration down to the 6 lines that actually explain why
  (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push
  comment to the same standard.
- chunkEncoder.test.ts: drop the duplicate WHY comment that restated
  the implementation comment in plain words.
- webm-concat-copy.test.ts: rewrite the file-header docstring to
  describe the contract being tested instead of the PR-8.1-gating
  history; strip "PR 8.2 / Path A / Path B" references from error
  messages (they belong in PR bodies, not in test output). Consolidate
  the yuva420p alpha smoke into a single `it()` block (was a full
  4-test describe with duplicated setup) — the yuv420p block already
  covers the probe/decode/frame-count contract; the alpha smoke only
  needs to prove the alpha args don't break concat-copy.
- plan.test.ts: drop the "PR 8.1 proved the contract" comment.
- webm-vp9 fixture: drop the aspirational "Other webm-with-audio
  fixtures cover the mux path separately when added" sentence (no
  other fixtures exist). Regenerated the baseline via
  `docker:test:update webm-vp9` to reflect the updated comment.
- migrating-to-hyperframes-lambda.mdx: add a paragraph about
  distributed webm's perf cost — ~10-25% larger files at constant CRF
  due to forced keyframes, and slower per-chunk encode due to
  `-cpu-used 2` being more conservative than the libvpx default.

All unit tests + the webm-vp9 distributed-simulated regression still
pass after these changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cli): accept --format=webm in `hyperframes lambda render`

The CLI's `lambda render` subcommand's FORMATS allowlist and the
`RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`,
so even though the producer + aws-lambda packages now support webm
end-to-end, the CLI surface rejected it with `--format must be mp4|mov|
png-sequence`. Add webm to both spots and update the --help description.

Surfaced during real-AWS deploy prep — the local lambda-local /
distributed-simulated tests didn't go through the CLI so the gap went
unnoticed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME)

The deterministic Google Fonts cache was rooted at
`\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the
runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's
read-only. `mkdirSync(..., { recursive: true })` can't create that
path and the plan stage trips with `ENOENT: no such file or directory,
mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on
every Lambda render that pulls a Google Font (i.e. every distributed
fixture using `@import url("https://fonts.googleapis.com/...")`).

Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to
`tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives
across invocations on a warm container, so cache hit rate is the same
as non-Lambda runs. Also honor an explicit
`\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a
different location regardless of the runtime.

Surfaced while verifying webm distributed end-to-end on real AWS — the
same bug affects mp4 fixtures using Google Fonts; webm just happened to
be the one I tried first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: extract DistributedFormat type + trim font-cache resolver

Second simplify-review pass on the webm stack flagged two cleanups:

1. **`DistributedFormat` type duplicated 10 times.** Every file in the
   distributed pipeline carried its own copy of
   `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format
   meant a 10-place edit with no compile-time guarantee they stayed in
   sync. Extract a single source of truth in
   `packages/producer/src/services/distributed/shared.ts`, re-export
   from `@hyperframes/producer/distributed` and
   `@hyperframes/aws-lambda/sdk`, and have all callers pull from
   there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's
   `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]`
   so the compiler enforces the runtime allowlist stays in sync with
   the type.

2. **`deterministicFonts.ts` font-cache resolver was over-commented.**
   Trim the 7-line block to 4 lines (drop the aspirational
   "and other read-only-FS execution environments" — only Lambda is
   detected — and the warm-container `/tmp` persistence narration —
   anyone reading already knows Lambda /tmp semantics). Collapse the
   two-step `if (explicit && explicit.length > 0)` into a single
   nullish-coalesce expression now that the empty-string defensive
   check is gone (`process.env.X` is `string | undefined`, no third
   shape to guard against).

Out-of-scope skips (called out by the agents, deferred):
- In-process `RenderConfig.format` and the in-process CLI's
  `render.ts` format union still carry their own inline copies. The
  union happens to coincide today but they're separate concerns —
  leaving them alone limits this PR's blast radius.
- `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry
  flagged as taste; skipping.
- Pre-existing redundant `existsSync` before `mkdirSync({ recursive:
  true })` in `fontCacheDir` — out of scope.

All tests + typecheck still pass. Lambda render still works
end-to-end (no functional changes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(lambda): drop plan-doc reference from migration checklist

PR review feedback: source/docs should not mention the
distributed-rendering planning doc. Tighten the migration checklist
sentence to describe the webm path directly rather than referencing
the doc's version label.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers

CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 —
the function interleaved (a) mp4 codec validation + dispatch, (b) the
non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting
into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the
top-level function's cyclomatic complexity below the threshold while
preserving every error message and code path. Behavior unchanged.

Also extracts an `EncoderTriple` type alias so the three functions
share the return shape declaratively rather than repeating it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 04:11:26 -04:00
James RussoandClaude Opus 4.7 21f5066832 feat(producer): enable webm in distributed mode via concat-copy (#951)
* feat(producer): enable webm in distributed mode via concat-copy

PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see
DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the
distributed pipeline now that PR 8.1 proved concat-copy works.

Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke
test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams).
The simpler architecture wins; no re-encode in assemble, no encode-
parallelism loss.

Changes:

- plan.ts:
  - DistributedRenderConfig.format and PlanResult.format now include
    "webm" — type-level acceptance matches the runtime gate.
  - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR
    mp4 remains the only refused configuration.
  - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p +
    preset="good" for format="webm". yuva420p preserves alpha — the
    format's main reason for existing for web delivery.
  - codec= remains rejected for non-mp4 formats (mov is always ProRes
    4444; webm is always libvpx-vp9). The error message lists all four
    distributed-supported formats.
  - FormatNotSupportedInDistributedError docstring updated to reflect
    the new reality (only HDR is unsupported).

- freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software".
  Mirrors libx265-software / prores-software / png-sequence in shape;
  the chunk worker reads this discriminant to decide encode args.

- renderChunk.ts: drops the now-incorrect cast that excluded webm from
  buildSyntheticRenderJob's format input; tightens the preset-format
  cast to include webm.

- assemble.ts: docstring + comment updates. The mp4/mov concat-copy
  path is format-agnostic — webm uses the exact same code (applyFaststart
  is a no-op for webm via the existing chunkEncoder.ts gate;
  muxVideoWithAudio already routes webm to libopus audio).

- planFormatBanlist.test.ts: webm-rejection tests removed; replaced with
  "accepts webm" tests + a HDR+webm combo test that verifies HDR is the
  trip regardless of format.

- plan.test.ts: new describe block pins the webm wiring contract:
  format="webm" produces an encoder=libvpx-vp9-software /
  pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize.

- webm-concat-copy.test.ts (smoke): extended with a yuva420p variant
  that proves the alpha pixel format the distributed pipeline actually
  emits also round-trips through concat-copy. 9/9 tests pass locally.

§8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally
left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end
fixture (PR 8.3) is green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke

PR review feedback from Miguel and Vai on #951 caught a real bug:
`plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan
stage froze `forceScreenshot: false` into the `LockedRenderConfig`
even though distributed webm uses `yuva420p`. Every chunk worker
captured opaque RGB via BeginFrame (which doesn't preserve alpha on
Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha
that the encoder then dropped — producing un-keyable webm.

Two changes:

1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the
   in-process renderer's logic at `renderOrchestrator.ts:1469`
   (`const needsAlpha = isWebm || isMov || isPngSequence`); the two
   sites must stay in sync since the distributed pipeline's PSNR
   regression compares against the in-process baseline.

2. **Smoke test (yuva420p describe)**: source frames now use a real
   alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of
   `testsrc2 + format=rgba` which was uniformly opaque. The decode-
   pix_fmt assertion is dropped (ffprobe reports `yuv420p` for
   VP9-with-alpha because the alpha lives in a Matroska
   `BlockAdditional` sidecar) and replaced with two stronger checks:
   - `TAG:ALPHA_MODE=1` is present on the stream — proves the
     encoder was actually configured for alpha
   - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba
     -vf extractplanes=a,signalstats` — proves the alpha sub-stream
     round-trips through concat-copy with spatially-varying content,
     not uniform/dropped alpha
   - decode-test gate is now exit-code-only (was `exitCode || stderr`
     which would flake on chatty ffmpeg `-v error` builds emitting
     non-fatal DTS/container notes)

These checks would have caught the `needsAlpha` bug before review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(aws-lambda): widen narrow format types to include webm

CI on PR #951 was failing at typecheck/build because the producer's
`DistributedRenderConfig.format` widened to include webm in this PR
but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"`
type literals in `events.ts`, `handler.ts`, and `validateConfig.ts`
hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now
including webm) into a parameter typed against the narrow union,
producing TS2345.

This widening originally landed in PR #952 (test fixture PR) but
needs to be atomic with the producer's widening here to keep each
PR independently typecheck-clean.

Also refactor `formatExtension` from a switch dispatch to a
`Record<DistributedFormat, string>` lookup. Adding the webm case
tipped the switch's CRAP to the 30.0 fallow threshold; the lookup
table drops cyclomatic from 5 to 1 with the same compile-time
exhaustiveness guarantee (TS errors on missing entries when
`DistributedFormat` adds a new format). The runtime
`_exhaustive: never` throw was only protecting against a string
slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already
gates untrusted input at the SDK boundary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 02:46:21 -04:00
James Russo 17f47f30dd fix(distributed): gate per-worker SwiftShader probe to worker 0 only (#956)
After #916 moved `assertSwiftShader` from `renderChunk()`'s eager probe
session into `executeWorkerTask`, every parallel worker began running its
own `chrome://gpu` / canvas-WebGL probe. At `chunkWorkerCount=6` (texture
launch at chunks=3) that's 6 concurrent CDP page-loads per chunk × 3
chunks = 18 simultaneous probes. Bench data on dev (12 producer pods × 22
vCPU) showed c=3 worst-case wall-clock at 67.3s, 24.7s above c=6 worst
(42.6s) — pod_total inflates 100s → 147s uniformly across all three
chunks per slow iter, the signature of cluster-level CDP contention
rather than within-pod contention.

Workers within a chunk share the same Chrome binary, flags, and OS/driver
state on a single pod, so worker 0's success is representative for the
rest. Gate the probe via `shouldVerifyWorkerGpu(workerId, config)` so
only worker 0 navigates to the probe page; workers 1..N-1 skip it. The
fail-fast contract still holds at the chunk level (worker 0 still aborts
the chunk if SwiftShader didn't load) — just without the concurrent CDP
traffic.

Expected wall-clock impact: c=3 worst drops from ~67s to in line with
c=6 worst (~42-44s). c=6 (3 workers/pod) and c=8 (2 workers/pod) should
see smaller wins; c=12 (1 worker/pod, sequential branch) is unaffected.

Closes #955.
2026-05-18 22:30:13 -04:00
James Russo f01fccb0ea perf(distributed): skip eager probe session when chunkWorkerCount > 1 (#916) 2026-05-17 04:18:47 -04:00
James Russo 22363a11c9 perf(distributed): parallelize chunk capture across multiple workers (#906)
* perf(distributed): parallelize chunk capture across multiple workers

The distributed `renderChunk` primitive hardcoded `workerCount: 1` and
`captureStage` explicitly forbade `workerCount > 1` when `frameRange` was
set, with the comment:

  "Distributed chunk workers fan out at the activity layer; reduce
   workerCount to 1 when passing frameRange."

The assumption was that orchestration-layer fan-out (Temporal / Lambda /
K8s Jobs / SSH) saturates the available CPU on its own. In practice
adopters that deploy chunks onto multi-core hosts (8-24 vCPU is the
standard producer-worker pod sizing) end up pinning only ~3-4 cores per
chunk while the rest sit idle: chunk-level fan-out at the orchestration
layer gives each pod one chunk at a time, and the chunk render itself
was single-threaded.

Validated against a real 1080p / 30fps / 22-second shader-heavy
composition on a 22-vCPU Temporal pod: each chunk rendered at
165-273ms per frame (vs 94-98ms for the in-process streaming render
which runs `workerCount=2` by default). The slowest chunk gates total
wall-clock under parallel chunk fan-out, so the 2-3x per-frame gap
compounds and `distributed` was net-slower than `in-process` on every
composition smaller than ~5min of texture-class content. Lifting the
restriction is a measured ~2x per-chunk speedup with no contract
change at the framesDir or encoder layer.

Wire-up:

  * `WorkerTask.outputFrameOffset` — optional offset subtracted from the
    absolute frame index when computing the captured file's name.
    Default 0 (the in-process contract; file name == absolute index).
    Distributed chunks set this to the chunk's startFrame so file names
    land 0-indexed within the chunk's range, matching the sequential
    chunk-capture contract and the encoder's expectation that frames
    are read sequentially without an `-start_number` override.

  * `distributeFrames(totalFrames, workerCount, workDir, rangeStart=0)` —
    offsets both `startFrame`/`endFrame` (used for per-frame time math
    on the page's virtual clock) by `rangeStart`, and threads
    `outputFrameOffset = rangeStart` onto each task it emits. With the
    default `rangeStart=0` it is a no-op for in-process renders.

  * `executeWorkerTask` — uses `i - (task.outputFrameOffset ?? 0)` for
    the captured file name, leaving the per-frame TIME computation
    `(i * fps.den) / fps.num` untouched so the page's virtual clock is
    unchanged.

  * `executeDiskCaptureWithAdaptiveRetry({ frameRangeStart? })` — accepts
    the chunk's absolute startFrame and forwards it to `distributeFrames`
    and `buildMissingFrameRetryBatches`. Default `undefined` preserves
    the in-process contract.

  * `buildMissingFrameRetryBatches(ranges, ..., rangeStart=0)` —
    `findMissingFrameRanges` walks LOCAL 0-indexed file names; the retry
    batch translates the local missing-range pair back to ABSOLUTE
    composition indices for `WorkerTask.startFrame/endFrame` and sets
    `outputFrameOffset = rangeStart` so the retried capture writes back
    to the same local file name.

  * `captureStage` — drops the assert; passes
    `frameRangeStart: frameRange?.startFrame` to the parallel branch so
    workers land on absolute composition frame indices for time math
    while file names stay 0-indexed within the chunk range. Docstring
    updated to reflect that the parallel branch is now supported.

  * `renderChunk` — `workerCount: 1` → `workerCount: 2`. The pre-warmed
    `probeSession` is consumed only by the sequential branch; the
    parallel branch closes it during stage entry and creates its own
    worker sessions. Documented as a follow-up: skip probeSession
    creation when `workerCount > 1` to recover the ~3-5s warmup cost.

Backwards compatibility: every change is gated on a parameter that
defaults to the prior behavior. In-process callers (`executeRenderJob`)
pass no `frameRangeStart`, so `rangeStart === 0`, `outputFrameOffset`
defaults to 0, and the file-name math collapses to the prior `i` value.
The framesDir contract (`frame_0..frame_(totalFrames-1)`) and the
WorkerTask interface are extended, not replaced.

Tests: 24 pass / 0 fail across the distributed test suite (renderChunk,
plan, assemble, planFormatBanlist, planSizeCap, publicExports). 7 pass /
0 fail in `parallelCoordinator.test.ts`. The renderOrchestrator suite
has one pre-existing Windows-only failure
(`writeCompiledArtifacts — external assets on Windows drive-letter
paths`) unrelated to this change; the other 56 tests pass.

Refs: distributed-vs-inprocess benchmark thread at
heygen-com/experiment-framework#36950

* perf(distributed): auto-size chunk workerCount via calculateOptimalWorkers

Match the in-process renderer's worker selection instead of hardcoding 2.
`calculateOptimalWorkers(framesInChunk, undefined, cfg)` is the same call
`resolveRenderWorkerCount` makes under the hood, minus the capture-cost
calibration reduction (which would require plumbing the chunk's compiled
metadata through — left as a follow-up).

For a typical 22-vCPU producer-worker pod with `cfg.concurrency: "auto"`
this resolves to ~6 workers for a 240-frame chunk (capped by
`defaultSafeMaxWorkers() = max(6, min(16, floor(cpuCount/8)))`), matching
what `executeRenderJob` (the in-process path) already does. The prior
hardcoded `workerCount: 2` was a safe-minimum starting point that
undersized chunks vs prod's auto behavior.

Tests: 12/12 pass in `renderChunk.test.ts` (unchanged — the test suite
mocks the inner runCaptureStage call so workerCount selection is opaque
to it).

* refactor(distributed): /simplify pass on PR #906

Review pass on the parallel-capture frame-range change. Four targeted
cleanups identified by code-quality and efficiency review agents:

1. Add the missing `frameRange.endFrame - frameRange.startFrame === totalFrames`
   assert. The parallel branch forwards `totalFrames` separately from
   `frameRangeStart`; a caller passing mismatched values would have got a
   silently wrong distribution. The sequential branch already implicitly
   relied on this via its `rangeFrames = rangeEnd - rangeStart` arithmetic.

2. Collapse three near-duplicate docstrings (on `WorkerTask.outputFrameOffset`,
   `executeDiskCaptureWithAdaptiveRetry.frameRangeStart`, and `runCaptureStage`'s
   `frameRange`) so only the WorkerTask field carries the full contract. The
   other two cross-reference it.

3. Drop the WHAT-narrating comments inside `executeWorkerTask`'s per-frame
   loop. The variable names (`fileFrameIdx = i - outputOffset`) already say
   what the line does; the only remaining comment flags the non-obvious
   contract that the streaming callback gets the absolute index.

4. Trim the 30-line `chunkWorkerCount` block in `renderChunk` to one paragraph
   explaining the one non-obvious thing (why we use `calculateOptimalWorkers`
   directly instead of `resolveRenderWorkerCount`). The probeSession-wasted-on-
   parallel acknowledgement stays as a 3-line follow-up flag — investigated
   skipping it in this pass, but the SwiftShader probe is safety-critical and
   has no per-worker equivalent, so deferred to a separate change with proper
   per-worker assertion plumbing.

Tests + format + lint clean:
  * `bun test parallelCoordinator.test.ts` — 7/7
  * `bun test distributed/{renderChunk,plan}.test.ts` — 24/24
  * `bunx oxfmt` + `bunx oxlint` — clean
2026-05-16 16:29:34 -07:00
James 38e08e81c7 style(producer): apply oxfmt to resolvePresetForLockedEncoder signature
The generic parameter constraint exceeded oxfmt's line width, so the
formatter wraps the type-param list onto its own line. Applies the same
formatting locally that CI's 'Format' job would have produced via
'bun run format:check' — no behavior change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-05-15 01:57:08 +00:00
James db62e31d39 fix(producer): reject unknown codec strings + extract testable preset-override helper
Address @vanceingalls review on #850:

1. Unknown codec strings (typos like 'H265', future additions like 'av1')
   silently fell through to libx264 in resolveEncoderTriple. Add an
   explicit throw symmetric to the non-mp4-format branch already there.
   A JS caller building config from JSON who passes 'codec: "h266"'
   now gets a clear error at plan time instead of unflagged h264 output.

2. The preset.codec override in renderChunk had no fast unit coverage —
   only the heavyweight Docker fixture in #851 would catch a regression
   if someone refactored the spread (e.g. moved it into getEncoderPreset
   itself). Extract resolvePresetForLockedEncoder() and add 4 fast unit
   tests pinning the four encoder shapes (libx265/libx264/prores/png-seq).

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-05-15 01:57:08 +00:00
James eccea88daf feat(producer): add codec knob to DistributedRenderConfig 2026-05-15 01:57:08 +00:00
JamesandClaude Opus 4.7 e50587496f fix(producer): address PR review feedback on harness mode + plan() copy filter
Miguel (approved) and Vai (commented) both flagged the same
PSNR-threshold doc/code mismatch; Vai additionally flagged a
path-anchoring bug in the projectDir-copy filter and a dishonest type
cast. Addressed all five findings:

PSNR threshold doc/code mismatch (important):
- Module docstring, `resolveMinPsnrForMode` JSDoc, and tests/README.md all
  claimed distributed-simulated tightens to ≥50 dB. The actual code uses
  `max(fixture.minPsnr, 10)` — 10 dB is a pathology floor, the per-test
  gate is the fixture's authored `minPsnr`. Updated all three doc sites
  to describe what the code does. The 50 dB target in §5.1 is a per-
  render distributed-vs-in-process contract; against the frozen baseline
  it's unreachable for either mode (shared encoder/JPEG jitter), so it
  can't be a per-fixture gate.

`PLAN_PROJECT_DIR_COPY_SKIP` regex matched absolute paths (important):
- `cpSync` calls the filter with the absolute source path, so a
  `projectDir` whose absolute path happens to contain a blocklisted
  segment (`/home/user/work/output/comp/`, `~/projects/dist/foo/`, etc.)
  caused the filter to return false for every descendant — empty
  compiled directory, broken render. Now matches relative-to-projectDir
  segments via `path.relative()` + `split(sep)`. Switched from a regex
  to a Set for clarity. Harness fixtures don't hit this because they
  live under `tests/<name>/src/`, but adapters call `plan()` with
  caller-supplied paths.

Dishonest type cast in regression-harness.ts (important):
- `as "mp4" | "mov" | "png-sequence"` claimed reachability for formats
  that `validateMetadata` doesn't accept (the schema is `"mp4" | "webm"`,
  and webm is rejected by `checkDistributedSupport`). Narrowed to
  hardcoded `format: "mp4"` with a comment naming the metadata-schema
  invariant that lets us do that.

Renamed `chunkVideoInjectorFactory` (nit):
- The variable was invoked once and never used again — "factory" implied
  repeated calls. Inlined as a plain `videoInjector: BeforeCaptureHook | null`
  ternary.

Replaced tautology test (nit):
- `expect(DISTRIBUTED_SIMULATED_MIN_PSNR_DB).toBe(10)` was a value-pin
  over an exported constant. The invariant the JSDoc actually asserts is
  "10 dB is below any real fixture's authored minPsnr"; if someone lands
  a permissive fixture (minPsnr: 5), the value-pin doesn't catch it.
  Replaced with a test that walks `tests/*/meta.json` and asserts every
  authored `minPsnr` is ≥ the floor.

Validated in `docker:test --mode=distributed-simulated`:
  font-variant-numeric, many-cuts, gsap-letters-render-compat,
  style-1-prod, sub-composition-video — all PASSED.
Unit tests: 15/15 pass (new fixture-scan test included).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 20:33:03 +00:00
JamesandClaude Opus 4.7 b8e8617f80 refactor(producer): apply /simplify cleanups across distributed harness stack
Code reuse:
- Move `PlanVideosJson` interface + `meta/videos.json` path constant into
  `services/distributed/shared.ts`; plan.ts and renderChunk.ts import from
  there instead of redeclaring the same shape with the "duplicated here so
  renderChunk doesn't import from plan.ts" comment.
- Replace hand-rolled `framePattern.slice(lastIndexOf("."))` with
  `extname()` from `node:path` in `rebuildExtractedFramesFromPlanDir`.

Efficiency:
- Hoist `rebuildExtractedFramesFromPlanDir` + `createFrameLookupTable` +
  `createVideoFrameInjector` out of the per-chunk closure in renderChunk.
  Computed once per chunk now, not once per `createRenderVideoFrameInjector`
  callsite (which `runCaptureStage` may invoke multiple times).
- Add a regex filter to `cpSync(projectDir → planDir/compiled/)` so
  `node_modules`, `.git`, `output/`, `failures/`, `dist/`, etc. are not
  copied. Real projects can have hundreds of MB in those directories;
  shipping them to S3/Lambda /tmp on every render bloats cost and time.
- Drop redundant `if (!existsSync(metaDir)) mkdirSync(metaDir, {recursive:true})`
  guards; `mkdirSync({recursive:true})` is already idempotent.

Quality:
- Strip narrative comments that told the story of debugging:
  - renderChunk's 30-line "Two failure modes made the call actively
    harmful" block → 4-line invariant on why `discardWarmupCapture` is
    omitted.
  - plan.ts's "DO NOT call cleanup()" block → 3 lines naming the
    invariant.
  - plan.ts's pre-seed-projectDir block → 7 lines on the file-server
    invariant.
  - renderChunk.ts top docstring's discardWarmupCapture paragraph.
  - regression-harness-distributed.ts's PSNR-drift table (belongs in
    DISTRIBUTED-RENDERING-PLAN.md, not the source).
  - test file's docstring about which tests live where.
- Drop the unreachable IIFE-throw on `format === "webm"` in the harness
  (the support check above rejected webm); replace with a plain
  `as` cast.
- Replace dynamic `await import("node:fs")` with a top-level import in
  `regression-harness-distributed.ts`.

All 54 distributed unit tests still pass in Docker. Full fixture sweep
in `docker:test --mode=distributed-simulated` (font-variant-numeric,
many-cuts, gsap-letters-render-compat, style-1-prod, sub-composition-video)
all PASSED.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 19:51:18 +00:00
JamesandClaude Opus 4.7 30f931b503 fix(producer): wire video frame injector into renderChunk
The chunk worker passed `createRenderVideoFrameInjector: () => null` to
`runCaptureStage`, leaving the page's `<video>` elements to decode the
source mp4 against the virtual clock. Chrome's native video pipeline
seeks ±1 frame off what the in-process renderer captures (which uses
pre-extracted frames injected as images via createVideoFrameInjector).
That ±1 frame drift produced the PSNR gap on sub-composition-video and
style-1-prod against the in-process baselines.

Two pieces:

1. `plan()` now persists the engine's `VideoElement[]` (composition.videos)
   and a serialized form of `extractionResult.extracted` (videoId,
   srcPath, framePattern, fps, totalFrames, metadata — paths omitted) to
   `<planDir>/meta/videos.json`. This is the data renderChunk needs to
   reconstruct a `FrameLookupTable` without re-running the extract stage.

2. `plan()` no longer calls `frameLookup.cleanup()` after extraction.
   That cleanup was rm-rf-ing each video's outputDir, which for the
   in-process orchestrator is a scratch tree the renderer owns — but for
   plan() that "scratch" IS `compiledDir/__hyperframes_video_frames/<videoId>/`,
   the source material that the subsequent rename moves into
   `planDir/video-frames/`. Cleaning it up before the rename left
   planDir/video-frames/ with only the `_downloads/` subdirectory and no
   actual frame files. Both `style-1-prod` and `sub-composition-video`
   reproduced this on every distributed-simulated run; both pass after
   the cleanup is dropped.

3. `renderChunk` reads `meta/videos.json`, rebuilds `ExtractedFrames[]`
   by re-listing `planDir/video-frames/<videoId>/` for each video, calls
   `createFrameLookupTable(videos, extracted)`, and wraps the result in
   `createVideoFrameInjector` — the same hook the in-process renderer
   uses. The rebuilt entries set `ownedByLookup: false` so any later
   cleanup() call from the engine doesn't rm the planDir bytes another
   worker may still be reading.

Validated in `docker:test --mode=distributed-simulated`:
  font-variant-numeric:           PASSED
  many-cuts:                      PASSED
  gsap-letters-render-compat:     PASSED
  style-1-prod:                   PASSED (was: 15 frames at 26-29 dB)
  sub-composition-video:          PASSED (was: most frames at 21-25 dB)

In-process unchanged; 54 distributed unit tests still pass in Docker.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 18:56:05 +00:00
JamesandClaude Opus 4.7 24f64f02c0 fix(producer): remove discardWarmupCapture call entirely
Validating the harness against a multi-chunk render (chunkSize=50 on
many-cuts, N=4 chunks) revealed that the previous "discard at startFrame-1
for chunk N>0" fix had a second deadlock mode: the discard's
frameTimeTicks (base + 49*interval) ended up LARGER than the captureStage
first-call's frameTimeTicks (base + 0). Chrome's compositor wedges when
asked to go backward in time as predictably as it wedges on a same-time
duplicate.

Both attempted fixes were trying to work around a problem that doesn't
exist: lastFrameCache is only consulted when Chrome returns
hasDamage=false, and every chunk frame seeks fresh DOM via __hf.seek()
before the screenshot, so hasDamage is always true and the cache is
never read. The priming step is unnecessary.

Validated:
- many-cuts at chunkSize=50 (N=4 chunks): distributed-simulated PASSED
- many-cuts at default chunkSize (N=1): distributed-simulated PASSED
- font-variant-numeric (N=1): distributed-simulated PASSED
- 39 unit tests across distributed/ : PASSED in Docker
- in-process mode unchanged: font-variant-numeric + many-cuts PASSED

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 17:43:09 +00:00
JamesandClaude Opus 4.7 9273eb2229 fix(producer): correct discardWarmupCapture chunk-0 deadlock and walk back probe overcorrection
Empirical investigation of --mode=distributed-simulated against many-cuts
revealed that the BeginFrame "hang" attributed earlier to a Chrome 148
SwiftShader compositor wedge was actually a renderChunk bug:
discardWarmupCapture was called with frameIndex=slice.startFrame, then
captureStage immediately captured frame 0 (relative) of the chunk's range.
For chunk 0 (slice.startFrame=0) these two calls produced the same
frameTimeTicks. Chrome's HeadlessExperimental.beginFrame deadlocks when
called twice in a row with the same frameTimeTicks — the compositor has no
new damage to advance for, and the second call hangs until the Puppeteer
protocolTimeout fires.

Tracing the chunk worker confirmed:
  warmup call 1 t=0  -> ok
  warmup call 60 t=1947 -> ok (loop exited)
  beginFrame call #1 t=2333.33 -> returned, hasData=true, hasDamage=true
  beginFrame call #2 t=2333.33 -> HANG

Fix: discardWarmupCapture skips chunk 0 (no prior frame to prime, and the
in-process renderer also has an empty cache at frame 0) and uses
slice.startFrame - 1 for chunk N>0 (the actual previous absolute frame,
which more accurately matches what the in-process renderer's cache holds
at the start of frame N).

The engine probe complications I added earlier — multi-step screenshot
test, inline data:URL pre-navigation, rastered-bytes assertion — were
chasing a phantom and are reverted to the original simple form.
chrome-headless-shell @stable on Linux with --use-angle=swiftshader
renders BeginFrame screenshots correctly after the warmup loop; what
looked like "wedged compositor" was the same frameTimeTicks deadlock
masquerading as a Chrome regression.

Also lowers the harness's distributed-simulated PSNR floor from 45 dB to
10 dB and switches to using the fixture's own minPsnr for both modes. The
45 dB floor was set against font-variant-numeric's static-content
baseline drift (~48 dB), but dynamic compositions like many-cuts produce
34-44 dB baseline drift even in-process — both renderers share the same
encoder/JPEG jitter floor, so requiring distributed to clear a tighter
threshold than in-process catches no real regression. 10 dB remains as an
absolute-pathology guard for fixtures with a permissive authored
threshold.

Validated end-to-end in `docker:test --mode=distributed-simulated`:
  font-variant-numeric: PASSED (PSNR ~48 dB, audio correlation 1.000)
  many-cuts:            PASSED (PSNR 37-44 dB across rapid transitions)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 16:42:21 +00:00
JamesandClaude Opus 4.7 e80bf61d61 fix(producer,engine): make distributed renderChunk actually work end-to-end
Three Phase 3 regressions surfaced when validating --mode=distributed-simulated:

engine: probeBeginFrameSupport approved chrome-headless-shell 148 even when
its SwiftShader compositor was wedged. The existing noDisplayUpdates:true
probe returns instantly on 148 and the screenshot variant returned empty
data without erroring. The real capture loop then hung on first frame with
"HeadlessExperimental.beginFrame timed out". Probe now navigates to a small
inline page (matching the real capture's compositor state, not about:blank)
and asserts that 3 back-to-back beginFrame calls each return non-empty
screenshotData. Catches the 148 soft-failure mode; falls back to
Page.captureScreenshot.

producer/plan: plan() didn't copy local assets (style.css, script.js, etc.
referenced by relative URL) into planDir/compiled/. The in-process file
server serves these from projectDir, but the distributed chunk worker's
file server only sees compiledDir. Result: every composition with external
local files rendered as unstyled HTML. Now plan() pre-seeds compiledDir
with cpSync(projectDir, ..., {dereference:true}) before compileStage
overwrites the entry HTML, so the planDir is the self-contained bundle
the docstring claims.

producer/renderChunk: force forceScreenshot:true in the chunk worker's
EngineConfig. Chrome 148's BeginFrame screenshot wedge is content-dependent
— the engine probe (now improved) catches it for some pages but not all,
and the real capture loop hangs on composition-shaped content the probe
can't simulate. Page.captureScreenshot works on every chrome-headless-shell
build we've tested, and executeRenderJob already takes this path for
multi-worker mp4, so the distributed pipeline inherits the proven Linux
reliability profile.

Also lowers the harness's distributed-simulated PSNR floor to 45 dB.
The plan's 50 dB target was written for per-render comparison; against
the frozen baseline file, the in-process renderer itself drifts ~2 dB
due to libx264/JPEG-capture jitter, so 50 dB is empirically unreachable
for either mode. 45 dB tracks the observed ~47-48 dB floor and stays
well above the 30 dB fixture threshold.

Validated:
- font-variant-numeric in distributed-simulated: PASSED (PSNR ~48 dB
  across 100 checkpoints, audio correlation 1.000).
- many-cuts surfaces a fourth Phase 3 issue: timing drift on compositions
  with external script src= files. First ~5 frames render the
  pre-script-execution state and later variants come in ~200 ms late vs
  baseline. Tracking separately — the harness mode is correctly detecting
  it as a regression, which is the point.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 08:19:34 +00:00
James e55a8d0a0c feat(producer): add services/distributed/renderChunk.ts 2026-05-14 01:44:40 +00:00