Linear: VA-1859
## Problem
For a `fit_to_scene` B-roll where the composition's intrinsic timeline (e.g. `data-duration=1.0s` → 30 frames) is shorter than the scene it fills (e.g. 4.8s narration), the producer renders only the intrinsic 30 frames and the downstream compositor frame-holds/PTS-stretches that fixed clip to the scene length. Spreading 30 unique frames over 4.8s starves motion to ~6 effective fps → a visibly choppy result. Root cause: the producer welds one `composition.duration` to both the frame count and the 1:1 seek mapping, with no notion of a target output length.
## Fix
Add optional `renderStretch: number` (default `1.0` = no-op), `renderStretch = intrinsic / target`:
- **Frame count** comes from the target: `outputDuration = intrinsic / renderStretch`, `totalFrames = outputDuration × fps` (`probeStage.ts`). `composition.duration` stays intrinsic (drives video/audio windows).
- **Per-frame seek** is scaled: `time = (frameIndex / fps) × renderStretch`, so the N output frames map across `[0, intrinsic]` — a fresh frame per output frame.
All seek sites go through a single shared `outputFrameToTimelineSeconds(frameIndex, fps, renderStretch)` helper (`core.types.ts`), consumed by every capture path so none can silently diverge:
- parallel (`parallelCoordinator.ts`), `sdr_streaming` (`captureStreamingStage.ts` ×3), `sdr_disk` (`captureStage.ts`), HDR loops.
- DrawElement + static self-verify (`frameCapture.ts`) — ground-truth seek uses the same mapping, so PSNR compares like-for-like (no spurious verification failure on stretched comps).
- Distributed path: `renderStretch` threaded through `DistributedRenderConfig` → chunk workers, and **folded into the plan hash only when `!= 1`** so a pre-stretch cached plan is never reused.
With `renderStretch = 1` (or omitted → `?? 1`): every seek is `×1.0` (IEEE-754 identity), frame counts unchanged, and the plan hash is byte-identical — a provable no-op. `player.ts` absolute-seek is untouched.
## Verify
- typecheck (core + engine + producer): pass. lint/format/fallow/commitlint: pass. `planHash` + `renderRequest` unit suites: pass.
- Adversarial self-review found + fixed three capture-path gaps (streaming, self-verify, distributed) before this revision.
- **Not yet runtime-verified** on a real render — needs a fit_to_scene render at `renderStretch < 1` confirming N distinct frames over the target length (draft until then).
Paired with experiment-framework#42766, which computes and forwards `renderStretch = hf intrinsic / scene duration`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* fix(core): stop the async media-metadata rebind once render capture starts seeking
scheduleMetadataDurationHydration re-resolves and can swap the captured
GSAP timeline off a debounced loadedmetadata/durationchange event, fully
uncoordinated with the producer's own per-frame renderSeek calls. When a
full-length <video>'s metadata resolves after capture has already begun
(slow I/O, Docker), this races the deterministic BeginFrame capture loop
and can reflow sub-composition state mid-render, producing phase-offset
duplicate content in captured frames (#2550).
Render-mode duration correction already happens deterministically during
the probe stage before capture starts, so once renderSeek has been called
once there is nothing left for this self-correction to do — gate it off
for the rest of the session.
* fix(core): scope the metadata-rebind guard to actual render/export pages
renderSeek isn't capture-exclusive — Studio's own preview iframe falls
back to it for compositions whose timeline overhangs every native
adapter's duration. Gating the HF#2550 fix on renderCaptureSeekStarted
alone silently disabled the metadata-driven duration self-correction for
that live-scrub case too, where it's still needed. Require the render/
export page signal (window.__HF_EXPORT_RENDER_SEEK_CONFIG, set only by
the producer's fileServer.ts) alongside it, and add a regression test
covering the Studio-preview case.
* fix(engine): stop requesting beyond-viewport capture for video comps that don't need it
Root-caused HF#2550 by reproducing the reporter's public repro end-to-end
(not just the timeline-rebind mechanism from the earlier commits in this
branch) on native Linux: instrumented the actual DOM state during a real
capture session and confirmed the sub-composition never double-mounts —
getBoundingClientRect and the timeline's own local time both match the
single, correct DOM tree throughout. The phantom second copy only exists
in the captured screenshot pixels.
Bisected it to captureBeyondViewport: resolveVideoCaptureBeyondViewport
(#1094's tall-portrait fix) forces `Page.captureScreenshot`'s beyond-viewport
path on for any render with a native <video>, regardless of whether the
page's content actually overflows the declared capture height. On
SwiftShader that beyond-viewport path can composite a stale, vertically
offset paint of the page alongside the fresh one for content that fits
entirely within the viewport — producing exactly the reported phase-offset
duplicate. Disabling captureBeyondViewport (repro's video still present)
eliminates the duplicate outright; re-enabling it reproduces the duplicate
byte-for-byte, isolating it as the actual cause.
Adds pageContentExceedsCaptureHeight, a ground-truth measurement of the
page's actual scrollHeight against the requested capture height, and wires
it into initializeSession to downgrade captureBeyondViewport back to false
once the page is settled and it's confirmed unnecessary — the "reliable
clip predictor" the original #1094 fix's ponytail comment flagged as
missing. This keeps #1094's fix intact for content that genuinely
overflows while closing the SwiftShader ghosting hazard for the (common)
case of video that fits inside its own viewport.
* test(producer): add HF#2550 video+sub-composition regression fixture
Checks in the reporter's confirmed real-world reproduction (media
regenerated via ffmpeg testsrc2, matching their public repro repo) as a
regression fixture, with a golden baseline rendered against the fix.
Verified end-to-end via the project's own Docker regression harness:
- Rendering this fixture with the fix produces the golden baseline
(clean, single flowchart instance, captureBeyondViewport correctly
downgraded).
- Direct CLI renders (not through this harness) against unpatched code
reproduce the reported phantom-duplicate artifact reliably (10/10).
Caveat documented in meta.json: the underlying bug is timing-dependent.
Two harness runs against unpatched code, using this same fixture, did
not reproduce the artifact (0/2) — the harness's in-process render path
apparently doesn't hit the same race window a direct CLI process does on
this host. This fixture is a best-effort regression guard and a
preserved real-world repro, not the sole protection — the deterministic
guard is packages/engine/src/services/screenshotService.test.ts's
pageContentExceedsCaptureHeight unit tests, which exercise the actual
fix logic directly.
Also adds an .gitattributes LFS rule for this fixture's source
index.html (744 KB — carries the real project's embedded base64
assets, over the largefiles hook's 500 KB non-LFS limit).
* fix: route HF#2550 fixture binaries through LFS (were committed raw)
filter.lfs.clean/smudge were locally configured as a no-op "cat" in
this repo's shared .git/config, silently disabling LFS filtering for
every worktree. The previous commit's large binaries (output.mp4,
compiled.html, source index.html, source video) landed as raw blobs
instead of LFS pointers as a result. Ran `git lfs install --local
--force` to restore the correct filter commands, then re-staged the
affected files so they commit as proper LFS pointers.
* fix(engine): address capture viewport review feedback
The cli, engine, and lint packages each carried their own copy of the
ffmpeg/ffprobe lookup, annotated fallow-ignore code-duplication, and the
copies had drifted: the engine copy handled Windows PATHEXT and executed
which/where without a shell but lacked the Homebrew-dirs fallback for
GUI-spawned processes; the cli copy had the opposite. One resolver in
@hyperframes/parsers (the dependency-graph bottom) now carries the union
of both hardenings, and all three packages delegate to it. Every
consumer gets strictly more robust resolution; env-override semantics
per call site are preserved via configuredMustExist.
Windows users with the OS temp dir on a small system drive have hit
C: exhaustion mid-render (Slack ts=1784219488 · CLI v0.7.58 · win32
15 GB / 8-core, ~5500 frames). The engine already honors
HYPERFRAMES_EXTRACT_CACHE_DIR for relocation, but the knob was
undocumented and invisible in diagnostics — the reporter had to piece
together a 4-flag compound workaround including EXTRACT_CACHE_DIR=off.
Changes:
- Extract the env-var resolver into a public engine API
(resolveExtractCacheDir, defaultExtractCacheDir,
EXTRACT_CACHE_DIR_DISABLED_ALIASES) with a typed resolution shape
distinguishing "disabled by user" vs "default" vs "env override".
- Add a Frames-cache check to `hyperframes doctor` that reports the
effective directory, its free space, source (env or default), and
fails with a relocation hint when <2 GB free at that mount.
- Add `hyperframes render --frames-cache-dir <path>` as discoverable
CLI sugar for the env var, including the opt-out aliases
(off/none/false/0) and CWD-safe absolute-path resolution.
- Document the flag in docs/packages/cli.mdx with the field-signal
citation, and add a render example row for the Windows workflow.
- Cover both surfaces with unit tests (6 doctor cases + 4 engine
cases including all disabled-alias variants).
Refs Slack #hyperframes-cli-feedback ts=1784219488 (win32 v0.7.58).
Co-authored-by: Via <via-heygen[bot]@users.noreply.github.com>
Field-signal baseline: >=2 fallbacks/hr on darwin/arm64 from filter:blur
and filter:drop-shadow triggers. Fallback path perf is currently untimed,
so we can't know if the overhead is 10% or 10x. This PR adds opt-in
per-frame timing (HF_PROFILE_FALLBACK_CAPTURE=true) that emits p50/p95/p99
+ trigger reason via the observeRenderStage telemetry channel extended in
#2510. Diagnostic surface only -- no perf fix, no behavior change on
healthy paths.
Stack: PR #9 (final) of 9 (base via/escape-hatch-fallback-reproducer).
Signed-off-by: Via
Field signals ts=1784049136 (hardware-GPU intermittent black rectangles →
resolved with --no-browser-gpu --low-memory-mode --workers 1) and
ts=1784032286 (clip-path animated image → intermittent black rectangles →
resolved with deterministic precompose). Pattern: hardware-GPU writes
solid-black on some composition shapes; software-GPU / screenshot bypass
restores correctness. Raw per-pixel diff alone false-positives on every
compositor jitter frame; the diagnostic-grade signal is asymmetric
black-only-in-A pixels (solid-black where B has content).
Adds `packages/engine/src/utils/gpuParityDiff.ts`: pure helpers
(`diffGpuParityFrames`, `diffGpuParityPngs`, `verifyGpuParity`) that
compare two RGBA frames captured via different GPU paths, count per-pixel
diffs above a tolerance, and isolate black-only-in-A / black-only-in-B
pixel counts + bounding boxes. Symmetric black regions (real black content
present in both captures) are NOT flagged. PNG wrapper preserves the
underlying decode error as Error.cause on either side. All exposed via
`@hyperframes/engine`'s package index for downstream wiring.
19 unit tests cover identity, per-pixel tolerance, the field-bug shape,
the shared-black no-op case, bounding-box tightness across multiple
regions, the inverse pattern, dimension mismatch, data-length mismatch,
overlapping threshold rejection, custom tolerance, verdict output, PNG
end-to-end, and cause-preservation on both A and B decode failures.
Reduced-scope first pass. Wiring a `hyperframes verify-gpu-parity` CLI
command, dual-mode capture orchestration, and integration coverage against
a known-bad composition is intentionally deferred to a follow-up so the
diagnostic primitive can land and be exercised in isolation. The exported
surface is stable — a follow-up need only add the capture-and-diff driver.
Stack: PR #7 of 9 (base via/parallel-capture-observability).
Signed-off-by: Via
Field signals ts=1784019503 (heartbeat reports 0 frames during 64s
browser calibration — reads as broken but is healthy) and ts=1784042064
(1292s Windows render hard-exited during video frame extraction with
no final error string — silent worker crash).
Add calibrating/capturing state to heartbeat labels; surface synthetic
terminal error on unexpected worker exit when no explicit error was
emitted.
Stack: PR #6 of 9 (base via/overlay-count-lint).
Signed-off-by: Via <vance@heygen.com>
Field signal ts=1784146416 (darwin/arm64, CLI 0.7.58, 7/10): host
page.goto hit Navigation timeout of 60000ms twice on a CSS 3D + audio
composition; Docker rendered the same composition successfully.
Puppeteer's stock "Navigation timeout of 60000 ms exceeded" text names
none of HyperFrames' existing escape hatches, so the reporter had no
signal that the failure had knobs.
Wraps main-render Puppeteer `page.goto` errors matching
/Navigation timeout|net::ERR_TIMED_OUT/i with an augmented message that
names:
- The effective timeout currently applied (`cfg.pageNavigationTimeout`).
- Raise-the-timeout: `PRODUCER_PAGE_NAVIGATION_TIMEOUT_MS` env,
`--browser-timeout` CLI flag (seconds).
- Browser-binary escape hatch: `HYPERFRAMES_BROWSER_PATH` env.
- Field-signal shape: darwin/arm64 + CSS 3D + audio compound Docker
hint — gated on all three inputs being explicitly true; falls back
to generic hints when any input is unknown.
Mirrors #2443's HYPERFRAMES_BROWSER_PATH surfacing pattern (which
covered download-time failures) at the runtime `page.goto` layer.
Non-matching errors flow through unchanged. Original error preserved
via `err.cause`.
Wired into `renderOrchestrator.executeRenderJob`'s top-level catch,
composed after `augmentProtocolTimeoutError` so the two augmenters
never both fire on the same error (mutually exclusive regexes).
Current wire-up passes no `hasCss3D` / `hasAudio` context — no
compile-time CSS-3D signal is threaded through the render pipeline,
and `hasAudio` is block-scoped inside the try. Per the helper's
fallback docs, unknown flags route to the generic env + browser-path
hints. A future compile-time CSS-3D scan can thread both flags to
enable the full compound Docker hint without touching this helper's
signature.
Stack: PR #3 of 9 (base via/win32-streaming-encode-autodisable).
Signed-off-by: Via <vance@heygen.com>
Field signal ts=1784131903 (win32/x64, CLI 0.7.58, 156s UI-heavy):
stable ONLY with four flags together — --workers 1 --no-browser-gpu
--low-memory-mode + PRODUCER_ENABLE_STREAMING_ENCODE=false. Since
--no-browser-gpu and --low-memory-mode already imply screenshot
capture, three of the four flags are structurally coupled. Auto-detect
the compound at resolveConfig time and disable streaming-encode on
the caller's behalf; user explicit-set (PRODUCER_ENABLE_STREAMING_ENCODE
or overrides.enableStreamingEncode) always wins.
Composition duration is not known at the config layer, so the wire-up
passes compositionDurationSec:undefined and the helper reduces to the
three-condition compound (platform + softwareGpuForced + workers=1).
The 4-arg helper stays exported for downstream callers that DO know
duration (e.g., renderOrchestrator) and want the >120s guard.
Trade-off documented in code + PR body: false positives possible for
short (~<120s) Windows software-GPU single-worker renders. Mitigation
is the explicit opt-in escape hatch.
Emits a single [hyperframes] log line naming the trigger + how to opt
back in, so operators can tell an auto-disable apart from an explicit
opt-out. Adds streamingEncodeAutoDisabledOnWin32Compound internal
provenance for downstream telemetry.
Stack: PR #2 of 9 (base via/protocol-timeout-discoverability).
Signed-off-by: Via
Field signal ts=1784047847 (darwin/arm64, 8GB M1, 9 videos + 22 images):
reporter hit Runtime.callFunctionOn timeout and switched to FFmpeg
because the error didn't surface HyperFrames' existing knobs
(PRODUCER_PUPPETEER_PROTOCOL_TIMEOUT_MS env, --protocol-timeout CLI).
Wraps main-render Puppeteer errors matching /Runtime\.callFunctionOn
timed out|Target closed|protocolTimeout/i with an augmented message that
names the effective timeout, the env var, the CLI flag, and the
field-signal shape. Non-matching errors pass through unchanged
(returned as the same instance). Original error preserved via err.cause.
Also adds a dedicated --protocol-timeout row to the CLI docs Flags table
so PRODUCER_PUPPETEER_PROTOCOL_TIMEOUT_MS is discoverable via search.
Signed-off-by: Via <noreply@heygen.com>
Deepwork's request-changes on #2411 (twice): deFallbackReason's blank/psnr
split still ran /blank/i.test(err.message) even after this PR's stated goal
of moving off message-text parsing — a reworded message, a translated
string, or a differently-shaped error crossing a module boundary could
silently relabel a blank failure as psnr (or vice versa), corrupting the
soak's telemetry taxonomy.
DrawElementVerificationDetails now carries a required `kind: "blank" | "psnr"`
field, set at all three real throw sites in captureStreamingStage.ts. The
orchestrator derives deFallbackReason from getDrawElementVerificationDetails's
kind instead of regexing the message. Making `kind` a required constructor
argument means any future throw site that omits it fails to compile, closing
the gap for good rather than just at today's three call sites.
New tests in frameCapture.test.ts prove message-independence directly: kind
survives a reworded message that says neither "blank" nor "psnr", and stays
correctly "psnr" even when the message adversarially contains the substring
"blank" — the exact scenario a regex-based classifier would get wrong.
de_fallback_reason only told you the fallback happened (blank/psnr/oom/
capture_error), not the failing PSNR or frame index — that data existed as
text inside the thrown error's message and was discarded on the way to
telemetry. DrawElementVerificationError now carries structured
frameIndex/failedDb/verifyThresholdDb; the orchestrator reads them via the
new getDrawElementVerificationDetails helper instead of regexing message
text, and both telemetry surfaces (the render_complete perfSummary path and
the crash-survival RenderCaptureObservability mirror) emit
de_fallback_failed_db / de_fallback_frame_index.
Needed to distinguish "32dB vs the 32dB threshold, tune it" from "12dB real
corruption, investigate" during the parallel-router soak — currently that
distinction is invisible.
## What
- cap static-dedup verification at 15 seconds of wall-clock time
- disable the optimization and continue normal capture when the verification budget is exhausted
- add a regression that models the reported 350-second / ~8,400-frame alpha render
## Why
Static-frame verification uses full-page screenshots. Its existing screenshot-count budget still scales with composition duration, so a long composition can spend minutes proving an optimization before frame capture starts. The reported 350.35-second ProRes alpha render spent about eight minutes in this phase before safely disabling dedup.
## How
The verifier now records a deadline before seeking verification frames. It checks the deadline before every full-page capture and returns the existing fail-closed `budgetExhausted` result when time is exhausted. This keeps the existing density-based safety checks while bounding their startup cost.
## Test plan
- [x] Unit tests added/updated
- [ ] Manual testing performed
- [ ] Documentation updated (not applicable)
- [x] Focused engine test: 9/9 passed
- [x] Engine typecheck passed
- [x] Pre-commit lint, format, tracked-artifact, fallow, and typecheck gates passed
- [x] Full engine suite: 988 passed, 3 skipped; 2 pre-existing environment failures because host FFmpeg 4.2 lacks `-fps_mode`
James