Follow-up on Miga's review of #2512. The regression fixture
`escape-hatch-fatal-fallback` is tagged `field-signal-reproducer` and
`known-broken` so it's skipped from the default sweep via
`--exclude-tags transparency,field-signal-reproducer` in the
`test:regression*` scripts in packages/producer/package.json. But
`Dockerfile.test`'s ENTRYPOINT invoked the harness directly (`bunx tsx
src/regression-harness.ts -- --sequential`), bypassing those scripts —
so `bun run docker:test*` and the aws-lambda smoke tests would still
try to run the known-broken fixture and fail. CI's own regression sweep
was insulated only because it hardcodes per-shard positional test names
that don't include this fixture, but that's incidental, not by design.
Bake the exclude-tags into the Dockerfile.test ENTRYPOINT itself so
every user of the image (local `docker:test*`, aws-lambda smoke, any
adopter running the reference image) picks up the same skip contract.
Docker CMD args appended after the entrypoint (e.g. matrix shard
positional test names in .github/workflows/regression.yml, or
`--mode=distributed-simulated`) still parse correctly — the harness
applies excludeTags after testNames-filtering (see discoverTestSuites
in regression-harness.ts).
Also exports `parseArgs()` from regression-harness.ts and adds
regression-harness-parse.test.ts to pin the `--exclude-tags` comma-parse
contract, so any future change to the parser or the values baked into
the Dockerfile / package.json will trip a red test rather than silently
diverging.
Verification A (harness comma-parses `--exclude-tags transparency,
field-signal-reproducer`) already worked pre-fix; the new test file
codifies it. Verification B (Docker ENTRYPOINT propagates the same
skip) is what this commit fixes.
Signed-off-by: Via
Field-signal baseline: >=2 fallbacks/hr on darwin/arm64 from filter:blur
and filter:drop-shadow triggers. Fallback path perf is currently untimed,
so we can't know if the overhead is 10% or 10x. This PR adds opt-in
per-frame timing (HF_PROFILE_FALLBACK_CAPTURE=true) that emits p50/p95/p99
+ trigger reason via the observeRenderStage telemetry channel extended in
#2510. Diagnostic surface only -- no perf fix, no behavior change on
healthy paths.
Stack: PR #9 (final) of 9 (base via/escape-hatch-fallback-reproducer).
Signed-off-by: Via
Field signal ts=1784039841 (win32/x64, CLI 0.7.57): shifted-DOM-image-
layer bug at frame 120 reproduces with BOTH PRODUCER_FORCE_SCREENSHOT=true
AND HF_DE_PARALLEL_ROUTER=false set. First case where both known escape
hatches fail simultaneously. Standalone 1920x1080 GSAP paused timeline,
absolute PNG layers, 4 independent scenes.
Ships a skipped regression fixture to codify the shape. NOT a fix — no
root cause identified. The composition is preserved so a future
diagnostic pass has a real, checked-in repro and a proposed fix can be
validated against the same shape the field reported.
Skip mechanism (belt-and-suspenders):
1. meta.json tagged `field-signal-reproducer` and `known-broken`;
producer/package.json test:regression* scripts add the tag to
--exclude-tags alongside the existing `transparency` skip.
2. Not registered in any .github/workflows/regression.yml shard's
args, so the CI regression sweep won't pick it up either.
Un-skip when a fix lands: drop the tags from meta.json AND add the
fixture id to a shard's args in the workflow. See src/README.md for
the field-signal envelope and diagnostic starting points.
Stack: PR #8 of 9 (base via/gpu-parity-gate).
Signed-off-by: Via
Field signals ts=1784019503 (heartbeat reports 0 frames during 64s
browser calibration — reads as broken but is healthy) and ts=1784042064
(1292s Windows render hard-exited during video frame extraction with
no final error string — silent worker crash).
Add calibrating/capturing state to heartbeat labels; surface synthetic
terminal error on unexpected worker exit when no explicit error was
emitted.
Stack: PR #6 of 9 (base via/overlay-count-lint).
Signed-off-by: Via <vance@heygen.com>
Field signal ts=1784146416 (darwin/arm64, CLI 0.7.58, 7/10): host
page.goto hit Navigation timeout of 60000ms twice on a CSS 3D + audio
composition; Docker rendered the same composition successfully.
Puppeteer's stock "Navigation timeout of 60000 ms exceeded" text names
none of HyperFrames' existing escape hatches, so the reporter had no
signal that the failure had knobs.
Wraps main-render Puppeteer `page.goto` errors matching
/Navigation timeout|net::ERR_TIMED_OUT/i with an augmented message that
names:
- The effective timeout currently applied (`cfg.pageNavigationTimeout`).
- Raise-the-timeout: `PRODUCER_PAGE_NAVIGATION_TIMEOUT_MS` env,
`--browser-timeout` CLI flag (seconds).
- Browser-binary escape hatch: `HYPERFRAMES_BROWSER_PATH` env.
- Field-signal shape: darwin/arm64 + CSS 3D + audio compound Docker
hint — gated on all three inputs being explicitly true; falls back
to generic hints when any input is unknown.
Mirrors #2443's HYPERFRAMES_BROWSER_PATH surfacing pattern (which
covered download-time failures) at the runtime `page.goto` layer.
Non-matching errors flow through unchanged. Original error preserved
via `err.cause`.
Wired into `renderOrchestrator.executeRenderJob`'s top-level catch,
composed after `augmentProtocolTimeoutError` so the two augmenters
never both fire on the same error (mutually exclusive regexes).
Current wire-up passes no `hasCss3D` / `hasAudio` context — no
compile-time CSS-3D signal is threaded through the render pipeline,
and `hasAudio` is block-scoped inside the try. Per the helper's
fallback docs, unknown flags route to the generic env + browser-path
hints. A future compile-time CSS-3D scan can thread both flags to
enable the full compound Docker hint without touching this helper's
signature.
Stack: PR #3 of 9 (base via/win32-streaming-encode-autodisable).
Signed-off-by: Via <vance@heygen.com>
Field signal ts=1784047847 (darwin/arm64, 8GB M1, 9 videos + 22 images):
reporter hit Runtime.callFunctionOn timeout and switched to FFmpeg
because the error didn't surface HyperFrames' existing knobs
(PRODUCER_PUPPETEER_PROTOCOL_TIMEOUT_MS env, --protocol-timeout CLI).
Wraps main-render Puppeteer errors matching /Runtime\.callFunctionOn
timed out|Target closed|protocolTimeout/i with an augmented message that
names the effective timeout, the env var, the CLI flag, and the
field-signal shape. Non-matching errors pass through unchanged
(returned as the same instance). Original error preserved via err.cause.
Also adds a dedicated --protocol-timeout row to the CLI docs Flags table
so PRODUCER_PUPPETEER_PROTOCOL_TIMEOUT_MS is discoverable via search.
Signed-off-by: Via <noreply@heygen.com>
Review feedback on #2473: HF_DE_PARALLEL_STALL_MS had no backwards-compat
shim after the rename to HF_DE_STALL_MS, and a parent abort during a wedged
sequential capture would surface as "stalled" instead of "aborted" in
downstream logs/telemetry (functionally harmless since isCancellation is
gated on abortSignal.aborted, not message text, but misleading to read).
* fix(cli): invalidate the skills nudge cache after a successful install/update/check
The passive "N skills out of date or missing" nudge reads a 24h config
cache that only the background check (on non-skills commands) ever wrote.
The skills commands themselves are excluded from the nudge pipeline, so a
successful `skills update`/install/check never refreshed or dropped the
cached verdict — the pre-install count kept printing on every other
command for up to 24h.
Reconcile commands now drop the cached verdict (counts + timestamp) so
the next command's background check re-runs for real. The offline
presence-only path deliberately keeps the cache: that run learned nothing
about freshness.
* fix(skills): win32-safe npx spawns in media-use + accurate whisper wording
The Whisper transcribe fallback and the Kokoro local-TTS delegation both
spawned a bare "npx" via execFileSync — on Windows npx is npx.cmd, which
spawn cannot exec, so both paths died with `spawnSync npx ENOENT`. Route
them through the skill's existing resolveSpawnCommand (node + npx-cli.js
on win32, no shell:true), same as the audio engine's TTS spawns.
Also corrects the "bundled with the hyperframes CLI" claim about
whisper.cpp: it is resolved from PATH / installed via Homebrew / built
from source with git+cmake on first use, and models download from
HuggingFace — nothing whisper is shipped in the package.
* feat(skills): canonical fully-silent marker + auth status exit-code docs
product-launch's Step 3.1 gate said "or the project is marked silent"
but nothing defined how to mark one, and audio.mjs unconditionally
retrieved BGM. Define the canonical marker — `music: none` in the
storyboard's top YAML block, plus no SCRIPT.md — and honor it:
audio generate produces nothing (removing stale audio_meta.json, since
absence is what assemble treats as silent), and `music: none` with
narration keeps TTS while turning BGM off.
Also documents the `auth status` exit-code contract (exit 1 while
signed out is the normal offline state, not a failure) in the
product-launch Step 0 note and the CLI skill's cloud reference.
* fix(skills): transient-init retry for standalone animation-map and contrast-report
The standalone helpers called initializeSession exactly once, so a valid
modular project — whose sub-composition timelines register asynchronously
— could hit the readiness deadline and die with the transient
"zero duration / Runtime ready: false" diagnostic the render pipeline
retries (probeStage). Add initializeSessionWithRetry to the shared
package-loader (both byte-identical copies): close the crashed session
and retry once with a fresh browser, gated by the engine's canonical
isTransientBrowserError — now re-exported from @hyperframes/producer,
with a frozen fallback pattern list for older published packages. The
"Runtime ready: true" fast-fail (a genuine authoring bug) still fails
without a retry.
* feat(skills): extend the fully-silent marker to faceless-explainer and pr-to-video
Both workflows reuse product-launch's audio model — their Step 3.1 gates
carried the same undefined "marked silent" phrase, and their (intentionally
identical) audio.mjs copies had the same unconditional BGM retrieve. Port
the `music: none` marker handling into both copies, define the marker in
their SKILL.md Step 3.1 and story-design references, and turn the
copies' "intentionally identical" header claim into a byte-identity pin
test so the next fix can't silently miss one of them.
* test(cli): reset the prune mock explicitly instead of relying on restoreAllMocks
The converge test's toHaveBeenCalledTimes(1) held only because vitest 3's
vi.restoreAllMocks() clears vi.fn() call state; vitest 4 restores spies
only, so the count would accumulate across tests and fail. Reset
pruneOrphanedLockEntries in beforeEach like the other manifest mocks —
passes under both vitest 3.2.4 (pinned) and vitest 4.
* test(skills): close review findings — package-loader pin, whisper win32 parity, quoted-none
Review follow-ups on #2476:
- package-loader.mjs byte-identity pin (the elevated concern): the two
copies now carry initializeSessionWithRetry + FALLBACK_TRANSIENT_PATTERNS,
exactly the shared-logic shape a future fix could land in one copy and
miss in the other — same enforcement as the audio.mjs pin.
- whisper win32 call-site parity: runWhisper's npx resolution lifted into
lib/npx-sync.mjs (resolveNpxInvocation, injectable params matching the
localTtsGenerate idiom) with the same three-branch coverage as the
Kokoro site — plus the hard-fail contract (throws actionably, since the
whisper fallback has no next provider to fall through to).
- quoted music: "none" pin: the vendored storyboard parser strips matching
quotes at parse time (stripQuotes), so the silent marker already accepts
the quoted spelling — pinned so that stays true.
Deepwork's request-changes on #2411 (twice): deFallbackReason's blank/psnr
split still ran /blank/i.test(err.message) even after this PR's stated goal
of moving off message-text parsing — a reworded message, a translated
string, or a differently-shaped error crossing a module boundary could
silently relabel a blank failure as psnr (or vice versa), corrupting the
soak's telemetry taxonomy.
DrawElementVerificationDetails now carries a required `kind: "blank" | "psnr"`
field, set at all three real throw sites in captureStreamingStage.ts. The
orchestrator derives deFallbackReason from getDrawElementVerificationDetails's
kind instead of regexing the message. Making `kind` a required constructor
argument means any future throw site that omits it fails to compile, closing
the gap for good rather than just at today's three call sites.
New tests in frameCapture.test.ts prove message-independence directly: kind
survives a reworded message that says neither "blank" nor "psnr", and stays
correctly "psnr" even when the message adversarially contains the substring
"blank" — the exact scenario a regex-based classifier would get wrong.
Review feedback on #2411 (Rames): the crash-survival RenderCaptureObservability
mirror passed deFallbackFailedDb raw/unrounded while the render_complete
perfSummary path rounded to 1 decimal — the same underlying PSNR could ship two
different values to PostHog depending on which event fired. Extracted the
existing inline round/clamp expression (previously duplicated for verifyMinDb
and fallbackFailedDb) into a shared roundDb helper, applied once at the single
point deFallbackFailedDb is derived from the thrown error so both downstream
consumers agree.
Also threads verifyThresholdDb (captured on the error but never propagated,
per the nit) through DrawElementPerfInput/RenderCaptureObservability/render.ts/
telemetry as de_fallback_threshold_db on both events — the HF_DE_VERIFY_MIN_DB
value the failing dB breached, letting ops read "28.4dB failed a 32dB
threshold" directly instead of cross-referencing config.
de_fallback_reason only told you the fallback happened (blank/psnr/oom/
capture_error), not the failing PSNR or frame index — that data existed as
text inside the thrown error's message and was discarded on the way to
telemetry. DrawElementVerificationError now carries structured
frameIndex/failedDb/verifyThresholdDb; the orchestrator reads them via the
new getDrawElementVerificationDetails helper instead of regexing message
text, and both telemetry surfaces (the render_complete perfSummary path and
the crash-survival RenderCaptureObservability mirror) emit
de_fallback_failed_db / de_fallback_frame_index.
Needed to distinguish "32dB vs the 32dB threshold, tune it" from "12dB real
corruption, investigate" during the parallel-router soak — currently that
distinction is invisible.
* fix(fonts): subset Google Fonts to composition text
* fix(producer): skip slow TTC recompression
* fix(fonts): include decoded composition text in subsets
* fix(producer): pass variables to duration probe
* fix(producer): tolerate rounded frame-boundary durations
* fix(cli): resolve relative data-start references in composition duration
`compositions --json` computed each timed child's start with a bare
parseFloat(data-start ?? "0") in parseCompositions (host duration) and
parseSubComposition (sub-comp duration). A relative reference like
data-start="s1" ("start when clip s1 ends") is not numeric, so parseFloat
returned NaN and that clip's contribution to the max-end was silently
dropped — a host with two 3s clips (2nd data-start="s1") reported duration 3
instead of 6, breaking compositions/inspect/snapshot for composition-clip
relative timing.
Resolve relative references the same way the extractor does (parseStartExpression
from @hyperframes/core + a findReferenceTargetEl/resolveReferencedStart port,
since the engine's referenceResolver isn't a public export across the package
boundary). Verified: host duration now 6; 3 tests pass.
(Implemented via Codex; verified independently.)
Miguel R4 blocker on #2359: my R3 fix at renderOrchestrator only updated
the observability copy, leaving the authoritative captureForceScreenshot
local at compileResult.forceScreenshot (false for auto→software). The
frameCapture side clamped its own local and correctly routed screenshot,
but downstream orchestrator code overwrote observability back to
beginframe from the still-false local at two sites:
- Parallel-stream label at renderOrchestrator.ts:2293 mis-labelled the
stream as 'beginframe' when actual capture was 'screenshot'.
- capture_strategy telemetry at renderOrchestrator.ts:2440-2450
overwrote the earlier observability correction, so the final
captureMode observation flipped back to 'beginframe' while the
engine actually captured screenshot.
Fix: extract the clamp into a caller-facing helper
applyConcreteGpuScreenshotClamp(current, resolvedGpuMode, cfg) that
returns the (possibly-promoted) new boolean. Callers assign it back to
their authoritative local, so routing + telemetry + strategy code read
one value.
Changes:
- packages/engine/src/config.ts: new exported
applyConcreteGpuScreenshotClamp; delegates to
shouldClampToScreenshotForConcreteGpu but computes the caller's
final value, not just the clamp decision. Reads the programmatic
opt-out from cfg.forceScreenshotExplicitlyOptedOut. Idempotent on
already-true input.
- packages/engine/src/index.ts: export the new helper.
- packages/engine/src/services/frameCapture.ts: replace the inline
OR expression with applyConcreteGpuScreenshotClamp.
- packages/producer/src/services/renderOrchestrator.ts: assign result
into the AUTHORITATIVE captureForceScreenshot local (was updating
only observability). Downstream parallel-stream label at :2293 and
capture_strategy telemetry at :2440-2450 now read the corrected
value.
Tests: 6 new caller-level cases for applyConcreteGpuScreenshotClamp
covering the exact matrix Miguel called out:
- resolved software + default false → promotes to true (screenshot)
- resolved software + programmatic opt-out → stays false (BeginFrame)
- resolved hardware + default false → stays false
- resolved software + already-true → stays true (idempotent)
- resolved software + env PRODUCER_FORCE_SCREENSHOT=false → stays false
- resolved software + undefined cfg → promotes to true (frameCapture path)
Local: 67/67 engine config tests pass (was 61). oxfmt clean.
Miguel R3 blocker on #2359: the runtime helper only checked the env opt-out
(PRODUCER_FORCE_SCREENSHOT=false), silently defeating the documented
programmatic escape hatch (overrides.forceScreenshot === false) on the
browserGpuMode:'auto' → software probe path. At the concrete-resolution
site the boolean forceScreenshot === false is ambiguous between default
and explicit opt-out — resolveConfig sees the provenance but the runtime
helper does not.
Fix: persist provenance on the resolved config.
- New INTERNAL EngineConfig field forceScreenshotExplicitlyOptedOut, set
by resolveConfig when EITHER env or programmatic explicit-false is
present. Purpose-documented in the type as 'not intended to be set by
callers'.
- shouldClampToScreenshotForConcreteGpu gains an opts.programmaticOptOut
parameter; returns false early when set. Env stays as the third arg
(backward compatibility with existing tests).
- frameCapture.ts and renderOrchestrator.ts pass
config.forceScreenshotExplicitlyOptedOut through at both call sites, so
the auto→software probe path preserves the same escape hatches as
literal browserGpuMode:'software'.
New tests: 5 additional cases across the helper (programmatic opt-out
alone; programmatic beats missing env) and resolveConfig provenance
(programmatic sets flag; env sets flag; neither leaves it undefined).
Local: 61/61 engine config tests pass (was 56).
Addresses Miguel's R1 blockers:
1. `browserGpuMode: "auto"` that runtime-probes to software slipped past the
`resolveConfig` clamp — that clamp only sees the pre-resolve string. Add
`shouldClampToScreenshotForConcreteGpu(resolvedGpuMode, currentForceScreenshot, env)`
in `packages/engine/src/config.ts` and apply it at BOTH concrete-resolution
sites:
- `packages/engine/src/services/frameCapture.ts`: downgrades `preMode`
from "beginframe" to "screenshot" when resolved GPU is software (respects
`PRODUCER_FORCE_SCREENSHOT=false` env opt-out), fixing the routing.
- `packages/producer/src/services/renderOrchestrator.ts`: updates
`captureObservability.forceScreenshot` (and thus `captureMode`) at the
same call site, fixing the observability truth on the auto → software
case.
2. New unit tests in `config.test.ts`:
- Documents the auto-branch gap (resolveConfig leaves auto as
forceScreenshot=false — the runtime companion closes it).
- 5 branch tests on `shouldClampToScreenshotForConcreteGpu` covering
software / hardware / already-forced / env-opt-out / non-"false" env
values.
Full suite: 56/56 pass.
Scope narrowing on Blocker 2: the distributed rendering path at
`packages/producer/src/services/distributed/plan.ts:753-754` and
`renderChunk.ts:462-466` explicitly hardcodes `browserGpuMode:"software",
forceScreenshot:false` post-resolveConfig and stays outside this PR's
invariant boundary. `compileStage` may still flip it to true for alpha
formats, but generic MP4 distributed renders on SwiftShader hosts remain
BeginFrame. That's a separate architectural cleanup (needs its own
behavior-change trace); the PR body now scopes the invariant to the
in-process CLI/orchestrator path.
Miguel Ángel