mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
perf(producer): skip wasted Chrome media work + injector page.evaluates during render (#1651)
## What Two small render-side improvements for video-heavy compositions: 1. **`packages/core/src/runtime/media.ts`** — gate the per-tick `el.currentTime = relTime` set + the `el.load()` drift-recovery retry on the *absence* of a `<img id="__render_frame_<id>__">` sibling (i.e., we're in render mode + this video's visual is bypassed by frame injection + its audio is mixed by ffmpeg from source files). 2. **`packages/engine/src/services/videoFrameInjector.ts`** — probe `window.__hfReseekGpu` and `window.__hf.colorGrading.redraw` once at the first injector call; cache the booleans; skip the per-frame `page.evaluate` round-trips when neither capability is registered. ## Why #### media.ts During render the runtime calls `el.currentTime = relTime` on every active video per sync tick. For frame-injected videos that's pure waste: - The visual comes from the `<img id="__render_frame_<id>__">` sibling injected by the producer's `videoFrameInjector` — the `<video>` element is `visibility: hidden`. - Audio is mixed by ffmpeg from the source files in `runAudioStage` (separate stage) — it never goes through the in-browser audio pipeline during render. So every per-tick seek just kicks Chrome's media pipeline (buffering checks, range fetches, decoder state changes) for no visible or audible benefit. On a 30 × 32 MB synth comp, that's ~2,400 wasted seeks per render — and the cost wasn't on the JS critical path, so it didn't show up in `avgBeforeCapture` directly. It bled into the BeginFrame compositor's per-frame screenshot time. Preview is unaffected: the injection sibling only exists during render. In preview `hasInjectionSibling` is always false → existing seek path runs unchanged. #### videoFrameInjector.ts The injector hook ran `__hfReseekGpu` and `redrawRuntimeColorGrading` via `page.evaluate` on every render frame. For comps that don't register either capability (the common case — anything without WebGL/WebGPU video sub-comps or a color-grading layer), each was a no-op page-side function preceded by a ~CDP-round-trip-worth of overhead. Probing once and caching `false` eliminates that for the rest of the render. ## How was this validated Stress shape: `synth-30-heavy` — 30 × 32 MB MP4 / 3 s each, sequenced end-to-end over a 90 s timeline (`data-composition-id` root + per-video `<video id="vid-NN" data-start data-duration data-track-index>`). Host: 8-core / 30 GB Linux. N=3 baseline against stock `origin/main` (post-#1630), N=3 with-fix on the same machine, same corpus, fresh worker pool each run. Phase timings via `[Render:trace]` JSON; per-frame sub-breakdown via a one-line `[CapturePerf]` stderr emit (kept locally, not in this PR — `dedupPerfs` already carries the data, this branch surfaces it). | | Baseline N=3 | With-fix N=3 | Δ | |---|---|---|---| | wall mean | 119.5 s ± 1.4 s | **117.3 s ± 0.9 s** | **-2.2 s (-1.8%)** | | avg screenshot / frame | 50.0 ms | **49.0 ms** | -2.0% | | avg beforeCapture / frame | 13.0 ms | **12.1 ms** | -7.0% | | avg total / frame | 66.0 ms | 63.9 ms | -3.2% | | output md5 | `5a22be64...` | identical ×3 | ✓ | The 1 ms screenshot drop is the load-bearing signal: it confirms the kicked Chrome media-pipeline work *was* bleeding into BeginFrame compositor time, even though it wasn't on the JS critical path. Per-frame budget improved 2.1 ms × 2700 / 3 workers ≈ 1.9 s of `capture_disk` savings, which matches the observed wall delta. This stacks cleanly with #1630 (which removed the injector's fileServer contention). #1630 moved the injector's PNG fetches off the fileServer's hot path; this PR keeps Chrome's media pipeline quiet during render so the BeginFrame compositor runs unhindered. ## Test plan - [x] Local-CLI render on `synth-30-heavy` × N=3 baseline + N=3 with-fix; wall, per-frame, md5 captured (above). - [x] Lint / format / typecheck via lefthook pre-commit (`oxlint`, `oxfmt`, `fallow audit`, `tsc --noEmit` across `@hyperframes/core` + `@hyperframes/engine` + `@hyperframes/producer`). - [ ] *Real-world video-heavy comp validation* — would love a Magi / Miga eye on a HF-heygen-stripe-shape or a Rahino-shape comp to confirm there's no audible artifact on unmuted videos. The change shouldn't affect them — in render mode the audio path is ffmpeg, not the in-browser pipeline — but a sanity-check render is cheap. ## Scope notes - *Not addressed in this PR*: the user-facing request for an upfront-extract concurrency cap (`Promise.all` in `extractAllVideoFrames` is currently unbounded across all videos). Filing as a follow-up PR — different layer of the pipeline, different user surface (CLI flag), worth keeping separate for review. - *Edge case*: in the calibration test-frame phase, the injection sibling may not yet exist when drift recovery first checks a video at the very start of its active window. The gate correctly defaults to "no sibling → run the seek" in that case, which is the existing behavior. _Authored by Jerrai (Rames team)._
This commit is contained in:
@@ -290,18 +290,33 @@ export function syncRuntimeMedia(params: {
|
|||||||
}
|
}
|
||||||
const forceSync = !isPlayingVideo && params.forceSync && drift > 0.02;
|
const forceSync = !isPlayingVideo && params.forceSync && drift > 0.02;
|
||||||
if (hardSync || strictSync || forceSync) {
|
if (hardSync || strictSync || forceSync) {
|
||||||
try {
|
// Skip the per-tick seek (and the `el.load()` drift-recovery retry
|
||||||
el.currentTime = relTime;
|
// below) for `<video>` elements that have a sibling
|
||||||
} catch (err) {
|
// `<img id="__render_frame_<id>__">`. The sibling is created only
|
||||||
swallow("runtime.media.site2", err);
|
// by the producer's frame-injection pipeline during render — its
|
||||||
}
|
// presence means the visual is painted from the `<img>` and the
|
||||||
if (Math.abs(el.currentTime - relTime) > 0.5 && !seekLoadRetried.has(el)) {
|
// `<video>` is `visibility: hidden`. Audio is mixed by ffmpeg from
|
||||||
seekLoadRetried.add(el);
|
// source files in `runAudioStage`, never via Chrome's in-browser
|
||||||
el.load();
|
// audio path. So the `<video>`'s `currentTime` has no observable
|
||||||
|
// effect during render, and the per-tick set just kicks Chrome's
|
||||||
|
// media pipeline for nothing. Preview is unaffected (the sibling
|
||||||
|
// only exists during render).
|
||||||
|
const skipForInjectedVideo =
|
||||||
|
el.tagName === "VIDEO" && el.id && !!document.getElementById(`__render_frame_${el.id}__`);
|
||||||
|
if (!skipForInjectedVideo) {
|
||||||
try {
|
try {
|
||||||
el.currentTime = relTime;
|
el.currentTime = relTime;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
swallow("runtime.media.site3", err);
|
swallow("runtime.media.site2", err);
|
||||||
|
}
|
||||||
|
if (Math.abs(el.currentTime - relTime) > 0.5 && !seekLoadRetried.has(el)) {
|
||||||
|
seekLoadRetried.add(el);
|
||||||
|
el.load();
|
||||||
|
try {
|
||||||
|
el.currentTime = relTime;
|
||||||
|
} catch (err) {
|
||||||
|
swallow("runtime.media.site3", err);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
playRequested.delete(el);
|
playRequested.delete(el);
|
||||||
|
|||||||
Reference in New Issue
Block a user