From b651a9d218b6a1d2649bfb7d8595ea83a04e69a5 Mon Sep 17 00:00:00 2001
From: James Russo <james.russo@heygen.com>
Date: Mon, 22 Jun 2026 17:12:28 -0700
Subject: [PATCH] perf(producer): skip wasted Chrome media work + injector
 page.evaluates during render (#1651)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## What

Two small render-side improvements for video-heavy compositions:

1. **`packages/core/src/runtime/media.ts`** — gate the per-tick `el.currentTime = relTime` set + the `el.load()` drift-recovery retry on the *absence* of a `<img id="__render_frame_<id>__">` sibling (i.e., we're in render mode + this video's visual is bypassed by frame injection + its audio is mixed by ffmpeg from source files).

2. **`packages/engine/src/services/videoFrameInjector.ts`** — probe `window.__hfReseekGpu` and `window.__hf.colorGrading.redraw` once at the first injector call; cache the booleans; skip the per-frame `page.evaluate` round-trips when neither capability is registered.

## Why

#### media.ts

During render the runtime calls `el.currentTime = relTime` on every active video per sync tick. For frame-injected videos that's pure waste:
- The visual comes from the `<img id="__render_frame_<id>__">` sibling injected by the producer's `videoFrameInjector` — the `<video>` element is `visibility: hidden`.
- Audio is mixed by ffmpeg from the source files in `runAudioStage` (separate stage) — it never goes through the in-browser audio pipeline during render.

So every per-tick seek just kicks Chrome's media pipeline (buffering checks, range fetches, decoder state changes) for no visible or audible benefit. On a 30 × 32 MB synth comp, that's ~2,400 wasted seeks per render — and the cost wasn't on the JS critical path, so it didn't show up in `avgBeforeCapture` directly. It bled into the BeginFrame compositor's per-frame screenshot time.

Preview is unaffected: the injection sibling only exists during render. In preview `hasInjectionSibling` is always false → existing seek path runs unchanged.

#### videoFrameInjector.ts

The injector hook ran `__hfReseekGpu` and `redrawRuntimeColorGrading` via `page.evaluate` on every render frame. For comps that don't register either capability (the common case — anything without WebGL/WebGPU video sub-comps or a color-grading layer), each was a no-op page-side function preceded by a ~CDP-round-trip-worth of overhead. Probing once and caching `false` eliminates that for the rest of the render.

## How was this validated

Stress shape: `synth-30-heavy` — 30 × 32 MB MP4 / 3 s each, sequenced end-to-end over a 90 s timeline (`data-composition-id` root + per-video `<video id="vid-NN" data-start data-duration data-track-index>`). Host: 8-core / 30 GB Linux.

N=3 baseline against stock `origin/main` (post-#1630), N=3 with-fix on the same machine, same corpus, fresh worker pool each run. Phase timings via `[Render:trace]` JSON; per-frame sub-breakdown via a one-line `[CapturePerf]` stderr emit (kept locally, not in this PR — `dedupPerfs` already carries the data, this branch surfaces it).

|  | Baseline N=3 | With-fix N=3 | Δ |
|---|---|---|---|
| wall mean | 119.5 s ± 1.4 s | **117.3 s ± 0.9 s** | **-2.2 s (-1.8%)** |
| avg screenshot / frame | 50.0 ms | **49.0 ms** | -2.0% |
| avg beforeCapture / frame | 13.0 ms | **12.1 ms** | -7.0% |
| avg total / frame | 66.0 ms | 63.9 ms | -3.2% |
| output md5 | `5a22be64...` | identical ×3 | ✓ |

The 1 ms screenshot drop is the load-bearing signal: it confirms the kicked Chrome media-pipeline work *was* bleeding into BeginFrame compositor time, even though it wasn't on the JS critical path. Per-frame budget improved 2.1 ms × 2700 / 3 workers ≈ 1.9 s of `capture_disk` savings, which matches the observed wall delta.

This stacks cleanly with #1630 (which removed the injector's fileServer contention). #1630 moved the injector's PNG fetches off the fileServer's hot path; this PR keeps Chrome's media pipeline quiet during render so the BeginFrame compositor runs unhindered.

## Test plan

- [x] Local-CLI render on `synth-30-heavy` × N=3 baseline + N=3 with-fix; wall, per-frame, md5 captured (above).
- [x] Lint / format / typecheck via lefthook pre-commit (`oxlint`, `oxfmt`, `fallow audit`, `tsc --noEmit` across `@hyperframes/core` + `@hyperframes/engine` + `@hyperframes/producer`).
- [ ] *Real-world video-heavy comp validation* — would love a Magi / Miga eye on a HF-heygen-stripe-shape or a Rahino-shape comp to confirm there's no audible artifact on unmuted videos. The change shouldn't affect them — in render mode the audio path is ffmpeg, not the in-browser pipeline — but a sanity-check render is cheap.

## Scope notes

- *Not addressed in this PR*: the user-facing request for an upfront-extract concurrency cap (`Promise.all` in `extractAllVideoFrames` is currently unbounded across all videos). Filing as a follow-up PR — different layer of the pipeline, different user surface (CLI flag), worth keeping separate for review.
- *Edge case*: in the calibration test-frame phase, the injection sibling may not yet exist when drift recovery first checks a video at the very start of its active window. The gate correctly defaults to "no sibling → run the seek" in that case, which is the existing behavior.

_Authored by Jerrai (Rames team)._
---
 packages/core/src/runtime/media.ts | 33 ++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 9 deletions(-)

diff --git a/packages/core/src/runtime/media.ts b/packages/core/src/runtime/media.ts
index bb67ab9d9..aa8e1b71d 100644
--- a/packages/core/src/runtime/media.ts
+++ b/packages/core/src/runtime/media.ts
@@ -290,18 +290,33 @@ export function syncRuntimeMedia(params: {
       }
       const forceSync = !isPlayingVideo && params.forceSync && drift > 0.02;
       if (hardSync || strictSync || forceSync) {
-        try {
-          el.currentTime = relTime;
-        } catch (err) {
-          swallow("runtime.media.site2", err);
-        }
-        if (Math.abs(el.currentTime - relTime) > 0.5 && !seekLoadRetried.has(el)) {
-          seekLoadRetried.add(el);
-          el.load();
+        // Skip the per-tick seek (and the `el.load()` drift-recovery retry
+        // below) for `<video>` elements that have a sibling
+        // `<img id="__render_frame_<id>__">`. The sibling is created only
+        // by the producer's frame-injection pipeline during render — its
+        // presence means the visual is painted from the `<img>` and the
+        // `<video>` is `visibility: hidden`. Audio is mixed by ffmpeg from
+        // source files in `runAudioStage`, never via Chrome's in-browser
+        // audio path. So the `<video>`'s `currentTime` has no observable
+        // effect during render, and the per-tick set just kicks Chrome's
+        // media pipeline for nothing. Preview is unaffected (the sibling
+        // only exists during render).
+        const skipForInjectedVideo =
+          el.tagName === "VIDEO" && el.id && !!document.getElementById(`__render_frame_${el.id}__`);
+        if (!skipForInjectedVideo) {
           try {
             el.currentTime = relTime;
           } catch (err) {
-            swallow("runtime.media.site3", err);
+            swallow("runtime.media.site2", err);
+          }
+          if (Math.abs(el.currentTime - relTime) > 0.5 && !seekLoadRetried.has(el)) {
+            seekLoadRetried.add(el);
+            el.load();
+            try {
+              el.currentTime = relTime;
+            } catch (err) {
+              swallow("runtime.media.site3", err);
+            }
           }
         }
         playRequested.delete(el);