ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-12 07:09:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(render): make WebGL video textures deterministic in headless render (#1403)

Browse Source 
* fix(render): make WebGL video textures deterministic in headless render

WebGL compositions that sample a `<video>` as a texture (e.g. a faceted
crystal with clips mapped onto its facets) rendered with flickering,
non-deterministic facets: a video would intermittently show a stale frame or
go black, and the same frame differed between two renders.

Two gaps caused this:

1. No WebGL analog of the WebGPU `patchVideoTextureCompat`. Chrome's headless
   compositor can't feed decoded `<video>` frames to the GPU, so the engine
   injects a decoded `<img class="__render_frame__">` sibling per video each
   frame. The WebGPU `copyExternalImageToTexture` path substitutes it, but
   `texImage2D` / `texSubImage2D` did not — so WebGL uploaded a stale/black
   frame. Add `patchWebGLVideoTextureCompat()` mirroring the WebGPU patch
   (shared `resolveRenderFrameImage` helper).

2. Capture ordering. Per frame the runtime seeks (GPU adapters render on
   `hf-seek`) BEFORE the engine injects the decoded frames, so the GPU render
   read a frame that didn't exist yet. After injecting, the engine now calls
   `window.__hfReseekGpu(t)` — a force-dispatch (`forceDispatchSeekEvent`) that
   bypasses the same-time `hf-seek` dedup — so GPU compositions re-upload their
   textures from the freshly-injected, decoded frames, deterministically.

Tests: unit tests for the texImage2D/texSubImage2D substitution and the
force-dispatch, plus a videoFrameInjector regression test asserting the
post-injection GPU reseek fires only when frames were injected. Verified
end-to-end: a WebGL prism with 8 live <video> facets renders byte-identical
across independent runs with no facet flicker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(render): add producer render-compat regression for WebGL video textures

A WebGL2 canvas samples a <video> as a texture every hf-seek (the natural
author pattern, distilled from the HeyGen prism). The render-compat harness
renders it and compares against the golden: with the video-texture fix the
render reproduces the decoded frames; revert the fix and the canvas renders
black, collapsing the comparison.

Golden verified to contain real, time-varying video content (not black), so a
regression is caught rather than passing vacuously.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
James Russo
2026-06-12 22:37:00 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 6364281ba0
commit d580f2a1d8
13 changed files with 585 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/core/src/runtime/window.d.ts
Vendored
+7
View File
@@ -45,6 +45,13 @@ declare global {
* imperative push signal: `window.addEventListener("hf-seek", e => render(e.detail.time))`.
*/
__hfTypegpuTime?: number;
/**
* Re-render GPU adapters (Three.js / WebGPU) at the given time, bypassing
* the `"hf-seek"` dedup. Called by the engine after injecting decoded
* video frames so GPU compositions re-upload their video textures from the
* freshly-injected `__render_frame__` images. See `forceDispatchSeekEvent`.
*/
__hfReseekGpu?: (time: number) => void;
__HF_PICKER_API?: HyperframePickerApi;
gsap?: {
timeline: (params?: { paused?: boolean }) => RuntimeTimelineLike;