mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-15 13:58:20 +00:00
fix(core): preload image decodes to eliminate jerky preview (GH #317)
Reported: compositions with many images stutter visibly during the
first ~second of preview playback, with the only workaround being to
render the composition to mp4 and watch the output file.
Root cause: every `<img>` that has never been painted before decodes
synchronously on the main thread the first time it enters the paint
tree. With a composition of several dozen images, that's several dozen
main-thread stalls stacked inside the first paint window — visible as
jank on every seek and the first few seconds of play.
Fix: a new `packages/core/src/runtime/imageDecode.ts` module that the
runtime binds alongside `bindMediaMetadataListeners`. For every `<img>`
in the document, it:
- Sets `decoding="async"` so the browser never falls back to sync-decode
on subsequent paints.
- Calls `img.decode()` off the main thread as soon as the image has a
`currentSrc` (either immediately if already loaded, or on the next
`load` event otherwise).
- Tracks bound images in a Set so repeated polling from the runtime's
observation loop is free after the first bind.
`img.decode()` is the canonical platform primitive for this — it's a
no-op for already-decoded images, runs on the decoder thread, and
resolves when the pixels are in the bitmap cache. Decode rejections
(CORS, corrupt file, 404) are intentionally swallowed because the
runtime's existing asset-error diagnostic pipeline already surfaces
those via the `error` event.
Lazy images (`loading="lazy"`) are intentionally NOT force-loaded — if
the author opted them out, we respect that; they'll decode when they're
needed.
The module is wired in three call sites in `init.ts`:
- Initial boot (after `runAdapters("discover")`)
- Post-load of external/inline compositions
- Every 10th poll tick (catches DOM changes from author scripts,
sub-composition mounts, and dynamically-inserted content)
Tests: `imageDecode.test.ts` covers nine cases — async-flag set,
immediate vs. load-deferred decode, idempotence, torn-down no-op,
swallowed rejections, missing currentSrc, multiple imgs in one pass,
and dynamically-added imgs across bind calls. 9/9 pass.
Docs: new "Preloading and decoding" subsection in `core.mdx` under
the Runtime section, documents the invariant for future contributors.
Closes #317.
This commit is contained in:
@@ -353,6 +353,25 @@ The pre-built runtime IIFE is available as a direct import:
|
||||
import runtime from '@hyperframes/core/runtime';
|
||||
```
|
||||
|
||||
### Preloading and decoding
|
||||
|
||||
The runtime eagerly prepares every asset in the composition so the first
|
||||
paint and the first play call never stall the compositor.
|
||||
|
||||
- **Media** — every `<video>` and `<audio>` has `preload="auto"` set and
|
||||
`.load()` kicked on bind. The first `play()` lands on buffered data
|
||||
instead of triggering a network fetch + decoder spin-up.
|
||||
- **Images** — every `<img>` gets `decoding="async"` and
|
||||
`img.decode()` invoked off the main thread. Compositions with many
|
||||
images (several dozen through several hundred) otherwise stutter
|
||||
during the first ~second of preview because each image decodes
|
||||
synchronously the first time it enters the paint tree (see
|
||||
[GH #317](https://github.com/heygen-com/hyperframes/issues/317)).
|
||||
|
||||
Both binders are **idempotent** — the runtime re-runs them periodically
|
||||
as sub-compositions mount and DOM changes land, and they skip elements
|
||||
they've already processed.
|
||||
|
||||
## Frame Adapters
|
||||
|
||||
The core package defines the [Frame Adapter](/concepts/frame-adapters) interface and provides the built-in GSAP adapter:
|
||||
|
||||
Reference in New Issue
Block a user