mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-19 11:26:02 +00:00
38d9aeb5ea1fa47a09d53c2db6df8079986b8a43
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d05bc4215d |
perf(core): preview runtime scales with what plays, not with the film's clip count (#5674)
* perf(core): probe media volume with one timeline walk per pass, not one per element Each bind, rebind and scene-swap pass probes every media element for volume automation, and each probe flattened the whole captured timeline (getChildren(true, true, false)) to ask whether any tween could move that one element's volume. A pass cost elements x tweens. Elements with no automation are never cached, so the cost came back on every rebind pass. On a two-hour podcast episode (2,354 media clips, ~100k caption tweens) that was several seconds of preview main thread per pass (CPU profile of the project's open in Desktop: IE/PE -> getChildren, ~6 s). volumeMoversOf() now walks the tweens once and lists the elements a tween names volume on, plus whether any non-DOM setter is tweened. Each pass builds it lazily and shares it with every probe; a probe without one walks as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(core): find the master bus from the first composition root The top-level root is the first [data-composition-id] in document order, so one match replaces a scan of every composition node, read once per clip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(core): lay out a clip again only when its layout inputs change applyClipLayout ran on every timeline post and, for every root clip, lifted display:none, read computed style and restored it. A clip whose inline layout, class and data-width/height are as last laid out gets the same answer, so it is skipped. On a 2,354-clip episode that was 4 of every 10 s of playback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(core): a clip is laid out again only when its layout inputs change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(core): play a long film without re-walking it every frame - The visibility pass skips root-level media far from the playhead during steady playback: hidden last pass, not loading or armed, and its window (widened by every look-ahead) clear of the span played and the loop start. Seeks, forced syncs and renders visit every node. - The GSAP re-render saves and restores only the landing marks near the playhead, placed once per timeline structure; the zero-duration-callback check is cached the same way. A caption track of 38k sets was walked twice a frame. - Registered timeline hosts, the root composition and the per-frame whole-film queries ([data-start], audio groups, timed audio) are kept for one timing revision; a render reads fresh. - The runtime's own media release (src reset) and load events of clips with an authored length no longer count as timing changes, which re-posted the timeline about twice a second. - The timeline payload resolves a stacking context once per parent. 2,354-clip episode in Desktop: playback 15 -> 67-89 fps (unminified runtime). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(core): check each element once per batch of raster-hint records, and cache the root from init - A script building a large tree (a caption track) made each attribute record re-walk its subtree with getComputedStyle; a batch now checks each element once, an element whose own attributes changed still reading its own hints. - The root composition cache installs the timing observer itself: a root with a declared length never reads the duration floors that installed it, so init looked the root up once per media clip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(core): type the playback visibility skip for any element, and cache clip layout only once loaded The skip's predicate took an HTMLElement where the pass hands it a StylableElement (tsconfig.runtime.json). The layout cache records a clip only once the document has loaded, so a pass before stylesheets apply is redone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(core): realm-safe element checks in the timing caches; cut added comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4ed269ad1e |
perf(core): playback housekeeping runs on a timer, and rebinds stop re-sampling silent media (#4595)
* perf(core): playback housekeeping runs on a timer, and rebinds stop re-sampling silent media
While playing, the transport re-posted the clip manifest every 20 frames and
re-bound media every 30, frame-counted, so a 120 Hz display did both twice as
often for a film that was not changing. Playing, those jobs now run when the
composition changes (rate-limited as before) and on a one-second poll for
what nothing can push.
A rebind re-probed every media element's volume by seeking the whole timeline
60 times per second of clip, about 1,500 seeks and 50-140 ms in one frame.
The probe seeks with events suppressed, so only a tween on the element that
names volume can move it; an element no tween touches is now answered from
the tweens without seeking.
* fix(core): a data-width or data-height change applies while playing and parked
postTimeline applies the root's sizing, and it now runs when the composition
revision moves. The revision only watched timing attributes, so a size edit
waited for a pause while playing, and on main it was missed while parked
too. The observer now also watches data-width and data-height. The tick
comment is trimmed to three lines, the tick reuses its playing flag, and the
cadence test is renamed to what it checks now.
* test(core): share one 120 Hz frame step in the transport park tests
The two playing tests each defined the same frame helper; one frame120Hz
now sits beside settle and all four loops use it. No source change.
* refactor(core): drop the playing poll doc comments
The comment ratchet does not let a file's comment share rise; the reason
lives in the PR body.
* fix(core): a playing clip's track, label and z-index reach Studio again
The playing path had dropped the periodic manifest post, and the
composition observer does not see data-track-index, data-timeline-label or
an inline z-index on an existing clip (watching style would fire on every
GSAP frame). On the playing poll the runtime now builds the manifest and
posts it only when it differs from the last one posted, so an edit lands
within a second and a steady film still posts nothing.
* fix(core): keep sampling volume when a tween drives a non-DOM object
A tween on a plain-object proxy whose setter writes the element's volume
names neither the element nor volume, so the fast skip dropped it and
WebAudio-routed preview played the static data-volume (0 where the film
asks for 0.375). The skip now applies only when every tween targets DOM
nodes; any other target falls back to the full volume probe, as on main.
* fix(core): skip the volume probe again for spacers, calls and counters
The non-DOM fallback fired on any plain-object or function target, so a
tl.to({}, {duration}) spacer, a tl.call or a tweened counter brought back the
full probe on ordinary films. It now falls back only when a tweened property
resolves to a setter on the target or its prototype chain, which is what a
gain proxy or a class with a volume setter uses; data properties skip.
* fix(core): post a clip edit made while playing when playback stops
A label, track or inline z-index edited while playing and then paused before
the next one-second poll was never posted: once parked, nothing watches it.
The tick where playback stops now runs the same manifest diff once and posts
only if the manifest changed.
|
||
|
|
73dfe35e46 |
fix(preview): show the correct scene instead of every scene at once on some loads (#3848)
* fix(runtime): show the right scene when the preview DOM comes from another window The Studio preview sometimes builds the composition body in the editor window and adopts it into the preview frame's document. Adopted nodes keep the prototypes of the realm that created them, so `node instanceof HTMLElement` is false for every element in the composition even though the elements are ordinary HTML sitting in that document. Every guard in the runtime written as `if (!(node instanceof HTMLElement))` then skipped the whole document, silently: nothing threw, nothing logged, and readiness still reported success. The timed-element visibility pass wrote no inline visibility at all, so on those loads the preview painted every scene on top of every other and the editor drew off-canvas markers for elements the user could not see. The auto-stamp pass stopped stamping too, which is why the composition came up one timeline clip short. Replace every realm-sensitive element check in the runtime with structural predicates that ask what a node IS (node type, namespace, tag name) rather than which window's constructor made it. Two local `doc.defaultView.HTMLElement` workarounds are deleted with it: they fix only the case where the nodes belong to the document's own realm, and adoption is exactly what breaks that. * fix(studio): scrub the music track the timeline named, not the first audio `resolveScrubAudioEl` tested `byId instanceof HTMLAudioElement` on a node from the preview iframe's document. That node is an instance of the IFRAME's `HTMLAudioElement`, never this module's, so the check was false on every load and the `musicId` hint was dead. Scrub fell through to the first `<audio>` in the document, which the comment right above it warns can be the voiceover, so dragging the playhead could preview the wrong track. Ask what the node is instead. Also close the gaps an independent review found in the runtime fix: - the cross-realm predicate test had no `<audio>` and no `<img>`, so reverting `isAudioElement` or `isImageElement` to `instanceof` left it green. It no longer does, and the audio case also pins `isMediaElement`, which composes from it and gates the media sync path. - nothing stopped the runtime regressing. `lint-runtime-preview-guards.ts` gains a second check kind: patterns that must be ABSENT under a directory, seeded with DOM-typed `instanceof` under `src/runtime`, pointing at domRealm.ts. Comment lines and tests are exempt, both on purpose. - domRealm.ts stated the adoption mechanism as settled fact. The mixed prototypes are measured; how the nodes get into the frame is not identified, and the docstring now says which is which. Its ownership claim is scoped to the runtime, since packages/studio still hand-rolls its own checks. |
||
|
|
5973273f8d |
fix(runtime): resolve media clip start once for playback, manifest, and audio (#3833)
* fix(core): play and draw media clips at the same time A clip placed inside a scene could be drawn on the editor's timeline at one time and actually play at another. Nothing failed; the timeline just showed the clip in the wrong place, and a clip pushed past the end of the composition disappeared from it entirely. The cause was that the code drawing the timeline and the code playing the video each worked out the clip's start time from the same HTML attributes in their own way, and only one of them knew about the marker that says "this start time is already measured from the beginning of the whole video". Both now ask the same function. The same function also answers for the visibility pass, the audio scheduling paths, and the volume-fade probe, all of which were reading the raw attribute and so placed a nested clip's audio at the wrong moment. Also folds the duplicated media-length helper into one shared version. * refactor(core): resolve the volume probe window once per element * fix(core): resolve media starts through the pass-scoped timing resolver After rebasing onto the resolver-reuse change, the shared media start resolver was building a fresh start-time resolver per call, which is the per-element construction that change removed. Route it through the scoped resolver so one pass shares one set of caches. |
||
|
|
e282ff15cc |
fix(audio): raise the authoring gain ceiling and carry it through the probes (#3333)
* fix(audio): raise the authoring gain ceiling and carry it through the probes Builds on #3328, which made the preview graph apply author gain and user volume exactly once each. That ownership is now correct but everything is still clamped to 1.0, so a clip authored above unity cannot be heard or rendered. `HTMLMediaElement.volume` is spec-clamped to [0,1], so both timeline probes lost a clip's authored gain the moment it also carried a fade: the probe seeded the element at the clamped value and every sample read back at or below 0 dB, and the mixer prefers probed keyframes over the static volume. Both probes now shadow the accessor for their own duration and forward the clamped value to the native setter, so the authored gain survives while nothing outside the probe ever sees an illegal volume. Measured on one 6 s composition, first 4 s: unity -32.8 LUFS, boosted-with-fade -32.8 before and -27.0 after — +5.8 dB, exactly the gain the clip was authored at. One ceiling, defined once in `audioGain.ts` and reachable from both sides: the render mixer imports it, and the page-serialized probe takes it as a parameter rather than re-literalling it. User volume stays spec-clamped — it is a fader, not a gain. Also holds the percent volume slider above unity in both property panels. That control tops out at 100%, so one touch would cap a boosted clip and drop up to 12 dB that now genuinely renders; the dB fader that can represent these levels replaces it in the next PR. * fix(audio): carry a static above-unity gain onto the preview gain node Review follow-up. `setElementVolume` receives the clip's author gain and clamped it to [0,1], so a static `data-volume` above unity was capped on the WebAudio preview path while the render honoured it — the exact preview/render divergence this ceiling exists to close. Automation lanes hid it: they schedule ramps onto the param directly and never pass through here. The master volume beside it stays spec-clamped, because a user fader is not a gain. Verified by mutation: restoring the [0,1] clamp reds the new case. Also scope the leveller's rationale to this rung — `VOLUME_RANGE` still stops at unity until the dB fader lands, so "both now span the same range" was premature — and say why the GSAP-tracking fallback is unity-capped: it reads back through `el.volume`, which the spec pins to [0,1], so it cannot observe an above-unity value however wide the clamp gets. * fix(audio): restore the live test files this branch overwrote, and uncap preview Review blocker: three files were wholesale copies from the abandoned #3304 branch laid over a two-day-newer base, so they silently reverted work that had landed in between. CI could not see it — deleted tests do not fail. - `audioMixer.test.ts` was byte-identical to #3304's head: 1186 lines against a base of 1353. Gone with it were the `data-playback-start` fallthrough cases from #3322 — merged 54 minutes before this branch's own merge base — and all retiming coverage (`playbackRate` 7 to 0, `atempo` 5 to 0), the strict literal-timing table, and the zero-window cases. - `mediaVolumeEnvelope.test.ts` dropped the trailing-garbage duration case and the plateau-retention case. - `packages/core/package.json` rolled the package version back 0.8.3 to 0.7.109. All three are restored from `main` with only this PR's additions re-applied on top, and the subpath export is regenerated by the repo's own script rather than hand-edited. Also closes the preview/render split the same review raised. Two clamps had to go, not one: `setElementVolume` capped the author gain at the transport, and the first-tick branch in `syncRuntimeMedia` trusted `el.volume` — which is spec-bound to [0,1] and so cannot represent a boost, opening a boosted clip at 0 dB for one tick before the steady-state branch took over. Both pinned by tests, both verified by mutation. |
||
|
|
afafca4b96 |
feat: make creator media edits render-safe (#3322)
* feat: make creator media edits render-safe * fix: align media playback timing * docs: add creator editing recipes * docs: expand creator editing guidance * fix: unify media source offsets * fix: scale natural media duration * fix: preserve natural media zero spans * fix: align compiled natural media timing * test: classify compiler media test as integration * fix: drop inactive media windows * fix: unify literal timing parsing * fix: keep browser media parsing serializable * fix: keep page timing readers strict * fix: close remaining preview timing gaps * fix(core): preserve Studio voice pitch at playback speed * chore: keep creator contract source-neutral |
||
|
|
eee9b26fb7 |
fix(core): key the preview volume envelope to the clip, not the timeline (#3198)
A GSAP volume fade on an audio clip that starts after t=0 left the preview silent for the clip's whole length while the encoded render was correct. `mediaVolumeEnvelope` is meant to keep preview and render on one envelope, and its contract is "normalise, then read with track-relative seconds". The preview skipped both halves. `probeElementVolumeKeyframes` stamps each keyframe with the TIMELINE seek time it sampled at, and `normaliseEnvelope` — the function that rebases those onto the track — had exactly one caller, the renderer's PCM baker. The preview handed the raw keyframes to `interpolateVolumeGain` along with `relTime`, so for a clip at t=2 every lookup fell two seconds before the first keyframe and clamped to its volume: 0 for a fade-in. Rebase once, where the cache is filled, so the cached envelope has a single documented time base. Read it with elapsed-time-in-clip rather than `relTime`, which is a position inside the media SOURCE — it carries `mediaStart` and the playback rate, and only coincides with the envelope's time base for an untrimmed clip playing at 1x from zero. That second half also fixes a latent sibling: a trimmed clip read the wrong envelope point even when it started at 0. |
||
|
|
04e0ccce42 |
fix: preserve plateaus in sampled audio automation (#2863)
* fix: preserve audio automation plateaus * fix(audio): align automation probe windows |
||
|
|
477defc7e7 | fix(core): avoid live volume probe during render | ||
|
|
0af07a07c5 | fix(core): enforce strict runtime safety | ||
|
|
16ab8b2935 | fix(core): preserve playhead during volume probing (#2143) | ||
|
|
d3c333b383 |
fix(core): apply renderer volume-automation solution to preview (#1118)
Preview audio with GSAP volume fades (e.g. data-volume="0" with a
gsap.to("#bgm", {volume:0.25, ...})) played ~1s then silenced. Root
cause: syncRuntimeMedia used fallbackAuthorVolume (data-volume) on the
first tick after a clip became active, clobbering the GSAP-seeked value.
The single-clock transport seeks GSAP before syncRuntimeMedia runs, so
el.volume already holds the animated value — we just need to trust it.
Fix — three layers, matching the renderer's approach (PR #1117):
1. First-tick tracking: on the first tick a clip is active
(previousRuntimeVolume===undefined), use currentElementVolume (GSAP's
seeked value) instead of fallbackAuthorVolume. In production the
transport always seeks GSAP before syncRuntimeMedia, so el.volume is
already at the correct animated position.
2. Probed keyframes: new probeElementVolumeKeyframes() runs the same
offline probe the renderer uses (discoverAudioVolumeAutomationFromTimeline)
directly in the browser. init.ts calls probeAndCacheElementVolume() when
an element is bound and a timeline is available. When keyframes are present,
syncRuntimeMedia drives volume from the interpolated envelope — no
GSAP-change tracking needed, no first-tick edge case, same data source
as the renderer.
3. Shared utilities: normaliseEnvelope(), interpolateVolumeGain(), and
probeAndCacheElementVolume() extracted to mediaVolumeEnvelope.ts and
exported from @hyperframes/core/media-volume-envelope. The engine's
audioVolumeEnvelope.ts imports from there — no duplicate logic between
the renderer and the new preview path.
Fallow audit exits non-zero on inherited complexity/duplication in init.ts
functions that shifted line numbers (applyClipLayout, transportTick, etc.),
unchanged by this PR — same known false-positive pattern noted in #1117.
Lint, format, typecheck, and unit tests all pass.
53 core/media tests pass (3 updated to pre-set el.volume to match the
runtime's bindMediaMetadataListeners — corrects a missing setup step).
audioVolumeEnvelope tests (6) still pass.
|