Commit Graph
12 Commits
Author SHA1 Message Date
Vance IngallsandClaude Opus 5.5 d05bc4215d perf(core): preview runtime scales with what plays, not with the film's clip count (#5674)
* perf(core): probe media volume with one timeline walk per pass, not one per element

Each bind, rebind and scene-swap pass probes every media element for volume
automation, and each probe flattened the whole captured timeline
(getChildren(true, true, false)) to ask whether any tween could move that one
element's volume. A pass cost elements x tweens. Elements with no automation are
never cached, so the cost came back on every rebind pass.

On a two-hour podcast episode (2,354 media clips, ~100k caption tweens) that was
several seconds of preview main thread per pass (CPU profile of the project's
open in Desktop: IE/PE -> getChildren, ~6 s).

volumeMoversOf() now walks the tweens once and lists the elements a tween names
volume on, plus whether any non-DOM setter is tweened. Each pass builds it lazily
and shares it with every probe; a probe without one walks as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(core): find the master bus from the first composition root

The top-level root is the first [data-composition-id] in document order, so
one match replaces a scan of every composition node, read once per clip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(core): lay out a clip again only when its layout inputs change

applyClipLayout ran on every timeline post and, for every root clip, lifted
display:none, read computed style and restored it. A clip whose inline
layout, class and data-width/height are as last laid out gets the same
answer, so it is skipped. On a 2,354-clip episode that was 4 of every 10 s
of playback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(core): a clip is laid out again only when its layout inputs change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(core): play a long film without re-walking it every frame

- The visibility pass skips root-level media far from the playhead during
  steady playback: hidden last pass, not loading or armed, and its window
  (widened by every look-ahead) clear of the span played and the loop start.
  Seeks, forced syncs and renders visit every node.
- The GSAP re-render saves and restores only the landing marks near the
  playhead, placed once per timeline structure; the zero-duration-callback
  check is cached the same way. A caption track of 38k sets was walked twice
  a frame.
- Registered timeline hosts, the root composition and the per-frame
  whole-film queries ([data-start], audio groups, timed audio) are kept for
  one timing revision; a render reads fresh.
- The runtime's own media release (src reset) and load events of clips with
  an authored length no longer count as timing changes, which re-posted the
  timeline about twice a second.
- The timeline payload resolves a stacking context once per parent.

2,354-clip episode in Desktop: playback 15 -> 67-89 fps (unminified runtime).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(core): check each element once per batch of raster-hint records, and cache the root from init

- A script building a large tree (a caption track) made each attribute record
  re-walk its subtree with getComputedStyle; a batch now checks each element
  once, an element whose own attributes changed still reading its own hints.
- The root composition cache installs the timing observer itself: a root
  with a declared length never reads the duration floors that installed it,
  so init looked the root up once per media clip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): type the playback visibility skip for any element, and cache clip layout only once loaded

The skip's predicate took an HTMLElement where the pass hands it a
StylableElement (tsconfig.runtime.json). The layout cache records a clip only
once the document has loaded, so a pass before stylesheets apply is redone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): realm-safe element checks in the timing caches; cut added comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 18:33:19 +00:00
Miguel Ángel 4ed269ad1e perf(core): playback housekeeping runs on a timer, and rebinds stop re-sampling silent media (#4595)
* perf(core): playback housekeeping runs on a timer, and rebinds stop re-sampling silent media

While playing, the transport re-posted the clip manifest every 20 frames and
re-bound media every 30, frame-counted, so a 120 Hz display did both twice as
often for a film that was not changing. Playing, those jobs now run when the
composition changes (rate-limited as before) and on a one-second poll for
what nothing can push.

A rebind re-probed every media element's volume by seeking the whole timeline
60 times per second of clip, about 1,500 seeks and 50-140 ms in one frame.
The probe seeks with events suppressed, so only a tween on the element that
names volume can move it; an element no tween touches is now answered from
the tweens without seeking.

* fix(core): a data-width or data-height change applies while playing and parked

postTimeline applies the root's sizing, and it now runs when the composition
revision moves. The revision only watched timing attributes, so a size edit
waited for a pause while playing, and on main it was missed while parked
too. The observer now also watches data-width and data-height. The tick
comment is trimmed to three lines, the tick reuses its playing flag, and the
cadence test is renamed to what it checks now.

* test(core): share one 120 Hz frame step in the transport park tests

The two playing tests each defined the same frame helper; one frame120Hz
now sits beside settle and all four loops use it. No source change.

* refactor(core): drop the playing poll doc comments

The comment ratchet does not let a file's comment share rise; the reason
lives in the PR body.

* fix(core): a playing clip's track, label and z-index reach Studio again

The playing path had dropped the periodic manifest post, and the
composition observer does not see data-track-index, data-timeline-label or
an inline z-index on an existing clip (watching style would fire on every
GSAP frame). On the playing poll the runtime now builds the manifest and
posts it only when it differs from the last one posted, so an edit lands
within a second and a steady film still posts nothing.

* fix(core): keep sampling volume when a tween drives a non-DOM object

A tween on a plain-object proxy whose setter writes the element's volume
names neither the element nor volume, so the fast skip dropped it and
WebAudio-routed preview played the static data-volume (0 where the film
asks for 0.375). The skip now applies only when every tween targets DOM
nodes; any other target falls back to the full volume probe, as on main.

* fix(core): skip the volume probe again for spacers, calls and counters

The non-DOM fallback fired on any plain-object or function target, so a
tl.to({}, {duration}) spacer, a tl.call or a tweened counter brought back the
full probe on ordinary films. It now falls back only when a tweened property
resolves to a setter on the target or its prototype chain, which is what a
gain proxy or a class with a volume setter uses; data properties skip.

* fix(core): post a clip edit made while playing when playback stops

A label, track or inline z-index edited while playing and then paused before
the next one-second poll was never posted: once parked, nothing watches it.
The tick where playback stops now runs the same manifest diff once and posts
only if the manifest changed.
2026-09-28 05:58:37 +00:00
Miguel Ángel 73dfe35e46 fix(preview): show the correct scene instead of every scene at once on some loads (#3848)
* fix(runtime): show the right scene when the preview DOM comes from another window

The Studio preview sometimes builds the composition body in the editor window
and adopts it into the preview frame's document. Adopted nodes keep the
prototypes of the realm that created them, so `node instanceof HTMLElement` is
false for every element in the composition even though the elements are
ordinary HTML sitting in that document.

Every guard in the runtime written as `if (!(node instanceof HTMLElement))`
then skipped the whole document, silently: nothing threw, nothing logged, and
readiness still reported success. The timed-element visibility pass wrote no
inline visibility at all, so on those loads the preview painted every scene on
top of every other and the editor drew off-canvas markers for elements the user
could not see. The auto-stamp pass stopped stamping too, which is why the
composition came up one timeline clip short.

Replace every realm-sensitive element check in the runtime with structural
predicates that ask what a node IS (node type, namespace, tag name) rather than
which window's constructor made it. Two local `doc.defaultView.HTMLElement`
workarounds are deleted with it: they fix only the case where the nodes belong
to the document's own realm, and adoption is exactly what breaks that.

* fix(studio): scrub the music track the timeline named, not the first audio

`resolveScrubAudioEl` tested `byId instanceof HTMLAudioElement` on a node from
the preview iframe's document. That node is an instance of the IFRAME's
`HTMLAudioElement`, never this module's, so the check was false on every load
and the `musicId` hint was dead. Scrub fell through to the first `<audio>` in
the document, which the comment right above it warns can be the voiceover, so
dragging the playhead could preview the wrong track. Ask what the node is
instead.

Also close the gaps an independent review found in the runtime fix:

- the cross-realm predicate test had no `<audio>` and no `<img>`, so reverting
  `isAudioElement` or `isImageElement` to `instanceof` left it green. It no
  longer does, and the audio case also pins `isMediaElement`, which composes
  from it and gates the media sync path.
- nothing stopped the runtime regressing. `lint-runtime-preview-guards.ts`
  gains a second check kind: patterns that must be ABSENT under a directory,
  seeded with DOM-typed `instanceof` under `src/runtime`, pointing at
  domRealm.ts. Comment lines and tests are exempt, both on purpose.
- domRealm.ts stated the adoption mechanism as settled fact. The mixed
  prototypes are measured; how the nodes get into the frame is not identified,
  and the docstring now says which is which. Its ownership claim is scoped to
  the runtime, since packages/studio still hand-rolls its own checks.
2026-09-10 08:06:43 -07:00
Miguel Ángel 5973273f8d fix(runtime): resolve media clip start once for playback, manifest, and audio (#3833)
* fix(core): play and draw media clips at the same time

A clip placed inside a scene could be drawn on the editor's timeline at one
time and actually play at another. Nothing failed; the timeline just showed
the clip in the wrong place, and a clip pushed past the end of the composition
disappeared from it entirely.

The cause was that the code drawing the timeline and the code playing the
video each worked out the clip's start time from the same HTML attributes in
their own way, and only one of them knew about the marker that says "this
start time is already measured from the beginning of the whole video".

Both now ask the same function. The same function also answers for the
visibility pass, the audio scheduling paths, and the volume-fade probe, all of
which were reading the raw attribute and so placed a nested clip's audio at
the wrong moment.

Also folds the duplicated media-length helper into one shared version.

* refactor(core): resolve the volume probe window once per element

* fix(core): resolve media starts through the pass-scoped timing resolver

After rebasing onto the resolver-reuse change, the shared media start
resolver was building a fresh start-time resolver per call, which is the
per-element construction that change removed. Route it through the
scoped resolver so one pass shares one set of caches.
2026-09-10 02:30:16 +00:00
Miguel Ángel e282ff15cc fix(audio): raise the authoring gain ceiling and carry it through the probes (#3333)
* fix(audio): raise the authoring gain ceiling and carry it through the probes

Builds on #3328, which made the preview graph apply author gain and user volume
exactly once each. That ownership is now correct but everything is still clamped
to 1.0, so a clip authored above unity cannot be heard or rendered.

`HTMLMediaElement.volume` is spec-clamped to [0,1], so both timeline probes lost
a clip's authored gain the moment it also carried a fade: the probe seeded the
element at the clamped value and every sample read back at or below 0 dB, and
the mixer prefers probed keyframes over the static volume. Both probes now
shadow the accessor for their own duration and forward the clamped value to the
native setter, so the authored gain survives while nothing outside the probe
ever sees an illegal volume.

Measured on one 6 s composition, first 4 s: unity -32.8 LUFS, boosted-with-fade
-32.8 before and -27.0 after — +5.8 dB, exactly the gain the clip was authored
at.

One ceiling, defined once in `audioGain.ts` and reachable from both sides: the
render mixer imports it, and the page-serialized probe takes it as a parameter
rather than re-literalling it. User volume stays spec-clamped — it is a fader,
not a gain.

Also holds the percent volume slider above unity in both property panels. That
control tops out at 100%, so one touch would cap a boosted clip and drop up to
12 dB that now genuinely renders; the dB fader that can represent these levels
replaces it in the next PR.

* fix(audio): carry a static above-unity gain onto the preview gain node

Review follow-up.

`setElementVolume` receives the clip's author gain and clamped it to [0,1],
so a static `data-volume` above unity was capped on the WebAudio preview path
while the render honoured it — the exact preview/render divergence this
ceiling exists to close. Automation lanes hid it: they schedule ramps onto the
param directly and never pass through here. The master volume beside it stays
spec-clamped, because a user fader is not a gain.

Verified by mutation: restoring the [0,1] clamp reds the new case.

Also scope the leveller's rationale to this rung — `VOLUME_RANGE` still stops
at unity until the dB fader lands, so "both now span the same range" was
premature — and say why the GSAP-tracking fallback is unity-capped: it reads
back through `el.volume`, which the spec pins to [0,1], so it cannot observe an
above-unity value however wide the clamp gets.

* fix(audio): restore the live test files this branch overwrote, and uncap preview

Review blocker: three files were wholesale copies from the abandoned #3304
branch laid over a two-day-newer base, so they silently reverted work that had
landed in between. CI could not see it — deleted tests do not fail.

- `audioMixer.test.ts` was byte-identical to #3304's head: 1186 lines against a
  base of 1353. Gone with it were the `data-playback-start` fallthrough cases
  from #3322 — merged 54 minutes before this branch's own merge base — and all
  retiming coverage (`playbackRate` 7 to 0, `atempo` 5 to 0), the strict
  literal-timing table, and the zero-window cases.
- `mediaVolumeEnvelope.test.ts` dropped the trailing-garbage duration case and
  the plateau-retention case.
- `packages/core/package.json` rolled the package version back 0.8.3 to 0.7.109.

All three are restored from `main` with only this PR's additions re-applied on
top, and the subpath export is regenerated by the repo's own script rather than
hand-edited.

Also closes the preview/render split the same review raised. Two clamps had to
go, not one: `setElementVolume` capped the author gain at the transport, and
the first-tick branch in `syncRuntimeMedia` trusted `el.volume` — which is
spec-bound to [0,1] and so cannot represent a boost, opening a boosted clip at
0 dB for one tick before the steady-state branch took over. Both pinned by
tests, both verified by mutation.
2026-08-18 20:07:42 -04:00
Miguel Ángel afafca4b96 feat: make creator media edits render-safe (#3322)
* feat: make creator media edits render-safe

* fix: align media playback timing

* docs: add creator editing recipes

* docs: expand creator editing guidance

* fix: unify media source offsets

* fix: scale natural media duration

* fix: preserve natural media zero spans

* fix: align compiled natural media timing

* test: classify compiler media test as integration

* fix: drop inactive media windows

* fix: unify literal timing parsing

* fix: keep browser media parsing serializable

* fix: keep page timing readers strict

* fix: close remaining preview timing gaps

* fix(core): preserve Studio voice pitch at playback speed

* chore: keep creator contract source-neutral
2026-08-18 10:17:02 -04:00
Miguel Ángel eee9b26fb7 fix(core): key the preview volume envelope to the clip, not the timeline (#3198)
A GSAP volume fade on an audio clip that starts after t=0 left the preview
silent for the clip's whole length while the encoded render was correct.

`mediaVolumeEnvelope` is meant to keep preview and render on one envelope, and
its contract is "normalise, then read with track-relative seconds". The preview
skipped both halves. `probeElementVolumeKeyframes` stamps each keyframe with the
TIMELINE seek time it sampled at, and `normaliseEnvelope` — the function that
rebases those onto the track — had exactly one caller, the renderer's PCM baker.
The preview handed the raw keyframes to `interpolateVolumeGain` along with
`relTime`, so for a clip at t=2 every lookup fell two seconds before the first
keyframe and clamped to its volume: 0 for a fade-in.

Rebase once, where the cache is filled, so the cached envelope has a single
documented time base. Read it with elapsed-time-in-clip rather than `relTime`,
which is a position inside the media SOURCE — it carries `mediaStart` and the
playback rate, and only coincides with the envelope's time base for an
untrimmed clip playing at 1x from zero. That second half also fixes a latent
sibling: a trimmed clip read the wrong envelope point even when it started at 0.
2026-08-10 23:38:32 -04:00
Miguel Ángel 04e0ccce42 fix: preserve plateaus in sampled audio automation (#2863)
* fix: preserve audio automation plateaus

* fix(audio): align automation probe windows
2026-07-29 20:51:33 +02:00
James 477defc7e7 fix(core): avoid live volume probe during render 2026-07-26 22:39:58 +00:00
James 0af07a07c5 fix(core): enforce strict runtime safety 2026-07-11 21:31:30 -07:00
Miguel Ángel 16ab8b2935 fix(core): preserve playhead during volume probing (#2143) 2026-07-10 17:18:16 -04:00
Miguel Ángel d3c333b383 fix(core): apply renderer volume-automation solution to preview (#1118)
Preview audio with GSAP volume fades (e.g. data-volume="0" with a
gsap.to("#bgm", {volume:0.25, ...})) played ~1s then silenced. Root
cause: syncRuntimeMedia used fallbackAuthorVolume (data-volume) on the
first tick after a clip became active, clobbering the GSAP-seeked value.
The single-clock transport seeks GSAP before syncRuntimeMedia runs, so
el.volume already holds the animated value — we just need to trust it.

Fix — three layers, matching the renderer's approach (PR #1117):

1. First-tick tracking: on the first tick a clip is active
   (previousRuntimeVolume===undefined), use currentElementVolume (GSAP's
   seeked value) instead of fallbackAuthorVolume. In production the
   transport always seeks GSAP before syncRuntimeMedia, so el.volume is
   already at the correct animated position.

2. Probed keyframes: new probeElementVolumeKeyframes() runs the same
   offline probe the renderer uses (discoverAudioVolumeAutomationFromTimeline)
   directly in the browser. init.ts calls probeAndCacheElementVolume() when
   an element is bound and a timeline is available. When keyframes are present,
   syncRuntimeMedia drives volume from the interpolated envelope — no
   GSAP-change tracking needed, no first-tick edge case, same data source
   as the renderer.

3. Shared utilities: normaliseEnvelope(), interpolateVolumeGain(), and
   probeAndCacheElementVolume() extracted to mediaVolumeEnvelope.ts and
   exported from @hyperframes/core/media-volume-envelope. The engine's
   audioVolumeEnvelope.ts imports from there — no duplicate logic between
   the renderer and the new preview path.

Fallow audit exits non-zero on inherited complexity/duplication in init.ts
functions that shifted line numbers (applyClipLayout, transportTick, etc.),
unchanged by this PR — same known false-positive pattern noted in #1117.
Lint, format, typecheck, and unit tests all pass.

53 core/media tests pass (3 updated to pre-set el.volume to match the
runtime's bindMediaMetadataListeners — corrects a missing setup step).
audioVolumeEnvelope tests (6) still pass.
2026-05-29 10:10:12 -04:00