The runtime plays audio two ways — a Web Audio transport (sample-accurate) and
the HTMLMediaElement as a fallback — and mutes the elements when Web Audio takes
over so they don't double-play. That mute gate was global: it muted every element
the moment ANY Web Audio source was active (webAudio.isActive()). A track Web
Audio had not claimed yet (its larger buffer decodes slower) was muted on the
fallback AND not playing on Web Audio = silent, while the other tracks played.
With TTS narration + BGM + SFX, the narration (largest buffer) lost the decode
race and dropped out intermittently, with every file fully loaded.
Make the mute per-element: an element is muted only when its own Web Audio source
is live, or the user / parent-proxy force-mute is set. A track Web Audio has not
claimed stays audible on the HTMLMedia fallback until the transport takes it over
— which also lets narration start immediately on cold play instead of waiting for
its buffer to decode.
Also in this change:
- Don't permanently blacklist a transient fetch failure in the Web Audio decoder
(_failedSrcs was never cleared); only blacklist genuinely undecodable bytes, so
a late-arriving asset (404 then available) self-heals on the next play.
- Stop re-issuing play() every tick on an errored / no-source element.
* fix(player): bound the parent audio proxy to its clip window
When iframe autoplay is blocked, audible playback is promoted to a parent-frame
audio proxy. The proxy read the clip's data-start/data-duration once at adopt
time and mirrorTime() only skipped (never paused) the element outside that
window — so a trimmed/moved music clip kept playing the full source past its
on-timeline end, even though the iframe element was correctly paused.
Fix: the proxy keeps a reference to its source iframe element and re-reads
data-start/data-duration each mirror tick (live trims/moves apply), pauses the
proxy when the playhead leaves [start, start+duration), and resumes it when the
playhead re-enters during parent-owned playback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
* fix(core,studio): bound trimmed audio playback to the clip window
Trimmed audio played to the source file's natural end instead of
stopping at the clip edge, on every audio path:
- WebAudio (the audible path in Studio): schedulePlayback now passes
the clip's data-duration as the third start() arg, so the decoded
buffer stops at the trimmed edge instead of running to the file end.
- Runtime element gating: the duration resolver caps each clip by its
own data-duration (min of source length, host window, authored
duration), so a trimmed <audio>/<video> element pauses at its edge.
Studio trim UX:
- Resize live-patches the media-start/playback-start offset, so a
start-edge drag trims into the source instead of only repositioning
the clip.
- AudioWaveform windows the rendered peaks to the trimmed slice so the
waveform tracks the clip edges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
* fix(player,core): gate proxy playback to the live clip window
Review follow-ups on the parent-audio-proxy / WebAudio bound:
- seekAll now re-reads live source bounds (_refreshEntryBounds) before
gating, so a paused scrub right after a trim/move uses the current clip
window instead of the adopt-time one.
- playAll and clip adoption only start a proxy when the playhead is inside
the clip's window (_playEntryIfActive), so bulk starts / promotion no
longer blip audio for clips outside their window until the next tick.
- The WebAudio buffer is now bounded by the host-composition window too
(matching resolveDurationSeconds), so a sub-composition-nested clip stops
at the same edge on the WebAudio and HTMLMedia paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
* fix(core,player): reschedule bounded WebAudio on rate change; guard NaN bounds
A bounded WebAudio source's wall-clock length is baked into start()'s duration
arg (in buffer-sample seconds) at its scheduling rate. Mutating playbackRate in
place on a later rate change does not rescale that bound, so a trimmed clip ends
early (fast) or late (slow). setRate now reports whether the rate changed and
exposes hasBoundedActiveSources(); the runtime stopAll()+reschedules active
clips at the new rate when any bounded source is live. The per-clip schedule
loop is extracted to a shared closure so play() and the rate path agree.
Also guard _refreshEntryBounds against a non-numeric duration attribute parsing
to NaN, which would make every window check false and let the proxy play past
its clip end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
---------
Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
onSetMuted/onSetMediaOutputMuted set el.muted = effective on every
<video> and <audio> element. When the bridge sent onSetMuted(false),
it unmuted avatar <video muted> elements whose baked-in lip-sync audio
should never play — causing double audio alongside the separate TTS.
Fix: el.muted = effective || el.defaultMuted.
AudioBufferSourceNode fires 'ended' when playback completes naturally,
but _activeSources was never cleaned up. This kept isActive() true
permanently, which force-muted all HTML audio elements via the
outputMuted flag in syncRuntimeMedia — causing audio to disappear
after the WebAudio buffer finished (~5s for short TTS clips).
Add onended listener that removes the source from _activeSources and
restores el.muted to its pre-WebAudio value. All side-effects are
guarded by idx !== -1 so a stale ended event after stopAll() is a
no-op and cannot clobber bridge state set between stop and the async
event delivery.
Two fixes for the 3M+ unhandled_promise_rejection events/day spike:
1. Filter: suppress "Error fetching ... 404" rejections from composition
code — these are asset-not-found content errors, not Studio bugs.
2. Rate-limit: cap both error and rejection telemetry at 50 per session.
After the cap, emit a single *_cap_reached event so we know capping
occurred without generating unlimited events.
3. Root cause: webAudioTransport now checks response.ok before decode
and caches failed URLs in _failedSrcs so repeat ticks don't re-fetch
the same 404 on every playback frame.
Also add playground/ to fallow ignorePatterns — local experiment
directory was tripping the audit gate.
- Hoist duplicated test mock helpers (createMockAudioContext / setupTransport /
mockBuffer / mockEl) from the two describe blocks to module scope.
- Drop redundant math-derivation comments in schedulePlayback; the dedicated
rate-aware tests are the canonical proof.
- Tighten setRate JSDoc.
- Add no-op guard in setRate when the new rate equals the current rate, so a
duplicate set-playback-rate postMessage doesn't re-anchor or walk active
sources for nothing.
- Add a regression test for the no-op guard, and strengthen the clamp test
to schedule at rate=2 first so the clamp-to-1 assertion is non-vacuous.
WebAudioTransport scheduled AudioBufferSourceNodes with the implicit
default playbackRate of 1, so non-1x transport rates desynced visuals
from audio: GSAP timelines, the transport clock, and native <video>
all sped up while WebAudio-routed <audio> clips kept playing at 1x.
- schedulePlayback now accepts a rate, sets sourceNode.playbackRate,
and scales the future-clip start delay by the rate (the in-progress
buffer offset stays elapsed + mediaStart, which is rate-independent).
- New setRate() updates active sources in place and rebases the
getTime() reference frame so the audio-master clock stays continuous
across mid-playback rate changes.
- Runtime onSetPlaybackRate now forwards into webAudio.setRate, and
player.play() schedules each clip with state.playbackRate.
Fixes#713
Clips whose compositionStart is ahead of the current timeline position
were starting immediately because sourceNode.start() always received
when=0. Use the AudioContext scheduling API to defer future clips:
sourceNode.start(ctx.currentTime + delay, mediaStart).
Closes#674
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the two-clock architecture (GSAP rAF ticker + HTMLMediaElement
pipeline reconciled by a 50ms polling loop) with a single TransportClock.
GSAP is always paused and seeked to clock.now() on each rAF tick.
Drift between visual timeline and audio is structurally impossible.
Architecture:
TransportClock.now() ──rAF──▶ timeline.seek(t) + el.currentTime
▲
AudioContext.currentTime (~21µs) ← WebAudio active
OR
audio.currentTime (~33ms) ← HTMLMediaElement fallback
OR
performance.now() (~1ms) ← no audio
Key changes:
- TransportClock class with monotonic + audio-master clock sources
- WebAudioTransport: routes audio through AudioBufferSourceNode for
sample-accurate scheduling, falls back gracefully to HTMLMediaElement
- rAF tick loop replaces 50ms setInterval poll; GSAP always paused
- Strict sync (40ms threshold, consecutive-sample gated) + forceSync
on play/pause/seek transitions for sub-frame media accuracy
- Buffer-stall: visuals freeze when audio is buffering instead of
running ahead
- Frame quantization preserved in seek/renderSeek (parity contract)
Browser-verified: 0.0ms drift after 40 pause/play cycles (was 400ms+).
Also fixes: CDN script HTML error responses in validate (pre-existing).
54 tests across clock, clock-drift, webAudioTransport, and media.
Closes#668
Miguel Ángel