fix(core,player,studio): bound trimmed audio playback to the clip window (#1430)

* fix(player): bound the parent audio proxy to its clip window

When iframe autoplay is blocked, audible playback is promoted to a parent-frame
audio proxy. The proxy read the clip's data-start/data-duration once at adopt
time and mirrorTime() only skipped (never paused) the element outside that
window — so a trimmed/moved music clip kept playing the full source past its
on-timeline end, even though the iframe element was correctly paused.

Fix: the proxy keeps a reference to its source iframe element and re-reads
data-start/data-duration each mirror tick (live trims/moves apply), pauses the
proxy when the playhead leaves [start, start+duration), and resumes it when the
playhead re-enters during parent-owned playback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* fix(core,studio): bound trimmed audio playback to the clip window

Trimmed audio played to the source file's natural end instead of
stopping at the clip edge, on every audio path:

- WebAudio (the audible path in Studio): schedulePlayback now passes
  the clip's data-duration as the third start() arg, so the decoded
  buffer stops at the trimmed edge instead of running to the file end.
- Runtime element gating: the duration resolver caps each clip by its
  own data-duration (min of source length, host window, authored
  duration), so a trimmed <audio>/<video> element pauses at its edge.

Studio trim UX:

- Resize live-patches the media-start/playback-start offset, so a
  start-edge drag trims into the source instead of only repositioning
  the clip.
- AudioWaveform windows the rendered peaks to the trimmed slice so the
  waveform tracks the clip edges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* fix(player,core): gate proxy playback to the live clip window

Review follow-ups on the parent-audio-proxy / WebAudio bound:
- seekAll now re-reads live source bounds (_refreshEntryBounds) before
  gating, so a paused scrub right after a trim/move uses the current clip
  window instead of the adopt-time one.
- playAll and clip adoption only start a proxy when the playhead is inside
  the clip's window (_playEntryIfActive), so bulk starts / promotion no
  longer blip audio for clips outside their window until the next tick.
- The WebAudio buffer is now bounded by the host-composition window too
  (matching resolveDurationSeconds), so a sub-composition-nested clip stops
  at the same edge on the WebAudio and HTMLMedia paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* fix(core,player): reschedule bounded WebAudio on rate change; guard NaN bounds

A bounded WebAudio source's wall-clock length is baked into start()'s duration
arg (in buffer-sample seconds) at its scheduling rate. Mutating playbackRate in
place on a later rate change does not rescale that bound, so a trimmed clip ends
early (fast) or late (slow). setRate now reports whether the rate changed and
exposes hasBoundedActiveSources(); the runtime stopAll()+reschedules active
clips at the new rate when any bounded source is live. The per-clip schedule
loop is extracted to a shared closure so play() and the rate path agree.

Also guard _refreshEntryBounds against a non-numeric duration attribute parsing
to NaN, which would make every window check false and let the proxy play past
its clip end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

---------

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
This commit is contained in:
Vance Ingalls
2026-06-14 17:14:49 -07:00
committed by GitHub
co-authored by Claude Opus 4.8 Miguel Ángel
parent 5c8b637369
commit a95e49dbda
8 changed files with 373 additions and 79 deletions
+76 -32
View File
@@ -1469,10 +1469,18 @@ export function initSandboxRuntimeModular(): void {
Number.isFinite(element.duration) && element.duration > mediaStart
? Math.max(0, element.duration - mediaStart)
: null;
if (sourceDuration != null && hostRemaining != null) {
return Math.min(sourceDuration, hostRemaining);
}
return sourceDuration ?? hostRemaining;
// The element's own data-duration is an explicit clip-length trim
// (the studio writes it when you drag the clip edge). It must bound
// playback so a trimmed track stops at its edge instead of running on
// to the source-file or host-composition end. Absent → no cap (an
// untrimmed clip plays its natural source length).
const ownDuration = Number.parseFloat(element.dataset.duration ?? "");
const explicitDuration =
Number.isFinite(ownDuration) && ownDuration > 0 ? ownDuration : null;
const candidates = [sourceDuration, hostRemaining, explicitDuration].filter(
(value): value is number => value != null,
);
return candidates.length > 0 ? Math.min(...candidates) : null;
},
});
// Attach probed volume keyframes to clips so syncRuntimeMedia can use the
@@ -1760,7 +1768,7 @@ export function initSandboxRuntimeModular(): void {
onSetPlaybackRate: (rate) => {
applyPlaybackRate(rate);
if (state.transportClock) state.transportClock.setRate(state.playbackRate);
webAudio.setRate(state.playbackRate);
applyWebAudioRate();
},
onTick: () => {
if (state.tornDown || !clock.isPlaying()) return;
@@ -2133,6 +2141,67 @@ export function initSandboxRuntimeModular(): void {
};
// Player methods route through the TransportClock.
// Schedule WebAudio playback for every in-window audio clip, bounding each
// buffer to its clip window (own data-duration AND the remaining host
// composition window) so trimmed / sub-composition-nested clips stop at the
// same edge as the HTMLMedia path. Reused by play() and by the rate-change
// handler (a rate change can't rescale a bounded source in place).
const scheduleWebAudioForActiveClips = () => {
const gen = webAudio.startGeneration();
const audioEls = document.querySelectorAll("audio[data-start]");
for (const rawEl of audioEls) {
if (!(rawEl instanceof HTMLMediaElement) || !rawEl.isConnected) continue;
const compStart = Number.parseFloat(rawEl.dataset.start ?? "");
if (!Number.isFinite(compStart)) continue;
const mediaStart =
Number.parseFloat(rawEl.dataset.playbackStart ?? rawEl.dataset.mediaStart ?? "0") || 0;
const volumeAttr = Number.parseFloat(rawEl.dataset.volume ?? "");
const vol = Number.isFinite(volumeAttr) ? volumeAttr : 1;
const durationAttr = Number.parseFloat(rawEl.dataset.duration ?? "");
let clipDuration =
Number.isFinite(durationAttr) && durationAttr > 0 ? durationAttr : Number.POSITIVE_INFINITY;
const compositionRoot = rawEl.closest("[data-composition-id]");
if (compositionRoot) {
const inheritedStart = resolveStartForElement(compositionRoot, 0);
const inheritedDuration = resolveDurationForElement(compositionRoot, {
includeAuthoredTimingAttrs: true,
});
if (inheritedDuration != null && inheritedDuration > 0) {
clipDuration = Math.min(
clipDuration,
Math.max(0, inheritedStart + inheritedDuration - compStart),
);
}
}
void webAudio.decodeAudioElement(rawEl).then((buffer) => {
if (!buffer || !clock.isPlaying()) return;
void webAudio.schedulePlayback(
rawEl,
buffer,
compStart,
mediaStart,
clock.now(),
vol * state.bridgeVolume,
gen,
state.playbackRate,
clipDuration,
);
});
}
};
// Apply a new playback rate to the WebAudio transport. Unbounded sources are
// rescaled in place; but a bounded source's window was baked into start()'s
// duration at its prior rate and can't be rescaled, so when one is active we
// stopAll()+reschedule at the new rate to keep trimmed clips ending on time.
const applyWebAudioRate = () => {
const changed = webAudio.setRate(state.playbackRate);
if (changed && webAudioReady && clock.isPlaying() && webAudio.hasBoundedActiveSources()) {
webAudio.stopAll();
scheduleWebAudioForActiveClips();
}
};
player.play = () => {
const tl = state.capturedTimeline;
if (clock.isPlaying()) return;
@@ -2157,32 +2226,7 @@ export function initSandboxRuntimeModular(): void {
// Schedule audio through WebAudio for sample-accurate timing.
// Falls back to HTMLMediaElement playback if WebAudio isn't ready
// or decoding fails (the syncRuntimeMedia path handles that).
if (webAudioReady) {
const gen = webAudio.startGeneration();
const audioEls = document.querySelectorAll("audio[data-start]");
for (const rawEl of audioEls) {
if (!(rawEl instanceof HTMLMediaElement) || !rawEl.isConnected) continue;
const compStart = Number.parseFloat(rawEl.dataset.start ?? "");
if (!Number.isFinite(compStart)) continue;
const mediaStart =
Number.parseFloat(rawEl.dataset.playbackStart ?? rawEl.dataset.mediaStart ?? "0") || 0;
const volumeAttr = Number.parseFloat(rawEl.dataset.volume ?? "");
const vol = Number.isFinite(volumeAttr) ? volumeAttr : 1;
void webAudio.decodeAudioElement(rawEl).then((buffer) => {
if (!buffer || !clock.isPlaying()) return;
void webAudio.schedulePlayback(
rawEl,
buffer,
compStart,
mediaStart,
clock.now(),
vol * state.bridgeVolume,
gen,
state.playbackRate,
);
});
}
}
if (webAudioReady) scheduleWebAudioForActiveClips();
runAdapters("play");
syncMediaForCurrentState();
postState(true);
@@ -2249,7 +2293,7 @@ export function initSandboxRuntimeModular(): void {
player.setPlaybackRate = (rate: number) => {
applyPlaybackRate(rate);
clock.setRate(state.playbackRate);
webAudio.setRate(state.playbackRate);
applyWebAudioRate();
};
// Sync clock duration from any captured timeline
@@ -171,6 +171,43 @@ describe("WebAudioTransport", () => {
});
});
describe("clip duration bound (trim)", () => {
it("bounds an in-progress clip to its remaining authored window", async () => {
const { transport, mock, gen } = setupTransport(100);
// compStart=5, mediaStart=0, compTime=8 → elapsed=3; clipDuration=10 → 7 left
await transport.schedulePlayback(mockEl, mockBuffer, 5, 0, 8, 1, gen, 1, 10);
expect(mock.startFn).toHaveBeenCalledWith(0, 3, 7);
});
it("bounds a future clip to its full authored window", async () => {
const { transport, mock, gen } = setupTransport(100);
// compStart=10, mediaStart=1.5, compTime=2 → elapsed=-8 → delay 8; clipDuration=4
await transport.schedulePlayback(mockEl, mockBuffer, 10, 1.5, 2, 1, gen, 1, 4);
expect(mock.startFn).toHaveBeenCalledWith(108, 1.5, 4);
});
it("does not schedule a clip whose window has already elapsed", async () => {
const { transport, mock, gen } = setupTransport(100);
// elapsed=15 > clipDuration=10 → nothing to play
const result = await transport.schedulePlayback(mockEl, mockBuffer, 5, 0, 20, 1, gen, 1, 10);
expect(result).toBeNull();
expect(mock.startFn).not.toHaveBeenCalled();
});
it("scales the bound by playback rate (buffer seconds)", async () => {
const { transport, mock, gen } = setupTransport(100);
// rate=2, clipDuration=10 → clipSourceLen=20; elapsed=3 → 17 buffer seconds left
await transport.schedulePlayback(mockEl, mockBuffer, 5, 0, 8, 1, gen, 2, 10);
expect(mock.startFn).toHaveBeenCalledWith(0, 3, 17);
});
it("plays unbounded when clipDuration is omitted (legacy behavior)", async () => {
const { transport, mock, gen } = setupTransport(100);
await transport.schedulePlayback(mockEl, mockBuffer, 5, 0, 8, 1, gen);
expect(mock.startFn).toHaveBeenCalledWith(0, 3);
});
});
describe("playback rate", () => {
it("sets sourceNode.playbackRate.value when rate is provided", async () => {
const { transport, mock, gen } = setupTransport(100);
+65 -7
View File
@@ -5,6 +5,42 @@ function normalizeRate(rate: number): number {
return rate;
}
/**
* Start a buffer source, bounding it to the clip's authored window
* (`data-duration`) so a trimmed clip stops at its edge instead of running the
* buffer to the source file's natural end. `clipSourceLen` is the clip span in
* buffer seconds; the third `start()` arg is the portion to play from the
* offset. An infinite `clipDuration` plays unbounded (legacy behavior).
*
* Returns false when the playhead is already past the clip end (nothing to
* play); the caller should discard the source.
*/
function startBoundedSource(
node: AudioBufferSourceNode,
opts: {
elapsed: number;
mediaStart: number;
scheduledAt: number;
safeRate: number;
clipDuration: number;
},
): boolean {
const { elapsed, mediaStart, scheduledAt, safeRate, clipDuration } = opts;
const hasBound = Number.isFinite(clipDuration) && clipDuration > 0;
const clipSourceLen = clipDuration * safeRate;
if (elapsed >= 0) {
const remaining = clipSourceLen - elapsed;
if (hasBound && remaining <= 0) return false;
if (hasBound) node.start(0, elapsed + mediaStart, remaining);
else node.start(0, elapsed + mediaStart);
return true;
}
const delay = -elapsed / safeRate;
if (hasBound) node.start(scheduledAt + delay, mediaStart, clipSourceLen);
else node.start(scheduledAt + delay, mediaStart);
return true;
}
export type ScheduledSource = {
el: HTMLMediaElement;
sourceNode: AudioBufferSourceNode;
@@ -13,6 +49,10 @@ export type ScheduledSource = {
mediaStart: number;
scheduledAt: number;
priorMuted: boolean;
// The clip had a finite window, so start() was given a fixed duration in
// buffer-sample seconds. That bound can't be rescaled in place on a rate
// change — callers must stopAll()+reschedule (see hasBoundedActiveSources).
bounded: boolean;
};
export class WebAudioTransport {
@@ -92,6 +132,7 @@ export class WebAudioTransport {
volume: number,
generation: number,
rate = 1,
clipDuration = Number.POSITIVE_INFINITY,
): Promise<ScheduledSource | null> {
if (!this._ctx || !this._masterGain) return null;
if (generation !== this._playGeneration) return null;
@@ -119,11 +160,19 @@ export class WebAudioTransport {
this._rateAnchorCtx = scheduledAt;
this._rateAnchorComp = compositionTime;
if (elapsed >= 0) {
sourceNode.start(0, elapsed + mediaStart);
} else {
const delay = -elapsed / safeRate;
sourceNode.start(scheduledAt + delay, mediaStart);
if (
!startBoundedSource(sourceNode, {
elapsed,
mediaStart,
scheduledAt,
safeRate,
clipDuration,
})
) {
// Playhead already past the clip end — discard the nodes we built.
sourceNode.disconnect();
gainNode.disconnect();
return null;
}
const priorMuted = el.muted;
@@ -137,6 +186,7 @@ export class WebAudioTransport {
mediaStart,
scheduledAt,
priorMuted,
bounded: Number.isFinite(clipDuration) && clipDuration > 0,
};
this._activeSources.push(scheduled);
this._paused = false;
@@ -163,9 +213,9 @@ export class WebAudioTransport {
* start in the future keep their original wallclock start time callers
* that need rate-correct future starts should `stopAll()` and reschedule.
*/
setRate(rate: number): void {
setRate(rate: number): boolean {
const safeRate = normalizeRate(rate);
if (safeRate === this._rate) return;
if (safeRate === this._rate) return false;
if (this._ctx && !this._paused) {
this._rateAnchorComp = this.getTime();
this._rateAnchorCtx = this._ctx.currentTime;
@@ -178,6 +228,14 @@ export class WebAudioTransport {
swallow("webAudioTransport.setRate", err);
}
}
return true;
}
// A bounded source's wall-clock duration was baked into start()'s duration
// arg at its original rate; a later rate change can't rescale it in place, so
// the caller must stopAll()+reschedule to keep trimmed clips ending on time.
hasBoundedActiveSources(): boolean {
return this._activeSources.some((s) => s.bounded);
}
stopAll(): void {
+49
View File
@@ -1,6 +1,22 @@
import { describe, it, expect } from "vitest";
import { ParentMediaManager, type ProxyEntry } from "./parent-media";
// A fake media element whose paused state is driven by play()/pause() stubs.
function makeFakeAudio(initiallyPaused: boolean): HTMLMediaElement {
const el = new Audio();
let paused = initiallyPaused;
Object.defineProperty(el, "paused", { get: () => paused });
el.pause = () => {
paused = true;
};
el.play = () => {
paused = false;
return Promise.resolve();
};
el.src = "https://example.test/music.mp3";
return el;
}
function makeManager(overrides: Partial<{ isPaused: boolean; owner: "runtime" | "parent" }> = {}) {
const mgr = new ParentMediaManager({
dispatchEvent: () => {},
@@ -73,6 +89,39 @@ describe("ParentMediaManager audio-src proxy lifecycle", () => {
expect(mgr.entries).toHaveLength(0);
});
it("pauses a proxy once the playhead passes the clip end (trimmed clip)", () => {
const mgr = makeManager({ owner: "parent", isPaused: false });
const el = makeFakeAudio(false); // already playing within the clip
mgr.entries.push({ el, start: 0, duration: 5, driftSamples: 0 });
mgr.mirrorTime(3); // inside [0, 5) — stays playing
expect(el.paused).toBe(false);
mgr.mirrorTime(6); // past the trimmed end — must pause
expect(el.paused).toBe(true);
});
it("re-reads the source element's live data-duration so trims bound the proxy", () => {
const mgr = makeManager({ owner: "parent", isPaused: false });
const source = new Audio();
source.setAttribute("data-start", "0");
source.setAttribute("data-duration", "30");
// jsdom reports isConnected=false unless attached; attach it.
document.body.appendChild(source);
const el = makeFakeAudio(false);
mgr.entries.push({ el, start: 0, duration: 30, driftSamples: 0, source });
mgr.mirrorTime(20); // within 30 → playing
expect(el.paused).toBe(false);
// User trims the clip to 10s; the proxy must pick it up and pause at 20s.
source.setAttribute("data-duration", "10");
mgr.mirrorTime(20);
expect(el.paused).toBe(true);
source.remove();
});
it("does not duplicate or hijack a clip the composition already owns", () => {
const mgr = makeManager();
// The composition already adopted a clip with this URL.
+56 -13
View File
@@ -28,6 +28,12 @@ export interface ProxyEntry {
el: HTMLMediaElement;
start: number;
duration: number;
/**
* The iframe media element this proxy mirrors, when adopted from the DOM.
* Its `data-start`/`data-duration` are re-read each tick so live timeline
* edits (trim/move) bound the proxy correctly. Null for URL-driven proxies.
*/
source?: HTMLMediaElement | null;
/**
* Count of consecutive steady-state samples in which the proxy's
* `currentTime` was found drifted beyond `MIRROR_DRIFT_THRESHOLD_SECONDS`.
@@ -118,11 +124,48 @@ export class ParentMediaManager {
for (const m of this._entries) m.el.playbackRate = rate;
}
playAll(): void {
for (const m of this._entries) {
if (!m.el.src) continue;
m.el.play().catch((err: unknown) => this._reportPlaybackError(err));
private _playEntry(m: ProxyEntry): void {
if (!m.el.src) return;
m.el.play().catch((err: unknown) => this._reportPlaybackError(err));
}
// Play only if the current playhead is inside the clip's (live) window, so
// bulk starts (playAll / adopt) don't blip audio for clips outside their
// window until the next mirrorTime tick gates them off.
private _playEntryIfActive(m: ProxyEntry): void {
this._refreshEntryBounds(m);
const relTime = this._getCurrentTime() - m.start;
if (relTime < 0 || relTime >= m.duration) return;
this._playEntry(m);
}
// Re-read the source clip's live timing so trims/moves bound the proxy
// (adopt-time values go stale when the timeline is edited).
private _refreshEntryBounds(m: ProxyEntry): void {
if (!m.source?.isConnected) return;
// Guard against a malformed (non-numeric) attribute parsing to NaN: an NaN
// duration makes every `relTime >= m.duration` window check false, so the
// gate never closes and the proxy plays past its clip end.
const start = parseFloat(m.source.getAttribute("data-start") || "0");
m.start = Number.isFinite(start) ? start : 0;
const duration = parseFloat(m.source.getAttribute("data-duration") || "");
m.duration = Number.isFinite(duration) && duration > 0 ? duration : Number.POSITIVE_INFINITY;
}
// Pause the proxy outside its clip window; resume it on re-entry during
// parent-owned playback. Returns whether the proxy is within the window.
private _gateEntryPlayback(m: ProxyEntry, relTime: number): boolean {
if (relTime < 0 || relTime >= m.duration) {
if (!m.el.paused) m.el.pause();
m.driftSamples = 0;
return false;
}
if (this._audioOwner === "parent" && !this._isPaused() && m.el.paused) this._playEntry(m);
return true;
}
playAll(): void {
for (const m of this._entries) this._playEntryIfActive(m);
}
pauseAll(): void {
@@ -131,6 +174,9 @@ export class ParentMediaManager {
seekAll(timeInSeconds: number): void {
for (const m of this._entries) {
// Re-read live bounds so a trim/move just before a paused scrub gates and
// positions against the current clip window, not the adopt-time one.
this._refreshEntryBounds(m);
const relTime = timeInSeconds - m.start;
if (relTime >= 0 && relTime < m.duration) m.el.currentTime = relTime;
}
@@ -145,11 +191,9 @@ export class ParentMediaManager {
mirrorTime(timelineSeconds: number, options?: { force?: boolean }): void {
const force = options?.force === true;
for (const m of this._entries) {
this._refreshEntryBounds(m);
const relTime = timelineSeconds - m.start;
if (relTime < 0 || relTime >= m.duration) {
m.driftSamples = 0;
continue;
}
if (!this._gateEntryPlayback(m, relTime)) continue;
if (Math.abs(m.el.currentTime - relTime) > MIRROR_DRIFT_THRESHOLD_SECONDS) {
m.driftSamples += 1;
if (force || m.driftSamples >= MIRROR_REQUIRED_CONSECUTIVE_DRIFT_SAMPLES) {
@@ -275,6 +319,7 @@ export class ParentMediaManager {
tag: "audio" | "video",
start: number,
duration: number,
source?: HTMLMediaElement | null,
): ProxyEntry | null {
if (this._entries.some((m) => m.el.src === src)) return null;
@@ -287,7 +332,7 @@ export class ParentMediaManager {
const rate = this._getPlaybackRate();
if (rate !== 1) el.playbackRate = rate;
const entry: ProxyEntry = { el, start, duration, driftSamples: 0 };
const entry: ProxyEntry = { el, start, duration, driftSamples: 0, source };
this._entries.push(entry);
return entry;
}
@@ -312,15 +357,13 @@ export class ParentMediaManager {
const duration = parseFloat(iframeEl.getAttribute("data-duration") || "Infinity");
const tag = iframeEl.tagName === "VIDEO" ? ("video" as const) : ("audio" as const);
const created = this._createEntry(src, tag, start, duration);
const created = this._createEntry(src, tag, start, duration, iframeEl);
// If already under parent ownership and playing, the new proxy must catch
// up immediately — bypass the jitter-coalescing gate.
if (created && this._audioOwner === "parent") {
this.mirrorTime(this._getCurrentTime(), { force: true });
if (!this._isPaused() && created.el.src) {
created.el.play().catch((err: unknown) => this._reportPlaybackError(err));
}
if (!this._isPaused()) this._playEntryIfActive(created);
}
}
@@ -22,6 +22,49 @@ export function normalizeCompositionSrc(
return compSrc;
}
/** Resolve a media src to its project-relative preview path, or null. */
function resolvePreviewRelative(src: string | undefined, pid: string): string | null {
if (!src) return null;
if (!src.startsWith("http")) return src;
const base = `/api/projects/${pid}/preview/`;
const idx = src.indexOf(base);
return idx !== -1 ? decodeURIComponent(src.slice(idx + base.length)) : null;
}
/**
* The trimmed source slice as start/end fractions (01) of the source, so the
* waveform can window its peaks to the clip edges. Undefined when the source
* length is unknown (renders full).
*/
function trimFractions(el: TimelineElement): { start?: number; end?: number } {
const sourceDur = el.sourceDuration;
if (sourceDur == null || sourceDur <= 0) return {};
const mediaStart = el.playbackStart ?? 0;
const rate = el.playbackRate ?? 1;
const start = Math.max(0, Math.min(1, mediaStart / sourceDur));
const end = Math.max(start, Math.min(1, (mediaStart + el.duration * rate) / sourceDur));
return { start, end };
}
/**
* Build the waveform element for an audio clip, windowing the rendered peaks to
* the trimmed source slice so the bars track the clip edges.
*/
function renderAudioClip(el: TimelineElement, pid: string, labelColor: string): ReactNode {
const srcRelative = resolvePreviewRelative(el.src, pid);
const audioUrl = srcRelative ? `/api/projects/${pid}/preview/${srcRelative}` : (el.src ?? "");
const waveformUrl = srcRelative ? `/api/projects/${pid}/waveform/${srcRelative}` : undefined;
const { start, end } = trimFractions(el);
return createElement(AudioWaveform, {
audioUrl,
waveformUrl,
label: getTimelineElementLabel(el),
labelColor,
trimStartFraction: start,
trimEndFraction: end,
});
}
interface UseRenderClipContentOptions {
projectIdRef: { current: string | null };
compIdToSrc: Map<string, string>;
@@ -36,6 +79,8 @@ export function useRenderClipContent({
effectiveTimelineDuration,
}: UseRenderClipContentOptions) {
return useCallback(
// Pre-existing clip-content dispatcher; reduced by extracting renderAudioClip.
// fallow-ignore-next-line complexity
(el: TimelineElement, style: { clip: string; label: string }): ReactNode => {
const pid = projectIdRef.current;
if (!pid) return null;
@@ -88,27 +133,7 @@ export function useRenderClipContent({
// Audio clips — waveform visualization
if (el.tag === "audio") {
const previewBase = `/api/projects/${pid}/preview/`;
const previewIdx = el.src?.startsWith("http") ? el.src.indexOf(previewBase) : -1;
const srcRelative = el.src
? previewIdx !== -1
? decodeURIComponent(el.src.slice(previewIdx + previewBase.length))
: el.src.startsWith("http")
? null
: el.src
: null;
const audioUrl = srcRelative
? `/api/projects/${pid}/preview/${srcRelative}`
: (el.src ?? "");
const waveformUrl = srcRelative
? `/api/projects/${pid}/waveform/${srcRelative}`
: undefined;
return createElement(AudioWaveform, {
audioUrl,
waveformUrl,
label: getTimelineElementLabel(el),
labelColor: style.label,
});
return renderAudioClip(el, pid, style.label);
}
if ((el.tag === "video" || el.tag === "img") && el.src) {
@@ -143,10 +143,21 @@ export function useTimelineEditing({
element: TimelineElement,
updates: Pick<TimelineElement, "start" | "duration" | "playbackStart">,
) => {
patchIframeDomTiming(previewIframeRef.current, element, [
const liveAttrs: Array<[string, string]> = [
["data-start", formatTimelineAttributeNumber(updates.start)],
["data-duration", formatTimelineAttributeNumber(updates.duration)],
]);
];
// A start-edge trim advances the media-start offset (skips into the
// source). Patch it live too — otherwise the iframe keeps the old offset
// and the clip only repositions instead of trimming the audio.
if (updates.playbackStart != null) {
const liveAttr =
element.playbackStartAttr === "playback-start"
? "data-playback-start"
: "data-media-start";
liveAttrs.push([liveAttr, formatTimelineAttributeNumber(updates.playbackStart)]);
}
patchIframeDomTiming(previewIframeRef.current, element, liveAttrs);
return enqueueEdit(element, "Resize timeline clip", (original, target) => {
const pbs = resolveResizePlaybackStart(original, target, element, updates);
let patched = applyPatchByTarget(original, target, {
@@ -173,6 +184,8 @@ export function useTimelineEditing({
);
const handleTimelineElementDelete = useCallback(
// Pre-existing handler complexity, unchanged by this PR.
// fallow-ignore-next-line complexity
async (element: TimelineElement) => {
if (isRecordingRef?.current) {
showToast("Cannot edit timeline while recording", "error");
@@ -247,6 +260,8 @@ export function useTimelineEditing({
);
const handleTimelineAssetDrop = useCallback(
// Pre-existing handler complexity, unchanged by this PR.
// fallow-ignore-next-line complexity
async (
assetPath: string,
placement: Pick<TimelineElement, "start" | "track">,
@@ -329,6 +344,8 @@ export function useTimelineEditing({
);
const handleTimelineFileDrop = useCallback(
// Pre-existing handler complexity, unchanged by this PR.
// fallow-ignore-next-line complexity
async (files: File[], placement?: Pick<TimelineElement, "start" | "track">) => {
if (isRecordingRef?.current) {
showToast("Cannot edit timeline while recording", "error");
@@ -5,6 +5,17 @@ interface AudioWaveformProps {
waveformUrl?: string;
label: string;
labelColor: string;
/**
* Fraction (01) of the source the clip starts at, after the media-start
* trim. Defaults to 0 (no front trim).
*/
trimStartFraction?: number;
/**
* Fraction (01) of the source the clip ends at. Defaults to 1 (no tail
* trim). Together these window the rendered peaks to the trimmed slice so the
* waveform tracks the clip edges instead of squeezing the whole file in.
*/
trimEndFraction?: number;
}
const BAR_W = 2;
@@ -62,6 +73,8 @@ export const AudioWaveform = memo(function AudioWaveform({
waveformUrl,
label,
labelColor,
trimStartFraction,
trimEndFraction,
}: AudioWaveformProps) {
const containerRef = useRef<HTMLDivElement | null>(null);
const barsRef = useRef<HTMLDivElement | null>(null);
@@ -116,20 +129,28 @@ export const AudioWaveform = memo(function AudioWaveform({
const barsEl = barsRef.current;
if (!container || !barsEl || !peaks) return;
// Window the peaks to the trimmed slice [start, end) of the source so the
// bars track the clip edges. Clamp to a valid, non-empty range.
const winStart = Math.max(0, Math.min(1, trimStartFraction ?? 0));
const winEnd = Math.max(winStart, Math.min(1, trimEndFraction ?? 1));
const lo = Math.floor(winStart * peaks.length);
const hi = Math.max(lo + 1, Math.ceil(winEnd * peaks.length));
const span = hi - lo;
const w = container.clientWidth || 400;
const barCount = Math.min(Math.floor(w / STEP), peaks.length);
const barCount = Math.min(Math.floor(w / STEP), span);
let html = "";
for (let i = 0; i < barCount; i++) {
// Map bar index to peak index (resample)
const peakIdx = Math.floor((i / barCount) * peaks.length);
// Map bar index to peak index within the windowed range (resample)
const peakIdx = lo + Math.floor((i / barCount) * span);
const amp = peaks[peakIdx] ?? 0;
const pct = Math.max(3, Math.round(amp * 100));
const opacity = (0.45 + amp * 0.4).toFixed(2);
html += `<div style="position:absolute;bottom:0;left:${i * STEP}px;width:${BAR_W}px;height:${pct}%;background:rgba(75,163,210,${opacity})"></div>`;
}
barsEl.innerHTML = html;
}, [peaks]);
}, [peaks, trimStartFraction, trimEndFraction]);
// Observe container size and redraw
const setContainerRef = useCallback(