mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 23:03:09 +00:00
feat(studio,cli): music beat detection with timeline guides + headless beats CLI (#1424)
* feat(studio,cli): music beat detection with timeline guides + headless beats CLI Beat detection for music tracks: the Studio draws beat guides on the active track, beats are user-editable and persist to a project file, and a new `hyperframes beats` CLI generates that file headlessly before the Studio opens. Detection lives in @hyperframes/core/beats (shared by Studio + CLI): an energy onset detector cross-validated with bpm-detective, regularized to an octave- aligned grid, silence-gated, with per-beat loudness. Music-only — an <audio data-timeline-role="music"> is analyzed; voiceover is excluded. Studio: green beat lines + draggable dots on the selected track; add at playhead, drag to move, double-click to delete (audio scrubs); edits persist to beats/<audio>.json and are undoable (interleaved with file history). CLI: `hyperframes beats [dir]` runs the same detection in headless Chrome (prebuilt browser bundle in dist) and writes the beat file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): timeline beat-grid + zoom UX refinements - Center-anchored magnify: zooming via the toolbar/slider keeps the time at the viewport center fixed instead of anchoring at the left. Pinch still anchors at the cursor. - Move-snap to beats: dragging a clip snaps whichever edge (start or end) is nearest a beat, matching the existing resize-edge snapping. - Beat lines on track backgrounds: faint full-height beat lines now paint behind the clips on every track lane (brightness scales with loudness); the green dots stay on the active track's top bar. - Waveform follows zoom: bars fill the full clip width and resample the windowed peaks, so the waveform stretches with zoom instead of stopping partway across a widened clip. - Beat dots centered in the top bar: align the dot band to the clip top (CLIP_Y) so the dots sit centered in the dark bar instead of being bisected by the clip's top border. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(studio): preserve media sourceDuration across element re-derivation Moving a non-music clip re-derived the timeline elements into fresh objects whose sourceDuration the DOM scan hadn't loaded yet. The async probe skips srcs already in its cache, so the value was silently dropped — trimFractions then returned no window and the trimmed music waveform reset to the full source pinned at the track start. Re-apply the cached probe duration synchronously on every derivation (applyCachedSourceDurations) and extract the async probe loop into probeMissingSourceDurations to keep useTimelinePlayer within the file size limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): skip beat-snap on the music track, highlight move-snap target The music track defines the beats, so moving or trimming it no longer snaps to its own beats (isMusicTrack guard on both the move and resize snap paths). Moving another clip snapped only on drop with no cue. snapMoveStartToBeat now also returns the beat it will snap to; BeatBackgroundLines draws that beat's line as a bright neon-green glow while the clip's edge is within the snap region, so the target is visible before drop. Also drops .commitmsg.tmp, accidentally committed via git add -A. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): hide playhead while dragging a beat; default beat dots to music track - Dragging a beat dot now hides the playhead guideline (new beatDragging store flag set on beat pointer down/up) so its line doesn't track the scrub and clutter the beat being moved. - Beat dots render on the selected track, falling back to the music track when nothing is selected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(core): remove polynomial-ReDoS regex from audioRelPathForSrc CodeQL js/polynomial-redos: the lazy `.+?` followed by an optional trailing `[?#].*$` backtracks polynomially on crafted `/preview/...` inputs. Parse the preview-relative path with indexOf/slice instead, and strip the query/hash with a single linear char-class search. Behavior is unchanged for all preview/absolute/blob/data/bare inputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(studio,core,cli): review hardening for beat detection + timeline UX - playerStore.reset() now clears beat state (analysis, edits, undo/redo, persist) so a project switch can't apply the previous project's beats, undo stack, or file-writer to the new one. - removeUserBeat returns the same reference on a no-op, and delete/move beat actions skip committing when nothing changed — no more phantom undo entries / debounced writes for no-op edits. - regularizeBeats bails to raw onsets when the (octave-misread) tempo would produce a sub-125ms grid, avoiding a tens-of-thousands-of-beats freeze. - parseBeats clamps strength to [0,1] and rejects non-finite time/strength, so a hand-edited file can't feed NaN into the gamma curve (Math.pow on a negative base) and blank out beat markers. - Start-edge beat-snap now also requires duration >= minDuration, matching the end-edge guard, so a rightward snap can't collapse the clip. - Center-anchor zoom effect always consumes its skip flag, so a pinch that produced no pps change can't leave it stranded and skip the next zoom. - Headless beats analyzer projects to {beatTimes,beatStrengths,bpm,confidence} before returning, so page.evaluate no longer serializes the full decoded PCM (channelData) across the CDP boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(core): gate parseBeats on schema version parseBeats accepted any object with a beats array, so a future v2 beat file (with changed semantics) would be parsed silently as v1. Reject anything whose version is not 1, treating an unknown version like an absent/invalid file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> --------- Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
Miguel Ángel
parent
a95e49dbda
commit
d9f69f61e7
@@ -137,13 +137,17 @@ export const AudioWaveform = memo(function AudioWaveform({
|
||||
const hi = Math.max(lo + 1, Math.ceil(winEnd * peaks.length));
|
||||
const span = hi - lo;
|
||||
|
||||
// Fill the full (possibly zoomed) clip width with STEP-spaced bars, resampling
|
||||
// the windowed peaks across them — upsampling (repeating peaks) when the clip
|
||||
// is wider than the slice has samples, so the waveform stretches with zoom
|
||||
// instead of stopping partway across.
|
||||
const w = container.clientWidth || 400;
|
||||
const barCount = Math.min(Math.floor(w / STEP), span);
|
||||
const barCount = Math.max(0, Math.floor(w / STEP));
|
||||
|
||||
let html = "";
|
||||
for (let i = 0; i < barCount; i++) {
|
||||
// Map bar index to peak index within the windowed range (resample)
|
||||
const peakIdx = lo + Math.floor((i / barCount) * span);
|
||||
const peakIdx = lo + Math.min(span - 1, Math.floor((i / barCount) * span));
|
||||
const amp = peaks[peakIdx] ?? 0;
|
||||
const pct = Math.max(3, Math.round(amp * 100));
|
||||
const opacity = (0.45 + amp * 0.4).toFixed(2);
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
import { memo, useRef, useState } from "react";
|
||||
import { moveBeatCompositionTime, deleteBeatAtCompositionTime } from "../../utils/beatEditActions";
|
||||
import { usePlayerStore } from "../store/playerStore";
|
||||
import { CLIP_Y } from "./timelineLayout";
|
||||
|
||||
const BEAT_BAND_H = 14; // dark band height at top of track
|
||||
const BEAT_HIT_W = 12; // grab width per beat (px)
|
||||
|
||||
/** Hide both layers when beats are packed tighter than this (px) — too dense to read. */
|
||||
function beatsTooDense(beatTimes: number[], pps: number): boolean {
|
||||
if (beatTimes.length < 2) return true;
|
||||
const avgInterval = (beatTimes[beatTimes.length - 1]! - beatTimes[0]!) / (beatTimes.length - 1);
|
||||
return avgInterval * pps < 5;
|
||||
}
|
||||
|
||||
/**
|
||||
* Faint full-height beat lines painted into a track lane's background. Rendered
|
||||
* behind the clips so they only show through the empty track area (the dots in
|
||||
* BeatStrip mark beats on the clips themselves). Brightness scales with beat
|
||||
* loudness. Drawn on every track lane for a global beat grid.
|
||||
*/
|
||||
export const BeatBackgroundLines = memo(function BeatBackgroundLines({
|
||||
beatTimes,
|
||||
beatStrengths,
|
||||
pps,
|
||||
highlightTime,
|
||||
}: {
|
||||
beatTimes: number[] | undefined;
|
||||
beatStrengths: number[] | undefined;
|
||||
pps: number;
|
||||
/** Beat time a dragged clip will snap to — drawn as a bright neon line. */
|
||||
highlightTime?: number | null;
|
||||
}) {
|
||||
if (!beatTimes || beatsTooDense(beatTimes, pps)) return null;
|
||||
return (
|
||||
<div className="absolute inset-0 pointer-events-none" style={{ zIndex: 0 }}>
|
||||
{beatTimes.map((t, i) => {
|
||||
const isHighlight = highlightTime != null && Math.abs(t - highlightTime) < 1e-3;
|
||||
const strength = Math.pow(Math.min(1, beatStrengths?.[i] ?? 0.5), 2.2);
|
||||
const opacity = isHighlight ? 1 : 0.06 + strength * 0.16;
|
||||
return (
|
||||
<div
|
||||
key={`${t}-${i}`}
|
||||
className="absolute top-0 bottom-0"
|
||||
style={{
|
||||
left: t * pps,
|
||||
width: isHighlight ? 2 : 1,
|
||||
background: `rgba(34,197,94,${opacity.toFixed(3)})`,
|
||||
boxShadow: isHighlight ? "0 0 6px rgba(34,197,94,0.9)" : undefined,
|
||||
zIndex: isHighlight ? 1 : undefined,
|
||||
}}
|
||||
/>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
});
|
||||
|
||||
/**
|
||||
* Green beat dots on the music track's row. Drag a dot to move its beat,
|
||||
* double-click to delete; both scrub the audio. Dot size/brightness scale with
|
||||
* beat loudness (gamma-curved for contrast).
|
||||
*/
|
||||
export const BeatStrip = memo(function BeatStrip({
|
||||
beatTimes,
|
||||
beatStrengths,
|
||||
pps,
|
||||
}: {
|
||||
beatTimes: number[] | undefined;
|
||||
beatStrengths: number[] | undefined;
|
||||
pps: number;
|
||||
}) {
|
||||
// Active drag: which beat and how far (px) it's been dragged.
|
||||
const [drag, setDrag] = useState<{ index: number; dx: number } | null>(null);
|
||||
const dragRef = useRef<{ index: number; startX: number; origTime: number } | null>(null);
|
||||
|
||||
if (!beatTimes || beatsTooDense(beatTimes, pps)) return null;
|
||||
const cy = BEAT_BAND_H / 2;
|
||||
|
||||
return (
|
||||
<div
|
||||
className="absolute left-0 right-0 pointer-events-none"
|
||||
style={{ top: CLIP_Y, height: BEAT_BAND_H, background: "rgba(0,0,0,0.28)", zIndex: 11 }}
|
||||
>
|
||||
{beatTimes.map((t, i) => {
|
||||
// Louder beats → larger, brighter dot. Gamma curve widens the contrast.
|
||||
const strength = Math.pow(Math.min(1, beatStrengths?.[i] ?? 0.5), 2.2);
|
||||
const r = 1.5 + strength * 2.5;
|
||||
const opacity = 0.25 + strength * 0.75;
|
||||
const dxPx = drag?.index === i ? drag.dx : 0;
|
||||
const x = t * pps + dxPx;
|
||||
return (
|
||||
<div
|
||||
key={`${t}-${i}`}
|
||||
className="absolute select-none"
|
||||
title="Drag to move · double-click to delete"
|
||||
draggable={false}
|
||||
style={{
|
||||
left: x - BEAT_HIT_W / 2,
|
||||
top: 0,
|
||||
width: BEAT_HIT_W,
|
||||
height: BEAT_BAND_H,
|
||||
cursor: "ew-resize",
|
||||
pointerEvents: "auto",
|
||||
touchAction: "none",
|
||||
}}
|
||||
onPointerDown={(e) => {
|
||||
// preventDefault stops the browser starting a native text/drag
|
||||
// selection (which otherwise "selects" the whole panel mid-drag).
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
e.currentTarget.setPointerCapture(e.pointerId);
|
||||
dragRef.current = { index: i, startX: e.clientX, origTime: t };
|
||||
setDrag({ index: i, dx: 0 });
|
||||
usePlayerStore.getState().setBeatDragging(true); // hide the playhead guideline
|
||||
usePlayerStore.getState().requestSeek(Math.max(0, t)); // scrub audio at beat
|
||||
}}
|
||||
onPointerMove={(e) => {
|
||||
const d = dragRef.current;
|
||||
if (!d || d.index !== i) return;
|
||||
e.preventDefault();
|
||||
const dx = e.clientX - d.startX;
|
||||
setDrag({ index: i, dx });
|
||||
// Scrub the audio (and move the playhead) to follow the dragged beat.
|
||||
usePlayerStore.getState().requestSeek(Math.max(0, d.origTime + dx / pps));
|
||||
}}
|
||||
onPointerUp={(e) => {
|
||||
const d = dragRef.current;
|
||||
dragRef.current = null;
|
||||
setDrag(null);
|
||||
usePlayerStore.getState().setBeatDragging(false);
|
||||
if (e.currentTarget.hasPointerCapture?.(e.pointerId)) {
|
||||
e.currentTarget.releasePointerCapture(e.pointerId);
|
||||
}
|
||||
if (!d || d.index !== i) return;
|
||||
const dx = e.clientX - d.startX;
|
||||
if (Math.abs(dx) > 2) {
|
||||
const newTime = Math.max(0, d.origTime + dx / pps);
|
||||
moveBeatCompositionTime(d.origTime, newTime);
|
||||
usePlayerStore.getState().requestSeek(newTime); // park scrubber at new beat
|
||||
}
|
||||
}}
|
||||
onDoubleClick={(e) => {
|
||||
e.stopPropagation();
|
||||
deleteBeatAtCompositionTime(t);
|
||||
usePlayerStore.getState().requestSeek(Math.max(0, t)); // park scrubber at deleted beat
|
||||
}}
|
||||
>
|
||||
<div
|
||||
className="absolute"
|
||||
style={{
|
||||
left: BEAT_HIT_W / 2 - r,
|
||||
top: cy - r,
|
||||
width: r * 2,
|
||||
height: r * 2,
|
||||
borderRadius: "50%",
|
||||
background: `rgba(34,197,94,${opacity.toFixed(3)})`,
|
||||
pointerEvents: "none",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
});
|
||||
@@ -1,4 +1,7 @@
|
||||
import { useRef, useMemo, useCallback, useState, useEffect, memo, type ReactNode } from "react";
|
||||
import { useMusicBeatAnalysis } from "../../hooks/useMusicBeatAnalysis";
|
||||
import { isMusicTrack } from "../../utils/timelineInspector";
|
||||
import { remapBeatAnalysisToComposition } from "../../utils/beatEditActions";
|
||||
import { usePlayerStore, type TimelineElement } from "../store/playerStore";
|
||||
import { useMountEffect } from "../../hooks/useMountEffect";
|
||||
import { EditPopover } from "./EditModal";
|
||||
@@ -78,7 +81,17 @@ export const Timeline = memo(function Timeline({
|
||||
onMoveKeyframe,
|
||||
} = useTimelineEditContext();
|
||||
const theme = useMemo(() => ({ ...defaultTimelineTheme, ...themeOverrides }), [themeOverrides]);
|
||||
useMusicBeatAnalysis();
|
||||
const elements = usePlayerStore((s) => s.elements);
|
||||
const beatAnalysis = usePlayerStore((s) => s.beatAnalysis);
|
||||
const musicElement = usePlayerStore((s) => s.elements.find(isMusicTrack) ?? null);
|
||||
|
||||
// Merge user edits + remap beats from audio-file → composition coordinates.
|
||||
const beatEdits = usePlayerStore((s) => s.beatEdits);
|
||||
const adjustedBeatAnalysis = useMemo(
|
||||
() => remapBeatAnalysisToComposition(beatAnalysis, musicElement, beatEdits),
|
||||
[beatAnalysis, musicElement, beatEdits],
|
||||
);
|
||||
const duration = usePlayerStore((s) => s.duration);
|
||||
const timelineReady = usePlayerStore((s) => s.timelineReady);
|
||||
const selectedElementId = usePlayerStore((s) => s.selectedElementId);
|
||||
@@ -439,6 +452,7 @@ export const Timeline = memo(function Timeline({
|
||||
keyframeCache={keyframeCache}
|
||||
selectedKeyframes={selectedKeyframes}
|
||||
currentTime={currentTime}
|
||||
beatAnalysis={adjustedBeatAnalysis}
|
||||
onClickKeyframe={(el, pct) => {
|
||||
usePlayerStore.getState().clearSelectedKeyframes();
|
||||
const elKey = el.key ?? el.id;
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import { memo, type ReactNode } from "react";
|
||||
import { BeatStrip, BeatBackgroundLines } from "./BeatStrip";
|
||||
import { TimelineClip } from "./TimelineClip";
|
||||
import { TimelineClipDiamonds } from "./TimelineClipDiamonds";
|
||||
import { TimelineRuler } from "./TimelineRuler";
|
||||
import type { MusicBeatAnalysis } from "@hyperframes/core/beats";
|
||||
import { PlayheadIndicator } from "./PlayheadIndicator";
|
||||
import {
|
||||
getTimelineEditCapabilities,
|
||||
@@ -20,6 +22,7 @@ import type { TrackVisualStyle } from "./timelineIcons";
|
||||
import { STUDIO_KEYFRAMES_ENABLED } from "../../components/editor/manualEditingAvailability";
|
||||
import { SPLIT_BOUNDARY_EPSILON_S } from "../../utils/timelineElementSplit";
|
||||
import { useTimelineEditContext } from "../../contexts/TimelineEditContext";
|
||||
import { isMusicTrack } from "../../utils/timelineInspector";
|
||||
|
||||
function ClipLabel({ element, color }: { element: TimelineElement; color: string }) {
|
||||
const lint = usePlayerStore((s) => s.lintFindingsByElement.get(element.key ?? element.id));
|
||||
@@ -91,6 +94,7 @@ interface TimelineCanvasProps {
|
||||
onDragKeyframe?: (element: TimelineElement, oldPct: number, newPct: number) => void;
|
||||
onContextMenuKeyframe?: (e: React.MouseEvent, elementId: string, percentage: number) => void;
|
||||
onContextMenuClip?: (e: React.MouseEvent, element: TimelineElement) => void;
|
||||
beatAnalysis?: MusicBeatAnalysis | null;
|
||||
}
|
||||
|
||||
export const TimelineCanvas = memo(function TimelineCanvas({
|
||||
@@ -138,9 +142,11 @@ export const TimelineCanvas = memo(function TimelineCanvas({
|
||||
onDragKeyframe,
|
||||
onContextMenuKeyframe,
|
||||
onContextMenuClip,
|
||||
beatAnalysis,
|
||||
}: TimelineCanvasProps) {
|
||||
const { onResizeElement, onMoveElement, onRazorSplit, onRazorSplitAll } =
|
||||
useTimelineEditContext();
|
||||
const beatDragging = usePlayerStore((s) => s.beatDragging);
|
||||
const draggedElement = draggedClip?.element ?? null;
|
||||
const activeDraggedElement =
|
||||
draggedClip?.started === true && draggedElement
|
||||
@@ -197,6 +203,7 @@ export const TimelineCanvas = memo(function TimelineCanvas({
|
||||
shiftHeld={shiftHeld}
|
||||
rangeSelection={rangeSelection}
|
||||
theme={theme}
|
||||
beatAnalysis={beatAnalysis}
|
||||
/>
|
||||
|
||||
{displayTrackOrder.map((trackNum) => {
|
||||
@@ -237,6 +244,25 @@ export const TimelineCanvas = memo(function TimelineCanvas({
|
||||
</div>
|
||||
</div>
|
||||
<div style={{ width: trackContentWidth }} className="relative">
|
||||
{/* Faint beat lines in every track's background (behind the clips);
|
||||
the active move-snap target is highlighted. */}
|
||||
<BeatBackgroundLines
|
||||
beatTimes={beatAnalysis?.beatTimes}
|
||||
beatStrengths={beatAnalysis?.beatStrengths}
|
||||
pps={pps}
|
||||
highlightTime={draggedClip?.started ? draggedClip.snapBeatTime : null}
|
||||
/>
|
||||
{/* Beat dots on the active track (the one holding the selection),
|
||||
falling back to the music track when nothing is selected. */}
|
||||
{(selectedElementId
|
||||
? els.some((e) => (e.key ?? e.id) === selectedElementId)
|
||||
: els.some(isMusicTrack)) && (
|
||||
<BeatStrip
|
||||
beatTimes={beatAnalysis?.beatTimes}
|
||||
beatStrengths={beatAnalysis?.beatStrengths}
|
||||
pps={pps}
|
||||
/>
|
||||
)}
|
||||
{isPendingTrack && (
|
||||
<div
|
||||
className="absolute inset-0 flex items-center"
|
||||
@@ -351,6 +377,7 @@ export const TimelineCanvas = memo(function TimelineCanvas({
|
||||
pointerOffsetY: e.clientY - rect.top,
|
||||
previewStart: el.start,
|
||||
previewTrack: el.track,
|
||||
snapBeatTime: null,
|
||||
started: false,
|
||||
});
|
||||
syncClipDragAutoScroll(e.clientX, e.clientY);
|
||||
@@ -472,11 +499,16 @@ export const TimelineCanvas = memo(function TimelineCanvas({
|
||||
/>
|
||||
)}
|
||||
|
||||
{/* Playhead */}
|
||||
{/* Playhead — hidden while dragging a beat so its guideline doesn't
|
||||
track the scrub and clutter the beat being moved. */}
|
||||
<div
|
||||
ref={playheadRef}
|
||||
className="absolute top-0 bottom-0 pointer-events-none"
|
||||
style={{ left: `${GUTTER}px`, zIndex: 100 }}
|
||||
style={{
|
||||
left: `${GUTTER}px`,
|
||||
zIndex: 100,
|
||||
display: beatDragging ? "none" : undefined,
|
||||
}}
|
||||
>
|
||||
<PlayheadIndicator />
|
||||
</div>
|
||||
|
||||
@@ -2,6 +2,7 @@ import { memo } from "react";
|
||||
import type { TimelineTheme } from "./timelineTheme";
|
||||
import type { TimelineRangeSelection } from "./timelineEditing";
|
||||
import { GUTTER, RULER_H, formatTimelineTickLabel } from "./timelineLayout";
|
||||
import type { MusicBeatAnalysis } from "@hyperframes/core/beats";
|
||||
|
||||
interface TimelineRulerProps {
|
||||
major: number[];
|
||||
@@ -14,6 +15,7 @@ interface TimelineRulerProps {
|
||||
shiftHeld: boolean;
|
||||
rangeSelection: TimelineRangeSelection | null;
|
||||
theme: TimelineTheme;
|
||||
beatAnalysis?: MusicBeatAnalysis | null;
|
||||
}
|
||||
|
||||
export const TimelineRuler = memo(function TimelineRuler({
|
||||
@@ -27,13 +29,25 @@ export const TimelineRuler = memo(function TimelineRuler({
|
||||
shiftHeld,
|
||||
rangeSelection,
|
||||
theme,
|
||||
beatAnalysis,
|
||||
}: TimelineRulerProps) {
|
||||
const beatTimes = beatAnalysis?.beatTimes ?? [];
|
||||
const beatStrengths = beatAnalysis?.beatStrengths ?? [];
|
||||
|
||||
// Only draw beat lines when they'd be at least 5px apart
|
||||
const avgBeatInterval =
|
||||
beatTimes.length > 1
|
||||
? (beatTimes[beatTimes.length - 1]! - beatTimes[0]!) / (beatTimes.length - 1)
|
||||
: null;
|
||||
const showBeats = avgBeatInterval !== null && avgBeatInterval * pps >= 5;
|
||||
|
||||
return (
|
||||
<>
|
||||
{/* Grid lines */}
|
||||
{/* Grid lines (major ticks + beat lines) — behind the tracks (background).
|
||||
Opaque track rows hide them; only the beat dots show on tracks. */}
|
||||
<svg
|
||||
className="absolute pointer-events-none"
|
||||
style={{ left: GUTTER, width: trackContentWidth }}
|
||||
style={{ left: GUTTER, width: trackContentWidth, zIndex: 0 }}
|
||||
height={totalH}
|
||||
>
|
||||
{major.map((t) => {
|
||||
@@ -50,6 +64,24 @@ export const TimelineRuler = memo(function TimelineRuler({
|
||||
/>
|
||||
);
|
||||
})}
|
||||
{showBeats &&
|
||||
beatTimes.map((t, i) => {
|
||||
const x = t * pps;
|
||||
// Louder beats → brighter line. Gamma curve widens the contrast.
|
||||
const strength = Math.pow(Math.min(1, beatStrengths[i] ?? 0.5), 2.2);
|
||||
const opacity = 0.08 + strength * 0.62;
|
||||
return (
|
||||
<line
|
||||
key={`b-${t}-${i}`}
|
||||
x1={x}
|
||||
y1={0}
|
||||
x2={x}
|
||||
y2={totalH}
|
||||
stroke={`rgba(34, 197, 94, ${opacity.toFixed(3)})`}
|
||||
strokeWidth="1"
|
||||
/>
|
||||
);
|
||||
})}
|
||||
</svg>
|
||||
|
||||
{/* Ruler */}
|
||||
@@ -64,11 +96,13 @@ export const TimelineRuler = memo(function TimelineRuler({
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{minor.map((t) => (
|
||||
<div key={`m-${t}`} className="absolute bottom-0" style={{ left: t * pps }}>
|
||||
<div className="w-px h-[3px]" style={{ background: theme.tickMinor }} />
|
||||
</div>
|
||||
))}
|
||||
|
||||
{major.map((t) => (
|
||||
<div
|
||||
key={`M-${t}`}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { useRef, useState, useCallback } from "react";
|
||||
import { useRef, useState, useCallback, useMemo } from "react";
|
||||
import { useMountEffect } from "../../hooks/useMountEffect";
|
||||
import {
|
||||
resolveTimelineMove,
|
||||
@@ -9,6 +9,64 @@ import {
|
||||
import { usePlayerStore } from "../store/playerStore";
|
||||
import type { TimelineElement } from "../store/playerStore";
|
||||
import { TRACK_H } from "./timelineLayout";
|
||||
import { isMusicTrack } from "../../utils/timelineInspector";
|
||||
import { mergeUserBeats } from "../../utils/beatEditing";
|
||||
|
||||
const BEAT_SNAP_PX = 8;
|
||||
const EMPTY_BEAT_TIMES: number[] = [];
|
||||
|
||||
function snapToNearestBeat(time: number, beatTimes: number[], thresholdSecs: number): number {
|
||||
let best = time;
|
||||
let bestDist = thresholdSecs;
|
||||
for (const bt of beatTimes) {
|
||||
const d = Math.abs(bt - time);
|
||||
if (d < bestDist) {
|
||||
bestDist = d;
|
||||
best = bt;
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
/**
|
||||
* Snap a moved clip so whichever edge (start or end) is nearest a beat lands on
|
||||
* it, keeping the duration fixed. Returns the (clamped) start plus the beat time
|
||||
* it snapped to (for the grid-line highlight), or `beat: null` when no edge is
|
||||
* within threshold.
|
||||
*/
|
||||
function snapMoveStartToBeat(
|
||||
start: number,
|
||||
duration: number,
|
||||
beatTimes: number[],
|
||||
pixelsPerSecond: number,
|
||||
timelineDuration: number,
|
||||
): { start: number; beat: number | null } {
|
||||
if (beatTimes.length === 0) return { start, beat: null };
|
||||
const snapSecs = BEAT_SNAP_PX / Math.max(pixelsPerSecond, 1);
|
||||
const snappedStart = snapToNearestBeat(start, beatTimes, snapSecs);
|
||||
const snappedEnd = snapToNearestBeat(start + duration, beatTimes, snapSecs);
|
||||
const startMoved = snappedStart !== start;
|
||||
const endMoved = snappedEnd !== start + duration;
|
||||
|
||||
let candidate = start;
|
||||
let beat: number | null = null;
|
||||
if (
|
||||
startMoved &&
|
||||
(!endMoved || Math.abs(snappedStart - start) <= Math.abs(snappedEnd - (start + duration)))
|
||||
) {
|
||||
candidate = snappedStart;
|
||||
beat = snappedStart;
|
||||
} else if (endMoved) {
|
||||
candidate = snappedEnd - duration;
|
||||
beat = snappedEnd;
|
||||
}
|
||||
|
||||
const maxStart = Math.max(0, timelineDuration - duration);
|
||||
const clamped = Math.max(0, Math.min(maxStart, Math.round(candidate * 1000) / 1000));
|
||||
// If clamping pulled the clip off the snap target, drop the highlight.
|
||||
if (beat != null && Math.abs(clamped - candidate) > 1e-6) beat = null;
|
||||
return { start: clamped, beat };
|
||||
}
|
||||
|
||||
/* ── Shared state types ─────────────────────────────────────────── */
|
||||
export interface DraggedClipState {
|
||||
@@ -23,6 +81,8 @@ export interface DraggedClipState {
|
||||
pointerOffsetY: number;
|
||||
previewStart: number;
|
||||
previewTrack: number;
|
||||
/** Beat time the clip will snap to on drop, for the grid-line highlight. */
|
||||
snapBeatTime: number | null;
|
||||
started: boolean;
|
||||
}
|
||||
|
||||
@@ -76,6 +136,36 @@ export function useTimelineClipDrag({
|
||||
setRangeSelectionRef,
|
||||
}: UseTimelineClipDragInput) {
|
||||
const updateElement = usePlayerStore((s) => s.updateElement);
|
||||
const rawBeatTimes = usePlayerStore((s) => s.beatAnalysis?.beatTimes ?? EMPTY_BEAT_TIMES);
|
||||
const rawBeatStrengths = usePlayerStore((s) => s.beatAnalysis?.beatStrengths ?? EMPTY_BEAT_TIMES);
|
||||
const beatEdits = usePlayerStore((s) => s.beatEdits);
|
||||
const musicStart = usePlayerStore((s) => s.elements.find(isMusicTrack)?.start ?? 0);
|
||||
const musicPlaybackStart = usePlayerStore(
|
||||
(s) => s.elements.find(isMusicTrack)?.playbackStart ?? 0,
|
||||
);
|
||||
const musicDuration = usePlayerStore((s) => s.elements.find(isMusicTrack)?.duration ?? 0);
|
||||
const musicSrc = usePlayerStore((s) => s.elements.find(isMusicTrack)?.src ?? null);
|
||||
|
||||
const adjustedBeatTimes = useMemo(() => {
|
||||
if (rawBeatTimes === EMPTY_BEAT_TIMES || musicDuration === 0) return EMPTY_BEAT_TIMES;
|
||||
const merged = mergeUserBeats(rawBeatTimes, rawBeatStrengths, beatEdits, musicSrc);
|
||||
const clipEnd = musicPlaybackStart + musicDuration;
|
||||
const offset = musicStart - musicPlaybackStart;
|
||||
return merged.times
|
||||
.filter((t) => t >= musicPlaybackStart && t <= clipEnd)
|
||||
.map((t) => Math.round((t + offset) * 1000) / 1000);
|
||||
}, [
|
||||
rawBeatTimes,
|
||||
rawBeatStrengths,
|
||||
beatEdits,
|
||||
musicSrc,
|
||||
musicStart,
|
||||
musicPlaybackStart,
|
||||
musicDuration,
|
||||
]);
|
||||
|
||||
const beatTimesRef = useRef<number[]>([]);
|
||||
beatTimesRef.current = adjustedBeatTimes;
|
||||
|
||||
const [draggedClip, setDraggedClip] = useState<DraggedClipState | null>(null);
|
||||
const draggedClipRef = useRef<DraggedClipState | null>(null);
|
||||
@@ -118,13 +208,24 @@ export function useTimelineClipDrag({
|
||||
clientX,
|
||||
clientY,
|
||||
);
|
||||
// The music track defines the beats, so it must not snap to itself.
|
||||
const snap = isMusicTrack(drag.element)
|
||||
? { start: nextMove.start, beat: null }
|
||||
: snapMoveStartToBeat(
|
||||
nextMove.start,
|
||||
drag.element.duration,
|
||||
beatTimesRef.current,
|
||||
ppsRef.current,
|
||||
durationRef.current,
|
||||
);
|
||||
return {
|
||||
...drag,
|
||||
started: true,
|
||||
pointerClientX: clientX,
|
||||
pointerClientY: clientY,
|
||||
previewStart: nextMove.start,
|
||||
previewStart: snap.start,
|
||||
previewTrack: nextMove.track,
|
||||
snapBeatTime: snap.beat,
|
||||
};
|
||||
},
|
||||
[scrollRef, ppsRef, durationRef, trackOrderRef],
|
||||
@@ -220,14 +321,16 @@ export function useTimelineClipDrag({
|
||||
: Number.POSITIVE_INFINITY;
|
||||
const normalizedTag = resize.element.tag.toLowerCase();
|
||||
const canSeedPlaybackStart = normalizedTag === "audio" || normalizedTag === "video";
|
||||
const nextResize = resolveTimelineResize(
|
||||
const playbackRate = Math.max(resize.element.playbackRate ?? 1, 0.1);
|
||||
const maxEnd = Math.min(durationRef.current, resize.element.start + sourceRemaining);
|
||||
let nextResize = resolveTimelineResize(
|
||||
{
|
||||
start: resize.element.start,
|
||||
duration: resize.element.duration,
|
||||
originClientX: resize.originClientX,
|
||||
pixelsPerSecond: ppsRef.current,
|
||||
minStart: 0,
|
||||
maxEnd: Math.min(durationRef.current, resize.element.start + sourceRemaining),
|
||||
maxEnd,
|
||||
playbackStart:
|
||||
resize.edge === "start" && canSeedPlaybackStart
|
||||
? (resize.element.playbackStart ?? 0)
|
||||
@@ -238,6 +341,54 @@ export function useTimelineClipDrag({
|
||||
e.clientX,
|
||||
);
|
||||
|
||||
// Snap edge to beat grid when beat analysis is available. The snap must
|
||||
// stay inside the same limits resolveTimelineResize enforces, or it would
|
||||
// push the edge past the available source media / composition end.
|
||||
// The music track defines the beats, so it must not snap to itself.
|
||||
const beatTimes = beatTimesRef.current;
|
||||
if (beatTimes.length > 0 && !isMusicTrack(resize.element)) {
|
||||
const snapSecs = BEAT_SNAP_PX / Math.max(ppsRef.current, 1);
|
||||
if (resize.edge === "end") {
|
||||
const edgeTime = nextResize.start + nextResize.duration;
|
||||
const snapped = snapToNearestBeat(edgeTime, beatTimes, snapSecs);
|
||||
// Stay within [start+minDuration, maxEnd] so the snap can't create a
|
||||
// degenerate clip or run past the source/composition limit.
|
||||
const snappedDuration = Math.round((snapped - nextResize.start) * 1000) / 1000;
|
||||
if (snapped !== edgeTime && snapped <= maxEnd + 1e-6 && snappedDuration >= 0.05) {
|
||||
nextResize = { ...nextResize, duration: snappedDuration };
|
||||
}
|
||||
} else {
|
||||
const snapped = snapToNearestBeat(nextResize.start, beatTimes, snapSecs);
|
||||
const delta = nextResize.start - snapped; // >0 when snapping left
|
||||
// Leftward snap reveals more source; cap so playbackStart can't go < 0.
|
||||
const maxLeftDelta =
|
||||
nextResize.playbackStart != null
|
||||
? nextResize.playbackStart / playbackRate
|
||||
: Number.POSITIVE_INFINITY;
|
||||
// Also require the resulting duration to stay >= minDuration so a
|
||||
// rightward snap (delta < 0) can't collapse the clip to zero/negative.
|
||||
const snappedDuration = Math.round((nextResize.duration + delta) * 1000) / 1000;
|
||||
if (
|
||||
snapped !== nextResize.start &&
|
||||
snapped >= 0 &&
|
||||
delta <= maxLeftDelta + 1e-6 &&
|
||||
snappedDuration >= 0.05
|
||||
) {
|
||||
nextResize = {
|
||||
...nextResize,
|
||||
start: snapped,
|
||||
duration: snappedDuration,
|
||||
playbackStart:
|
||||
nextResize.playbackStart != null
|
||||
? Math.round(
|
||||
Math.max(0, nextResize.playbackStart - delta * playbackRate) * 1000,
|
||||
) / 1000
|
||||
: undefined,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setResizingClip((prev) =>
|
||||
prev
|
||||
? {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { useRef, useCallback, useEffect } from "react";
|
||||
import { useRef, useCallback, useEffect, useLayoutEffect } from "react";
|
||||
import { liveTime, usePlayerStore, type ZoomMode } from "../store/playerStore";
|
||||
import { useMountEffect } from "../../hooks/useMountEffect";
|
||||
import { getPinchTimelineZoomPercent } from "./timelineZoom";
|
||||
@@ -54,6 +54,33 @@ export function useTimelinePlayhead({
|
||||
}: UseTimelinePlayheadInput) {
|
||||
const dragScrollRaf = useRef(0);
|
||||
const previousZoomModeRef = useRef<ZoomMode | null>(zoomMode);
|
||||
// Center-anchored magnify: keep the time at the viewport center fixed when
|
||||
// the zoom level (pps) changes via the toolbar / slider. The pinch handler
|
||||
// anchors at the cursor instead, so it opts out via `skipCenterAnchorRef`.
|
||||
const previousAnchorPpsRef = useRef(pps);
|
||||
const skipCenterAnchorRef = useRef(false);
|
||||
|
||||
useLayoutEffect(() => {
|
||||
const scroll = scrollRef.current;
|
||||
const prevPps = previousAnchorPpsRef.current;
|
||||
previousAnchorPpsRef.current = pps;
|
||||
// Always consume the skip flag, even when pps didn't change — otherwise a
|
||||
// pinch that produced no pps change (already at the zoom clamp) would strand
|
||||
// it true and the next toolbar zoom would wrongly skip center-anchoring.
|
||||
const skip = skipCenterAnchorRef.current;
|
||||
skipCenterAnchorRef.current = false;
|
||||
if (!scroll || pps === prevPps || skip) return;
|
||||
const nextScrollLeft = getTimelineScrollLeftForZoomAnchor({
|
||||
pointerX: scroll.clientWidth / 2,
|
||||
currentScrollLeft: scroll.scrollLeft,
|
||||
gutter: GUTTER,
|
||||
currentPixelsPerSecond: prevPps,
|
||||
nextPixelsPerSecond: pps,
|
||||
duration: durationRef.current,
|
||||
});
|
||||
const maxScrollLeft = Math.max(0, scroll.scrollWidth - scroll.clientWidth);
|
||||
scroll.scrollLeft = Math.max(0, Math.min(maxScrollLeft, nextScrollLeft));
|
||||
}, [pps, scrollRef, durationRef]);
|
||||
|
||||
const syncPlayheadPosition = useCallback(
|
||||
(time: number) => {
|
||||
@@ -169,6 +196,8 @@ export function useTimelinePlayhead({
|
||||
nextPixelsPerSecond: nextPps,
|
||||
duration: durationRef.current,
|
||||
});
|
||||
// Pinch anchors at the cursor (below), so skip the center-anchor effect.
|
||||
skipCenterAnchorRef.current = true;
|
||||
setZoomMode("manual");
|
||||
setManualZoomPercent(nextZoomPercent);
|
||||
requestAnimationFrame(() => {
|
||||
|
||||
@@ -41,9 +41,30 @@ import {
|
||||
setPreviewPlaybackRate,
|
||||
shouldMutePreviewAudio,
|
||||
} from "../lib/timelineIframeHelpers";
|
||||
import { probeMediaUrl, getCachedProbe } from "../lib/mediaProbe";
|
||||
import { scrubMusicAtSeek, stopScrubPreviewAudio } from "../lib/playbackScrub";
|
||||
import { applyCachedSourceDurations, probeMissingSourceDurations } from "../lib/mediaProbe";
|
||||
import { shouldResumeForwardPlaybackAfterSeek, shouldStopAfterSeek } from "../lib/playbackSeek";
|
||||
|
||||
/**
|
||||
* Whether the derived elements differ from the current ones in any field that
|
||||
* affects rendering (identity, timing, track, or source length) — used to skip
|
||||
* redundant store writes.
|
||||
*/
|
||||
function timelineElementsChanged(prev: TimelineElement[], next: TimelineElement[]): boolean {
|
||||
if (next.length !== prev.length) return true;
|
||||
return next.some((el, i) => {
|
||||
const p = prev[i];
|
||||
return (
|
||||
!p ||
|
||||
el.id !== p.id ||
|
||||
el.start !== p.start ||
|
||||
el.duration !== p.duration ||
|
||||
el.track !== p.track ||
|
||||
el.sourceDuration !== p.sourceDuration
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
export function useTimelinePlayer() {
|
||||
const iframeRef = useRef<HTMLIFrameElement | null>(null);
|
||||
const rafRef = useRef<number>(0);
|
||||
@@ -65,27 +86,19 @@ export function useTimelinePlayer() {
|
||||
(elements: TimelineElement[], nextDuration?: number) => {
|
||||
const state = usePlayerStore.getState();
|
||||
const resolvedDuration = nextDuration ?? state.duration;
|
||||
const mergedElements = mergeTimelineElementsPreservingDowngrades(
|
||||
state.elements,
|
||||
elements,
|
||||
state.duration,
|
||||
resolvedDuration,
|
||||
// applyCachedSourceDurations re-applies the cached probe duration: re-derived
|
||||
// elements (e.g. after a clip move) can arrive without sourceDuration, which
|
||||
// otherwise makes trimmed waveforms lose their window.
|
||||
const mergedElements = applyCachedSourceDurations(
|
||||
mergeTimelineElementsPreservingDowngrades(
|
||||
state.elements,
|
||||
elements,
|
||||
state.duration,
|
||||
resolvedDuration,
|
||||
),
|
||||
);
|
||||
|
||||
const elementsChanged =
|
||||
mergedElements.length !== state.elements.length ||
|
||||
mergedElements.some((el, i) => {
|
||||
const prev = state.elements[i];
|
||||
return (
|
||||
!prev ||
|
||||
el.id !== prev.id ||
|
||||
el.start !== prev.start ||
|
||||
el.duration !== prev.duration ||
|
||||
el.track !== prev.track
|
||||
);
|
||||
});
|
||||
|
||||
if (elementsChanged) {
|
||||
if (timelineElementsChanged(state.elements, mergedElements)) {
|
||||
setElements(mergedElements);
|
||||
}
|
||||
if (
|
||||
@@ -99,31 +112,17 @@ export function useTimelinePlayer() {
|
||||
setTimelineReady(true);
|
||||
}
|
||||
|
||||
// Asynchronously enrich media elements missing sourceDuration via mediabunny.
|
||||
// The probe reads file headers only — no full decode — so this is cheap.
|
||||
const needsProbe = mergedElements.filter(
|
||||
(el) =>
|
||||
el.src &&
|
||||
el.sourceDuration == null &&
|
||||
["video", "audio"].includes(el.tag.toLowerCase()) &&
|
||||
!getCachedProbe(el.src),
|
||||
);
|
||||
if (needsProbe.length > 0) {
|
||||
void Promise.allSettled(
|
||||
needsProbe.map(async (el) => {
|
||||
const result = await probeMediaUrl(el.src!);
|
||||
if (!result) return;
|
||||
const key = el.key ?? el.id;
|
||||
usePlayerStore.setState((state) => {
|
||||
const idx = state.elements.findIndex((e) => (e.key ?? e.id) === key);
|
||||
if (idx === -1 || state.elements[idx].sourceDuration != null) return {};
|
||||
const patched = state.elements.slice();
|
||||
patched[idx] = { ...state.elements[idx], sourceDuration: result.duration };
|
||||
return { elements: patched };
|
||||
});
|
||||
}),
|
||||
);
|
||||
}
|
||||
// Asynchronously enrich media elements still missing sourceDuration
|
||||
// (header-only probe, cheap), applying each resolved value to the store.
|
||||
void probeMissingSourceDurations(mergedElements, (key, durationSeconds) => {
|
||||
usePlayerStore.setState((state) => {
|
||||
const idx = state.elements.findIndex((e) => (e.key ?? e.id) === key);
|
||||
if (idx === -1 || state.elements[idx].sourceDuration != null) return {};
|
||||
const patched = state.elements.slice();
|
||||
patched[idx] = { ...state.elements[idx], sourceDuration: durationSeconds };
|
||||
return { elements: patched };
|
||||
});
|
||||
});
|
||||
},
|
||||
[setElements, setTimelineReady, setDuration],
|
||||
);
|
||||
@@ -280,6 +279,7 @@ export function useTimelinePlayer() {
|
||||
const play = useCallback(() => {
|
||||
stopRAFLoop();
|
||||
stopReverseLoop();
|
||||
stopScrubPreviewAudio();
|
||||
const adapter = getAdapter();
|
||||
if (!adapter) return;
|
||||
if (adapter.getTime() >= adapter.getDuration()) {
|
||||
@@ -392,6 +392,7 @@ export function useTimelinePlayer() {
|
||||
adapter.seek(nextTime, options);
|
||||
liveTime.notify(nextTime); // Direct DOM updates (playhead, timecode, progress) — no re-render
|
||||
setCurrentTime(nextTime); // sync store so Split/Delete have accurate time
|
||||
if (!shouldResumeAfterSeek && !keepPlaying) scrubMusicAtSeek(iframeRef.current, nextTime);
|
||||
if (shouldResumeAfterSeek) {
|
||||
stopRAFLoop();
|
||||
applyPlaybackRate(usePlayerStore.getState().playbackRate);
|
||||
@@ -557,6 +558,7 @@ export function useTimelinePlayer() {
|
||||
document.removeEventListener("visibilitychange", handleVisibilityChange);
|
||||
stopRAFLoop();
|
||||
stopReverseLoop();
|
||||
stopScrubPreviewAudio();
|
||||
releaseStaticSeekCache(staticSeekAdapterRef, staticSeekWarnedRef);
|
||||
if (probeIntervalRef.current) clearInterval(probeIntervalRef.current);
|
||||
};
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
export interface MediaProbeResult {
|
||||
interface MediaProbeResult {
|
||||
duration: number;
|
||||
width?: number;
|
||||
height?: number;
|
||||
@@ -61,11 +61,54 @@ async function probeOne(url: string): Promise<MediaProbeResult | null> {
|
||||
}
|
||||
}
|
||||
|
||||
export function getCachedProbe(url: string): MediaProbeResult | undefined {
|
||||
function getCachedProbe(url: string): MediaProbeResult | undefined {
|
||||
return cache.get(normalizeUrl(url));
|
||||
}
|
||||
|
||||
export async function probeMediaUrl(url: string): Promise<MediaProbeResult | null> {
|
||||
/**
|
||||
* Re-apply the cached probe `sourceDuration` to media elements that arrive
|
||||
* without it. Re-deriving the timeline (e.g. after a clip move) produces fresh
|
||||
* objects whose duration the DOM scan may not have, and the async probe skips
|
||||
* already-cached srcs — so without this, trimmed waveforms lose their window.
|
||||
*/
|
||||
export function applyCachedSourceDurations<
|
||||
T extends { src?: string; tag: string; sourceDuration?: number },
|
||||
>(elements: T[]): T[] {
|
||||
return elements.map((el) => {
|
||||
const tag = el.tag.toLowerCase();
|
||||
if (!el.src || el.sourceDuration != null || (tag !== "audio" && tag !== "video")) return el;
|
||||
const cached = getCachedProbe(el.src);
|
||||
return cached?.duration && cached.duration > 0
|
||||
? { ...el, sourceDuration: cached.duration }
|
||||
: el;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe (header-only, cheap) any media elements still missing sourceDuration
|
||||
* after the cache pass, applying each resolved duration via `apply(key, secs)`.
|
||||
* Skips already-cached srcs.
|
||||
*/
|
||||
export async function probeMissingSourceDurations<
|
||||
T extends { src?: string; tag: string; sourceDuration?: number; key?: string; id: string },
|
||||
>(elements: T[], apply: (key: string, durationSeconds: number) => void): Promise<void> {
|
||||
const needs = elements.filter(
|
||||
(el) =>
|
||||
el.src &&
|
||||
el.sourceDuration == null &&
|
||||
["video", "audio"].includes(el.tag.toLowerCase()) &&
|
||||
!getCachedProbe(el.src),
|
||||
);
|
||||
if (needs.length === 0) return;
|
||||
await Promise.allSettled(
|
||||
needs.map(async (el) => {
|
||||
const result = await probeMediaUrl(el.src!);
|
||||
if (result) apply(el.key ?? el.id, result.duration);
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async function probeMediaUrl(url: string): Promise<MediaProbeResult | null> {
|
||||
const key = normalizeUrl(url);
|
||||
const cached = cache.get(key);
|
||||
if (cached) return cached;
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
import { usePlayerStore } from "../store/playerStore";
|
||||
import { isMusicTrack } from "../../utils/timelineInspector";
|
||||
import { scrubPreviewAudio, stopScrubPreviewAudio } from "./timelineIframeHelpers";
|
||||
|
||||
export { stopScrubPreviewAudio };
|
||||
|
||||
// Scrub the music track's audio at a seeked composition time (paused-seek only).
|
||||
// Skipped when audio is muted or the time falls outside the music clip.
|
||||
export function scrubMusicAtSeek(iframe: HTMLIFrameElement | null, nextTime: number): void {
|
||||
const s = usePlayerStore.getState();
|
||||
const music = s.elements.find(isMusicTrack);
|
||||
if (!music || s.audioMuted) return;
|
||||
const rel = nextTime - music.start;
|
||||
const audioFileTime = rel >= 0 && rel <= music.duration ? (music.playbackStart ?? 0) + rel : null;
|
||||
scrubPreviewAudio(iframe, audioFileTime, music.domId ?? music.id);
|
||||
}
|
||||
@@ -115,6 +115,8 @@ export function createTimelineElementFromManifestClip(params: {
|
||||
|
||||
if (hostEl) {
|
||||
applyMediaMetadataFromElement(entry, hostEl);
|
||||
const timelineRole = hostEl.getAttribute("data-timeline-role");
|
||||
if (timelineRole) entry.timelineRole = timelineRole;
|
||||
}
|
||||
if (clip.assetUrl) entry.src = clip.assetUrl;
|
||||
if (clip.kind === "composition" && clip.compositionId) {
|
||||
@@ -286,17 +288,23 @@ export function parseTimelineFromDOM(doc: Document, rootDuration: number): Timel
|
||||
if (mediaEl.tagName === "IMG") {
|
||||
entry.tag = "img";
|
||||
}
|
||||
const src = mediaEl.getAttribute("src");
|
||||
if (src) entry.src = src;
|
||||
const vol = el.getAttribute("data-volume") ?? mediaEl.getAttribute("data-volume");
|
||||
if (vol) entry.volume = parseFloat(vol);
|
||||
applyMediaMetadataFromElement(entry, el);
|
||||
// Override AFTER the helper (which sets the raw relative attribute) so the
|
||||
// resolved absolute URL wins — the Studio can then fetch the asset
|
||||
// regardless of whether the attribute value was relative or absolute.
|
||||
const resolvedSrc = (mediaEl as HTMLMediaElement | HTMLImageElement).src || undefined;
|
||||
if (resolvedSrc) entry.src = resolvedSrc;
|
||||
}
|
||||
|
||||
if (el.hasAttribute("data-timeline-locked")) {
|
||||
entry.timelineLocked = true;
|
||||
}
|
||||
|
||||
const timelineRole = el.getAttribute("data-timeline-role");
|
||||
if (timelineRole) entry.timelineRole = timelineRole;
|
||||
|
||||
// Sub-compositions
|
||||
const compSrc =
|
||||
el.getAttribute("data-composition-src") || el.getAttribute("data-composition-file");
|
||||
|
||||
@@ -170,6 +170,95 @@ export function resolveIframe(el: Element | null): HTMLIFrameElement | null {
|
||||
return el.shadowRoot?.querySelector("iframe") ?? el.querySelector("iframe") ?? null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Audio scrubbing
|
||||
// ---------------------------------------------------------------------------
|
||||
// Plays a brief slice of the music track while the user drags the playhead,
|
||||
// like an NLE scrub. Repeated calls keep playback alive; it auto-pauses shortly
|
||||
// after scrubbing stops and restores the element's prior muted state.
|
||||
|
||||
const SCRUB_VOLUME = 0.25;
|
||||
|
||||
let scrubAudioEl: HTMLAudioElement | null = null;
|
||||
let scrubStopTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
let scrubPrevMuted: boolean | null = null;
|
||||
let scrubPrevVolume: number | null = null;
|
||||
|
||||
// Resolve the SAME element the store identified as music: prefer its id, then
|
||||
// the role attribute, and only fall back to the first <audio> (which could be a
|
||||
// voiceover, so the id hint matters).
|
||||
function resolveScrubAudioEl(doc: Document, musicId?: string | null): HTMLAudioElement | null {
|
||||
if (musicId) {
|
||||
const byId = doc.getElementById(musicId);
|
||||
if (byId instanceof HTMLAudioElement) return byId;
|
||||
}
|
||||
return (
|
||||
doc.querySelector<HTMLAudioElement>("audio[data-timeline-role='music']") ??
|
||||
doc.querySelector<HTMLAudioElement>("audio")
|
||||
);
|
||||
}
|
||||
|
||||
function applyScrub(el: HTMLAudioElement, audioFileTime: number): void {
|
||||
if (scrubAudioEl && scrubAudioEl !== el) stopScrubPreviewAudio();
|
||||
if (scrubPrevMuted === null) scrubPrevMuted = el.muted;
|
||||
if (scrubPrevVolume === null) scrubPrevVolume = el.volume;
|
||||
scrubAudioEl = el;
|
||||
try {
|
||||
el.muted = false;
|
||||
el.volume = SCRUB_VOLUME;
|
||||
if (Math.abs(el.currentTime - audioFileTime) > 0.04) el.currentTime = audioFileTime;
|
||||
if (el.paused) void el.play().catch(() => {});
|
||||
} catch {
|
||||
/* element not ready */
|
||||
}
|
||||
if (scrubStopTimer) clearTimeout(scrubStopTimer);
|
||||
scrubStopTimer = setTimeout(stopScrubPreviewAudio, 140);
|
||||
}
|
||||
|
||||
/**
|
||||
* Scrub the preview music audio to `audioFileTime` (seconds into the source
|
||||
* file). Pass `null` to stop. Safe to call rapidly during a playhead drag.
|
||||
*/
|
||||
export function scrubPreviewAudio(
|
||||
iframe: HTMLIFrameElement | null,
|
||||
audioFileTime: number | null,
|
||||
musicId?: string | null,
|
||||
): void {
|
||||
if (!iframe) return;
|
||||
if (audioFileTime === null) {
|
||||
stopScrubPreviewAudio();
|
||||
return;
|
||||
}
|
||||
let doc: Document | null = null;
|
||||
try {
|
||||
doc = iframe.contentDocument;
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
if (!doc) return;
|
||||
const el = resolveScrubAudioEl(doc, musicId);
|
||||
if (el) applyScrub(el, audioFileTime);
|
||||
}
|
||||
|
||||
export function stopScrubPreviewAudio(): void {
|
||||
if (scrubStopTimer) {
|
||||
clearTimeout(scrubStopTimer);
|
||||
scrubStopTimer = null;
|
||||
}
|
||||
const el = scrubAudioEl;
|
||||
scrubAudioEl = null;
|
||||
if (!el) return;
|
||||
try {
|
||||
el.pause();
|
||||
if (scrubPrevMuted !== null) el.muted = scrubPrevMuted;
|
||||
if (scrubPrevVolume !== null) el.volume = scrubPrevVolume;
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
scrubPrevMuted = null;
|
||||
scrubPrevVolume = null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Enrich missing compositions from DOM
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { create } from "zustand";
|
||||
import type { MusicBeatAnalysis } from "@hyperframes/core/beats";
|
||||
import type { BeatEditState } from "../../utils/beatEditing";
|
||||
import { readStudioUiPreferences, writeStudioUiPreferences } from "../../utils/studioUiPreferences";
|
||||
|
||||
/** Minimal keyframe cache types — mirrors GsapKeyframesData without pulling in Node-only gsap-parser. */
|
||||
@@ -46,6 +48,8 @@ export interface TimelineElement {
|
||||
timingSource?: "authored" | "implicit";
|
||||
/** Set by data-timeline-locked on the host element — disables move and trim in Studio. */
|
||||
timelineLocked?: boolean;
|
||||
/** Value of data-timeline-role attribute — used to identify music vs. voiceover. */
|
||||
timelineRole?: string;
|
||||
}
|
||||
|
||||
export type ZoomMode = "fit" | "manual";
|
||||
@@ -56,6 +60,8 @@ interface PlayerState {
|
||||
currentTime: number;
|
||||
duration: number;
|
||||
timelineReady: boolean;
|
||||
/** True while a beat dot is being dragged — hides the playhead guideline. */
|
||||
beatDragging: boolean;
|
||||
elements: TimelineElement[];
|
||||
selectedElementId: string | null;
|
||||
playbackRate: number;
|
||||
@@ -99,6 +105,7 @@ interface PlayerState {
|
||||
setAudioMuted: (muted: boolean) => void;
|
||||
setLoopEnabled: (enabled: boolean) => void;
|
||||
setTimelineReady: (ready: boolean) => void;
|
||||
setBeatDragging: (dragging: boolean) => void;
|
||||
setElements: (elements: TimelineElement[]) => void;
|
||||
setSelectedElementId: (id: string | null) => void;
|
||||
updateElement: (
|
||||
@@ -121,6 +128,32 @@ interface PlayerState {
|
||||
|
||||
lintFindingsByElement: Map<string, { count: number; messages: string[] }>;
|
||||
setLintFindingsByElement: (map: Map<string, { count: number; messages: string[] }>) => void;
|
||||
|
||||
beatAnalysis: MusicBeatAnalysis | null;
|
||||
setBeatAnalysis: (analysis: MusicBeatAnalysis | null) => void;
|
||||
|
||||
/** User edits (add/move/delete) layered over the detected beat grid. */
|
||||
beatEdits: BeatEditState | null;
|
||||
setBeatEdits: (edits: BeatEditState | null) => void;
|
||||
/** Undo/redo stacks for beat edits (in-memory, session-only). */
|
||||
beatUndo: BeatHistoryEntry[];
|
||||
beatRedo: BeatHistoryEntry[];
|
||||
/** Apply a beat edit and record it for undo. */
|
||||
commitBeatEdits: (next: BeatEditState | null, label: string) => void;
|
||||
/** Undo/redo the most recent beat edit; returns its label or null if none. */
|
||||
undoBeatEdits: () => string | null;
|
||||
redoBeatEdits: () => string | null;
|
||||
/** Clear beat edit history (e.g. when the music track changes). */
|
||||
resetBeatHistory: () => void;
|
||||
/** Callback that persists current beats to disk; registered by the analysis hook. */
|
||||
beatPersist: (() => void) | null;
|
||||
setBeatPersist: (fn: (() => void) | null) => void;
|
||||
}
|
||||
|
||||
interface BeatHistoryEntry {
|
||||
restore: BeatEditState | null; // state to restore when this entry is applied
|
||||
at: number; // original edit timestamp (for global undo ordering)
|
||||
label: string;
|
||||
}
|
||||
|
||||
// Lightweight pub-sub for current time during playback.
|
||||
@@ -141,6 +174,7 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
|
||||
currentTime: 0,
|
||||
duration: 0,
|
||||
timelineReady: false,
|
||||
beatDragging: false,
|
||||
elements: [],
|
||||
selectedElementId: null,
|
||||
playbackRate: readStudioUiPreferences().playbackRate ?? 1,
|
||||
@@ -193,6 +227,50 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
|
||||
lintFindingsByElement: new Map(),
|
||||
setLintFindingsByElement: (map) => set({ lintFindingsByElement: map }),
|
||||
|
||||
beatAnalysis: null,
|
||||
setBeatAnalysis: (analysis) => set({ beatAnalysis: analysis }),
|
||||
|
||||
beatEdits: null,
|
||||
setBeatEdits: (edits) => set({ beatEdits: edits }),
|
||||
|
||||
beatUndo: [],
|
||||
beatRedo: [],
|
||||
beatPersist: null,
|
||||
setBeatPersist: (fn) => set({ beatPersist: fn }),
|
||||
commitBeatEdits: (next, label) => {
|
||||
set((s) => ({
|
||||
beatEdits: next,
|
||||
beatUndo: [...s.beatUndo, { restore: s.beatEdits, at: Date.now(), label }],
|
||||
beatRedo: [],
|
||||
}));
|
||||
get().beatPersist?.();
|
||||
},
|
||||
undoBeatEdits: () => {
|
||||
const s = get();
|
||||
const entry = s.beatUndo[s.beatUndo.length - 1];
|
||||
if (!entry) return null;
|
||||
set({
|
||||
beatEdits: entry.restore,
|
||||
beatUndo: s.beatUndo.slice(0, -1),
|
||||
beatRedo: [...s.beatRedo, { restore: s.beatEdits, at: entry.at, label: entry.label }],
|
||||
});
|
||||
get().beatPersist?.();
|
||||
return entry.label;
|
||||
},
|
||||
resetBeatHistory: () => set({ beatUndo: [], beatRedo: [] }),
|
||||
redoBeatEdits: () => {
|
||||
const s = get();
|
||||
const entry = s.beatRedo[s.beatRedo.length - 1];
|
||||
if (!entry) return null;
|
||||
set({
|
||||
beatEdits: entry.restore,
|
||||
beatRedo: s.beatRedo.slice(0, -1),
|
||||
beatUndo: [...s.beatUndo, { restore: s.beatEdits, at: entry.at, label: entry.label }],
|
||||
});
|
||||
get().beatPersist?.();
|
||||
return entry.label;
|
||||
},
|
||||
|
||||
setIsPlaying: (playing) => {
|
||||
if (get().isPlaying === playing) return;
|
||||
set({ isPlaying: playing });
|
||||
@@ -233,6 +311,7 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
|
||||
setCurrentTime: (time) => set({ currentTime: Number.isFinite(time) ? time : 0 }),
|
||||
setDuration: (duration) => set({ duration: Number.isFinite(duration) ? duration : 0 }),
|
||||
setTimelineReady: (ready) => set({ timelineReady: ready }),
|
||||
setBeatDragging: (dragging) => set({ beatDragging: dragging }),
|
||||
setElements: (elements) => set({ elements }),
|
||||
setSelectedElementId: (id) => set({ selectedElementId: id }),
|
||||
updateElement: (elementId, updates) =>
|
||||
@@ -250,6 +329,7 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
|
||||
currentTime: 0,
|
||||
duration: 0,
|
||||
timelineReady: false,
|
||||
beatDragging: false,
|
||||
elements: [],
|
||||
selectedElementId: null,
|
||||
inPoint: null,
|
||||
@@ -258,5 +338,12 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
|
||||
selectedKeyframes: new Set(),
|
||||
selectedElementIds: new Set(),
|
||||
keyframeCache: new Map(),
|
||||
// Beat state is project-specific — clear it so a project switch can't
|
||||
// apply the previous project's beats/undo/persist to the new one.
|
||||
beatAnalysis: null,
|
||||
beatEdits: null,
|
||||
beatUndo: [],
|
||||
beatRedo: [],
|
||||
beatPersist: null,
|
||||
}),
|
||||
}));
|
||||
|
||||
Reference in New Issue
Block a user