feat(studio,cli): music beat detection with timeline guides + headless beats CLI (#1424)

* feat(studio,cli): music beat detection with timeline guides + headless beats CLI Beat detection for music tracks: the Studio draws beat guides on the active track, beats are user-editable and persist to a project file, and a new `hyperframes beats` CLI generates that file headlessly before the Studio opens. Detection lives in @hyperframes/core/beats (shared by Studio + CLI): an energy onset detector cross-validated with bpm-detective, regularized to an octave- aligned grid, silence-gated, with per-beat loudness. Music-only — an <audio data-timeline-role="music"> is analyzed; voiceover is excluded. Studio: green beat lines + draggable dots on the selected track; add at playhead, drag to move, double-click to delete (audio scrubs); edits persist to beats/<audio>.json and are undoable (interleaved with file history). CLI: `hyperframes beats [dir]` runs the same detection in headless Chrome (prebuilt browser bundle in dist) and writes the beat file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): timeline beat-grid + zoom UX refinements - Center-anchored magnify: zooming via the toolbar/slider keeps the time at the viewport center fixed instead of anchoring at the left. Pinch still anchors at the cursor. - Move-snap to beats: dragging a clip snaps whichever edge (start or end) is nearest a beat, matching the existing resize-edge snapping. - Beat lines on track backgrounds: faint full-height beat lines now paint behind the clips on every track lane (brightness scales with loudness); the green dots stay on the active track's top bar. - Waveform follows zoom: bars fill the full clip width and resample the windowed peaks, so the waveform stretches with zoom instead of stopping partway across a widened clip. - Beat dots centered in the top bar: align the dot band to the clip top (CLIP_Y) so the dots sit centered in the dark bar instead of being bisected by the clip's top border. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(studio): preserve media sourceDuration across element re-derivation Moving a non-music clip re-derived the timeline elements into fresh objects whose sourceDuration the DOM scan hadn't loaded yet. The async probe skips srcs already in its cache, so the value was silently dropped — trimFractions then returned no window and the trimmed music waveform reset to the full source pinned at the track start. Re-apply the cached probe duration synchronously on every derivation (applyCachedSourceDurations) and extract the async probe loop into probeMissingSourceDurations to keep useTimelinePlayer within the file size limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): skip beat-snap on the music track, highlight move-snap target The music track defines the beats, so moving or trimming it no longer snaps to its own beats (isMusicTrack guard on both the move and resize snap paths). Moving another clip snapped only on drop with no cue. snapMoveStartToBeat now also returns the beat it will snap to; BeatBackgroundLines draws that beat's line as a bright neon-green glow while the clip's edge is within the snap region, so the target is visible before drop. Also drops .commitmsg.tmp, accidentally committed via git add -A. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): hide playhead while dragging a beat; default beat dots to music track - Dragging a beat dot now hides the playhead guideline (new beatDragging store flag set on beat pointer down/up) so its line doesn't track the scrub and clutter the beat being moved. - Beat dots render on the selected track, falling back to the music track when nothing is selected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(core): remove polynomial-ReDoS regex from audioRelPathForSrc CodeQL js/polynomial-redos: the lazy `.+?` followed by an optional trailing `[?#].*$` backtracks polynomially on crafted `/preview/...` inputs. Parse the preview-relative path with indexOf/slice instead, and strip the query/hash with a single linear char-class search. Behavior is unchanged for all preview/absolute/blob/data/bare inputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(studio,core,cli): review hardening for beat detection + timeline UX - playerStore.reset() now clears beat state (analysis, edits, undo/redo, persist) so a project switch can't apply the previous project's beats, undo stack, or file-writer to the new one. - removeUserBeat returns the same reference on a no-op, and delete/move beat actions skip committing when nothing changed — no more phantom undo entries / debounced writes for no-op edits. - regularizeBeats bails to raw onsets when the (octave-misread) tempo would produce a sub-125ms grid, avoiding a tens-of-thousands-of-beats freeze. - parseBeats clamps strength to [0,1] and rejects non-finite time/strength, so a hand-edited file can't feed NaN into the gamma curve (Math.pow on a negative base) and blank out beat markers. - Start-edge beat-snap now also requires duration >= minDuration, matching the end-edge guard, so a rightward snap can't collapse the clip. - Center-anchor zoom effect always consumes its skip flag, so a pinch that produced no pps change can't leave it stranded and skip the next zoom. - Headless beats analyzer projects to {beatTimes,beatStrengths,bpm,confidence} before returning, so page.evaluate no longer serializes the full decoded PCM (channelData) across the CDP boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(core): gate parseBeats on schema version parseBeats accepted any object with a beats array, so a future v2 beat file (with changed semantics) would be parsed silently as v1. Reject anything whose version is not 1, treating an unknown version like an absent/invalid file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> --------- Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
2026-09-03 04:38:33 +00:00 · 2026-06-14 17:17:13 -07:00
parent a95e49dbda
commit d9f69f61e7
33 changed files with 1945 additions and 73 deletions
@@ -0,0 +1,285 @@
+// bpm-detective touches `window` at module top-level, so it must NOT be a static
+// import (that would crash any non-browser import of this module graph, e.g.
+// vitest/SSR). It's loaded lazily inside analyzeMusicFromBuffer, which only runs
+// in the browser.
+type BpmDetect = (buffer: AudioBuffer) => number;
+let bpmDetectivePromise: Promise<BpmDetect | null> | null = null;
+function loadBpmDetective(): Promise<BpmDetect | null> {
+  if (!bpmDetectivePromise) {
+    bpmDetectivePromise = import(
+      // @ts-ignore -- no type declarations for bpm-detective
+      "bpm-detective"
+    )
+      .then((m) => ((m as { default?: BpmDetect }).default ?? (m as unknown as BpmDetect)) || null)
+      .catch(() => null);
+  }
+  return bpmDetectivePromise;
+}
+
+const WINDOW_SIZE = 1024;
+const HOP_SIZE = 512;
+
+export interface MusicBeatAnalysis {
+  beatTimes: number[];
+  /** Per-beat loudness 0–1 (local RMS / peak), aligned by index with beatTimes. */
+  beatStrengths: number[];
+  bpm: number | null;
+  bpmConfidence: "high" | "low" | "uncertain";
+  /** Decoded mono samples — retained so strength can be measured at user-added
+   *  beats. Audio-file coordinates. May be null if decode data was dropped. */
+  channelData: Float32Array | null;
+  sampleRate: number;
+  /** Reference peak RMS used to normalize beat strengths. */
+  peak: number;
+}
+
+const STRENGTH_WINDOW_S = 0.05; // ±50ms RMS window
+
+/** Local RMS amplitude at a given audio-file time. */
+export function computeRmsAt(channelData: Float32Array, sampleRate: number, time: number): number {
+  const halfWindow = Math.floor(sampleRate * STRENGTH_WINDOW_S);
+  const center = Math.floor(time * sampleRate);
+  const start = Math.max(0, center - halfWindow);
+  const end = Math.min(channelData.length, center + halfWindow);
+  let sum = 0;
+  for (let i = start; i < end; i++) {
+    const s = channelData[i] ?? 0;
+    sum += s * s;
+  }
+  return Math.sqrt(sum / Math.max(end - start, 1));
+}
+
+/** Normalized beat strength (0–1) at an audio-file time, using a track peak. */
+export function strengthAtTime(
+  analysis: Pick<MusicBeatAnalysis, "channelData" | "sampleRate" | "peak">,
+  time: number,
+): number {
+  if (!analysis.channelData || analysis.peak <= 0) return 0.5;
+  return Math.min(1, computeRmsAt(analysis.channelData, analysis.sampleRate, time) / analysis.peak);
+}
+
+// fallow-ignore-next-line complexity
+export async function detectBeats(audioBuffer: AudioBuffer): Promise<number[]> {
+  const channelData = audioBuffer.getChannelData(0);
+  const sampleRate = audioBuffer.sampleRate;
+
+  const energies: number[] = [];
+  for (let i = 0; i < channelData.length - WINDOW_SIZE; i += HOP_SIZE) {
+    let sum = 0;
+    for (let j = 0; j < WINDOW_SIZE; j++) {
+      const sample = channelData[i + j]!;
+      sum += sample * sample;
+    }
+    energies.push(sum / WINDOW_SIZE);
+  }
+
+  const beats: number[] = [];
+  const localWindowSize = 20;
+
+  for (let i = localWindowSize; i < energies.length - localWindowSize; i++) {
+    let localMean = 0;
+    for (let j = i - localWindowSize; j < i + localWindowSize; j++) {
+      localMean += energies[j]!;
+    }
+    localMean /= localWindowSize * 2;
+
+    const threshold = localMean * 1.5;
+    const current = energies[i]!;
+
+    if (
+      current > threshold &&
+      current > (energies[i - 1] ?? 0) &&
+      current > (energies[i + 1] ?? 0)
+    ) {
+      const timeInSeconds = (i * HOP_SIZE) / sampleRate;
+      if (beats.length === 0 || timeInSeconds - beats[beats.length - 1]! > 0.1) {
+        beats.push(Math.round(timeInSeconds * 1000) / 1000);
+      }
+    }
+  }
+
+  return beats;
+}
+
+function computeBpmFromBeats(beatTimes: number[]): number | null {
+  if (beatTimes.length < 4) return null;
+  const iois: number[] = [];
+  for (let i = 1; i < beatTimes.length; i++) {
+    iois.push(beatTimes[i]! - beatTimes[i - 1]!);
+  }
+  const sorted = [...iois].sort((a, b) => a - b);
+  const mid = Math.floor(sorted.length / 2);
+  const medianIoi = sorted.length % 2 === 0 ? (sorted[mid - 1]! + sorted[mid]!) / 2 : sorted[mid]!;
+  if (medianIoi <= 0) return null;
+  return Math.round((60 / medianIoi) * 10) / 10;
+}
+
+// Fold into 60–120 for octave-safe BPM comparison
+function canonicalizeBpm(bpm: number): number {
+  let b = bpm;
+  while (b > 120) b /= 2;
+  while (b < 60) b *= 2;
+  return b;
+}
+
+// Pick the *2/÷2 octave of `bpm` whose beat interval is closest to the onset
+// pulse, so a half-time detective reading (e.g. 174 for an 87bpm song) doesn't
+// produce a double-density grid.
+function octaveAlignBpm(bpm: number, reference: number): number {
+  const candidates = [bpm / 2, bpm, bpm * 2];
+  let best = bpm;
+  let bestDist = Number.POSITIVE_INFINITY;
+  for (const c of candidates) {
+    if (c <= 0) continue;
+    const dist = Math.abs(c - reference);
+    if (dist < bestDist) {
+      bestDist = dist;
+      best = c;
+    }
+  }
+  return best;
+}
+
+// fallow-ignore-next-line complexity
+function regularizeBeats(rawBeats: number[], bpm: number, duration: number): number[] {
+  if (rawBeats.length === 0 || bpm <= 0 || duration <= 0) return rawBeats;
+  const beatInterval = 60 / bpm;
+  // Guard against a pathological (octave-misread) tempo producing a millisecond
+  // interval → tens of thousands of grid beats that freeze the timeline. 480 BPM
+  // (0.125s) is well above any real music tempo; bail to the raw onsets instead.
+  if (beatInterval < 0.125) return rawBeats;
+  const threshold = beatInterval * 0.25;
+
+  // Find phase offset that maximally aligns with raw onsets
+  let bestOffset = 0;
+  let bestScore = -1;
+  for (const anchor of rawBeats.slice(0, 10)) {
+    const offset = ((anchor % beatInterval) + beatInterval) % beatInterval;
+    let score = 0;
+    for (const rb of rawBeats) {
+      const phase = ((rb % beatInterval) + beatInterval) % beatInterval;
+      const dist = Math.min(Math.abs(phase - offset), beatInterval - Math.abs(phase - offset));
+      if (dist < threshold) score++;
+    }
+    if (score > bestScore) {
+      bestScore = score;
+      bestOffset = offset;
+    }
+  }
+
+  const beats: number[] = [];
+  for (let t = bestOffset; t <= duration + 0.001; t += beatInterval) {
+    beats.push(Math.round(t * 1000) / 1000);
+  }
+  return beats;
+}
+
+// Drop beats that fall in silent/near-silent regions (e.g. intro/outro) and
+// score each surviving beat's loudness (local RMS / peak) for brightness.
+function gateBeatsBySilence(
+  beats: number[],
+  channelData: Float32Array,
+  sampleRate: number,
+): { times: number[]; strengths: number[]; peak: number } {
+  if (beats.length === 0) return { times: beats, strengths: [], peak: 1e-6 };
+  const energies = beats.map((t) => computeRmsAt(channelData, sampleRate, t));
+  const peak = Math.max(...energies, 1e-6);
+  const threshold = peak * 0.12;
+  const times: number[] = [];
+  const strengths: number[] = [];
+  for (let i = 0; i < beats.length; i++) {
+    if (energies[i]! >= threshold) {
+      times.push(beats[i]!);
+      strengths.push(Math.min(1, energies[i]! / peak));
+    }
+  }
+  return { times, strengths, peak };
+}
+
+// fallow-ignore-next-line complexity
+export async function analyzeMusicFromBuffer(audioBuffer: AudioBuffer): Promise<MusicBeatAnalysis> {
+  const channelData = audioBuffer.getChannelData(0);
+  const sampleRate = audioBuffer.sampleRate;
+  const duration = audioBuffer.duration;
+
+  const rawBeats = await detectBeats(audioBuffer);
+  const onsetBpm = computeBpmFromBeats(rawBeats);
+
+  let detectiveBpm: number | null = null;
+  try {
+    const detect = await loadBpmDetective();
+    if (detect) detectiveBpm = detect(audioBuffer);
+  } catch {
+    // Not enough peaks or browser context unavailable
+  }
+
+  let bpm: number | null = onsetBpm;
+  let confidence: MusicBeatAnalysis["bpmConfidence"] = "uncertain";
+  let regularizeBpm: number | null = null;
+
+  if (onsetBpm !== null && detectiveBpm !== null) {
+    const pctDiff =
+      Math.abs(canonicalizeBpm(onsetBpm) - canonicalizeBpm(detectiveBpm)) /
+      canonicalizeBpm(detectiveBpm);
+    if (pctDiff < 0.05) {
+      // Detective folds tempo into 90–180; re-pick the octave nearest the onset
+      // pulse so a half-time track isn't gridded at double density.
+      bpm = octaveAlignBpm(detectiveBpm, onsetBpm);
+      confidence = "high";
+      regularizeBpm = bpm;
+    } else if (pctDiff < 0.1) {
+      bpm = Math.round((onsetBpm + detectiveBpm) / 2);
+      confidence = "low";
+      regularizeBpm = bpm;
+    } else {
+      bpm = onsetBpm;
+      confidence = "uncertain";
+    }
+  } else if (onsetBpm !== null) {
+    bpm = onsetBpm;
+    confidence = "low";
+    regularizeBpm = onsetBpm;
+  } else if (detectiveBpm !== null) {
+    bpm = detectiveBpm;
+    confidence = "low";
+    regularizeBpm = detectiveBpm;
+  }
+
+  const gridBeats =
+    regularizeBpm !== null ? regularizeBeats(rawBeats, regularizeBpm, duration) : rawBeats;
+  const gated = gateBeatsBySilence(gridBeats, channelData, sampleRate);
+
+  return {
+    beatTimes: gated.times,
+    beatStrengths: gated.strengths,
+    bpm,
+    bpmConfidence: confidence,
+    channelData,
+    sampleRate,
+    peak: gated.peak,
+  };
+}
+
+export async function detectBeatsFromUrl(url: string): Promise<number[]> {
+  const audioContext = new AudioContext();
+  try {
+    const response = await fetch(url);
+    const arrayBuffer = await response.arrayBuffer();
+    const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
+    return detectBeats(audioBuffer);
+  } finally {
+    await audioContext.close();
+  }
+}
+
+export async function analyzeMusicFromUrl(url: string): Promise<MusicBeatAnalysis> {
+  const audioContext = new AudioContext();
+  try {
+    const response = await fetch(url);
+    const arrayBuffer = await response.arrayBuffer();
+    const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
+    return analyzeMusicFromBuffer(audioBuffer);
+  } finally {
+    await audioContext.close();
+  }
+}
@@ -0,0 +1,113 @@
+// Persistence for beat data: one JSON file per audio file, matched by the
+// audio's project-relative path. Lives under `beats/` in the project so it
+// survives the audio being removed and re-added.
+
+interface BeatFileData {
+  version: 1;
+  audio: string;
+  beats: { time: number; strength: number }[];
+}
+
+/** Project-relative path of the audio file behind a (possibly absolute) src URL. */
+export function audioRelPathForSrc(src: string | null | undefined): string | null {
+  if (!src) return null;
+  // blob:/data: URLs have no stable identity across sessions — not persistable.
+  if (/^(blob:|data:)/i.test(src)) return null;
+  // Studio preview URLs: /api/projects/<id>/preview[/comp]/<relpath>.
+  // Parsed with indexOf/slice (not a regex) to avoid polynomial backtracking
+  // on adversarial inputs (CodeQL js/polynomial-redos).
+  let rel: string | null = null;
+  const PREVIEW = "/preview/";
+  const previewIdx = src.indexOf(PREVIEW);
+  if (previewIdx !== -1) {
+    let after = src.slice(previewIdx + PREVIEW.length);
+    // Strip query/hash (single char class — linear, ReDoS-safe).
+    const queryOrHash = after.search(/[?#]/);
+    if (queryOrHash !== -1) after = after.slice(0, queryOrHash);
+    if (after.startsWith("comp/")) after = after.slice("comp/".length);
+    rel = after ? decodeURIComponent(after) : null;
+  }
+  if (!rel) {
+    // Fall back to the FULL pathname (not just basename) so two files with the
+    // same name in different folders don't collide on one beat file.
+    try {
+      rel = decodeURIComponent(new URL(src, "http://_").pathname);
+    } catch {
+      rel = src;
+    }
+  }
+  if (!rel) return null;
+  rel = rel.replace(/^\/+/, "");
+  return rel || null;
+}
+
+/** Path of the beat file for a given audio src, or null if it can't be derived. */
+export function beatFilePathForSrc(src: string | null | undefined): string | null {
+  const rel = audioRelPathForSrc(src);
+  return rel ? `beats/${rel}.json` : null;
+}
+
+export function serializeBeats(times: number[], strengths: number[], audio: string): string {
+  const beats = times.map((t, i) => ({
+    time: Math.round(t * 1000) / 1000,
+    strength: Math.round((strengths[i] ?? 0.5) * 1000) / 1000,
+  }));
+  const data: BeatFileData = { version: 1, audio, beats };
+  return `${JSON.stringify(data, null, 2)}\n`;
+}
+
+export function parseBeats(content: string): { times: number[]; strengths: number[] } | null {
+  try {
+    const data = JSON.parse(content) as BeatFileData;
+    // Gate on the schema version so a future v2 file (with changed semantics)
+    // isn't silently parsed as v1 — an unknown version is treated as absent.
+    if (!data || data.version !== 1 || !Array.isArray(data.beats)) return null;
+    const times: number[] = [];
+    const strengths: number[] = [];
+    for (const b of data.beats) {
+      if (b && typeof b.time === "number" && Number.isFinite(b.time)) {
+        times.push(b.time);
+        // Clamp to [0,1] — a hand-edited file could carry an out-of-range or
+        // non-finite strength, and the renderers feed it into Math.pow(s, 2.2)
+        // (NaN for a negative base).
+        const s = typeof b.strength === "number" && Number.isFinite(b.strength) ? b.strength : 0.5;
+        strengths.push(Math.max(0, Math.min(1, s)));
+      }
+    }
+    return { times, strengths };
+  } catch {
+    return null;
+  }
+}
+
+const MUSIC_ID_RE = /\b(music|bgm|soundtrack|background[-_]?music)\b/i;
+
+function attr(tag: string, name: string): string | null {
+  const m = tag.match(new RegExp(`\\b${name}\\s*=\\s*["']([^"']*)["']`, "i"));
+  return m ? m[1]! : null;
+}
+
+/**
+ * Find the music track's src in composition HTML, applying the SAME rules as the
+ * Studio's `isMusicTrack` so the CLI and Studio agree on which `<audio>` is music:
+ * the FIRST `<audio>` (in document order) where data-timeline-role="music", or —
+ * when no role is set — whose id matches the music regex. An explicit non-music
+ * role excludes the element. Returns the raw src attribute, or null.
+ */
+export function findMusicAudioSrc(html: string): string | null {
+  // `[^>]*` spans newlines (it's a negated class, not `.`), so multi-line opening
+  // tags are handled. HyperFrames authors src as an attribute on <audio>.
+  const tags = html.match(/<audio\b[^>]*>/gi) ?? [];
+  for (const tag of tags) {
+    const src = attr(tag, "src");
+    if (!src) continue;
+    const role = attr(tag, "data-timeline-role");
+    if (role) {
+      if (role === "music") return src;
+      continue; // explicit non-music role excludes
+    }
+    const id = attr(tag, "id");
+    if (id && MUSIC_ID_RE.test(id)) return src;
+  }
+  return null;
+}
@@ -0,0 +1,2 @@
+export * from "./beatDetection";
+export * from "./beatFile";