feat(studio,cli): music beat detection with timeline guides + headless beats CLI (#1424)

* feat(studio,cli): music beat detection with timeline guides + headless beats CLI

Beat detection for music tracks: the Studio draws beat guides on the active
track, beats are user-editable and persist to a project file, and a new
`hyperframes beats` CLI generates that file headlessly before the Studio opens.

Detection lives in @hyperframes/core/beats (shared by Studio + CLI): an energy
onset detector cross-validated with bpm-detective, regularized to an octave-
aligned grid, silence-gated, with per-beat loudness. Music-only — an
<audio data-timeline-role="music"> is analyzed; voiceover is excluded.

Studio: green beat lines + draggable dots on the selected track; add at playhead,
drag to move, double-click to delete (audio scrubs); edits persist to
beats/<audio>.json and are undoable (interleaved with file history).

CLI: `hyperframes beats [dir]` runs the same detection in headless Chrome
(prebuilt browser bundle in dist) and writes the beat file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* feat(studio): timeline beat-grid + zoom UX refinements

- Center-anchored magnify: zooming via the toolbar/slider keeps the time
  at the viewport center fixed instead of anchoring at the left. Pinch
  still anchors at the cursor.
- Move-snap to beats: dragging a clip snaps whichever edge (start or end)
  is nearest a beat, matching the existing resize-edge snapping.
- Beat lines on track backgrounds: faint full-height beat lines now paint
  behind the clips on every track lane (brightness scales with loudness);
  the green dots stay on the active track's top bar.
- Waveform follows zoom: bars fill the full clip width and resample the
  windowed peaks, so the waveform stretches with zoom instead of stopping
  partway across a widened clip.
- Beat dots centered in the top bar: align the dot band to the clip top
  (CLIP_Y) so the dots sit centered in the dark bar instead of being
  bisected by the clip's top border.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* fix(studio): preserve media sourceDuration across element re-derivation

Moving a non-music clip re-derived the timeline elements into fresh
objects whose sourceDuration the DOM scan hadn't loaded yet. The async
probe skips srcs already in its cache, so the value was silently
dropped — trimFractions then returned no window and the trimmed music
waveform reset to the full source pinned at the track start.

Re-apply the cached probe duration synchronously on every derivation
(applyCachedSourceDurations) and extract the async probe loop into
probeMissingSourceDurations to keep useTimelinePlayer within the file
size limit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* feat(studio): skip beat-snap on the music track, highlight move-snap target

The music track defines the beats, so moving or trimming it no longer
snaps to its own beats (isMusicTrack guard on both the move and resize
snap paths).

Moving another clip snapped only on drop with no cue. snapMoveStartToBeat
now also returns the beat it will snap to; BeatBackgroundLines draws that
beat's line as a bright neon-green glow while the clip's edge is within
the snap region, so the target is visible before drop.

Also drops .commitmsg.tmp, accidentally committed via git add -A.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* feat(studio): hide playhead while dragging a beat; default beat dots to music track

- Dragging a beat dot now hides the playhead guideline (new beatDragging
  store flag set on beat pointer down/up) so its line doesn't track the
  scrub and clutter the beat being moved.
- Beat dots render on the selected track, falling back to the music track
  when nothing is selected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* fix(core): remove polynomial-ReDoS regex from audioRelPathForSrc

CodeQL js/polynomial-redos: the lazy `.+?` followed by an optional
trailing `[?#].*$` backtracks polynomially on crafted `/preview/...`
inputs. Parse the preview-relative path with indexOf/slice instead, and
strip the query/hash with a single linear char-class search. Behavior is
unchanged for all preview/absolute/blob/data/bare inputs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* fix(studio,core,cli): review hardening for beat detection + timeline UX

- playerStore.reset() now clears beat state (analysis, edits, undo/redo,
  persist) so a project switch can't apply the previous project's beats,
  undo stack, or file-writer to the new one.
- removeUserBeat returns the same reference on a no-op, and delete/move beat
  actions skip committing when nothing changed — no more phantom undo
  entries / debounced writes for no-op edits.
- regularizeBeats bails to raw onsets when the (octave-misread) tempo would
  produce a sub-125ms grid, avoiding a tens-of-thousands-of-beats freeze.
- parseBeats clamps strength to [0,1] and rejects non-finite time/strength,
  so a hand-edited file can't feed NaN into the gamma curve (Math.pow on a
  negative base) and blank out beat markers.
- Start-edge beat-snap now also requires duration >= minDuration, matching
  the end-edge guard, so a rightward snap can't collapse the clip.
- Center-anchor zoom effect always consumes its skip flag, so a pinch that
  produced no pps change can't leave it stranded and skip the next zoom.
- Headless beats analyzer projects to {beatTimes,beatStrengths,bpm,confidence}
  before returning, so page.evaluate no longer serializes the full decoded
  PCM (channelData) across the CDP boundary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

* fix(core): gate parseBeats on schema version

parseBeats accepted any object with a beats array, so a future v2 beat file
(with changed semantics) would be parsed silently as v1. Reject anything whose
version is not 1, treating an unknown version like an absent/invalid file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>

---------

Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
This commit is contained in:
Vance Ingalls
2026-06-14 17:17:13 -07:00
committed by GitHub
co-authored by Claude Opus 4.8 Miguel Ángel
parent a95e49dbda
commit d9f69f61e7
33 changed files with 1945 additions and 73 deletions
+14 -10
View File
@@ -22,7 +22,7 @@
}, },
"packages/aws-lambda": { "packages/aws-lambda": {
"name": "@hyperframes/aws-lambda", "name": "@hyperframes/aws-lambda",
"version": "0.6.90", "version": "0.6.95",
"dependencies": { "dependencies": {
"@aws-sdk/client-s3": "^3.700.0", "@aws-sdk/client-s3": "^3.700.0",
"@aws-sdk/client-sfn": "^3.700.0", "@aws-sdk/client-sfn": "^3.700.0",
@@ -54,7 +54,7 @@
}, },
"packages/cli": { "packages/cli": {
"name": "@hyperframes/cli", "name": "@hyperframes/cli",
"version": "0.6.90", "version": "0.6.95",
"bin": { "bin": {
"hyperframes": "./dist/cli.js", "hyperframes": "./dist/cli.js",
}, },
@@ -101,10 +101,11 @@
}, },
"packages/core": { "packages/core": {
"name": "@hyperframes/core", "name": "@hyperframes/core",
"version": "0.6.90", "version": "0.6.95",
"dependencies": { "dependencies": {
"@babel/parser": "^7.27.0", "@babel/parser": "^7.27.0",
"@chenglou/pretext": "^0.0.5", "@chenglou/pretext": "^0.0.5",
"bpm-detective": "^2.0.5",
"postcss": "^8.5.8", "postcss": "^8.5.8",
"postcss-selector-parser": "^7.1.2", "postcss-selector-parser": "^7.1.2",
"recast": "^0.23.11", "recast": "^0.23.11",
@@ -131,7 +132,7 @@
}, },
"packages/engine": { "packages/engine": {
"name": "@hyperframes/engine", "name": "@hyperframes/engine",
"version": "0.6.90", "version": "0.6.95",
"dependencies": { "dependencies": {
"@hono/node-server": "^1.13.0", "@hono/node-server": "^1.13.0",
"@hyperframes/core": "workspace:^", "@hyperframes/core": "workspace:^",
@@ -149,7 +150,7 @@
}, },
"packages/gcp-cloud-run": { "packages/gcp-cloud-run": {
"name": "@hyperframes/gcp-cloud-run", "name": "@hyperframes/gcp-cloud-run",
"version": "0.6.90", "version": "0.6.95",
"dependencies": { "dependencies": {
"@google-cloud/storage": "^7.14.0", "@google-cloud/storage": "^7.14.0",
"@google-cloud/workflows": "^4.2.0", "@google-cloud/workflows": "^4.2.0",
@@ -169,7 +170,7 @@
}, },
"packages/player": { "packages/player": {
"name": "@hyperframes/player", "name": "@hyperframes/player",
"version": "0.6.90", "version": "0.6.95",
"devDependencies": { "devDependencies": {
"@types/bun": "^1.1.0", "@types/bun": "^1.1.0",
"gsap": "^3.12.5", "gsap": "^3.12.5",
@@ -181,7 +182,7 @@
}, },
"packages/producer": { "packages/producer": {
"name": "@hyperframes/producer", "name": "@hyperframes/producer",
"version": "0.6.90", "version": "0.6.95",
"dependencies": { "dependencies": {
"@fontsource/archivo-black": "^5.2.8", "@fontsource/archivo-black": "^5.2.8",
"@fontsource/eb-garamond": "^5.2.7", "@fontsource/eb-garamond": "^5.2.7",
@@ -222,7 +223,7 @@
}, },
"packages/sdk": { "packages/sdk": {
"name": "@hyperframes/sdk", "name": "@hyperframes/sdk",
"version": "0.6.86", "version": "0.6.91",
"dependencies": { "dependencies": {
"@hyperframes/core": "workspace:*", "@hyperframes/core": "workspace:*",
"linkedom": "^0.18.12", "linkedom": "^0.18.12",
@@ -235,7 +236,7 @@
}, },
"packages/shader-transitions": { "packages/shader-transitions": {
"name": "@hyperframes/shader-transitions", "name": "@hyperframes/shader-transitions",
"version": "0.6.90", "version": "0.6.95",
"dependencies": { "dependencies": {
"html2canvas": "^1.4.1", "html2canvas": "^1.4.1",
}, },
@@ -247,7 +248,7 @@
}, },
"packages/studio": { "packages/studio": {
"name": "@hyperframes/studio", "name": "@hyperframes/studio",
"version": "0.6.90", "version": "0.6.95",
"dependencies": { "dependencies": {
"@codemirror/autocomplete": "^6.20.1", "@codemirror/autocomplete": "^6.20.1",
"@codemirror/commands": "^6.10.3", "@codemirror/commands": "^6.10.3",
@@ -262,6 +263,7 @@
"@hyperframes/core": "workspace:*", "@hyperframes/core": "workspace:*",
"@hyperframes/player": "workspace:*", "@hyperframes/player": "workspace:*",
"@phosphor-icons/react": "^2.1.10", "@phosphor-icons/react": "^2.1.10",
"bpm-detective": "^2.0.5",
"mediabunny": "^1.45.3", "mediabunny": "^1.45.3",
}, },
"devDependencies": { "devDependencies": {
@@ -1158,6 +1160,8 @@
"bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="], "bowser": ["bowser@2.14.1", "", {}, "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg=="],
"bpm-detective": ["bpm-detective@2.0.5", "", {}, "sha512-FaHFT5WDCR5zwtjVTqxk01o2MDaf22ORHityX1lxMamBcVNX6NWXB6hw7nx4fmlEseRMv0xfuSybINZylljLfA=="],
"brace-expansion": ["brace-expansion@5.0.6", "", { "dependencies": { "balanced-match": "^4.0.2" } }, "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g=="], "brace-expansion": ["brace-expansion@5.0.6", "", { "dependencies": { "balanced-match": "^4.0.2" } }, "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g=="],
"braces": ["braces@3.0.3", "", { "dependencies": { "fill-range": "^7.1.1" } }, "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA=="], "braces": ["braces@3.0.3", "", { "dependencies": { "fill-range": "^7.1.1" } }, "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA=="],
+27
View File
@@ -512,6 +512,33 @@ This is suppressed in CI environments, non-TTY shells, and when `HYPERFRAMES_NO_
The linter detects missing attributes, missing adapter libraries (GSAP, Lottie, Three.js), structural problems, and more. See [Common Mistakes](/guides/common-mistakes) for details on each rule. The linter detects missing attributes, missing adapter libraries (GSAP, Lottie, Three.js), structural problems, and more. See [Common Mistakes](/guides/common-mistakes) for details on each rule.
### `beats`
Detect the beats in a composition's music track and write them to a beat file the Studio uses to draw beat guides on the timeline:
```bash
npx hyperframes beats [dir]
npx hyperframes beats [dir] --json # machine-readable JSON output
```
The command finds the music track (an `<audio>` element with `data-timeline-role="music"`, or an id like `music`/`bgm`/`soundtrack`), runs the **same** detection the Studio uses inside a headless Chrome (identical decode + BPM analysis), and writes `beats/<audio-path>.json`:
```json
{
"version": 1,
"audio": "music.wav",
"beats": [{ "time": 2.027, "strength": 0.924 }]
}
```
Run it when authoring a composition so the beat file exists **before** the Studio is opened — the Studio loads this file as-is (it only auto-generates one when none exists). `time` is in seconds into the audio file; `strength` (01) is the beat's relative loudness. Beats edited in the Studio (add/move/delete) persist back to the same file.
| Flag | Description |
|------|-------------|
| `--json` | Output `{ ok, file, count, bpm }` as JSON |
Requires a local Chrome (the same one used by `render`; run `npx hyperframes browser ensure` if missing). Detection runs the **same** algorithm the Studio uses; results are near-identical (a different headless-Chrome audio sample rate can shift beat times by a frame or two).
### `inspect` ### `inspect`
Inspect rendered visual layout across the composition timeline: Inspect rendered visual layout across the composition timeline:
+2 -1
View File
@@ -17,9 +17,10 @@
"scripts": { "scripts": {
"test": "vitest run", "test": "vitest run",
"dev": "tsx src/cli.ts", "dev": "tsx src/cli.ts",
"build": "bun run build:fonts && tsup && bun run build:runtime && bun run build:copy", "build": "bun run build:fonts && tsup && bun run build:runtime && bun run build:beat-analyzer && bun run build:copy",
"build:fonts": "node scripts/build-fonts.mjs", "build:fonts": "node scripts/build-fonts.mjs",
"build:runtime": "tsx scripts/build-runtime.ts", "build:runtime": "tsx scripts/build-runtime.ts",
"build:beat-analyzer": "node scripts/build-beat-analyzer.mjs",
"build:copy": "node scripts/build-copy.mjs", "build:copy": "node scripts/build-copy.mjs",
"typecheck": "tsc --noEmit" "typecheck": "tsc --noEmit"
}, },
@@ -0,0 +1,28 @@
// Prebuild the beat-detection browser bundle into dist so `hyperframes beats`
// works in the published CLI (which ships only dist, not source). Mirrors how
// the runtime IIFE is shipped. headlessAnalyzer.ts loads this at runtime and
// injects it into a headless page.
import { build } from "esbuild";
import { createRequire } from "node:module";
import { dirname, join } from "node:path";
const require = createRequire(import.meta.url);
const coreRoot = dirname(require.resolve("@hyperframes/core/package.json"));
const entry = join(coreRoot, "src/beats/beatDetection.ts");
await build({
stdin: {
contents:
`import { analyzeMusicFromBuffer } from ${JSON.stringify(entry)};\n` +
`globalThis.__hfAnalyze = analyzeMusicFromBuffer;`,
resolveDir: coreRoot,
loader: "ts",
},
bundle: true,
format: "iife",
platform: "browser",
target: "es2020",
outfile: "dist/beat-analyzer.global.js",
});
console.log("built dist/beat-analyzer.global.js");
+152
View File
@@ -0,0 +1,152 @@
// Run the shared beat detection (@hyperframes/core/beats) in a headless Chrome
// so results match the Studio exactly — same Web Audio decode + same
// bpm-detective. Used by the `beats` CLI command to write the beat file before
// the Studio is ever opened.
import { existsSync, readFileSync } from "node:fs";
import { createRequire } from "node:module";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import type { Browser, Page } from "puppeteer-core";
const require = createRequire(import.meta.url);
// The detection is browser code. We need it as an IIFE that exposes
// analyzeMusicFromBuffer on the page. Prefer the artifact prebuilt at CLI build
// time (shipped in dist); fall back to bundling from core source at runtime
// (dev/monorepo, where core's src is on disk).
let bundlePromise: Promise<string> | null = null;
function findPrebuiltBundle(): string | null {
const here = dirname(fileURLToPath(import.meta.url));
const candidates = [
join(here, "beat-analyzer.global.js"), // dist root (tsup-bundled cli)
join(here, "../beat-analyzer.global.js"), // dist/beats → dist
join(here, "../dist/beat-analyzer.global.js"),
];
for (const p of candidates) {
if (existsSync(p)) return p;
}
return null;
}
async function buildFromCoreSource(): Promise<string> {
const esbuild = await import("esbuild");
const coreRoot = dirname(require.resolve("@hyperframes/core/package.json"));
const entry = join(coreRoot, "src/beats/beatDetection.ts");
const result = await esbuild.build({
stdin: {
contents:
`import { analyzeMusicFromBuffer } from ${JSON.stringify(entry)};\n` +
`globalThis.__hfAnalyze = analyzeMusicFromBuffer;`,
resolveDir: coreRoot,
loader: "ts",
},
bundle: true,
format: "iife",
platform: "browser",
target: "es2020",
write: false,
});
const out = result.outputFiles?.[0];
if (!out) throw new Error("Failed to bundle beat analyzer");
return out.text;
}
function buildAnalyzerBundle(): Promise<string> {
if (bundlePromise) return bundlePromise;
bundlePromise = (async () => {
const prebuilt = findPrebuiltBundle();
if (prebuilt) return readFileSync(prebuilt, "utf8");
return buildFromCoreSource();
})().catch((err) => {
bundlePromise = null; // don't poison the process with a cached rejection
throw err;
});
return bundlePromise;
}
export interface HeadlessBeatResult {
beatTimes: number[];
beatStrengths: number[];
bpm: number | null;
bpmConfidence: string;
}
// Guard against pathological inputs that would blow CDP message limits when
// transferred to the page as base64 (≈ +33% over the raw bytes).
const MAX_AUDIO_BYTES = 80 * 1024 * 1024;
// Runs inside the headless page: decode the base64 audio and analyze it.
function inPageAnalyze(data: string) {
const bin = atob(data);
const bytes = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
const win = window as unknown as {
AudioContext: typeof AudioContext;
webkitAudioContext?: typeof AudioContext;
__hfAnalyze?: (buffer: AudioBuffer) => Promise<HeadlessBeatResult>;
};
if (typeof win.__hfAnalyze !== "function") throw new Error("beat analyzer not loaded");
const ctx = new (win.AudioContext || win.webkitAudioContext!)();
return (
ctx
.decodeAudioData(bytes.buffer)
.then((buf) => win.__hfAnalyze!(buf))
// analyzeMusicFromBuffer also returns the decoded PCM (channelData) + sampleRate;
// project to only the fields we need so page.evaluate doesn't serialize an
// ~8-million-element Float32Array back across the CDP boundary.
.then((r) => ({
beatTimes: r.beatTimes,
beatStrengths: r.beatStrengths,
bpm: r.bpm,
bpmConfidence: r.bpmConfidence,
}))
.finally(() => ctx.close())
);
}
// Load the analyzer bundle into the page, run analysis, and surface in-page
// errors (decode/codec failures, missing global) instead of an opaque rejection.
async function detectOnPage(page: Page, bundle: string, b64: string): Promise<HeadlessBeatResult> {
const pageErrors: string[] = [];
page.on("pageerror", (e) => {
pageErrors.push((e as Error).message);
});
page.on("console", (m) => {
if (m.type() === "error") pageErrors.push(m.text());
});
await page.setContent("<!doctype html><html><body></body></html>");
await page.addScriptTag({ content: bundle });
try {
return (await page.evaluate(inPageAnalyze, b64)) as HeadlessBeatResult;
} catch (err) {
const detail = pageErrors.length ? ` (${pageErrors.join("; ")})` : "";
throw new Error(`${err instanceof Error ? err.message : String(err)}${detail}`);
}
}
/** Decode + analyze the given audio bytes in headless Chrome. */
export async function analyzeBeatsHeadless(audioBytes: Buffer): Promise<HeadlessBeatResult> {
if (audioBytes.length > MAX_AUDIO_BYTES) {
const mb = Math.round(audioBytes.length / 1e6);
throw new Error(
`Audio file too large for headless analysis (${mb}MB > ${MAX_AUDIO_BYTES / 1e6}MB).`,
);
}
const bundle = await buildAnalyzerBundle();
const { ensureBrowser } = await import("../browser/manager.js");
const puppeteer = await import("puppeteer-core");
const browser = await ensureBrowser();
const chrome: Browser = await puppeteer.default.launch({
headless: true,
executablePath: browser.executablePath,
args: ["--no-sandbox", "--disable-dev-shm-usage", "--autoplay-policy=no-user-gesture-required"],
});
try {
const page = await chrome.newPage();
return await detectOnPage(page, bundle, audioBytes.toString("base64"));
} finally {
await chrome.close();
}
}
+1
View File
@@ -117,6 +117,7 @@ const subCommands = {
publish: () => import("./commands/publish.js").then((m) => m.default), publish: () => import("./commands/publish.js").then((m) => m.default),
render: () => import("./commands/render.js").then((m) => m.default), render: () => import("./commands/render.js").then((m) => m.default),
lint: () => import("./commands/lint.js").then((m) => m.default), lint: () => import("./commands/lint.js").then((m) => m.default),
beats: () => import("./commands/beats.js").then((m) => m.default),
inspect: () => import("./commands/inspect.js").then((m) => m.default), inspect: () => import("./commands/inspect.js").then((m) => m.default),
layout: () => import("./commands/layout.js").then((m) => m.default), layout: () => import("./commands/layout.js").then((m) => m.default),
info: () => import("./commands/info.js").then((m) => m.default), info: () => import("./commands/info.js").then((m) => m.default),
+87
View File
@@ -0,0 +1,87 @@
import { defineCommand } from "citty";
import { existsSync, readFileSync, mkdirSync, writeFileSync } from "node:fs";
import { resolve, join, dirname } from "node:path";
import { findMusicAudioSrc, audioRelPathForSrc, serializeBeats } from "@hyperframes/core/beats";
import type { Example } from "./_examples.js";
import { resolveProject, type ProjectDir } from "../utils/project.js";
import { analyzeBeatsHeadless, type HeadlessBeatResult } from "../beats/headlessAnalyzer.js";
import { c } from "../ui/colors.js";
export const examples: Example[] = [
["Generate the beat file for the current project", "hyperframes beats"],
["Generate for a specific directory", "hyperframes beats ./my-video"],
];
function fail(message: string): never {
console.error(c.error(message));
process.exit(1);
}
/** Locate the music track + its on-disk audio, or fail with a clear message. */
function resolveMusicTarget(project: ProjectDir): { rel: string; audioPath: string } {
const src = findMusicAudioSrc(readFileSync(project.indexPath, "utf-8"));
if (!src) {
fail(
'No music track found. Add data-timeline-role="music" to the <audio> element ' +
"(or give it an id like music/bgm/soundtrack).",
);
}
const rel = audioRelPathForSrc(src); // same derivation the Studio uses
if (!rel) fail(`Cannot derive a beat-file path for music src: ${src}`);
const audioPath = resolve(project.dir, rel);
if (!existsSync(audioPath)) fail(`Audio file not found: ${rel}`);
return { rel, audioPath };
}
async function detect(audioPath: string): Promise<HeadlessBeatResult> {
try {
return await analyzeBeatsHeadless(readFileSync(audioPath));
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
const hint = /chrome|executable|browser|ENOENT/i.test(msg)
? "\nRun: npx hyperframes browser ensure"
: "";
fail(`Beat detection failed: ${msg}${hint}`);
}
}
function report(file: string, result: HeadlessBeatResult, json: boolean): void {
if (json) {
console.log(
JSON.stringify({ ok: true, file, count: result.beatTimes.length, bpm: result.bpm }, null, 2),
);
return;
}
console.log(
c.success(
`✓ Wrote ${result.beatTimes.length} beats → ${file} (bpm ${result.bpm ?? "?"}, ${result.bpmConfidence})`,
),
);
}
export default defineCommand({
meta: {
name: "beats",
description: "Detect beats in the music track (headless) and write beats/<audio>.json",
},
args: {
dir: { type: "positional", description: "Project directory", required: false },
json: { type: "boolean", description: "Output result as JSON", default: false },
},
async run({ args }) {
const project = resolveProject(args.dir);
const { rel, audioPath } = resolveMusicTarget(project);
if (!args.json) console.log(c.dim(`Analyzing ${rel} in headless Chrome…`));
const result = await detect(audioPath);
// The Studio ignores a 0-beat file (treats it as a stale seed), so don't write one.
if (result.beatTimes.length === 0) {
fail(`No beats detected in ${rel} — nothing written. (Track may be silent/ambient.)`);
}
const outPath = join(project.dir, "beats", `${rel}.json`);
mkdirSync(dirname(outPath), { recursive: true });
writeFileSync(outPath, serializeBeats(result.beatTimes, result.beatStrengths, rel));
report(`beats/${rel}.json`, result, Boolean(args.json));
},
});
+1
View File
@@ -32,6 +32,7 @@ const GROUPS: Group[] = [
title: "Project", title: "Project",
commands: [ commands: [
["lint", "Validate a composition for common mistakes"], ["lint", "Validate a composition for common mistakes"],
["beats", "Detect beats in the music track and write beats/<audio>.json"],
["inspect", "Inspect rendered visual layout across the timeline"], ["inspect", "Inspect rendered visual layout across the timeline"],
["snapshot", "Capture key frames as PNG screenshots for visual verification"], ["snapshot", "Capture key frames as PNG screenshots for visual verification"],
["info", "Print project metadata"], ["info", "Print project metadata"],
+6
View File
@@ -21,6 +21,11 @@
"import": "./src/index.ts", "import": "./src/index.ts",
"types": "./src/index.ts" "types": "./src/index.ts"
}, },
"./package.json": "./package.json",
"./beats": {
"import": "./src/beats/index.ts",
"types": "./src/beats/index.ts"
},
"./lint": { "./lint": {
"import": "./src/lint/index.ts", "import": "./src/lint/index.ts",
"types": "./src/lint/index.ts" "types": "./src/lint/index.ts"
@@ -208,6 +213,7 @@
"dependencies": { "dependencies": {
"@babel/parser": "^7.27.0", "@babel/parser": "^7.27.0",
"@chenglou/pretext": "^0.0.5", "@chenglou/pretext": "^0.0.5",
"bpm-detective": "^2.0.5",
"postcss": "^8.5.8", "postcss": "^8.5.8",
"postcss-selector-parser": "^7.1.2", "postcss-selector-parser": "^7.1.2",
"recast": "^0.23.11" "recast": "^0.23.11"
+285
View File
@@ -0,0 +1,285 @@
// bpm-detective touches `window` at module top-level, so it must NOT be a static
// import (that would crash any non-browser import of this module graph, e.g.
// vitest/SSR). It's loaded lazily inside analyzeMusicFromBuffer, which only runs
// in the browser.
type BpmDetect = (buffer: AudioBuffer) => number;
let bpmDetectivePromise: Promise<BpmDetect | null> | null = null;
function loadBpmDetective(): Promise<BpmDetect | null> {
if (!bpmDetectivePromise) {
bpmDetectivePromise = import(
// @ts-ignore -- no type declarations for bpm-detective
"bpm-detective"
)
.then((m) => ((m as { default?: BpmDetect }).default ?? (m as unknown as BpmDetect)) || null)
.catch(() => null);
}
return bpmDetectivePromise;
}
const WINDOW_SIZE = 1024;
const HOP_SIZE = 512;
export interface MusicBeatAnalysis {
beatTimes: number[];
/** Per-beat loudness 01 (local RMS / peak), aligned by index with beatTimes. */
beatStrengths: number[];
bpm: number | null;
bpmConfidence: "high" | "low" | "uncertain";
/** Decoded mono samples — retained so strength can be measured at user-added
* beats. Audio-file coordinates. May be null if decode data was dropped. */
channelData: Float32Array | null;
sampleRate: number;
/** Reference peak RMS used to normalize beat strengths. */
peak: number;
}
const STRENGTH_WINDOW_S = 0.05; // ±50ms RMS window
/** Local RMS amplitude at a given audio-file time. */
export function computeRmsAt(channelData: Float32Array, sampleRate: number, time: number): number {
const halfWindow = Math.floor(sampleRate * STRENGTH_WINDOW_S);
const center = Math.floor(time * sampleRate);
const start = Math.max(0, center - halfWindow);
const end = Math.min(channelData.length, center + halfWindow);
let sum = 0;
for (let i = start; i < end; i++) {
const s = channelData[i] ?? 0;
sum += s * s;
}
return Math.sqrt(sum / Math.max(end - start, 1));
}
/** Normalized beat strength (01) at an audio-file time, using a track peak. */
export function strengthAtTime(
analysis: Pick<MusicBeatAnalysis, "channelData" | "sampleRate" | "peak">,
time: number,
): number {
if (!analysis.channelData || analysis.peak <= 0) return 0.5;
return Math.min(1, computeRmsAt(analysis.channelData, analysis.sampleRate, time) / analysis.peak);
}
// fallow-ignore-next-line complexity
export async function detectBeats(audioBuffer: AudioBuffer): Promise<number[]> {
const channelData = audioBuffer.getChannelData(0);
const sampleRate = audioBuffer.sampleRate;
const energies: number[] = [];
for (let i = 0; i < channelData.length - WINDOW_SIZE; i += HOP_SIZE) {
let sum = 0;
for (let j = 0; j < WINDOW_SIZE; j++) {
const sample = channelData[i + j]!;
sum += sample * sample;
}
energies.push(sum / WINDOW_SIZE);
}
const beats: number[] = [];
const localWindowSize = 20;
for (let i = localWindowSize; i < energies.length - localWindowSize; i++) {
let localMean = 0;
for (let j = i - localWindowSize; j < i + localWindowSize; j++) {
localMean += energies[j]!;
}
localMean /= localWindowSize * 2;
const threshold = localMean * 1.5;
const current = energies[i]!;
if (
current > threshold &&
current > (energies[i - 1] ?? 0) &&
current > (energies[i + 1] ?? 0)
) {
const timeInSeconds = (i * HOP_SIZE) / sampleRate;
if (beats.length === 0 || timeInSeconds - beats[beats.length - 1]! > 0.1) {
beats.push(Math.round(timeInSeconds * 1000) / 1000);
}
}
}
return beats;
}
function computeBpmFromBeats(beatTimes: number[]): number | null {
if (beatTimes.length < 4) return null;
const iois: number[] = [];
for (let i = 1; i < beatTimes.length; i++) {
iois.push(beatTimes[i]! - beatTimes[i - 1]!);
}
const sorted = [...iois].sort((a, b) => a - b);
const mid = Math.floor(sorted.length / 2);
const medianIoi = sorted.length % 2 === 0 ? (sorted[mid - 1]! + sorted[mid]!) / 2 : sorted[mid]!;
if (medianIoi <= 0) return null;
return Math.round((60 / medianIoi) * 10) / 10;
}
// Fold into 60120 for octave-safe BPM comparison
function canonicalizeBpm(bpm: number): number {
let b = bpm;
while (b > 120) b /= 2;
while (b < 60) b *= 2;
return b;
}
// Pick the *2/÷2 octave of `bpm` whose beat interval is closest to the onset
// pulse, so a half-time detective reading (e.g. 174 for an 87bpm song) doesn't
// produce a double-density grid.
function octaveAlignBpm(bpm: number, reference: number): number {
const candidates = [bpm / 2, bpm, bpm * 2];
let best = bpm;
let bestDist = Number.POSITIVE_INFINITY;
for (const c of candidates) {
if (c <= 0) continue;
const dist = Math.abs(c - reference);
if (dist < bestDist) {
bestDist = dist;
best = c;
}
}
return best;
}
// fallow-ignore-next-line complexity
function regularizeBeats(rawBeats: number[], bpm: number, duration: number): number[] {
if (rawBeats.length === 0 || bpm <= 0 || duration <= 0) return rawBeats;
const beatInterval = 60 / bpm;
// Guard against a pathological (octave-misread) tempo producing a millisecond
// interval → tens of thousands of grid beats that freeze the timeline. 480 BPM
// (0.125s) is well above any real music tempo; bail to the raw onsets instead.
if (beatInterval < 0.125) return rawBeats;
const threshold = beatInterval * 0.25;
// Find phase offset that maximally aligns with raw onsets
let bestOffset = 0;
let bestScore = -1;
for (const anchor of rawBeats.slice(0, 10)) {
const offset = ((anchor % beatInterval) + beatInterval) % beatInterval;
let score = 0;
for (const rb of rawBeats) {
const phase = ((rb % beatInterval) + beatInterval) % beatInterval;
const dist = Math.min(Math.abs(phase - offset), beatInterval - Math.abs(phase - offset));
if (dist < threshold) score++;
}
if (score > bestScore) {
bestScore = score;
bestOffset = offset;
}
}
const beats: number[] = [];
for (let t = bestOffset; t <= duration + 0.001; t += beatInterval) {
beats.push(Math.round(t * 1000) / 1000);
}
return beats;
}
// Drop beats that fall in silent/near-silent regions (e.g. intro/outro) and
// score each surviving beat's loudness (local RMS / peak) for brightness.
function gateBeatsBySilence(
beats: number[],
channelData: Float32Array,
sampleRate: number,
): { times: number[]; strengths: number[]; peak: number } {
if (beats.length === 0) return { times: beats, strengths: [], peak: 1e-6 };
const energies = beats.map((t) => computeRmsAt(channelData, sampleRate, t));
const peak = Math.max(...energies, 1e-6);
const threshold = peak * 0.12;
const times: number[] = [];
const strengths: number[] = [];
for (let i = 0; i < beats.length; i++) {
if (energies[i]! >= threshold) {
times.push(beats[i]!);
strengths.push(Math.min(1, energies[i]! / peak));
}
}
return { times, strengths, peak };
}
// fallow-ignore-next-line complexity
export async function analyzeMusicFromBuffer(audioBuffer: AudioBuffer): Promise<MusicBeatAnalysis> {
const channelData = audioBuffer.getChannelData(0);
const sampleRate = audioBuffer.sampleRate;
const duration = audioBuffer.duration;
const rawBeats = await detectBeats(audioBuffer);
const onsetBpm = computeBpmFromBeats(rawBeats);
let detectiveBpm: number | null = null;
try {
const detect = await loadBpmDetective();
if (detect) detectiveBpm = detect(audioBuffer);
} catch {
// Not enough peaks or browser context unavailable
}
let bpm: number | null = onsetBpm;
let confidence: MusicBeatAnalysis["bpmConfidence"] = "uncertain";
let regularizeBpm: number | null = null;
if (onsetBpm !== null && detectiveBpm !== null) {
const pctDiff =
Math.abs(canonicalizeBpm(onsetBpm) - canonicalizeBpm(detectiveBpm)) /
canonicalizeBpm(detectiveBpm);
if (pctDiff < 0.05) {
// Detective folds tempo into 90180; re-pick the octave nearest the onset
// pulse so a half-time track isn't gridded at double density.
bpm = octaveAlignBpm(detectiveBpm, onsetBpm);
confidence = "high";
regularizeBpm = bpm;
} else if (pctDiff < 0.1) {
bpm = Math.round((onsetBpm + detectiveBpm) / 2);
confidence = "low";
regularizeBpm = bpm;
} else {
bpm = onsetBpm;
confidence = "uncertain";
}
} else if (onsetBpm !== null) {
bpm = onsetBpm;
confidence = "low";
regularizeBpm = onsetBpm;
} else if (detectiveBpm !== null) {
bpm = detectiveBpm;
confidence = "low";
regularizeBpm = detectiveBpm;
}
const gridBeats =
regularizeBpm !== null ? regularizeBeats(rawBeats, regularizeBpm, duration) : rawBeats;
const gated = gateBeatsBySilence(gridBeats, channelData, sampleRate);
return {
beatTimes: gated.times,
beatStrengths: gated.strengths,
bpm,
bpmConfidence: confidence,
channelData,
sampleRate,
peak: gated.peak,
};
}
export async function detectBeatsFromUrl(url: string): Promise<number[]> {
const audioContext = new AudioContext();
try {
const response = await fetch(url);
const arrayBuffer = await response.arrayBuffer();
const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
return detectBeats(audioBuffer);
} finally {
await audioContext.close();
}
}
export async function analyzeMusicFromUrl(url: string): Promise<MusicBeatAnalysis> {
const audioContext = new AudioContext();
try {
const response = await fetch(url);
const arrayBuffer = await response.arrayBuffer();
const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
return analyzeMusicFromBuffer(audioBuffer);
} finally {
await audioContext.close();
}
}
+113
View File
@@ -0,0 +1,113 @@
// Persistence for beat data: one JSON file per audio file, matched by the
// audio's project-relative path. Lives under `beats/` in the project so it
// survives the audio being removed and re-added.
interface BeatFileData {
version: 1;
audio: string;
beats: { time: number; strength: number }[];
}
/** Project-relative path of the audio file behind a (possibly absolute) src URL. */
export function audioRelPathForSrc(src: string | null | undefined): string | null {
if (!src) return null;
// blob:/data: URLs have no stable identity across sessions — not persistable.
if (/^(blob:|data:)/i.test(src)) return null;
// Studio preview URLs: /api/projects/<id>/preview[/comp]/<relpath>.
// Parsed with indexOf/slice (not a regex) to avoid polynomial backtracking
// on adversarial inputs (CodeQL js/polynomial-redos).
let rel: string | null = null;
const PREVIEW = "/preview/";
const previewIdx = src.indexOf(PREVIEW);
if (previewIdx !== -1) {
let after = src.slice(previewIdx + PREVIEW.length);
// Strip query/hash (single char class — linear, ReDoS-safe).
const queryOrHash = after.search(/[?#]/);
if (queryOrHash !== -1) after = after.slice(0, queryOrHash);
if (after.startsWith("comp/")) after = after.slice("comp/".length);
rel = after ? decodeURIComponent(after) : null;
}
if (!rel) {
// Fall back to the FULL pathname (not just basename) so two files with the
// same name in different folders don't collide on one beat file.
try {
rel = decodeURIComponent(new URL(src, "http://_").pathname);
} catch {
rel = src;
}
}
if (!rel) return null;
rel = rel.replace(/^\/+/, "");
return rel || null;
}
/** Path of the beat file for a given audio src, or null if it can't be derived. */
export function beatFilePathForSrc(src: string | null | undefined): string | null {
const rel = audioRelPathForSrc(src);
return rel ? `beats/${rel}.json` : null;
}
export function serializeBeats(times: number[], strengths: number[], audio: string): string {
const beats = times.map((t, i) => ({
time: Math.round(t * 1000) / 1000,
strength: Math.round((strengths[i] ?? 0.5) * 1000) / 1000,
}));
const data: BeatFileData = { version: 1, audio, beats };
return `${JSON.stringify(data, null, 2)}\n`;
}
export function parseBeats(content: string): { times: number[]; strengths: number[] } | null {
try {
const data = JSON.parse(content) as BeatFileData;
// Gate on the schema version so a future v2 file (with changed semantics)
// isn't silently parsed as v1 — an unknown version is treated as absent.
if (!data || data.version !== 1 || !Array.isArray(data.beats)) return null;
const times: number[] = [];
const strengths: number[] = [];
for (const b of data.beats) {
if (b && typeof b.time === "number" && Number.isFinite(b.time)) {
times.push(b.time);
// Clamp to [0,1] — a hand-edited file could carry an out-of-range or
// non-finite strength, and the renderers feed it into Math.pow(s, 2.2)
// (NaN for a negative base).
const s = typeof b.strength === "number" && Number.isFinite(b.strength) ? b.strength : 0.5;
strengths.push(Math.max(0, Math.min(1, s)));
}
}
return { times, strengths };
} catch {
return null;
}
}
const MUSIC_ID_RE = /\b(music|bgm|soundtrack|background[-_]?music)\b/i;
function attr(tag: string, name: string): string | null {
const m = tag.match(new RegExp(`\\b${name}\\s*=\\s*["']([^"']*)["']`, "i"));
return m ? m[1]! : null;
}
/**
* Find the music track's src in composition HTML, applying the SAME rules as the
* Studio's `isMusicTrack` so the CLI and Studio agree on which `<audio>` is music:
* the FIRST `<audio>` (in document order) where data-timeline-role="music", or —
* when no role is set — whose id matches the music regex. An explicit non-music
* role excludes the element. Returns the raw src attribute, or null.
*/
export function findMusicAudioSrc(html: string): string | null {
// `[^>]*` spans newlines (it's a negated class, not `.`), so multi-line opening
// tags are handled. HyperFrames authors src as an attribute on <audio>.
const tags = html.match(/<audio\b[^>]*>/gi) ?? [];
for (const tag of tags) {
const src = attr(tag, "src");
if (!src) continue;
const role = attr(tag, "data-timeline-role");
if (role) {
if (role === "music") return src;
continue; // explicit non-music role excludes
}
const id = attr(tag, "id");
if (id && MUSIC_ID_RE.test(id)) return src;
}
return null;
}
+2
View File
@@ -0,0 +1,2 @@
export * from "./beatDetection";
export * from "./beatFile";
+3 -1
View File
@@ -16,7 +16,8 @@
"types": "./src/index.ts", "types": "./src/index.ts",
"exports": { "exports": {
".": "./src/index.ts", ".": "./src/index.ts",
"./tailwind-preset": "./src/styles/tailwind-preset.ts" "./tailwind-preset": "./src/styles/tailwind-preset.ts",
"./package.json": "./package.json"
}, },
"scripts": { "scripts": {
"dev": "vite", "dev": "vite",
@@ -39,6 +40,7 @@
"@hyperframes/core": "workspace:*", "@hyperframes/core": "workspace:*",
"@hyperframes/player": "workspace:*", "@hyperframes/player": "workspace:*",
"@phosphor-icons/react": "^2.1.10", "@phosphor-icons/react": "^2.1.10",
"bpm-detective": "^2.0.5",
"mediabunny": "^1.45.3" "mediabunny": "^1.45.3"
}, },
"devDependencies": { "devDependencies": {
@@ -16,6 +16,7 @@ import { Scissors } from "../icons/SystemIcons";
import type { GsapAnimation } from "@hyperframes/core/gsap-parser"; import type { GsapAnimation } from "@hyperframes/core/gsap-parser";
import type { DomEditSelection } from "./editor/domEditingTypes"; import type { DomEditSelection } from "./editor/domEditingTypes";
import { canSplitElement } from "../utils/timelineElementSplit"; import { canSplitElement } from "../utils/timelineElementSplit";
import { canAddBeatAt, addBeatAtCompositionTime } from "../utils/beatEditActions";
interface DomEditSessionSlice extends EnableKeyframesSession { interface DomEditSessionSlice extends EnableKeyframesSession {
domEditSelection: DomEditSelection | null; domEditSelection: DomEditSelection | null;
@@ -70,6 +71,9 @@ export function TimelineToolbar({
}: TimelineToolbarProps) { }: TimelineToolbarProps) {
const activeTool = usePlayerStore((s) => s.activeTool); const activeTool = usePlayerStore((s) => s.activeTool);
const setActiveTool = usePlayerStore((s) => s.setActiveTool); const setActiveTool = usePlayerStore((s) => s.setActiveTool);
// Subscribe so the add-beat button reacts to playhead movement and analysis load.
const currentTime = usePlayerStore((s) => s.currentTime);
const beatAnalysisReady = usePlayerStore((s) => s.beatAnalysis !== null);
const { zoomMode, manualZoomPercent, setZoomMode, setManualZoomPercent } = useTimelineZoom(); const { zoomMode, manualZoomPercent, setZoomMode, setManualZoomPercent } = useTimelineZoom();
const displayedTimelineZoomPercent = getTimelineZoomPercent(zoomMode, manualZoomPercent); const displayedTimelineZoomPercent = getTimelineZoomPercent(zoomMode, manualZoomPercent);
const { state: keyframeState, onToggle: onToggleKeyframe } = useKeyframeToggle(domEditSession); const { state: keyframeState, onToggle: onToggleKeyframe } = useKeyframeToggle(domEditSession);
@@ -178,6 +182,27 @@ export function TimelineToolbar({
</Tooltip> </Tooltip>
); );
})()} })()}
{beatAnalysisReady &&
canAddBeatAt(currentTime) &&
(() => (
<Tooltip label="Add beat at playhead">
<button
type="button"
onClick={() => addBeatAtCompositionTime(currentTime)}
className="flex h-7 w-7 items-center justify-center rounded text-neutral-500 transition-colors hover:text-[#22c55e]"
>
<svg width="16" height="16" viewBox="0 0 24 24" fill="none">
<path
d="M21 10C21 12.2091 16.9706 14 12 14M21 10C21 7.79086 16.9706 6 12 6C7.02944 6 3 7.79086 3 10M21 10V16C21 18.2091 16.9706 20 12 20M12 14C7.02944 14 3 12.2091 3 10M12 14V20M3 10V16C3 18.2091 7.02944 20 12 20M7 19.3264V13.3264M17 19.3264V13.3264M12 10L20 4"
stroke="currentColor"
strokeWidth="2"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
</button>
</Tooltip>
))()}
</div> </div>
<div className="flex items-center gap-1"> <div className="flex items-center gap-1">
<Tooltip label="Fit timeline to width"> <Tooltip label="Fit timeline to width">
@@ -48,6 +48,31 @@ function handleUndoRedoKey(event: KeyboardEvent, onUndo: () => void, onRedo: ()
return false; return false;
} }
// Beat edits live in an in-memory stack interleaved with file history by
// timestamp. Undo steps to the NEWER op (beatAt >= fileAt); redo replays the
// inverse, stepping to the OLDER op (beatAt <= fileAt). Returns true when it
// handled the keystroke (so the file-history path is skipped).
// fallow-ignore-next-line complexity
function tryApplyBeatHistory(
direction: "undo" | "redo",
fileState: {
undo: ReadonlyArray<{ createdAt: number }>;
redo: ReadonlyArray<{ createdAt: number }>;
},
showToast: (message: string, tone?: "error" | "info") => void,
): boolean {
const ps = usePlayerStore.getState();
const beatStack = direction === "undo" ? ps.beatUndo : ps.beatRedo;
const beatAt = beatStack[beatStack.length - 1]?.at ?? null;
if (beatAt === null) return false;
const fileStack = fileState[direction];
const fileAt = fileStack[fileStack.length - 1]?.createdAt ?? null;
if (fileAt !== null && (direction === "undo" ? beatAt < fileAt : beatAt > fileAt)) return false;
const label = direction === "undo" ? ps.undoBeatEdits() : ps.redoBeatEdits();
if (label) showToast(`${direction === "undo" ? "Undid" : "Redid"} ${label}`, "info");
return true;
}
// ── Types ── // ── Types ──
interface HistoryResult { interface HistoryResult {
@@ -63,6 +88,10 @@ interface HistoryFileCallbacks {
interface EditHistoryHandle { interface EditHistoryHandle {
undo: (cb: HistoryFileCallbacks) => Promise<HistoryResult>; undo: (cb: HistoryFileCallbacks) => Promise<HistoryResult>;
redo: (cb: HistoryFileCallbacks) => Promise<HistoryResult>; redo: (cb: HistoryFileCallbacks) => Promise<HistoryResult>;
state: {
undo: ReadonlyArray<{ createdAt: number }>;
redo: ReadonlyArray<{ createdAt: number }>;
};
} }
interface UseAppHotkeysParams { interface UseAppHotkeysParams {
@@ -294,6 +323,9 @@ export function useAppHotkeys({
const applyHistory = useCallback( const applyHistory = useCallback(
async (direction: "undo" | "redo") => { async (direction: "undo" | "redo") => {
// Beat edits interleave with file history by timestamp; handle them first.
if (tryApplyBeatHistory(direction, editHistory.state, showToast)) return;
await waitForPendingDomEditSaves(); await waitForPendingDomEditSaves();
const result = await editHistory[direction]({ const result = await editHistory[direction]({
readFile: readHistoryFile, readFile: readHistoryFile,
@@ -0,0 +1,152 @@
import { useEffect, useMemo, useRef } from "react";
import { usePlayerStore } from "../player/store/playerStore";
import { isMusicTrack } from "../utils/timelineInspector";
import { analyzeMusicFromUrl } from "@hyperframes/core/beats";
import { useFileManagerContext } from "../contexts/FileManagerContext";
import { mergeUserBeats } from "../utils/beatEditing";
import {
audioRelPathForSrc,
beatFilePathForSrc,
serializeBeats,
parseBeats,
} from "@hyperframes/core/beats";
// Module-level cache so the same URL isn't re-decoded/analyzed on re-mount.
// Capped so decoded PCM buffers don't accumulate unbounded across a session.
const analysisCache = new Map<string, ReturnType<typeof analyzeMusicFromUrl>>();
const MAX_ANALYSIS_CACHE = 4;
const PERSIST_DEBOUNCE_MS = 350;
function cacheAnalysis(url: string, promise: ReturnType<typeof analyzeMusicFromUrl>): void {
analysisCache.set(url, promise);
while (analysisCache.size > MAX_ANALYSIS_CACHE) {
const oldest = analysisCache.keys().next().value;
if (oldest === undefined) break;
analysisCache.delete(oldest);
}
}
type ProjectIo = { readOptionalProjectFile: (p: string) => Promise<string> };
/**
* Resolve the effective beat list for a track: a saved file with real beats
* wins; otherwise the detected beats are used (and `hasFile` is false so the
* caller seeds a new file). An empty saved file is ignored so detection retries.
*/
async function resolveBeats(
beatPath: string | null,
detected: { times: number[]; strengths: number[] },
io: ProjectIo,
): Promise<{ times: number[]; strengths: number[]; hasFile: boolean }> {
if (!beatPath) return { ...detected, hasFile: false };
try {
const content = await io.readOptionalProjectFile(beatPath);
const parsed = content ? parseBeats(content) : null;
if (parsed && parsed.times.length > 0) {
return { times: parsed.times, strengths: parsed.strengths, hasFile: true };
}
} catch {
/* fall back to detected beats */
}
return { ...detected, hasFile: false };
}
export function useMusicBeatAnalysis(): void {
const elements = usePlayerStore((s) => s.elements);
const setBeatAnalysis = usePlayerStore((s) => s.setBeatAnalysis);
const setBeatEdits = usePlayerStore((s) => s.setBeatEdits);
const setBeatPersist = usePlayerStore((s) => s.setBeatPersist);
const resetBeatHistory = usePlayerStore((s) => s.resetBeatHistory);
const { readOptionalProjectFile, writeProjectFile } = useFileManagerContext();
// File IO via ref so the effects only re-run when the track changes.
const ioRef = useRef({ readOptionalProjectFile, writeProjectFile });
ioRef.current = { readOptionalProjectFile, writeProjectFile };
const musicSrc = useMemo(() => {
const el = elements.find((e) => isMusicTrack(e));
return el?.src ?? null;
}, [elements]);
// ── Load: decode for strength data, then use the saved beat file if present,
// otherwise seed it from detection. Resets edits + history on track change. ──
useEffect(() => {
if (!musicSrc) {
setBeatAnalysis(null);
setBeatEdits(null);
resetBeatHistory();
return;
}
let cancelled = false;
let promise = analysisCache.get(musicSrc);
if (!promise) {
promise = analyzeMusicFromUrl(musicSrc);
cacheAnalysis(musicSrc, promise);
}
const beatPath = beatFilePathForSrc(musicSrc);
promise
.then(async (analysis) => {
const detected = { times: analysis.beatTimes, strengths: analysis.beatStrengths };
const { times, strengths, hasFile } = await resolveBeats(beatPath, detected, ioRef.current);
if (cancelled) return;
setBeatEdits(null);
resetBeatHistory();
setBeatAnalysis({ ...analysis, beatTimes: times, beatStrengths: strengths });
// Seed a missing file through the SAME debounced writer the edits use, so
// the initial write can't race a near-simultaneous edit's persist.
if (beatPath && !hasFile && times.length > 0) usePlayerStore.getState().beatPersist?.();
})
.catch(() => {
if (cancelled) return;
setBeatAnalysis(null);
analysisCache.delete(musicSrc);
});
return () => {
cancelled = true;
};
}, [musicSrc, setBeatAnalysis, setBeatEdits, resetBeatHistory]);
// ── Persist: register a debounced writer fired by every beat edit/undo/redo.
// Flushes any pending write on cleanup so the last edit is never lost. ──
useEffect(() => {
const beatPath = beatFilePathForSrc(musicSrc);
if (!musicSrc || !beatPath) {
setBeatPersist(null);
return;
}
const audio = audioRelPathForSrc(musicSrc) ?? "audio";
let timer: ReturnType<typeof setTimeout> | null = null;
let pending: string | null = null;
const flush = () => {
if (pending === null) return;
const content = pending;
pending = null;
void ioRef.current.writeProjectFile(beatPath, content).catch(() => {});
};
const persist = () => {
const s = usePlayerStore.getState();
const a = s.beatAnalysis;
if (!a) return;
const merged = mergeUserBeats(a.beatTimes, a.beatStrengths, s.beatEdits, musicSrc);
pending = serializeBeats(merged.times, merged.strengths, audio);
if (timer) clearTimeout(timer);
timer = setTimeout(() => {
timer = null;
flush();
}, PERSIST_DEBOUNCE_MS);
};
setBeatPersist(persist);
return () => {
if (timer) clearTimeout(timer);
flush(); // write the last pending edit before tearing down
setBeatPersist(null);
};
}, [musicSrc, setBeatPersist]);
}
@@ -137,13 +137,17 @@ export const AudioWaveform = memo(function AudioWaveform({
const hi = Math.max(lo + 1, Math.ceil(winEnd * peaks.length)); const hi = Math.max(lo + 1, Math.ceil(winEnd * peaks.length));
const span = hi - lo; const span = hi - lo;
// Fill the full (possibly zoomed) clip width with STEP-spaced bars, resampling
// the windowed peaks across them — upsampling (repeating peaks) when the clip
// is wider than the slice has samples, so the waveform stretches with zoom
// instead of stopping partway across.
const w = container.clientWidth || 400; const w = container.clientWidth || 400;
const barCount = Math.min(Math.floor(w / STEP), span); const barCount = Math.max(0, Math.floor(w / STEP));
let html = ""; let html = "";
for (let i = 0; i < barCount; i++) { for (let i = 0; i < barCount; i++) {
// Map bar index to peak index within the windowed range (resample) // Map bar index to peak index within the windowed range (resample)
const peakIdx = lo + Math.floor((i / barCount) * span); const peakIdx = lo + Math.min(span - 1, Math.floor((i / barCount) * span));
const amp = peaks[peakIdx] ?? 0; const amp = peaks[peakIdx] ?? 0;
const pct = Math.max(3, Math.round(amp * 100)); const pct = Math.max(3, Math.round(amp * 100));
const opacity = (0.45 + amp * 0.4).toFixed(2); const opacity = (0.45 + amp * 0.4).toFixed(2);
@@ -0,0 +1,166 @@
import { memo, useRef, useState } from "react";
import { moveBeatCompositionTime, deleteBeatAtCompositionTime } from "../../utils/beatEditActions";
import { usePlayerStore } from "../store/playerStore";
import { CLIP_Y } from "./timelineLayout";
const BEAT_BAND_H = 14; // dark band height at top of track
const BEAT_HIT_W = 12; // grab width per beat (px)
/** Hide both layers when beats are packed tighter than this (px) — too dense to read. */
function beatsTooDense(beatTimes: number[], pps: number): boolean {
if (beatTimes.length < 2) return true;
const avgInterval = (beatTimes[beatTimes.length - 1]! - beatTimes[0]!) / (beatTimes.length - 1);
return avgInterval * pps < 5;
}
/**
* Faint full-height beat lines painted into a track lane's background. Rendered
* behind the clips so they only show through the empty track area (the dots in
* BeatStrip mark beats on the clips themselves). Brightness scales with beat
* loudness. Drawn on every track lane for a global beat grid.
*/
export const BeatBackgroundLines = memo(function BeatBackgroundLines({
beatTimes,
beatStrengths,
pps,
highlightTime,
}: {
beatTimes: number[] | undefined;
beatStrengths: number[] | undefined;
pps: number;
/** Beat time a dragged clip will snap to — drawn as a bright neon line. */
highlightTime?: number | null;
}) {
if (!beatTimes || beatsTooDense(beatTimes, pps)) return null;
return (
<div className="absolute inset-0 pointer-events-none" style={{ zIndex: 0 }}>
{beatTimes.map((t, i) => {
const isHighlight = highlightTime != null && Math.abs(t - highlightTime) < 1e-3;
const strength = Math.pow(Math.min(1, beatStrengths?.[i] ?? 0.5), 2.2);
const opacity = isHighlight ? 1 : 0.06 + strength * 0.16;
return (
<div
key={`${t}-${i}`}
className="absolute top-0 bottom-0"
style={{
left: t * pps,
width: isHighlight ? 2 : 1,
background: `rgba(34,197,94,${opacity.toFixed(3)})`,
boxShadow: isHighlight ? "0 0 6px rgba(34,197,94,0.9)" : undefined,
zIndex: isHighlight ? 1 : undefined,
}}
/>
);
})}
</div>
);
});
/**
* Green beat dots on the music track's row. Drag a dot to move its beat,
* double-click to delete; both scrub the audio. Dot size/brightness scale with
* beat loudness (gamma-curved for contrast).
*/
export const BeatStrip = memo(function BeatStrip({
beatTimes,
beatStrengths,
pps,
}: {
beatTimes: number[] | undefined;
beatStrengths: number[] | undefined;
pps: number;
}) {
// Active drag: which beat and how far (px) it's been dragged.
const [drag, setDrag] = useState<{ index: number; dx: number } | null>(null);
const dragRef = useRef<{ index: number; startX: number; origTime: number } | null>(null);
if (!beatTimes || beatsTooDense(beatTimes, pps)) return null;
const cy = BEAT_BAND_H / 2;
return (
<div
className="absolute left-0 right-0 pointer-events-none"
style={{ top: CLIP_Y, height: BEAT_BAND_H, background: "rgba(0,0,0,0.28)", zIndex: 11 }}
>
{beatTimes.map((t, i) => {
// Louder beats → larger, brighter dot. Gamma curve widens the contrast.
const strength = Math.pow(Math.min(1, beatStrengths?.[i] ?? 0.5), 2.2);
const r = 1.5 + strength * 2.5;
const opacity = 0.25 + strength * 0.75;
const dxPx = drag?.index === i ? drag.dx : 0;
const x = t * pps + dxPx;
return (
<div
key={`${t}-${i}`}
className="absolute select-none"
title="Drag to move · double-click to delete"
draggable={false}
style={{
left: x - BEAT_HIT_W / 2,
top: 0,
width: BEAT_HIT_W,
height: BEAT_BAND_H,
cursor: "ew-resize",
pointerEvents: "auto",
touchAction: "none",
}}
onPointerDown={(e) => {
// preventDefault stops the browser starting a native text/drag
// selection (which otherwise "selects" the whole panel mid-drag).
e.preventDefault();
e.stopPropagation();
e.currentTarget.setPointerCapture(e.pointerId);
dragRef.current = { index: i, startX: e.clientX, origTime: t };
setDrag({ index: i, dx: 0 });
usePlayerStore.getState().setBeatDragging(true); // hide the playhead guideline
usePlayerStore.getState().requestSeek(Math.max(0, t)); // scrub audio at beat
}}
onPointerMove={(e) => {
const d = dragRef.current;
if (!d || d.index !== i) return;
e.preventDefault();
const dx = e.clientX - d.startX;
setDrag({ index: i, dx });
// Scrub the audio (and move the playhead) to follow the dragged beat.
usePlayerStore.getState().requestSeek(Math.max(0, d.origTime + dx / pps));
}}
onPointerUp={(e) => {
const d = dragRef.current;
dragRef.current = null;
setDrag(null);
usePlayerStore.getState().setBeatDragging(false);
if (e.currentTarget.hasPointerCapture?.(e.pointerId)) {
e.currentTarget.releasePointerCapture(e.pointerId);
}
if (!d || d.index !== i) return;
const dx = e.clientX - d.startX;
if (Math.abs(dx) > 2) {
const newTime = Math.max(0, d.origTime + dx / pps);
moveBeatCompositionTime(d.origTime, newTime);
usePlayerStore.getState().requestSeek(newTime); // park scrubber at new beat
}
}}
onDoubleClick={(e) => {
e.stopPropagation();
deleteBeatAtCompositionTime(t);
usePlayerStore.getState().requestSeek(Math.max(0, t)); // park scrubber at deleted beat
}}
>
<div
className="absolute"
style={{
left: BEAT_HIT_W / 2 - r,
top: cy - r,
width: r * 2,
height: r * 2,
borderRadius: "50%",
background: `rgba(34,197,94,${opacity.toFixed(3)})`,
pointerEvents: "none",
}}
/>
</div>
);
})}
</div>
);
});
@@ -1,4 +1,7 @@
import { useRef, useMemo, useCallback, useState, useEffect, memo, type ReactNode } from "react"; import { useRef, useMemo, useCallback, useState, useEffect, memo, type ReactNode } from "react";
import { useMusicBeatAnalysis } from "../../hooks/useMusicBeatAnalysis";
import { isMusicTrack } from "../../utils/timelineInspector";
import { remapBeatAnalysisToComposition } from "../../utils/beatEditActions";
import { usePlayerStore, type TimelineElement } from "../store/playerStore"; import { usePlayerStore, type TimelineElement } from "../store/playerStore";
import { useMountEffect } from "../../hooks/useMountEffect"; import { useMountEffect } from "../../hooks/useMountEffect";
import { EditPopover } from "./EditModal"; import { EditPopover } from "./EditModal";
@@ -78,7 +81,17 @@ export const Timeline = memo(function Timeline({
onMoveKeyframe, onMoveKeyframe,
} = useTimelineEditContext(); } = useTimelineEditContext();
const theme = useMemo(() => ({ ...defaultTimelineTheme, ...themeOverrides }), [themeOverrides]); const theme = useMemo(() => ({ ...defaultTimelineTheme, ...themeOverrides }), [themeOverrides]);
useMusicBeatAnalysis();
const elements = usePlayerStore((s) => s.elements); const elements = usePlayerStore((s) => s.elements);
const beatAnalysis = usePlayerStore((s) => s.beatAnalysis);
const musicElement = usePlayerStore((s) => s.elements.find(isMusicTrack) ?? null);
// Merge user edits + remap beats from audio-file → composition coordinates.
const beatEdits = usePlayerStore((s) => s.beatEdits);
const adjustedBeatAnalysis = useMemo(
() => remapBeatAnalysisToComposition(beatAnalysis, musicElement, beatEdits),
[beatAnalysis, musicElement, beatEdits],
);
const duration = usePlayerStore((s) => s.duration); const duration = usePlayerStore((s) => s.duration);
const timelineReady = usePlayerStore((s) => s.timelineReady); const timelineReady = usePlayerStore((s) => s.timelineReady);
const selectedElementId = usePlayerStore((s) => s.selectedElementId); const selectedElementId = usePlayerStore((s) => s.selectedElementId);
@@ -439,6 +452,7 @@ export const Timeline = memo(function Timeline({
keyframeCache={keyframeCache} keyframeCache={keyframeCache}
selectedKeyframes={selectedKeyframes} selectedKeyframes={selectedKeyframes}
currentTime={currentTime} currentTime={currentTime}
beatAnalysis={adjustedBeatAnalysis}
onClickKeyframe={(el, pct) => { onClickKeyframe={(el, pct) => {
usePlayerStore.getState().clearSelectedKeyframes(); usePlayerStore.getState().clearSelectedKeyframes();
const elKey = el.key ?? el.id; const elKey = el.key ?? el.id;
@@ -1,7 +1,9 @@
import { memo, type ReactNode } from "react"; import { memo, type ReactNode } from "react";
import { BeatStrip, BeatBackgroundLines } from "./BeatStrip";
import { TimelineClip } from "./TimelineClip"; import { TimelineClip } from "./TimelineClip";
import { TimelineClipDiamonds } from "./TimelineClipDiamonds"; import { TimelineClipDiamonds } from "./TimelineClipDiamonds";
import { TimelineRuler } from "./TimelineRuler"; import { TimelineRuler } from "./TimelineRuler";
import type { MusicBeatAnalysis } from "@hyperframes/core/beats";
import { PlayheadIndicator } from "./PlayheadIndicator"; import { PlayheadIndicator } from "./PlayheadIndicator";
import { import {
getTimelineEditCapabilities, getTimelineEditCapabilities,
@@ -20,6 +22,7 @@ import type { TrackVisualStyle } from "./timelineIcons";
import { STUDIO_KEYFRAMES_ENABLED } from "../../components/editor/manualEditingAvailability"; import { STUDIO_KEYFRAMES_ENABLED } from "../../components/editor/manualEditingAvailability";
import { SPLIT_BOUNDARY_EPSILON_S } from "../../utils/timelineElementSplit"; import { SPLIT_BOUNDARY_EPSILON_S } from "../../utils/timelineElementSplit";
import { useTimelineEditContext } from "../../contexts/TimelineEditContext"; import { useTimelineEditContext } from "../../contexts/TimelineEditContext";
import { isMusicTrack } from "../../utils/timelineInspector";
function ClipLabel({ element, color }: { element: TimelineElement; color: string }) { function ClipLabel({ element, color }: { element: TimelineElement; color: string }) {
const lint = usePlayerStore((s) => s.lintFindingsByElement.get(element.key ?? element.id)); const lint = usePlayerStore((s) => s.lintFindingsByElement.get(element.key ?? element.id));
@@ -91,6 +94,7 @@ interface TimelineCanvasProps {
onDragKeyframe?: (element: TimelineElement, oldPct: number, newPct: number) => void; onDragKeyframe?: (element: TimelineElement, oldPct: number, newPct: number) => void;
onContextMenuKeyframe?: (e: React.MouseEvent, elementId: string, percentage: number) => void; onContextMenuKeyframe?: (e: React.MouseEvent, elementId: string, percentage: number) => void;
onContextMenuClip?: (e: React.MouseEvent, element: TimelineElement) => void; onContextMenuClip?: (e: React.MouseEvent, element: TimelineElement) => void;
beatAnalysis?: MusicBeatAnalysis | null;
} }
export const TimelineCanvas = memo(function TimelineCanvas({ export const TimelineCanvas = memo(function TimelineCanvas({
@@ -138,9 +142,11 @@ export const TimelineCanvas = memo(function TimelineCanvas({
onDragKeyframe, onDragKeyframe,
onContextMenuKeyframe, onContextMenuKeyframe,
onContextMenuClip, onContextMenuClip,
beatAnalysis,
}: TimelineCanvasProps) { }: TimelineCanvasProps) {
const { onResizeElement, onMoveElement, onRazorSplit, onRazorSplitAll } = const { onResizeElement, onMoveElement, onRazorSplit, onRazorSplitAll } =
useTimelineEditContext(); useTimelineEditContext();
const beatDragging = usePlayerStore((s) => s.beatDragging);
const draggedElement = draggedClip?.element ?? null; const draggedElement = draggedClip?.element ?? null;
const activeDraggedElement = const activeDraggedElement =
draggedClip?.started === true && draggedElement draggedClip?.started === true && draggedElement
@@ -197,6 +203,7 @@ export const TimelineCanvas = memo(function TimelineCanvas({
shiftHeld={shiftHeld} shiftHeld={shiftHeld}
rangeSelection={rangeSelection} rangeSelection={rangeSelection}
theme={theme} theme={theme}
beatAnalysis={beatAnalysis}
/> />
{displayTrackOrder.map((trackNum) => { {displayTrackOrder.map((trackNum) => {
@@ -237,6 +244,25 @@ export const TimelineCanvas = memo(function TimelineCanvas({
</div> </div>
</div> </div>
<div style={{ width: trackContentWidth }} className="relative"> <div style={{ width: trackContentWidth }} className="relative">
{/* Faint beat lines in every track's background (behind the clips);
the active move-snap target is highlighted. */}
<BeatBackgroundLines
beatTimes={beatAnalysis?.beatTimes}
beatStrengths={beatAnalysis?.beatStrengths}
pps={pps}
highlightTime={draggedClip?.started ? draggedClip.snapBeatTime : null}
/>
{/* Beat dots on the active track (the one holding the selection),
falling back to the music track when nothing is selected. */}
{(selectedElementId
? els.some((e) => (e.key ?? e.id) === selectedElementId)
: els.some(isMusicTrack)) && (
<BeatStrip
beatTimes={beatAnalysis?.beatTimes}
beatStrengths={beatAnalysis?.beatStrengths}
pps={pps}
/>
)}
{isPendingTrack && ( {isPendingTrack && (
<div <div
className="absolute inset-0 flex items-center" className="absolute inset-0 flex items-center"
@@ -351,6 +377,7 @@ export const TimelineCanvas = memo(function TimelineCanvas({
pointerOffsetY: e.clientY - rect.top, pointerOffsetY: e.clientY - rect.top,
previewStart: el.start, previewStart: el.start,
previewTrack: el.track, previewTrack: el.track,
snapBeatTime: null,
started: false, started: false,
}); });
syncClipDragAutoScroll(e.clientX, e.clientY); syncClipDragAutoScroll(e.clientX, e.clientY);
@@ -472,11 +499,16 @@ export const TimelineCanvas = memo(function TimelineCanvas({
/> />
)} )}
{/* Playhead */} {/* Playhead — hidden while dragging a beat so its guideline doesn't
track the scrub and clutter the beat being moved. */}
<div <div
ref={playheadRef} ref={playheadRef}
className="absolute top-0 bottom-0 pointer-events-none" className="absolute top-0 bottom-0 pointer-events-none"
style={{ left: `${GUTTER}px`, zIndex: 100 }} style={{
left: `${GUTTER}px`,
zIndex: 100,
display: beatDragging ? "none" : undefined,
}}
> >
<PlayheadIndicator /> <PlayheadIndicator />
</div> </div>
@@ -2,6 +2,7 @@ import { memo } from "react";
import type { TimelineTheme } from "./timelineTheme"; import type { TimelineTheme } from "./timelineTheme";
import type { TimelineRangeSelection } from "./timelineEditing"; import type { TimelineRangeSelection } from "./timelineEditing";
import { GUTTER, RULER_H, formatTimelineTickLabel } from "./timelineLayout"; import { GUTTER, RULER_H, formatTimelineTickLabel } from "./timelineLayout";
import type { MusicBeatAnalysis } from "@hyperframes/core/beats";
interface TimelineRulerProps { interface TimelineRulerProps {
major: number[]; major: number[];
@@ -14,6 +15,7 @@ interface TimelineRulerProps {
shiftHeld: boolean; shiftHeld: boolean;
rangeSelection: TimelineRangeSelection | null; rangeSelection: TimelineRangeSelection | null;
theme: TimelineTheme; theme: TimelineTheme;
beatAnalysis?: MusicBeatAnalysis | null;
} }
export const TimelineRuler = memo(function TimelineRuler({ export const TimelineRuler = memo(function TimelineRuler({
@@ -27,13 +29,25 @@ export const TimelineRuler = memo(function TimelineRuler({
shiftHeld, shiftHeld,
rangeSelection, rangeSelection,
theme, theme,
beatAnalysis,
}: TimelineRulerProps) { }: TimelineRulerProps) {
const beatTimes = beatAnalysis?.beatTimes ?? [];
const beatStrengths = beatAnalysis?.beatStrengths ?? [];
// Only draw beat lines when they'd be at least 5px apart
const avgBeatInterval =
beatTimes.length > 1
? (beatTimes[beatTimes.length - 1]! - beatTimes[0]!) / (beatTimes.length - 1)
: null;
const showBeats = avgBeatInterval !== null && avgBeatInterval * pps >= 5;
return ( return (
<> <>
{/* Grid lines */} {/* Grid lines (major ticks + beat lines) — behind the tracks (background).
Opaque track rows hide them; only the beat dots show on tracks. */}
<svg <svg
className="absolute pointer-events-none" className="absolute pointer-events-none"
style={{ left: GUTTER, width: trackContentWidth }} style={{ left: GUTTER, width: trackContentWidth, zIndex: 0 }}
height={totalH} height={totalH}
> >
{major.map((t) => { {major.map((t) => {
@@ -50,6 +64,24 @@ export const TimelineRuler = memo(function TimelineRuler({
/> />
); );
})} })}
{showBeats &&
beatTimes.map((t, i) => {
const x = t * pps;
// Louder beats → brighter line. Gamma curve widens the contrast.
const strength = Math.pow(Math.min(1, beatStrengths[i] ?? 0.5), 2.2);
const opacity = 0.08 + strength * 0.62;
return (
<line
key={`b-${t}-${i}`}
x1={x}
y1={0}
x2={x}
y2={totalH}
stroke={`rgba(34, 197, 94, ${opacity.toFixed(3)})`}
strokeWidth="1"
/>
);
})}
</svg> </svg>
{/* Ruler */} {/* Ruler */}
@@ -64,11 +96,13 @@ export const TimelineRuler = memo(function TimelineRuler({
</span> </span>
</div> </div>
)} )}
{minor.map((t) => ( {minor.map((t) => (
<div key={`m-${t}`} className="absolute bottom-0" style={{ left: t * pps }}> <div key={`m-${t}`} className="absolute bottom-0" style={{ left: t * pps }}>
<div className="w-px h-[3px]" style={{ background: theme.tickMinor }} /> <div className="w-px h-[3px]" style={{ background: theme.tickMinor }} />
</div> </div>
))} ))}
{major.map((t) => ( {major.map((t) => (
<div <div
key={`M-${t}`} key={`M-${t}`}
@@ -1,4 +1,4 @@
import { useRef, useState, useCallback } from "react"; import { useRef, useState, useCallback, useMemo } from "react";
import { useMountEffect } from "../../hooks/useMountEffect"; import { useMountEffect } from "../../hooks/useMountEffect";
import { import {
resolveTimelineMove, resolveTimelineMove,
@@ -9,6 +9,64 @@ import {
import { usePlayerStore } from "../store/playerStore"; import { usePlayerStore } from "../store/playerStore";
import type { TimelineElement } from "../store/playerStore"; import type { TimelineElement } from "../store/playerStore";
import { TRACK_H } from "./timelineLayout"; import { TRACK_H } from "./timelineLayout";
import { isMusicTrack } from "../../utils/timelineInspector";
import { mergeUserBeats } from "../../utils/beatEditing";
const BEAT_SNAP_PX = 8;
const EMPTY_BEAT_TIMES: number[] = [];
function snapToNearestBeat(time: number, beatTimes: number[], thresholdSecs: number): number {
let best = time;
let bestDist = thresholdSecs;
for (const bt of beatTimes) {
const d = Math.abs(bt - time);
if (d < bestDist) {
bestDist = d;
best = bt;
}
}
return best;
}
/**
* Snap a moved clip so whichever edge (start or end) is nearest a beat lands on
* it, keeping the duration fixed. Returns the (clamped) start plus the beat time
* it snapped to (for the grid-line highlight), or `beat: null` when no edge is
* within threshold.
*/
function snapMoveStartToBeat(
start: number,
duration: number,
beatTimes: number[],
pixelsPerSecond: number,
timelineDuration: number,
): { start: number; beat: number | null } {
if (beatTimes.length === 0) return { start, beat: null };
const snapSecs = BEAT_SNAP_PX / Math.max(pixelsPerSecond, 1);
const snappedStart = snapToNearestBeat(start, beatTimes, snapSecs);
const snappedEnd = snapToNearestBeat(start + duration, beatTimes, snapSecs);
const startMoved = snappedStart !== start;
const endMoved = snappedEnd !== start + duration;
let candidate = start;
let beat: number | null = null;
if (
startMoved &&
(!endMoved || Math.abs(snappedStart - start) <= Math.abs(snappedEnd - (start + duration)))
) {
candidate = snappedStart;
beat = snappedStart;
} else if (endMoved) {
candidate = snappedEnd - duration;
beat = snappedEnd;
}
const maxStart = Math.max(0, timelineDuration - duration);
const clamped = Math.max(0, Math.min(maxStart, Math.round(candidate * 1000) / 1000));
// If clamping pulled the clip off the snap target, drop the highlight.
if (beat != null && Math.abs(clamped - candidate) > 1e-6) beat = null;
return { start: clamped, beat };
}
/* ── Shared state types ─────────────────────────────────────────── */ /* ── Shared state types ─────────────────────────────────────────── */
export interface DraggedClipState { export interface DraggedClipState {
@@ -23,6 +81,8 @@ export interface DraggedClipState {
pointerOffsetY: number; pointerOffsetY: number;
previewStart: number; previewStart: number;
previewTrack: number; previewTrack: number;
/** Beat time the clip will snap to on drop, for the grid-line highlight. */
snapBeatTime: number | null;
started: boolean; started: boolean;
} }
@@ -76,6 +136,36 @@ export function useTimelineClipDrag({
setRangeSelectionRef, setRangeSelectionRef,
}: UseTimelineClipDragInput) { }: UseTimelineClipDragInput) {
const updateElement = usePlayerStore((s) => s.updateElement); const updateElement = usePlayerStore((s) => s.updateElement);
const rawBeatTimes = usePlayerStore((s) => s.beatAnalysis?.beatTimes ?? EMPTY_BEAT_TIMES);
const rawBeatStrengths = usePlayerStore((s) => s.beatAnalysis?.beatStrengths ?? EMPTY_BEAT_TIMES);
const beatEdits = usePlayerStore((s) => s.beatEdits);
const musicStart = usePlayerStore((s) => s.elements.find(isMusicTrack)?.start ?? 0);
const musicPlaybackStart = usePlayerStore(
(s) => s.elements.find(isMusicTrack)?.playbackStart ?? 0,
);
const musicDuration = usePlayerStore((s) => s.elements.find(isMusicTrack)?.duration ?? 0);
const musicSrc = usePlayerStore((s) => s.elements.find(isMusicTrack)?.src ?? null);
const adjustedBeatTimes = useMemo(() => {
if (rawBeatTimes === EMPTY_BEAT_TIMES || musicDuration === 0) return EMPTY_BEAT_TIMES;
const merged = mergeUserBeats(rawBeatTimes, rawBeatStrengths, beatEdits, musicSrc);
const clipEnd = musicPlaybackStart + musicDuration;
const offset = musicStart - musicPlaybackStart;
return merged.times
.filter((t) => t >= musicPlaybackStart && t <= clipEnd)
.map((t) => Math.round((t + offset) * 1000) / 1000);
}, [
rawBeatTimes,
rawBeatStrengths,
beatEdits,
musicSrc,
musicStart,
musicPlaybackStart,
musicDuration,
]);
const beatTimesRef = useRef<number[]>([]);
beatTimesRef.current = adjustedBeatTimes;
const [draggedClip, setDraggedClip] = useState<DraggedClipState | null>(null); const [draggedClip, setDraggedClip] = useState<DraggedClipState | null>(null);
const draggedClipRef = useRef<DraggedClipState | null>(null); const draggedClipRef = useRef<DraggedClipState | null>(null);
@@ -118,13 +208,24 @@ export function useTimelineClipDrag({
clientX, clientX,
clientY, clientY,
); );
// The music track defines the beats, so it must not snap to itself.
const snap = isMusicTrack(drag.element)
? { start: nextMove.start, beat: null }
: snapMoveStartToBeat(
nextMove.start,
drag.element.duration,
beatTimesRef.current,
ppsRef.current,
durationRef.current,
);
return { return {
...drag, ...drag,
started: true, started: true,
pointerClientX: clientX, pointerClientX: clientX,
pointerClientY: clientY, pointerClientY: clientY,
previewStart: nextMove.start, previewStart: snap.start,
previewTrack: nextMove.track, previewTrack: nextMove.track,
snapBeatTime: snap.beat,
}; };
}, },
[scrollRef, ppsRef, durationRef, trackOrderRef], [scrollRef, ppsRef, durationRef, trackOrderRef],
@@ -220,14 +321,16 @@ export function useTimelineClipDrag({
: Number.POSITIVE_INFINITY; : Number.POSITIVE_INFINITY;
const normalizedTag = resize.element.tag.toLowerCase(); const normalizedTag = resize.element.tag.toLowerCase();
const canSeedPlaybackStart = normalizedTag === "audio" || normalizedTag === "video"; const canSeedPlaybackStart = normalizedTag === "audio" || normalizedTag === "video";
const nextResize = resolveTimelineResize( const playbackRate = Math.max(resize.element.playbackRate ?? 1, 0.1);
const maxEnd = Math.min(durationRef.current, resize.element.start + sourceRemaining);
let nextResize = resolveTimelineResize(
{ {
start: resize.element.start, start: resize.element.start,
duration: resize.element.duration, duration: resize.element.duration,
originClientX: resize.originClientX, originClientX: resize.originClientX,
pixelsPerSecond: ppsRef.current, pixelsPerSecond: ppsRef.current,
minStart: 0, minStart: 0,
maxEnd: Math.min(durationRef.current, resize.element.start + sourceRemaining), maxEnd,
playbackStart: playbackStart:
resize.edge === "start" && canSeedPlaybackStart resize.edge === "start" && canSeedPlaybackStart
? (resize.element.playbackStart ?? 0) ? (resize.element.playbackStart ?? 0)
@@ -238,6 +341,54 @@ export function useTimelineClipDrag({
e.clientX, e.clientX,
); );
// Snap edge to beat grid when beat analysis is available. The snap must
// stay inside the same limits resolveTimelineResize enforces, or it would
// push the edge past the available source media / composition end.
// The music track defines the beats, so it must not snap to itself.
const beatTimes = beatTimesRef.current;
if (beatTimes.length > 0 && !isMusicTrack(resize.element)) {
const snapSecs = BEAT_SNAP_PX / Math.max(ppsRef.current, 1);
if (resize.edge === "end") {
const edgeTime = nextResize.start + nextResize.duration;
const snapped = snapToNearestBeat(edgeTime, beatTimes, snapSecs);
// Stay within [start+minDuration, maxEnd] so the snap can't create a
// degenerate clip or run past the source/composition limit.
const snappedDuration = Math.round((snapped - nextResize.start) * 1000) / 1000;
if (snapped !== edgeTime && snapped <= maxEnd + 1e-6 && snappedDuration >= 0.05) {
nextResize = { ...nextResize, duration: snappedDuration };
}
} else {
const snapped = snapToNearestBeat(nextResize.start, beatTimes, snapSecs);
const delta = nextResize.start - snapped; // >0 when snapping left
// Leftward snap reveals more source; cap so playbackStart can't go < 0.
const maxLeftDelta =
nextResize.playbackStart != null
? nextResize.playbackStart / playbackRate
: Number.POSITIVE_INFINITY;
// Also require the resulting duration to stay >= minDuration so a
// rightward snap (delta < 0) can't collapse the clip to zero/negative.
const snappedDuration = Math.round((nextResize.duration + delta) * 1000) / 1000;
if (
snapped !== nextResize.start &&
snapped >= 0 &&
delta <= maxLeftDelta + 1e-6 &&
snappedDuration >= 0.05
) {
nextResize = {
...nextResize,
start: snapped,
duration: snappedDuration,
playbackStart:
nextResize.playbackStart != null
? Math.round(
Math.max(0, nextResize.playbackStart - delta * playbackRate) * 1000,
) / 1000
: undefined,
};
}
}
}
setResizingClip((prev) => setResizingClip((prev) =>
prev prev
? { ? {
@@ -1,4 +1,4 @@
import { useRef, useCallback, useEffect } from "react"; import { useRef, useCallback, useEffect, useLayoutEffect } from "react";
import { liveTime, usePlayerStore, type ZoomMode } from "../store/playerStore"; import { liveTime, usePlayerStore, type ZoomMode } from "../store/playerStore";
import { useMountEffect } from "../../hooks/useMountEffect"; import { useMountEffect } from "../../hooks/useMountEffect";
import { getPinchTimelineZoomPercent } from "./timelineZoom"; import { getPinchTimelineZoomPercent } from "./timelineZoom";
@@ -54,6 +54,33 @@ export function useTimelinePlayhead({
}: UseTimelinePlayheadInput) { }: UseTimelinePlayheadInput) {
const dragScrollRaf = useRef(0); const dragScrollRaf = useRef(0);
const previousZoomModeRef = useRef<ZoomMode | null>(zoomMode); const previousZoomModeRef = useRef<ZoomMode | null>(zoomMode);
// Center-anchored magnify: keep the time at the viewport center fixed when
// the zoom level (pps) changes via the toolbar / slider. The pinch handler
// anchors at the cursor instead, so it opts out via `skipCenterAnchorRef`.
const previousAnchorPpsRef = useRef(pps);
const skipCenterAnchorRef = useRef(false);
useLayoutEffect(() => {
const scroll = scrollRef.current;
const prevPps = previousAnchorPpsRef.current;
previousAnchorPpsRef.current = pps;
// Always consume the skip flag, even when pps didn't change — otherwise a
// pinch that produced no pps change (already at the zoom clamp) would strand
// it true and the next toolbar zoom would wrongly skip center-anchoring.
const skip = skipCenterAnchorRef.current;
skipCenterAnchorRef.current = false;
if (!scroll || pps === prevPps || skip) return;
const nextScrollLeft = getTimelineScrollLeftForZoomAnchor({
pointerX: scroll.clientWidth / 2,
currentScrollLeft: scroll.scrollLeft,
gutter: GUTTER,
currentPixelsPerSecond: prevPps,
nextPixelsPerSecond: pps,
duration: durationRef.current,
});
const maxScrollLeft = Math.max(0, scroll.scrollWidth - scroll.clientWidth);
scroll.scrollLeft = Math.max(0, Math.min(maxScrollLeft, nextScrollLeft));
}, [pps, scrollRef, durationRef]);
const syncPlayheadPosition = useCallback( const syncPlayheadPosition = useCallback(
(time: number) => { (time: number) => {
@@ -169,6 +196,8 @@ export function useTimelinePlayhead({
nextPixelsPerSecond: nextPps, nextPixelsPerSecond: nextPps,
duration: durationRef.current, duration: durationRef.current,
}); });
// Pinch anchors at the cursor (below), so skip the center-anchor effect.
skipCenterAnchorRef.current = true;
setZoomMode("manual"); setZoomMode("manual");
setManualZoomPercent(nextZoomPercent); setManualZoomPercent(nextZoomPercent);
requestAnimationFrame(() => { requestAnimationFrame(() => {
@@ -41,9 +41,30 @@ import {
setPreviewPlaybackRate, setPreviewPlaybackRate,
shouldMutePreviewAudio, shouldMutePreviewAudio,
} from "../lib/timelineIframeHelpers"; } from "../lib/timelineIframeHelpers";
import { probeMediaUrl, getCachedProbe } from "../lib/mediaProbe"; import { scrubMusicAtSeek, stopScrubPreviewAudio } from "../lib/playbackScrub";
import { applyCachedSourceDurations, probeMissingSourceDurations } from "../lib/mediaProbe";
import { shouldResumeForwardPlaybackAfterSeek, shouldStopAfterSeek } from "../lib/playbackSeek"; import { shouldResumeForwardPlaybackAfterSeek, shouldStopAfterSeek } from "../lib/playbackSeek";
/**
* Whether the derived elements differ from the current ones in any field that
* affects rendering (identity, timing, track, or source length) — used to skip
* redundant store writes.
*/
function timelineElementsChanged(prev: TimelineElement[], next: TimelineElement[]): boolean {
if (next.length !== prev.length) return true;
return next.some((el, i) => {
const p = prev[i];
return (
!p ||
el.id !== p.id ||
el.start !== p.start ||
el.duration !== p.duration ||
el.track !== p.track ||
el.sourceDuration !== p.sourceDuration
);
});
}
export function useTimelinePlayer() { export function useTimelinePlayer() {
const iframeRef = useRef<HTMLIFrameElement | null>(null); const iframeRef = useRef<HTMLIFrameElement | null>(null);
const rafRef = useRef<number>(0); const rafRef = useRef<number>(0);
@@ -65,27 +86,19 @@ export function useTimelinePlayer() {
(elements: TimelineElement[], nextDuration?: number) => { (elements: TimelineElement[], nextDuration?: number) => {
const state = usePlayerStore.getState(); const state = usePlayerStore.getState();
const resolvedDuration = nextDuration ?? state.duration; const resolvedDuration = nextDuration ?? state.duration;
const mergedElements = mergeTimelineElementsPreservingDowngrades( // applyCachedSourceDurations re-applies the cached probe duration: re-derived
state.elements, // elements (e.g. after a clip move) can arrive without sourceDuration, which
elements, // otherwise makes trimmed waveforms lose their window.
state.duration, const mergedElements = applyCachedSourceDurations(
resolvedDuration, mergeTimelineElementsPreservingDowngrades(
state.elements,
elements,
state.duration,
resolvedDuration,
),
); );
const elementsChanged = if (timelineElementsChanged(state.elements, mergedElements)) {
mergedElements.length !== state.elements.length ||
mergedElements.some((el, i) => {
const prev = state.elements[i];
return (
!prev ||
el.id !== prev.id ||
el.start !== prev.start ||
el.duration !== prev.duration ||
el.track !== prev.track
);
});
if (elementsChanged) {
setElements(mergedElements); setElements(mergedElements);
} }
if ( if (
@@ -99,31 +112,17 @@ export function useTimelinePlayer() {
setTimelineReady(true); setTimelineReady(true);
} }
// Asynchronously enrich media elements missing sourceDuration via mediabunny. // Asynchronously enrich media elements still missing sourceDuration
// The probe reads file headers only — no full decode — so this is cheap. // (header-only probe, cheap), applying each resolved value to the store.
const needsProbe = mergedElements.filter( void probeMissingSourceDurations(mergedElements, (key, durationSeconds) => {
(el) => usePlayerStore.setState((state) => {
el.src && const idx = state.elements.findIndex((e) => (e.key ?? e.id) === key);
el.sourceDuration == null && if (idx === -1 || state.elements[idx].sourceDuration != null) return {};
["video", "audio"].includes(el.tag.toLowerCase()) && const patched = state.elements.slice();
!getCachedProbe(el.src), patched[idx] = { ...state.elements[idx], sourceDuration: durationSeconds };
); return { elements: patched };
if (needsProbe.length > 0) { });
void Promise.allSettled( });
needsProbe.map(async (el) => {
const result = await probeMediaUrl(el.src!);
if (!result) return;
const key = el.key ?? el.id;
usePlayerStore.setState((state) => {
const idx = state.elements.findIndex((e) => (e.key ?? e.id) === key);
if (idx === -1 || state.elements[idx].sourceDuration != null) return {};
const patched = state.elements.slice();
patched[idx] = { ...state.elements[idx], sourceDuration: result.duration };
return { elements: patched };
});
}),
);
}
}, },
[setElements, setTimelineReady, setDuration], [setElements, setTimelineReady, setDuration],
); );
@@ -280,6 +279,7 @@ export function useTimelinePlayer() {
const play = useCallback(() => { const play = useCallback(() => {
stopRAFLoop(); stopRAFLoop();
stopReverseLoop(); stopReverseLoop();
stopScrubPreviewAudio();
const adapter = getAdapter(); const adapter = getAdapter();
if (!adapter) return; if (!adapter) return;
if (adapter.getTime() >= adapter.getDuration()) { if (adapter.getTime() >= adapter.getDuration()) {
@@ -392,6 +392,7 @@ export function useTimelinePlayer() {
adapter.seek(nextTime, options); adapter.seek(nextTime, options);
liveTime.notify(nextTime); // Direct DOM updates (playhead, timecode, progress) — no re-render liveTime.notify(nextTime); // Direct DOM updates (playhead, timecode, progress) — no re-render
setCurrentTime(nextTime); // sync store so Split/Delete have accurate time setCurrentTime(nextTime); // sync store so Split/Delete have accurate time
if (!shouldResumeAfterSeek && !keepPlaying) scrubMusicAtSeek(iframeRef.current, nextTime);
if (shouldResumeAfterSeek) { if (shouldResumeAfterSeek) {
stopRAFLoop(); stopRAFLoop();
applyPlaybackRate(usePlayerStore.getState().playbackRate); applyPlaybackRate(usePlayerStore.getState().playbackRate);
@@ -557,6 +558,7 @@ export function useTimelinePlayer() {
document.removeEventListener("visibilitychange", handleVisibilityChange); document.removeEventListener("visibilitychange", handleVisibilityChange);
stopRAFLoop(); stopRAFLoop();
stopReverseLoop(); stopReverseLoop();
stopScrubPreviewAudio();
releaseStaticSeekCache(staticSeekAdapterRef, staticSeekWarnedRef); releaseStaticSeekCache(staticSeekAdapterRef, staticSeekWarnedRef);
if (probeIntervalRef.current) clearInterval(probeIntervalRef.current); if (probeIntervalRef.current) clearInterval(probeIntervalRef.current);
}; };
+46 -3
View File
@@ -1,4 +1,4 @@
export interface MediaProbeResult { interface MediaProbeResult {
duration: number; duration: number;
width?: number; width?: number;
height?: number; height?: number;
@@ -61,11 +61,54 @@ async function probeOne(url: string): Promise<MediaProbeResult | null> {
} }
} }
export function getCachedProbe(url: string): MediaProbeResult | undefined { function getCachedProbe(url: string): MediaProbeResult | undefined {
return cache.get(normalizeUrl(url)); return cache.get(normalizeUrl(url));
} }
export async function probeMediaUrl(url: string): Promise<MediaProbeResult | null> { /**
* Re-apply the cached probe `sourceDuration` to media elements that arrive
* without it. Re-deriving the timeline (e.g. after a clip move) produces fresh
* objects whose duration the DOM scan may not have, and the async probe skips
* already-cached srcs — so without this, trimmed waveforms lose their window.
*/
export function applyCachedSourceDurations<
T extends { src?: string; tag: string; sourceDuration?: number },
>(elements: T[]): T[] {
return elements.map((el) => {
const tag = el.tag.toLowerCase();
if (!el.src || el.sourceDuration != null || (tag !== "audio" && tag !== "video")) return el;
const cached = getCachedProbe(el.src);
return cached?.duration && cached.duration > 0
? { ...el, sourceDuration: cached.duration }
: el;
});
}
/**
* Probe (header-only, cheap) any media elements still missing sourceDuration
* after the cache pass, applying each resolved duration via `apply(key, secs)`.
* Skips already-cached srcs.
*/
export async function probeMissingSourceDurations<
T extends { src?: string; tag: string; sourceDuration?: number; key?: string; id: string },
>(elements: T[], apply: (key: string, durationSeconds: number) => void): Promise<void> {
const needs = elements.filter(
(el) =>
el.src &&
el.sourceDuration == null &&
["video", "audio"].includes(el.tag.toLowerCase()) &&
!getCachedProbe(el.src),
);
if (needs.length === 0) return;
await Promise.allSettled(
needs.map(async (el) => {
const result = await probeMediaUrl(el.src!);
if (result) apply(el.key ?? el.id, result.duration);
}),
);
}
async function probeMediaUrl(url: string): Promise<MediaProbeResult | null> {
const key = normalizeUrl(url); const key = normalizeUrl(url);
const cached = cache.get(key); const cached = cache.get(key);
if (cached) return cached; if (cached) return cached;
@@ -0,0 +1,16 @@
import { usePlayerStore } from "../store/playerStore";
import { isMusicTrack } from "../../utils/timelineInspector";
import { scrubPreviewAudio, stopScrubPreviewAudio } from "./timelineIframeHelpers";
export { stopScrubPreviewAudio };
// Scrub the music track's audio at a seeked composition time (paused-seek only).
// Skipped when audio is muted or the time falls outside the music clip.
export function scrubMusicAtSeek(iframe: HTMLIFrameElement | null, nextTime: number): void {
const s = usePlayerStore.getState();
const music = s.elements.find(isMusicTrack);
if (!music || s.audioMuted) return;
const rel = nextTime - music.start;
const audioFileTime = rel >= 0 && rel <= music.duration ? (music.playbackStart ?? 0) + rel : null;
scrubPreviewAudio(iframe, audioFileTime, music.domId ?? music.id);
}
+10 -2
View File
@@ -115,6 +115,8 @@ export function createTimelineElementFromManifestClip(params: {
if (hostEl) { if (hostEl) {
applyMediaMetadataFromElement(entry, hostEl); applyMediaMetadataFromElement(entry, hostEl);
const timelineRole = hostEl.getAttribute("data-timeline-role");
if (timelineRole) entry.timelineRole = timelineRole;
} }
if (clip.assetUrl) entry.src = clip.assetUrl; if (clip.assetUrl) entry.src = clip.assetUrl;
if (clip.kind === "composition" && clip.compositionId) { if (clip.kind === "composition" && clip.compositionId) {
@@ -286,17 +288,23 @@ export function parseTimelineFromDOM(doc: Document, rootDuration: number): Timel
if (mediaEl.tagName === "IMG") { if (mediaEl.tagName === "IMG") {
entry.tag = "img"; entry.tag = "img";
} }
const src = mediaEl.getAttribute("src");
if (src) entry.src = src;
const vol = el.getAttribute("data-volume") ?? mediaEl.getAttribute("data-volume"); const vol = el.getAttribute("data-volume") ?? mediaEl.getAttribute("data-volume");
if (vol) entry.volume = parseFloat(vol); if (vol) entry.volume = parseFloat(vol);
applyMediaMetadataFromElement(entry, el); applyMediaMetadataFromElement(entry, el);
// Override AFTER the helper (which sets the raw relative attribute) so the
// resolved absolute URL wins — the Studio can then fetch the asset
// regardless of whether the attribute value was relative or absolute.
const resolvedSrc = (mediaEl as HTMLMediaElement | HTMLImageElement).src || undefined;
if (resolvedSrc) entry.src = resolvedSrc;
} }
if (el.hasAttribute("data-timeline-locked")) { if (el.hasAttribute("data-timeline-locked")) {
entry.timelineLocked = true; entry.timelineLocked = true;
} }
const timelineRole = el.getAttribute("data-timeline-role");
if (timelineRole) entry.timelineRole = timelineRole;
// Sub-compositions // Sub-compositions
const compSrc = const compSrc =
el.getAttribute("data-composition-src") || el.getAttribute("data-composition-file"); el.getAttribute("data-composition-src") || el.getAttribute("data-composition-file");
@@ -170,6 +170,95 @@ export function resolveIframe(el: Element | null): HTMLIFrameElement | null {
return el.shadowRoot?.querySelector("iframe") ?? el.querySelector("iframe") ?? null; return el.shadowRoot?.querySelector("iframe") ?? el.querySelector("iframe") ?? null;
} }
// ---------------------------------------------------------------------------
// Audio scrubbing
// ---------------------------------------------------------------------------
// Plays a brief slice of the music track while the user drags the playhead,
// like an NLE scrub. Repeated calls keep playback alive; it auto-pauses shortly
// after scrubbing stops and restores the element's prior muted state.
const SCRUB_VOLUME = 0.25;
let scrubAudioEl: HTMLAudioElement | null = null;
let scrubStopTimer: ReturnType<typeof setTimeout> | null = null;
let scrubPrevMuted: boolean | null = null;
let scrubPrevVolume: number | null = null;
// Resolve the SAME element the store identified as music: prefer its id, then
// the role attribute, and only fall back to the first <audio> (which could be a
// voiceover, so the id hint matters).
function resolveScrubAudioEl(doc: Document, musicId?: string | null): HTMLAudioElement | null {
if (musicId) {
const byId = doc.getElementById(musicId);
if (byId instanceof HTMLAudioElement) return byId;
}
return (
doc.querySelector<HTMLAudioElement>("audio[data-timeline-role='music']") ??
doc.querySelector<HTMLAudioElement>("audio")
);
}
function applyScrub(el: HTMLAudioElement, audioFileTime: number): void {
if (scrubAudioEl && scrubAudioEl !== el) stopScrubPreviewAudio();
if (scrubPrevMuted === null) scrubPrevMuted = el.muted;
if (scrubPrevVolume === null) scrubPrevVolume = el.volume;
scrubAudioEl = el;
try {
el.muted = false;
el.volume = SCRUB_VOLUME;
if (Math.abs(el.currentTime - audioFileTime) > 0.04) el.currentTime = audioFileTime;
if (el.paused) void el.play().catch(() => {});
} catch {
/* element not ready */
}
if (scrubStopTimer) clearTimeout(scrubStopTimer);
scrubStopTimer = setTimeout(stopScrubPreviewAudio, 140);
}
/**
* Scrub the preview music audio to `audioFileTime` (seconds into the source
* file). Pass `null` to stop. Safe to call rapidly during a playhead drag.
*/
export function scrubPreviewAudio(
iframe: HTMLIFrameElement | null,
audioFileTime: number | null,
musicId?: string | null,
): void {
if (!iframe) return;
if (audioFileTime === null) {
stopScrubPreviewAudio();
return;
}
let doc: Document | null = null;
try {
doc = iframe.contentDocument;
} catch {
return;
}
if (!doc) return;
const el = resolveScrubAudioEl(doc, musicId);
if (el) applyScrub(el, audioFileTime);
}
export function stopScrubPreviewAudio(): void {
if (scrubStopTimer) {
clearTimeout(scrubStopTimer);
scrubStopTimer = null;
}
const el = scrubAudioEl;
scrubAudioEl = null;
if (!el) return;
try {
el.pause();
if (scrubPrevMuted !== null) el.muted = scrubPrevMuted;
if (scrubPrevVolume !== null) el.volume = scrubPrevVolume;
} catch {
/* ignore */
}
scrubPrevMuted = null;
scrubPrevVolume = null;
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Enrich missing compositions from DOM // Enrich missing compositions from DOM
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -1,4 +1,6 @@
import { create } from "zustand"; import { create } from "zustand";
import type { MusicBeatAnalysis } from "@hyperframes/core/beats";
import type { BeatEditState } from "../../utils/beatEditing";
import { readStudioUiPreferences, writeStudioUiPreferences } from "../../utils/studioUiPreferences"; import { readStudioUiPreferences, writeStudioUiPreferences } from "../../utils/studioUiPreferences";
/** Minimal keyframe cache types — mirrors GsapKeyframesData without pulling in Node-only gsap-parser. */ /** Minimal keyframe cache types — mirrors GsapKeyframesData without pulling in Node-only gsap-parser. */
@@ -46,6 +48,8 @@ export interface TimelineElement {
timingSource?: "authored" | "implicit"; timingSource?: "authored" | "implicit";
/** Set by data-timeline-locked on the host element — disables move and trim in Studio. */ /** Set by data-timeline-locked on the host element — disables move and trim in Studio. */
timelineLocked?: boolean; timelineLocked?: boolean;
/** Value of data-timeline-role attribute — used to identify music vs. voiceover. */
timelineRole?: string;
} }
export type ZoomMode = "fit" | "manual"; export type ZoomMode = "fit" | "manual";
@@ -56,6 +60,8 @@ interface PlayerState {
currentTime: number; currentTime: number;
duration: number; duration: number;
timelineReady: boolean; timelineReady: boolean;
/** True while a beat dot is being dragged — hides the playhead guideline. */
beatDragging: boolean;
elements: TimelineElement[]; elements: TimelineElement[];
selectedElementId: string | null; selectedElementId: string | null;
playbackRate: number; playbackRate: number;
@@ -99,6 +105,7 @@ interface PlayerState {
setAudioMuted: (muted: boolean) => void; setAudioMuted: (muted: boolean) => void;
setLoopEnabled: (enabled: boolean) => void; setLoopEnabled: (enabled: boolean) => void;
setTimelineReady: (ready: boolean) => void; setTimelineReady: (ready: boolean) => void;
setBeatDragging: (dragging: boolean) => void;
setElements: (elements: TimelineElement[]) => void; setElements: (elements: TimelineElement[]) => void;
setSelectedElementId: (id: string | null) => void; setSelectedElementId: (id: string | null) => void;
updateElement: ( updateElement: (
@@ -121,6 +128,32 @@ interface PlayerState {
lintFindingsByElement: Map<string, { count: number; messages: string[] }>; lintFindingsByElement: Map<string, { count: number; messages: string[] }>;
setLintFindingsByElement: (map: Map<string, { count: number; messages: string[] }>) => void; setLintFindingsByElement: (map: Map<string, { count: number; messages: string[] }>) => void;
beatAnalysis: MusicBeatAnalysis | null;
setBeatAnalysis: (analysis: MusicBeatAnalysis | null) => void;
/** User edits (add/move/delete) layered over the detected beat grid. */
beatEdits: BeatEditState | null;
setBeatEdits: (edits: BeatEditState | null) => void;
/** Undo/redo stacks for beat edits (in-memory, session-only). */
beatUndo: BeatHistoryEntry[];
beatRedo: BeatHistoryEntry[];
/** Apply a beat edit and record it for undo. */
commitBeatEdits: (next: BeatEditState | null, label: string) => void;
/** Undo/redo the most recent beat edit; returns its label or null if none. */
undoBeatEdits: () => string | null;
redoBeatEdits: () => string | null;
/** Clear beat edit history (e.g. when the music track changes). */
resetBeatHistory: () => void;
/** Callback that persists current beats to disk; registered by the analysis hook. */
beatPersist: (() => void) | null;
setBeatPersist: (fn: (() => void) | null) => void;
}
interface BeatHistoryEntry {
restore: BeatEditState | null; // state to restore when this entry is applied
at: number; // original edit timestamp (for global undo ordering)
label: string;
} }
// Lightweight pub-sub for current time during playback. // Lightweight pub-sub for current time during playback.
@@ -141,6 +174,7 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
currentTime: 0, currentTime: 0,
duration: 0, duration: 0,
timelineReady: false, timelineReady: false,
beatDragging: false,
elements: [], elements: [],
selectedElementId: null, selectedElementId: null,
playbackRate: readStudioUiPreferences().playbackRate ?? 1, playbackRate: readStudioUiPreferences().playbackRate ?? 1,
@@ -193,6 +227,50 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
lintFindingsByElement: new Map(), lintFindingsByElement: new Map(),
setLintFindingsByElement: (map) => set({ lintFindingsByElement: map }), setLintFindingsByElement: (map) => set({ lintFindingsByElement: map }),
beatAnalysis: null,
setBeatAnalysis: (analysis) => set({ beatAnalysis: analysis }),
beatEdits: null,
setBeatEdits: (edits) => set({ beatEdits: edits }),
beatUndo: [],
beatRedo: [],
beatPersist: null,
setBeatPersist: (fn) => set({ beatPersist: fn }),
commitBeatEdits: (next, label) => {
set((s) => ({
beatEdits: next,
beatUndo: [...s.beatUndo, { restore: s.beatEdits, at: Date.now(), label }],
beatRedo: [],
}));
get().beatPersist?.();
},
undoBeatEdits: () => {
const s = get();
const entry = s.beatUndo[s.beatUndo.length - 1];
if (!entry) return null;
set({
beatEdits: entry.restore,
beatUndo: s.beatUndo.slice(0, -1),
beatRedo: [...s.beatRedo, { restore: s.beatEdits, at: entry.at, label: entry.label }],
});
get().beatPersist?.();
return entry.label;
},
resetBeatHistory: () => set({ beatUndo: [], beatRedo: [] }),
redoBeatEdits: () => {
const s = get();
const entry = s.beatRedo[s.beatRedo.length - 1];
if (!entry) return null;
set({
beatEdits: entry.restore,
beatRedo: s.beatRedo.slice(0, -1),
beatUndo: [...s.beatUndo, { restore: s.beatEdits, at: entry.at, label: entry.label }],
});
get().beatPersist?.();
return entry.label;
},
setIsPlaying: (playing) => { setIsPlaying: (playing) => {
if (get().isPlaying === playing) return; if (get().isPlaying === playing) return;
set({ isPlaying: playing }); set({ isPlaying: playing });
@@ -233,6 +311,7 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
setCurrentTime: (time) => set({ currentTime: Number.isFinite(time) ? time : 0 }), setCurrentTime: (time) => set({ currentTime: Number.isFinite(time) ? time : 0 }),
setDuration: (duration) => set({ duration: Number.isFinite(duration) ? duration : 0 }), setDuration: (duration) => set({ duration: Number.isFinite(duration) ? duration : 0 }),
setTimelineReady: (ready) => set({ timelineReady: ready }), setTimelineReady: (ready) => set({ timelineReady: ready }),
setBeatDragging: (dragging) => set({ beatDragging: dragging }),
setElements: (elements) => set({ elements }), setElements: (elements) => set({ elements }),
setSelectedElementId: (id) => set({ selectedElementId: id }), setSelectedElementId: (id) => set({ selectedElementId: id }),
updateElement: (elementId, updates) => updateElement: (elementId, updates) =>
@@ -250,6 +329,7 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
currentTime: 0, currentTime: 0,
duration: 0, duration: 0,
timelineReady: false, timelineReady: false,
beatDragging: false,
elements: [], elements: [],
selectedElementId: null, selectedElementId: null,
inPoint: null, inPoint: null,
@@ -258,5 +338,12 @@ export const usePlayerStore = create<PlayerState>((set, get) => ({
selectedKeyframes: new Set(), selectedKeyframes: new Set(),
selectedElementIds: new Set(), selectedElementIds: new Set(),
keyframeCache: new Map(), keyframeCache: new Map(),
// Beat state is project-specific — clear it so a project switch can't
// apply the previous project's beats/undo/persist to the new one.
beatAnalysis: null,
beatEdits: null,
beatUndo: [],
beatRedo: [],
beatPersist: null,
}), }),
})); }));
@@ -0,0 +1,109 @@
// Imperative beat-edit operations driven by the player store. Times passed in
// are COMPOSITION coordinates (timeline seconds); they're converted to audio-file
// coordinates internally and strength is measured from the decoded audio.
import { usePlayerStore, type TimelineElement } from "../player/store/playerStore";
import { isMusicTrack } from "./timelineInspector";
import { strengthAtTime, type MusicBeatAnalysis } from "@hyperframes/core/beats";
import {
addUserBeat,
removeUserBeat,
moveUserBeat,
mergeUserBeats,
type BeatEditState,
} from "./beatEditing";
/**
* Merge user beat edits into the detected analysis and remap from audio-file to
* composition coordinates (filtered to the music clip's visible range). Returns
* null when there's no music element, so beats never paint at wrong positions.
*/
export function remapBeatAnalysisToComposition(
beatAnalysis: MusicBeatAnalysis | null,
musicElement: Pick<TimelineElement, "src" | "start" | "playbackStart" | "duration"> | null,
beatEdits: BeatEditState | null,
): MusicBeatAnalysis | null {
if (!beatAnalysis || !musicElement) return null;
const merged = mergeUserBeats(
beatAnalysis.beatTimes,
beatAnalysis.beatStrengths,
beatEdits,
musicElement.src ?? null,
);
const playbackStart = musicElement.playbackStart ?? 0;
const clipEnd = playbackStart + musicElement.duration;
const offset = musicElement.start - playbackStart;
const times: number[] = [];
const strengths: number[] = [];
merged.times.forEach((t, i) => {
if (t >= playbackStart && t <= clipEnd) {
times.push(Math.round((t + offset) * 1000) / 1000);
strengths.push(merged.strengths[i] ?? 1);
}
});
return { ...beatAnalysis, beatTimes: times, beatStrengths: strengths };
}
function ctx() {
const s = usePlayerStore.getState();
const music = s.elements.find(isMusicTrack);
const analysis = s.beatAnalysis;
if (!music || !analysis || !music.src) return null;
return { s, music, analysis, src: music.src };
}
function compToAudio(start: number, playbackStart: number, compT: number): number {
return playbackStart + (compT - start);
}
// Clip length on the timeline. Falls back to source/analysis length when the
// media duration hasn't been probed yet (0), so the add window isn't degenerate.
function clipDuration(music: { duration: number; sourceDuration?: number }): number {
if (music.duration > 0) return music.duration;
if (music.sourceDuration && music.sourceDuration > 0) return music.sourceDuration;
return Number.POSITIVE_INFINITY;
}
/** True when a music track with analysis exists and the time is inside the clip. */
export function canAddBeatAt(compT: number): boolean {
const c = ctx();
if (!c) return false;
return compT >= c.music.start && compT <= c.music.start + clipDuration(c.music);
}
export function addBeatAtCompositionTime(compT: number): void {
const c = ctx();
if (!c) return;
const playbackStart = c.music.playbackStart ?? 0;
const audioT = compToAudio(c.music.start, playbackStart, compT);
if (audioT < playbackStart || audioT > playbackStart + clipDuration(c.music)) return;
const strength = strengthAtTime(c.analysis, audioT);
const next = addUserBeat(c.s.beatEdits, c.src, { time: audioT, strength }, c.analysis.beatTimes);
// No-op when the beat lands on an existing one — skip the undo entry + write.
if (next !== c.s.beatEdits) c.s.commitBeatEdits(next, "add beat");
}
export function deleteBeatAtCompositionTime(compT: number): void {
const c = ctx();
if (!c) return;
const audioT = compToAudio(c.music.start, c.music.playbackStart ?? 0, compT);
const next = removeUserBeat(c.s.beatEdits, c.src, c.analysis.beatTimes, audioT);
// No-op when there was no beat to remove — skip the undo entry + write.
if (next !== c.s.beatEdits) c.s.commitBeatEdits(next, "delete beat");
}
export function moveBeatCompositionTime(fromCompT: number, toCompT: number): void {
const c = ctx();
if (!c) return;
const playbackStart = c.music.playbackStart ?? 0;
const fromAudio = compToAudio(c.music.start, playbackStart, fromCompT);
const toAudio = compToAudio(c.music.start, playbackStart, toCompT);
const clamped = Math.max(playbackStart, Math.min(playbackStart + clipDuration(c.music), toAudio));
const strength = strengthAtTime(c.analysis, clamped);
const next = moveUserBeat(c.s.beatEdits, c.src, c.analysis.beatTimes, fromAudio, {
time: clamped,
strength,
});
// No-op when the move resolves to no change — skip the undo entry + write.
if (next !== c.s.beatEdits) c.s.commitBeatEdits(next, "move beat");
}
+136
View File
@@ -0,0 +1,136 @@
// User edits to the detected beat grid. All times are in AUDIO-FILE coordinates
// (offsets into the music source), matching MusicBeatAnalysis.beatTimes, so edits
// survive moving/trimming the music clip on the timeline.
export interface UserBeat {
time: number; // audio-file seconds
strength: number; // 01, measured from audio
}
export interface BeatEditState {
/** Music src these edits apply to; edits reset when the src changes. */
src: string;
/** Beats the user added (audio-file coords). */
added: UserBeat[];
/** Audio-file times of detected beats the user removed. */
removed: number[];
}
// Two beat times within this many seconds are treated as the same beat.
const MATCH_EPS = 0.015;
function near(a: number, b: number): boolean {
return Math.abs(a - b) < MATCH_EPS;
}
function activeEdits(edits: BeatEditState | null, src: string | null): BeatEditState | null {
return edits && src && edits.src === src ? edits : null;
}
/** Merge detected beats with user edits → effective beats (audio-file coords). */
export function mergeUserBeats(
detectedTimes: number[],
detectedStrengths: number[],
edits: BeatEditState | null,
src: string | null,
): { times: number[]; strengths: number[] } {
const e = activeEdits(edits, src);
const removed = e?.removed ?? [];
const merged: UserBeat[] = [];
for (let i = 0; i < detectedTimes.length; i++) {
const t = detectedTimes[i]!;
if (removed.some((r) => near(r, t))) continue;
merged.push({ time: t, strength: detectedStrengths[i] ?? 0.5 });
}
if (e) {
// Skip added beats that land on an already-present (detected) beat so an
// "add" near an existing beat doesn't create a near-duplicate.
for (const b of e.added) {
if (!merged.some((m) => near(m.time, b.time))) merged.push(b);
}
}
merged.sort((a, b) => a.time - b.time);
return { times: merged.map((b) => b.time), strengths: merged.map((b) => b.strength) };
}
function base(edits: BeatEditState | null, src: string): BeatEditState {
const e = activeEdits(edits, src);
return e
? { ...e, added: [...e.added], removed: [...e.removed] }
: { src, added: [], removed: [] };
}
/**
* Add a beat at an audio-file time. `detectedTimes` lets us no-op when the beat
* lands on an existing (non-removed) detected beat — otherwise the merge would
* drop it anyway and we'd record a phantom edit/undo/write. Returns the SAME
* reference when nothing changed so callers can skip persisting.
*/
export function addUserBeat(
edits: BeatEditState | null,
src: string,
beat: UserBeat,
detectedTimes: number[] = [],
): BeatEditState | null {
const active = activeEdits(edits, src);
// Already covered by a surviving detected beat → nothing to do.
const onLiveDetected =
detectedTimes.some((t) => near(t, beat.time)) &&
!(active?.removed ?? []).some((r) => near(r, beat.time));
if (onLiveDetected) return edits;
// Already an added beat here → nothing to do.
if ((active?.added ?? []).some((b) => near(b.time, beat.time))) return edits;
const next = base(edits, src);
// If a detected beat here was previously removed, drop the removal instead of stacking.
const ri = next.removed.findIndex((r) => near(r, beat.time));
if (ri >= 0) {
next.removed.splice(ri, 1);
return next;
}
next.added.push(beat);
return next;
}
/**
* Remove the beat nearest `time` — drops a user-added beat or hides a detected
* one. Returns the SAME reference when nothing changed (no added beat near
* `time`, and no live detected beat to hide) so callers can skip persisting a
* phantom edit/undo/write.
*/
export function removeUserBeat(
edits: BeatEditState | null,
src: string,
detectedTimes: number[],
time: number,
): BeatEditState | null {
const active = activeEdits(edits, src);
const hasAdded = (active?.added ?? []).some((b) => near(b.time, time));
const detected = detectedTimes.find((t) => near(t, time));
const alreadyHidden =
detected !== undefined && (active?.removed ?? []).some((r) => near(r, detected));
if (!hasAdded && (detected === undefined || alreadyHidden)) return edits;
const next = base(edits, src);
const ai = next.added.findIndex((b) => near(b.time, time));
if (ai >= 0) {
next.added.splice(ai, 1);
return next;
}
if (detected !== undefined && !next.removed.some((r) => near(r, detected))) {
next.removed.push(detected);
}
return next;
}
/** Move the beat at `fromTime` to `toBeat` (delete original, add new). */
export function moveUserBeat(
edits: BeatEditState | null,
src: string,
detectedTimes: number[],
fromTime: number,
toBeat: UserBeat,
): BeatEditState | null {
const removed = removeUserBeat(edits, src, detectedTimes, fromTime);
return addUserBeat(removed, src, toBeat, detectedTimes) ?? removed;
}
@@ -0,0 +1,31 @@
import type { TimelineElement } from "../player";
const AUDIO_TIMELINE_TAGS = new Set(["audio", "music", "sfx", "sound", "narration"]);
const AUDIO_SOURCE_EXT_RE = /\.(aac|flac|m4a|mp3|ogg|opus|wav)(?:[?#].*)?$/i;
const MUSIC_ID_RE = /\b(music|bgm|soundtrack|background[-_]?music)\b/i;
function isAudioTimelineElement(
element: Pick<TimelineElement, "tag" | "src"> | null | undefined,
): boolean {
if (!element) return false;
const tag = element.tag.trim().toLowerCase();
if (AUDIO_TIMELINE_TAGS.has(tag)) return true;
return Boolean(element.src && AUDIO_SOURCE_EXT_RE.test(element.src));
}
/** True for the music track: an audio element with data-timeline-role="music",
* or — when no role is set — an id matching the music regex. Voiceover/other
* audio (explicit non-music role) is excluded. */
export function isMusicTrack(
element:
| Pick<TimelineElement, "tag" | "src" | "id" | "domId" | "timelineRole">
| null
| undefined,
): boolean {
if (!element) return false;
if (!isAudioTimelineElement(element)) return false;
if (element.timelineRole === "music") return true;
if (element.timelineRole && element.timelineRole !== "music") return false;
const id = element.domId ?? element.id ?? "";
return MUSIC_ID_RE.test(id);
}
+3
View File
@@ -180,6 +180,9 @@ export default defineConfig({
outDir: "dist", outDir: "dist",
emptyOutDir: true, emptyOutDir: true,
}, },
optimizeDeps: {
include: ["bpm-detective"],
},
server: { server: {
port: 5190, port: 5190,
}, },