mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
feat(studio,cli): music beat detection with timeline guides + headless beats CLI (#1424)
* feat(studio,cli): music beat detection with timeline guides + headless beats CLI Beat detection for music tracks: the Studio draws beat guides on the active track, beats are user-editable and persist to a project file, and a new `hyperframes beats` CLI generates that file headlessly before the Studio opens. Detection lives in @hyperframes/core/beats (shared by Studio + CLI): an energy onset detector cross-validated with bpm-detective, regularized to an octave- aligned grid, silence-gated, with per-beat loudness. Music-only — an <audio data-timeline-role="music"> is analyzed; voiceover is excluded. Studio: green beat lines + draggable dots on the selected track; add at playhead, drag to move, double-click to delete (audio scrubs); edits persist to beats/<audio>.json and are undoable (interleaved with file history). CLI: `hyperframes beats [dir]` runs the same detection in headless Chrome (prebuilt browser bundle in dist) and writes the beat file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): timeline beat-grid + zoom UX refinements - Center-anchored magnify: zooming via the toolbar/slider keeps the time at the viewport center fixed instead of anchoring at the left. Pinch still anchors at the cursor. - Move-snap to beats: dragging a clip snaps whichever edge (start or end) is nearest a beat, matching the existing resize-edge snapping. - Beat lines on track backgrounds: faint full-height beat lines now paint behind the clips on every track lane (brightness scales with loudness); the green dots stay on the active track's top bar. - Waveform follows zoom: bars fill the full clip width and resample the windowed peaks, so the waveform stretches with zoom instead of stopping partway across a widened clip. - Beat dots centered in the top bar: align the dot band to the clip top (CLIP_Y) so the dots sit centered in the dark bar instead of being bisected by the clip's top border. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(studio): preserve media sourceDuration across element re-derivation Moving a non-music clip re-derived the timeline elements into fresh objects whose sourceDuration the DOM scan hadn't loaded yet. The async probe skips srcs already in its cache, so the value was silently dropped — trimFractions then returned no window and the trimmed music waveform reset to the full source pinned at the track start. Re-apply the cached probe duration synchronously on every derivation (applyCachedSourceDurations) and extract the async probe loop into probeMissingSourceDurations to keep useTimelinePlayer within the file size limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): skip beat-snap on the music track, highlight move-snap target The music track defines the beats, so moving or trimming it no longer snaps to its own beats (isMusicTrack guard on both the move and resize snap paths). Moving another clip snapped only on drop with no cue. snapMoveStartToBeat now also returns the beat it will snap to; BeatBackgroundLines draws that beat's line as a bright neon-green glow while the clip's edge is within the snap region, so the target is visible before drop. Also drops .commitmsg.tmp, accidentally committed via git add -A. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * feat(studio): hide playhead while dragging a beat; default beat dots to music track - Dragging a beat dot now hides the playhead guideline (new beatDragging store flag set on beat pointer down/up) so its line doesn't track the scrub and clutter the beat being moved. - Beat dots render on the selected track, falling back to the music track when nothing is selected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(core): remove polynomial-ReDoS regex from audioRelPathForSrc CodeQL js/polynomial-redos: the lazy `.+?` followed by an optional trailing `[?#].*$` backtracks polynomially on crafted `/preview/...` inputs. Parse the preview-relative path with indexOf/slice instead, and strip the query/hash with a single linear char-class search. Behavior is unchanged for all preview/absolute/blob/data/bare inputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(studio,core,cli): review hardening for beat detection + timeline UX - playerStore.reset() now clears beat state (analysis, edits, undo/redo, persist) so a project switch can't apply the previous project's beats, undo stack, or file-writer to the new one. - removeUserBeat returns the same reference on a no-op, and delete/move beat actions skip committing when nothing changed — no more phantom undo entries / debounced writes for no-op edits. - regularizeBeats bails to raw onsets when the (octave-misread) tempo would produce a sub-125ms grid, avoiding a tens-of-thousands-of-beats freeze. - parseBeats clamps strength to [0,1] and rejects non-finite time/strength, so a hand-edited file can't feed NaN into the gamma curve (Math.pow on a negative base) and blank out beat markers. - Start-edge beat-snap now also requires duration >= minDuration, matching the end-edge guard, so a rightward snap can't collapse the clip. - Center-anchor zoom effect always consumes its skip flag, so a pinch that produced no pps change can't leave it stranded and skip the next zoom. - Headless beats analyzer projects to {beatTimes,beatStrengths,bpm,confidence} before returning, so page.evaluate no longer serializes the full decoded PCM (channelData) across the CDP boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> * fix(core): gate parseBeats on schema version parseBeats accepted any object with a beats array, so a future v2 beat file (with changed semantics) would be parsed silently as v1. Reject anything whose version is not 1, treating an unknown version like an absent/invalid file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com> --------- Co-authored-by: Miguel Ángel <miguel07alm@protonmail.com>
This commit is contained in:
Vance Ingalls
co-authored by
Claude Opus 4.8
Miguel Ángel
Claude Opus 4.8
Miguel Ángel
parent
a95e49dbda
commit
d9f69f61e7
@@ -0,0 +1,152 @@
|
||||
// Run the shared beat detection (@hyperframes/core/beats) in a headless Chrome
|
||||
// so results match the Studio exactly — same Web Audio decode + same
|
||||
// bpm-detective. Used by the `beats` CLI command to write the beat file before
|
||||
// the Studio is ever opened.
|
||||
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import type { Browser, Page } from "puppeteer-core";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
|
||||
// The detection is browser code. We need it as an IIFE that exposes
|
||||
// analyzeMusicFromBuffer on the page. Prefer the artifact prebuilt at CLI build
|
||||
// time (shipped in dist); fall back to bundling from core source at runtime
|
||||
// (dev/monorepo, where core's src is on disk).
|
||||
let bundlePromise: Promise<string> | null = null;
|
||||
|
||||
function findPrebuiltBundle(): string | null {
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const candidates = [
|
||||
join(here, "beat-analyzer.global.js"), // dist root (tsup-bundled cli)
|
||||
join(here, "../beat-analyzer.global.js"), // dist/beats → dist
|
||||
join(here, "../dist/beat-analyzer.global.js"),
|
||||
];
|
||||
for (const p of candidates) {
|
||||
if (existsSync(p)) return p;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function buildFromCoreSource(): Promise<string> {
|
||||
const esbuild = await import("esbuild");
|
||||
const coreRoot = dirname(require.resolve("@hyperframes/core/package.json"));
|
||||
const entry = join(coreRoot, "src/beats/beatDetection.ts");
|
||||
const result = await esbuild.build({
|
||||
stdin: {
|
||||
contents:
|
||||
`import { analyzeMusicFromBuffer } from ${JSON.stringify(entry)};\n` +
|
||||
`globalThis.__hfAnalyze = analyzeMusicFromBuffer;`,
|
||||
resolveDir: coreRoot,
|
||||
loader: "ts",
|
||||
},
|
||||
bundle: true,
|
||||
format: "iife",
|
||||
platform: "browser",
|
||||
target: "es2020",
|
||||
write: false,
|
||||
});
|
||||
const out = result.outputFiles?.[0];
|
||||
if (!out) throw new Error("Failed to bundle beat analyzer");
|
||||
return out.text;
|
||||
}
|
||||
|
||||
function buildAnalyzerBundle(): Promise<string> {
|
||||
if (bundlePromise) return bundlePromise;
|
||||
bundlePromise = (async () => {
|
||||
const prebuilt = findPrebuiltBundle();
|
||||
if (prebuilt) return readFileSync(prebuilt, "utf8");
|
||||
return buildFromCoreSource();
|
||||
})().catch((err) => {
|
||||
bundlePromise = null; // don't poison the process with a cached rejection
|
||||
throw err;
|
||||
});
|
||||
return bundlePromise;
|
||||
}
|
||||
|
||||
export interface HeadlessBeatResult {
|
||||
beatTimes: number[];
|
||||
beatStrengths: number[];
|
||||
bpm: number | null;
|
||||
bpmConfidence: string;
|
||||
}
|
||||
|
||||
// Guard against pathological inputs that would blow CDP message limits when
|
||||
// transferred to the page as base64 (≈ +33% over the raw bytes).
|
||||
const MAX_AUDIO_BYTES = 80 * 1024 * 1024;
|
||||
|
||||
// Runs inside the headless page: decode the base64 audio and analyze it.
|
||||
function inPageAnalyze(data: string) {
|
||||
const bin = atob(data);
|
||||
const bytes = new Uint8Array(bin.length);
|
||||
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
|
||||
const win = window as unknown as {
|
||||
AudioContext: typeof AudioContext;
|
||||
webkitAudioContext?: typeof AudioContext;
|
||||
__hfAnalyze?: (buffer: AudioBuffer) => Promise<HeadlessBeatResult>;
|
||||
};
|
||||
if (typeof win.__hfAnalyze !== "function") throw new Error("beat analyzer not loaded");
|
||||
const ctx = new (win.AudioContext || win.webkitAudioContext!)();
|
||||
return (
|
||||
ctx
|
||||
.decodeAudioData(bytes.buffer)
|
||||
.then((buf) => win.__hfAnalyze!(buf))
|
||||
// analyzeMusicFromBuffer also returns the decoded PCM (channelData) + sampleRate;
|
||||
// project to only the fields we need so page.evaluate doesn't serialize an
|
||||
// ~8-million-element Float32Array back across the CDP boundary.
|
||||
.then((r) => ({
|
||||
beatTimes: r.beatTimes,
|
||||
beatStrengths: r.beatStrengths,
|
||||
bpm: r.bpm,
|
||||
bpmConfidence: r.bpmConfidence,
|
||||
}))
|
||||
.finally(() => ctx.close())
|
||||
);
|
||||
}
|
||||
|
||||
// Load the analyzer bundle into the page, run analysis, and surface in-page
|
||||
// errors (decode/codec failures, missing global) instead of an opaque rejection.
|
||||
async function detectOnPage(page: Page, bundle: string, b64: string): Promise<HeadlessBeatResult> {
|
||||
const pageErrors: string[] = [];
|
||||
page.on("pageerror", (e) => {
|
||||
pageErrors.push((e as Error).message);
|
||||
});
|
||||
page.on("console", (m) => {
|
||||
if (m.type() === "error") pageErrors.push(m.text());
|
||||
});
|
||||
await page.setContent("<!doctype html><html><body></body></html>");
|
||||
await page.addScriptTag({ content: bundle });
|
||||
try {
|
||||
return (await page.evaluate(inPageAnalyze, b64)) as HeadlessBeatResult;
|
||||
} catch (err) {
|
||||
const detail = pageErrors.length ? ` (${pageErrors.join("; ")})` : "";
|
||||
throw new Error(`${err instanceof Error ? err.message : String(err)}${detail}`);
|
||||
}
|
||||
}
|
||||
|
||||
/** Decode + analyze the given audio bytes in headless Chrome. */
|
||||
export async function analyzeBeatsHeadless(audioBytes: Buffer): Promise<HeadlessBeatResult> {
|
||||
if (audioBytes.length > MAX_AUDIO_BYTES) {
|
||||
const mb = Math.round(audioBytes.length / 1e6);
|
||||
throw new Error(
|
||||
`Audio file too large for headless analysis (${mb}MB > ${MAX_AUDIO_BYTES / 1e6}MB).`,
|
||||
);
|
||||
}
|
||||
const bundle = await buildAnalyzerBundle();
|
||||
const { ensureBrowser } = await import("../browser/manager.js");
|
||||
const puppeteer = await import("puppeteer-core");
|
||||
const browser = await ensureBrowser();
|
||||
const chrome: Browser = await puppeteer.default.launch({
|
||||
headless: true,
|
||||
executablePath: browser.executablePath,
|
||||
args: ["--no-sandbox", "--disable-dev-shm-usage", "--autoplay-policy=no-user-gesture-required"],
|
||||
});
|
||||
try {
|
||||
const page = await chrome.newPage();
|
||||
return await detectOnPage(page, bundle, audioBytes.toString("base64"));
|
||||
} finally {
|
||||
await chrome.close();
|
||||
}
|
||||
}
|
||||
@@ -117,6 +117,7 @@ const subCommands = {
|
||||
publish: () => import("./commands/publish.js").then((m) => m.default),
|
||||
render: () => import("./commands/render.js").then((m) => m.default),
|
||||
lint: () => import("./commands/lint.js").then((m) => m.default),
|
||||
beats: () => import("./commands/beats.js").then((m) => m.default),
|
||||
inspect: () => import("./commands/inspect.js").then((m) => m.default),
|
||||
layout: () => import("./commands/layout.js").then((m) => m.default),
|
||||
info: () => import("./commands/info.js").then((m) => m.default),
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
import { defineCommand } from "citty";
|
||||
import { existsSync, readFileSync, mkdirSync, writeFileSync } from "node:fs";
|
||||
import { resolve, join, dirname } from "node:path";
|
||||
import { findMusicAudioSrc, audioRelPathForSrc, serializeBeats } from "@hyperframes/core/beats";
|
||||
import type { Example } from "./_examples.js";
|
||||
import { resolveProject, type ProjectDir } from "../utils/project.js";
|
||||
import { analyzeBeatsHeadless, type HeadlessBeatResult } from "../beats/headlessAnalyzer.js";
|
||||
import { c } from "../ui/colors.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
["Generate the beat file for the current project", "hyperframes beats"],
|
||||
["Generate for a specific directory", "hyperframes beats ./my-video"],
|
||||
];
|
||||
|
||||
function fail(message: string): never {
|
||||
console.error(c.error(message));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
/** Locate the music track + its on-disk audio, or fail with a clear message. */
|
||||
function resolveMusicTarget(project: ProjectDir): { rel: string; audioPath: string } {
|
||||
const src = findMusicAudioSrc(readFileSync(project.indexPath, "utf-8"));
|
||||
if (!src) {
|
||||
fail(
|
||||
'No music track found. Add data-timeline-role="music" to the <audio> element ' +
|
||||
"(or give it an id like music/bgm/soundtrack).",
|
||||
);
|
||||
}
|
||||
const rel = audioRelPathForSrc(src); // same derivation the Studio uses
|
||||
if (!rel) fail(`Cannot derive a beat-file path for music src: ${src}`);
|
||||
const audioPath = resolve(project.dir, rel);
|
||||
if (!existsSync(audioPath)) fail(`Audio file not found: ${rel}`);
|
||||
return { rel, audioPath };
|
||||
}
|
||||
|
||||
async function detect(audioPath: string): Promise<HeadlessBeatResult> {
|
||||
try {
|
||||
return await analyzeBeatsHeadless(readFileSync(audioPath));
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
const hint = /chrome|executable|browser|ENOENT/i.test(msg)
|
||||
? "\nRun: npx hyperframes browser ensure"
|
||||
: "";
|
||||
fail(`Beat detection failed: ${msg}${hint}`);
|
||||
}
|
||||
}
|
||||
|
||||
function report(file: string, result: HeadlessBeatResult, json: boolean): void {
|
||||
if (json) {
|
||||
console.log(
|
||||
JSON.stringify({ ok: true, file, count: result.beatTimes.length, bpm: result.bpm }, null, 2),
|
||||
);
|
||||
return;
|
||||
}
|
||||
console.log(
|
||||
c.success(
|
||||
`✓ Wrote ${result.beatTimes.length} beats → ${file} (bpm ${result.bpm ?? "?"}, ${result.bpmConfidence})`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "beats",
|
||||
description: "Detect beats in the music track (headless) and write beats/<audio>.json",
|
||||
},
|
||||
args: {
|
||||
dir: { type: "positional", description: "Project directory", required: false },
|
||||
json: { type: "boolean", description: "Output result as JSON", default: false },
|
||||
},
|
||||
async run({ args }) {
|
||||
const project = resolveProject(args.dir);
|
||||
const { rel, audioPath } = resolveMusicTarget(project);
|
||||
if (!args.json) console.log(c.dim(`Analyzing ${rel} in headless Chrome…`));
|
||||
|
||||
const result = await detect(audioPath);
|
||||
// The Studio ignores a 0-beat file (treats it as a stale seed), so don't write one.
|
||||
if (result.beatTimes.length === 0) {
|
||||
fail(`No beats detected in ${rel} — nothing written. (Track may be silent/ambient.)`);
|
||||
}
|
||||
|
||||
const outPath = join(project.dir, "beats", `${rel}.json`);
|
||||
mkdirSync(dirname(outPath), { recursive: true });
|
||||
writeFileSync(outPath, serializeBeats(result.beatTimes, result.beatStrengths, rel));
|
||||
report(`beats/${rel}.json`, result, Boolean(args.json));
|
||||
},
|
||||
});
|
||||
@@ -32,6 +32,7 @@ const GROUPS: Group[] = [
|
||||
title: "Project",
|
||||
commands: [
|
||||
["lint", "Validate a composition for common mistakes"],
|
||||
["beats", "Detect beats in the music track and write beats/<audio>.json"],
|
||||
["inspect", "Inspect rendered visual layout across the timeline"],
|
||||
["snapshot", "Capture key frames as PNG screenshots for visual verification"],
|
||||
["info", "Print project metadata"],
|
||||
|
||||
Reference in New Issue
Block a user