diff --git a/skills/gsap-effects/SKILL.md b/skills/gsap-effects/SKILL.md index 2f71fe8ce..5369015c0 100644 --- a/skills/gsap-effects/SKILL.md +++ b/skills/gsap-effects/SKILL.md @@ -1,16 +1,17 @@ --- name: gsap-effects -description: Ready-made GSAP animation effects for HyperFrames compositions. Use when adding typewriter text, text reveals, or character-by-character animation to a composition. Reference files contain copy-paste patterns. +description: Ready-made animation effects for HyperFrames compositions. Use when adding typewriter text, text reveals, character-by-character animation, audio visualizations, spectrum bars, waveform displays, or any reactive audio-driven animation to a composition. Also use when audio has been analyzed or transcribed in the current session and music is detected — the audio visualizer can enhance the composition with reactive visuals. Reference files contain patterns and data contracts. --- # GSAP Effects -Drop-in animation patterns for HyperFrames compositions. Each effect is a self-contained reference with the HTML, CSS, and GSAP code needed to add it to a composition. +Drop-in animation patterns for HyperFrames compositions. Each effect is a self-contained reference with the HTML, CSS, and code needed to add it to a composition. These effects follow all HyperFrames composition rules — deterministic, no randomness, timelines registered via `window.__timelines`. ## Available Effects -| Effect | File | Use when | -| ---------- | -------------------------------- | ---------------------------------------------------------------------------- | -| Typewriter | [typewriter.md](./typewriter.md) | Text should appear character by character, with or without a blinking cursor | +| Effect | File | Use when | +| ---------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| Typewriter | [typewriter.md](./typewriter.md) | Text should appear character by character, with or without a blinking cursor | +| Audio Visualizer | [audio-visualizer.md](./audio-visualizer.md) | Reactive bars, waveforms, circles, or glow that respond to audio. Includes extraction script and Canvas 2D patterns | diff --git a/skills/gsap-effects/audio-visualizer.md b/skills/gsap-effects/audio-visualizer.md new file mode 100644 index 000000000..c9edb2592 --- /dev/null +++ b/skills/gsap-effects/audio-visualizer.md @@ -0,0 +1,238 @@ +# Audio Visualizer + +Reactive audio visualizations for HyperFrames compositions. Pre-extracts amplitude and frequency data from an audio file, then drives rendering from the GSAP timeline. + +## Why Pre-Extraction + +HyperFrames renders frame-by-frame in headless Chrome — there's no audio playing during rendering, so the Web Audio API's real-time `AnalyserNode` won't work. Instead, extract all audio data before the composition runs and bake it as a static JSON array. The composition reads the array by frame index. This is fully deterministic and seekable. + +## Step 1: Extract Audio Data + +```bash +python skills/gsap-effects/scripts/extract-audio-data.py audio.mp3 -o audio-data.json +python skills/gsap-effects/scripts/extract-audio-data.py video.mp4 --fps 30 --bands 16 -o audio-data.json +``` + +Requires ffmpeg and numpy (`pip install numpy`). + +| Flag | Default | Description | +| --------- | --------------- | -------------------------------------------------------- | +| `--fps` | 30 | Must match the composition/render FPS | +| `--bands` | 16 | Number of frequency bands (more = finer spectrum detail) | +| `-o` | audio-data.json | Output path | + +The script uses a 4096-sample FFT window (not the per-frame sample count) to ensure each frequency band maps to distinct FFT bins. Bands are logarithmically spaced from 30Hz to 16kHz — the useful range for music. Each band is normalized independently across the full track so treble activity is visible even when bass is louder in absolute terms. + +## Step 2: Understanding the Data + +```json +{ + "duration": 180.5, + "fps": 30, + "bands": 16, + "totalFrames": 5415, + "frames": [ + { "time": 0.0, "rms": 0.0, "bands": [0.0, 0.0, 0.0, ...] }, + { "time": 0.0333, "rms": 0.42, "bands": [0.8, 0.6, 0.3, ...] } + ] +} +``` + +**`rms`** (0-1) — overall loudness of this frame, normalized across the full track. 0 is silence, 1 is the loudest moment in the entire audio. Use this for anything that should respond to overall energy: scaling, pulsing, glow intensity, opacity, movement speed. + +**`bands`** (array of 0-1 values) — frequency magnitudes. Each value is normalized independently for that band across the full track, so a 0.8 in treble means "this is 80% of the loudest this treble band gets anywhere in the audio" — not that treble is as loud as bass in absolute terms. This is what makes all frequency ranges visually active. + +- Index 0 = lowest bass (~30Hz). Index `n-1` = highest treble (~16kHz). +- Low indices (0-3) react to kick drums, bass lines, sub-bass rumble. +- Mid indices (4-9) react to vocals, guitars, synths, most melodic content. +- High indices (10-15) react to hi-hats, cymbals, sibilance, brightness. + +## Loading the Data + +Embed the data in the composition so it's available when the timeline runs. + +```js +// Option A: inline (small files, under ~500KB) +const AUDIO_DATA = { + /* paste audio-data.json contents */ +}; +setupTimeline(AUDIO_DATA); + +// Option B: fetch (large files) +fetch("audio-data.json") + .then((r) => r.json()) + .then((data) => { + setupTimeline(data); + }); + +function setupTimeline(AUDIO_DATA) { + // Register tl.call() draws here — AUDIO_DATA is guaranteed to be loaded + for (let f = 0; f < AUDIO_DATA.totalFrames; f++) { + tl.call( + () => { + draw(AUDIO_DATA.frames[f]); + }, + [], + f / AUDIO_DATA.fps, + ); + } +} +``` + +With fetch, wrap all timeline setup inside the callback so `AUDIO_DATA` is available when the `for` loop reads `totalFrames`. The fetch completes before the renderer's first seek because it waits for `window.__hf` readiness. + +## Step 3: Drive Rendering from the Timeline + +Register a `tl.call()` at every frame interval. Each call reads the pre-computed data and renders. This is deterministic and seekable — scrubbing in the studio works because each frame's draw is tied to a specific timeline position. + +## Rendering Approaches + +The data is framework-agnostic. Here's how to wire it up in each approach. + +### Canvas 2D + +Best for: bars, waveforms, circles, gradients, particles. Most common choice. + +```js +const canvas = document.querySelector("#viz-canvas"); +const ctx = canvas.getContext("2d"); + +for (let f = 0; f < AUDIO_DATA.totalFrames; f++) { + tl.call( + () => { + const frame = AUDIO_DATA.frames[f]; + if (!frame) return; + ctx.clearRect(0, 0, canvas.width, canvas.height); + // read frame.rms and frame.bands, draw whatever you want + }, + [], + f / AUDIO_DATA.fps, + ); +} +``` + +### WebGL / Three.js + +HyperFrames has a Three.js adapter that patches `THREE.Clock` for deterministic time. Create your scene normally, then update uniforms or object properties from the audio data each frame. + +```js +// In your Three.js setup: +const uniforms = { uBass: { value: 0 }, uMid: { value: 0 }, uRms: { value: 0 } }; + +for (let f = 0; f < AUDIO_DATA.totalFrames; f++) { + tl.call( + () => { + const frame = AUDIO_DATA.frames[f]; + if (!frame) return; + uniforms.uBass.value = Math.max(frame.bands[0], frame.bands[1], frame.bands[2]); + uniforms.uMid.value = Math.max(frame.bands[6], frame.bands[7], frame.bands[8]); + uniforms.uRms.value = frame.rms; + }, + [], + f / AUDIO_DATA.fps, + ); +} +``` + +### DOM Elements + +For simpler visualizations (a few bars, a pulsing element), you can animate DOM elements directly. Less performant than Canvas for many elements, but fine for under ~20. + +```js +const bars = document.querySelectorAll(".bar"); +for (let f = 0; f < AUDIO_DATA.totalFrames; f++) { + tl.call( + () => { + const frame = AUDIO_DATA.frames[f]; + if (!frame) return; + bars.forEach((bar, i) => { + bar.style.height = frame.bands[i] * 100 + "%"; + }); + }, + [], + f / AUDIO_DATA.fps, + ); +} +``` + +## Spatial Mapping + +When laying out frequency data spatially, follow these conventions so visualizations read naturally: + +- **Horizontal layouts**: low frequencies (bass) on the left, high frequencies (treble) on the right. Iterate the bands array left-to-right. +- **Vertical layouts**: low frequencies at the bottom, high frequencies at the top. +- **Circular layouts**: bass starts at the top (12 o'clock) and wraps clockwise. Mirror the bands array for a full circle. + +## Motion Principles + +### Smoothing + +Raw per-frame data changes abruptly. Blend with the previous frame for fluid motion: + +```js +let prev = null; +const smoothing = 0.25; // 0 = no smoothing, higher = more lag + +function smooth(f) { + const raw = AUDIO_DATA.frames[f]; + if (!raw) return prev; + if (!prev) { + prev = { rms: raw.rms, bands: [...raw.bands] }; + return prev; + } + prev = { + rms: prev.rms * smoothing + raw.rms * (1 - smoothing), + bands: raw.bands.map((b, i) => prev.bands[i] * smoothing + b * (1 - smoothing)), + }; + return prev; +} +``` + +Lower smoothing (0.1-0.2) feels snappy and responsive — good for percussive music. Higher smoothing (0.3-0.5) feels languid and flowing — good for ambient or orchestral. + +### Value Mapping + +Audio data is 0-1 but visual properties need different ranges. Map with intention: + +- **Scale/size**: multiply by a max value. A bar's height = `bands[i] * maxHeight`. Don't let elements disappear at 0 — add a minimum: `minHeight + bands[i] * (maxHeight - minHeight)`. +- **Opacity**: low values should still be slightly visible. `0.15 + bands[i] * 0.85` keeps elements present during quiet moments. +- **Color intensity**: shift between a muted base and a vivid peak. Interpolate HSL lightness or RGB channels based on the value. +- **Position/offset**: use rms to drive drift or wobble. Small movements (5-20px) feel organic; large movements look chaotic. + +### What Makes It Feel Good + +- **Bass drives the big moves.** Scale, position shifts, and glow should react to low bands. Bass is what makes a visualization feel like it's "hitting." +- **Treble drives the detail.** Small particle movements, edge shimmer, opacity flicker. Treble adds texture without dominating. +- **RMS drives global properties.** Background brightness, overall scale, color warmth. It's the "energy level" of the whole frame. +- **Don't animate everything at once.** Pick 2-3 visual properties to tie to the audio. More than that looks noisy. +- **Quiet sections should still have life.** A completely static frame during a soft passage looks broken. Keep minimum values above zero. + +## Band Count Guide + +| Bands | Detail level | Good for | +| ----- | ------------ | ------------------------------------------- | +| 4 | Low | Simple pulsing, background glow | +| 8 | Medium | Bar visualizations, basic spectrum | +| 16 | High | Detailed EQ, circular visualizers (default) | +| 32 | Very high | Smooth curves, dense radial layouts | + +More bands = larger JSON file. 16 is a good default. + +## Layering + +Layer multiple canvases with CSS z-index for depth: + +```html + + +``` + +A background layer driven by bass/rms and a foreground layer driven by individual bands creates depth without complexity. + +## HyperFrames Integration Notes + +- The `` element needs `data-start`, `data-duration`, and `data-track-index` like any other clip +- Set canvas `width`/`height` attributes to match the composition dimensions (1920x1080) +- The extraction script FPS must match the render FPS (default: 30) +- For large audio files, the JSON can be several MB — load via `fetch` rather than inlining +- Each canvas in the composition needs its own `data-track-index` — don't put multiple canvases on the same track diff --git a/skills/gsap-effects/scripts/extract-audio-data.py b/skills/gsap-effects/scripts/extract-audio-data.py new file mode 100644 index 000000000..b4efba788 --- /dev/null +++ b/skills/gsap-effects/scripts/extract-audio-data.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +""" +Extract per-frame audio visualization data from an audio or video file. + +Outputs JSON with RMS amplitude and frequency band data at the target FPS, +ready to embed in a HyperFrames composition. + +Usage: + python extract-audio-data.py input.mp3 -o audio-data.json + python extract-audio-data.py input.mp4 --fps 30 --bands 16 -o audio-data.json + +Requirements: + - Python 3.9+ + - ffmpeg (for decoding audio) + - numpy (pip install numpy) +""" + +import argparse +import json +import subprocess +import sys + +import numpy as np + +# --------------------------------------------------------------------------- +# FFT parameters +# +# A 4096-sample window gives ~10.8 Hz per bin at 44100Hz — enough to resolve +# low-frequency bands cleanly. The per-frame audio slice (44100/30 = 1470 +# samples at 30fps) is too small and causes low bands to map to the same bins. +# +# Frequency range 30Hz–16kHz covers the useful range for music. Below 30Hz is +# sub-bass most speakers can't reproduce; above 16kHz is noise/harmonics that +# don't contribute to perceived rhythm or melody. +# --------------------------------------------------------------------------- + +SAMPLE_RATE = 44100 +FFT_SIZE = 4096 +MIN_FREQ = 30.0 +MAX_FREQ = 16000.0 + + +def decode_audio(path: str) -> np.ndarray: + """Decode audio to mono float32 samples via ffmpeg.""" + cmd = [ + "ffmpeg", "-i", path, + "-vn", "-ac", "1", "-ar", str(SAMPLE_RATE), + "-f", "s16le", "-acodec", "pcm_s16le", + "-loglevel", "error", + "pipe:1", + ] + result = subprocess.run(cmd, capture_output=True) + if result.returncode != 0: + print(f"ffmpeg error: {result.stderr.decode()}", file=sys.stderr) + sys.exit(1) + return np.frombuffer(result.stdout, dtype=np.int16).astype(np.float32) / 32768.0 + + +def compute_band_edges(n_bands: int) -> np.ndarray: + """Logarithmically-spaced frequency band edges from MIN_FREQ to MAX_FREQ.""" + return np.array([ + MIN_FREQ * (MAX_FREQ / MIN_FREQ) ** (i / n_bands) + for i in range(n_bands + 1) + ]) + + +def compute_fft_bands( + windowed: np.ndarray, freq_per_bin: float, n_bins: int, + band_edges: np.ndarray, n_bands: int, +) -> np.ndarray: + """Compute peak magnitude in logarithmically-spaced frequency bands.""" + magnitudes = np.abs(np.fft.rfft(windowed)) + + bands = np.zeros(n_bands) + for b in range(n_bands): + low_bin = max(0, int(band_edges[b] / freq_per_bin)) + high_bin = min(n_bins, int(band_edges[b + 1] / freq_per_bin)) + if high_bin <= low_bin: + high_bin = low_bin + 1 + # Clamp to valid range to avoid empty slices + low_bin = min(low_bin, n_bins - 1) + high_bin = min(high_bin, n_bins) + bands[b] = np.max(magnitudes[low_bin:high_bin]) + + return bands + + +def extract(path: str, fps: int, n_bands: int) -> dict: + """Extract per-frame audio data.""" + print(f"Decoding audio from {path}...", file=sys.stderr) + samples = decode_audio(path) + duration = len(samples) / SAMPLE_RATE + frame_step = SAMPLE_RATE // fps + total_frames = int(duration * fps) + + print(f"Duration: {duration:.1f}s, {total_frames} frames at {fps}fps", file=sys.stderr) + print(f"FFT window: {FFT_SIZE} samples ({SAMPLE_RATE / FFT_SIZE:.1f} Hz/bin)", file=sys.stderr) + print(f"Frequency range: {MIN_FREQ:.0f}-{MAX_FREQ:.0f} Hz, {n_bands} bands", file=sys.stderr) + + # Precompute constants + hann = np.hanning(FFT_SIZE) + band_edges = compute_band_edges(n_bands) + freq_per_bin = SAMPLE_RATE / FFT_SIZE + n_bins = FFT_SIZE // 2 + 1 + half_fft = FFT_SIZE // 2 + + # Pass 1: extract raw values + rms_values = np.zeros(total_frames) + band_values = np.zeros((total_frames, n_bands)) + + for f in range(total_frames): + # RMS from the frame's audio slice + rms_start = f * frame_step + rms_end = rms_start + frame_step + frame_slice = samples[rms_start:min(rms_end, len(samples))] + if len(frame_slice) > 0: + rms_values[f] = np.sqrt(np.mean(frame_slice ** 2)) + + # FFT from a centered 4096-sample window + center = rms_start + frame_step // 2 + win_start = center - half_fft + win_end = center + half_fft + + if win_start >= 0 and win_end <= len(samples): + window = samples[win_start:win_end] * hann + else: + # Zero-pad at edges + padded = np.zeros(FFT_SIZE) + src_start = max(0, win_start) + src_end = min(len(samples), win_end) + dst_start = src_start - win_start + dst_end = dst_start + (src_end - src_start) + padded[dst_start:dst_end] = samples[src_start:src_end] + window = padded * hann + + band_values[f] = compute_fft_bands(window, freq_per_bin, n_bins, band_edges, n_bands) + + # Pass 2: normalize + peak_rms = rms_values.max() if total_frames > 0 else 1.0 + if peak_rms > 0: + rms_values /= peak_rms + + # Per-band normalization so treble is visible alongside louder bass + band_peaks = band_values.max(axis=0) + band_peaks[band_peaks == 0] = 1.0 + band_values /= band_peaks + + # Build output + frames = [] + for f in range(total_frames): + frames.append({ + "time": round(f / fps, 4), + "rms": round(float(rms_values[f]), 4), + "bands": [round(float(b), 4) for b in band_values[f]], + }) + + return { + "duration": round(duration, 4), + "fps": fps, + "bands": n_bands, + "totalFrames": total_frames, + "frames": frames, + } + + +def main(): + parser = argparse.ArgumentParser(description="Extract per-frame audio visualization data") + parser.add_argument("input", help="Audio or video file") + parser.add_argument("-o", "--output", default="audio-data.json", help="Output JSON path") + parser.add_argument("--fps", type=int, default=30, help="Frames per second (default: 30)") + parser.add_argument("--bands", type=int, default=16, help="Number of frequency bands (default: 16)") + args = parser.parse_args() + + if args.fps < 1: + parser.error("--fps must be at least 1") + if args.bands < 1: + parser.error("--bands must be at least 1") + + data = extract(args.input, args.fps, args.bands) + + with open(args.output, "w") as f: + json.dump(data, f) + + print(f"Wrote {args.output} ({data['totalFrames']} frames, {data['bands']} bands)", file=sys.stderr) + + +if __name__ == "__main__": + main()