mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-11 06:30:03 +00:00
WaterrrForeverandClaude Opus 4.8
d0f0ec29e7
* feat(hyperframes-creative): add frame-preset library Add a library of ready-made visual frame presets (claude, biennale-yellow, blockframe, blue-professional, bold-poster, broadside, capsule, cartesian, cobalt-grid, coral, creative-mode, daisy-days, editorial-forest, …), each with a FRAME.md spec, a frame-showcase.html, and a per-preset caption-skin.html. Registered in the creative design-spec so workflows can remix a preset onto brand tokens. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hyperframes-media): shared TTS/BGM/SFX audio engine Add a shared audio engine under hyperframes-media (scripts/audio.mjs + lib/ tts.mjs, bgm.mjs, sfx.mjs, heygen.mjs) plus a bundled SFX pack and manifest. Workflows resolve this engine by path (../../hyperframes-media/scripts/ audio.mjs) for text-to-speech, background music, and sound effects, so audio is authored once and reused across skills instead of duplicated per workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(skills): gate render on user review; refresh router, core, general-video - hyperframes-cli: render is now user-gated — preview opens Studio (the timeline editor where the user can hand-edit anything, not just watch); never auto-render once checks pass, pause at preview and render only after approval. - hyperframes (router): tighten the entry SKILL.md description + routing. - hyperframes-core: rewrite SKILL.md and add script-format.md + storyboard-format.md references for the script-driven authoring architecture. - general-video: tidy the fallback-workflow description and routing table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(hyperframes-creative): reformat frame-preset showcase HTML Run the HTML formatter over the frame-showcase.html files (indentation, self-closing void tags, one CSS declaration per line). Formatting only — no content or markup changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hyperframes-media): correct wait-bgm field mapping and guard credential parse Two correctness fixes from review (#1632): - wait-bgm.mjs read audioMeta.bgm_path / audioMeta.bgm_enabled, but audio.mjs writes the path nested as bgm.path and the flag as bgm_pending. The detached generate path (Lyria/MusicGen) therefore always saw an empty path and exited status: disabled, silently dropping the music track even while generation was running. Read audioMeta.bgm?.path and gate on bgm_pending. - heygenCredential() had an unguarded JSON.parse despite documenting that it never throws — a malformed ~/.heygen credentials file crashed the engine at startup instead of degrading to no-credential. Wrap the parse and return null. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(hyperframes): add router tag to entry skill metadata Fold the router metadata tag into the foundation rewrite of the entry SKILL.md. This file is owned by this PR (the full router rewrite); keeping the tag tweak here — instead of a separate edit on the pre-rewrite version in another PR — avoids a guaranteed merge conflict between the two. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
82 lines
8.8 KiB
Markdown
82 lines
8.8 KiB
Markdown
---
|
|
name: hyperframes-media
|
|
description: Audio and media assets for HyperFrames compositions, produced by one shared audio engine (`scripts/audio.mjs`) — multi-provider TTS (HeyGen / ElevenLabs / Kokoro local), background music + sound effects (HeyGen audio-library retrieval by default, with local Lyria / MusicGen BGM generation and a bundled SFX library as the no-credential fallback), Whisper transcription, background removal, and caption authoring. Use for voiceover / TTS, BGM, SFX / sound effects, transcription, captions / subtitles / lyrics / karaoke / per-word styling, voice + provider selection, and music-mood prompting.
|
|
---
|
|
|
|
# HyperFrames Media
|
|
|
|
Create the audio and media assets a composition needs — voiceover (TTS), background music + sound effects, transcription, captions, background removal — then consume and animate that data in HTML. For placing assets into compositions, see `hyperframes-core`.
|
|
|
|
## The audio engine — one source for TTS · BGM · SFX
|
|
|
|
Workflows do NOT hand-roll audio or vendor a copy. There is one engine — **`scripts/audio.mjs`** — that takes a neutral `audio_request.json` and writes `audio_meta.json` (plus assets under `assets/voice|bgm|sfx`):
|
|
|
|
```bash
|
|
# <MEDIA_DIR> = this skill's directory
|
|
node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json
|
|
```
|
|
|
|
All three capabilities degrade on **ONE switch** — whether a HeyGen credential is present (resolved from `$HEYGEN_API_KEY` / `$HYPERFRAMES_API_KEY` / `~/.heygen`, **not** the CLI):
|
|
|
|
| Capability | HeyGen credential present | absent |
|
|
| ---------- | -------------------------------------------------- | ---------------------------------------------------- |
|
|
| TTS | HeyGen Starfish REST (native word timestamps) | → ElevenLabs → Kokoro (chain `transcribe` for words) |
|
|
| BGM | HeyGen music **retrieval** | Lyria → MusicGen local **generation** (detached) |
|
|
| SFX | HeyGen sound-effects **retrieval** (min_score 0.4) | bundled 21-file library (`assets/sfx/`) |
|
|
|
|
- **Request** (`audio_request.json`): `{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`. `id` joins each line back to the caller's model (a frame number, a scene id, …). `bgm.mode` = `retrieve | generate | none`; omit for auto (retrieve when credentialed, else generate). An **explicit** `retrieve` is strict — it skips rather than starting a detached generate (for callers with no `wait-bgm` step).
|
|
- **Output** (`audio_meta.json`, id-keyed): `{ tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }`.
|
|
- `--only tts,bgm,sfx` runs a subset and **merges** into an existing `--out` (e.g. TTS+BGM early, SFX once cues exist).
|
|
- BGM generate is spawned **detached** (`bgm_pending: true`) — run `scripts/wait-bgm.mjs` before assembling.
|
|
- `scripts/heygen-tts.mjs` is a single-shot CLI over the same code (one text → wav + words) for when you just need HeyGen TTS without a request file.
|
|
|
|
Full flag list + the `audio_meta.json` schema live in the header of `scripts/audio.mjs`. The references below cover the provider details and edge cases behind each capability.
|
|
|
|
## Provider chains (the detail behind the engine)
|
|
|
|
**TTS** — first available provider wins (the engine, or `npx hyperframes tts "..."`):
|
|
|
|
| Order | Provider | Detected when | Word timestamps |
|
|
| ----- | ----------------------------- | -------------------------------------------- | ---------------------------------------------------------------- |
|
|
| 1 | HeyGen (Starfish) | `$HEYGEN_API_KEY` / `hyperframes auth login` | **Yes, native** — pass `--words narration.words.json` to capture |
|
|
| 2 | ElevenLabs | `$ELEVENLABS_API_KEY` set | No — chain `transcribe` after |
|
|
| 3 | Kokoro-82M (local, 54 voices) | always (no key required) | No — chain `transcribe` after |
|
|
|
|
> The published `hyperframes tts` CLI is often the local-only build (its `--help` says "Kokoro-82M", no `--provider`/`--words`) and silently falls back to Kokoro even with `$HEYGEN_API_KEY` set. That is why the engine's HeyGen path is the self-contained `scripts/heygen-tts.mjs` (REST), NOT the CLI; the CLI is used only for the Kokoro path. See `references/tts.md`.
|
|
|
|
**BGM & SFX** — by default **retrieved** from the HeyGen audio library (`/v3/audio/sounds`), same credential as HeyGen TTS, with the no-credential fallback from the switch above:
|
|
|
|
| Asset | HeyGen `type` | Lands in | Fallback (no credential) |
|
|
| ----- | ------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------- |
|
|
| BGM | `music` | `assets/bgm/track.mp3` (retrieve) · `track.wav` (generate) | Lyria / MusicGen generation |
|
|
| SFX | `sound_effects` (min_score 0.4) | `assets/sfx/<slug>.mp3` | bundled 21-file library (`assets/sfx/*` + `manifest.json`) |
|
|
|
|
See `references/bgm.md` and `references/sfx.md`.
|
|
|
|
## Routing
|
|
|
|
| Task | Read |
|
|
| ------------------------------------------------------------------- | -------------------------------------------- |
|
|
| The audio engine — request/meta schema, `--only`, the switch | `scripts/audio.mjs` (header comment) |
|
|
| `npx hyperframes tts` / `heygen-tts.mjs` — providers, voices, words | `references/tts.md` |
|
|
| BGM — HeyGen retrieval + local Lyria / MusicGen generation | `references/bgm.md` |
|
|
| SFX — HeyGen retrieval (min_score 0.4) + bundled local library | `references/sfx.md` |
|
|
| `npx hyperframes transcribe` — Whisper, model rules, output shape | `references/transcribe.md` |
|
|
| `npx hyperframes remove-background` — transparent cutouts | `references/remove-background.md` |
|
|
| TTS → transcription → captions (no recorded voiceover) | `references/tts-to-captions.md` |
|
|
| Caption authoring — style detection, layout, word grouping, exit | `references/captions/authoring.md` |
|
|
| Transcript handling — input formats, quality gates, cleanup, APIs | `references/captions/transcript-handling.md` |
|
|
| Caption motion — karaoke, marker effects, audio-reactive | `references/captions/motion.md` |
|
|
| Model caches, system dependencies, troubleshooting | `references/requirements.md` |
|
|
|
|
## Non-negotiable rules
|
|
|
|
- **One engine, no vendored copies.** Produce audio via `scripts/audio.mjs` (or `heygen-tts.mjs` for one-shot HeyGen TTS). Don't re-implement TTS/BGM/SFX inside a workflow — write an `audio_request.json` adapter and call the engine.
|
|
- **"HeyGen available" = a resolvable credential, not the CLI.** The whole switch keys off `heygenCredential()`; the published `hyperframes tts` may be Kokoro-only, and there is no `hyperframes bgm` / `hyperframes sfx` command at all.
|
|
- **Voice IDs are provider-specific.** `am_michael` is Kokoro-only; HeyGen UUIDs don't work on Kokoro. If you pass `--voice`, also pin `--provider` to avoid silent provider drift when the user's env changes.
|
|
- **Always pass `--model` to `transcribe`.** The CLI default `small.en` silently translates non-English audio. See `references/transcribe.md` → "Language Rule".
|
|
- **HeyGen returns word timestamps; ElevenLabs / Kokoro do not.** The engine chains `transcribe` automatically for the latter two; standalone, pass `--words` to HeyGen or run `transcribe` against the audio file.
|
|
- **Captions consume the flat word-array format** with `{ id, text, start, end }`. See `references/transcribe.md` → "Output Shape".
|
|
- **`remove-background --background-output` is hole-cut, not inpainted.** For "scene without the person", a different tool is needed. See `references/remove-background.md` → "When NOT the right tool".
|
|
- **BGM/SFX default to HeyGen retrieval; the no-credential fallback is generation (BGM) or the bundled library (SFX).** `/audio/sounds` ranks by a text query — name effects concretely (`glass shatter`, not `dramatic sound`); a no-match **skips**, never blocks the render. SFX sit at volume ~0.35 under voice + BGM. See `references/sfx.md` / `references/bgm.md`.
|