mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-07 01:56:04 +00:00
WaterrrForever
4d3cdc3e4b
* feat(media-use): resolve official brand logos via a four-tier cascade Third-party brand logos (the meeting's 'credibility signals lost' gap) had no acquisition path: capture only grabs the product's own site assets, and HeyGen asset search returns generic look-alike icons for brand queries (0/3 in testing — an X-in-a-circle for LinkedIn). Workers could only fake a mark or drop it. New resolve type 'logo', four tiers verified by a 54-brand stress test (100% cascade hit across dev tools / big tech / non-tech / CN brands): - svgl — official full-color vector SVGs + wordmark variants (40/54 first-hits); search is substring-based, so entities pass through alias normalization (nextjs → 'next.js', aws → 'amazon web services') - simple-icons (pinned CDN build) — official monochrome glyphs; catches the long tail (nike, visa, toyota, wechat, bytedance) - github org avatar — known-org map only; a brand name is not a GitHub login, guessing risks same-named personal accounts - domain favicon (DuckDuckGo ip3) — small-raster last resort; sub-500B responses are DDG's placeholder and rejected; frozen with a low_res provenance flag (chip-size use only) logo joins the icon/image equivalence group (typesMatch) and the images/ subdir, so entity cache hits interop with figma-imported marks. A total miss falls through resolve's normal failure path — no special casing. HeyGen search stays the icon provider; it is deliberately absent from the logo cascade. Docs: media-use gap/types/providers tables + example; the five workflow banners now cover logos (catalog claim kept for media, 'from their official sources' added for logos); product-launch story-design and motion-graphics logo-reveal point at the new type; catalog surfaces (CLAUDE.md / README / docs) updated in lockstep. Verified: 19 unit tests + coverage row green; live smoke across all four tiers (linkedin→svgl, nike→simple-icons, heygen→github.avatar, amazon→favicon) plus a fabricated brand exiting 1 on the default miss path. oxlint + oxfmt clean. * test(media-use): sanction the four logo providers in the registry allowlist svgl / simple-icons / github.avatar / favicon.ddg join the sanctioned list — the logo cascade added in the previous commit. Full lib suite 95/95 green. * test(media-use): gate the logo cascade behavior in CI + single-fetch favicon tier Review follow-ups (miga-heygen, jrusso1020 on #2061): - Eight mocked-network tests pin what the manual 54-brand stress test only asserted: descriptor shape, alias retry (svgl non-array payload → next query, simple-icons 404 → next slug), network-error → null fallthrough, the sub-500B placeholder rejection, github's no-guessing (zero fetches for unmapped entities), and the real cascade order landing tier by tier under a mocked network. - faviconSearch now hands its verified bytes over as a local file, so the freeze step copies instead of re-downloading — one round-trip, and the size check is authoritative over what gets frozen. - The header's hit counts are labeled as a stress-test snapshot, not a live invariant. Full lib suite 103/103; live smoke re-verified (amazon → favicon.ddg, frozen .ico).
259 lines
19 KiB
Markdown
259 lines
19 KiB
Markdown
---
|
||
name: media-use
|
||
description: Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, or voice into a frozen local file + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Keeps search noise on disk, hands the agent a path. Use for any audio, image, icon, voiceover, caption, or media-asset need.
|
||
---
|
||
|
||
# media-use
|
||
|
||
The media OS for HyperFrames: resolve · generate · operate · remember, every media type, one skill, zero context noise.
|
||
|
||
## What it owns (the gaps HyperFrames leaves)
|
||
|
||
HyperFrames owns media _playback_; media-use owns everything else. Each row is enforced by `scripts/lib/coverage.test.mjs` so the claim can't rot.
|
||
|
||
| HyperFrames gap | media-use owns it via |
|
||
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| Audio-only, no image/icon | `resolve --type image\|icon` (heygen asset search) |
|
||
| No third-party brand logos | `resolve --type logo` (svgl → simple-icons → GitHub org avatar → domain favicon) |
|
||
| No voice / audio generation | `resolve --type voice` + the audio engine (`audio/scripts/audio.mjs`) |
|
||
| Scattered/duplicated audio engine | one consolidated engine under `audio/` (hyperframes-media retired) |
|
||
| No agent media-ops (cut/reframe/transform) | `references/operations.md` + `resolve --from` to register outputs |
|
||
| No transcript-driven cutting | `scripts/transcript-cut.mjs` compiles word-timestamp edits into cut lists |
|
||
| No auto-duck / publish loudness | `scripts/audio-duck.mjs` + `references/operations.md` loudnorm/sidechain recipes |
|
||
| No cross-project memory | global content-addressed cache + auto-promote (`~/.media`) |
|
||
| No image generation | RAM-graded local mflux (FLUX) via `scripts/lib/mflux-provider.mjs`, codex `image_gen` upsell (`scripts/lib/codex-provider.mjs`) |
|
||
| No video generation | spec-gated local LTX (`videogen` in `scripts/lib/local-models.mjs`); `heygen video create` avatar upsell |
|
||
| Weak local-model defaults | free-usage HeyGen first (TTS, bg-removal) via the `heygen` CLI; local open-source only as an opt-out fallback (`scripts/lib/local-run.mjs`) |
|
||
|
||
## When to use
|
||
|
||
Call `resolve` whenever a composition needs media: background music, sound effects, images, icons, brand logos, or voice. For voiceover / TTS, music, SFX, and caption timing, use the **audio engine** (below); background removal is delegated to the `hyperframes` CLI; transcription defaults to Parakeet (better than whisper.cpp: 6.05% vs 7.44% WER, 5-10x faster) via `scripts/transcribe.mjs`, with whisper.cpp auto-fallback (see `references/operations.md`). For cutting / reframing / transforming existing media, see `references/operations.md`. media-use searches the HeyGen catalog first, freezes the best match locally, registers it in a manifest, and hands the agent one line; all search noise stays on disk.
|
||
|
||
## Resolve
|
||
|
||
```bash
|
||
node <SKILL_DIR>/scripts/resolve.mjs --type <type> --intent "<description>" --project <dir>
|
||
```
|
||
|
||
Returns one line: `resolved <id> → <path> (<type>, <metadata>)`
|
||
|
||
### Types
|
||
|
||
| Type | What it finds | Provider |
|
||
| ------- | -------------------- | -------------------------------------------------------- |
|
||
| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) |
|
||
| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog |
|
||
| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) |
|
||
| `icon` | Icons, symbols | HeyGen asset search (type=icon) |
|
||
| `logo` | Official brand marks | svgl → simple-icons → GitHub org avatar → domain favicon |
|
||
| `voice` | TTS voiceover | Local Kokoro (free); HeyGen TTS upsell |
|
||
|
||
### Examples
|
||
|
||
```bash
|
||
# Background music
|
||
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --project .
|
||
# → resolved bgm_001 → .media/audio/bgm/bgm_001.mp3 (bgm, 25s)
|
||
|
||
# Sound effect
|
||
node <SKILL_DIR>/scripts/resolve.mjs --type sfx --intent "whoosh" --project .
|
||
# → resolved sfx_001 → .media/audio/sfx/sfx_001.mp3 (sfx, 0.57s)
|
||
|
||
# Image
|
||
node <SKILL_DIR>/scripts/resolve.mjs --type image --intent "gradient tech background" --project .
|
||
# → resolved image_001 → .media/images/image_001.jpg (image)
|
||
|
||
# Icon
|
||
node <SKILL_DIR>/scripts/resolve.mjs --type icon --intent "rocket" --project .
|
||
# → resolved icon_001 → .media/images/icon_001.png (icon, transparent)
|
||
|
||
# Brand logo (official mark — never redrawn by hand)
|
||
node <SKILL_DIR>/scripts/resolve.mjs --type logo --entity linkedin --intent "LinkedIn logo" --project .
|
||
# → resolved logo_001 → .media/images/logo_001.svg (logo, official mark)
|
||
```
|
||
|
||
### Flags
|
||
|
||
| Flag | Description |
|
||
| --------------- | ------------------------------------------------------------------------------------ |
|
||
| `--type, -t` | Media type: bgm, sfx, image, icon, voice |
|
||
| `--intent, -i` | What you need (natural language) |
|
||
| `--entity, -e` | Entity name for cache matching (optional) |
|
||
| `--project, -p` | Project directory (default: .) |
|
||
| `--candidates` | List reusable assets (project + global cache) for `--type`; no download, no mutation |
|
||
| `--reuse <sha>` | Import a specific global-cache asset (by content sha/prefix, from `--candidates`) |
|
||
| `--from` | Freeze a local file or direct public URL (ingest) |
|
||
| `--local-only` | Offline: skip every network provider (cache + local only) |
|
||
| `--provider` | Force one generator (e.g. `codex`, `mflux`, `kokoro`, `heygen`) |
|
||
| `--adopt` | Bulk-import existing assets/ into manifest |
|
||
| `--json` | Output JSON instead of one-line result |
|
||
|
||
## Reuse before you resolve
|
||
|
||
Before resolving bgm/sfx/image/icon, **check what already exists and reuse it when it fits.** media-use does not semantically match for you — you are the judge. It surfaces candidates; you decide.
|
||
|
||
```bash
|
||
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --candidates --project .
|
||
# [project] upbeat tech launch (25s, heygen.audio.sounds)
|
||
# .media/audio/bgm/bgm_001.wav
|
||
# [global] energetic tech intro (22s, heygen.audio.sounds)
|
||
# --reuse 06e052c075fd2b80
|
||
```
|
||
|
||
Read the list and judge semantic fit yourself — "upbeat tech launch" ≈ "energetic tech intro" is a call only you can make from the descriptions. Then:
|
||
|
||
- **A project candidate fits** → just reference its path in your composition. Nothing else to run.
|
||
- **A global candidate fits** → `resolve --type bgm --reuse <sha>` copies it into this project (self-contained render) and records it.
|
||
- **Nothing fits** → resolve fresh (`--type ... --intent ...`).
|
||
|
||
**Trust guardrail — when unsure, resolve fresh.** A redundant download is cheap; shipping the wrong asset is not. Judge fit from description + prompt + type + duration/dims. For **brand/entity** assets, reuse a _global_ candidate only when the entity matches exactly — the global cache aggregates every project you have worked on, so a `--candidates` list can surface another client's brand mark and its prompt text. Never reuse a cross-project brand asset on a loose match.
|
||
|
||
The deterministic floor still runs automatically: an identical (case/whitespace-insensitive) repeat auto-reuses with no `--candidates` step. `--candidates` is only for the semantic layer above that floor — and a fuzzy match is **never** auto-applied; reuse is always your explicit call. On a resolve that misses the floor and is about to fetch, media-use prints a one-line stderr hint when similar cached assets exist, pointing you back here.
|
||
|
||
## Providers
|
||
|
||
media-use holds no keys; every external tool owns its auth. Generation is
|
||
local-first with a cloud upsell where one helps. `resolve` spec-checks
|
||
AVAILABLE RAM and auto-picks the best local model that fits (a RAM-graded
|
||
ladder, `describeModelLadder`); the agent can see the ladder and override.
|
||
|
||
| Type | Provider (in order) |
|
||
| ------------- | --------------------------------------------------------------------------------------- |
|
||
| bgm/sfx | heygen catalog (free) |
|
||
| image | heygen search, then local mflux (best FLUX for your RAM), then codex `image_gen` upsell |
|
||
| voice | local **Kokoro** (free, on-device), then **heygen tts** paid upsell |
|
||
| icon | heygen asset search |
|
||
| logo | svgl, then simple-icons, then GitHub org avatar, then domain favicon (all free) |
|
||
| video (local) | local LTX (`videogen` ladder); `heygen video create` avatar upsell |
|
||
|
||
Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free,
|
||
private, offline once cached). Paid/cloud upsells sit behind them: HeyGen TTS
|
||
for voice, the `codex` CLI (ChatGPT sub) for a better image, the `heygen` CLI
|
||
for avatar video. Cost rule (X4): the agent confirms before an agent-initiated
|
||
paid call; a user-requested one just runs.
|
||
|
||
To force a specific generator (e.g. a user says "make this image with codex"),
|
||
pass `--provider codex`: it pins resolution to that provider and skips the
|
||
free-first default. See `references/operations.md` for the RAM ladders and
|
||
upsell recipes.
|
||
|
||
`--local-only` skips every network provider, including the free HeyGen ones,
|
||
leaving the project + global cache and any local provider.
|
||
|
||
## How it works
|
||
|
||
`resolve` runs an automatic floor, then falls through to fetching:
|
||
|
||
1. Check project `.media/manifest.jsonl` for a prompt match (case- and whitespace-insensitive) — auto-reuse
|
||
2. Scan existing `assets/` directory for unregistered files that share a word with the need
|
||
3. Check global cache `~/.media/` for a reusable asset matched on the same normalized prompt — auto-reuse
|
||
4. Search via provider (HeyGen audio catalog, HeyGen asset search), then generate
|
||
5. Freeze file to `.media/<type>/`, register in manifest, regenerate `index.md`, auto-promote to `~/.media/`
|
||
|
||
Steps 1 and 3 are the **deterministic floor**: they only auto-reuse an exact-normalized match, never a fuzzy one. Semantic reuse ("close enough") is the agent's explicit call via [Reuse before you resolve](#reuse-before-you-resolve) — it never happens automatically. The agent gets back **one line**; candidates, scores, provenance stay on disk.
|
||
|
||
## Adopt existing projects
|
||
|
||
Most HyperFrames projects already have assets in `assets/`. media-use adopts them:
|
||
|
||
```bash
|
||
node <SKILL_DIR>/scripts/resolve.mjs --adopt --project .
|
||
# → adopted 9 assets from assets/
|
||
# bgm_001 → assets/bgm/mango-fizz.mp3 (bgm, 146.6s)
|
||
# image_001 → assets/images/avatar.jpg (image, 400×400)
|
||
```
|
||
|
||
`ffprobe` extracts real duration and dimensions. During resolve, unregistered files in `assets/` matching the intent are adopted on the fly.
|
||
|
||
## Reading the inventory
|
||
|
||
After resolve or adopt, read `.media/index.md` for the full inventory:
|
||
|
||
```
|
||
# .media · 4 assets
|
||
|
||
id type dur dims path description
|
||
bgm_001 bgm 25s - .media/audio/bgm/bgm_001.mp3 upbeat tech launch
|
||
sfx_001 sfx 0.6s - .media/audio/sfx/sfx_001.mp3 whoosh
|
||
image_001 image - 1920×1080 .media/images/image_001.jpg gradient tech background
|
||
icon_001 icon - 200×200 .media/images/icon_001.png rocket
|
||
```
|
||
|
||
## Cross-project reuse
|
||
|
||
Assets are cached automatically on resolve. Every resolved/ingested asset is auto-promoted to the global cache at `~/.media/`, so subsequent resolves for the same (or near-identical) prompt, in any project, hit the cache with no re-download and no provider call.
|
||
|
||
For a _semantically_ similar (not identical) need in another project, the exact-match floor won't fire — use [Reuse before you resolve](#reuse-before-you-resolve): `--candidates` lists the global assets, and `--reuse <sha>` imports the one you pick. This is how a track resolved in one project gets reused in the next when the wording differs.
|
||
|
||
## Files
|
||
|
||
- `.media/manifest.jsonl`: machine SSOT, one JSON record per line
|
||
- `.media/index.md`: agent-readable table (id, type, dur, dims, path, description)
|
||
- `~/.media/`: global cross-project reuse cache (content-addressed, SHA-256)
|
||
|
||
## Audio engine: voiceover, music, SFX, captions, transcription
|
||
|
||
For a full audio pass (TTS voiceover + background music + sound effects in one
|
||
shot), use the shared engine at `audio/scripts/audio.mjs`. It takes a neutral
|
||
`audio_request.json` and writes `audio_meta.json` plus assets under
|
||
`.media/audio/{voice,bgm,sfx}`:
|
||
|
||
```bash
|
||
node <SKILL_DIR>/audio/scripts/audio.mjs --request ./audio_request.json --out ./audio_meta.json
|
||
```
|
||
|
||
- **Request** `{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`: `id` joins each line back to your model; `bgm.mode` = `retrieve | generate | none` (omit for auto). `--only tts,bgm,sfx` runs a subset and merges into an existing `--out`.
|
||
- **Output** `audio_meta.json` (id-keyed): `voices[].{path,duration_s,words[]}` (word timestamps for captions), `sfx[]`, `bgm`, `total_duration_s`.
|
||
- **Auto-degrades on one switch**: HeyGen credential present → HeyGen TTS + music/SFX retrieval; absent → ElevenLabs/Kokoro TTS, Lyria/MusicGen BGM generation, and the bundled SFX library (no credential needed).
|
||
- If BGM took the generate path (`bgm_pending: true`), run `audio/scripts/wait-bgm.mjs` before final render.
|
||
|
||
Single-shot helpers: `audio/scripts/heygen-tts.mjs` (one voice file). Transcription / background removal / captions use the `hyperframes` CLI (`transcribe`, `remove-background`), see the per-topic guides in `audio/references/` (`tts.md`, `bgm.md`, `sfx.md`, `transcribe.md`, `remove-background.md`, `captions/`).
|
||
|
||
## Operating on media (cut, reframe, transform)
|
||
|
||
media-use resolves + remembers; for **operating** on assets see
|
||
`references/operations.md`: local-tool recipes (ffmpeg trim/reframe/montage,
|
||
auto-editor, scenedetect) and the local-vs-HeyGen transform table (background
|
||
removal, upscale, lipsync, translate). Run the tool, then register the output
|
||
with `resolve --from <output> --type <type>` so it joins the ledger + global
|
||
cache.
|
||
|
||
## CLI tools used (what to run, and how to enable each)
|
||
|
||
`resolve` auto-cascades; each provider shells one CLI. Local tools are OPT-IN:
|
||
if a local tool is absent, resolve degrades gracefully to the free/cloud path,
|
||
so nothing here is strictly required except `ffmpeg`/`ffprobe`. Install a local
|
||
tool to unlock its free, private, on-device path. media-use holds no keys.
|
||
|
||
| Tool | Serves | Install |
|
||
| ------------------ | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
|
||
| `ffmpeg`/`ffprobe` | adopt probing, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
|
||
| `heygen` | catalog (bgm/sfx/image/icon), TTS + avatar upsell | `curl -fsSL https://static.heygen.ai/cli/install.sh \| bash` then `heygen auth login --key <key>` (needs >= v0.1.6) |
|
||
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
|
||
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
|
||
| `parakeet-mlx` | local transcription (default ASR, best) | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
|
||
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
|
||
| `npx hyperframes` | Kokoro TTS (voice), whisper.cpp (transcribe fallback), remove-background | bundled with the hyperframes CLI |
|
||
|
||
The RAM-graded local-model shortlist + exact per-tier install/invoke lives in
|
||
`scripts/lib/local-models.mjs` (the agent can read `describeModelLadder(cap, specs)`
|
||
to see which model fits this machine). Without a tool on PATH, its provider
|
||
prints a one-line diagnostic to stderr and resolve falls through to the next
|
||
provider (e.g. no `mflux` -> codex image upsell; no `parakeet-mlx` -> whisper.cpp).
|
||
|
||
`heygen asset search` is a pre-launch command hidden from `heygen --help`, but it
|
||
runs; providers tag requests with the allowlisted `X-HeyGen-Client-Source` header
|
||
(v0.1.6+).
|
||
|
||
## Telemetry
|
||
|
||
`resolve` and the edit tools (transcribe / transcript-cut / audio-duck) send an
|
||
anonymous usage event to PostHog (`scripts/lib/telemetry.mjs`), so we can see
|
||
which capabilities are actually used. It records only the media TYPE, the
|
||
resolution SOURCE, and the winning PROVIDER: never the intent text, file names,
|
||
or paths, and `$ip:null` so no IP is stored. Best-effort and non-blocking (a
|
||
resolve never waits on or fails from telemetry).
|
||
|
||
Opt out with `DO_NOT_TRACK=1` or `HYPERFRAMES_NO_TELEMETRY=1` (also off in CI and
|
||
dev). Same public PostHog project key and opt-outs as the `hyperframes` CLI.
|