ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-13 07:40:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(media-use): add video generation (HeyGen avatar-video + local LTX fallback) (#2614)

Browse Source 
* fix(media-use): tag HeyGen TTS generation with attribution header

Centralizes the X-HeyGen-Client-Source header into HEYGEN_CLIENT_SOURCE_ARGV
in heygen-cli.mjs and reuses it in heygen-search.mjs (dropping the duplicated
inline literal) so voice-provider's `voice speech create` call carries it too.
The generation call was previously untagged, making media-use TTS usage
invisible in HeyGen's billing/analytics warehouse; the read-only `voice list`
discovery call intentionally stays untagged.

* feat(media-use): add local LTX video generate provider

* feat(media-use): add HeyGen avatar-video generate provider

* feat(media-use): register video as a real provider type

* docs(media-use): document the wired video type and full HeyGen tagging coverage

resolve --type video is now the default path (HeyGen avatar video first,
local LTX fallback, sign-in nudge on auth failure) instead of a manual
recipe; correct the claim that only search requests are tagged now that
TTS and avatar-video generation carry the attribution header too.

* fix(media-use): wire --avatar-id/--voice-id CLI flags and close video-provider auth/cache gaps

- resolve.mjs never implemented the --avatar-id/--voice-id override that
  operations.md documented, so following the docs crashed with
  ERR_PARSE_ARGS_UNKNOWN_OPTION; wire the flags through to ctx.
- defaultAvatarId/defaultStarfishVoiceId cached a failed discovery lookup
  as a permanent null, disabling heygen.video after one transient miss;
  cache only a truthy id, matching the same fix in voice-provider.mjs's
  defaultVoiceId.
- the avatar-video onboarding nudge only fired on a video-create failure,
  never when avatar/voice discovery itself was unauthenticated (the
  common unauthenticated case) -- propagate the discovery failure reason
  so onboarding fires either way.
- dedupe the CLI-shelling JSON helper (heygen-cli.mjs's new runHeygenJson)
  and the local-model argv-template builder (local-models.mjs's new
  buildArgv) instead of leaving byte-identical copies in each provider.

* fix(media-use): address avatar-video PR review feedback

- heygenVideoGenerate short-circuits after the first discovery-call
  failure instead of always attempting both avatar list and voice list,
  so an unauthenticated caller gets one onboarding message and one
  provider-error telemetry ping instead of a double-fire.
- runHeygenJson logs a diagnostic when a CLI call succeeds but returns
  unparseable JSON, instead of silently returning null.
- dedupe the "avatar video is free" onboarding string into one constant
  (was duplicated across three call sites).

* fix(media-use): match review-requested naming and message conventions

- export AVATAR_VIDEO_SIGNIN_MESSAGE from heygen-video-provider.mjs so
  the test imports the canonical string instead of redeclaring it.
- runHeygenJson's non-JSON diagnostic now matches heygen-search.mjs's
  existing wording ("returned non-JSON output").
This commit is contained in:
Miguel Ángel
2026-07-17 03:56:57 -04:00
committed by GitHub
parent f084a7217d
commit 0a66671fc5
18 changed files with 1021 additions and 122 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/media-use/SKILL.md
+31 -26
View File
@@ -25,20 +25,20 @@ node <SKILL_DIR>/scripts/resolve.mjs --doctor
HyperFrames owns media _playback_; media-use owns everything else. Each row is enforced by `scripts/lib/coverage.test.mjs` so the claim can't rot.
| HyperFrames gap | media-use owns it via |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Audio-only, no image/icon | `resolve --type image\|icon` (heygen asset search) |
| No third-party brand logos | `resolve --type logo` (svgl → simple-icons → GitHub org avatar → domain favicon) |
| No voice / audio generation | `resolve --type voice` (HeyGen TTS free-usage path; optional local Kokoro) + the audio engine (`audio/scripts/audio.mjs`) |
| Scattered/duplicated audio engine | one consolidated engine under `audio/` (hyperframes-media retired) |
| No agent media-ops (cut/reframe/transform) | `references/operations.md` + `resolve --from` to register outputs |
| No transcript-driven cutting | `scripts/transcript-cut.mjs` compiles word-timestamp edits into cut lists |
| No auto-duck / publish loudness | `scripts/audio-duck.mjs` + `references/operations.md` loudnorm/sidechain recipes |
| No cross-project memory | global content-addressed cache + auto-promote (`~/.media`) |
| No color-grade authoring | `resolve --type grade` emits a paste-ready `data-color-grading` block; `resolve --type lut` freezes validated `.cube` files |
| No image generation | RAM-graded local mflux (FLUX) via `scripts/lib/mflux-provider.mjs`, codex `image_gen` upsell (`scripts/lib/codex-provider.mjs`) |
| No video generation | HeyGen avatar video + image-to-video (animate any still into a talking clip), photo-avatar, dub/translate — `heygen` CLI free-usage path (`references/operations.md`); optional spec-gated local LTX (`videogen` in `scripts/lib/local-models.mjs`) |
| Weak local-model defaults | HeyGen free-usage path via the `heygen` CLI; local open-source tools only as opt-in alternatives (`scripts/lib/local-run.mjs`) |
| HyperFrames gap | media-use owns it via |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Audio-only, no image/icon | `resolve --type image\|icon` (heygen asset search) |
| No third-party brand logos | `resolve --type logo` (svgl → simple-icons → GitHub org avatar → domain favicon) |
| No voice / audio generation | `resolve --type voice` (HeyGen TTS free-usage path; optional local Kokoro) + the audio engine (`audio/scripts/audio.mjs`) |
| Scattered/duplicated audio engine | one consolidated engine under `audio/` (hyperframes-media retired) |
| No agent media-ops (cut/reframe/transform) | `references/operations.md` + `resolve --from` to register outputs |
| No transcript-driven cutting | `scripts/transcript-cut.mjs` compiles word-timestamp edits into cut lists |
| No auto-duck / publish loudness | `scripts/audio-duck.mjs` + `references/operations.md` loudnorm/sidechain recipes |
| No cross-project memory | global content-addressed cache + auto-promote (`~/.media`) |
| No color-grade authoring | `resolve --type grade` emits a paste-ready `data-color-grading` block; `resolve --type lut` freezes validated `.cube` files |
| No image generation | RAM-graded local mflux (FLUX) via `scripts/lib/mflux-provider.mjs`, codex `image_gen` upsell (`scripts/lib/codex-provider.mjs`) |
| No video generation | `resolve --type video` — HeyGen avatar video first (free-usage path, sign-in nudge on auth failure), local LTX fallback (`videogen` in `scripts/lib/local-models.mjs`); image-to-video, photo-avatar, dub/translate remain manual `heygen` CLI recipes (`references/operations.md`) |
| Weak local-model defaults | HeyGen free-usage path via the `heygen` CLI; local open-source tools only as opt-in alternatives (`scripts/lib/local-run.mjs`) |
## When to use
@@ -249,20 +249,21 @@ transcription, and LTX for local video generation. `resolve` spec-checks
AVAILABLE RAM for those local ladders (`describeModelLadder`); the agent can
see the ladder and override.
| Type | Provider / path |
| --------- | ----------------------------------------------------------------------------------------------------------- |
| bgm/sfx | heygen catalog free-usage path |
| image | heygen search free-usage path; optional local mflux; codex `image_gen` upsell |
| voice | heygen tts free-usage path; optional local **Kokoro** (free, on-device) |
| icon | heygen asset search free-usage path |
| logo | svgl, then simple-icons, then GitHub org avatar, then domain favicon (all free) |
| grade/lut | local core-preset map, params/CDN look index, deterministic `buildCube` fallback |
| video | heygen avatar / image-to-video / photo-avatar / dub free-usage path; optional local LTX (`videogen` ladder) |
| Type | Provider / path |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| bgm/sfx | heygen catalog free-usage path |
| image | heygen search free-usage path; optional local mflux; codex `image_gen` upsell |
| voice | heygen tts free-usage path; optional local **Kokoro** (free, on-device) |
| icon | heygen asset search free-usage path |
| logo | svgl, then simple-icons, then GitHub org avatar, then domain favicon (all free) |
| grade/lut | local core-preset map, params/CDN look index, deterministic `buildCube` fallback |
| video | heygen avatar video free-usage path (sign-in nudge on auth failure); optional local LTX (`videogen` ladder). Image-to-video / photo-avatar / dub stay manual `heygen` recipes |
Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free,
private, offline once cached). The `codex` CLI remains the ChatGPT-sub image
upsell. Cost rule (X4): the agent confirms before an agent-initiated paid call;
a user-requested one just runs.
a user-requested one just runs — `heygen.video` is flagged paid (metered free
allowance) so an agent-initiated `resolve --type video` confirms first.
To force a specific generator (e.g. a user says "make this image with codex"),
pass `--provider codex`: it pins resolution to that provider and skips the
@@ -425,8 +426,12 @@ prints a one-line diagnostic to stderr and resolve falls through where another
provider exists (e.g. no `mflux` -> codex image upsell; no `parakeet-mlx` -> whisper.cpp).
`heygen asset search` is a pre-launch command hidden from `heygen --help`, but it
runs; providers tag requests with the allowlisted `X-HeyGen-Client-Source` header
(v0.3.0+).
runs. Every media-use call that shells `heygen` — catalog search AND every
generating call (TTS, avatar video) — tags requests with the allowlisted
`X-HeyGen-Client-Source: media-use` header (v0.3.0+), sourced from one shared
constant (`HEYGEN_CLIENT_SOURCE_ARGV` in `scripts/lib/heygen-cli.mjs`) so a
future call site can't silently ship untagged. Read-only discovery calls
(`voice list`, `avatar list`) are intentionally left untagged.
## Telemetry