mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-04 04:46:59 +00:00
feat(skills): style templates on the HyperFrames plan (#6842)
* fix(cli,skills): say a step's credits are its total, and keep cost out of retries An out-of-credits refusal named what the step needs, and an agent read that as the shortfall. It now says the number is the step's total. After its last try, an agent told the person what was spent and what another try would cost. The HeyGen reference now says not to. * fix(cli): plan avatar credits from the HyperFrames price rows An avatar's credits came from its old API price, about five times what the plan charges, so --max-credits refused calls that fit and a top-up asked for too much. They now use the Avatar IV photo row plus the general avatar video row, 950 credits a minute. * fix(cli): pin the avatar price test and keep pricing under the comment share * fix(cli): plan avatar seconds on the Avatar IV row alone The general avatar row is only checked at admission and billed on seconds left after Avatar IV, so adding it to every second made a 15 s script plan at 238 instead of 200 and a 220 limit refuse a call that fits. * feat(skills): creator templates on the HyperFrames plan A Pro account asks for a video in a creator's format and the agent pulls that template on demand. The packs sit in the private bucket behind the plan-gated bundle route, so a Free account gets a 402 and the upgrade step. A pull costs no credits; making the film spends them as Reverse does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): make creator-remix the one template skill SKILL.md holds the catalog, the Pro openings, the pull and the credit rules. references/remix.md is how to make the episode, from Desktop's creator-remix with only what the app alone has swapped for the CLI. The packs now hold only the template, so their hashes change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): the creator-remix skill and its packs SKILL.md holds the catalog, the Pro openings, the pull and the credit rules. references/remix.md is how to make the episode, from Desktop's creator-remix with only what the app alone has swapped for the CLI. The packs now hold only the template, so their hashes change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(skills): word creator-remix's sign-in steps the way media-use does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): a creator-remix presenter runs on the plan by default The person's own HeyGen avatar is recommended, not required: without one, 小Lin's presenter is made with HeyGen Video 1 from their photo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): pin template packs whose voice bills the plan first Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cli): offload media named inside template tags References in a composition's <template> were never rewritten, because the DOM query stops at the tag and template.content does not serialize, so the file stayed in the zip. A stem matched as a plain word and a same-named file elsewhere counted as a mention too. A 5-minute presented film went from 196 MB to 71 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): lipsync, and avatars that speak a finished voice Video 1 takes audio as a reference and re-paces it, so a shot's lips follow its own track and drift up to a second from the voice. generate --type lipsync redoes the mouth to the exact voice. generate --type avatar takes --audio, and --avatar for the person's own look, so the lips land on the voice from one job. Also moves the plan rows to the 2026-10-30 price book, estimates a voice's credits so --max-credits can hold it, logs jobs from a folder with no project, and describes the resolve flags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): presenters go avatar first, and a voice can be stock The person's own avatar, then Avatar IV from their photo, then Video 1 with lipsync. With no clone or recording, a stock voice named in the credits. Pins the five rebuilt packs. The archive test keeps its ../ under GNU tar with -P. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cli): fix avatar price and sfx provider tests, trim comments A wav gives the avatar price test an exact length on every platform. The sfx tests now expect HeyGen first. Comment share is back under the base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(core): read a font shorthand whose size nests parentheses `font: calc(var(--size) * 1px)/1 "X"` cut at the first closing parenthesis, so a bad family name failed the render. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cli): one ledger row per job, and search skips a template's own pictures A job done at once logs one row, transcribe logs to the project, and resolve no longer offers a component's file for a search. --reuse and --provider get help text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): pin the tested template packs, and a board needs no voice Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(cli): read the transcribe ledger row as JSON, for Windows paths Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cli): pin the Big Questions pack without the film length check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): call them style templates, and the docs example pulls a real slug Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cli): pin the cleaned template packs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): drop the stock-presenter ban and the no-spend-before-storyboard rule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cli): pin the Paper Trail and Long Story Short packs without the stock-presenter ban Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): rename creator-remix to style-remix, and the five templates to names that say the look Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): style-remix lists the five templates when none is named Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cli): pin the Finance Explainer pack that varies its motion graphics Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): media-use generate --type cutout takes a shot out of its background on the plan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cli): pin the Finance Explainer pack without the provisional variety rule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cli): pin the five style packs with variety rules measured on real films Animated Science gets a worlds-and-boards budget, 3D True Crime a main board and a brief corner note, 3D Brand caps by measured rate, Finance and Collage set pieces and more than one look. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): media-use names the on-plan cutout and lipsync commands, not the heygen CLI ones Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cli): cutout retries dedupe on request_id, uploads carry their real type; style-remix is public The background-removal route dedupes on the body's request_id and ignores the Idempotency-Key header, so a retry after a lost answer started a second job: runJob now also sends the key in the body for such a route. Uploads typed every non-MP4 file as audio, so WebM and MOV inputs were refused: they now take their type from the extension. Fails without the fix: the new cutout test in generate.test.ts sees audio/mpeg for WebM and MOV. style-remix drops metadata.internal and is listed with the other published skills. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(cli): keep the comment share of the touched files at main's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: count style-remix in the README and CLAUDE.md skill lists Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(skills): rename style-remix to style-sampling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WaterrrForever
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
3f42dc1c73
commit
2daedcdafc
@@ -57,3 +57,4 @@ skills/figma/SKILL.md
|
||||
skills/slideshow/SKILL.md
|
||||
skills/remotion-to-hyperframes/SKILL.md
|
||||
skills/reverse-any-video/SKILL.md
|
||||
skills/style-sampling/SKILL.md
|
||||
|
||||
@@ -8,7 +8,7 @@ This repo ships AI agent skills via [vercel-labs/skills](https://github.com/verc
|
||||
|
||||
```bash
|
||||
npx hyperframes skills update # default: installs/refreshes the core set plus anything already installed
|
||||
npx hyperframes skills # all 8 published skills at once
|
||||
npx hyperframes skills # all 9 published skills at once
|
||||
npx skills add heygen-com/hyperframes # interactive picker (terminal only; repo-internal skills are excluded by default)
|
||||
```
|
||||
|
||||
@@ -22,6 +22,7 @@ Make any video by asking for it; the `/hyperframes` skill carries the contract a
|
||||
- `/slideshow` — a presentation, pitch deck or interactive deck. Output is a navigable deck, not a rendered MP4.
|
||||
- `/remotion-to-hyperframes` — bring existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, or renders from other tools) and verify it frame by frame against the original.
|
||||
- `/reverse-any-video` — rebuild any video file as an editable HyperFrames project (motion graphics, talking heads, live action or AI-generated footage) and check the render against the original frame by frame. A HyperFrames Pro feature.
|
||||
- `/style-sampling` — make a new episode on your topic in one of five proven styles (Collage Explainer, Finance Explainer, Animated Science Explainer, 3D Brand Explainer, 3D True Crime). A HyperFrames Pro feature.
|
||||
|
||||
## Issue and PR triage
|
||||
|
||||
|
||||
@@ -4,11 +4,11 @@ Open-source video rendering framework: write HTML, render video.
|
||||
|
||||
## Skills
|
||||
|
||||
This repo ships 8 AI agent skills via [vercel-labs/skills](https://github.com/vercel-labs/skills). Install them before writing compositions — they encode framework-specific patterns that generic docs don't cover. **Default to the core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`); `/slideshow`, `/remotion-to-hyperframes`, `/reverse-any-video` and `/figma` install on demand. Install all 8 only when the user explicitly asks for the full set.
|
||||
This repo ships 9 AI agent skills via [vercel-labs/skills](https://github.com/vercel-labs/skills). Install them before writing compositions — they encode framework-specific patterns that generic docs don't cover. **Default to the core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`); `/slideshow`, `/remotion-to-hyperframes`, `/reverse-any-video`, `/style-sampling` and `/figma` install on demand. Install all 9 only when the user explicitly asks for the full set.
|
||||
|
||||
```bash
|
||||
npx hyperframes skills update # default: installs/refreshes the core set plus anything already installed
|
||||
npx hyperframes skills # all 8 published skills at once — only on explicit request
|
||||
npx hyperframes skills # all 9 published skills at once — only on explicit request
|
||||
npx skills add heygen-com/hyperframes # interactive picker; internal skills require INSTALL_INTERNAL_SKILLS=1 for discovery
|
||||
npx skills add heygen-com/hyperframes --all # excludes internal skills unless INSTALL_INTERNAL_SKILLS=1
|
||||
npx skills add heygen-com/hyperframes --skill <name> # a named match installs regardless of metadata.internal (bare name, no leading slash)
|
||||
@@ -28,6 +28,7 @@ Make any video by asking for it; the `/hyperframes` skill carries the render con
|
||||
- `/slideshow` — a presentation, pitch deck or interactive deck with discrete slides, fragment reveals, branching, hotspot navigation and presenter mode with speaker notes; also converts an existing page into a deck. Output is a navigable deck, not a rendered MP4.
|
||||
- `/remotion-to-hyperframes` — bring existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, or renders from other tools) and verify it frame by frame against the original.
|
||||
- `/reverse-any-video` — rebuild any video file as an editable HyperFrames project (motion graphics, talking heads, live action or AI-generated footage) and check the render against the original frame by frame. A HyperFrames Pro feature.
|
||||
- `/style-sampling` — make a new episode on your topic in one of five proven styles (Collage Explainer, Finance Explainer, Animated Science Explainer, 3D Brand Explainer, 3D True Crime). A HyperFrames Pro feature.
|
||||
|
||||
## Skill catalog maintenance
|
||||
|
||||
|
||||
@@ -64,7 +64,7 @@ For standalone skills (including OpenCode), use:
|
||||
npx skills add heygen-com/hyperframes
|
||||
```
|
||||
|
||||
> The picker opens with nothing pre-selected — the **Core Skills** group is all you need; `/slideshow`, `/remotion-to-hyperframes`, `/reverse-any-video` and `/figma` install on demand. Agents and non-interactive runs should use `npx hyperframes skills update` instead — it installs exactly the core set, whereas `skills add --all` installs all 8 published skills. The six repo-internal skills under `.claude/skills` / `.agents/skills` are excluded by default. For the full published set use `npx hyperframes skills`.
|
||||
> The picker opens with nothing pre-selected — the **Core Skills** group is all you need; `/slideshow`, `/remotion-to-hyperframes`, `/reverse-any-video`, `/style-sampling` and `/figma` install on demand. Agents and non-interactive runs should use `npx hyperframes skills update` instead — it installs exactly the core set, whereas `skills add --all` installs all 9 published skills. The six repo-internal skills under `.claude/skills` / `.agents/skills` are excluded by default. For the full published set use `npx hyperframes skills`.
|
||||
>
|
||||
> `skills add` resolves the skills.sh registry blob, which can lag `main` by hours. `npx hyperframes skills update` installs from the current `main`, so reach for it when you need the newest copy of a skill.
|
||||
|
||||
@@ -81,9 +81,9 @@ The skills teach agents the HyperFrames production loop: plan the video, write v
|
||||
|
||||
## Skills
|
||||
|
||||
HyperFrames ships 8 skills. Make any video by asking for it; the `/hyperframes` skill carries the contract and the references (film, captions, overlays, motion, look). Read it first.
|
||||
HyperFrames ships 9 skills. Make any video by asking for it; the `/hyperframes` skill carries the contract and the references (film, captions, overlays, motion, look). Read it first.
|
||||
|
||||
For standalone installation, default to the **core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`). `npx hyperframes skills update` installs exactly that from anywhere; the interactive picker (`npx skills add heygen-com/hyperframes`) lists it as the "Core Skills" group, nothing pre-selected. The picker is interactive-only — a non-interactive or agent run without `--skill` installs all 8. Use `npx skills add heygen-com/hyperframes --all` to install the complete published set deliberately (skips the picker), or `npx skills add heygen-com/hyperframes --skill <name>` for just one (bare name, no leading `/`).
|
||||
For standalone installation, default to the **core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`). `npx hyperframes skills update` installs exactly that from anywhere; the interactive picker (`npx skills add heygen-com/hyperframes`) lists it as the "Core Skills" group, nothing pre-selected. The picker is interactive-only — a non-interactive or agent run without `--skill` installs all 9. Use `npx skills add heygen-com/hyperframes --all` to install the complete published set deliberately (skips the picker), or `npx skills add heygen-com/hyperframes --skill <name>` for just one (bare name, no leading `/`).
|
||||
|
||||
Standalone installs stay lean after that: `npx hyperframes init` installs the **core set** only when it is missing and never touches a skill you already have (`npx hyperframes skills update` refreshes them, and never removes one); `/figma`, `/slideshow`, `/remotion-to-hyperframes` and `/reverse-any-video` install **on demand** with `npx hyperframes skills update <name>`. Nothing re-pulls the full set behind your back.
|
||||
|
||||
@@ -120,6 +120,7 @@ This writes `dist/hyperframes-plugin.zip` with a `hyperframes/` root folder and
|
||||
| `/slideshow` | A presentation, pitch deck or interactive deck — discrete slides, fragment reveals, branching, hotspot navigation, presenter mode with speaker notes. Output is a navigable deck, not a rendered MP4. |
|
||||
| `/remotion-to-hyperframes` | Bringing existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, other tools' renders), verified frame by frame. |
|
||||
| `/reverse-any-video` | Rebuilding any video file as an editable HyperFrames project (motion graphics, talking heads, live action, AI-generated footage), checked frame by frame against the original. HyperFrames Pro. |
|
||||
| `/style-sampling` | Making a new episode on your topic in one of five proven styles: Collage Explainer, Finance Explainer, Animated Science Explainer, 3D Brand Explainer, 3D True Crime. HyperFrames Pro. |
|
||||
|
||||
For visual design handoff workflows, see the [Claude Design guide](https://hyperframes.heygen.com/guides/claude-design) and [Open Design guide](https://hyperframes.heygen.com/guides/open-design).
|
||||
|
||||
|
||||
@@ -126,6 +126,7 @@ Make any video by asking for it; the `/hyperframes` skill carries the contract a
|
||||
| `/slideshow` | A presentation, pitch deck or interactive deck. Output is a navigable deck, not a rendered MP4. |
|
||||
| `/remotion-to-hyperframes` | Bringing existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, other tools' renders), verified frame by frame. |
|
||||
| `/reverse-any-video` | Rebuilding any video file as an editable HyperFrames project (motion graphics, talking heads, live action, AI-generated footage), checked frame by frame against the original. HyperFrames Pro. |
|
||||
| `/style-sampling` | Making a new episode on your topic in one of five proven styles: Collage Explainer, Finance Explainer, Animated Science Explainer, 3D Brand Explainer, 3D True Crime. HyperFrames Pro. |
|
||||
|
||||
You can name a skill directly, but you do not need to memorize the names to start.
|
||||
|
||||
|
||||
+15
-1
@@ -33,7 +33,7 @@ npm install -g hyperframes
|
||||
| Make a file | [`render`](#render), [`benchmark`](#benchmark) |
|
||||
| Render somewhere else | [`cloud`](#hyperframes-cloud), [`lambda`](#hyperframes-lambda), [`cloudrun`](#hyperframes-cloudrun) |
|
||||
| Fix your setup | [`doctor`](#doctor), [`browser`](#browser), [`upgrade`](#upgrade), [`docs`](#docs), [`info`](#info-and-compositions), [`compositions`](#info-and-compositions), [`clean`](#clean) |
|
||||
| Connect other tools | [`auth`](#hyperframes-auth), [`billing`](#billing), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
|
||||
| Connect other tools | [`auth`](#hyperframes-auth), [`billing`](#billing), [`templates`](#templates), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
|
||||
|
||||
`validate`, `inspect`, and `layout` still work but are
|
||||
[deprecated](#deprecated-validate-inspect-and-layout). Use `check` instead.
|
||||
@@ -2288,6 +2288,20 @@ npx hyperframes billing top-up --needed 180
|
||||
| `--tool` | The step that is waiting, shown on the page |
|
||||
| `--timeout` | Seconds to wait (default 600) |
|
||||
|
||||
### `templates`
|
||||
|
||||
Pulls a style template into the project's `templates/<slug>/`, for a HyperFrames
|
||||
Pro account: `collage-explainer`, `finance-explainer`, `animated-science-explainer`,
|
||||
`3d-brand-explainer` or `3d-true-crime`. A pull costs no credits, and a template already there is left alone.
|
||||
|
||||
```bash
|
||||
npx hyperframes templates pull collage-explainer
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
| ------- | ----------------------------------------------- |
|
||||
| `--dir` | Project folder (default: the current folder) |
|
||||
|
||||
### `figma` and `events`
|
||||
|
||||
`npx hyperframes figma` imports through the Figma REST API, with three
|
||||
|
||||
@@ -423,7 +423,6 @@ async function transcribeOnHeygen(
|
||||
try {
|
||||
const result = await transcribeWithHeygen(inputPath, join(dir, TRANSCRIPT_FILE), {
|
||||
diarize: options.speakers,
|
||||
projectDir: dir,
|
||||
onStatus: options.onStatus,
|
||||
});
|
||||
return finishTranscript(result, "heygen", dir, sidecar);
|
||||
|
||||
@@ -180,6 +180,7 @@ const commandLoaders = {
|
||||
measure: () => import("./commands/measure.js").then((m) => m.default),
|
||||
reverse: () => import("./commands/reverse.js").then((m) => m.default),
|
||||
billing: () => import("./commands/billing.js").then((m) => m.default),
|
||||
templates: () => import("./commands/templates.js").then((m) => m.default),
|
||||
capture: () => import("./commands/capture.js").then((m) => m.default),
|
||||
lambda: () => import("./commands/lambda.js").then((m) => m.default),
|
||||
cloudrun: () => import("./commands/cloudrun.js").then((m) => m.default),
|
||||
|
||||
@@ -894,6 +894,106 @@ describe("offloadHeavyMedia", () => {
|
||||
expect(fileMap.has("media/clip.mp4")).toBe(true);
|
||||
});
|
||||
|
||||
it("offloads a heavy file only a composition's <template> plays", async () => {
|
||||
const dir = project({
|
||||
"index.html": html(`<div data-composition-src="compositions/a.html"></div>`),
|
||||
"compositions/a.html":
|
||||
`<template id="a"><div data-composition-id="a">` +
|
||||
`<video class="clip" src="media/clip.mp4" data-start="0" data-duration="2"></video>` +
|
||||
`</div></template>`,
|
||||
"media/clip.mp4": HEAVY,
|
||||
});
|
||||
const fileMap = buildPublishFileMap(dir);
|
||||
|
||||
const [item] = await offloadHeavyMedia(dir, fileMap, {
|
||||
apiBaseUrl: "https://api.example",
|
||||
upload: fakeUpload(),
|
||||
cachePath: cachePath(),
|
||||
});
|
||||
|
||||
const composition = String(fileMap.get("compositions/a.html"));
|
||||
expect(composition.startsWith(`<template id="a">`)).toBe(true);
|
||||
expect(composition).toContain(`src="${item?.url}"`);
|
||||
expect(fileMap.has("media/clip.mp4")).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps a heavy file a script inside a <template> builds from split strings", async () => {
|
||||
const dir = project({
|
||||
"index.html": html(
|
||||
`<video id="v" src="media/clip.mp4"></video>` +
|
||||
`<template><script>v.src = "media/" + "clip" + ".mp4";</script></template>`,
|
||||
),
|
||||
"media/clip.mp4": HEAVY,
|
||||
});
|
||||
const fileMap = buildPublishFileMap(dir);
|
||||
|
||||
await offloadHeavyMedia(dir, fileMap, {
|
||||
apiBaseUrl: "https://api.example",
|
||||
upload: fakeUpload(),
|
||||
cachePath: cachePath(),
|
||||
});
|
||||
|
||||
expect(fileMap.has("media/clip.mp4")).toBe(true);
|
||||
});
|
||||
|
||||
it("drops an offloaded file whose stem is only a word in script code", async () => {
|
||||
const dir = project({
|
||||
"index.html": html(
|
||||
`<video id="v" src="media/host.mp4"></video>` +
|
||||
`<script>const host = "video/mp4"; const hosted = host.wave;</script>`,
|
||||
),
|
||||
"media/host.mp4": HEAVY,
|
||||
});
|
||||
const fileMap = buildPublishFileMap(dir);
|
||||
|
||||
const [item] = await offloadHeavyMedia(dir, fileMap, {
|
||||
apiBaseUrl: "https://api.example",
|
||||
upload: fakeUpload(),
|
||||
cachePath: cachePath(),
|
||||
});
|
||||
|
||||
expect(item?.kept_in_archive).toBeUndefined();
|
||||
expect(fileMap.has("media/host.mp4")).toBe(false);
|
||||
});
|
||||
|
||||
it("drops an offloaded file when only another folder's file shares its name", async () => {
|
||||
const dir = project({
|
||||
"index.html": html(
|
||||
`<video id="v" src="media/host.mp4"></video>` +
|
||||
`<video id="w" src="media/outro/host.mp4"></video>`,
|
||||
),
|
||||
"media/host.mp4": HEAVY,
|
||||
"media/outro/host.mp4": Buffer.alloc(16, 2),
|
||||
});
|
||||
const fileMap = buildPublishFileMap(dir);
|
||||
|
||||
await offloadHeavyMedia(dir, fileMap, {
|
||||
apiBaseUrl: "https://api.example",
|
||||
upload: fakeUpload(),
|
||||
cachePath: cachePath(),
|
||||
});
|
||||
|
||||
expect(fileMap.has("media/host.mp4")).toBe(false);
|
||||
expect(fileMap.has("media/outro/host.mp4")).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps a zip copy of an offloaded file another file names by a relative path", async () => {
|
||||
const dir = project({
|
||||
"index.html": html(`<video id="v" src="media/clip.mp4"></video>`),
|
||||
"notes.txt": "see ../media/clip.mp4",
|
||||
"media/clip.mp4": HEAVY,
|
||||
});
|
||||
const fileMap = buildPublishFileMap(dir);
|
||||
|
||||
await offloadHeavyMedia(dir, fileMap, {
|
||||
apiBaseUrl: "https://api.example",
|
||||
upload: fakeUpload(),
|
||||
cachePath: cachePath(),
|
||||
});
|
||||
|
||||
expect(fileMap.has("media/clip.mp4")).toBe(true);
|
||||
});
|
||||
|
||||
it("prepares an entry GIF staged on disk from a checked private copy", async () => {
|
||||
const files = {
|
||||
"index.html": html(`<img src="media/loop.gif" />`),
|
||||
|
||||
@@ -523,7 +523,9 @@ function scriptCode(fileMap: Map<string, StagedFile>): string {
|
||||
const text = textOf(path, content);
|
||||
if (text === undefined) continue;
|
||||
if (/\.html?$/i.test(path)) {
|
||||
for (const script of parseHTML(text).document.querySelectorAll("script:not([src])")) {
|
||||
const { document } = parseHTML(text);
|
||||
const pages = [document, ...[...document.querySelectorAll("template")].map((t) => t.content)];
|
||||
for (const script of pages.flatMap((p) => [...p.querySelectorAll("script:not([src])")])) {
|
||||
parts.push(script.textContent ?? "");
|
||||
}
|
||||
} else if (/\.(m?js|cjs|json)$/i.test(path)) parts.push(text);
|
||||
@@ -834,10 +836,17 @@ function dropReplacedFiles(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a reference rewriting could not replace may still load the file: its name anywhere, or its
|
||||
* stem and extension both in script code, which can join them at runtime ("clip" + ".mp4").
|
||||
*/
|
||||
const escapeRegExp = (text: string) => text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
|
||||
function namesFile(text: string, path: string): boolean {
|
||||
const tokens = text.match(new RegExp(`[\\w./-]*${escapeRegExp(basename(path))}`, "g")) ?? [];
|
||||
return tokens.some((token) => {
|
||||
const tail = token.replace(/^[./]+/, "");
|
||||
return path === tail || path.endsWith(`/${tail}`);
|
||||
});
|
||||
}
|
||||
|
||||
/** A path ending in its name, or in script code its stem as a string with the extension nearby ("clip" + ".mp4"). */
|
||||
function isStillMentioned(
|
||||
fileMap: Map<string, StagedFile>,
|
||||
path: string,
|
||||
@@ -845,12 +854,13 @@ function isStillMentioned(
|
||||
withoutUrls: (text: string) => string,
|
||||
): boolean {
|
||||
const ext = extname(path);
|
||||
if (code.includes(basename(path, ext)) && code.includes(ext.slice(1))) return true;
|
||||
const stem = new RegExp(`["'\`/]${escapeRegExp(basename(path, ext))}\\.?["'\`$]`);
|
||||
if (stem.test(code) && code.includes(ext.slice(1))) return true;
|
||||
const name = basename(path);
|
||||
for (const [entryPath, content] of fileMap) {
|
||||
const contentText = textOf(entryPath, content);
|
||||
if (entryPath === path || !contentText?.includes(name)) continue;
|
||||
if (withoutUrls(contentText).includes(name)) return true;
|
||||
if (namesFile(withoutUrls(contentText), path)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
@@ -937,7 +947,29 @@ function rewriteCssEntry(absDir: string, entryPath: string, css: string, urlFor:
|
||||
return result.modified ? result.css : null;
|
||||
}
|
||||
|
||||
const TEMPLATE_SCENE_RE = /(<template\b[^>]*>)([\s\S]*?)(<\/template>)/gi;
|
||||
|
||||
/** An entry's HTML with each <template> scene rewritten too: the runtime stamps those scenes into the page. */
|
||||
function rewriteHtmlEntry(
|
||||
absDir: string,
|
||||
entryPath: string,
|
||||
source: string,
|
||||
urlFor: UrlFor,
|
||||
fileMap: Map<string, StagedFile>,
|
||||
): string | null {
|
||||
const html = source.replace(
|
||||
TEMPLATE_SCENE_RE,
|
||||
(full, open: string, inner: string, close: string) => {
|
||||
const next = rewriteHtmlEntry(absDir, entryPath, inner, urlFor, fileMap);
|
||||
return next === null ? full : open + next + close;
|
||||
},
|
||||
);
|
||||
return (
|
||||
rewriteHtmlDocument(absDir, entryPath, html, urlFor, fileMap) ?? (html === source ? null : html)
|
||||
);
|
||||
}
|
||||
|
||||
function rewriteHtmlDocument(
|
||||
absDir: string,
|
||||
entryPath: string,
|
||||
html: string,
|
||||
|
||||
@@ -7,26 +7,32 @@ import { finishCommand } from "../utils/commandResult.js";
|
||||
import { withMediaAuthEnvironment } from "../auth/media-environment.js";
|
||||
|
||||
const MEDIA_USE_ARGS = {
|
||||
type: { type: "string" },
|
||||
intent: { type: "string" },
|
||||
entity: { type: "string" },
|
||||
project: { type: "string", alias: "p" },
|
||||
adopt: { type: "boolean" },
|
||||
candidates: { type: "boolean" },
|
||||
type: { type: "string", description: "What to find: bgm, sfx, image, icon, logo, voice" },
|
||||
intent: { type: "string", description: "What it should be, in plain words" },
|
||||
entity: { type: "string", description: "The brand or thing a logo belongs to" },
|
||||
project: { type: "string", alias: "p", description: "The project folder (default: here)" },
|
||||
adopt: { type: "boolean", description: "Keep a file you found yourself, with --from" },
|
||||
candidates: { type: "boolean", description: "List reusable assets instead of resolving one" },
|
||||
doctor: { type: "boolean" },
|
||||
stats: { type: "boolean" },
|
||||
days: { type: "string" },
|
||||
"dry-run": { type: "boolean" },
|
||||
reuse: { type: "string" },
|
||||
from: { type: "string" },
|
||||
reuse: {
|
||||
type: "string",
|
||||
description: "Import the global-cache asset with this sha, from --candidates",
|
||||
},
|
||||
from: { type: "string", description: "Bring in this file or URL instead of searching" },
|
||||
params: { type: "string" },
|
||||
for: { type: "string" },
|
||||
analyze: { type: "boolean" },
|
||||
"local-only": { type: "boolean" },
|
||||
provider: { type: "string" },
|
||||
provider: {
|
||||
type: "string",
|
||||
description: "Skip the caches and use this provider only, e.g. heygen",
|
||||
},
|
||||
"avatar-id": { type: "string" },
|
||||
"voice-id": { type: "string" },
|
||||
json: { type: "boolean" },
|
||||
json: { type: "boolean", description: "One JSON result on stdout" },
|
||||
help: { type: "boolean", alias: "h" },
|
||||
} as const;
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ import {
|
||||
checkRequest,
|
||||
generate,
|
||||
parseMask,
|
||||
pickVoice,
|
||||
voiceFor,
|
||||
type GenerateRequest,
|
||||
type GenerateType,
|
||||
} from "../../heygen/generate.js";
|
||||
@@ -26,6 +26,18 @@ export const examples: Example[] = [
|
||||
"The original's presenter re-shot clean, saying an 8 s stretch of the original voice",
|
||||
'hyperframes media-use generate --type video --reference assets/presenter-clean.png --audio assets/s02-voice.m4a --motion assets/s02-original.mp4 --mask 110,615,390,165 --prompt "the woman talks to camera, seated, soft daylight" --seconds 8 --aspect 9:16 --out assets/s02-presenter.mp4',
|
||||
],
|
||||
[
|
||||
"That shot's mouth redone to the exact voice, which stays as it is",
|
||||
"hyperframes media-use generate --type lipsync --video assets/s02-presenter.mp4 --audio assets/s02-voice.mp3 --out assets/s02-lips.mp4",
|
||||
],
|
||||
[
|
||||
"The presenter cut out of their room, on a transparent background",
|
||||
"hyperframes media-use generate --type cutout --video assets/presenter.mp4 --out assets/presenter-cutout.webm",
|
||||
],
|
||||
[
|
||||
"The person in a photo saying a finished voice, which stays as it is",
|
||||
'hyperframes media-use generate --type avatar --photo assets/presenter.png --audio assets/vo.mp3 --prompt "seated, talks with both hands" --aspect 16:9 --out assets/host.mp4',
|
||||
],
|
||||
[
|
||||
"The person in a photo saying a script in a chosen voice",
|
||||
'hyperframes media-use generate --type avatar --photo assets/presenter.png --script "Hi, I am Ana" --voice <voice-id> --out assets/ana.mp4',
|
||||
@@ -49,7 +61,7 @@ export default defineCommand({
|
||||
meta: {
|
||||
name: "generate",
|
||||
description:
|
||||
"Make new media on HeyGen: an image (edited from references), a video shot (from a first frame), a person speaking from a photo, a voice, or music. --plan prices it first",
|
||||
"Make new media on HeyGen: an image (edited from references), a video shot (from a first frame), a shot's mouth redone to a voice, a person speaking from a photo, a voice, or music. --plan prices it first",
|
||||
},
|
||||
args: {
|
||||
type: {
|
||||
@@ -62,7 +74,10 @@ export default defineCommand({
|
||||
description: "The project file to write, e.g. assets/plate.png",
|
||||
required: true,
|
||||
},
|
||||
prompt: { type: "string", description: "What it shows or sounds like (image, video, music)" },
|
||||
prompt: {
|
||||
type: "string",
|
||||
description: "What it shows or sounds like (image, video, music); avatar: how a photo moves",
|
||||
},
|
||||
reference: {
|
||||
type: "string",
|
||||
description:
|
||||
@@ -70,7 +85,13 @@ export default defineCommand({
|
||||
},
|
||||
audio: {
|
||||
type: "string",
|
||||
description: "video: speech the referenced person says, lips in sync (5–15 s)",
|
||||
description:
|
||||
"video: speech the referenced person says (5–15 s), a reference the model re-paces; lipsync, avatar: the exact voice",
|
||||
},
|
||||
video: {
|
||||
type: "string",
|
||||
description:
|
||||
"lipsync: the shot whose mouth is redone; cutout: the shot cut out of its background",
|
||||
},
|
||||
motion: {
|
||||
type: "string",
|
||||
@@ -88,6 +109,10 @@ export default defineCommand({
|
||||
},
|
||||
"first-frame": { type: "string", description: "video: the picture the shot starts from" },
|
||||
photo: { type: "string", description: "avatar: the person's photo" },
|
||||
avatar: {
|
||||
type: "string",
|
||||
description: "avatar: the person's own HeyGen avatar look id, instead of --photo",
|
||||
},
|
||||
script: { type: "string", description: "avatar, voice: the words spoken" },
|
||||
voice: {
|
||||
type: "string",
|
||||
@@ -135,10 +160,12 @@ export default defineCommand({
|
||||
.filter(Boolean),
|
||||
firstFrame: args["first-frame"],
|
||||
audio: args.audio,
|
||||
video: args.video,
|
||||
motion: args.motion,
|
||||
allowCuts: args["allow-cuts"],
|
||||
masks: repeated(rawArgs, "--mask").map(parseMask),
|
||||
photo: args.photo,
|
||||
avatar: args.avatar,
|
||||
script: args.script,
|
||||
voice: args.voice,
|
||||
seconds: num(args.seconds, "--seconds"),
|
||||
@@ -147,7 +174,7 @@ export default defineCommand({
|
||||
bpm: num(args.bpm, "--bpm"),
|
||||
instrumental: !args.vocals,
|
||||
};
|
||||
const voice = type === "voice" || type === "avatar" ? await pickVoice(request) : undefined;
|
||||
const voice = await voiceFor(request);
|
||||
if (voice) request.voice = voice.voice_id;
|
||||
const price = priceOf(request);
|
||||
const credits = creditsOf(price);
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const signedIn = vi.hoisted(() => ({ value: true }));
|
||||
vi.mock("../cloud/auth.js", () => ({
|
||||
resolveCloudAuthHeaders: async () => {
|
||||
if (!signedIn.value) throw new Error("no credential");
|
||||
return { authorization: "Bearer test" };
|
||||
},
|
||||
resolveCloudBaseUrl: () => "https://api.heygen.test",
|
||||
}));
|
||||
|
||||
import { pullTemplate, PullRefused, TEMPLATES } from "./templates.js";
|
||||
|
||||
const LINK = "https://signed.test/bundle.tar.gz?sig=1";
|
||||
let dir: string;
|
||||
let archive: Buffer;
|
||||
|
||||
function tarOf(files: Record<string, string>): Buffer {
|
||||
const src = mkdtempSync(join(dir, "src-"));
|
||||
for (const [name, text] of Object.entries(files)) writeFileSync(join(src, name), text);
|
||||
const out = join(src, "..", `${Date.now()}-${Math.random()}.tar.gz`);
|
||||
execFileSync("tar", ["-czf", out, "-C", src, "."]);
|
||||
return readFileSync(out);
|
||||
}
|
||||
|
||||
function serve(link: Partial<{ status: number }>, body = archive): ReturnType<typeof vi.fn> {
|
||||
const fetchMock = vi.fn(async (url: string) =>
|
||||
url === LINK
|
||||
? new Response(new Uint8Array(body))
|
||||
: new Response(JSON.stringify({ data: { url: LINK, expires_in: 300 } }), {
|
||||
status: link.status ?? 200,
|
||||
}),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
return fetchMock;
|
||||
}
|
||||
|
||||
function pin(body: Buffer): void {
|
||||
const sha256 = createHash("sha256").update(body).digest("hex");
|
||||
TEMPLATES["sample"] = { id: "M9001", sha256, bytes: body.length };
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "hf-templates-"));
|
||||
signedIn.value = true;
|
||||
archive = tarOf({ "SKILL.md": "# sample\n" });
|
||||
pin(archive);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
delete TEMPLATES["sample"];
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe("pullTemplate", () => {
|
||||
it("asks the plan-gated route for the pinned bundle and unpacks it whole", async () => {
|
||||
const fetchMock = serve({});
|
||||
const path = await pullTemplate("sample", dir);
|
||||
expect(path).toBe(join(dir, "templates", "sample"));
|
||||
expect(readFileSync(join(path, "SKILL.md"), "utf8")).toBe("# sample\n");
|
||||
expect(String(fetchMock.mock.calls[0]?.[0])).toContain(
|
||||
`/v1/hyperframes/moments/M9001/bundle?sha256=${TEMPLATES["sample"]?.sha256}`,
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves a template that is already there alone and spends no call", async () => {
|
||||
const fetchMock = serve({});
|
||||
mkdirSync(join(dir, "templates", "sample"), { recursive: true });
|
||||
await pullTemplate("sample", dir);
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("turns a 402 into the upgrade step and writes nothing", async () => {
|
||||
serve({ status: 402 });
|
||||
await expect(pullTemplate("sample", dir)).rejects.toThrow("billing subscribe");
|
||||
expect(existsSync(join(dir, "templates", "sample"))).toBe(false);
|
||||
});
|
||||
|
||||
it("asks for a sign-in when there is no credential or the server says 401", async () => {
|
||||
serve({ status: 401 });
|
||||
await expect(pullTemplate("sample", dir)).rejects.toThrow("auth login");
|
||||
signedIn.value = false;
|
||||
await expect(pullTemplate("sample", dir)).rejects.toThrow("auth login");
|
||||
});
|
||||
|
||||
it("retries a 503 once, then stops without a public fallback", async () => {
|
||||
const fetchMock = serve({ status: 503 });
|
||||
await expect(pullTemplate("sample", dir)).rejects.toBeInstanceOf(PullRefused);
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("refuses a download that does not match its hash", async () => {
|
||||
serve({}, tarOf({ "SKILL.md": "changed\n" }));
|
||||
await expect(pullTemplate("sample", dir)).rejects.toThrow("hash");
|
||||
expect(existsSync(join(dir, "templates", "sample"))).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses an archive that climbs out of its folder and leaves no partial folder", async () => {
|
||||
const src = mkdtempSync(join(dir, "evil-"));
|
||||
mkdirSync(join(src, "a"));
|
||||
writeFileSync(join(src, "x.md"), "x");
|
||||
const out = join(dir, "evil.tar.gz");
|
||||
execFileSync("tar", ["-czPf", out, "-C", join(src, "a"), "../x.md"]); // -P: GNU tar strips the ../ otherwise
|
||||
const evil = readFileSync(out);
|
||||
pin(evil);
|
||||
serve({}, evil);
|
||||
await expect(pullTemplate("sample", dir)).rejects.toThrow("unsafe path");
|
||||
expect(existsSync(join(dir, "templates", "sample"))).toBe(false);
|
||||
});
|
||||
|
||||
it("names the templates when the slug is unknown", async () => {
|
||||
await expect(pullTemplate("nope", dir)).rejects.toThrow("sample");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,160 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { existsSync, mkdirSync, mkdtempSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { resolveCloudAuthHeaders } from "../cloud/auth.js";
|
||||
import { billingBaseUrl } from "../heygen/plan.js";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { failCommand } from "../utils/commandResult.js";
|
||||
import { defineCommand } from "../utils/defineCommand.js";
|
||||
|
||||
export interface TemplatePack {
|
||||
/** The private bundle's id on the plan-gated route. */
|
||||
id: string;
|
||||
sha256: string;
|
||||
bytes: number;
|
||||
}
|
||||
|
||||
// ponytail: packs are pinned here, so a new pack ships with a CLI release; move to a public index if templates change faster.
|
||||
export const TEMPLATES: Record<string, TemplatePack> = {
|
||||
"animated-science-explainer": {
|
||||
id: "M9003",
|
||||
sha256: "4ead405395892545c658a03b48630529943f5598cc15770930c0ffa6b8de58e9",
|
||||
bytes: 25559788,
|
||||
},
|
||||
"finance-explainer": {
|
||||
id: "M9002",
|
||||
sha256: "b10efcfad35097ddac8118062701eeb3ef4012f3c9b41704e474964d09d37d11",
|
||||
bytes: 140688140,
|
||||
},
|
||||
"3d-true-crime": {
|
||||
id: "M9005",
|
||||
sha256: "53d1f51eb5b78e3cb60a23625d160952236d5f5fb8d41118798d782a6168eec1",
|
||||
bytes: 5977486,
|
||||
},
|
||||
"collage-explainer": {
|
||||
id: "M9001",
|
||||
sha256: "395362e3c629706143724e99d72dacb4997d35b978ecc4173901cdef6e70205c",
|
||||
bytes: 36711101,
|
||||
},
|
||||
"3d-brand-explainer": {
|
||||
id: "M9004",
|
||||
sha256: "4e5c5eb574f977d3f97a170a5a0d8d442234d34dd51f925dc4a2b93e32826ae8",
|
||||
bytes: 20181051,
|
||||
},
|
||||
};
|
||||
|
||||
/** A pull that stopped, with the one line the agent relays. Nothing was written. */
|
||||
export class PullRefused extends Error {}
|
||||
|
||||
const LINK_TIMEOUT_MS = 30_000;
|
||||
const DOWNLOAD_TIMEOUT_MS = 20 * 60_000;
|
||||
|
||||
async function bundleUrl(pack: TemplatePack, retries = 1): Promise<string> {
|
||||
const headers = await resolveCloudAuthHeaders().catch(() => {
|
||||
throw new PullRefused("Not signed in. Run `npx hyperframes auth login`.");
|
||||
});
|
||||
const route = `${billingBaseUrl()}/v1/hyperframes/moments/${pack.id}/bundle?sha256=${pack.sha256}`;
|
||||
const res = await fetch(route, {
|
||||
headers: { ...headers, accept: "application/json" },
|
||||
signal: AbortSignal.timeout(LINK_TIMEOUT_MS),
|
||||
});
|
||||
if (res.status === 503 && retries > 0) return bundleUrl(pack, retries - 1);
|
||||
if (res.status === 401) throw new PullRefused("Not signed in. Run `npx hyperframes auth login`.");
|
||||
if (res.status === 402)
|
||||
throw new PullRefused(
|
||||
"Creator templates are part of HyperFrames Pro; nothing was downloaded. Run `npx hyperframes billing subscribe`: it opens the page and waits.",
|
||||
);
|
||||
if (res.status === 404) throw new PullRefused("That template version is no longer available.");
|
||||
if (!res.ok) throw new PullRefused(`HeyGen could not hand over the template (${res.status}).`);
|
||||
const body = (await res.json()) as { data?: { url?: string } };
|
||||
if (!body.data?.url) throw new PullRefused("HeyGen answered without a download link.");
|
||||
return body.data.url;
|
||||
}
|
||||
|
||||
async function download(url: string, pack: TemplatePack): Promise<Buffer> {
|
||||
const res = await fetch(url, {
|
||||
signal: AbortSignal.timeout(DOWNLOAD_TIMEOUT_MS),
|
||||
});
|
||||
if (!res.ok) throw new PullRefused(`The download failed (${res.status}). Run it again.`);
|
||||
const data = Buffer.from(await res.arrayBuffer());
|
||||
const sha256 = createHash("sha256").update(data).digest("hex");
|
||||
if (data.length !== pack.bytes || sha256 !== pack.sha256)
|
||||
throw new PullRefused("The download does not match the template's hash; nothing was unpacked.");
|
||||
return data;
|
||||
}
|
||||
|
||||
function assertSafeEntries(archive: string): void {
|
||||
const names = execFileSync("tar", ["-tzf", archive], {
|
||||
encoding: "utf8",
|
||||
maxBuffer: 1 << 26,
|
||||
});
|
||||
for (const name of names.split("\n"))
|
||||
if (name.startsWith("/") || name.split("/").includes(".."))
|
||||
throw new PullRefused(`The template holds an unsafe path (${name}); nothing was unpacked.`);
|
||||
}
|
||||
|
||||
/** Unpacks the pack beside `dest` and moves it in whole, so `dest` only ever holds a complete template. */
|
||||
function unpack(data: Buffer, dest: string): void {
|
||||
mkdirSync(dirname(dest), { recursive: true });
|
||||
const work = mkdtempSync(`${dest}.part-`);
|
||||
try {
|
||||
const archive = join(work, "template.tar.gz");
|
||||
const out = join(work, "template");
|
||||
writeFileSync(archive, data);
|
||||
assertSafeEntries(archive);
|
||||
mkdirSync(out);
|
||||
execFileSync("tar", ["-xzf", archive, "-C", out, "--no-same-owner"]);
|
||||
renameSync(out, dest);
|
||||
} finally {
|
||||
rmSync(work, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
/** Fetches the template into `<dir>/templates/<slug>/` and returns that path; one already there stays as it is. */
|
||||
export async function pullTemplate(slug: string, dir: string): Promise<string> {
|
||||
const pack = TEMPLATES[slug];
|
||||
if (!pack)
|
||||
throw new PullRefused(
|
||||
`No template "${slug}". Templates: ${Object.keys(TEMPLATES).join(", ")}.`,
|
||||
);
|
||||
const dest = resolve(dir, "templates", slug);
|
||||
if (existsSync(dest)) return dest;
|
||||
unpack(await download(await bundleUrl(pack), pack), dest);
|
||||
return dest;
|
||||
}
|
||||
|
||||
/** Downloads a style template for a HyperFrames Pro account. The server decides who may have it. */
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "templates",
|
||||
description: "Pull a style template into this project (HyperFrames Pro)",
|
||||
},
|
||||
args: {
|
||||
action: { type: "positional", description: "pull", required: true },
|
||||
slug: {
|
||||
type: "positional",
|
||||
description: "The template's slug, from the style-sampling skill",
|
||||
required: true,
|
||||
},
|
||||
dir: {
|
||||
type: "string",
|
||||
description: "Project folder (default: the current folder)",
|
||||
},
|
||||
},
|
||||
async run({ args }) {
|
||||
if (args.action !== "pull") {
|
||||
console.error(c.error("Use `templates pull <slug>`."));
|
||||
failCommand();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const path = await pullTemplate(String(args.slug), args.dir ?? ".");
|
||||
console.log(c.success(`✓ ${path}`));
|
||||
} catch (error) {
|
||||
if (!(error instanceof PullRefused)) throw error;
|
||||
console.error(c.error(error.message));
|
||||
failCommand();
|
||||
}
|
||||
},
|
||||
});
|
||||
@@ -56,7 +56,7 @@ export default defineCommand({
|
||||
engine: {
|
||||
type: "string",
|
||||
description:
|
||||
"ASR engine: auto (Parakeet if installed and it covers --language, else whisper), parakeet, whisper, or heygen (HeyGen's speech model; needs an API key, charged to API credits). Default: auto. Parakeet is more accurate and faster than whisper; install it with `hyperframes models install parakeet`.",
|
||||
"ASR engine: auto (Parakeet if installed and it covers --language, else whisper), parakeet, whisper, or heygen (HeyGen's speech model; bills the HyperFrames plan when signed in, or API credits with a key). Default: auto. Parakeet is more accurate and faster than whisper; install it with `hyperframes models install parakeet`.",
|
||||
alias: "e",
|
||||
},
|
||||
speakers: {
|
||||
|
||||
@@ -66,6 +66,7 @@ const GROUPS: Group[] = [
|
||||
],
|
||||
["reverse", "Open a reverse-any-video run: note it with HeyGen, then probe the video"],
|
||||
["billing", "Open the HyperFrames plans or credits page and wait until it is done"],
|
||||
["templates", "Pull a style template into this project (HyperFrames Pro)"],
|
||||
["info", "Print project metadata"],
|
||||
["compositions", "List all compositions in a project"],
|
||||
["timeline", "Print the project's tracks and clips"],
|
||||
|
||||
@@ -12,6 +12,15 @@ vi.mock("../cloud/auth.js", () => ({
|
||||
resolveCloudBaseUrl: () => "https://api.test",
|
||||
}));
|
||||
|
||||
vi.mock("../cloud/index.js", () => ({ createCloudClient: async () => ({}) }));
|
||||
const uploads = vi.hoisted(() => [] as { filename: string; contentType: string }[]);
|
||||
vi.mock("../cloud/upload.js", () => ({
|
||||
uploadAssetViaDirectUpload: async (u: { filename: string; contentType: string }) => {
|
||||
uploads.push({ filename: u.filename, contentType: u.contentType });
|
||||
return { asset_id: `asset-${u.filename}` };
|
||||
},
|
||||
}));
|
||||
|
||||
vi.stubEnv("HEYGEN_API_KEY", "k");
|
||||
vi.stubEnv("HEYGEN_API_URL", "https://api.test");
|
||||
|
||||
@@ -649,6 +658,7 @@ describe("generate", () => {
|
||||
}).usd,
|
||||
).toBe(0.07);
|
||||
expect(priceOf({ type: "voice", out: "a.mp3" }).usd).toBeNull();
|
||||
expect(creditsOf(priceOf({ type: "voice", out: "a.mp3", script: "今天聊黄金" }))).toBe(1);
|
||||
});
|
||||
|
||||
it("prices only a request the real call would accept", () => {
|
||||
@@ -673,12 +683,118 @@ describe("generate", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("redoes a shot's mouth to the exact voice", async () => {
|
||||
const video = shots("lips-shot.mp4", ["color=c=red:d=2"]);
|
||||
const audio = join(project, "lips-voice.mp3");
|
||||
writeFileSync(audio, "voice");
|
||||
const sent: { url: string; body?: Record<string, unknown> }[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
sent.push({ url, ...(init?.body ? { body: JSON.parse(String(init.body)) } : {}) });
|
||||
if (url === "https://files.test/lips.mp4") return new Response(new Uint8Array([1, 2, 3]));
|
||||
if (init?.method === "POST") return json({ data: { lipsync_id: "lip1" } });
|
||||
return json({
|
||||
data: { id: "lip1", status: "completed", video_url: "https://files.test/lips.mp4" },
|
||||
});
|
||||
});
|
||||
const out = join(project, "lips-out.mp4");
|
||||
const request = { type: "lipsync" as const, video, audio, out };
|
||||
const made = await generate(request);
|
||||
expect(made.job).toBe("lip1");
|
||||
expect(sent[0]!.url).toBe("https://api.test/v3/lipsyncs");
|
||||
expect(sent[0]!.body).toMatchObject({
|
||||
video: { type: "asset_id", asset_id: "asset-lips-shot.mp4" },
|
||||
audio: { type: "asset_id", asset_id: "asset-lips-voice.mp3" },
|
||||
mode: "speed",
|
||||
enable_dynamic_duration: false,
|
||||
});
|
||||
expect(sent[1]!.url).toBe("https://api.test/v3/lipsyncs/lip1");
|
||||
expect(existsSync(out)).toBe(true);
|
||||
expect(creditsOf(priceOf(request))).toBe(14);
|
||||
expect(() => checkRequest({ type: "lipsync", audio, out })).toThrow("--video");
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("cuts out a WebM or MOV shot as video, with its key in the body", async () => {
|
||||
const posts: Record<string, unknown>[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
if (url === "https://files.test/fg.webm") return new Response(new Uint8Array([1, 2, 3]));
|
||||
if (init?.method === "POST") posts.push(JSON.parse(String(init.body)));
|
||||
return json({
|
||||
data: {
|
||||
id: "bg1",
|
||||
status: "completed",
|
||||
layers: { foreground: "https://files.test/fg.webm" },
|
||||
},
|
||||
});
|
||||
});
|
||||
uploads.length = 0;
|
||||
for (const ext of ["webm", "mov"]) {
|
||||
const video = join(project, `cut-shot.${ext}`);
|
||||
execFileSync(ffmpeg!, [
|
||||
"-v",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=red:s=64x36:d=1",
|
||||
video,
|
||||
]);
|
||||
const out = join(project, `cut-${ext}.webm`);
|
||||
await generate({ type: "cutout", video, out });
|
||||
await generate({ type: "cutout", video, out });
|
||||
}
|
||||
expect(uploads.map((u) => u.contentType)).toEqual([
|
||||
"video/webm",
|
||||
"video/webm",
|
||||
"video/quicktime",
|
||||
"video/quicktime",
|
||||
]);
|
||||
// the route dedupes on the body's request_id alone, so a re-run must send the same one
|
||||
const ids = posts.map((p) => p.request_id);
|
||||
expect(ids[0]).toMatch(/\S/);
|
||||
expect(ids[1]).toBe(ids[0]);
|
||||
expect(ids[2]).toBe(ids[3]);
|
||||
expect(ids[2]).not.toBe(ids[0]);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("speaks a finished voice as it is, from a photo or a look", async () => {
|
||||
const photo = join(project, "avatar-photo.png");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "color=c=red:s=64x36"],
|
||||
...["-frames:v", "1", photo],
|
||||
]);
|
||||
const audio = join(project, "avatar-voice.wav");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "sine=frequency=440:duration=6", audio],
|
||||
]);
|
||||
const sent: Record<string, unknown>[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
if (url === "https://files.test/a.mp4") return new Response(new Uint8Array([1]));
|
||||
if (init?.body) sent.push(JSON.parse(String(init.body)));
|
||||
return json({
|
||||
data: { id: "v1", status: "completed", video_url: "https://files.test/a.mp4" },
|
||||
});
|
||||
});
|
||||
const out = join(project, "avatar-out.mp4");
|
||||
await generate({ type: "avatar", photo, audio, prompt: "talks with both hands", out });
|
||||
await generate({ type: "avatar", avatar: "look1", audio, out });
|
||||
expect(sent[0]).toMatchObject({
|
||||
type: "image",
|
||||
audio_asset_id: "asset-avatar-voice.wav",
|
||||
motion_prompt: "talks with both hands",
|
||||
});
|
||||
expect(sent[0]).not.toHaveProperty("script");
|
||||
expect(sent[1]).toMatchObject({ type: "avatar", avatar_id: "look1" });
|
||||
expect(creditsOf(priceOf({ type: "avatar", photo, audio, out }))).toBe(116);
|
||||
expect(() => checkRequest({ type: "avatar", photo, out })).toThrow("--script");
|
||||
});
|
||||
|
||||
it("prices an avatar on the plan's avatar rows, not its API price", () => {
|
||||
const script = Array.from({ length: 30 }, () => "word").join(" ");
|
||||
const avatar = priceOf({ type: "avatar", out: "a.mp4", script });
|
||||
expect(creditsOf(avatar)).toBe(200);
|
||||
expect(creditsOf(avatar)).toBe(289);
|
||||
expect(avatar.list).toBeCloseTo(1.2167);
|
||||
expect(() => checkLimit(creditsOf(avatar), 220)).not.toThrow();
|
||||
expect(() => checkLimit(creditsOf(avatar), 300)).not.toThrow();
|
||||
});
|
||||
|
||||
it("charges a reference shot's pictures after the first two", () => {
|
||||
@@ -689,18 +805,18 @@ describe("generate", () => {
|
||||
references: Array.from({ length: pictures }, (_, i) => `${i}.png`),
|
||||
});
|
||||
const credits = [1, 2, 3, 9].map((n) => creditsOf(priceOf(shot(n))));
|
||||
expect(credits).toEqual([167, 167, 180, 260]);
|
||||
expect(credits).toEqual([250, 250, 270, 390]);
|
||||
expect(priceOf(shot(9)).list).toBeCloseTo(0.39);
|
||||
expect(() => checkLimit(credits[3]!, 250)).toThrow("about 260 credits, over --max-credits 250");
|
||||
expect(() => checkLimit(credits[3]!, 260)).not.toThrow();
|
||||
expect(() => checkLimit(credits[3]!, 380)).toThrow("about 390 credits, over --max-credits 380");
|
||||
expect(() => checkLimit(credits[3]!, 390)).not.toThrow();
|
||||
});
|
||||
|
||||
it("keeps a call under the person's credit limit, or refuses it before sending", () => {
|
||||
const shot = creditsOf(priceOf({ type: "video", out: "a.mp4", seconds: 6 }));
|
||||
expect(shot).toBe(72);
|
||||
expect(shot).toBe(108);
|
||||
expect(creditsOf(priceOf({ type: "image", out: "a.png" }))).toBe(67);
|
||||
expect(() => checkLimit(shot, 72)).not.toThrow();
|
||||
expect(() => checkLimit(shot, 30)).toThrow("about 72 credits, over --max-credits 30");
|
||||
expect(() => checkLimit(shot, 108)).not.toThrow();
|
||||
expect(() => checkLimit(shot, 30)).toThrow("about 108 credits, over --max-credits 30");
|
||||
expect(() => checkLimit(null, 30)).toThrow("cannot be priced ahead");
|
||||
expect(() => checkLimit(null, undefined)).not.toThrow();
|
||||
});
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
// New media on HeyGen for `media-use generate`: a still (optionally edited from a reference), a shot (optionally from
|
||||
// a first frame), a person speaking from one photo, a voice reading text, or music. Every call is priced first with
|
||||
// --plan, keyed for idempotency, downloaded into the project and recorded in the media ledger as generated.
|
||||
// New media on HeyGen for `media-use generate`: a still, a shot, a shot's mouth redone to a voice, a person speaking
|
||||
// from one photo, a voice, or music. Every call is priced first with --plan, keyed for idempotency, downloaded into
|
||||
// the project and recorded in the media ledger as generated.
|
||||
import { randomUUID } from "node:crypto";
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
@@ -11,8 +12,10 @@ import {
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import { basename, dirname, extname, join, relative, resolve } from "node:path";
|
||||
import type {
|
||||
CreateBackgroundRemovalRequest,
|
||||
CreateLipsyncRequest,
|
||||
CreateModelVideoRequest,
|
||||
CreateSpeechV3Request,
|
||||
CreateVideoV3RequestBody,
|
||||
@@ -33,11 +36,13 @@ import {
|
||||
type Voice,
|
||||
} from "./v3.js";
|
||||
|
||||
export type GenerateType = "image" | "video" | "avatar" | "voice" | "music";
|
||||
export type GenerateType = "image" | "video" | "lipsync" | "avatar" | "cutout" | "voice" | "music";
|
||||
export const GENERATE_TYPES: readonly GenerateType[] = [
|
||||
"image",
|
||||
"video",
|
||||
"lipsync",
|
||||
"avatar",
|
||||
"cutout",
|
||||
"voice",
|
||||
"music",
|
||||
];
|
||||
@@ -48,8 +53,10 @@ export interface GenerateRequest {
|
||||
prompt?: string;
|
||||
/** image: pictures the result is edited from or styled after. video: the person or things to keep in the shot. */
|
||||
references?: string[];
|
||||
/** video: speech the person in the references says, lips in sync (a stretch of the original's voice). */
|
||||
/** video: speech as a reference, which the model re-paces. lipsync and avatar: the exact voice, kept as it is. */
|
||||
audio?: string;
|
||||
/** lipsync: the shot whose mouth is redone. cutout: the shot whose background is removed. */
|
||||
video?: string;
|
||||
/** video: a clip whose camera and body motion the shot follows (the original's same stretch, up to 15 s). */
|
||||
motion?: string;
|
||||
/** video: skip the one-shot check of the motion clip (a guide a person has looked at). */
|
||||
@@ -60,6 +67,8 @@ export interface GenerateRequest {
|
||||
firstFrame?: string;
|
||||
/** avatar: the photo of the person who speaks. */
|
||||
photo?: string;
|
||||
/** avatar: the person's own HeyGen avatar look, instead of a photo. */
|
||||
avatar?: string;
|
||||
/** avatar and voice: what is said. */
|
||||
script?: string;
|
||||
voice?: string;
|
||||
@@ -157,10 +166,7 @@ interface Route extends JobRoute {
|
||||
ext: string[];
|
||||
}
|
||||
|
||||
type Media = (
|
||||
path: string,
|
||||
kind: "image" | "audio" | "video",
|
||||
) => { type: "base64"; media_type: string; data: string };
|
||||
type Media = (path: string, kind: "image" | "audio" | "video") => Record<string, unknown>;
|
||||
|
||||
const DEFAULTS = { image: "nano-banana-2", video: "heygen-video-1", music: "lyria-3-pro" } as const;
|
||||
const VIDEO_RESOLUTION = "768p";
|
||||
@@ -200,7 +206,6 @@ const ROUTES: Record<GenerateType, Route> = {
|
||||
status: (c, id, signal) => c.getModelVideo({ video_id: id, signal }),
|
||||
// fallow-ignore-next-line complexity
|
||||
body: (r, media) => {
|
||||
// A person speaking given audio is a reference shot: the references show who, the audio drives the lips.
|
||||
const spoken = !!r.audio || !!r.references?.length || !!r.motion;
|
||||
if (spoken && r.firstFrame)
|
||||
throw new Fix(
|
||||
@@ -229,17 +234,60 @@ const ROUTES: Record<GenerateType, Route> = {
|
||||
ledger: "video",
|
||||
ext: [".mp4", ".mov", ".webm"],
|
||||
},
|
||||
lipsync: {
|
||||
path: "/v3/lipsyncs",
|
||||
start: (c, body, key, signal) =>
|
||||
c.createLipsyncV3({ body: body as CreateLipsyncRequest, idempotencyKey: key, signal }),
|
||||
status: (c, id, signal) => c.getLipsyncV3({ lipsync_id: id, signal }),
|
||||
body: (r, media) => ({
|
||||
video: media(need(r.video, "--video", "--video work/s01.mp4"), "video"),
|
||||
audio: media(need(r.audio, "--audio", "--audio work/s01.mp3"), "audio"),
|
||||
mode: "speed",
|
||||
enable_dynamic_duration: false,
|
||||
fps_mode: "cfr",
|
||||
keep_the_same_format: true,
|
||||
}),
|
||||
urls: ["video_url"],
|
||||
ledger: "video",
|
||||
ext: [".mp4"],
|
||||
},
|
||||
cutout: {
|
||||
path: "/v3/background-removals",
|
||||
start: (c, body, key, signal) =>
|
||||
c.createBackgroundRemoval({
|
||||
body: { ...(body as CreateBackgroundRemovalRequest), request_id: key },
|
||||
signal,
|
||||
}),
|
||||
status: (c, id, signal) => c.getBackgroundRemoval({ job_id: id, signal }),
|
||||
body: (r, media) => ({
|
||||
video: media(need(r.video, "--video", "--video assets/presenter.mp4"), "video"),
|
||||
layers: ["foreground"],
|
||||
}),
|
||||
urls: ["layers.foreground"],
|
||||
ledger: "video",
|
||||
ext: [".webm", ".mov"],
|
||||
},
|
||||
avatar: {
|
||||
path: "/v3/videos",
|
||||
start: (c, body, key, signal) =>
|
||||
c.createVideo({ body: body as CreateVideoV3RequestBody, idempotencyKey: key, signal }),
|
||||
status: (c, id, signal) => c.getVideoV3({ video_id: id, signal }),
|
||||
body: (r, media) => ({
|
||||
type: "image",
|
||||
image: media(need(r.photo, "--photo", "--photo assets/presenter.png"), "image"),
|
||||
script: need(r.script, "--script", '--script "What they say"'),
|
||||
voice_id: need(r.voice, "--voice", "--voice <voice id or language>"),
|
||||
...(r.avatar
|
||||
? { type: "avatar", avatar_id: r.avatar }
|
||||
: {
|
||||
type: "image",
|
||||
image: media(need(r.photo, "--photo", "--photo assets/presenter.png"), "image"),
|
||||
...(r.prompt ? { motion_prompt: r.prompt } : {}),
|
||||
}),
|
||||
...(r.audio
|
||||
? { audio_asset_id: media(r.audio, "audio").asset_id }
|
||||
: {
|
||||
script: need(r.script, "--script", '--script "What they say"'),
|
||||
voice_id: need(r.voice, "--voice", "--voice <voice id or language>"),
|
||||
}),
|
||||
...(r.aspect ? { aspect_ratio: r.aspect } : {}),
|
||||
resolution: "1080p",
|
||||
output_format: "mp4",
|
||||
}),
|
||||
urls: ["video_url"],
|
||||
@@ -341,6 +389,34 @@ function inline(work: string, masks: Mask[] = []): Media {
|
||||
};
|
||||
}
|
||||
|
||||
/** Files a route takes as asset ids, not inline. */
|
||||
function assetsOf(r: GenerateRequest): string[] {
|
||||
if (r.type === "lipsync") return [r.video!, r.audio!];
|
||||
if (r.type === "cutout") return [r.video!];
|
||||
return r.type === "avatar" && r.audio ? [r.audio] : [];
|
||||
}
|
||||
|
||||
/** Uploads them whole; the asset id of each path. */
|
||||
async function uploaded(paths: string[]): Promise<Map<string, string>> {
|
||||
const ids = new Map<string, string>();
|
||||
if (!paths.length) return ids;
|
||||
const { createCloudClient } = await import("../cloud/index.js");
|
||||
const { uploadAssetViaDirectUpload } = await import("../cloud/upload.js");
|
||||
const { getMimeType } = await import("@hyperframes/studio-server");
|
||||
const client = await createCloudClient();
|
||||
for (const path of paths) {
|
||||
const asset = await uploadAssetViaDirectUpload({
|
||||
client,
|
||||
bytes: new Uint8Array(readFileSync(path)),
|
||||
filename: basename(path),
|
||||
contentType: getMimeType(path),
|
||||
idempotencyKey: randomUUID(),
|
||||
});
|
||||
ids.set(path, asset.asset_id);
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
/**
|
||||
* The seconds where a new shot starts in a video's first `seconds`, and the length read. A cut within a frame of
|
||||
* either end leaves a stray frame, not a second shot, and does not count.
|
||||
@@ -419,7 +495,7 @@ function fillMasks(path: string, masks: Mask[], work: string): string {
|
||||
* HeyGen answers a 16:9 request at 1344x768 (1.75:1). The result is cropped, centred, to the requested ratio
|
||||
* (1344x756) rather than stretched; returns a note on what was done, or nothing when it already fits.
|
||||
*/
|
||||
export function fitAspect(out: string, aspect: string): string | undefined {
|
||||
function fitAspect(out: string, aspect: string): string | undefined {
|
||||
const [aw, ah] = aspect.split(":").map(Number);
|
||||
if (!aw || !ah) return;
|
||||
const { width, height } = probeVideo(out);
|
||||
@@ -497,6 +573,10 @@ function checkGuide(r: Pick<GenerateRequest, "motion" | "allowCuts">): void {
|
||||
* The route's body runs with a stand-in for the media, so its required flags and conflicts are checked as sent. */
|
||||
const VOICE_ID = /^[0-9a-f]{32}$/i;
|
||||
|
||||
export async function voiceFor(r: GenerateRequest): Promise<Voice | undefined> {
|
||||
return r.type === "voice" || (r.type === "avatar" && !r.audio) ? pickVoice(r) : undefined;
|
||||
}
|
||||
|
||||
export async function pickVoice(r: GenerateRequest): Promise<Voice> {
|
||||
if (r.voice && VOICE_ID.test(r.voice)) return { voice_id: r.voice, name: r.voice };
|
||||
if (!r.voice) {
|
||||
@@ -565,11 +645,18 @@ export async function generate(
|
||||
const project = projectOf(dirname(out));
|
||||
const work = mkdtempSync(join(tmpdir(), "hf-generate-"));
|
||||
try {
|
||||
const body = route.body(r, inline(work, r.masks));
|
||||
const key = idempotencyKey(route.path, [JSON.stringify(body)]);
|
||||
const sources = assetsOf(r);
|
||||
const ids = await uploaded(sources);
|
||||
const local = inline(work, r.masks);
|
||||
const body = route.body(r, (path, kind) =>
|
||||
ids.has(path) ? { type: "asset_id", asset_id: ids.get(path) } : local(path, kind),
|
||||
);
|
||||
// An asset id is new on every upload; the files say whether this is the same job.
|
||||
const sent = [...ids.values()].reduce((t, id) => t.replaceAll(id, ""), JSON.stringify(body));
|
||||
const key = idempotencyKey(route.path, [...sources.map((p) => readFileSync(p)), sent]);
|
||||
const job = await runJob(route, body, key, {
|
||||
log: {
|
||||
projectDir: project ?? undefined,
|
||||
projectDir: project ?? process.cwd(),
|
||||
what: `generate ${r.type} ${relative(project ?? ".", out)}`,
|
||||
},
|
||||
onWait: (j) => onWait?.(j.status),
|
||||
@@ -634,7 +721,9 @@ export async function generate(
|
||||
...(r.references?.length ? { references: r.references.map(describe) } : {}),
|
||||
...(r.firstFrame ? { first_frame: describe(r.firstFrame) } : {}),
|
||||
...(r.photo ? { photo: describe(r.photo) } : {}),
|
||||
...(r.avatar ? { avatar: r.avatar } : {}),
|
||||
...(r.audio ? { audio: describe(r.audio) } : {}),
|
||||
...(r.video ? { video: describe(r.video) } : {}),
|
||||
...(r.motion ? { motion: describe(r.motion) } : {}),
|
||||
...(r.masks?.length
|
||||
? { motion_masks: r.masks.map((m) => [...m.box, ...(m.window ?? [])]) }
|
||||
|
||||
@@ -22,7 +22,7 @@ function mockPlan(): Plan | null {
|
||||
}
|
||||
|
||||
// Payment routes live on api2, beside the OAuth ones; the v3 generation routes stay on api.
|
||||
const billingBaseUrl = () => resolveCloudBaseUrl().replace("://api.", "://api2.");
|
||||
export const billingBaseUrl = () => resolveCloudBaseUrl().replace("://api.", "://api2.");
|
||||
|
||||
export async function readPlan(): Promise<Plan | null> {
|
||||
try {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Prices before a paid call, from the HyperFrames app's creation pricing. HeyGen charges in units of $1/60, rounded
|
||||
// up; the label rounds up to the cent. Routes HeyGen does not price ahead say so instead of guessing.
|
||||
import { Fix, probeVideo, round } from "../measure/media.js";
|
||||
import { Fix, probeSeconds, probeVideo, round } from "../measure/media.js";
|
||||
import { GUIDE_MAX_S, type GenerateRequest } from "./generate.js";
|
||||
|
||||
export interface Price {
|
||||
@@ -38,14 +38,19 @@ const VIDEO_REF_768P_PER_S = 0.05;
|
||||
// A reference shot's first two pictures are included; each one after them bills on its own line.
|
||||
const VIDEO_INCLUDED_IMAGES = 2;
|
||||
const VIDEO_EXTRA_IMAGE = 0.02;
|
||||
// The price book's HyperFrames rows (experiment-framework price_book rules.json, 2026-10-28).
|
||||
const HF_VIDEO_768P_PER_S = 12;
|
||||
const HF_VIDEO_REF_768P_PER_S = 100 / 3;
|
||||
const HF_VIDEO_EXTRA_IMAGE = 40 / 3;
|
||||
const HF_AVATAR_IV_PHOTO_PER_S = 800 / 60;
|
||||
// The price book's HyperFrames rows (experiment-framework price_book rules.json, 2026-10-30).
|
||||
const HF_VIDEO_768P_PER_S = 18;
|
||||
const HF_VIDEO_REF_768P_PER_S = 50;
|
||||
const HF_VIDEO_EXTRA_IMAGE = 20;
|
||||
const HF_AVATAR_IV_PHOTO_PER_S = 1155 / 60;
|
||||
const HF_AVATAR_IV_TWIN_PER_S = 2415 / 60;
|
||||
// Billed on the whole video, not the voice: measured 2026-11-01, a 13.75 s shot took 97 credits.
|
||||
const HF_LIPSYNC_SPEED_PER_S = 7;
|
||||
const MUSIC_PER_TRACK = 0.08;
|
||||
const AVATAR_IV_PER_MIN = 4.83;
|
||||
const SCRIPT_WORDS_PER_S = 2;
|
||||
const HF_TTS_PER_S = 1;
|
||||
const SPEECH_CHARS_PER_S = { zh: 5, en: 15 };
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
export function priceOf(r: GenerateRequest): Price {
|
||||
@@ -78,24 +83,46 @@ export function priceOf(r: GenerateRequest): Price {
|
||||
seconds * HF_VIDEO_768P_PER_S,
|
||||
);
|
||||
}
|
||||
case "lipsync": {
|
||||
const seconds = r.video ? probeVideo(r.video).pictureDuration : 0;
|
||||
return {
|
||||
usd: null,
|
||||
list: null,
|
||||
credits: seconds * HF_LIPSYNC_SPEED_PER_S,
|
||||
basis: `lipsync at speed, ${round(seconds, 2)} s at ${HF_LIPSYNC_SPEED_PER_S} credits a second`,
|
||||
};
|
||||
}
|
||||
case "avatar": {
|
||||
const words = (r.script ?? "").split(/\s+/).filter(Boolean).length;
|
||||
const seconds = Math.max(1, Math.floor(words / SCRIPT_WORDS_PER_S));
|
||||
const seconds = r.audio
|
||||
? probeSeconds(r.audio)
|
||||
: Math.max(1, Math.floor(words / SCRIPT_WORDS_PER_S));
|
||||
return at(
|
||||
Math.ceil(seconds * AVATAR_IV_PER_MIN) / 60,
|
||||
`avatar video, about ${seconds} s from ${words} words at $4.83 a minute`,
|
||||
seconds * HF_AVATAR_IV_PHOTO_PER_S,
|
||||
`avatar video, about ${round(seconds, 1)} s${r.audio ? "" : ` from ${words} words`} at $4.83 a minute`,
|
||||
seconds * (r.avatar ? HF_AVATAR_IV_TWIN_PER_S : HF_AVATAR_IV_PHOTO_PER_S),
|
||||
);
|
||||
}
|
||||
case "music":
|
||||
return at(MUSIC_PER_TRACK, "lyria-3-pro, $0.08 a track");
|
||||
case "voice":
|
||||
case "cutout":
|
||||
return {
|
||||
usd: null,
|
||||
list: null,
|
||||
credits: null,
|
||||
basis: "HeyGen does not price catalog speech ahead; it is charged per character",
|
||||
basis: "background removal, not priced ahead",
|
||||
};
|
||||
case "music":
|
||||
return at(MUSIC_PER_TRACK, "lyria-3-pro, $0.08 a track");
|
||||
case "voice": {
|
||||
const text = r.script ?? "";
|
||||
const cjk = (text.match(/\p{Script=Han}/gu) ?? []).length;
|
||||
const seconds = cjk / SPEECH_CHARS_PER_S.zh + (text.length - cjk) / SPEECH_CHARS_PER_S.en;
|
||||
return {
|
||||
usd: null,
|
||||
list: null,
|
||||
credits: seconds * HF_TTS_PER_S,
|
||||
basis: `about ${round(seconds, 1)} s of speech at ${HF_TTS_PER_S} credit a second, an estimate: it is charged per character`,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||
@@ -15,6 +15,7 @@ vi.stubEnv("HEYGEN_API_KEY", "k");
|
||||
const { transcribeWithHeygen, wordsOf } = await import("./transcribe.js");
|
||||
const ffmpeg = findFFmpeg();
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-heygen-asr-test-"));
|
||||
writeFileSync(join(dir, "index.html"), "");
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
@@ -70,6 +71,16 @@ describe("HeyGen transcription", () => {
|
||||
expect(JSON.parse(readFileSync(out, "utf8"))).toHaveLength(2);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("logs the job in the project, wherever the transcript goes", async () => {
|
||||
const audio = join(dir, "a.wav");
|
||||
execFileSync(ffmpeg!, ["-v", "error", "-y", "-f", "lavfi", "-i", "sine=f=300:d=1.5", audio]);
|
||||
vi.stubGlobal("fetch", async () => new Response(JSON.stringify({ data: job })));
|
||||
mkdirSync(join(dir, "work"));
|
||||
await transcribeWithHeygen(audio, join(dir, "work", "transcript.json"));
|
||||
const rows = readFileSync(join(dir, ".hyperframes/heygen-jobs.jsonl"), "utf8");
|
||||
expect(JSON.parse(rows.trim().split("\n")[0]!).what).toBe(`transcribe ${audio}`);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("keys the request and polls a pending transcription to the end", async () => {
|
||||
const audio = join(dir, "b.wav");
|
||||
execFileSync(ffmpeg!, ["-v", "error", "-y", "-f", "lavfi", "-i", "sine=f=200:d=1", audio]);
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
// The route runs inside the POST and takes an API key; the Idempotency-Key makes a re-run free.
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { dirname, join } from "node:path";
|
||||
import type { TranscriptionCreateRequest } from "../cloud/_gen/types.js";
|
||||
import { runFfmpeg } from "../measure/media.js";
|
||||
import { projectOf, runFfmpeg } from "../measure/media.js";
|
||||
import { idempotencyKey, runJob, type JobRoute } from "./v3.js";
|
||||
|
||||
const ROUTE: JobRoute = {
|
||||
@@ -53,7 +53,7 @@ export function wordsOf(job: Record<string, unknown>) {
|
||||
export async function transcribeWithHeygen(
|
||||
inputPath: string,
|
||||
transcriptPath: string,
|
||||
opts: { diarize?: boolean; projectDir?: string; onStatus?: (message: string) => void } = {},
|
||||
opts: { diarize?: boolean; onStatus?: (message: string) => void } = {},
|
||||
): Promise<HeygenTranscript> {
|
||||
const work = mkdtempSync(join(tmpdir(), "hf-heygen-asr-"));
|
||||
try {
|
||||
@@ -86,7 +86,10 @@ export async function transcribeWithHeygen(
|
||||
},
|
||||
idempotencyKey(ROUTE.path, [audio, String(diarize)]),
|
||||
{
|
||||
log: { projectDir: opts.projectDir, what: `transcribe ${inputPath}` },
|
||||
log: {
|
||||
projectDir: projectOf(dirname(transcriptPath)) ?? process.cwd(),
|
||||
what: `transcribe ${inputPath}`,
|
||||
},
|
||||
timeoutMs: 10 * 60_000,
|
||||
},
|
||||
);
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
import { mkdtempSync, writeFileSync } from "node:fs";
|
||||
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { CreateModelVideoRequest } from "../cloud/_gen/types.js";
|
||||
import type {
|
||||
CreateBackgroundRemovalRequest,
|
||||
CreateModelVideoRequest,
|
||||
} from "../cloud/_gen/types.js";
|
||||
import type { JobRoute } from "./v3.js";
|
||||
|
||||
vi.mock("../cloud/auth.js", () => ({
|
||||
@@ -55,6 +58,25 @@ describe("runJob", () => {
|
||||
expect(new Headers(calls[0]!.init.headers).get("Idempotency-Key")).toBe("hf-key");
|
||||
});
|
||||
|
||||
it("logs a job that is done at once one time, and a waited-for job at each end", async () => {
|
||||
const projectDir = mkdtempSync(join(tmpdir(), "hf-v3-log-"));
|
||||
const rows = () =>
|
||||
readFileSync(join(projectDir, ".hyperframes/heygen-jobs.jsonl"), "utf8")
|
||||
.trim()
|
||||
.split("\n")
|
||||
.map((row) => JSON.parse(row).status);
|
||||
const log = { projectDir, what: "a line" };
|
||||
vi.stubGlobal("fetch", async () => reply(200, { data: { id: "j1", status: "completed" } }));
|
||||
await runJob(X, {}, "k1", { log });
|
||||
expect(rows()).toEqual(["completed"]);
|
||||
const answers = ["pending", "completed"].map((status) =>
|
||||
reply(200, { data: { id: "j2", status } }),
|
||||
);
|
||||
vi.stubGlobal("fetch", async () => answers.shift()!);
|
||||
await runJob(X, {}, "k2", { log, intervalMs: 1 });
|
||||
expect(rows()).toEqual(["completed", "pending", "completed"]);
|
||||
});
|
||||
|
||||
it("turns refusals into what to do next", async () => {
|
||||
vi.stubGlobal("fetch", async () =>
|
||||
reply(402, { error: { code: "insufficient_credit", message: "no" } }),
|
||||
@@ -143,6 +165,43 @@ describe("runJob", () => {
|
||||
expect(keys).toEqual(["hf-broke", "hf-broke-1"]);
|
||||
});
|
||||
|
||||
it("puts the key in the body for a route that dedupes there, and moves it only past a failed job", async () => {
|
||||
const jobs = new Map<string, string>();
|
||||
const sent: string[] = [];
|
||||
let status = "completed";
|
||||
let lose = false;
|
||||
vi.stubGlobal("fetch", async (_url: string, init: RequestInit) => {
|
||||
if (init.method !== "POST") return reply(200, { data: { id: "j", status } });
|
||||
// like /v3/background-removals: the header is ignored, the body's request_id picks the job
|
||||
const id = String(JSON.parse(String(init.body)).request_id);
|
||||
sent.push(id);
|
||||
if (!jobs.has(id)) jobs.set(id, `job-${jobs.size}`);
|
||||
if (lose) return reply(408, { message: "timeout" });
|
||||
return reply(200, { data: { id: jobs.get(id), status } });
|
||||
});
|
||||
const cut: JobRoute = {
|
||||
path: "/v3/background-removals",
|
||||
start: (c, body, key, signal) =>
|
||||
c.createBackgroundRemoval({
|
||||
body: { ...(body as CreateBackgroundRemovalRequest), request_id: key },
|
||||
signal,
|
||||
}),
|
||||
status: (c, id, signal) => c.getBackgroundRemoval({ job_id: id, signal }),
|
||||
};
|
||||
const go = () => runJob(cut, { layers: ["foreground"] }, "hf-cut", { intervalMs: 1 });
|
||||
lose = true;
|
||||
await expect(go()).rejects.toThrow();
|
||||
lose = false;
|
||||
await go();
|
||||
expect(jobs.size).toBe(1);
|
||||
status = "failed";
|
||||
await expect(go()).rejects.toThrow();
|
||||
status = "completed";
|
||||
await go();
|
||||
expect(sent).toEqual(["hf-cut", "hf-cut", "hf-cut", "hf-cut-1"]);
|
||||
expect(jobs.size).toBe(2);
|
||||
});
|
||||
|
||||
it.each([401, 403, 408, 409, 425, 429])(
|
||||
"keeps the key after a %i, which says nothing about the job",
|
||||
async (status) => {
|
||||
|
||||
@@ -180,7 +180,7 @@ async function call(
|
||||
if (data == null) throw new Fix(`HeyGen answered ${route} with no readable answer`, resume);
|
||||
if (!isRecord(data))
|
||||
throw new Fix(`HeyGen answered ${route} with no job`, "run the same command again");
|
||||
const id = data.id ?? data.job_id ?? data.video_id ?? data.generation_id;
|
||||
const id = data.id ?? data.job_id ?? data.video_id ?? data.generation_id ?? data.lipsync_id;
|
||||
return {
|
||||
...data,
|
||||
id: typeof id === "string" ? id : "",
|
||||
@@ -239,7 +239,8 @@ export async function runJob(
|
||||
if (err instanceof Refused && !KEEPS_KEY.has(err.status)) markFailed(sent, scope);
|
||||
throw err;
|
||||
}
|
||||
if (opts.log) log(opts.log, { route, id: job.id, key: sent, status: job.status });
|
||||
const waits = RUNNING.has(job.status.toLowerCase());
|
||||
if (opts.log && waits) log(opts.log, { route, id: job.id, key: sent, status: job.status });
|
||||
const status = jobRoute.status;
|
||||
while (status && RUNNING.has(job.status.toLowerCase())) {
|
||||
if (Date.now() - started > (opts.timeoutMs ?? 15 * 60_000))
|
||||
|
||||
@@ -96,6 +96,17 @@ const rate = (text: string | undefined): number => {
|
||||
return b ? (a ?? 0) / b : (a ?? 0);
|
||||
};
|
||||
|
||||
/** A sound's length in seconds. */
|
||||
export function probeSeconds(path: string): number {
|
||||
if (!existsSync(path)) throw new Fix(`no file at ${path}`, "pass a sound from the project");
|
||||
const text = execFileSync(
|
||||
ffprobeBin(),
|
||||
["-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", "--", path],
|
||||
{ maxBuffer: MAX_BUFFER, windowsHide: true },
|
||||
).toString();
|
||||
return Number(text) || 0;
|
||||
}
|
||||
|
||||
/** Size (as displayed, rotation applied), clock and sound of a video. */
|
||||
// fallow-ignore-next-line complexity
|
||||
export function probeVideo(path: string): VideoInfo {
|
||||
|
||||
@@ -8,6 +8,7 @@ import { matchTokens } from "./match.mjs";
|
||||
const AUDIO_EXT = new Set([".mp3", ".wav", ".ogg", ".m4a", ".aac"]);
|
||||
const IMAGE_EXT = new Set([".jpg", ".jpeg", ".png", ".gif", ".webp", ".svg", ".ico"]);
|
||||
const VIDEO_EXT = new Set([".mp4", ".webm", ".mov"]);
|
||||
const TEMPLATE_FOLDERS = new Set(["previews", "textures"]);
|
||||
|
||||
function audioKind(filePath) {
|
||||
const words = filePath.toLowerCase().split(/[^a-z0-9]+/);
|
||||
@@ -104,6 +105,15 @@ export function adoptExistingAssets(projectDir) {
|
||||
return adopted;
|
||||
}
|
||||
|
||||
// A component's own pictures, and a template's textures and previews, are not media for the film to pick up.
|
||||
function belongsToTemplate(projectDir, rel) {
|
||||
const [folder, ...rest] = rel.split("/");
|
||||
if (rest.length === 0) return false;
|
||||
return (
|
||||
TEMPLATE_FOLDERS.has(folder) || existsSync(join(projectDir, "compositions", `${folder}.html`))
|
||||
);
|
||||
}
|
||||
|
||||
// Adopt a pre-existing assets/ file only when it shares a meaningful word with
|
||||
// the intent. The old test — `name.includes(intent) || intent.includes(name)` —
|
||||
// silently returned the WRONG file: "whoosh" grabbed a stray who.mp3, and a
|
||||
@@ -116,6 +126,7 @@ export function findExistingAsset(projectDir, intent, type) {
|
||||
const intentTokens = matchTokens(intent);
|
||||
if (intentTokens.size === 0) return null;
|
||||
for (const rel of walkDir(assetsDir)) {
|
||||
if (belongsToTemplate(projectDir, rel)) continue;
|
||||
const t = inferType(rel);
|
||||
if (!t || (type && t !== type)) continue;
|
||||
const stem = basename(rel, extname(rel));
|
||||
|
||||
@@ -51,6 +51,32 @@ function runTests() {
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("a template's own pictures are not search hits", () => {
|
||||
setup();
|
||||
drop("halftone-collage/clip-paper.png");
|
||||
drop("textures/paper-grain.png");
|
||||
drop("previews/paper-board.jpg");
|
||||
mkdirSync(join(tmp, "compositions"));
|
||||
writeFileSync(join(tmp, "compositions", "halftone-collage.html"), "x");
|
||||
assert.equal(findExistingAsset(tmp, "paper form", "image"), null);
|
||||
drop("img/paper-form.png");
|
||||
assert.equal(
|
||||
findExistingAsset(tmp, "paper form", "image").relativePath,
|
||||
"assets/img/paper-form.png",
|
||||
);
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("no search takes a component's sample page for its own, however many ask", () => {
|
||||
setup();
|
||||
drop("highlight-wipe/page-sample.png");
|
||||
mkdirSync(join(tmp, "compositions"));
|
||||
writeFileSync(join(tmp, "compositions", "highlight-wipe.html"), "x");
|
||||
assert.equal(findExistingAsset(tmp, "criminal complaint court filing page", "image"), null);
|
||||
assert.equal(findExistingAsset(tmp, "appeals court opinion first page", "image"), null);
|
||||
cleanup();
|
||||
});
|
||||
|
||||
test("a shared stopword alone does not match", () => {
|
||||
setup();
|
||||
drop("video/the-video.mp4");
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
// (e.g. "make an image with codex").
|
||||
|
||||
import { bgmProvider } from "./bgm-provider.mjs";
|
||||
import { OPENVERSE_SFX_PROVIDER, sfxProvider } from "./sfx-provider.mjs";
|
||||
import { heygenSfxProvider, OPENVERSE_SFX_PROVIDER, sfxProvider } from "./sfx-provider.mjs";
|
||||
import { bundledSfxProvider } from "./bundled-sfx-provider.mjs";
|
||||
import { imageProvider, iconProvider } from "./image-provider.mjs";
|
||||
import { brandProvider } from "./brand-provider.mjs";
|
||||
@@ -48,6 +48,7 @@ const P = (name, caps) => ({ name, network: true, paid: true, ...caps }); // rem
|
||||
const REGISTRY = {
|
||||
bgm: [N("heygen.audio.sounds", { search: bgmProvider.search })],
|
||||
sfx: [
|
||||
N("heygen.audio.sounds", { search: heygenSfxProvider.search }),
|
||||
N(OPENVERSE_SFX_PROVIDER, { search: sfxProvider.search }),
|
||||
A("bundled.sfx", { search: bundledSfxProvider.search }),
|
||||
],
|
||||
|
||||
@@ -33,12 +33,11 @@ test("listTypes exposes the v2 media types", () => {
|
||||
});
|
||||
|
||||
test("heygen provider is first for every type it serves", () => {
|
||||
for (const t of ["bgm", "image", "icon"]) {
|
||||
for (const t of ["bgm", "sfx", "image", "icon"]) {
|
||||
const first = getProviders(t)[0];
|
||||
assert.ok(first, `no enabled provider for ${t}`);
|
||||
assert.match(first.name, /^heygen/, `${t} first provider is ${first.name}`);
|
||||
}
|
||||
assert.ok(!getProviders("sfx").some((p) => p.name.startsWith("heygen")), "sfx never asks HeyGen");
|
||||
});
|
||||
|
||||
test("sanctioned providers only: heygen, Openverse, local mflux/kokoro/ltx, codex, design spec, logo tiers", () => {
|
||||
@@ -85,14 +84,14 @@ test("video cascade: HeyGen first, LTX local fallback, generate-only", async ()
|
||||
assert.equal(await runCapability("video", "search", "x", {}), null);
|
||||
});
|
||||
|
||||
test("sfx cascade: Openverse first, the bundled library is the offline fallback", () => {
|
||||
test("sfx cascade: HeyGen, then Openverse, the bundled library is the offline fallback", () => {
|
||||
const ps = getProviders("sfx");
|
||||
assert.deepEqual(
|
||||
ps.map((p) => p.name),
|
||||
["openverse.audio", "bundled.sfx"],
|
||||
["heygen.audio.sounds", "openverse.audio", "bundled.sfx"],
|
||||
);
|
||||
assert.ok(ps[0].network && !ps[0].paid, "Openverse is free and network-only");
|
||||
assert.ok(!ps[1].network, "bundled SFX remain available offline");
|
||||
assert.ok(ps[1].network && !ps[1].paid, "Openverse is free and network-only");
|
||||
assert.ok(!ps[2].network, "bundled SFX remain available offline");
|
||||
});
|
||||
|
||||
test("ctx.provider forces one generator (e.g. 'make an image WITH codex')", async () => {
|
||||
|
||||
@@ -1,7 +1,30 @@
|
||||
import { heygenSearch } from "./heygen-search.mjs";
|
||||
import { OpenverseUnavailableError, searchOpenverseSfx } from "./openverse.mjs";
|
||||
|
||||
export const OPENVERSE_SFX_PROVIDER = "openverse.audio";
|
||||
|
||||
export const heygenSfxProvider = {
|
||||
async search(intent) {
|
||||
// Good effects score 0.5-0.67, under the server's default floor of 0.7.
|
||||
const results = heygenSearch("audio sounds list", intent, {
|
||||
type: "sound_effects",
|
||||
minScore: 0.4,
|
||||
});
|
||||
if (!results) return null;
|
||||
const best = results[0];
|
||||
return {
|
||||
url: best.audio_url,
|
||||
source: "search",
|
||||
metadata: {
|
||||
description: best.description || intent,
|
||||
duration: best.duration || null,
|
||||
provider: "heygen.audio.sounds",
|
||||
provenance: { track_id: best.id, score: best.score, query: intent },
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
export const sfxProvider = {
|
||||
async search(intent) {
|
||||
let sound;
|
||||
|
||||
@@ -1265,7 +1265,9 @@ async function reuseGlobal(shaArg) {
|
||||
process.exit(2);
|
||||
}
|
||||
if (!rec) {
|
||||
console.error(`error: no reusable global asset matches sha "${shaArg}"`);
|
||||
console.error(
|
||||
`error: no reusable global asset matches sha "${shaArg}". --reuse takes a sha from \`resolve --candidates\`; to skip the caches, pass --provider <name> instead`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
// Do not import an asset under the wrong type.
|
||||
|
||||
@@ -18,6 +18,7 @@ MP4.
|
||||
- `/slideshow` — a presentation or pitch deck. Output is a navigable deck, not a rendered MP4.
|
||||
- `/remotion-to-hyperframes` — bring existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, or renders from other tools) and verify it frame by frame against the original.
|
||||
- `/reverse-any-video` — rebuild any video file as an editable HyperFrames project and check it against the original frame by frame (HyperFrames Pro).
|
||||
- `/style-sampling` — make a new episode on your topic in one of five proven styles (HyperFrames Pro).
|
||||
|
||||
**Changing how real footage or images look or reveal?** Load `/media-use` and read its `references/media-treatments.md` before editing, even when the request only says dark, flat, boring, retro, private, or “make the reveal cooler.” It governs how footage is treated, never whether media may be used. Use canonical media treatments and seek-safe motion; do not improvise equivalent CSS/SVG filters or overlays.
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@ MP4.
|
||||
- `/slideshow` — a presentation or pitch deck. Output is a navigable deck, not a rendered MP4.
|
||||
- `/remotion-to-hyperframes` — bring existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, or renders from other tools) and verify it frame by frame against the original.
|
||||
- `/reverse-any-video` — rebuild any video file as an editable HyperFrames project and check it against the original frame by frame (HyperFrames Pro).
|
||||
- `/style-sampling` — make a new episode on your topic in one of five proven styles (HyperFrames Pro).
|
||||
|
||||
**Changing how real footage or images look or reveal?** Load `/media-use` and read its `references/media-treatments.md` before editing, even when the request only says dark, flat, boring, retro, private, or “make the reveal cooler.” It governs how footage is treated, never whether media may be used. Use canonical media treatments and seek-safe motion; do not improvise equivalent CSS/SVG filters or overlays.
|
||||
|
||||
|
||||
@@ -110,7 +110,7 @@ const FETCH_TIMEOUT_MS = 4000;
|
||||
// "call /media-use" preambles). These must be present and current
|
||||
// for any video work, so `init` / `skills update` keep them fresh.
|
||||
// on-demand — everything else: decks, Remotion ports, video rebuilds
|
||||
// (reverse-any-video) and optional integrations (figma).
|
||||
// (reverse-any-video, style-sampling) and optional integrations (figma).
|
||||
// They install when named (`hyperframes skills update <name>`),
|
||||
// instead of being sprayed onto every machine that runs `init`.
|
||||
|
||||
|
||||
@@ -37,6 +37,7 @@ describe("a bundled family named only in the font shorthand loads", () => {
|
||||
["style, variant, weight, size", 'p { font: italic small-caps bold 1.5rem "Inter", serif; }'],
|
||||
["a keyword size", "p { font: bold large Inter; }"],
|
||||
["a calc() size", "p { font: calc(10px + 2vw) Inter; }"],
|
||||
["a nested calc() size", 'p { font: calc(var(--size) * 1px)/1 "Inter"; }'],
|
||||
])("%s", async (_case, css) => {
|
||||
expect(await inject(css)).toMatch(/@font-face\s*\{[^}]*font-family:\s*"Inter"/i);
|
||||
});
|
||||
|
||||
@@ -268,11 +268,23 @@ const FONT_SIZE_KEYWORD =
|
||||
/^(?:xx-small|x-small|small|medium|large|x-large|xx-large|xxx-large|smaller|larger)$/i;
|
||||
const FONT_SIZE_TOKEN = /^(?:\d*\.?\d+[a-z%]+|(?:calc|min|max|clamp|var)\(.*\))(?:\/\S+)?$/i;
|
||||
|
||||
function maskParens(value: string): string {
|
||||
let depth = 0;
|
||||
return value.replace(/[\s\S]/g, (char) => {
|
||||
if (char === "(") depth += 1;
|
||||
const inside = depth > 0;
|
||||
if (char === ")") depth -= 1;
|
||||
return inside ? "_" : char;
|
||||
});
|
||||
}
|
||||
|
||||
function fontShorthandFamilies(value: string): string {
|
||||
for (const token of value.matchAll(/(?:[^\s(]|\([^)]*\))+/g)) {
|
||||
const size = token[0].split("/")[0] ?? "";
|
||||
if (FONT_SIZE_KEYWORD.test(size) || FONT_SIZE_TOKEN.test(token[0])) {
|
||||
return value.slice(token.index + token[0].length).trim();
|
||||
for (const { 0: masked, index } of maskParens(value).matchAll(/\S+/g)) {
|
||||
const end = index + masked.length;
|
||||
const token = value.slice(index, end);
|
||||
const size = token.split("/")[0] ?? "";
|
||||
if (FONT_SIZE_KEYWORD.test(size) || FONT_SIZE_TOKEN.test(token)) {
|
||||
return value.slice(end).trim();
|
||||
}
|
||||
}
|
||||
return "";
|
||||
|
||||
@@ -20,6 +20,7 @@ export const CORE_SKILLS = {
|
||||
slideshow: 5120,
|
||||
"remotion-to-hyperframes": 5120,
|
||||
"reverse-any-video": 5120,
|
||||
"style-sampling": 5120,
|
||||
};
|
||||
|
||||
/** @param {string} skill */
|
||||
|
||||
@@ -254,7 +254,7 @@ fi
|
||||
say "Verifying skill installation..."
|
||||
|
||||
SKILLS=(hyperframes hyperframes-cli hyperframes-registry media-use \
|
||||
figma slideshow remotion-to-hyperframes reverse-any-video)
|
||||
figma slideshow remotion-to-hyperframes reverse-any-video style-sampling)
|
||||
|
||||
MISSING=()
|
||||
check_skill() { if [[ -d "$SKILLS_DIR/$1" ]]; then ok "$SKILLS_DIR/$1/"; else MISSING+=("$1"); fi; }
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
"files": 13
|
||||
},
|
||||
"media-use": {
|
||||
"hash": "23d3e8801d9cb398",
|
||||
"hash": "48db8eb3e041c589",
|
||||
"files": 117
|
||||
},
|
||||
"remotion-to-hyperframes": {
|
||||
@@ -32,6 +32,10 @@
|
||||
"slideshow": {
|
||||
"hash": "c8a591e2aa7a6d86",
|
||||
"files": 2
|
||||
},
|
||||
"style-sampling": {
|
||||
"hash": "2c7a5d72f475437c",
|
||||
"files": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,13 +20,13 @@ It prints one line, `resolved <id> → <path> (<type>, <metadata>)`. List reusab
|
||||
| Need | Free | HyperFrames credits | Local or the person's own key (only when they ask) |
|
||||
| --- | --- | --- | --- |
|
||||
| Music (`bgm`) | HeyGen catalog of 10k+ tracks (signed in) | Generated track | Local MusicGen |
|
||||
| Sound effects (`sfx`) | Openverse (credit CC BY sounds); 21 bundled offline | | |
|
||||
| Sound effects (`sfx`) | HeyGen (signed in); Openverse (credit CC BY); bundled | | |
|
||||
| Images, icons | HeyGen asset search, 75k+ (signed in) | Generated images | Local FLUX (mflux), if installed |
|
||||
| Logos (`logo`) | theSVG, GitHub avatar, favicon; never redrawn | | |
|
||||
| Voiceover (`voice`) | Voice Turbo, Free plan: 3 min a month | Voice Turbo on a paid plan, other HeyGen voices, cloning | `npx hyperframes tts` (Kokoro on device) |
|
||||
| A presenter speaking | | Avatar video (`--type video`) | Local LTX for generic clips, if installed |
|
||||
| Transcripts | `npx hyperframes transcribe` (Parakeet or whisper) | | |
|
||||
| Cutouts | | `heygen background-removal` | `npx hyperframes remove-background` |
|
||||
| Cutouts | | Video: `--type cutout` | `npx hyperframes remove-background` |
|
||||
| Grades, LUTs, looks | `npx hyperframes media-treatment` | | |
|
||||
|
||||
**Sign in.** Every time a request needs a HeyGen model or the catalog, run `npx hyperframes auth status` first. Not signed in: tell the person in one line that sign-in is free and gives the music catalog, asset search and 3 min of Voice Turbo a month on the Free plan, ask them to type an exclamation mark and `npx hyperframes auth login` in the prompt, and wait until they say it is done. Use local tools only when the person asks, not as a fallback.
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
| Credential | Pays with | Covers |
|
||||
| -------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------- |
|
||||
| API key | API credits | everything below |
|
||||
| Sign-in (OAuth) | the plan's allowance on subscription credits | avatar videos, video shots, images and voices; music and transcription take an API key |
|
||||
| Sign-in (OAuth) | the plan's allowance on subscription credits | avatar videos, video shots, lipsync, cut-outs, images, voices, music and transcription |
|
||||
| Inside the HyperFrames app | the app | whatever the app allows; nothing to set up |
|
||||
|
||||
A refusal names its fix: `api_key_required` needs an API key, `402` or `insufficient_credit` needs a top-up, `hf_usage_exhausted` means the plan's allowance is used up.
|
||||
@@ -28,10 +28,16 @@ npx hyperframes media-use generate --type video --first-frame assets/s03-first.p
|
||||
# pixels and seconds) blanks text in the guide, which HeyGen would copy. A guide with a cut inside is refused; pass
|
||||
# --allow-cuts only when you have looked at those frames and they are one shot
|
||||
npx hyperframes media-use generate --type video --reference assets/person.png --audio assets/s02-voice.m4a --motion assets/s02-original.mp4 --mask 110,615,390,165,0.4-2.0 --prompt "talks to camera, seated" --seconds 8 --aspect 9:16 --out assets/s02-person.mp4
|
||||
# a speaking shot follows its own re-paced audio, not --audio: to land it on a set voice, redo its mouth to that voice
|
||||
npx hyperframes media-use generate --type lipsync --video assets/s02-person.mp4 --audio assets/s02-voice.mp3 --out assets/s02-lips.mp4
|
||||
# the same for a person who does not speak: no --audio, the guide clip carries the motion
|
||||
npx hyperframes media-use generate --type video --reference assets/person.png --motion assets/s05-original.mp4 --prompt "holds up a jar, smiles" --seconds 6 --aspect 9:16 --out assets/s05-person.mp4
|
||||
# a filmed person (or object) cut out of its background, as a transparent WebM
|
||||
npx hyperframes media-use generate --type cutout --video assets/host.mp4 --out assets/host-cutout.webm
|
||||
# a person in a photo speaking a script in a HeyGen voice
|
||||
npx hyperframes media-use generate --type avatar --photo assets/person.png --script "…" --voice English --out assets/person.mp4
|
||||
# a finished voice stays exactly as it is: --audio replaces --script and --voice, --avatar <look id> replaces --photo, --prompt moves a photo
|
||||
npx hyperframes media-use generate --type avatar --photo assets/person.png --audio assets/vo.mp3 --prompt "seated, talks with both hands" --aspect 16:9 --out assets/host.mp4
|
||||
# a voice reading text; --voice takes a language (HeyGen's first voice in it, named in the output) or a voice id
|
||||
npx hyperframes media-use generate --type voice --script "…" --voice Chinese --out assets/vo.mp3
|
||||
# music, 30, 60, 120 or 180 s (instrumental unless --vocals), trimmed to length in the composition
|
||||
@@ -40,13 +46,15 @@ npx hyperframes media-use generate --type music --prompt "warm lo-fi, soft keys"
|
||||
|
||||
The file lands at `--out` and the media ledger records it as generated, with the route, model, job, prompt and the files it was made from. Pictures and shots carry no text or logos: ask for clean plates and draw words in HyperFrames.
|
||||
|
||||
| Type | Model | Price |
|
||||
| ------ | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| image | nano-banana-2 (default), or gpt-image-2.5 with `--model` | $0.067 an image; gpt-image-2.5 $0.036–0.053 by aspect, plus $0.0082 a reference after the first |
|
||||
| video | heygen-video-1, 768p | $0.02 a second from a prompt or first frame; $0.05 a second with a reference picture, voice or guide clip, and the guide's seconds count too (1080p costs about twice) |
|
||||
| avatar | talking photo | $4.83 a minute of speech |
|
||||
| music | lyria-3-pro | $0.08 a track |
|
||||
| voice | the voice's engine | charged per character, not priced ahead |
|
||||
| Type | Model | Price |
|
||||
| ------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| image | nano-banana-2 (default), or gpt-image-2.5 with `--model` | $0.067 an image; gpt-image-2.5 $0.036–0.053 by aspect, plus $0.0082 a reference after the first |
|
||||
| video | heygen-video-1, 768p | $0.02 a second from a prompt or first frame; $0.05 a second with a reference picture, voice or guide clip, and the guide's seconds count too (1080p costs about twice) |
|
||||
| lipsync | HeyGen lipsync, speed | 7 credits a second of the video on a plan |
|
||||
| avatar | talking photo | $4.83 a minute of speech |
|
||||
| cutout | HeyGen background removal | not priced ahead |
|
||||
| music | lyria-3-pro | $0.08 a track |
|
||||
| voice | the voice's engine | charged per character, not priced ahead |
|
||||
|
||||
`--plan`'s `usd` is what an API key is charged (HeyGen rounds up to units of $1/60); its `credits` is what a HyperFrames plan charges, on the plan's own rates: a video shot's seconds and guide seconds, plus each picture past the first two. Without `--voice`, generate lists voice ids to pick from on the sign-in alone; a cloned voice from `heygen voice clone create` (`operations.md`).
|
||||
|
||||
@@ -56,4 +64,4 @@ The file lands at `--out` and the media ledger records it as generated, with the
|
||||
npx hyperframes transcribe interview.mp4 --engine heygen --speakers
|
||||
```
|
||||
|
||||
Word times from HeyGen's speech model, each word with its speaker under `--speakers`, in the same `transcript.json` local engines write. It takes an API key. The local engines (`--engine auto`) are free and good for one clear voice; HeyGen is the better choice for several speakers, noise or music under the voice.
|
||||
Word times from HeyGen's speech model, each word with its speaker under `--speakers`, in the same `transcript.json` local engines write. The local engines (`--engine auto`) are free and good for one clear voice; HeyGen is the better choice for several speakers, noise or music under the voice.
|
||||
|
||||
@@ -48,19 +48,20 @@ scenedetect -i in.mp4 detect-adaptive list-scenes # pip install scened
|
||||
|
||||
## Transforms with a quality choice (process)
|
||||
|
||||
These have a local option AND a higher-quality HeyGen-CLI option. Run the local
|
||||
one for free/offline; use the HeyGen CLI when quality matters. Showing the user
|
||||
a **side-by-side** (local vs HeyGen) is the honest way to let them choose.
|
||||
These have a local option AND a higher-quality HeyGen option. Run the local
|
||||
one for free/offline; use HeyGen when quality matters: `media-use generate` on the
|
||||
sign-in alone, the rest through the `heygen` CLI. Showing the user a
|
||||
**side-by-side** (local vs HeyGen) is the honest way to let them choose.
|
||||
|
||||
| Op | Local (free) | HeyGen CLI (quality) |
|
||||
| ----------------------- | -------------------------------------------------- | --------------------------- |
|
||||
| Background removal | `hyperframes remove-background in.png` (u2net) | `heygen background-removal` |
|
||||
| Upscale | `realesrgan-ncnn-vulkan -i in.png -o out.png -s 4` | n/a |
|
||||
| Lipsync (dub) | n/a | `heygen lipsync` |
|
||||
| Translate | n/a | `heygen video-translate` |
|
||||
| Clone a voice | n/a | `heygen voice clone create` |
|
||||
| Text to video | n/a | `heygen video-agent create` |
|
||||
| Clips from a long video | n/a | `heygen ai-clipping` |
|
||||
| Op | Local (free) | HeyGen (quality) |
|
||||
| ----------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||||
| Background removal | `hyperframes remove-background in.png` (u2net) | `hyperframes media-use generate --type cutout --video in.mp4`; stills stay local |
|
||||
| Upscale | `realesrgan-ncnn-vulkan -i in.png -o out.png -s 4` | n/a |
|
||||
| Lipsync (dub) | n/a | `hyperframes media-use generate --type lipsync` |
|
||||
| Translate | n/a | `heygen video-translate` |
|
||||
| Clone a voice | n/a | `heygen voice clone create` |
|
||||
| Text to video | n/a | `heygen video-agent create` |
|
||||
| Clips from a long video | n/a | `heygen ai-clipping` |
|
||||
|
||||
After any op: `resolve --from out.ext --type <type>` to register the derived
|
||||
asset (it records provenance and auto-promotes to the global cache).
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: style-sampling
|
||||
description: Make a new episode in a proven video style - Collage Explainer, Finance Explainer, Animated Science Explainer, 3D Brand Explainer, 3D True Crime - on the person's topic. Use for "an episode of", "in the style of", "sample this style", 「用这个风格做一集」. One video to copy is reverse-any-video.
|
||||
---
|
||||
|
||||
**Plugin installs:** Before setup or freshness commands, follow [plugin execution rules](../hyperframes/references/plugin-installation.md) when this skill is inside a HyperFrames plugin.
|
||||
|
||||
# A new episode from a style template
|
||||
|
||||
A good episode is one the series' own viewers would take for the next upload: its structure, pacing, look, graphics and sound, on a new topic, in the person's voice and, when the series has a presenter, their own face. Never use a creator's face, voice, footage, logo or show name.
|
||||
|
||||
Style templates are a HyperFrames Pro feature. The CLI pulls them and the server checks the plan.
|
||||
|
||||
| Template | Slug | Series | Makes | The person brings |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Collage Explainer | `collage-explainer` | `american-policy` | English explainer on one American policy or history question | Voice |
|
||||
| Finance Explainer (财经讲解) | `finance-explainer` | `deep-dive` | 中文主持人讲透一个国家、商品或机构 | Voice and face |
|
||||
| Animated Science Explainer | `animated-science-explainer` | `science-explainer` | English fully drawn science explainer | Voice |
|
||||
| 3D Brand Explainer | `3d-brand-explainer` | `brand-explained` | English breakdown of a brand or design, in one graphic world with real 3D | Voice |
|
||||
| 3D True Crime | `3d-true-crime` | `crime-hunt` | English reconstruction of a hunt or a closed case, in a dark 3D world | Voice |
|
||||
|
||||
## Which template
|
||||
|
||||
The person named a template, or their topic fits only one: go on with it. Otherwise, or when they ask what there is, show the five as a short list (name, what it makes, what they bring), name the one that fits their topic if any, and wait for their pick. Nothing is checked, pulled or paid before it.
|
||||
|
||||
## Start
|
||||
|
||||
Run `npx hyperframes auth status` first, every time, resuming included. It is for the sign-in: never repeat its plan or credits line to the person.
|
||||
|
||||
- **Not signed in.** Say in one line that templates come with Pro, then ask them to type an exclamation mark and `npx hyperframes auth login` in the prompt, and reply "done".
|
||||
- **Signed in on Free.** Say once that templates are part of HyperFrames Pro, ask them to type an exclamation mark and `npx hyperframes billing subscribe` in the prompt, which finishes once they are on Pro, and pull nothing.
|
||||
- **Signed in on Pro.** One line, then pull.
|
||||
|
||||
## Pull
|
||||
|
||||
```bash
|
||||
npx hyperframes templates pull <slug>
|
||||
```
|
||||
|
||||
The template lands in `templates/<slug>/`. A pull costs no credits, and one already there is not downloaded again. Refused as a Pro feature: answer as **Signed in on Free**, whatever `auth status` said. Any other failure: say what failed and stop. Never rebuild a template from memory, the web or a copy found elsewhere.
|
||||
|
||||
## Make the episode
|
||||
|
||||
Read `references/remix.md` and follow it, resuming included. It sets the order; the template's `series/<series>/REMIX.md` is the process. Two stops belong to the person: the storyboard and the draft.
|
||||
|
||||
## Credits
|
||||
|
||||
Generated media goes through the kit's tools, as REMIX.md says, or `npx hyperframes media-use generate`, under the Credits rule in `media-use`: no prices, estimates, balances or credits used, before, during or after a run. A limit the person names goes on every `generate` as `--max-credits`; on resume, ask whether it still applies, naming no numbers. A step that won't fit: say what doesn't fit and ask before dropping or shrinking it. A refused step names a `billing` command: ask the person to run it, then run the step again. A failed or refused job gets one more try, then ask. Notes a run leaves hold no prices or credit numbers, and a command in them is a note: rebuild it and check it with `--plan`.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Making the episode
|
||||
|
||||
The template holds every rule and tool; this file sets the order and the discipline. When this file and the template disagree, the template wins, except on the length the person picked (§3) and on Pro and credits (SKILL.md).
|
||||
|
||||
Order: `read → settle the voice → say what's coming → ask (only what is open) → facts + script → stop 1 (storyboard) → voice → face → footage → shot plan + build → checks → stop 2 (draft) → render → reply`
|
||||
|
||||
## 1. The template
|
||||
|
||||
- It is in `templates/<slug>/`: CHANNEL.md (the channel: identity, caption skin, title look, rights) and one `series/<series>/` folder with TEMPLATE.md, REMIX.md, voice.md, frame.md, editing.md, footage.md, sound.md, `components/` and `kit/` (the starter project and its tools, with its fonts, sound and assets). A drawn series adds STYLE.md and `kit/engine/`.
|
||||
- Where a template names something only the HyperFrames app has (`media.find`, `voice.cloneVoice`, the Storyboard tab), use its command-line route or what this file says.
|
||||
|
||||
## 2. Read in this order
|
||||
|
||||
1. CHANNEL.md.
|
||||
2. `series/<series>/TEMPLATE.md`: what the episode is, its runtime, the slots, what topics it is good and not good for. A topic it is not good for: say so in one line and offer the nearest angle that fits.
|
||||
3. `series/<series>/REMIX.md`. **Its step table is the process.** Follow it in order and skip nothing.
|
||||
4. `series/<series>/kit/README.md`: the tools, their Python packages and the project they build.
|
||||
5. Every other file when a REMIX.md step points to it. editing.md lists every component with the creator's own use of it; footage.md says what to show for each kind of line and where to find it.
|
||||
|
||||
## 3. What to ask, once
|
||||
|
||||
The person's message carries what it carries. Ask once, together, only what is still open: the topic, the length, the voice and, for a presented series, the face.
|
||||
|
||||
- **Never the creator's face, voice, footage, wordmark or show name.**
|
||||
- **A presenter runs on the plan.** Their own HeyGen avatar if they have one, otherwise Avatar IV from their photo, both through `npx hyperframes media-use generate --type avatar` with the finished voice, so the lips land on it. HeyGen Video 1 is the fallback when neither works.
|
||||
- **Settle the voice before any research.** A HeyGen voice id the person gave is the voice: use it and ask nothing. Anything else: with `heygen` signed in, look in `heygen voice list`. One clone: use it. Several: ask which. None, or no `heygen`: ask once for their voice id, a clone they make in HeyGen from 30 s to 3 min of their own reading, or their own recorded narration; with none of these, say in one line that you'll use a HeyGen stock voice (`npx hyperframes media-use resolve --type voice`), name it in the credits as AI-generated, and go on. Nothing else starts until this is settled.
|
||||
- **Then say what's coming** in one short message that is not a question, and go on without waiting: the working title and hook, the length, and roughly how long it takes. No prices.
|
||||
- **The length is the person's, and it beats the template's runtime.** Offer about 1 minute first: the cold open, the title beat and one chapter, the end card short. 2 to 4 minutes: REMIX.md's short cut. A full episode: the series' own runtime. Their own words ("90 seconds", "5 min"): that length. No length answered and none in the topic: about 1 minute, and offer the longer cut in the reply. Scale everything to it: script words (the template's script stats at that runtime), facts, sections, music cues, review rounds (one for a minute, two for a full episode); never pad or trim to reach the series' runtime instead.
|
||||
- Everything else REMIX.md §0 asks (chapters, language) takes REMIX.md's default; write each one, and the length, with why, in the project's `CLAUDE.md`.
|
||||
|
||||
## 4. Facts and script, then stop 1
|
||||
|
||||
- Facts first, per REMIX.md: every number, date and quote with its source. A research sub-agent is fine.
|
||||
- Write the script to voice.md and the series' script anatomy; run the script checks REMIX.md names.
|
||||
- **Stop 1:** pose the script and the shot plan as stills on the board, written as the hyperframes skill's `references/storyboard.md` says: one still per beat, its line under it, what each shot shows and which component makes it. A series kit builds its film from the voice's timing, so never voice to get a board: pose the beats on a plain page with the real components, words and fonts, and give each its estimated seconds. Point the person to `boards/storyboard.html` and wait for their go.
|
||||
|
||||
## 5. Voice and face
|
||||
|
||||
- Voice: the person's voice id, through the kit's voice tool, as REMIX.md says. Then the kit's timing and pause tools. The voice is final before any presenter.
|
||||
- **Audition before the whole script:** one paragraph (the kit's voice tool, as REMIX.md says), measure speech-only words a minute, and set the speed that lands on the series' rate. An instant clone takes no speed: change the words. Voice the script once; re-voice only the paragraphs that changed.
|
||||
- Face (presented series): REMIX.md's host step, on the final voice: their own avatar, Avatar IV from their photo, or the HeyGen Video 1 fallback. Their own takes: cut them per editing.md's presenter rules.
|
||||
- A changed line after the presenter take exists means the whole take is made again, or the lips drift.
|
||||
|
||||
## 6. Footage, shot plan, build
|
||||
|
||||
- Find footage per footage.md, from the sources it names (Wikimedia Commons, Openverse, NARA, Library of Congress, Prelinger, Pexels, Mixkit, official media). Download with curl. One sub-agent per chapter is fine. Every file gets a licence line in the record REMIX.md names; credit what needs credit.
|
||||
- Never take footage from YouTube, TikTok, Instagram or X unless its licence is stated.
|
||||
- Write the shot plan and assemble with the kit's tools, as REMIX.md says. **Before any graphic, find the creator's precedent for that kind of moment in editing.md; with none, don't add it.** A component the template lacks: use the closest one and note it in GAPS.md.
|
||||
- Keep `templates/` as pulled.
|
||||
|
||||
## 7. Checks, stop 2, render
|
||||
|
||||
- One heavy job at a time (renders, long ffmpeg, transcription): renders with `--workers 1`, ffmpeg with `-threads 2 -nostdin`. For a change, render only that window.
|
||||
- Run REMIX.md's checks. Report each of our numbers beside the series' own: cuts a minute, pause density, longest hold, loudness, sound effects on picture changes, the script fingerprint. Name every check that failed.
|
||||
- **Stop 2:** ask once on the draft: a contact sheet and `npx hyperframes preview`, with what is still a stand-in.
|
||||
- Then the one full render REMIX.md's delivery calls for. A render over 10 minutes starts detached and is watched through its log, because a render stops when the shell that started it exits: `bash -c 'set -m; HYPERFRAMES_RENDER_DETACHED=1 npx hyperframes render -o <out> < /dev/null > render.log 2>&1 &'`.
|
||||
|
||||
## 8. Reply and hand back
|
||||
|
||||
- Say what the episode is, its length (and, for a short cut, that the full episode is one ask away), every stand-in, every check that failed and why, and the footage licence table. Update the project's `CLAUDE.md` with how it was made.
|
||||
- Keep LOG.md (each step and how long, no prices or credit numbers) and GAPS.md (where the template was missing, unclear or wrong, with file and section) in the project root. GAPS.md is how the template gets better: never skip it.
|
||||
Reference in New Issue
Block a user