ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-10-30 09:16:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(skills): satisfy CI gates for reverse-any-video

Browse Source 
- docs: list diff, measure, analyze and billing in "Find your command" in
  docs/packages/cli.mdx, each with its own section.
- skills: bring media-use/SKILL.md under its 5120 B budget by pointing setup,
  direct-recipe and treatment detail at the references that already hold it;
  the sign-in, credits, --max-credits and billing rules stay in SKILL.md.
- comments: register transcript.json and REVERSE.md as project files for the
  citation checker, and drop a history note from face.test.ts.
- comments: cut restating prose or move it into names so every changed file is
  at or under its comment share. Allowlist the two deleted remotion scripts in
  check-no-main-deletions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
James
2026-10-29 23:30:10 +00:00
co-authored by Claude Opus 5.5
parent 395512802f
commit d016247e0e
18 changed files with 161 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/packages/cli.mdx
+100 -1
View File
@@ -28,11 +28,12 @@ npm install -g hyperframes
| Bring in source material | [`capture`](#capture), [`media-use`](#media-use), [`transcribe`](#transcribe), [`tts`](#tts), [`remove-background`](#remove-background), [`media-treatment`](#media-treatment), [`beats`](#beats), [`audio`](#audio), [`normalize-audio`](#normalize-audio), [`models`](#models) |
| Look at it, or share it | [`preview`](#preview), [`present`](#present-and-play), [`play`](#present-and-play), [`publish`](#publish), [`open`](#open-and-catch-up), [`catch-up`](#open-and-catch-up) |
| Find problems | [`lint`](#lint), [`check`](#check), [`snapshot`](#snapshot), [`keyframes`](#keyframes), [`compare`](#compare-and-grade-compare), [`grade-compare`](#compare-and-grade-compare) |
| Rebuild a video | [`measure`](#measure), [`analyze`](#analyze), [`diff`](#diff) |
| Edit from the terminal | [`timeline`](#timeline), [`history`](#history) |
| Make a file | [`render`](#render), [`benchmark`](#benchmark) |
| Render somewhere else | [`cloud`](#hyperframes-cloud), [`lambda`](#hyperframes-lambda), [`cloudrun`](#hyperframes-cloudrun) |
| Fix your setup | [`doctor`](#doctor), [`browser`](#browser), [`upgrade`](#upgrade), [`docs`](#docs), [`info`](#info-and-compositions), [`compositions`](#info-and-compositions), [`clean`](#clean) |
| Connect other tools | [`auth`](#hyperframes-auth), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
| Connect other tools | [`auth`](#hyperframes-auth), [`billing`](#billing), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
`validate`, `inspect`, and `layout` still work but are
[deprecated](#deprecated-validate-inspect-and-layout). Use `check` instead.
@@ -1503,6 +1504,88 @@ session, and it is where new work should go. Their flags are unchanged — run
`npx hyperframes inspect --help` if you are maintaining automation that still
calls them.
## Rebuild an existing video
These commands serve the `reverse-any-video` skill: measure a video, have HeyGen
draft its timeline, then score your rebuild against it.
### `measure`
Measures a video file and prints JSON. Each measurement is a subcommand; run
`npx hyperframes measure <subcommand> --help` for its flags.
```bash
npx hyperframes measure probe ref.mp4
npx hyperframes measure cuts ref.mp4
npx hyperframes measure frames ref.mp4 --events
npx hyperframes measure box ref.mp4 --start 1 --end 2.5 --roi 200,300,600,200 --track
```
| Subcommand | What it reports |
| ------------ | --------------------------------------------------------------------------------------- |
| `probe` | Displayed size, frame rate, duration, sound, and what to fix before rebuilding |
| `cuts` | Hard cuts from the frame-to-frame difference, and the shots between them |
| `events` | Cuts, flashes, appears, disappears and holds; `--map` adds the sound's hits |
| `frames` | Frames as pictures: chosen moments, an overview sheet, a range, or strips around events |
| `trail` | An onion skin of a window's frames, the moving part's path, and a heat map |
| `extract` | A still or a clip cut from the video, cropped to a region, as project media |
| `box` | One graphic's box, ink, colours, text lines and cap height; over a range, its moves |
| `ink` | Where text ink sits in a region, for timing and font size |
| `curve` | Each move of one element: start, duration, overshoot and the closest GSAP ease |
| `rhythm` | Moving share, holds, update rate, cuts, and each shot's zoom, pan or handheld |
| `objects` | Separate elements at one time: count, sizes and arrangement |
| `energy` | How much two clips move, and whether at the same moments |
| `look` | Brightness, contrast, saturation and warmth, alone or against another video |
| `track` | One textured thing followed frame by frame from its box |
| `flow` | One element's motion from optical flow inside its box |
| `face` | The face's size and position over a window |
| `sync` | The lag between mouth motion and the voice |
| `continuity` | Seams your rebuild has and the original does not |
### `analyze`
HeyGen's video model watches a video and returns JSON: with `--preset reverse`
(the default) a draft timeline, with `--prompt` an answer to your question and
the moments it cites. It uploads the video and needs a HeyGen sign-in or API key.
```bash
npx hyperframes analyze .reverse/source.mp4 --preset reverse
npx hyperframes analyze clip.mp4 --prompt "Does the logo animate in or cut in?"
```
| Flag | Description |
| ------------------- | ------------------------------------------------------------------- |
| `--preset` | `reverse`: the draft of a reverse spec (default) |
| `--prompt` | Ask your own question instead |
| `--start` / `--end` | The window in seconds (default: the whole video) |
| `--fps` | Frames a second the model sees (default 6; long parts get fewer) |
| `--model` | HeyGen's analysis model |
| `--out` | Where to write the draft (default `.reverse/<video>-analysis.json`) |
### `diff`
Scores a render, or a project folder without rendering it, against the original
video frame by frame: SSIM, the worst moments, missed and extra cuts, duration
and the sound's offset. It exits non-zero when the scores fall under the bar.
```bash
npx hyperframes diff .reverse/source.mp4 renders/final.mp4
npx hyperframes diff .reverse/source.mp4 . --at 0.5,2.4,3.1
npx hyperframes diff .reverse/source.mp4 renders/final.mp4 --segments REVERSE.md --sbs
```
| Flag | Description |
| ----------------------------------------- | -------------------------------------------------------------------- |
| `--at` | Compare only these seconds |
| `--range` | Compare every frame of `start,end` seconds |
| `--segments` | A verdict per `## S01 · a–b s` segment of a `REVERSE.md` |
| `--drawn` | Score one drawn element's box `x,y,w,h[,a-b]` by itself (repeatable) |
| `--min-mean` / `--min-p05` / `--min-cell` | The SSIM bars to pass |
| `--sbs` | Also write an original-beside-yours video |
| `--explain` | Ask HeyGen's video model what differs in each failing element |
| `--out` | Folder for the sheet and log (default `.hyperframes/diff/<yours>`) |
| `--json` | Print one JSON result |
## Edit from the command line
### `timeline`
@@ -2176,6 +2259,22 @@ goes only to that provider's own usage endpoint. The command sends no telemetry
and skips the update check. The numbers are a snapshot: they do not reserve
allowance or estimate what the next video will cost.
### `billing`
Opens the HyperFrames plans or credits page in the browser, then waits until the
account shows the change, so an agent's run carries on by itself.
```bash
npx hyperframes billing subscribe
npx hyperframes billing top-up --needed 180
```
| Flag | Description |
| ----------- | -------------------------------------------------------------------------- |
| `--needed` | Credits the waiting step needs; `top-up` waits until the balance covers it |
| `--tool` | The step that is waiting, shown on the page |
| `--timeout` | Seconds to wait (default 600) |
### `figma` and `events`
`npx hyperframes figma` imports through the Figma REST API, with three
packages/cli/src/auth/browser.ts
+3 -4
View File
@@ -1,7 +1,7 @@
/**
* Open a URL in the user's default browser. Falls back to printing the
* URL when no browser is openable (SSH session, CI, `BROWSER=none`,
* or `open` rejects).
* Open a URL in the user's default browser. Falls back to printing the URL when no browser is openable
* (SSH session, CI, `BROWSER=none`, or `open` rejects). ponytail: `HF_BROWSER_LOG` is a test seam for
* scripted journeys: the URL goes to that file, and a harness plays the person in its own browser.
*/
import { appendFileSync } from "node:fs";
@@ -14,7 +14,6 @@ export interface OpenBrowserResult {
}
export async function openBrowser(url: string): Promise<OpenBrowserResult> {
// ponytail: test seam for scripted journeys; a harness reads the URLs and plays the person in its own browser.
const log = process.env["HF_BROWSER_LOG"];
if (log) {
appendFileSync(log, `${url}\n`);
packages/cli/src/auth/oauth.ts
+3 -4
View File
@@ -474,10 +474,9 @@ function buildAuthorizeUrl(args: {
code_challenge: args.challenge,
code_challenge_method: "S256",
});
// Keep a query the endpoint already carries (e.g. `?lang=en` for the sign-in page's language).
const url = new URL(authorizeEndpoint());
for (const [key, value] of params) url.searchParams.set(key, value);
return url.toString();
const urlWithEndpointQuery = new URL(authorizeEndpoint());
for (const [key, value] of params) urlWithEndpointQuery.searchParams.set(key, value);
return urlWithEndpointQuery.toString();
}
async function exchangeCodeForTokens(args: {
packages/cli/src/commands/auth/login.ts
+5 -2
View File
@@ -115,11 +115,14 @@ function isRemoteOrHeadless(): boolean {
process.env["BROWSER"] === "none" ||
process.env["HF_NO_BROWSER"] === "1" ||
remoteEnvironment ||
// A coding agent's shell has no terminal, yet the browser opens on this machine and the person finishes there.
(process.stdout.isTTY !== true && (isCI() || !isAgentDriven())),
(process.stdout.isTTY !== true && !agentShellOnThisMachine()),
);
}
function agentShellOnThisMachine(): boolean {
return !isCI() && isAgentDriven();
}
function assertAttendedDeviceFlow(): void {
if (!isAttendedTerminal()) {
console.error(
packages/cli/src/commands/auth/status-guidance.ts
-1
View File
@@ -69,7 +69,6 @@ export function buildUnconfiguredLines(
ctx: UnconfiguredContext,
engines?: OfflineEngineLine[],
): string[] {
// The agent relays one step and waits; local engines and API keys are for the person to raise.
if (ctx.agent)
return [
c.warn("Not signed in to HeyGen."),
packages/cli/src/heygen/plan.ts
+1 -3
View File
@@ -24,7 +24,6 @@ function mockPlan(): Plan | null {
// Payment routes live on api2, beside the OAuth ones; the v3 generation routes stay on api.
const billingBaseUrl = () => resolveCloudBaseUrl().replace("://api.", "://api2.");
/** The plan from `/v1/payment/hyperframes/subscription`, else the mock, else null. */
export async function readPlan(): Promise<Plan | null> {
try {
const res = await fetch(`${billingBaseUrl()}/v1/payment/hyperframes/subscription`, {
@@ -35,8 +34,7 @@ export async function readPlan(): Promise<Plan | null> {
const data = isRecord(body) && isRecord(body.data) ? body.data : body;
if (!res.ok || !isRecord(data)) return mockPlan();
const usage = isRecord(data.usage) ? data.usage : {};
// Shaped as studio-contracts' BillingSubscription: `subscription` is null on Free, and an unpaid placeholder
// carries its plan's name before the first invoice funds it.
// studio-contracts' BillingSubscription: `subscription` is null on Free; an unpaid placeholder still names a plan.
const sub = isRecord(data.subscription) ? data.subscription : null;
const paid = !!sub && !UNPAID.has(String(sub.status));
return {
packages/cli/src/heygen/pricing.ts
+2 -3
View File
@@ -34,8 +34,7 @@ const VIDEO_768P_PER_S = 0.018;
const VIDEO_REF_768P_PER_S = 0.05;
const MUSIC_PER_TRACK = 0.08;
const AVATAR_IV_PER_MIN = 4.83;
// An avatar's length is estimated from its script, as the app does.
const WORDS_PER_S = 2;
const SCRIPT_WORDS_PER_S = 2;
// fallow-ignore-next-line complexity
export function priceOf(r: GenerateRequest): Price {
@@ -66,7 +65,7 @@ export function priceOf(r: GenerateRequest): Price {
}
case "avatar": {
const words = (r.script ?? "").split(/\s+/).filter(Boolean).length;
const seconds = Math.max(1, Math.floor(words / WORDS_PER_S));
const seconds = Math.max(1, Math.floor(words / SCRIPT_WORDS_PER_S));
return at(
Math.ceil(seconds * AVATAR_IV_PER_MIN) / 60,
`avatar video, about ${seconds} s from ${words} words at $4.83 a minute`,
packages/cli/src/measure/cuts.ts
+2 -11
View File
@@ -1,20 +1,14 @@
// Cuts from the frame-to-frame difference itself, ported from the Studio's videoShots.mjs: ffmpeg's scene score
// missed two of the James Blake source's cuts at 0.25, so a cut is a frame far unlike the one before and several
// times unlike its neighbours' changes. A punch-in at the same angle changes less (dan's 9.743 s: 11 grey levels),
// so a smaller change also counts when nothing else near it changes anywhere near as much: a soft cut.
// Cuts from the frame-to-frame difference, ported from the Studio's videoShots.mjs; ffmpeg's scene score misses some.
// A punch-in at the same angle changes less than a cut, so a smaller change that stands alone is a soft cut.
import { decode, meanAbsDiff, type Frames } from "./media.js";
// Small enough to read a minute quickly, large enough to tell a cut from motion.
export const SIG_W = 64;
export const SIG_H = 36;
const CUT_LEVEL = 12;
const CUT_RATIO = 4;
// A soft cut: at least SOFT_LEVEL, SOFT_RATIO times its neighbours' median and SOFT_ALONE times the largest of them.
const SOFT_LEVEL = 10;
const SOFT_RATIO = 8;
const SOFT_ALONE = 4;
// The other side confirms a soft cut with a change of CONFIRM_LEVEL, CUT_RATIO times its neighbours' median and
// CONFIRM_SHARE of the cut's own change, within a frame of it.
const CONFIRM_LEVEL = 6;
const CONFIRM_SHARE = 0.5;
const NEIGHBOURS = 12;
@@ -25,7 +19,6 @@ export interface Cut {
frames: number;
/** Its largest change, in mean grey levels. */
change: number;
/** Found only by the soft rule. */
soft?: true;
}
@@ -39,7 +32,6 @@ const median = (values: number[]) => {
return sorted[Math.floor(sorted.length / 2)] ?? 0;
};
/** Each frame's mean grey difference from the one before (0 for the first). */
export function frameChanges(frames: Uint8Array[]): number[] {
return frames.map((f, i) => (i ? meanAbsDiff(frames[i - 1]!, f) : 0));
}
@@ -47,7 +39,6 @@ export function frameChanges(frames: Uint8Array[]): number[] {
const neighbours = (diffs: number[], i: number) =>
diffs.slice(Math.max(1, i - NEIGHBOURS), i).concat(diffs.slice(i + 1, i + NEIGHBOURS + 1));
/** The frame indexes where a new shot starts, from each frame's change (`frameChanges`). */
export function cutsIn(diffs: number[]): Cut[] {
const hits = diffs.flatMap((d, i): { frame: number; change: number; soft?: true }[] => {
if (i === 0) return [];
packages/cli/src/measure/drawn.ts
+8 -12
View File
@@ -1,9 +1,8 @@
// A drawn element (text, a badge, a card) over a regenerated person, scored by itself: what is behind it differs on
// purpose, so SSIM over its box measures the background. Each frame finds the element's ink on both sides (pixels
// far from the median colour of their neighbourhood, so a letter stands out from any background behind it), aligns
// the two within a few pixels, and asks how much of either side's ink the other matches in colour. Wrong weight,
// tracking, slant, size, fill, a doubled letter or a late onset all leave ink unmatched; a different background
// leaves none. Calibrated on the reverse bench (10-23): correct titles and captions 0.82–0.94, wrong ones 0.40–0.66.
// purpose, so SSIM over its box measures the background. Each frame finds the element's ink on both sides, aligns
// them within a few pixels, and asks how much of either side's ink the other matches in colour. Wrong weight,
// tracking, slant, size, fill, a doubled letter or a late onset leave ink unmatched; a different background leaves
// none. Calibrated on the reverse bench (10-23): correct titles and captions 0.82–0.94, wrong ones 0.40–0.66.
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -18,7 +17,6 @@ const MATCH_COLOUR = 45;
const ALIGN = 3;
// Fewer ink pixels than this on both sides: nothing drawn in that frame on either side.
const MIN_INK = 30;
// Elements are read at most this often.
const MAX_FPS = 10;
export interface DrawnFrame {
@@ -89,8 +87,7 @@ export function scoreDrawnFrame(a: Uint8Array, b: Uint8Array, w: number, h: numb
const [ia, ib] = [inkOf(a, w, h), inkOf(b, w, h)];
const [na, nb] = [ia.reduce((s, v) => s + v, 0), ib.reduce((s, v) => s + v, 0)];
if (na + nb < MIN_INK) return { score: null, offset: [0, 0], inkOriginal: na, inkYours: nb };
// The shift of yours whose ink overlaps the original's most.
let best = { iou: -1, dx: 0, dy: 0 };
let mostOverlappingShift = { iou: -1, dx: 0, dy: 0 };
for (let dy = -ALIGN; dy <= ALIGN; dy++)
for (let dx = -ALIGN; dx <= ALIGN; dx++) {
let [both, either] = [0, 0];
@@ -105,7 +102,7 @@ export function scoreDrawnFrame(a: Uint8Array, b: Uint8Array, w: number, h: numb
}
}
const iou = either ? both / either : 0;
if (iou > best.iou) best = { iou, dx, dy };
if (iou > mostOverlappingShift.iou) mostOverlappingShift = { iou, dx, dy };
}
// Ink on either side counts as matched when the other side has a pixel of its colour within one pixel.
const at = (x: number, y: number) =>
@@ -123,7 +120,7 @@ export function scoreDrawnFrame(a: Uint8Array, b: Uint8Array, w: number, h: numb
for (let y = 0; y < h; y++)
for (let x = 0; x < w; x++) {
const i = y * w + x;
const [sx, sy] = [x - best.dx, y - best.dy];
const [sx, sy] = [x - mostOverlappingShift.dx, y - mostOverlappingShift.dy];
const inB = sy >= 0 && sy < h && sx >= 0 && sx < w && ib[sy * w + sx] === 1;
if (ia[i]) {
inkA++;
@@ -137,7 +134,7 @@ export function scoreDrawnFrame(a: Uint8Array, b: Uint8Array, w: number, h: numb
const [recall, precision] = [inkA ? hitA / inkA : 0, inkB ? hitB / inkB : 0];
return {
score: recall + precision ? (2 * recall * precision) / (recall + precision) : 0,
offset: [0 - best.dx, 0 - best.dy],
offset: [0 - mostOverlappingShift.dx, 0 - mostOverlappingShift.dy],
inkOriginal: na,
inkYours: nb,
};
@@ -145,7 +142,6 @@ export function scoreDrawnFrame(a: Uint8Array, b: Uint8Array, w: number, h: numb
const median = (v: number[]) => [...v].sort((x, y) => x - y)[v.length >> 1] ?? 0;
/** The verdict of an element from its frames. */
export function drawnVerdict(
box: Roi,
window: [number, number],
packages/cli/src/measure/energy.ts
+1 -2
View File
@@ -1,6 +1,5 @@
// Body and camera motion where the pixels differ (a regenerated shot, another person, another room), ported from
// the Studio's motion-energy.mjs. Both clips are sampled on one 12 fps grid (mixed 24/30 fps skews per-frame
// differences), ROI scaled to 270 px wide, luma. Energy(t) = mean |frame_t - frame_t-1|.
// the Studio's motion-energy.mjs. Both clips share one 12 fps grid, as mixed 24/30 fps skews per-frame differences.
import { meanAbsDiff, pearson, readClip, round, type Clip, type Roi } from "./media.js";
const ENERGY_RATE = 12;
packages/cli/src/measure/face.test.ts
+1 -1
View File
@@ -39,7 +39,7 @@ describe("againstStart", () => {
it("reads a generated shot from its own start and a full render on film time", () => {
// A 5.2 s HeyGen shot for the original's 9.5-12.27 s: its face is at 0-2.77 s of the clip, not 9.5.
expect(againstStart(5.17, 12.27, 9.5)).toBe(0);
// A shot longer than the window but shorter than the film is still its own clip (it used to be read from --from).
// A shot longer than the window but shorter than the film is still its own clip.
expect(againstStart(7.29, 12.27, 2.77)).toBe(0);
expect(againstStart(12.27, 12.27, 9.5)).toBe(9.5);
expect(againstStart(12.3, 12.27, 9.5)).toBe(9.5);
packages/cli/src/measure/face.ts
+2 -4
View File
@@ -8,10 +8,8 @@ const FACE_HOLDS = 0.25;
const FACE_SLIDES = 0.15;
const FACE_FLOOR = 0.06;
/**
* The smallest face looked for by default, as a share of the frame's height: well under FACE_FLOOR (a smaller face
* fails either way), and it skips the cascades' smallest, costliest windows. Those start at 20 px and grow by 10% a
* step, so a minimum on one of the steps finds the faces the full search finds from that size up, with the same
* boxes; scaling the frame down instead moves every box by up to a step.
* The smallest face looked for by default, as a share of the frame's height: well under FACE_FLOOR, and it skips the
* cascades' costliest windows. A minimum on their ladder keeps the full search's boxes; scaling the frame moves them.
*/
const FACE_MIN = 0.04;
const WORK_W = 960;
packages/cli/src/measure/segments.ts
+3 -6
View File
@@ -1,9 +1,6 @@
// The segments of a REVERSE.md: `## S01 · 0.00–2.40 s · title` headings, each with an optional
// `Pixels: rebuilt | reused | regenerated` line saying where that window's picture comes from, and an optional
// `Regenerated: x, y, w, h` line for a rebuilt or reused segment with a regenerated region (a re-shot presenter
// under drawn graphics): the region's box in the source's pixels, or as shares of the frame when all are ≤ 1. Each
// `Drawn: x, y, w, h` line is a drawn element over a regenerated person, still judged by pixels, with an optional
// `a–b s` when it shows for part of the segment. A `Copy:` line that quotes a string says the segment draws text.
// The segments of a REVERSE.md: `## S01 · 0.00–2.40 s · title` headings, each with optional lines
// `Pixels: rebuilt | reused | regenerated`, `Regenerated: x, y, w, h` (source pixels, or shares of the frame when
// all are ≤ 1), `Drawn: x, y, w, h` with an optional `a–b s`, and `Copy:` quoting a string; see `Segment`.
import { existsSync, readFileSync } from "node:fs";
import { Fix } from "./media.js";
scripts/check-comment-citations.mjs
+2
View File
@@ -139,6 +139,8 @@ const PROJECT_FILES = new Set([
"REVIEW.md",
"STORYLINES.md",
"assets/transcript.json",
"transcript.json",
"REVERSE.md",
"asset-descriptions.md",
"extracted/page.html",
"extracted/visible-text.txt",
scripts/check-no-main-deletions.mjs
+8
View File
@@ -2080,6 +2080,14 @@ export const ALLOWED_DELETIONS = new Map([
".github/workflows/lambda-ffmpeg.yml",
"the ffmpeg build moves into lambda-arm64-image.yml, which also builds and smokes the image",
],
[
"skills/remotion-to-hyperframes/scripts/frame_strip.sh",
"remotion-to-hyperframes now compares with `hyperframes diff` instead of its own shell scripts",
],
[
"skills/remotion-to-hyperframes/scripts/render_diff.sh",
"remotion-to-hyperframes now compares with `hyperframes diff` instead of its own shell scripts",
],
]);
export function parseBase(argv, fallback = "origin/main") {
skills-manifest.json
+1 -1
View File
@@ -18,7 +18,7 @@
"files": 13
},
"media-use": {
"hash": "6f5914c6a81b6106",
"hash": "6af3426f9dc2809f",
"files": 117
},
"remotion-to-hyperframes": {
skills/media-use/SKILL.md
+10 -14
View File
@@ -19,11 +19,11 @@ It prints one line, `resolved <id> → <path> (<type>, <metadata>)`. List reusab
| Need | Free | HyperFrames credits | Local or the person's own key (only when they ask) |
| --- | --- | --- | --- |
| Music (`bgm`) | HeyGen catalog of 10k+ tracks (signed in) | Generated track | Local MusicGen (`audio/references/bgm.md`) |
| Sound effects (`sfx`) | Openverse (credit CC BY: `audio/references/sfx.md`); 21 bundled offline | | |
| Music (`bgm`) | HeyGen catalog of 10k+ tracks (signed in) | Generated track | Local MusicGen |
| Sound effects (`sfx`) | Openverse (credit CC BY sounds); 21 bundled offline | | |
| Images, icons | HeyGen asset search, 75k+ (signed in) | Generated images | Local FLUX (mflux), if installed |
| Logos (`logo`) | theSVG, GitHub avatar, favicon; never redrawn | | |
| Voiceover (`voice`) | HeyGen Voice Turbo (signed in) | Other HeyGen voices, voice cloning | `npx hyperframes tts` (Kokoro on device; Python 3.10+ with `pip install kokoro-onnx soundfile`) |
| Voiceover (`voice`) | HeyGen Voice Turbo (signed in) | Other HeyGen voices, voice cloning | `npx hyperframes tts` (Kokoro on device) |
| A presenter speaking | | Avatar video (`--type video`) | Local LTX for generic clips, if installed |
| Transcripts | `npx hyperframes transcribe` (Parakeet or whisper) | | |
| Cutouts | | `heygen background-removal` | `npx hyperframes remove-background` |
@@ -31,19 +31,15 @@ It prints one line, `resolved <id> → <path> (<type>, <metadata>)`. List reusab
**Sign in.** Every time a request needs a HeyGen model or the catalog, run `npx hyperframes auth status` first. Not signed in: tell the person in one line that sign-in is free and gives Voice Turbo, the music catalog and asset search, ask them to type an exclamation mark and `npx hyperframes auth login` in the prompt, and wait until they say it is done. Use local tools only when the person asks, not as a fallback.
**Host apps.** Inside an app with its own media tools, use those first. When `HEYGEN_API_BASE` is set, the host pays for HeyGen: don't ask for a sign-in, and when a call is refused, relay the host's message as written and stop.
**Host apps.** Inside an app with its own media tools, use those first. When `HEYGEN_API_BASE` is set, the host pays: don't ask for a sign-in; relay a refused call's message as written and stop.
**HeyGen.** Install the [`heygen` CLI](https://developers.heygen.com/cli), then sign in with `npx hyperframes auth login`. HyperFrames supplies its credential to the helper; it does not read the separate HeyGen CLI login. An explicit API key bills API credits, and the server decides OAuth eligibility and charges. Check setup with `npx hyperframes media-use resolve --doctor`. If a required provider is missing or signed out, give the install link and HyperFrames sign-in command, then wait until they say it is done.
**HeyGen.** HeyGen calls also need the [`heygen` CLI](https://developers.heygen.com/cli); if it is missing, give the link and wait until they say it is installed. `npx hyperframes media-use generate` makes images, shots, talking photos, voices and music. The direct `heygen` recipes bill the HeyGen CLI's own login, not HyperFrames: use them only when that separate payer is selected.
**Direct HeyGen recipes.** Commands such as `heygen video-agent create`, `heygen video create`, `heygen voice clone create`, `heygen lipsync`, `heygen video-translate` and `heygen ai-clipping` in `references/operations.md` use the HeyGen CLI's own credential and HeyGen/API billing. They do not inherit HyperFrames OAuth or its subscription. Use only when that separate payer is selected.
**Generate.** `npx hyperframes media-use generate` makes images, shots, talking photos, voices and music (`references/heygen.md`).
**Credits.** Plan for the best result; don't swap in cheaper or free options, or drop items, to save credits unless the person asks to save cost. Never tell the person prices, estimates or what a run used; money comes up only when a call is refused. A credit limit the person sets is the one exception: pass what is left of it as `--max-credits` on every `generate`, subtract each call's `credits`, and when a call is refused for the limit, say what does not fit and ask; never go past it. A refused call names a `billing` command: tell the person why, ask them to run it the same way, then retry the step. `--provider <name>` pins a generator the person names. `--local-only` skips every network provider.
**Credits.** Plan for the best result: don't swap in cheaper or free options, or drop items, unless the person asks to save cost. Never tell the person prices, estimates or what a run used; money comes up only when a call is refused. A credit limit the person sets is the one exception: pass what is left of it as `--max-credits` on every `generate`, subtract each call's `credits`, and when a call is refused for the limit, say what does not fit and ask; never go past it. A refused call names a `billing` command: tell the person why, ask them to run it the same way, then retry the step.
## Looks on real footage
Feedback that footage is dark, flat or dull, should feel retro or printed, needs a face hidden, or needs a reveal is media intent: read `references/media-treatments.md`. Inspect the real `<img>`/`<video>`, pick one intent, start from `npx hyperframes media-treatment --capabilities --json`, then `--capability <id>`; never `--all`. Persist the final treatment with `npx hyperframes media-treatment`; no CSS or SVG stand-ins for supported effects, and no generated `.cube` just to encode exposure or warmth.
Feedback that footage is dark, flat or dull, should feel retro or printed, needs a face hidden, or needs a reveal is media intent. Inspect the real `<img>`/`<video>`, pick one intent, and follow `references/media-treatments.md`; persist with `npx hyperframes media-treatment`, never a CSS, SVG or `.cube` stand-in.
## Offer once, never add silently
@@ -56,10 +52,10 @@ When you build or review a piece, scan once for concrete gaps (on-screen text wi
| Resolve, reuse, adopt, ingest, flags | `references/resolve.md` |
| Grading, LUTs, `grade-compare` | `references/grading.md` |
| Voiceover, music, sound effects, captions, transcription | `references/audio.md` |
| Cut, reframe or transform media; HeyGen recipes | `references/operations.md` |
| Cut, reframe or transform media; direct `heygen` recipes | `references/operations.md` |
| Treatments, realtime effects, overlays, reveals | `references/media-treatments.md` |
| Install, sign-in, providers, RAM ladders | `references/setup-providers.md` |
| HeyGen generation, transcription, video analysis, prices | `references/heygen.md` |
| Install, sign-in, credentials, `--provider`, `--local-only`, RAM ladders | `references/setup-providers.md` |
| `media-use generate`, HeyGen transcription and video analysis, prices | `references/heygen.md` |
| Fades, gain, ducking, automation, effects, buses on placed tracks | `references/mixing.md` |
| Local commands: grade, cutout, loudness, beats | `references/local.md` |
| Remembered preferences | `references/memory.md` |
skills/media-use/references/setup-providers.md
+9 -9
View File
@@ -74,15 +74,15 @@ tools are OPT-IN alternatives where they exist; install one to unlock its free,
private, on-device path instead of or ahead of HeyGen for that type. Only
`ffmpeg`/`ffprobe` are strictly required for the tool to run at all.
| Tool | Serves | Install |
| ------------------ | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg`/`ffprobe` | adopt probing, smart-grade signalstats, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
| `heygen` | catalog (bgm/image/icon) + TTS (voice) + avatar video — the HyperFrames-authenticated path | install through [verified HeyGen release instructions](https://developers.heygen.com/cli), then `npx hyperframes auth login` (helper needs >= v0.3.0) |
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
| `parakeet-mlx` | local transcription (default ASR, best) on Apple Silicon | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
| `npx hyperframes` | Kokoro TTS (voice), Parakeet or whisper.cpp (transcribe), remove-background | via the hyperframes CLI; Parakeet on macOS, Windows and glibc 2.32+ Linux: `npx hyperframes models install parakeet`; whisper.cpp is built on first use (Homebrew on macOS, else git+cmake) |
| Tool | Serves | Install |
| ------------------ | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg`/`ffprobe` | adopt probing, smart-grade signalstats, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
| `heygen` | catalog (bgm/image/icon) + TTS (voice) + avatar video — the HyperFrames-authenticated path | install through [verified HeyGen release instructions](https://developers.heygen.com/cli), then `npx hyperframes auth login` (helper needs >= v0.3.0) |
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
| `parakeet-mlx` | local transcription (default ASR, best) on Apple Silicon | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
| `npx hyperframes` | Kokoro TTS (voice), Parakeet or whisper.cpp (transcribe), remove-background | via the hyperframes CLI; Kokoro needs Python 3.10+ with `pip install kokoro-onnx soundfile`; Parakeet on macOS, Windows and glibc 2.32+ Linux: `npx hyperframes models install parakeet`; whisper.cpp is built on first use (Homebrew on macOS, else git+cmake) |
The RAM-graded local-model shortlist + exact per-tier install/invoke lives in
the CLI's bundled media-use engine, not in the standalone skill. Without a tool on PATH, its provider