mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-10-30 17:27:09 +00:00
* revert: feat(skills): reverse any video (#6754)
Keep the reverse-any-video skill and its CLI billing commands out of main until
web subscriptions are live (/pricing and /account/plan return 404 today).
Re-land together with #6833 once those pages ship.
This reverts commit cb2389cf35.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* chore(scripts): allow the #6754 revert deletions
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
James Russo
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
746ce956b3
commit
b45e602faa
@@ -56,4 +56,3 @@ skills/media-use/SKILL.md
|
||||
skills/figma/SKILL.md
|
||||
skills/slideshow/SKILL.md
|
||||
skills/remotion-to-hyperframes/SKILL.md
|
||||
skills/reverse-any-video/SKILL.md
|
||||
|
||||
+1
-73
@@ -28,12 +28,11 @@ npm install -g hyperframes
|
||||
| Bring in source material | [`capture`](#capture), [`media-use`](#media-use), [`transcribe`](#transcribe), [`tts`](#tts), [`remove-background`](#remove-background), [`media-treatment`](#media-treatment), [`beats`](#beats), [`audio`](#audio), [`normalize-audio`](#normalize-audio), [`models`](#models) |
|
||||
| Look at it, or share it | [`preview`](#preview), [`present`](#present-and-play), [`play`](#present-and-play), [`publish`](#publish), [`open`](#open-and-catch-up), [`catch-up`](#open-and-catch-up) |
|
||||
| Find problems | [`lint`](#lint), [`check`](#check), [`snapshot`](#snapshot), [`keyframes`](#keyframes), [`compare`](#compare-and-grade-compare), [`grade-compare`](#compare-and-grade-compare) |
|
||||
| Rebuild a video | [`measure`](#measure), [`diff`](#diff) |
|
||||
| Edit from the terminal | [`timeline`](#timeline), [`history`](#history) |
|
||||
| Make a file | [`render`](#render), [`benchmark`](#benchmark) |
|
||||
| Render somewhere else | [`cloud`](#hyperframes-cloud), [`lambda`](#hyperframes-lambda), [`cloudrun`](#hyperframes-cloudrun) |
|
||||
| Fix your setup | [`doctor`](#doctor), [`browser`](#browser), [`upgrade`](#upgrade), [`docs`](#docs), [`info`](#info-and-compositions), [`compositions`](#info-and-compositions), [`clean`](#clean) |
|
||||
| Connect other tools | [`auth`](#hyperframes-auth), [`billing`](#billing), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
|
||||
| Connect other tools | [`auth`](#hyperframes-auth), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
|
||||
|
||||
`validate`, `inspect`, and `layout` still work but are
|
||||
[deprecated](#deprecated-validate-inspect-and-layout). Use `check` instead.
|
||||
@@ -1505,59 +1504,6 @@ session, and it is where new work should go. Their flags are unchanged — run
|
||||
`npx hyperframes inspect --help` if you are maintaining automation that still
|
||||
calls them.
|
||||
|
||||
## Rebuild an existing video
|
||||
|
||||
These commands serve the `reverse-any-video` skill: measure a video, have HeyGen
|
||||
draft its timeline, then score your rebuild against it.
|
||||
|
||||
### `measure`
|
||||
|
||||
Measures a video file and prints JSON. Each measurement is a subcommand; run
|
||||
`npx hyperframes measure <subcommand> --help` for its flags.
|
||||
|
||||
```bash
|
||||
npx hyperframes measure probe ref.mp4
|
||||
npx hyperframes measure cuts ref.mp4
|
||||
npx hyperframes measure frames ref.mp4 --events
|
||||
npx hyperframes measure face ref.mp4 --start 1 --end 2.5
|
||||
```
|
||||
|
||||
| Subcommand | What it reports |
|
||||
| ------------ | --------------------------------------------------------------------------------------- |
|
||||
| `probe` | Displayed size, frame rate, duration, sound, and what to fix before rebuilding |
|
||||
| `cuts` | Hard cuts from the frame-to-frame difference, and the shots between them |
|
||||
| `events` | Cuts, flashes, appears, disappears and holds; `--map` adds the sound's hits |
|
||||
| `frames` | Frames as pictures: chosen moments, an overview sheet, a range, or strips around events |
|
||||
| `extract` | A still or a clip cut from the video, cropped to a region, as project media |
|
||||
| `curve` | Each move of one element: start, duration, overshoot and the closest GSAP ease |
|
||||
| `energy` | How much two clips move, and whether at the same moments |
|
||||
| `look` | Brightness, contrast, saturation and warmth, alone or against another video |
|
||||
| `face` | The face's size and position over a window |
|
||||
| `sync` | The lag between mouth motion and the voice |
|
||||
|
||||
### `diff`
|
||||
|
||||
Scores a render, or a project folder without rendering it, against the original
|
||||
video frame by frame: SSIM, the worst moments, missed and extra cuts, duration
|
||||
and the sound's offset. It exits non-zero when the scores fall under the bar.
|
||||
|
||||
```bash
|
||||
npx hyperframes diff original.mp4 renders/final.mp4
|
||||
npx hyperframes diff original.mp4 . --at 0.5,2.4,3.1
|
||||
npx hyperframes diff original.mp4 renders/final.mp4 --segments segments.md --sbs
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
| ----------------------------------------- | -------------------------------------------------------------------- |
|
||||
| `--at` | Compare only these seconds |
|
||||
| `--range` | Compare every frame of `start,end` seconds |
|
||||
| `--segments` | A verdict per `## S01 · a–b s` segment of a file |
|
||||
| `--drawn` | Score one drawn element's box `x,y,w,h[,a-b]` by itself (repeatable) |
|
||||
| `--min-mean` / `--min-p05` / `--min-cell` | The SSIM bars to pass |
|
||||
| `--sbs` | Also write an original-beside-yours video |
|
||||
| `--out` | Folder for the sheet and log (default `.hyperframes/diff/<yours>`) |
|
||||
| `--json` | Print one JSON result |
|
||||
|
||||
## Edit from the command line
|
||||
|
||||
### `timeline`
|
||||
@@ -2231,24 +2177,6 @@ goes only to that provider's own usage endpoint. The command sends no telemetry
|
||||
and skips the update check. The numbers are a snapshot: they do not reserve
|
||||
allowance or estimate what the next video will cost.
|
||||
|
||||
### `billing`
|
||||
|
||||
Opens the HyperFrames plans or credits page in the browser, then waits until the
|
||||
account shows the change, so an agent's run carries on by itself. The page is on
|
||||
the web app for `HYPERFRAMES_ENV`: hyperframes.studio, or dev.hyperframes.dev
|
||||
with `HYPERFRAMES_ENV=dev`.
|
||||
|
||||
```bash
|
||||
npx hyperframes billing subscribe
|
||||
npx hyperframes billing top-up --needed 180
|
||||
```
|
||||
|
||||
| Flag | Description |
|
||||
| ----------- | -------------------------------------------------------------------------- |
|
||||
| `--needed` | Credits the waiting step needs; `top-up` waits until the balance covers it |
|
||||
| `--tool` | The step that is waiting, shown on the page |
|
||||
| `--timeout` | Seconds to wait (default 600) |
|
||||
|
||||
### `figma` and `events`
|
||||
|
||||
`npx hyperframes figma` imports through the Figma REST API, with three
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import { existsSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs";
|
||||
import { basename, dirname, extname, join, resolve } from "node:path";
|
||||
import { transcribeWithHeygen } from "../heygen/transcribe.js";
|
||||
import { normalizeErrorMessage } from "../utils/errorMessage.js";
|
||||
import {
|
||||
createRenderCancellationScope,
|
||||
@@ -39,7 +38,7 @@ import {
|
||||
// finishing even on the fastest hardware for the shortest clip.
|
||||
export const MIN_TIMEOUT_MS = 5000;
|
||||
|
||||
export type TranscribeEngine = "auto" | "parakeet" | "whisper" | "heygen";
|
||||
export type TranscribeEngine = "auto" | "parakeet" | "whisper";
|
||||
export type CaptionFormat = "srt" | "vtt";
|
||||
/** The program that does the work: Parakeet runs on sherpa-onnx or parakeet-mlx. */
|
||||
export type TranscribeRunner = ParakeetRunner | "whisper";
|
||||
@@ -70,10 +69,8 @@ export interface TranscribeOptions {
|
||||
input: string;
|
||||
/** Where transcript.json goes; caption scripts in this folder's .html files are patched. Default: the input's folder. */
|
||||
dir?: string;
|
||||
/** Default "auto": Parakeet when installed and it covers `language`, else whisper; "heygen" runs on HeyGen. */
|
||||
/** Default "auto": Parakeet when installed and it covers `language`, else whisper. */
|
||||
engine?: TranscribeEngine;
|
||||
/** HeyGen only: label each word with its speaker. */
|
||||
speakers?: boolean;
|
||||
/** Whisper model. Parakeet has its own fixed model. */
|
||||
model?: string;
|
||||
language?: string;
|
||||
@@ -100,7 +97,7 @@ export interface TranscribeOptions {
|
||||
export type TranscribeResult =
|
||||
| {
|
||||
kind: "transcribed";
|
||||
engine: "parakeet" | "whisper" | "heygen";
|
||||
engine: "parakeet" | "whisper";
|
||||
model: string;
|
||||
detectedLanguage: string | null;
|
||||
wordCount: number;
|
||||
@@ -370,71 +367,6 @@ function failureMessage(err: unknown): string {
|
||||
return stderr ? `${base}\n${stderr}` : base;
|
||||
}
|
||||
|
||||
interface EngineTranscript {
|
||||
transcriptPath: string;
|
||||
model: string;
|
||||
detectedLanguage: string | null;
|
||||
durationSeconds: number;
|
||||
speechOnsetSeconds: number | null;
|
||||
}
|
||||
|
||||
/** Every engine's words end the same way: normalized, patched into caption scripts, and as a sidecar if asked. */
|
||||
function finishTranscript(
|
||||
result: EngineTranscript,
|
||||
engine: "parakeet" | "whisper" | "heygen",
|
||||
dir: string,
|
||||
sidecar: CaptionSidecar | undefined,
|
||||
): TranscribeResult {
|
||||
const { words } = loadTranscript(result.transcriptPath);
|
||||
|
||||
writeFileSync(result.transcriptPath, JSON.stringify(words, null, 2));
|
||||
patchCaptionHtml(dir, words);
|
||||
if (sidecar) {
|
||||
try {
|
||||
writeCaptionSidecar(words, sidecar, false);
|
||||
} catch (err) {
|
||||
throw new TranscribeError(
|
||||
"caption_write_failed",
|
||||
`Transcript saved to ${result.transcriptPath}, but the caption file ${sidecar.outPath} could not be written: ${normalizeErrorMessage(err)}`,
|
||||
{ cause: err, transcriptPath: result.transcriptPath },
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
kind: "transcribed",
|
||||
engine,
|
||||
model: result.model,
|
||||
detectedLanguage: result.detectedLanguage,
|
||||
wordCount: words.length,
|
||||
durationSeconds: result.durationSeconds,
|
||||
speechOnsetSeconds: result.speechOnsetSeconds,
|
||||
transcriptPath: result.transcriptPath,
|
||||
...(sidecar && { captions: { format: sidecar.to, outputPath: sidecar.outPath } }),
|
||||
};
|
||||
}
|
||||
|
||||
async function transcribeOnHeygen(
|
||||
inputPath: string,
|
||||
dir: string,
|
||||
sidecar: CaptionSidecar | undefined,
|
||||
options: TranscribeRequest,
|
||||
): Promise<TranscribeResult> {
|
||||
try {
|
||||
const result = await transcribeWithHeygen(inputPath, join(dir, TRANSCRIPT_FILE), {
|
||||
diarize: options.speakers,
|
||||
projectDir: dir,
|
||||
onStatus: options.onStatus,
|
||||
});
|
||||
return finishTranscript(result, "heygen", dir, sidecar);
|
||||
} catch (err) {
|
||||
if (err instanceof TranscribeError) throw err;
|
||||
const fix =
|
||||
err instanceof Error && "fix" in err && typeof err.fix === "string" ? ` Fix: ${err.fix}` : "";
|
||||
throw new TranscribeError("failed", `${normalizeErrorMessage(err)}.${fix}`, { cause: err });
|
||||
}
|
||||
}
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
async function transcribeAudio(
|
||||
inputPath: string,
|
||||
@@ -443,11 +375,10 @@ async function transcribeAudio(
|
||||
options: TranscribeRequest,
|
||||
): Promise<TranscribeResult> {
|
||||
const engine = (options.engine ?? "auto").toLowerCase();
|
||||
if (engine === "heygen") return transcribeOnHeygen(inputPath, dir, sidecar, options);
|
||||
if (engine !== "auto" && engine !== "parakeet" && engine !== "whisper") {
|
||||
throw new TranscribeError(
|
||||
"invalid_option",
|
||||
`Unknown --engine: ${options.engine}. Use auto, parakeet, whisper, or heygen.`,
|
||||
`Unknown --engine: ${options.engine}. Use auto, parakeet, or whisper.`,
|
||||
);
|
||||
}
|
||||
const unsupported = sherpaUnsupportedReason();
|
||||
@@ -557,7 +488,33 @@ async function transcribeAudio(
|
||||
}
|
||||
}
|
||||
|
||||
return finishTranscript(result, engineOf(runner), dir, sidecar);
|
||||
const { words } = loadTranscript(result.transcriptPath);
|
||||
|
||||
writeFileSync(result.transcriptPath, JSON.stringify(words, null, 2));
|
||||
patchCaptionHtml(dir, words);
|
||||
if (sidecar) {
|
||||
try {
|
||||
writeCaptionSidecar(words, sidecar, false);
|
||||
} catch (err) {
|
||||
throw new TranscribeError(
|
||||
"caption_write_failed",
|
||||
`Transcript saved to ${result.transcriptPath}, but the caption file ${sidecar.outPath} could not be written: ${normalizeErrorMessage(err)}`,
|
||||
{ cause: err, transcriptPath: result.transcriptPath },
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
kind: "transcribed",
|
||||
engine: engineOf(runner),
|
||||
model: result.model,
|
||||
detectedLanguage: result.detectedLanguage,
|
||||
wordCount: words.length,
|
||||
durationSeconds: result.durationSeconds,
|
||||
speechOnsetSeconds: result.speechOnsetSeconds,
|
||||
transcriptPath: result.transcriptPath,
|
||||
...(sidecar && { captions: { format: sidecar.to, outputPath: sidecar.outPath } }),
|
||||
};
|
||||
} catch (err) {
|
||||
if (err instanceof TranscribeError) throw err;
|
||||
if (err instanceof DecodeCancelled || cancellation?.signal.aborted) {
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
/**
|
||||
* Open a URL in the user's default browser. Falls back to printing the URL when no browser is openable
|
||||
* (SSH session, CI, `BROWSER=none`, or `open` rejects). ponytail: `HF_BROWSER_LOG` is a test seam for
|
||||
* scripted journeys: the URL goes to that file, and a harness plays the person in its own browser.
|
||||
* Open a URL in the user's default browser. Falls back to printing the
|
||||
* URL when no browser is openable (SSH session, CI, `BROWSER=none`,
|
||||
* or `open` rejects).
|
||||
*/
|
||||
|
||||
import { appendFileSync } from "node:fs";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { normalizeErrorMessage } from "../utils/errorMessage.js";
|
||||
|
||||
@@ -14,11 +13,6 @@ export interface OpenBrowserResult {
|
||||
}
|
||||
|
||||
export async function openBrowser(url: string): Promise<OpenBrowserResult> {
|
||||
const log = process.env["HF_BROWSER_LOG"];
|
||||
if (log) {
|
||||
appendFileSync(log, `${url}\n`);
|
||||
return { opened: true };
|
||||
}
|
||||
if (process.env["BROWSER"] === "none" || process.env["HF_NO_BROWSER"] === "1") {
|
||||
printManualInstructions(url);
|
||||
return { opened: false };
|
||||
|
||||
@@ -45,7 +45,7 @@ export interface LoopbackHandle {
|
||||
}
|
||||
|
||||
const CALLBACK_PATH = "/oauth/callback";
|
||||
const DEFAULT_TIMEOUT_MS = 300_000;
|
||||
const DEFAULT_TIMEOUT_MS = 120_000;
|
||||
|
||||
export async function startLoopback(opts: LoopbackOptions): Promise<LoopbackHandle> {
|
||||
const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
@@ -191,7 +191,7 @@ function respond(res: ServerResponse, status: number, body: string): void {
|
||||
}
|
||||
|
||||
function successPage(): string {
|
||||
return `<!doctype html><html><head><meta charset="utf-8"><title>Signed in to HyperFrames</title>
|
||||
return `<!doctype html><html><head><meta charset="utf-8"><title>Signed in to HeyGen</title>
|
||||
<style>body{font-family:system-ui,sans-serif;display:flex;align-items:center;justify-content:center;height:100vh;margin:0;background:#0b0f14;color:#e6e8eb}main{max-width:480px;text-align:center;padding:32px;border-radius:12px;background:#11161d;border:1px solid #1f2630}h1{font-weight:600;margin:0 0 8px;color:#3CE6AC}p{margin:0;color:#9aa3ad}</style>
|
||||
</head><body><main><h1>You're signed in.</h1><p>You can close this tab and return to your terminal.</p></main></body></html>`;
|
||||
}
|
||||
|
||||
@@ -474,9 +474,7 @@ function buildAuthorizeUrl(args: {
|
||||
code_challenge: args.challenge,
|
||||
code_challenge_method: "S256",
|
||||
});
|
||||
const urlWithEndpointQuery = new URL(authorizeEndpoint());
|
||||
for (const [key, value] of params) urlWithEndpointQuery.searchParams.set(key, value);
|
||||
return urlWithEndpointQuery.toString();
|
||||
return `${authorizeEndpoint()}?${params.toString()}`;
|
||||
}
|
||||
|
||||
async function exchangeCodeForTokens(args: {
|
||||
|
||||
@@ -175,9 +175,6 @@ const commandLoaders = {
|
||||
import("./commands/media-treatment.js").then((m) => m.mediaTreatmentCommand),
|
||||
"grade-compare": () => import("./commands/grade-compare.js").then((m) => m.default),
|
||||
compare: () => import("./commands/compare.js").then((m) => m.default),
|
||||
diff: () => import("./commands/diff.js").then((m) => m.default),
|
||||
measure: () => import("./commands/measure.js").then((m) => m.default),
|
||||
billing: () => import("./commands/billing.js").then((m) => m.default),
|
||||
capture: () => import("./commands/capture.js").then((m) => m.default),
|
||||
lambda: () => import("./commands/lambda.js").then((m) => m.default),
|
||||
cloudrun: () => import("./commands/cloudrun.js").then((m) => m.default),
|
||||
|
||||
@@ -42,7 +42,7 @@ ${c.bold("ENV VARS:")}
|
||||
${c.accent("HYPERFRAMES_API_KEY")} Alias for HEYGEN_API_KEY.
|
||||
${c.accent("HEYGEN_ACCESS_TOKEN")} OAuth token from a host app; used after the API keys, never refreshed or saved.
|
||||
${c.accent("HEYGEN_API_URL")} Override the API base URL (default https://api.heygen.com).
|
||||
${c.accent("HYPERFRAMES_ENV")} prod (default) or dev; selects OAuth, API and plans-page hosts together.
|
||||
${c.accent("HYPERFRAMES_ENV")} prod (default) or dev; selects OAuth/API hosts together.
|
||||
${c.accent("HYPERFRAMES_CONFIG_DIR")} Override the credentials directory (default ~/.hyperframes).
|
||||
${c.accent("HYPERFRAMES_OAUTH_CLIENT_ID")} Override the OAuth client_id (for dev/test).
|
||||
${c.accent("HYPERFRAMES_OAUTH_DEVICE_URL")} Override the RFC 8628 device endpoint (for dev/test).
|
||||
|
||||
@@ -109,8 +109,6 @@ describe("auth login", () => {
|
||||
"REMOTE_CONTAINERS",
|
||||
"GITPOD_WORKSPACE_ID",
|
||||
"container",
|
||||
"CLAUDECODE",
|
||||
"CLAUDE_CODE_ENTRYPOINT",
|
||||
].map((key) => [key, process.env[key]]),
|
||||
);
|
||||
for (const key of Object.keys(runtimeEnv)) delete process.env[key];
|
||||
@@ -132,7 +130,6 @@ describe("auth login", () => {
|
||||
for (const token of Object.keys(usersByToken)) delete usersByToken[token];
|
||||
for (const fn of Object.values(telemetry)) fn.mockClear();
|
||||
for (const fn of Object.values(deviceAuth)) fn.mockClear();
|
||||
for (const fn of Object.values(browserAuth)) fn.mockClear();
|
||||
vi.spyOn(console, "log").mockImplementation(() => {});
|
||||
vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
});
|
||||
@@ -320,21 +317,6 @@ describe("auth login", () => {
|
||||
expect(deviceAuth.start).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("lets a coding agent on this machine open the browser sign-in without a terminal", async () => {
|
||||
process.env["CLAUDECODE"] = "1";
|
||||
Object.defineProperty(process.stdout, "isTTY", { configurable: true, value: undefined });
|
||||
await runCommand({});
|
||||
expect(browserAuth.start).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("keeps a coding agent in CI off the browser sign-in", async () => {
|
||||
process.env["CLAUDECODE"] = "1";
|
||||
process.env["CI"] = "true";
|
||||
Object.defineProperty(process.stdout, "isTTY", { configurable: true, value: undefined });
|
||||
await expect(runCommand({})).rejects.toThrow(/Invalid command usage/);
|
||||
expect(browserAuth.start).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("opens verification_uri_complete without asking the user to re-enter the code", async () => {
|
||||
deviceChallenge.verificationUriComplete =
|
||||
"https://app.heygen.com/oauth/device?user_code=ABCD-2345";
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import { failCommand, failUsage } from "../../utils/commandResult.js";
|
||||
import { envFlagEnabled, isAttendedTerminal, isCI } from "../../utils/attendedTerminal.js";
|
||||
import { isAgentDriven } from "../../telemetry/agent_runtime.js";
|
||||
import { envFlagEnabled, isAttendedTerminal } from "../../utils/attendedTerminal.js";
|
||||
/**
|
||||
* `hyperframes auth login` — sign in to HeyGen.
|
||||
*
|
||||
@@ -115,14 +114,10 @@ function isRemoteOrHeadless(): boolean {
|
||||
process.env["BROWSER"] === "none" ||
|
||||
process.env["HF_NO_BROWSER"] === "1" ||
|
||||
remoteEnvironment ||
|
||||
(process.stdout.isTTY !== true && !agentShellOnThisMachine()),
|
||||
process.stdout.isTTY !== true,
|
||||
);
|
||||
}
|
||||
|
||||
function agentShellOnThisMachine(): boolean {
|
||||
return !isCI() && isAgentDriven();
|
||||
}
|
||||
|
||||
function assertAttendedDeviceFlow(): void {
|
||||
if (!isAttendedTerminal()) {
|
||||
console.error(
|
||||
|
||||
@@ -12,8 +12,6 @@ import { c } from "../../ui/colors.js";
|
||||
export interface UnconfiguredContext {
|
||||
/** A human can act on guidance now — a TTY, or a coding agent driving the CLI. */
|
||||
interactive: boolean;
|
||||
/** A coding agent runs this; the person reads what it relays. */
|
||||
agent?: boolean;
|
||||
}
|
||||
|
||||
/** The local engine a workflow will fall back to, and whether it's ready. */
|
||||
@@ -69,11 +67,6 @@ export function buildUnconfiguredLines(
|
||||
ctx: UnconfiguredContext,
|
||||
engines?: OfflineEngineLine[],
|
||||
): string[] {
|
||||
if (ctx.agent)
|
||||
return [
|
||||
c.warn("Not signed in to HeyGen."),
|
||||
`Ask the person to run ${c.accent("! npx hyperframes auth login")} in this prompt, then wait until they say it is done.`,
|
||||
];
|
||||
if (!ctx.interactive) {
|
||||
return [
|
||||
c.warn("Not signed in to HeyGen (non-interactive)."),
|
||||
|
||||
@@ -51,7 +51,6 @@ describe("auth status — persisted user block surface", () => {
|
||||
afterEach(async () => {
|
||||
consumeCommandResult();
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
await envFixture.restore();
|
||||
});
|
||||
|
||||
@@ -132,31 +131,6 @@ describe("auth status — persisted user block surface", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("gives no 'balance unavailable' notice when the HyperFrames plan is shown", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
async () =>
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
data: {
|
||||
subscription: { plan_name: "pro", status: "active" },
|
||||
usage: { remaining_units: 25000 },
|
||||
},
|
||||
}),
|
||||
),
|
||||
);
|
||||
await configureBillingCredential("hyperframes");
|
||||
expect(await runStatus(true)).toBe(0);
|
||||
const payload = lastJson();
|
||||
expect(payload["hyperframes"]).toEqual({ plan: "pro", credits: 25000 });
|
||||
expect(payload).not.toHaveProperty("billing_notice");
|
||||
stdout = [];
|
||||
expect(await runStatus(false)).toBe(0);
|
||||
const text = stdout.join("\n");
|
||||
expect(text).toContain("pro plan, 25,000 credits");
|
||||
expect(text).not.toContain("balance unavailable");
|
||||
});
|
||||
|
||||
it("surfaces the persisted user block (with resolved display_name) for a file credential", async () => {
|
||||
await writeStore({
|
||||
api_key: "hg_x",
|
||||
|
||||
@@ -97,15 +97,6 @@ describe("buildUnconfiguredLines — offline engine readiness", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildUnconfiguredLines — a coding agent", () => {
|
||||
const text = joined({ interactive: true, agent: true });
|
||||
|
||||
it("asks the person to sign in from the prompt, with no local or API-key detour", () => {
|
||||
expect(text).toContain("! npx hyperframes auth login");
|
||||
expect(text).not.toMatch(/local engines|api-key|Kokoro/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildUnconfiguredJson", () => {
|
||||
it("recommends auth login and reports the local fallback", () => {
|
||||
for (const ctx of [INTERACTIVE, NON_INTERACTIVE]) {
|
||||
|
||||
@@ -31,7 +31,6 @@ import {
|
||||
} from "../../auth/index.js";
|
||||
import { getSystemMeta } from "../../telemetry/system.js";
|
||||
import { c } from "../../ui/colors.js";
|
||||
import { pageUrl, readPlan, type Plan } from "../../heygen/plan.js";
|
||||
import { resolveMusic, resolveVoice } from "../../audio/providers.js";
|
||||
import {
|
||||
buildUnconfiguredJson,
|
||||
@@ -62,8 +61,6 @@ interface VerifiedStatus {
|
||||
*/
|
||||
persistedUser: StoredUserInfo | null;
|
||||
apiError: string | null;
|
||||
/** The HyperFrames plan and credits, when the server (or the mock) answers. */
|
||||
plan?: Plan | null;
|
||||
}
|
||||
|
||||
/** True for credentials resolved from the HyperFrames store (not env). */
|
||||
@@ -109,10 +106,7 @@ export default defineCommand({
|
||||
*/
|
||||
function detectUnconfiguredContext(): UnconfiguredContext {
|
||||
const sys = getSystemMeta();
|
||||
return {
|
||||
interactive: !sys.is_ci && (sys.is_tty || sys.agent_runtime !== null),
|
||||
agent: !sys.is_ci && sys.agent_runtime !== null,
|
||||
};
|
||||
return { interactive: !sys.is_ci && (sys.is_tty || sys.agent_runtime !== null) };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -170,7 +164,7 @@ async function verify(credential: ResolvedCredential): Promise<VerifiedStatus> {
|
||||
const persistedUser = await loadPersistedUser(credential);
|
||||
try {
|
||||
const user = await client.getCurrentUser(credential);
|
||||
return { credential, user, persistedUser, apiError: null, plan: await readPlan() };
|
||||
return { credential, user, persistedUser, apiError: null };
|
||||
} catch (err) {
|
||||
if (!isAuthError(err)) throw err;
|
||||
return {
|
||||
@@ -211,17 +205,13 @@ function printJsonStatus(s: VerifiedStatus): void {
|
||||
// unchanged. Lets callers read identity offline / on an API blip.
|
||||
persisted_user: persistedUserJson(s.persistedUser),
|
||||
api_error: s.apiError,
|
||||
hyperframes: s.plan ?? null,
|
||||
};
|
||||
if (s.credential.type === "oauth") {
|
||||
payload["hyperframes_balance"] = null;
|
||||
// With the plan read, `hyperframes` above holds the plan and credits, so there is no notice to give.
|
||||
if (product !== "hyperframes")
|
||||
payload["billing_notice"] =
|
||||
"OAuth billing product is not verified here; user billing fields describe the HeyGen account.";
|
||||
else if (!s.plan)
|
||||
payload["billing_notice"] =
|
||||
"HyperFrames balance is unavailable here; user billing fields describe the HeyGen account.";
|
||||
payload["billing_notice"] =
|
||||
product === "hyperframes"
|
||||
? "HyperFrames balance is unavailable here; user billing fields describe the HeyGen account."
|
||||
: "OAuth billing product is not verified here; user billing fields describe the HeyGen account.";
|
||||
payload["expires_at"] = s.credential.expires_at?.toISOString() ?? null;
|
||||
payload["refreshable"] = s.credential.refreshable;
|
||||
payload["scope"] = s.credential.scope ?? null;
|
||||
@@ -267,17 +257,7 @@ function collectStatusRows(s: VerifiedStatus): [string, string][] {
|
||||
if (cached) rows.push(["Account:", `${cached} ${c.dim("(cached)")}`]);
|
||||
return rows;
|
||||
}
|
||||
if (s.user) rows.push(...identityRows(s.user, billingProduct(s.credential), !!s.plan));
|
||||
const plan = s.plan;
|
||||
if (plan) {
|
||||
const tag = plan.mock ? ` ${c.dim("(mock)")}` : "";
|
||||
rows.push([
|
||||
"HyperFrames:",
|
||||
plan.plan === "none"
|
||||
? `Free · Upgrade to Pro: ${pageUrl("plans")}${tag}`
|
||||
: `${plan.plan} plan, ${plan.credits.toLocaleString("en-US")} credits${tag}`,
|
||||
]);
|
||||
}
|
||||
if (s.user) rows.push(...identityRows(s.user, billingProduct(s.credential)));
|
||||
return rows;
|
||||
}
|
||||
|
||||
@@ -294,21 +274,14 @@ function oauthRows(credential: Extract<ResolvedCredential, { type: "oauth" }>):
|
||||
return rows;
|
||||
}
|
||||
|
||||
/** `hasPlan`: the HyperFrames line below shows the plan and credits, so no "unavailable" balance above it. */
|
||||
function identityRows(
|
||||
user: UserInfo,
|
||||
product: BillingProduct,
|
||||
hasPlan = false,
|
||||
): [string, string][] {
|
||||
function identityRows(user: UserInfo, product: BillingProduct): [string, string][] {
|
||||
const identity = user.email ?? user.username ?? "(unknown user)";
|
||||
const rows: [string, string][] = [["Account:", identity]];
|
||||
if (product === "hyperframes")
|
||||
return [
|
||||
...rows,
|
||||
["Billing:", "HyperFrames"],
|
||||
...(hasPlan
|
||||
? []
|
||||
: [["Balance:", "HyperFrames balance unavailable here."] as [string, string]]),
|
||||
["Balance:", "HyperFrames balance unavailable here."],
|
||||
];
|
||||
if (product === "unknown")
|
||||
return [
|
||||
|
||||
@@ -1,63 +0,0 @@
|
||||
import { openBrowser } from "../auth/browser.js";
|
||||
import { pageUrl, readPlan, type Plan } from "../heygen/plan.js";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { failCommand } from "../utils/commandResult.js";
|
||||
import { defineCommand } from "../utils/defineCommand.js";
|
||||
|
||||
const describe = (p: Plan | null) =>
|
||||
p && p.plan !== "none" ? `${p.plan} plan, ${p.credits.toLocaleString("en-US")} credits` : "Free";
|
||||
|
||||
/** Opens the plans or credits page, then waits until the account shows the change, so the run continues on its own. */
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "billing",
|
||||
description:
|
||||
"Open the HyperFrames plans or credits page and wait until the person is done (subscribe | top-up)",
|
||||
},
|
||||
args: {
|
||||
action: { type: "positional", description: "subscribe | top-up", required: true },
|
||||
needed: {
|
||||
type: "string",
|
||||
description: "Credits the waiting step needs (top-up waits until the balance covers it)",
|
||||
},
|
||||
tool: { type: "string", description: "The step that is waiting, shown on the page" },
|
||||
timeout: { type: "string", description: "Seconds to wait (default 600)" },
|
||||
},
|
||||
// fallow-ignore-next-line complexity
|
||||
async run({ args }) {
|
||||
const action = String(args.action);
|
||||
if (action !== "subscribe" && action !== "top-up") {
|
||||
console.error(c.error("Use `billing subscribe` or `billing top-up`."));
|
||||
failCommand();
|
||||
}
|
||||
const needed = Number(args.needed) || undefined;
|
||||
const before = await readPlan();
|
||||
const done = (p: Plan | null) =>
|
||||
action === "subscribe"
|
||||
? !!p && p.plan !== "none"
|
||||
: !!p && p.plan !== "none" && p.credits >= (needed ?? (before?.credits ?? 0) + 1);
|
||||
if (done(before) && (action === "subscribe" || needed)) {
|
||||
console.log(c.success(`✓ Already covered: ${describe(before)}.`));
|
||||
return;
|
||||
}
|
||||
const url = pageUrl(action === "subscribe" ? "plans" : "credits", { needed, tool: args.tool });
|
||||
console.log(`${action === "subscribe" ? "Plans" : "Credits"} page: ${c.accent(url)}`);
|
||||
await openBrowser(url);
|
||||
console.log("Waiting for the person to finish there ...");
|
||||
const deadline = Date.now() + (Number(args.timeout) || 600) * 1000;
|
||||
while (Date.now() < deadline) {
|
||||
await new Promise((r) => setTimeout(r, 2000));
|
||||
const now = await readPlan();
|
||||
if (done(now)) {
|
||||
console.log(c.success(`✓ Done: ${describe(now)}. Carry on.`));
|
||||
return;
|
||||
}
|
||||
}
|
||||
console.error(
|
||||
c.error(
|
||||
`Still waiting on the ${action === "subscribe" ? "plans" : "credits"} page; run this again once it is done.`,
|
||||
),
|
||||
);
|
||||
failCommand();
|
||||
},
|
||||
});
|
||||
@@ -1,256 +0,0 @@
|
||||
import { basename, extname, join, resolve } from "node:path";
|
||||
import { defineCommand } from "../utils/defineCommand.js";
|
||||
import type { Example } from "./_examples.js";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { failCommand, setCommandExitCode } from "../utils/commandResult.js";
|
||||
import { normalizeErrorMessage } from "../utils/errorMessage.js";
|
||||
import { withMeta } from "../utils/updateCheck.js";
|
||||
import {
|
||||
DEFAULT_MIN_CELL,
|
||||
DEFAULT_MIN_MEAN,
|
||||
DEFAULT_MIN_P05,
|
||||
diffVideos,
|
||||
findLedger,
|
||||
type DiffResult,
|
||||
} from "../measure/diff.js";
|
||||
import { Fix, formatJson, outDir, parseNumbers, parseRange } from "../measure/media.js";
|
||||
import { readSegments } from "../measure/segments.js";
|
||||
import { repeated } from "./media-use/generate.js";
|
||||
import { parseMask } from "../heygen/generate.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"Score a render against the original, frame by frame",
|
||||
"hyperframes diff original.mp4 renders/final.mp4",
|
||||
],
|
||||
[
|
||||
"Score one segment of a project without rendering it",
|
||||
"hyperframes diff original.mp4 . --range 2.1,4.5",
|
||||
],
|
||||
[
|
||||
"Score chosen moments of a project without rendering it",
|
||||
"hyperframes diff original.mp4 . --at 0.5,2.4,3.1",
|
||||
],
|
||||
[
|
||||
"Score one drawn element (a title's box) by itself at chosen moments of a project",
|
||||
"hyperframes diff original.mp4 . --at 1,1.5,2 --drawn 120,670,485,115",
|
||||
],
|
||||
[
|
||||
"Judge each segment of the final render and write a side-by-side",
|
||||
"hyperframes diff original.mp4 renders/final.mp4 --segments segments.md --sbs",
|
||||
],
|
||||
];
|
||||
|
||||
function number(value: unknown, flag: string): number | undefined {
|
||||
if (value === undefined) return undefined;
|
||||
const [n] = parseNumbers(value, flag);
|
||||
return n;
|
||||
}
|
||||
|
||||
/** `x,y,w,h` or `x,y,w,h,a-b` in the original's pixels and seconds. */
|
||||
function parseDrawn(text: string): ReturnType<typeof parseMask> {
|
||||
try {
|
||||
return parseMask(text);
|
||||
} catch {
|
||||
throw new Fix(`--drawn ${text} is not x,y,w,h[,a-b]`, "--drawn 120,670,485,115,0-2.77");
|
||||
}
|
||||
}
|
||||
|
||||
function personLine(p: NonNullable<DiffResult["segments"]>[number]["person"] & object): string {
|
||||
const off = [
|
||||
...(p.framing ?? [])
|
||||
.filter((f) => !f.pass)
|
||||
.map(
|
||||
(f) => `framing off ${f.window[0]}–${f.window[1]} s (shift ${f.shift}, size ×${f.scale})`,
|
||||
),
|
||||
...(p.text ?? []).filter((t) => t.carried).map((t) => `${t.clip} carries ${t.drawn}'s text`),
|
||||
...(p.sync ?? [])
|
||||
.filter((x) => x.pass === false)
|
||||
.map((x) => `lips ${x.lag_ms} ms off at ${x.window[0]} s`),
|
||||
];
|
||||
const moved = p.energy ? `, motion ×${p.energy.mean_ratio} (reported)` : "";
|
||||
return `person ${p.pass === null ? "—" : p.pass ? "✓" : "✗"}${off.length ? ` ${off.join("; ")}` : ""}${moved}`;
|
||||
}
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
function printHuman(r: DiffResult) {
|
||||
const mark = r.pass ? c.success("✓ pass") : c.error("✗ fail");
|
||||
console.log(
|
||||
`${mark} SSIM mean ${r.ssim.mean}, p05 ${r.ssim.p05}, worst cell p05 ${r.ssim.cell_p05} over ${r.ssim.frames} frames (${r.window[0]}–${r.window[1]} s)`,
|
||||
);
|
||||
if (!r.missing_content.pass)
|
||||
console.log(
|
||||
` missing content in ${Math.round(r.missing_content.share * 100)}% of frames: ${r.missing_content.spans.map(([a, b]) => `${a}–${b} s`).join(", ")}`,
|
||||
);
|
||||
if (r.missing_content.texture.share > 0)
|
||||
console.log(
|
||||
` missing texture (reported, not failed) in ${Math.round(r.missing_content.texture.share * 100)}% of frames: ${r.missing_content.texture.spans.map(([a, b]) => `${a}–${b} s`).join(", ")}`,
|
||||
);
|
||||
for (const m of r.worst)
|
||||
console.log(` worst ${m.t}s ssim ${m.ssim} cell ${m.cell.roi.join(",")} at ${m.cell.ssim}`);
|
||||
if (r.whole_frame_reuse?.spans.length)
|
||||
console.log(
|
||||
` whole frame of the original in ${Math.round(r.whole_frame_reuse.share * 100)}% of frames${r.whole_frame_reuse.pass ? "" : `, with its drawn things at ${r.whole_frame_reuse.drawn.map(([a, b]) => `${a}–${b} s`).join(", ")} (a copy)`}`,
|
||||
);
|
||||
if (r.cuts)
|
||||
console.log(
|
||||
` cuts: ${r.cuts.reference.length} in the original${r.cuts.pass ? ", all matched" : `; missing ${r.cuts.missing.join(", ") || "none"}, extra ${r.cuts.extra.join(", ") || "none"}`}`,
|
||||
);
|
||||
if (r.duration)
|
||||
console.log(
|
||||
` duration: original ${r.duration.reference} s, yours ${r.duration.yours} s${r.duration.pass ? "" : " (differ)"}`,
|
||||
);
|
||||
if (r.sound)
|
||||
console.log(
|
||||
` sound: ${"offset_ms" in r.sound ? (r.sound.offset_ms === null ? (r.sound.note ?? "not aligned") : `${r.sound.offset_ms} ms off`) : `original ${r.sound.original ? "has" : "has no"} sound, yours ${r.sound.yours ? "has" : "has no"}`}${r.sound.pass ? "" : " (fails)"}`,
|
||||
);
|
||||
if (r.components)
|
||||
console.log(
|
||||
` components: ${r.components.passed} of ${r.components.total} pass${r.components.mean === null ? "" : `, mean ${r.components.mean}`}`,
|
||||
);
|
||||
if (r.person) {
|
||||
const g = r.person.gates;
|
||||
const mark = (v: boolean | null, failed: string[]) =>
|
||||
v === null ? "unmeasured" : v ? "ok" : `fails in ${failed.join(", ")}`;
|
||||
console.log(
|
||||
` person: framing ${mark(g.framing, r.person.failed.framing)}, text in the clip ${mark(g.text, r.person.failed.text)}, lips ${mark(g.sync, r.person.failed.sync)}; reported: motion ×${r.person.energy ?? "?"}, ${r.person.shots_at_cuts.at_cuts} of ${r.person.shots_at_cuts.total} clips start at a cut${r.person.shots_at_cuts.cuts_without_shot.length ? `, no new clip at ${r.person.shots_at_cuts.cuts_without_shot.join(", ")} s` : ""}`,
|
||||
);
|
||||
}
|
||||
console.log(` every frame: ${r.frames}`);
|
||||
for (const s of r.segments ?? []) {
|
||||
const state = s.pass === null ? c.dim("—") : s.pass ? c.success("✓") : c.error("✗");
|
||||
const detail =
|
||||
[
|
||||
s.ssim && `ssim ${s.ssim.mean} / p05 ${s.ssim.p05} / cell ${s.ssim.cell_p05}`,
|
||||
s.energy && `motion ×${s.energy.mean_ratio}, luma ×${s.look?.luma_ratio}`,
|
||||
s.person && personLine(s.person),
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join("; ") ||
|
||||
(s.note ?? "");
|
||||
const region = s.regenerated ? `; person in ${s.regenerated.roi.join(",")}` : "";
|
||||
console.log(
|
||||
` ${state} ${s.id} ${s.window[0]}–${s.window[1]} s ${s.pixels}${s.regenerated ? " outside the region" : ""}: ${detail}${region}`,
|
||||
);
|
||||
}
|
||||
for (const d of r.drawn ?? []) {
|
||||
const state = d.pass === null ? c.dim("—") : d.pass ? c.success("✓") : c.error("✗");
|
||||
const detail =
|
||||
d.pass === null
|
||||
? "no ink found in the box (crop it tight around the letters)"
|
||||
: `score ${d.score} / p05 ${d.p05}, off by ${d.offset!.join(",")} px${d.ink === undefined ? "" : `, ink ×${d.ink}`}`;
|
||||
console.log(
|
||||
` ${state} ${d.id} ${d.box.join(",")} ${d.window[0]}–${d.window[1]} s: ${detail}`,
|
||||
);
|
||||
}
|
||||
for (const w of r.warnings) console.log(` ${c.warn("⚠")} ${w}`);
|
||||
if (r.sheet) console.log(` sheet ${r.sheet}`);
|
||||
if (r.sbs) console.log(` side by side ${r.sbs}`);
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "diff",
|
||||
description:
|
||||
"Score a render or a project against the original video frame by frame: SSIM, worst moments, cuts, segments",
|
||||
},
|
||||
args: {
|
||||
reference: { type: "positional", description: "The original video", required: true },
|
||||
yours: {
|
||||
type: "positional",
|
||||
description:
|
||||
"A rendered video, or a project folder (captured at the original's frame times, no render)",
|
||||
required: true,
|
||||
},
|
||||
at: { type: "string", description: "Compare only these seconds, e.g. 0.5,2.4" },
|
||||
range: { type: "string", description: "Compare every frame of start,end seconds" },
|
||||
segments: {
|
||||
type: "string",
|
||||
description:
|
||||
"A file of `## S01 · a–b s` segments: a verdict per segment (Pixels: regenerated compares motion and look)",
|
||||
},
|
||||
ledger: {
|
||||
type: "string",
|
||||
description:
|
||||
"The project's media ledger, to find whole-frame cuts of the original (default: .media/manifest.jsonl beside the project or REVERSE.md)",
|
||||
},
|
||||
project: {
|
||||
type: "string",
|
||||
description:
|
||||
"The project folder, to find the ledger's generated clips and check them for the original's text (default: the folder holding .media)",
|
||||
},
|
||||
"min-mean": { type: "string", description: `Mean SSIM to pass (default ${DEFAULT_MIN_MEAN})` },
|
||||
"min-p05": {
|
||||
type: "string",
|
||||
description: `5th-percentile SSIM to pass (default ${DEFAULT_MIN_P05})`,
|
||||
},
|
||||
"min-cell": {
|
||||
type: "string",
|
||||
description: `5th percentile of each frame's worst 4x4 cell to pass (default ${DEFAULT_MIN_CELL})`,
|
||||
},
|
||||
worst: { type: "string", description: "Worst moments to show (default 4)" },
|
||||
sbs: {
|
||||
type: "boolean",
|
||||
description: "Also write an original | yours side-by-side video",
|
||||
default: false,
|
||||
},
|
||||
out: {
|
||||
type: "string",
|
||||
description: "Folder for the sheet and log (default .hyperframes/diff/<yours>)",
|
||||
},
|
||||
timeout: {
|
||||
type: "string",
|
||||
description: "Render-ready timeout in ms for a project folder (default 5000)",
|
||||
},
|
||||
drawn: {
|
||||
type: "string",
|
||||
description:
|
||||
"A drawn element's box x,y,w,h in the original's pixels, optionally its seconds a-b (repeatable): scored by itself whatever is behind it, with its offset and ink (yours over the original's: under 1 is lighter, thinner or smaller)",
|
||||
},
|
||||
json: { type: "boolean", description: "Print one JSON result", default: false },
|
||||
},
|
||||
// fallow-ignore-next-line complexity
|
||||
async run({ args, rawArgs }) {
|
||||
const json = args.json === true;
|
||||
try {
|
||||
if (args.at !== undefined && args.range !== undefined)
|
||||
throw new Fix("--at and --range both given", "pass one of them");
|
||||
const yours = resolve(String(args.yours));
|
||||
const stem = basename(yours, extname(yours)) || "project";
|
||||
const segmentsFile = args.segments === undefined ? undefined : resolve(String(args.segments));
|
||||
const result = await diffVideos({
|
||||
reference: resolve(String(args.reference)),
|
||||
yours,
|
||||
at: args.at === undefined ? undefined : parseNumbers(args.at, "--at"),
|
||||
range: args.range === undefined ? undefined : parseRange(args.range),
|
||||
segments: segmentsFile === undefined ? undefined : readSegments(segmentsFile),
|
||||
ledger:
|
||||
args.ledger === undefined
|
||||
? findLedger(yours, segmentsFile)
|
||||
: resolve(String(args.ledger)),
|
||||
project: args.project === undefined ? undefined : resolve(String(args.project)),
|
||||
drawn: repeated(rawArgs, "--drawn").map(parseDrawn),
|
||||
thresholds: {
|
||||
mean: number(args["min-mean"], "--min-mean"),
|
||||
p05: number(args["min-p05"], "--min-p05"),
|
||||
cell: number(args["min-cell"], "--min-cell"),
|
||||
},
|
||||
worst: number(args.worst, "--worst"),
|
||||
sbs: args.sbs === true,
|
||||
outDir: outDir(args.out, join("diff", stem)),
|
||||
timeoutMs: number(args.timeout, "--timeout"),
|
||||
log: (message) => console.error(message),
|
||||
});
|
||||
if (json) console.log(formatJson(withMeta({ ok: true, ...result })));
|
||||
else printHuman(result);
|
||||
if (!result.pass) setCommandExitCode(1);
|
||||
} catch (err) {
|
||||
const fix = err instanceof Fix ? err.fix : undefined;
|
||||
const message = normalizeErrorMessage(err);
|
||||
if (json)
|
||||
console.log(formatJson(withMeta({ ok: false, error: message, ...(fix ? { fix } : {}) })));
|
||||
else console.error(`${c.error("✗")} ${message}${fix ? `\n Fix: ${fix}` : ""}`);
|
||||
failCommand();
|
||||
}
|
||||
},
|
||||
});
|
||||
@@ -1,34 +0,0 @@
|
||||
import { defineCommand } from "../utils/defineCommand.js";
|
||||
import type { Example } from "./_examples.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"Size, frame rate and what to fix before rebuilding a video",
|
||||
"hyperframes measure probe ref.mp4",
|
||||
],
|
||||
[
|
||||
"Everything that changes when, with the sound's hits",
|
||||
"hyperframes measure events ref.mp4 --map",
|
||||
],
|
||||
["The frames around every event, as strips", "hyperframes measure frames ref.mp4 --events"],
|
||||
];
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "measure",
|
||||
description:
|
||||
"Measure a video file as JSON: clock, cuts, events, frames, crops, eases, motion energy, look, faces, lip sync",
|
||||
},
|
||||
subCommands: {
|
||||
probe: () => import("./measure/probe.js").then((m) => m.default),
|
||||
cuts: () => import("./measure/cuts.js").then((m) => m.default),
|
||||
events: () => import("./measure/events.js").then((m) => m.default),
|
||||
frames: () => import("./measure/frames.js").then((m) => m.default),
|
||||
extract: () => import("./measure/extract.js").then((m) => m.default),
|
||||
curve: () => import("./measure/curve.js").then((m) => m.default),
|
||||
energy: () => import("./measure/energy.js").then((m) => m.default),
|
||||
look: () => import("./measure/look.js").then((m) => m.default),
|
||||
face: () => import("./measure/face.js").then((m) => m.default),
|
||||
sync: () => import("./measure/sync.js").then((m) => m.default),
|
||||
},
|
||||
});
|
||||
@@ -1,84 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import {
|
||||
compareWindowArgs,
|
||||
jsonArg,
|
||||
num,
|
||||
roiArg,
|
||||
runMeasure,
|
||||
videoAndRoi,
|
||||
videoArg,
|
||||
window,
|
||||
} from "../../measure/command.js";
|
||||
import { compareCurves, curves, printable } from "../../measure/curve.js";
|
||||
import { Fix, parseRoi, probeVideo } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"The ease, duration and overshoot of one element's move",
|
||||
"hyperframes measure curve original.mp4 --start 2.1 --end 3 --roi 240,280,600,540",
|
||||
],
|
||||
[
|
||||
"The same move in your render, paired with the original's",
|
||||
"hyperframes measure curve original.mp4 --start 2.1 --end 3 --roi 240,280,600,540 --compare renders/final.mp4 --compare-start 2.1",
|
||||
],
|
||||
];
|
||||
|
||||
/** "#rrggbb" as RGB. */
|
||||
function hex(text: string): number[] {
|
||||
const m = /^#?([0-9a-f]{6})$/i.exec(text);
|
||||
if (!m) throw new Fix(`--bg must be #rrggbb, got "${text}"`, "--bg #101820");
|
||||
return [0, 2, 4].map((i) => parseInt(m[1]!.slice(i, i + 2), 16));
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "curve",
|
||||
description:
|
||||
"Each move of one element on a flat background: start, duration, overshoot and the closest GSAP ease",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
start: { type: "string", description: "Window start in seconds (default 0)" },
|
||||
end: { type: "string", description: "Window end in seconds (default: the end)" },
|
||||
roi: {
|
||||
...roiArg,
|
||||
description: "Region holding the whole move, overshoot included, and nothing else that moves",
|
||||
},
|
||||
bg: { type: "string", description: "Background #rrggbb (default: the ROI border's median)" },
|
||||
thr: {
|
||||
type: "string",
|
||||
description: "Pixel distance from the background that counts as the element (24)",
|
||||
},
|
||||
compare: {
|
||||
type: "string",
|
||||
description: "A second video (your render) to measure the same way",
|
||||
},
|
||||
...compareWindowArgs,
|
||||
json: jsonArg,
|
||||
},
|
||||
run: ({ args }) =>
|
||||
runMeasure(() => {
|
||||
const { info, roi } = videoAndRoi(args.video, args.roi);
|
||||
const [start, end] = window(info, args.start, args.end);
|
||||
const opts = { bg: args.bg ? hex(args.bg) : null, thr: num(args.thr, "--thr", 24) };
|
||||
const ref = curves(info.path, start, end - start, roi, opts);
|
||||
if (!args.compare) return { video: info.path, ...ref, moves: printable(ref.moves) };
|
||||
const other = probeVideo(args.compare);
|
||||
const yoursRoi = args["compare-roi"]
|
||||
? parseRoi(args["compare-roi"], other.width, other.height)
|
||||
: roi;
|
||||
const yours = curves(
|
||||
other.path,
|
||||
num(args["compare-start"], "--compare-start", start),
|
||||
end - start,
|
||||
yoursRoi,
|
||||
opts,
|
||||
);
|
||||
return {
|
||||
compare: compareCurves(ref, yours),
|
||||
reference: printable(ref.moves),
|
||||
yours: printable(yours.moves),
|
||||
};
|
||||
}),
|
||||
});
|
||||
@@ -1,52 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, runMeasure, videoArg, window } from "../../measure/command.js";
|
||||
import { findCuts, signatures } from "../../measure/cuts.js";
|
||||
import { probeVideo, round, videoClip } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
["Every hard cut and the shots between them", "hyperframes measure cuts ref.mp4"],
|
||||
["Cuts in the first 20 seconds", "hyperframes measure cuts ref.mp4 --start 0 --end 20"],
|
||||
];
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "cuts",
|
||||
description:
|
||||
"Hard cuts from the frame-to-frame difference (scene scores miss some), and the shots between them",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
start: { type: "string", description: "First second to read (default 0)" },
|
||||
end: { type: "string", description: "Last second to read (default: the end)" },
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
await runMeasure(() => {
|
||||
const info = probeVideo(String(args.video));
|
||||
const [start, end] = window(info, args.start, args.end);
|
||||
const clip = videoClip(info.path, start, end - start, info);
|
||||
const found = findCuts(signatures(clip.input, clip.filters).frames);
|
||||
const f0 = Math.ceil(start * info.fps - 1e-6);
|
||||
const cuts = found.map((c) => ({
|
||||
t: round((f0 + c.frame) / info.fps, 3),
|
||||
frame: f0 + c.frame,
|
||||
...(c.frames > 1 ? { frames: c.frames } : {}),
|
||||
change: round(c.change, 1),
|
||||
}));
|
||||
const edges = [start, ...cuts.map((c) => c.t), end];
|
||||
return {
|
||||
video: info.path,
|
||||
start,
|
||||
end: round(end, 3),
|
||||
fps: round(info.fps, 3),
|
||||
cuts,
|
||||
shots: edges.slice(1).map((e, i) => ({
|
||||
start: round(edges[i]!, 3),
|
||||
end: round(e, 3),
|
||||
seconds: round(e - edges[i]!, 2),
|
||||
})),
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,66 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, num, runMeasure } from "../../measure/command.js";
|
||||
import { compareEnergy, energyCurve } from "../../measure/energy.js";
|
||||
import { Fix, parseRoi, probeVideo, round, videoClip, type Roi } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"Does a regenerated shot move as much as the original's, and when",
|
||||
"hyperframes measure energy ref.mp4 renders/v1.mp4 --start 4 --seconds 5",
|
||||
],
|
||||
];
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "energy",
|
||||
description:
|
||||
"How much two clips move, and whether they move at the same moments, when their pixels differ",
|
||||
},
|
||||
args: {
|
||||
reference: { type: "positional", description: "The original", required: true },
|
||||
yours: { type: "positional", description: "The other video", required: true },
|
||||
start: { type: "string", description: "Second the window starts in the original (default 0)" },
|
||||
"yours-start": { type: "string", description: "Second it starts in yours (default: --start)" },
|
||||
seconds: { type: "string", description: "Window length (default: the rest of the original)" },
|
||||
roi: { type: "string", description: "Region of the original, x,y,w,h" },
|
||||
"yours-roi": {
|
||||
type: "string",
|
||||
description: "Region of yours (default: --roi scaled by the widths)",
|
||||
},
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
await runMeasure(() => {
|
||||
const [a, b] = [probeVideo(String(args.reference)), probeVideo(String(args.yours))];
|
||||
const start = num(args.start, "--start", 0);
|
||||
const yoursStart = num(args["yours-start"], "--yours-start", start);
|
||||
const seconds = num(args.seconds, "--seconds", a.duration - start);
|
||||
if (seconds <= 0.5) throw new Fix("the window is under half a second", "--seconds 2 or more");
|
||||
const roi = args.roi === undefined ? undefined : parseRoi(args.roi, a.width, a.height);
|
||||
const k = b.width / a.width;
|
||||
const scaled = (r: Roi): Roi => [
|
||||
Math.round(r[0] * k),
|
||||
Math.round(r[1] * k),
|
||||
Math.max(2, Math.round(r[2] * k)),
|
||||
Math.max(2, Math.round(r[3] * k)),
|
||||
];
|
||||
const yoursRoi =
|
||||
args["yours-roi"] !== undefined
|
||||
? parseRoi(args["yours-roi"], b.width, b.height)
|
||||
: roi && scaled(roi);
|
||||
const r = compareEnergy(
|
||||
energyCurve(videoClip(a.path, start, seconds, a), roi),
|
||||
energyCurve(videoClip(b.path, yoursStart, seconds, b), yoursRoi),
|
||||
);
|
||||
return {
|
||||
reference: a.path,
|
||||
yours: b.path,
|
||||
window_s: round(seconds, 3),
|
||||
...r,
|
||||
reading:
|
||||
"mean_ratio is how much yours moves (0.8–1.25 is the same amount); r near 1 at a small lag_s is the same action at the same time",
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,190 +0,0 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import {
|
||||
jsonArg,
|
||||
num,
|
||||
outArg,
|
||||
roiArg,
|
||||
runMeasure,
|
||||
videoAndRoi,
|
||||
videoArg,
|
||||
window,
|
||||
} from "../../measure/command.js";
|
||||
import {
|
||||
countEvents,
|
||||
detectEvents,
|
||||
drawStrip,
|
||||
lookWindows,
|
||||
type ScreenEvent,
|
||||
} from "../../measure/events.js";
|
||||
import { Fix, outDir, round, type VideoInfo } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"The film's event map: cuts, flashes, what appears where, holds, and the sound's hits, grouped into windows to look at",
|
||||
"hyperframes measure events ref.mp4 --map",
|
||||
],
|
||||
[
|
||||
"When one region changes, frame by frame",
|
||||
"hyperframes measure events ref.mp4 --roi 0,800,1080,300 --start 2 --end 6",
|
||||
],
|
||||
];
|
||||
|
||||
/** "6x10" as columns and rows. */
|
||||
function parseGrid(text: unknown): { cols: number; rows: number } {
|
||||
const m = /^(\d+)x(\d+)$/.exec(String(text));
|
||||
if (!m || Number(m[1]) < 1 || Number(m[2]) < 1)
|
||||
throw new Fix(`--grid must be COLSxROWS, got "${String(text)}"`, "--grid 6x10");
|
||||
return { cols: Number(m[1]), rows: Number(m[2]) };
|
||||
}
|
||||
|
||||
const isRecord = (v: unknown): v is Record<string, unknown> =>
|
||||
typeof v === "object" && v !== null && !Array.isArray(v);
|
||||
const numbers = (v: unknown): number[] =>
|
||||
Array.isArray(v) ? v.filter((x): x is number => typeof x === "number") : [];
|
||||
|
||||
interface Sound {
|
||||
bpm: number | null;
|
||||
beats: number[];
|
||||
onsets: { t: number; strength: number }[];
|
||||
}
|
||||
|
||||
/** The sound's onsets, beats and tempo, measured on this machine. */
|
||||
async function soundOf(info: VideoInfo, start: number, end: number): Promise<Sound | null> {
|
||||
if (!info.audio) return null;
|
||||
const { analyzeTargets, previewTarget } = await import("../../audio/analysis/run.js");
|
||||
const report = await analyzeTargets([await previewTarget(info.path, info.path, null)], {
|
||||
capabilities: ["onsets", "beats", "bpm"],
|
||||
quality: "local",
|
||||
});
|
||||
const item = report.items[0];
|
||||
if (!item) return null;
|
||||
const inside = (t: number) => t >= start && t <= end;
|
||||
const onsets =
|
||||
isRecord(item.onsets) && Array.isArray(item.onsets.events) ? item.onsets.events : [];
|
||||
return {
|
||||
bpm: isRecord(item.bpm) && typeof item.bpm.value === "number" ? round(item.bpm.value, 1) : null,
|
||||
beats: numbers(isRecord(item.beats) ? item.beats.times : undefined).filter(inside),
|
||||
onsets: onsets
|
||||
.filter(isRecord)
|
||||
.flatMap((e) =>
|
||||
typeof e.t === "number" && typeof e.strength === "number" && inside(e.t)
|
||||
? [{ t: e.t, strength: e.strength }]
|
||||
: [],
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
const LIST_MAX = 64;
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "events",
|
||||
description:
|
||||
"When things change on screen: cuts, flashes, appears, disappears, changes and holds; --map adds the sound and the windows worth looking at",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
map: {
|
||||
type: "boolean",
|
||||
description:
|
||||
"The whole window on a 4x4 grid with the sound's onsets and beats, grouped into windows to look at",
|
||||
default: false,
|
||||
},
|
||||
roi: roiArg,
|
||||
grid: {
|
||||
type: "string",
|
||||
description: "Split the frame into COLSxROWS cells, e.g. 6x10 (default with --map: 4x4)",
|
||||
},
|
||||
start: { type: "string", description: "First second (default 0)" },
|
||||
end: { type: "string", description: "Last second (default: the end)" },
|
||||
thr: {
|
||||
type: "string",
|
||||
description: "Mean grey-level change of an appear or a cut (default 5)",
|
||||
},
|
||||
out: outArg,
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
// fallow-ignore-next-line complexity
|
||||
await runMeasure(async () => {
|
||||
const { info, roi } = videoAndRoi(args.video, args.roi);
|
||||
const [start, end] = window(info, args.start, args.end);
|
||||
const grid =
|
||||
args.grid !== undefined
|
||||
? parseGrid(args.grid)
|
||||
: args.map && args.roi === undefined
|
||||
? { cols: 4, rows: 4 }
|
||||
: undefined;
|
||||
const thr = num(args.thr, "--thr", 5);
|
||||
const found = detectEvents({ info, start, end, roi, grid, thr });
|
||||
if (found.frames < 3)
|
||||
throw new Fix(`under 3 frames between ${start}s and ${end}s`, "widen --start/--end");
|
||||
const dir = outDir(args.out, "measure/events");
|
||||
const name = `events-${start}-${round(end, 2)}s`;
|
||||
const strip = found.shown.length ? drawStrip(found, join(dir, `${name}.png`)) : null;
|
||||
const holds = found.events.filter((e) => e.type === "hold");
|
||||
const head = {
|
||||
video: info.path,
|
||||
...(grid ? { grid: `${grid.cols}x${grid.rows}` } : { roi }),
|
||||
start,
|
||||
end: round(end, 3),
|
||||
fps: round(info.fps, 3),
|
||||
};
|
||||
if (!args.map) {
|
||||
const lengths = holds.map((e) => e.seconds ?? 0).sort((a, b) => a - b);
|
||||
return {
|
||||
...head,
|
||||
frames: found.frames,
|
||||
thr,
|
||||
counts: countEvents(found.events),
|
||||
holds: {
|
||||
longest_s: lengths.at(-1) ?? 0,
|
||||
median_s: lengths[(lengths.length - 1) >> 1] ?? 0,
|
||||
count: lengths.length,
|
||||
},
|
||||
strip,
|
||||
strip_events: `${found.shown.length} of ${found.events.length - holds.length} (cuts and flashes, then the largest steps, then changes)`,
|
||||
events: found.events,
|
||||
};
|
||||
}
|
||||
const sound = await soundOf(info, start, end);
|
||||
const cuts = found.events.filter((e) => e.type === "cut").map((e) => e.t);
|
||||
const edges = [start, ...cuts, end];
|
||||
const windows = lookWindows(found.events, info.fps, sound?.onsets.map((o) => o.t) ?? []);
|
||||
const file = join(dir, `${name}-map.json`);
|
||||
writeFileSync(
|
||||
file,
|
||||
JSON.stringify({ ...head, events: found.events, sound, windows }, null, 1),
|
||||
);
|
||||
const strong = sound?.onsets.filter((o) => o.strength >= 0.5).map((o) => round(o.t, 3)) ?? [];
|
||||
const longHolds = holds
|
||||
.filter((e: ScreenEvent) => (e.seconds ?? 0) >= 0.5)
|
||||
.map((e) => ({ t: e.t, seconds: e.seconds }));
|
||||
return {
|
||||
...head,
|
||||
duration: round(end - start, 3),
|
||||
counts: countEvents(found.events),
|
||||
shots: edges.slice(1).map((e, i) => ({ start: round(edges[i]!, 3), end: round(e, 3) })),
|
||||
holds: longHolds,
|
||||
sound: sound
|
||||
? {
|
||||
bpm: sound.bpm,
|
||||
beats: sound.beats.length,
|
||||
...(sound.beats.length
|
||||
? { beat_times: sound.beats.slice(0, LIST_MAX).map((t) => round(t, 3)) }
|
||||
: {}),
|
||||
onsets: sound.onsets.length,
|
||||
strong_onsets: strong.slice(0, LIST_MAX),
|
||||
}
|
||||
: "none",
|
||||
windows,
|
||||
strip,
|
||||
file,
|
||||
next: "look at each window's frames: npx hyperframes measure frames <video> --events",
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,274 +0,0 @@
|
||||
// Pixels from the original, as project media: a still at one moment or a clip over a window, cropped to a region,
|
||||
// frame-exact, recorded in the media ledger as cut from the original. A rebuild reuses footage, photos and people
|
||||
// this way instead of drawing or generating them.
|
||||
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, num, roiArg, runMeasure, videoAndRoi, videoArg } from "../../measure/command.js";
|
||||
import { Fix, frameAt, round, runFfmpeg, type Roi, type VideoInfo } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"A photo from the original at 3.2 s, cropped to its box",
|
||||
"hyperframes measure extract original.mp4 --at 3.2 --roi 120,340,840,620 --out assets/photo.png",
|
||||
],
|
||||
[
|
||||
"The original's whole soundtrack",
|
||||
"hyperframes measure extract original.mp4 --out assets/sound.m4a",
|
||||
],
|
||||
[
|
||||
"The talking head from 4.0 s to 9.5 s with its sound",
|
||||
"hyperframes measure extract original.mp4 --from 4 --to 9.5 --roi 0,0,1080,1350 --audio --out assets/presenter.mp4",
|
||||
],
|
||||
];
|
||||
|
||||
/** H.264 needs even sizes: shrink an odd width or height by a pixel. */
|
||||
const even = ([x, y, w, h]: Roi): Roi => [x, y, w - (w % 2), h - (h % 2)];
|
||||
|
||||
/** The nearest folder at or above `from` holding index.html: the project the file belongs to. */
|
||||
function projectOf(from: string): string | null {
|
||||
for (let dir = resolve(from); ; dir = dirname(dir)) {
|
||||
if (existsSync(join(dir, "index.html"))) return dir;
|
||||
if (dirname(dir) === dir) return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function record(
|
||||
out: string,
|
||||
type: "image" | "video" | "bgm",
|
||||
provenance: Record<string, unknown>,
|
||||
duration?: number,
|
||||
) {
|
||||
const project = projectOf(dirname(out));
|
||||
if (!project) return null;
|
||||
const { recordInPlace } = await import("../../media-use/lib/manifest.mjs");
|
||||
const { regenerateIndex } = await import("../../media-use/lib/index-gen.mjs");
|
||||
const rel = (p: string) => relative(project, p).split("\\").join("/");
|
||||
const path = rel(out);
|
||||
const from = String(provenance.from);
|
||||
if (!rel(from).startsWith("..")) provenance = { ...provenance, from: rel(from) };
|
||||
const entry = recordInPlace(project, {
|
||||
type,
|
||||
path,
|
||||
source: "extracted",
|
||||
description: `cut from the original ${String(provenance.window ?? provenance.at)}`,
|
||||
...(duration !== undefined ? { duration } : {}),
|
||||
provenance,
|
||||
});
|
||||
regenerateIndex(project);
|
||||
return entry.id;
|
||||
}
|
||||
|
||||
const crop = (roi: Roi) => `crop=${roi[2]}:${roi[3]}:${roi[0]}:${roi[1]}`;
|
||||
|
||||
/** The frame shown at `at` seconds, cropped, as a picture. */
|
||||
export async function cutStill(info: VideoInfo, at: number, roi: Roi, out: string) {
|
||||
const k = frameAt(at, info.fps);
|
||||
const t = k / info.fps;
|
||||
const { extractVideoFrameToBuffer } = await import("../snapshot.js");
|
||||
const png = await extractVideoFrameToBuffer(info.path, t);
|
||||
if (!png)
|
||||
throw new Fix(`no frame at ${round(t, 3)}s`, `pick a time before ${round(info.duration, 2)}s`);
|
||||
const work = mkdtempSync(join(tmpdir(), "hf-extract-"));
|
||||
try {
|
||||
const full = join(work, "frame.png");
|
||||
writeFileSync(full, png);
|
||||
runFfmpeg(["-y", "-i", full, "-vf", crop(roi), "-frames:v", "1", out]);
|
||||
} finally {
|
||||
rmSync(work, { recursive: true, force: true });
|
||||
}
|
||||
return { at: round(t, 3), frame: k };
|
||||
}
|
||||
|
||||
/** The frames shown from `from` to `to` seconds, cropped (to even sizes), as an H.264 clip. */
|
||||
export function cutClip(
|
||||
info: VideoInfo,
|
||||
from: number,
|
||||
to: number,
|
||||
asked: Roi,
|
||||
out: string,
|
||||
opts: { audio?: boolean; crf?: number } = {},
|
||||
) {
|
||||
const k0 = frameAt(from, info.fps);
|
||||
const k1 = Math.min(info.frames, Math.ceil(to * info.fps - 1e-6));
|
||||
if (k1 <= k0) throw new Fix("--to is not after --from", "--from 4 --to 9.5");
|
||||
const roi = even(asked);
|
||||
const [a, n] = [k0 / info.fps, k1 - k0];
|
||||
const sound = Boolean(opts.audio && info.audio);
|
||||
// Frames picked by index, as diff and the event map count them: a seek keeps whichever frame is showing at the
|
||||
// seek time, which at some rates is the one before k0.
|
||||
runFfmpeg([
|
||||
...["-y", "-i", info.path],
|
||||
...[
|
||||
"-vf",
|
||||
`trim=start_frame=${k0}:end_frame=${k1},setpts=PTS-STARTPTS,${crop(roi)},format=yuv420p`,
|
||||
],
|
||||
...["-r", info.fpsText, "-c:v", "libx264", "-crf", String(opts.crf ?? 14), "-preset", "medium"],
|
||||
...(sound
|
||||
? [
|
||||
"-af",
|
||||
`atrim=start=${a}:end=${k1 / info.fps},asetpts=PTS-STARTPTS`,
|
||||
"-c:a",
|
||||
"aac",
|
||||
"-b:a",
|
||||
"192k",
|
||||
]
|
||||
: ["-an"]),
|
||||
...["-movflags", "+faststart", out],
|
||||
]);
|
||||
return {
|
||||
window: [round(a, 3), round(k1 / info.fps, 3)],
|
||||
frames: n,
|
||||
duration: round(n / info.fps, 3),
|
||||
roi,
|
||||
audio: sound,
|
||||
};
|
||||
}
|
||||
|
||||
/** The sound from `from` to `to` seconds as an audio file, on the same clock as the picture. */
|
||||
export function cutSound(info: VideoInfo, from: number, to: number, out: string) {
|
||||
if (!info.audio) throw new Fix(`${info.path} has no sound`, "leave <audio> out of the rebuild");
|
||||
const [a, b] = [Math.max(0, from), Math.min(info.duration, to)];
|
||||
if (b <= a) throw new Fix("--to is not after --from", "--from 0 --to 12");
|
||||
const codec = /\.wav$/i.test(out)
|
||||
? ["-c:a", "pcm_s16le"]
|
||||
: /\.mp3$/i.test(out)
|
||||
? ["-c:a", "libmp3lame", "-q:a", "2"]
|
||||
: ["-c:a", "aac", "-b:a", "256k"];
|
||||
runFfmpeg(["-y", "-ss", String(a), "-i", info.path, "-t", String(b - a), "-vn", ...codec, out]);
|
||||
return { window: [round(a, 3), round(b, 3)], duration: round(b - a, 3) };
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "extract",
|
||||
description:
|
||||
"Cut a still (--at) or a clip (--from/--to) out of a video, cropped to a region, as project media",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
out: {
|
||||
type: "string",
|
||||
description: "The file to write: .png/.jpg for a still, .mp4 for a clip",
|
||||
required: true,
|
||||
},
|
||||
at: { type: "string", description: "A still of the frame shown at this second" },
|
||||
from: { type: "string", description: "A clip's first second" },
|
||||
to: { type: "string", description: "A clip's end second" },
|
||||
audio: {
|
||||
type: "boolean",
|
||||
description: "Keep the clip's sound (default: picture only)",
|
||||
default: false,
|
||||
},
|
||||
crf: { type: "string", description: "Clip quality, lower is better (default 14)" },
|
||||
roi: roiArg,
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
// fallow-ignore-next-line complexity
|
||||
await runMeasure(async () => {
|
||||
const { info, roi: asked } = videoAndRoi(args.video, args.roi);
|
||||
const out = resolve(String(args.out));
|
||||
mkdirSync(dirname(out), { recursive: true });
|
||||
if (/\.(m4a|aac|mp3|wav)$/i.test(out)) {
|
||||
const sound = cutSound(
|
||||
info,
|
||||
num(args.from, "--from", 0),
|
||||
num(args.to, "--to", info.duration),
|
||||
out,
|
||||
);
|
||||
const ledger = await record(
|
||||
out,
|
||||
"bgm",
|
||||
{ provider: "original", from: info.path, window: sound.window },
|
||||
sound.duration,
|
||||
);
|
||||
return {
|
||||
out,
|
||||
kind: "sound",
|
||||
...sound,
|
||||
...(ledger ? { ledger } : {}),
|
||||
place: {
|
||||
"data-start": sound.window[0],
|
||||
"data-duration": sound.duration,
|
||||
note: "an <audio id> with this src",
|
||||
},
|
||||
};
|
||||
}
|
||||
const isClip = args.from !== undefined || args.to !== undefined;
|
||||
if (isClip === (args.at !== undefined))
|
||||
throw new Fix(
|
||||
"pass --at for a still, or --from and --to for a clip",
|
||||
"--at 3.2, or --from 4 --to 9.5",
|
||||
);
|
||||
if (!isClip) {
|
||||
if (!/\.(png|jpe?g|webp)$/i.test(out))
|
||||
throw new Fix(`${out} is not a picture`, "--out assets/name.png");
|
||||
const { at, frame: k } = await cutStill(info, num(args.at, "--at", 0), asked, out);
|
||||
const ledger = await record(out, "image", {
|
||||
provider: "original",
|
||||
from: info.path,
|
||||
at,
|
||||
roi: asked,
|
||||
});
|
||||
return {
|
||||
out,
|
||||
kind: "image",
|
||||
at,
|
||||
frame: k,
|
||||
roi: asked,
|
||||
...(ledger ? { ledger } : {}),
|
||||
place: { left: asked[0], top: asked[1], width: asked[2], height: asked[3] },
|
||||
};
|
||||
}
|
||||
|
||||
if (!/\.(mp4|mov|webm)$/i.test(out))
|
||||
throw new Fix(`${out} is not a video`, "--out assets/name.mp4");
|
||||
const clip = cutClip(
|
||||
info,
|
||||
num(args.from, "--from", 0),
|
||||
num(args.to, "--to", info.duration),
|
||||
asked,
|
||||
out,
|
||||
{
|
||||
audio: args.audio,
|
||||
crf: num(args.crf, "--crf", 14),
|
||||
},
|
||||
);
|
||||
const { window, duration, roi } = clip;
|
||||
const askedEnd = num(args.to, "--to", info.duration);
|
||||
const ledger = await record(
|
||||
out,
|
||||
"video",
|
||||
{ provider: "original", from: info.path, window, roi, audio: clip.audio },
|
||||
duration,
|
||||
);
|
||||
return {
|
||||
out,
|
||||
kind: "clip",
|
||||
window,
|
||||
frames: clip.frames,
|
||||
duration,
|
||||
roi,
|
||||
audio: clip.audio,
|
||||
...(askedEnd > info.duration + 0.5 / info.fps
|
||||
? {
|
||||
note: `--to ${askedEnd} is past the end; the clip stops at ${round(info.duration, 3)} s`,
|
||||
}
|
||||
: {}),
|
||||
...(ledger ? { ledger } : {}),
|
||||
place: {
|
||||
"data-start": window[0],
|
||||
"data-duration": duration,
|
||||
left: roi[0],
|
||||
top: roi[1],
|
||||
width: roi[2],
|
||||
height: roi[3],
|
||||
note: "a <video> clip at the original's time and box; muted unless it carries the film's sound",
|
||||
},
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,108 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, num, runMeasure, videoArg, window } from "../../measure/command.js";
|
||||
import {
|
||||
FIT_MIN_FACES,
|
||||
againstStart,
|
||||
faceBoxes,
|
||||
faceFit,
|
||||
faceSummary,
|
||||
} from "../../measure/face.js";
|
||||
import { Fix, probeVideo, round } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"How big and where the presenter's face is in a shot",
|
||||
"hyperframes measure face ref.mp4 --start 4 --end 9.5",
|
||||
],
|
||||
[
|
||||
"The scale and x, y that put a generated clip's face on the original's face over 2-6 s",
|
||||
"hyperframes measure face presenter.mp4 --against ref.mp4 --from 2 --to 6",
|
||||
],
|
||||
];
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "face",
|
||||
description:
|
||||
"How big the face is and where, over a window: its box per sampled frame and whether a generated presenter can keep the likeness at that size",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
start: {
|
||||
type: "string",
|
||||
description:
|
||||
"First second (default 0; with --against, 0 for a shot of its own and --from for a render as long as the original)",
|
||||
},
|
||||
end: {
|
||||
type: "string",
|
||||
description:
|
||||
"Last second (default: the end; with --against, --start plus the length of --from/--to)",
|
||||
},
|
||||
samples: {
|
||||
type: "string",
|
||||
description: "Frames sampled, evenly, the last one included (default 8)",
|
||||
},
|
||||
cast: {
|
||||
type: "boolean",
|
||||
description:
|
||||
"The person is one face in a crowd: grade the smallest face (and look for faces down to 20 px)",
|
||||
default: false,
|
||||
},
|
||||
against: {
|
||||
type: "string",
|
||||
description:
|
||||
"The original video: print the GSAP scale and x, y (canvas pixels) that put this clip's face on the original's over --from/--to",
|
||||
},
|
||||
from: { type: "string", description: "With --against: the original's first second" },
|
||||
to: { type: "string", description: "With --against: the original's last second" },
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
await runMeasure(async () => {
|
||||
const info = probeVideo(String(args.video));
|
||||
const cast = args.cast === true;
|
||||
const samples = Math.max(1, num(args.samples, "--samples", 8));
|
||||
const opts = cast ? { minFace: 0 } : {};
|
||||
if (args.against !== undefined) {
|
||||
const ref = probeVideo(String(args.against));
|
||||
const [a, b] = window(ref, args.from, args.to, ["--from", "--to"]);
|
||||
const s = num(args.start, "--start", againstStart(info.duration, ref.duration, a));
|
||||
const e = Math.min(info.duration, s + (b - a));
|
||||
if (!(e > s))
|
||||
throw new Fix(`--start ${s} is past this clip's end`, `--start 0 (or leave it out)`);
|
||||
const [fo, fc] = [
|
||||
await faceBoxes(ref, a, b, samples, opts),
|
||||
await faceBoxes(info, s, e, samples, opts),
|
||||
];
|
||||
const rel = (f: { t: number; boxes: number[][] }[], t0: number) =>
|
||||
f.map((x) => ({ t: x.t - t0, boxes: x.boxes }));
|
||||
const fit = faceFit(rel(fo, a), rel(fc, s), ref, info, cast);
|
||||
return {
|
||||
video: info.path,
|
||||
against: ref.path,
|
||||
window: [round(s, 3), round(e, 3)],
|
||||
against_window: [a, b],
|
||||
canvas: [ref.width, ref.height],
|
||||
...fit,
|
||||
...(fit.transform === null
|
||||
? {
|
||||
note: `fewer than ${FIT_MIN_FACES} moments with a face in both: try a wider window or more --samples`,
|
||||
}
|
||||
: {}),
|
||||
reading:
|
||||
"gsap.set(clip, { scale, x, y }) on a clip element that fills the canvas (object-fit: cover) with the default transform origin (its centre); x, y and the faces' centre and height are in the canvas's (the original's) pixels, clip_face before the transform; residual is the median face-centre miss left after it, in face heights",
|
||||
};
|
||||
}
|
||||
const [a, b] = window(info, args.start, args.end);
|
||||
return {
|
||||
video: info.path,
|
||||
start: a,
|
||||
end: b,
|
||||
...faceSummary(await faceBoxes(info, a, b, samples, opts), cast),
|
||||
reading:
|
||||
"face_height is the face's height over the frame's; boxes are x, y, w, h as shares of the frame",
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,338 +0,0 @@
|
||||
import { mkdtempSync, readdirSync, renameSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import {
|
||||
jsonArg,
|
||||
num,
|
||||
outArg,
|
||||
roiArg,
|
||||
runMeasure,
|
||||
videoAndRoi,
|
||||
videoArg,
|
||||
window,
|
||||
} from "../../measure/command.js";
|
||||
import { detectEvents, lookWindows } from "../../measure/events.js";
|
||||
import {
|
||||
Fix,
|
||||
frameAt,
|
||||
outDir,
|
||||
parseNumbers,
|
||||
parseRange,
|
||||
readClip,
|
||||
round,
|
||||
runFfmpeg,
|
||||
videoClip,
|
||||
type Frames,
|
||||
type Roi,
|
||||
type VideoInfo,
|
||||
} from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
["An overview sheet, one frame a second", "hyperframes measure frames ref.mp4 --every 1"],
|
||||
[
|
||||
"The consecutive frames around every event, one strip per window",
|
||||
"hyperframes measure frames ref.mp4 --events",
|
||||
],
|
||||
[
|
||||
"Every frame of a transition, at full size",
|
||||
"hyperframes measure frames ref.mp4 --range 2.1,2.6 --width 0",
|
||||
],
|
||||
[
|
||||
"Two moments, cropped to read the text",
|
||||
"hyperframes measure frames ref.mp4 --at 1.2,3.4 --roi 100,800,880,200",
|
||||
],
|
||||
];
|
||||
|
||||
const TILE_W = 360;
|
||||
const SHEET_TILES = 24;
|
||||
const STRIP_MAX = 12;
|
||||
const WINDOWS_MAX = 60;
|
||||
|
||||
/** Runs of consecutive indexes, for a short select expression. */
|
||||
function runsOf(ks: number[]): [number, number][] {
|
||||
const out: [number, number][] = [];
|
||||
for (const k of ks) {
|
||||
const last = out.at(-1);
|
||||
if (last && k === last[1] + 1) last[1] = k;
|
||||
else out.push([k, k]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The frames at these indexes (sorted, unique), in order, from one decode pass. */
|
||||
function framesAt(info: VideoInfo, ks: number[], roi: Roi, width: number): Frames {
|
||||
const expr = runsOf(ks)
|
||||
.map(([a, b]) => (a === b ? `eq(n\\,${a})` : `between(n\\,${a}\\,${b})`))
|
||||
.join("+");
|
||||
// Renumbered so the raw output is exactly the picked frames, no gap filled by a repeat.
|
||||
const clip = {
|
||||
...videoClip(info.path, 0, undefined, info),
|
||||
filters: [`select=${expr}`, `setpts=N/(${info.fpsText})/TB`],
|
||||
};
|
||||
const full = roi[0] === 0 && roi[1] === 0 && roi[2] === info.width && roi[3] === info.height;
|
||||
return readClip(clip, { ...(full ? {} : { roi }), width: width || roi[2] });
|
||||
}
|
||||
|
||||
/** Writes frames as numbered PNGs in one ffmpeg call; returns their paths in order. */
|
||||
function writeFrames(
|
||||
f: Frames,
|
||||
from: number,
|
||||
count: number,
|
||||
dir: string,
|
||||
prefix: string,
|
||||
): string[] {
|
||||
// Beside the output, so the move below stays on one filesystem (a container's /tmp often is not the project's).
|
||||
const work = mkdtempSync(join(dir, ".frames-"));
|
||||
try {
|
||||
const raw = Buffer.concat(
|
||||
f.frames
|
||||
.slice(from, from + count)
|
||||
.map((px) => Buffer.from(px.buffer, px.byteOffset, px.byteLength)),
|
||||
);
|
||||
runFfmpeg(
|
||||
[
|
||||
"-y",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-pix_fmt",
|
||||
"rgb24",
|
||||
"-s",
|
||||
`${f.w}x${f.h}`,
|
||||
"-i",
|
||||
"-",
|
||||
"-start_number",
|
||||
"0",
|
||||
join(work, "%04d.png"),
|
||||
],
|
||||
raw,
|
||||
);
|
||||
return readdirSync(work)
|
||||
.sort()
|
||||
.map((name, i) => {
|
||||
const to = join(dir, `${prefix}-${String(i).padStart(2, "0")}.png`);
|
||||
renameSync(join(work, name), to);
|
||||
return to;
|
||||
});
|
||||
} finally {
|
||||
rmSync(work, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function sheet(
|
||||
paths: string[],
|
||||
labels: string[],
|
||||
out: string,
|
||||
width: number,
|
||||
): Promise<string | null> {
|
||||
const { createContactSheet } = await import("../../capture/contactSheet.js");
|
||||
return createContactSheet(paths, out, {
|
||||
cols: Math.min(6, paths.length),
|
||||
maxImages: paths.length,
|
||||
cellWidth: width,
|
||||
labelMode: "custom",
|
||||
labels,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Up to `max` of a window's frames: all when they fit, else the frame before, at and after each event's onset
|
||||
* (uniform thinning turns a cut into a dissolve and drops one-frame flashes), the rest evenly spaced.
|
||||
*/
|
||||
function spread(
|
||||
k0: number,
|
||||
k1: number,
|
||||
max: number,
|
||||
onsets: number[] = [],
|
||||
): { ks: number[]; step: number } {
|
||||
const n = k1 - k0 + 1;
|
||||
if (n <= max) return { ks: Array.from({ length: n }, (_, i) => k0 + i), step: 1 };
|
||||
const must = new Set(onsets.flatMap((k) => [k - 1, k, k + 1]).filter((k) => k >= k0 && k <= k1));
|
||||
// At least half the strip stays evenly spaced, so what happens between the onsets is seen too.
|
||||
const room = Math.max(Math.ceil(max / 2), max - must.size);
|
||||
for (let i = 0; i < room; i++) must.add(k0 + Math.round((i * (n - 1)) / (room - 1)));
|
||||
const ks = [...must].sort((a, b) => a - b);
|
||||
return { ks, step: Math.ceil((n - 1) / Math.max(1, ks.length - 1)) };
|
||||
}
|
||||
|
||||
const label = (k: number, fps: number) => `f${k} · ${round(k / fps, 3)}s`;
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "frames",
|
||||
description:
|
||||
"Frames of a video as pictures: chosen moments, an overview sheet, consecutive native frames of a range, or strips around every event",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
at: { type: "string", description: "Seconds, e.g. 1.2,3.4: one picture each" },
|
||||
every: {
|
||||
type: "string",
|
||||
description: "An overview: one frame every N seconds on sheets of 24",
|
||||
},
|
||||
range: {
|
||||
type: "string",
|
||||
description: "start,end seconds: consecutive native frames as one strip",
|
||||
},
|
||||
events: {
|
||||
type: "boolean",
|
||||
description: "One strip of consecutive frames per window from measure events --map",
|
||||
default: false,
|
||||
},
|
||||
start: { type: "string", description: "With --events or --every: first second (default 0)" },
|
||||
end: {
|
||||
type: "string",
|
||||
description: "With --events or --every: last second (default: the end)",
|
||||
},
|
||||
max: {
|
||||
type: "string",
|
||||
description: `Frames per strip (default ${STRIP_MAX}; longer ranges are evenly thinned)`,
|
||||
},
|
||||
width: {
|
||||
type: "string",
|
||||
description: `Picture width in pixels (default ${TILE_W}; 0 keeps the ROI's size)`,
|
||||
},
|
||||
roi: roiArg,
|
||||
out: outArg,
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
// fallow-ignore-next-line complexity
|
||||
await runMeasure(async () => {
|
||||
const modes = [args.at, args.every, args.range, args.events || undefined].filter(
|
||||
(v) => v !== undefined,
|
||||
);
|
||||
if (modes.length !== 1)
|
||||
throw new Fix(
|
||||
"pass exactly one of --at, --every, --range or --events",
|
||||
"npx hyperframes measure frames ref.mp4 --every 1",
|
||||
);
|
||||
const { info, roi } = videoAndRoi(args.video, args.roi);
|
||||
const dir = outDir(args.out, "measure/frames");
|
||||
const width = num(args.width, "--width", TILE_W);
|
||||
const max = num(args.max, "--max", STRIP_MAX);
|
||||
const last = info.frames - 1;
|
||||
const head = {
|
||||
video: info.path,
|
||||
fps: round(info.fps, 3),
|
||||
...(args.roi !== undefined ? { roi } : {}),
|
||||
};
|
||||
|
||||
if (args.at !== undefined) {
|
||||
const ks = [
|
||||
...new Set(
|
||||
parseNumbers(args.at, "--at").map((t) => Math.min(last, frameAt(t, info.fps))),
|
||||
),
|
||||
].sort((a, b) => a - b);
|
||||
const f = framesAt(info, ks, roi, num(args.width, "--width", 0));
|
||||
const paths = writeFrames(f, 0, ks.length, dir, "at");
|
||||
return {
|
||||
...head,
|
||||
frames: ks.map((k, i) => ({ t: round(k / info.fps, 3), frame: k, path: paths[i] })),
|
||||
};
|
||||
}
|
||||
|
||||
if (args.every !== undefined) {
|
||||
const step = num(args.every, "--every", 1);
|
||||
const [a, b] = window(info, args.start, args.end);
|
||||
const ks = [
|
||||
...new Set(
|
||||
Array.from({ length: Math.floor((b - a) / step) + 1 }, (_, i) =>
|
||||
Math.min(last, frameAt(a + i * step, info.fps)),
|
||||
),
|
||||
),
|
||||
];
|
||||
const f = framesAt(info, ks, roi, width);
|
||||
const sheets: string[] = [];
|
||||
for (let s = 0; s < ks.length; s += SHEET_TILES) {
|
||||
const part = ks.slice(s, s + SHEET_TILES);
|
||||
const paths = writeFrames(f, s, part.length, dir, `every-${s / SHEET_TILES}`);
|
||||
const out = await sheet(
|
||||
paths,
|
||||
part.map((k) => `${round(k / info.fps, 2)}s`),
|
||||
join(dir, `every-${step}s-${s / SHEET_TILES + 1}.jpg`),
|
||||
f.w,
|
||||
);
|
||||
for (const p of paths) rmSync(p, { force: true });
|
||||
if (out) sheets.push(out);
|
||||
}
|
||||
return { ...head, every_s: step, frames: ks.length, sheets };
|
||||
}
|
||||
|
||||
if (args.range !== undefined) {
|
||||
const [a, b] = parseRange(args.range);
|
||||
const k0 = frameAt(a, info.fps);
|
||||
const k1 = Math.min(last, Math.max(k0, Math.ceil(b * info.fps - 1e-6) - 1));
|
||||
const { ks, step } = spread(k0, k1, max);
|
||||
const f = framesAt(info, ks, roi, width);
|
||||
const paths = writeFrames(f, 0, ks.length, dir, "range");
|
||||
const out = await sheet(
|
||||
paths,
|
||||
ks.map((k) => label(k, info.fps)),
|
||||
join(dir, `range-${a}-${b}s.jpg`),
|
||||
f.w,
|
||||
);
|
||||
for (const p of paths) rmSync(p, { force: true });
|
||||
return { ...head, start: a, end: b, frames: ks.length, every_nth_frame: step, strip: out };
|
||||
}
|
||||
|
||||
// --events: the windows the event map groups, each as a strip of its consecutive frames.
|
||||
const [a, b] = window(info, args.start, args.end);
|
||||
const found = detectEvents({
|
||||
info,
|
||||
start: a,
|
||||
end: b,
|
||||
roi: [0, 0, info.width, info.height],
|
||||
grid: { cols: 4, rows: 4 },
|
||||
});
|
||||
const windows = lookWindows(found.events, info.fps).slice(0, WINDOWS_MAX);
|
||||
const picks = windows.map((wnd) =>
|
||||
spread(
|
||||
frameAt(wnd.start, info.fps),
|
||||
Math.min(last, frameAt(wnd.end, info.fps)),
|
||||
max,
|
||||
wnd.onset_frames,
|
||||
),
|
||||
);
|
||||
const all = [...new Set(picks.flatMap((p) => p.ks))].sort((x, y) => x - y);
|
||||
if (!all.length)
|
||||
return {
|
||||
...head,
|
||||
windows: [],
|
||||
note: "nothing changes on screen: one overview frame is enough (--every)",
|
||||
};
|
||||
const f = framesAt(info, all, roi, width);
|
||||
const strips = [];
|
||||
for (const [i, wnd] of windows.entries()) {
|
||||
const { ks, step } = picks[i]!;
|
||||
const from = all.indexOf(ks[0]!);
|
||||
const paths = writeFrames(f, from, ks.length, dir, `w${i + 1}`);
|
||||
const out = await sheet(
|
||||
paths,
|
||||
ks.map((k) => label(k, info.fps)),
|
||||
join(dir, `window-${String(i + 1).padStart(2, "0")}-${wnd.start}s.jpg`),
|
||||
f.w,
|
||||
);
|
||||
for (const p of paths) rmSync(p, { force: true });
|
||||
strips.push({
|
||||
start: wnd.start,
|
||||
end: wnd.end,
|
||||
what: wnd.what,
|
||||
frames: ks.length,
|
||||
...(step > 1 ? { thinned: `onsets kept, about every ${step}th frame between` } : {}),
|
||||
strip: out,
|
||||
});
|
||||
}
|
||||
const total = lookWindows(found.events, info.fps).length;
|
||||
return {
|
||||
...head,
|
||||
windows: strips,
|
||||
...(total > WINDOWS_MAX
|
||||
? {
|
||||
note: `${total} windows; the first ${WINDOWS_MAX} shown: use --start/--end for the rest`,
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,114 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import {
|
||||
jsonArg,
|
||||
num,
|
||||
roiArg,
|
||||
runMeasure,
|
||||
videoAndRoi,
|
||||
videoArg,
|
||||
window,
|
||||
} from "../../measure/command.js";
|
||||
import { compareLook, look, sameLook } from "../../measure/look.js";
|
||||
import { Fix, parseRoi, probeVideo, readFrameAt, videoClip } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
["The grade of the first 5 seconds", "hyperframes measure look ref.mp4 --start 0 --end 5"],
|
||||
[
|
||||
"Does the rebuild's grade match the original's",
|
||||
"hyperframes measure look ref.mp4 --compare renders/v1.mp4",
|
||||
],
|
||||
];
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "look",
|
||||
description:
|
||||
"Brightness, contrast, saturation, black-and-white share and warmth of a window, or compared with another video",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
start: { type: "string", description: "First second (default 0)" },
|
||||
end: { type: "string", description: "Last second (default: the end)" },
|
||||
fps: { type: "string", description: "Frames sampled per second (default 2)" },
|
||||
roi: roiArg,
|
||||
at: { type: "string", description: "With --grid: the moment to read" },
|
||||
grid: {
|
||||
type: "string",
|
||||
description: "With --at: the mean colour of each cell, COLSxROWS, e.g. 8x6",
|
||||
},
|
||||
compare: { type: "string", description: "Another video measured over the same window" },
|
||||
"compare-start": {
|
||||
type: "string",
|
||||
description: "Second the window starts in it (default: --start)",
|
||||
},
|
||||
"compare-roi": { type: "string", description: "Region of it (default: the whole frame)" },
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
await runMeasure(() => {
|
||||
const { info, roi } = videoAndRoi(args.video, args.roi);
|
||||
if (args.grid !== undefined || args.at !== undefined) {
|
||||
const m = /^(\d+)x(\d+)$/.exec(String(args.grid ?? ""));
|
||||
if (!m || args.at === undefined)
|
||||
throw new Fix("--grid and --at go together", "--at 2 --grid 8x6");
|
||||
const [cols, rows] = [Number(m[1]), Number(m[2])];
|
||||
const at = num(args.at, "--at", 0);
|
||||
const { frames, w, h } = readFrameAt(info, at, { roi, width: Math.min(640, roi[2]) });
|
||||
const px = frames[0];
|
||||
if (!px) throw new Fix(`no frame at ${at}s`, "pick a time inside the video");
|
||||
const hex = (v: number[]) =>
|
||||
`#${v.map((c) => Math.round(c).toString(16).padStart(2, "0")).join("")}`;
|
||||
const colors = Array.from({ length: rows }, (_, r) =>
|
||||
Array.from({ length: cols }, (_, c) => {
|
||||
const [x0, x1, y0, y1] = [
|
||||
Math.floor((c * w) / cols),
|
||||
Math.floor(((c + 1) * w) / cols),
|
||||
Math.floor((r * h) / rows),
|
||||
Math.floor(((r + 1) * h) / rows),
|
||||
];
|
||||
const sum = [0, 0, 0];
|
||||
for (let y = y0; y < y1; y++)
|
||||
for (let x = x0; x < x1; x++)
|
||||
for (let k = 0; k < 3; k++) sum[k]! += px[(y * w + x) * 3 + k]!;
|
||||
const n = Math.max(1, (x1 - x0) * (y1 - y0));
|
||||
return hex(sum.map((v) => v / n));
|
||||
}),
|
||||
);
|
||||
return {
|
||||
video: info.path,
|
||||
at,
|
||||
roi,
|
||||
grid: `${cols}x${rows}`,
|
||||
colors,
|
||||
reading: "rows top to bottom, each cell's mean colour",
|
||||
};
|
||||
}
|
||||
const [a, b] = window(info, args.start, args.end);
|
||||
const fps = num(args.fps, "--fps", 2);
|
||||
const ref = look(videoClip(info.path, a, b - a, info), fps, roi);
|
||||
if (args.compare === undefined) return { video: info.path, start: a, end: b, ...ref };
|
||||
const other = probeVideo(String(args.compare));
|
||||
const otherRoi =
|
||||
args["compare-roi"] === undefined
|
||||
? undefined
|
||||
: parseRoi(args["compare-roi"], other.width, other.height);
|
||||
const theirs = look(
|
||||
videoClip(other.path, num(args["compare-start"], "--compare-start", a), b - a, other),
|
||||
fps,
|
||||
otherRoi,
|
||||
);
|
||||
const diff = compareLook(ref, theirs);
|
||||
return {
|
||||
video: info.path,
|
||||
compare: other.path,
|
||||
start: a,
|
||||
end: b,
|
||||
reference: ref,
|
||||
other: theirs,
|
||||
...diff,
|
||||
same_look: sameLook(diff),
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,129 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, runMeasure, videoArg } from "../../measure/command.js";
|
||||
import {
|
||||
ffmpegBin,
|
||||
Fix,
|
||||
probeVideo,
|
||||
round,
|
||||
runFfmpeg,
|
||||
type VideoInfo,
|
||||
} from "../../measure/media.js";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdirSync } from "node:fs";
|
||||
import { dirname, resolve } from "node:path";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"Size, frame rate, sound, and what to fix before rebuilding",
|
||||
"hyperframes measure probe ref.mp4",
|
||||
],
|
||||
[
|
||||
"Make a constant-rate SDR working copy of a phone or HDR video",
|
||||
"hyperframes measure probe ref.mp4 --normalize original.mp4",
|
||||
],
|
||||
];
|
||||
|
||||
// Sections longer than this are analysed and checked one at a time.
|
||||
const SECTION_S = 60;
|
||||
const STANDARD = ["24000/1001", "24", "25", "30000/1001", "30", "50", "60000/1001", "60"];
|
||||
const rateOf = (text: string) => {
|
||||
const [a, b] = text.split("/").map(Number);
|
||||
return (a ?? 0) / (b || 1);
|
||||
};
|
||||
|
||||
/** The standard frame rate closest to a variable-rate video's average. */
|
||||
const standardRate = (info: VideoInfo) =>
|
||||
STANDARD.reduce((best, r) =>
|
||||
Math.abs(rateOf(r) - info.avgFps) < Math.abs(rateOf(best) - info.avgFps) ? r : best,
|
||||
);
|
||||
|
||||
const hasFilter = (name: string) =>
|
||||
new RegExp(`\\s${name}\\s`).test(
|
||||
execFileSync(ffmpegBin(), ["-hide_banner", "-filters"], { windowsHide: true }).toString(),
|
||||
);
|
||||
|
||||
/** A constant-rate SDR H.264 copy: what the measuring tools, diff and the render all read the same way. */
|
||||
function normalize(info: VideoInfo, out: string): { path: string; fps: string } {
|
||||
const fps = info.vfr ? standardRate(info) : info.fpsText;
|
||||
const filters = [`fps=${fps}`];
|
||||
if (info.hdr) {
|
||||
if (!hasFilter("zscale"))
|
||||
throw new Fix(
|
||||
"this ffmpeg cannot tone-map HDR (no zscale filter)",
|
||||
"install an ffmpeg built with libzimg, e.g. brew install ffmpeg",
|
||||
);
|
||||
filters.push(
|
||||
"zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv",
|
||||
);
|
||||
}
|
||||
mkdirSync(dirname(out), { recursive: true });
|
||||
runFfmpeg([
|
||||
...["-y", "-i", info.path, "-vf", [...filters, "format=yuv420p"].join(",")],
|
||||
...["-c:v", "libx264", "-crf", "12", "-preset", "medium", "-c:a", "aac", "-b:a", "192k", out],
|
||||
]);
|
||||
return { path: out, fps };
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "probe",
|
||||
description:
|
||||
"Size (as displayed), frame rate, duration, sound, and what to fix before measuring or rebuilding",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
normalize: {
|
||||
type: "string",
|
||||
description:
|
||||
"Write a constant-frame-rate SDR copy here (for variable-rate or HDR videos); the original is left alone",
|
||||
},
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
// fallow-ignore-next-line complexity
|
||||
await runMeasure(() => {
|
||||
const info = probeVideo(String(args.video));
|
||||
const notes: string[] = [];
|
||||
if (info.vfr)
|
||||
notes.push(
|
||||
"variable frame rate: frame times drift, so measure and rebuild a constant-rate copy (--normalize)",
|
||||
);
|
||||
if (info.hdr) notes.push("HDR: --normalize tone-maps an SDR copy; the render is SDR");
|
||||
if (info.duration > SECTION_S)
|
||||
notes.push(
|
||||
`longer than ${SECTION_S} s: analyse and check it in sections of at most ${SECTION_S} s, split at cuts (measure cuts)`,
|
||||
);
|
||||
if (!info.audio) notes.push("no sound: the rebuild has no <audio>");
|
||||
const made = args.normalize ? normalize(info, resolve(String(args.normalize))) : undefined;
|
||||
return {
|
||||
video: info.path,
|
||||
width: info.width,
|
||||
height: info.height,
|
||||
fps: round(info.fps, 3),
|
||||
fps_text: info.fpsText,
|
||||
duration: round(info.duration, 3),
|
||||
frames: info.frames,
|
||||
picture_duration: round(info.frames / info.fps, 3),
|
||||
audio: info.audio,
|
||||
vfr: info.vfr,
|
||||
hdr: info.hdr,
|
||||
codec: info.codec,
|
||||
canvas: {
|
||||
width: info.width,
|
||||
height: info.height,
|
||||
fps: made?.fps ?? info.fpsText,
|
||||
duration: round(info.frames / info.fps, 3),
|
||||
note: "data-width, data-height and data-duration (the picture's length) on the root composition; render with --fps at this rate",
|
||||
},
|
||||
...(made ? { normalized: made.path } : {}),
|
||||
notes,
|
||||
...((info.vfr || info.hdr) && !made
|
||||
? {
|
||||
fix: `npx hyperframes measure probe ${String(args.video)} --normalize <working-copy.mp4>`,
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -1,53 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import {
|
||||
jsonArg,
|
||||
num,
|
||||
roiArg,
|
||||
runMeasure,
|
||||
videoAndRoi,
|
||||
videoArg,
|
||||
window,
|
||||
} from "../../measure/command.js";
|
||||
import { Fix } from "../../measure/media.js";
|
||||
import { avSync } from "../../measure/sync.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"How far the presenter's mouth trails the voice",
|
||||
"hyperframes measure sync renders/v1.mp4 --roi 470,620,140,90 --start 2 --end 8",
|
||||
],
|
||||
];
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "sync",
|
||||
description:
|
||||
"Picture against sound for a person speaking: the lag between mouth motion and the voice",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
roi: { ...roiArg, description: "The mouth, x,y,w,h in the video's pixels", required: true },
|
||||
start: { type: "string", description: "First second (default 0)" },
|
||||
end: { type: "string", description: "Last second (default: the end)" },
|
||||
"max-lag-ms": { type: "string", description: "Lags searched each way (default 400)" },
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
await runMeasure(() => {
|
||||
if (args.roi === undefined)
|
||||
throw new Fix("--roi is required", "--roi x,y,w,h round the mouth");
|
||||
const { info, roi } = videoAndRoi(args.video, args.roi);
|
||||
const [start, end] = window(info, args.start, args.end);
|
||||
return {
|
||||
video: info.path,
|
||||
roi,
|
||||
start,
|
||||
end,
|
||||
...avSync(info, roi, start, end, num(args["max-lag-ms"], "--max-lag-ms", 400)),
|
||||
reading:
|
||||
"positive best_lag_ms means the picture is late; trust high confidence, repeat a medium one on another window without cuts",
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -94,17 +94,13 @@ function subcommand(name: MediaUseVerb, invoke: InvokeMediaUse) {
|
||||
}
|
||||
|
||||
export function createMediaUseCommand(invoke: InvokeMediaUse = invokeEngine) {
|
||||
const subCommands = {
|
||||
...Object.fromEntries(MEDIA_USE_VERBS.map((name) => [name, () => subcommand(name, invoke)])),
|
||||
generate: () => import("./media-use/generate.js").then((m) => m.default),
|
||||
};
|
||||
const subCommands = Object.fromEntries(
|
||||
MEDIA_USE_VERBS.map((name) => [name, () => subcommand(name, invoke)]),
|
||||
);
|
||||
return defineCommand({
|
||||
meta: { name: "media-use", description: "Resolve and operate on project media" },
|
||||
subCommands,
|
||||
run: ({ args }) => {
|
||||
if (!args._?.[0])
|
||||
console.log("Run `hyperframes media-use <resolve|generate|doctor|stats|...> --help`");
|
||||
},
|
||||
run: () => console.log("Run `hyperframes media-use <resolve|doctor|stats|...> --help`"),
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -1,162 +0,0 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, num, runMeasure } from "../../measure/command.js";
|
||||
import { Fix } from "../../measure/media.js";
|
||||
import {
|
||||
GENERATE_TYPES,
|
||||
checkRequest,
|
||||
generate,
|
||||
parseMask,
|
||||
type GenerateRequest,
|
||||
type GenerateType,
|
||||
} from "../../heygen/generate.js";
|
||||
import { checkLimit, creditsOf, priceOf } from "../../heygen/pricing.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"Price a clean plate of a frame, then make it",
|
||||
'hyperframes media-use generate --type image --reference f120.png --prompt "the same shot with no text" --out assets/plate.png --plan',
|
||||
],
|
||||
[
|
||||
"A 6 s shot that starts from the original's first frame",
|
||||
'hyperframes media-use generate --type video --first-frame assets/s03-first.png --prompt "slow push in" --seconds 6 --out assets/s03.mp4',
|
||||
],
|
||||
[
|
||||
"The original's presenter re-shot clean, saying an 8 s stretch of the original voice",
|
||||
'hyperframes media-use generate --type video --reference assets/presenter-clean.png --audio assets/s02-voice.m4a --motion assets/s02-original.mp4 --mask 110,615,390,165 --prompt "the woman talks to camera, seated, soft daylight" --seconds 8 --aspect 9:16 --out assets/s02-presenter.mp4',
|
||||
],
|
||||
[
|
||||
"The person in a photo saying a script in a chosen voice",
|
||||
'hyperframes media-use generate --type avatar --photo assets/presenter.png --script "Hi, I am Ana" --voice <voice-id> --out assets/ana.mp4',
|
||||
],
|
||||
];
|
||||
|
||||
const isType = (v: string): v is GenerateType => GENERATE_TYPES.some((t) => t === v);
|
||||
|
||||
/** Every value of a flag given more than once (citty keeps only the last). */
|
||||
export function repeated(raw: string[], flag: string): string[] {
|
||||
return raw.flatMap((a, i) =>
|
||||
a === flag && raw[i + 1] !== undefined
|
||||
? [raw[i + 1]!]
|
||||
: a.startsWith(`${flag}=`)
|
||||
? [a.slice(flag.length + 1)]
|
||||
: [],
|
||||
);
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "generate",
|
||||
description:
|
||||
"Make new media on HeyGen: an image (edited from references), a video shot (from a first frame), a person speaking from a photo, a voice, or music. --plan prices it first",
|
||||
},
|
||||
args: {
|
||||
type: {
|
||||
type: "string",
|
||||
description: `What to make: ${GENERATE_TYPES.join(", ")}`,
|
||||
required: true,
|
||||
},
|
||||
out: {
|
||||
type: "string",
|
||||
description: "The project file to write, e.g. assets/plate.png",
|
||||
required: true,
|
||||
},
|
||||
prompt: { type: "string", description: "What it shows or sounds like (image, video, music)" },
|
||||
reference: {
|
||||
type: "string",
|
||||
description:
|
||||
"image: comma-separated pictures to edit from or follow; video: the person to show speaking",
|
||||
},
|
||||
audio: {
|
||||
type: "string",
|
||||
description: "video: speech the referenced person says, lips in sync (5–15 s)",
|
||||
},
|
||||
motion: {
|
||||
type: "string",
|
||||
description: "video: a clip whose camera and body motion the shot follows (up to 15 s)",
|
||||
},
|
||||
mask: {
|
||||
type: "string",
|
||||
description:
|
||||
"video: x,y,w,h[,a-b] of the --motion clip to blank before sending (its drawn text; a-b in the clip's seconds); repeat for more",
|
||||
},
|
||||
"first-frame": { type: "string", description: "video: the picture the shot starts from" },
|
||||
photo: { type: "string", description: "avatar: the person's photo" },
|
||||
script: { type: "string", description: "avatar, voice: the words spoken" },
|
||||
voice: { type: "string", description: "avatar, voice: a HeyGen voice id" },
|
||||
seconds: {
|
||||
type: "string",
|
||||
description: "video 5–15 (default 5); music: 30, 60, 120 or 180 (default 30)",
|
||||
},
|
||||
aspect: {
|
||||
type: "string",
|
||||
description: "Aspect ratio, e.g. 9:16 (image, video without a first frame, avatar)",
|
||||
},
|
||||
model: { type: "string", description: "image: nano-banana-2 (default) or gpt-image-2.5" },
|
||||
bpm: { type: "string", description: "music: tempo, 60–180" },
|
||||
vocals: {
|
||||
type: "boolean",
|
||||
description: "music: allow singing (default instrumental)",
|
||||
default: false,
|
||||
},
|
||||
"max-credits": {
|
||||
type: "string",
|
||||
description:
|
||||
"The person's credit limit, less what earlier calls used: make nothing when this call would cost more",
|
||||
},
|
||||
plan: {
|
||||
type: "boolean",
|
||||
description: "Print the price and what would be sent; make nothing",
|
||||
default: false,
|
||||
},
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args, rawArgs }) {
|
||||
await runMeasure(async () => {
|
||||
const type = String(args.type);
|
||||
if (!isType(type))
|
||||
throw new Fix(`unknown --type ${type}`, `--type ${GENERATE_TYPES.join("|")}`);
|
||||
const request: GenerateRequest = {
|
||||
type,
|
||||
out: String(args.out),
|
||||
prompt: args.prompt,
|
||||
references: args.reference
|
||||
?.split(",")
|
||||
.map((p) => p.trim())
|
||||
.filter(Boolean),
|
||||
firstFrame: args["first-frame"],
|
||||
audio: args.audio,
|
||||
motion: args.motion,
|
||||
masks: repeated(rawArgs, "--mask").map(parseMask),
|
||||
photo: args.photo,
|
||||
script: args.script,
|
||||
voice: args.voice,
|
||||
seconds: num(args.seconds, "--seconds"),
|
||||
aspect: args.aspect,
|
||||
model: args.model,
|
||||
bpm: num(args.bpm, "--bpm"),
|
||||
instrumental: !args.vocals,
|
||||
};
|
||||
const price = priceOf(request);
|
||||
const credits = creditsOf(price);
|
||||
if (args.plan) {
|
||||
checkRequest(request);
|
||||
return {
|
||||
plan: true,
|
||||
type,
|
||||
out: request.out,
|
||||
price,
|
||||
credits,
|
||||
next: "run the same command without --plan to make it",
|
||||
};
|
||||
}
|
||||
checkLimit(credits, num(args["max-credits"], "--max-credits"));
|
||||
const made = await generate(
|
||||
request,
|
||||
(status) => console.error(`generate: ${status}…`),
|
||||
credits ?? undefined,
|
||||
);
|
||||
return { ...made, price, credits };
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -881,7 +881,7 @@ export function serveCapturePage(projectDir: string, html: string, proxy: boolea
|
||||
* Render key frames from a composition as PNG screenshots.
|
||||
* The agent can Read these to verify its output visually.
|
||||
*/
|
||||
export async function captureSnapshots(
|
||||
async function captureSnapshots(
|
||||
projectDir: string,
|
||||
opts: {
|
||||
frames?: number;
|
||||
@@ -904,17 +904,13 @@ export async function captureSnapshots(
|
||||
review?: boolean;
|
||||
/** Progress lines; `--json` sends them to stderr so stdout stays one envelope. */
|
||||
log?: (message: string) => void;
|
||||
record?: boolean;
|
||||
},
|
||||
): Promise<CapturedSnapshots> {
|
||||
const startedAt = performance.now();
|
||||
const log = opts.log ?? ((message: string) => console.log(message));
|
||||
const { bundleWithLocalizedFonts } = await import("../utils/bundleWithLocalizedFonts.js");
|
||||
|
||||
const capture =
|
||||
opts.record === false
|
||||
? undefined
|
||||
: await captureArtifactRevision(projectDir).catch(warnSnapshotsNotRecorded);
|
||||
const capture = await captureArtifactRevision(projectDir).catch(warnSnapshotsNotRecorded);
|
||||
const invocation = randomUUID();
|
||||
const artifacts: Array<{
|
||||
id: string;
|
||||
@@ -1447,7 +1443,7 @@ export async function captureSnapshots(
|
||||
log,
|
||||
)
|
||||
: false;
|
||||
if (!recorded && opts.record !== false) {
|
||||
if (!recorded) {
|
||||
const paths = [...replaced, ...artifacts.map((artifact) => artifact.image)];
|
||||
if (paths.length > 0) await forgetRecordedArtifacts(projectDir, paths).catch(() => undefined);
|
||||
}
|
||||
|
||||
@@ -19,10 +19,6 @@ import { jsonFailure } from "../utils/jsonError.js";
|
||||
export const examples: Example[] = [
|
||||
["Transcribe an audio file", "hyperframes transcribe audio.mp3"],
|
||||
["Transcribe a video file", "hyperframes transcribe video.mp4"],
|
||||
[
|
||||
"Transcribe on HeyGen, with speakers",
|
||||
"hyperframes transcribe interview.mp4 --engine heygen --speakers",
|
||||
],
|
||||
["Use a larger model for better accuracy", "hyperframes transcribe audio.mp3 --model medium.en"],
|
||||
["Set language to filter non-target speech", "hyperframes transcribe audio.mp3 --language en"],
|
||||
["Import an existing SRT file", "hyperframes transcribe subtitles.srt"],
|
||||
@@ -56,14 +52,9 @@ export default defineCommand({
|
||||
engine: {
|
||||
type: "string",
|
||||
description:
|
||||
"ASR engine: auto (Parakeet if installed and it covers --language, else whisper), parakeet, whisper, or heygen (HeyGen's speech model; needs an API key, charged to API credits). Default: auto. Parakeet is more accurate and faster than whisper; install it with `hyperframes models install parakeet`.",
|
||||
"ASR engine: auto (Parakeet if installed and it covers --language, else whisper), parakeet, or whisper. Default: auto. Parakeet is more accurate and faster; install it with `hyperframes models install parakeet`.",
|
||||
alias: "e",
|
||||
},
|
||||
speakers: {
|
||||
type: "boolean",
|
||||
description: "With --engine heygen: label each word with its speaker",
|
||||
default: false,
|
||||
},
|
||||
model: {
|
||||
type: "string",
|
||||
description: `Whisper model (default: ${DEFAULT_MODEL}). Options: tiny.en, base.en, small.en, medium.en, small, medium, large-v3`,
|
||||
@@ -128,7 +119,6 @@ export default defineCommand({
|
||||
input: args.input,
|
||||
dir: args.dir,
|
||||
engine: args.engine,
|
||||
speakers: args.speakers,
|
||||
model: args.model,
|
||||
language: args.language,
|
||||
installRuntime: args["runtime-install"],
|
||||
|
||||
@@ -59,12 +59,6 @@ const GROUPS: Group[] = [
|
||||
"Render candidate color grades onto a reference frame as one labeled comparison PNG",
|
||||
],
|
||||
["compare", "Render composition variants into one labeled comparison sheet"],
|
||||
["diff", "Score a render or a project against the original video, frame by frame"],
|
||||
[
|
||||
"measure",
|
||||
"Measure a video file: clock, cuts, events, frames, motion, crops, boxes, eases, look",
|
||||
],
|
||||
["billing", "Open the HyperFrames plans or credits page and wait until it is done"],
|
||||
["info", "Print project metadata"],
|
||||
["compositions", "List all compositions in a project"],
|
||||
["timeline", "Print the project's tracks and clips"],
|
||||
|
||||
@@ -1,399 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { findFFmpeg } from "../browser/ffmpeg.js";
|
||||
|
||||
vi.mock("../cloud/auth.js", () => ({
|
||||
resolveCloudAuthHeaders: async () => ({ "x-api-key": "k" }),
|
||||
refreshIfNeeded: async <T>(credential: T) => credential,
|
||||
resolveCloudBaseUrl: () => "https://api.test",
|
||||
}));
|
||||
|
||||
vi.stubEnv("HEYGEN_API_KEY", "k");
|
||||
|
||||
const { checkRequest, generate, parseMask, urlOf } = await import("./generate.js");
|
||||
const { checkLimit, creditsOf, priceOf } = await import("./pricing.js");
|
||||
const ffmpeg = findFFmpeg();
|
||||
const project = mkdtempSync(join(tmpdir(), "hf-generate-test-"));
|
||||
writeFileSync(join(project, "index.html"), "<html></html>");
|
||||
afterAll(() => rmSync(project, { recursive: true, force: true }));
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
const json = (body: unknown) => new Response(JSON.stringify(body), { status: 200 });
|
||||
|
||||
/** A 30 fps clip of these lavfi sources one after another: each new source is a cut. */
|
||||
function shots(name: string, sources: string[], size = "180x320"): string {
|
||||
const path = join(project, name);
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y"],
|
||||
...sources.flatMap((src) => ["-f", "lavfi", "-i", `${src}:s=${size}:r=30`]),
|
||||
...["-filter_complex", `concat=n=${sources.length}:v=1`, "-pix_fmt", "yuv420p", path],
|
||||
]);
|
||||
return path;
|
||||
}
|
||||
|
||||
describe("generate", () => {
|
||||
it("finds the file URL wherever the route keeps it", () => {
|
||||
expect(
|
||||
urlOf({ id: "a", status: "completed", images: [{ url: "https://x/i.png" }] }, [
|
||||
"images.0.url",
|
||||
]),
|
||||
).toBe("https://x/i.png");
|
||||
expect(
|
||||
urlOf({ id: "a", status: "completed", video_url: "ftp://no" }, ["video_url"]),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("edits a picture from a reference, downloads it and records it", async () => {
|
||||
const ref = join(project, "ref.png");
|
||||
execFileSync(ffmpeg!, [
|
||||
"-v",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=red:s=64x36",
|
||||
"-frames:v",
|
||||
"1",
|
||||
ref,
|
||||
]);
|
||||
const sent: { url: string; body?: Record<string, unknown> }[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
sent.push({ url, ...(init?.body ? { body: JSON.parse(String(init.body)) } : {}) });
|
||||
if (url === "https://files.test/plate.png") return new Response(new Uint8Array([1, 2, 3]));
|
||||
return json({
|
||||
data: {
|
||||
id: "img1",
|
||||
status: "completed",
|
||||
images: [{ url: "https://files.test/plate.png" }],
|
||||
},
|
||||
});
|
||||
});
|
||||
const r = await generate({
|
||||
type: "image",
|
||||
out: join(project, "assets/plate.png"),
|
||||
prompt: "empty room",
|
||||
references: [ref],
|
||||
});
|
||||
expect(sent[0]!.body).toMatchObject({
|
||||
model: "nano-banana-2",
|
||||
prompt: "empty room",
|
||||
reference_images: [{ type: "base64", media_type: "image/jpeg" }],
|
||||
});
|
||||
expect(readFileSync(r.out)).toEqual(Buffer.from([1, 2, 3]));
|
||||
const ledger = readFileSync(join(project, ".media/manifest.jsonl"), "utf8");
|
||||
expect(ledger).toContain('"source":"generated"');
|
||||
expect(ledger).toContain('"references":["ref.png"]');
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("re-shoots a person speaking a stretch of the original voice", async () => {
|
||||
const [face, voice] = [join(project, "face.png"), join(project, "voice.m4a")];
|
||||
execFileSync(ffmpeg!, [
|
||||
"-v",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=gray:s=36x64",
|
||||
"-frames:v",
|
||||
"1",
|
||||
face,
|
||||
]);
|
||||
execFileSync(ffmpeg!, ["-v", "error", "-y", "-f", "lavfi", "-i", "sine=d=1", voice]);
|
||||
const guide = join(project, "guide.mp4");
|
||||
execFileSync(ffmpeg!, [
|
||||
"-v",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc2=d=1:s=180x320",
|
||||
guide,
|
||||
]);
|
||||
const sent: Record<string, unknown>[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
if (init?.body) sent.push(JSON.parse(String(init.body)));
|
||||
if (url === "https://files.test/s.mp4") return new Response(new Uint8Array([7]));
|
||||
return json({
|
||||
data: { id: "v2", status: "completed", video_url: "https://files.test/s.mp4" },
|
||||
});
|
||||
});
|
||||
await generate({
|
||||
type: "video",
|
||||
out: join(project, "assets/presenter.mp4"),
|
||||
prompt: "talks to camera",
|
||||
references: [face],
|
||||
audio: voice,
|
||||
motion: guide,
|
||||
seconds: 8,
|
||||
aspect: "9:16",
|
||||
});
|
||||
expect(sent[0]).toMatchObject({
|
||||
mode: "reference_to_video",
|
||||
duration: 8,
|
||||
aspect_ratio: "9:16",
|
||||
reference_images: [{ media_type: "image/jpeg" }],
|
||||
reference_audio: [{ media_type: "audio/mpeg" }],
|
||||
reference_videos: [{ media_type: "video/mp4" }],
|
||||
});
|
||||
expect(readFileSync(join(project, ".media/manifest.jsonl"), "utf8")).toContain(
|
||||
'"audio":"voice.m4a"',
|
||||
);
|
||||
await expect(
|
||||
generate({ type: "video", out: "x.mp4", prompt: "p", audio: voice, firstFrame: face }),
|
||||
).rejects.toThrow(/--first-frame cannot be used/);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("blanks the motion clip's masked regions before sending it", async () => {
|
||||
// A guide with white letters at 40,100 over a moving picture; the mask covers them for its first 0.5 s.
|
||||
const guide = join(project, "titled.mp4");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "testsrc2=d=1:s=180x320", "-vf"],
|
||||
"drawbox=x=40:y=100:w=100:h=10:color=white:t=fill,drawbox=x=40:y=120:w=100:h=10:color=white:t=fill",
|
||||
guide,
|
||||
]);
|
||||
const face = join(project, "face2.png");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "color=c=gray:s=36x64", "-frames:v", "1"],
|
||||
face,
|
||||
]);
|
||||
const sent: Record<string, unknown>[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
if (init?.body) sent.push(JSON.parse(String(init.body)));
|
||||
if (url === "https://files.test/m.mp4") return new Response(new Uint8Array([7]));
|
||||
return json({
|
||||
data: { id: "v3", status: "completed", video_url: "https://files.test/m.mp4" },
|
||||
});
|
||||
});
|
||||
await generate({
|
||||
type: "video",
|
||||
out: join(project, "assets/masked.mp4"),
|
||||
prompt: "walks",
|
||||
references: [face],
|
||||
motion: guide,
|
||||
masks: [parseMask("30,90,120,50,0-0.5")],
|
||||
});
|
||||
const [video] = sent[0]!.reference_videos as { data: string }[];
|
||||
const sentClip = join(project, "sent.mp4");
|
||||
writeFileSync(sentClip, Buffer.from(video!.data, "base64"));
|
||||
// The letters are two white rows of bars: the largest step between neighbouring rows' brightness is their edge.
|
||||
// Filled from around, no such step is left; after 0.5 s they are back.
|
||||
const edge = (t: number) => {
|
||||
const px = execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-ss", String(t), "-i", sentClip, "-frames:v", "1"],
|
||||
...["-vf", "crop=100:40:40:95,format=gray", "-f", "rawvideo", "-"],
|
||||
]);
|
||||
const rows = Array.from({ length: 40 }, (_, y) => px.subarray(y * 100, y * 100 + 100));
|
||||
const mean = rows.map((r) => r.reduce((a, v) => a + v, 0) / r.length);
|
||||
return Math.max(...mean.slice(1).map((m, y) => Math.abs(m - mean[y]!)));
|
||||
};
|
||||
expect(edge(0.2)).toBeLessThan(30);
|
||||
expect(edge(0.8)).toBeGreaterThan(60);
|
||||
expect(readFileSync(join(project, ".media/manifest.jsonl"), "utf8")).toContain(
|
||||
'"motion_masks":[[30,90,120,50,0,0.5]]',
|
||||
);
|
||||
expect(() => parseMask("30,90,120")).toThrow(/x,y,w,h/);
|
||||
const { repeated } = await import("../commands/media-use/generate.js");
|
||||
expect(
|
||||
repeated(["--mask", "1,1,2,2", "--out", "a.mp4", "--mask=3,3,4,4,0-1"], "--mask"),
|
||||
).toEqual(["1,1,2,2", "3,3,4,4,0-1"]);
|
||||
await expect(
|
||||
generate({ type: "video", out: "x.mp4", prompt: "p", masks: [parseMask("1,1,2,2")] }),
|
||||
).rejects.toThrow(/--mask blanks regions of the --motion clip/);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("refuses a guide with a cut inside before sending anything", async () => {
|
||||
const guide = shots("two-shots.mp4", ["color=c=red:d=1", "testsrc2=d=1"]);
|
||||
const face = join(project, "face3.png");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "color=c=gray:s=36x64", "-frames:v", "1"],
|
||||
face,
|
||||
]);
|
||||
const fetched = vi.fn();
|
||||
vi.stubGlobal("fetch", fetched);
|
||||
const request = {
|
||||
type: "video" as const,
|
||||
out: join(project, "assets/refused.mp4"),
|
||||
prompt: "talks",
|
||||
references: [face],
|
||||
motion: guide,
|
||||
};
|
||||
await expect(generate(request)).rejects.toMatchObject({
|
||||
message: `--motion ${guide} has a cut at 1.00 s`,
|
||||
fix: "guide has a cut at 1.00 s: generate 0–1.00 s and 1.00–2.00 s as two shots, each with its own --motion clip",
|
||||
});
|
||||
expect(fetched).not.toHaveBeenCalled();
|
||||
expect(existsSync(request.out)).toBe(false);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("reports where HeyGen cut inside the returned shot", async () => {
|
||||
// The guide's last frame is another picture: a cut within a frame of the end is no second shot, so it is sent.
|
||||
const guide = shots("stray-frame.mp4", ["testsrc2=d=1", "color=c=blue:d=0.02"]);
|
||||
const made = shots("heygen-cut.mp4", ["testsrc2=d=1", "color=c=red:d=1"]);
|
||||
const clean = shots("heygen-clean.mp4", ["testsrc2=d=2"]);
|
||||
const face = join(project, "face4.png");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "color=c=gray:s=36x64", "-frames:v", "1"],
|
||||
face,
|
||||
]);
|
||||
let file = made;
|
||||
const sent: Record<string, unknown>[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
if (init?.body) sent.push(JSON.parse(String(init.body)));
|
||||
if (url === "https://files.test/c.mp4") return new Response(readFileSync(file));
|
||||
return json({
|
||||
data: { id: `c${sent.length}`, status: "completed", video_url: "https://files.test/c.mp4" },
|
||||
});
|
||||
});
|
||||
const request = { type: "video" as const, prompt: "walks", references: [face], motion: guide };
|
||||
const cut = await generate({ ...request, out: join(project, "assets/cut.mp4") });
|
||||
expect(sent).toHaveLength(1);
|
||||
expect(cut.cuts_inside).toEqual([1]);
|
||||
expect(readFileSync(join(project, ".media/manifest.jsonl"), "utf8")).toContain(
|
||||
'"cuts_inside":[1]',
|
||||
);
|
||||
file = clean;
|
||||
const one = await generate({ ...request, out: join(project, "assets/one.mp4"), seconds: 6 });
|
||||
expect(one.cuts_inside).toEqual([]);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)(
|
||||
"asks for 768p whatever the source's size, as 1080p bills about twice",
|
||||
async () => {
|
||||
const guide = shots("tall.mp4", ["testsrc2=d=1"], "1080x1920");
|
||||
const sent: Record<string, unknown>[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
|
||||
if (init?.body) sent.push(JSON.parse(String(init.body)));
|
||||
if (url === "https://files.test/t.mp4") return new Response(new Uint8Array([7]));
|
||||
return json({
|
||||
data: { id: "t1", status: "completed", video_url: "https://files.test/t.mp4" },
|
||||
});
|
||||
});
|
||||
const r = await generate({
|
||||
type: "video",
|
||||
out: join(project, "assets/tall.mp4"),
|
||||
prompt: "walks",
|
||||
motion: guide,
|
||||
aspect: "9:16",
|
||||
});
|
||||
expect(sent[0]).toMatchObject({ resolution: "768p", aspect_ratio: "9:16" });
|
||||
// An unreadable download is kept and its cuts reported as unknown.
|
||||
expect(r.cuts_inside).toBeNull();
|
||||
// A reference shot bills its seconds and the guide's at $0.05: 5 s and the 1 s guide.
|
||||
expect(priceOf({ type: "video", out: "a.mp4", motion: guide }).usd).toBe(0.3);
|
||||
},
|
||||
);
|
||||
|
||||
it("waits for a queued shot and downloads it", async () => {
|
||||
const answers = [
|
||||
json({ data: { video_id: "v1", status: "pending" } }),
|
||||
json({ data: { id: "v1", status: "processing" } }),
|
||||
json({ data: { id: "v1", status: "completed", video_url: "https://files.test/v1.mp4" } }),
|
||||
new Response(new Uint8Array([9])),
|
||||
];
|
||||
const urls: string[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string) => {
|
||||
urls.push(url);
|
||||
return answers.shift()!;
|
||||
});
|
||||
vi.useFakeTimers({ toFake: ["setTimeout"] });
|
||||
const pending = generate({
|
||||
type: "video",
|
||||
out: join(project, "assets/shot.mp4"),
|
||||
prompt: "push in",
|
||||
seconds: 6,
|
||||
});
|
||||
await vi.runAllTimersAsync();
|
||||
const r = await pending;
|
||||
vi.useRealTimers();
|
||||
expect(urls.slice(0, 3)).toEqual([
|
||||
"https://api.test/v3/models/videos",
|
||||
"https://api.test/v3/models/videos/v1",
|
||||
"https://api.test/v3/models/videos/v1",
|
||||
]);
|
||||
expect(existsSync(r.out)).toBe(true);
|
||||
expect(r.job).toBe("v1");
|
||||
});
|
||||
|
||||
it("names its own price when HeyGen refuses for credits without saying how many", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
async () =>
|
||||
new Response(JSON.stringify({ error: { code: "hf_usage_exhausted" } }), { status: 402 }),
|
||||
);
|
||||
const request = {
|
||||
type: "image",
|
||||
out: join(project, "assets/broke.png"),
|
||||
prompt: "a desk",
|
||||
} as const;
|
||||
await expect(
|
||||
generate(request, undefined, creditsOf(priceOf(request)) ?? undefined),
|
||||
).rejects.toThrow("this step needs about 67");
|
||||
});
|
||||
|
||||
it("prices before making", () => {
|
||||
expect(priceOf({ type: "video", out: "a.mp4", seconds: 6 }).usd).toBe(0.12);
|
||||
expect(priceOf({ type: "music", out: "a.mp3" }).usd).toBe(0.09);
|
||||
expect(
|
||||
priceOf({
|
||||
type: "image",
|
||||
out: "a.png",
|
||||
model: "gpt-image-2.5",
|
||||
aspect: "1:1",
|
||||
references: ["a", "b"],
|
||||
}).usd,
|
||||
).toBe(0.07);
|
||||
expect(priceOf({ type: "voice", out: "a.mp3" }).usd).toBeNull();
|
||||
});
|
||||
|
||||
it("prices only a request the real call would accept", () => {
|
||||
// --plan runs the same checks as the real call, so it never prices a request the call would refuse.
|
||||
const [plate, voice] = [join(project, "plate.png"), join(project, "voice.m4a")];
|
||||
writeFileSync(plate, "");
|
||||
writeFileSync(voice, "");
|
||||
const shot = { type: "video" as const, out: "a.mp4", references: [plate], audio: voice };
|
||||
expect(() => checkRequest(shot)).toThrow("--prompt is required for this type");
|
||||
expect(() =>
|
||||
checkRequest({ ...shot, prompt: "x", audio: join(project, "missing.m4a") }),
|
||||
).toThrow("no file at");
|
||||
expect(() => checkRequest({ ...shot, prompt: "she talks to the camera" })).not.toThrow();
|
||||
expect(() => checkRequest({ ...shot, prompt: "x", firstFrame: "f.png" })).toThrow(
|
||||
"--first-frame cannot be used",
|
||||
);
|
||||
expect(() => checkRequest({ ...shot, prompt: "x", out: "a.png" })).toThrow(
|
||||
"--out a.png does not fit a video",
|
||||
);
|
||||
expect(() => checkRequest({ ...shot, prompt: "x", masks: [parseMask("1,1,2,2")] })).toThrow(
|
||||
"--mask blanks regions of the --motion clip",
|
||||
);
|
||||
});
|
||||
|
||||
it("charges a reference shot's pictures after the first two", () => {
|
||||
const shot = (pictures: number) => ({
|
||||
type: "video" as const,
|
||||
out: "a.mp4",
|
||||
seconds: 5,
|
||||
references: Array.from({ length: pictures }, (_, i) => `${i}.png`),
|
||||
});
|
||||
const credits = [1, 2, 3, 9].map((n) => creditsOf(priceOf(shot(n))));
|
||||
expect(credits).toEqual([167, 167, 180, 260]);
|
||||
expect(priceOf(shot(9)).list).toBeCloseTo(0.39);
|
||||
expect(() => checkLimit(credits[3]!, 250)).toThrow("about 260 credits, over --max-credits 250");
|
||||
expect(() => checkLimit(credits[3]!, 260)).not.toThrow();
|
||||
});
|
||||
|
||||
it("keeps a call under the person's credit limit, or refuses it before sending", () => {
|
||||
const shot = creditsOf(priceOf({ type: "video", out: "a.mp4", seconds: 6 }));
|
||||
expect(shot).toBe(72);
|
||||
expect(creditsOf(priceOf({ type: "image", out: "a.png" }))).toBe(67);
|
||||
expect(() => checkLimit(shot, 72)).not.toThrow();
|
||||
expect(() => checkLimit(shot, 30)).toThrow("about 72 credits, over --max-credits 30");
|
||||
expect(() => checkLimit(null, 30)).toThrow("cannot be priced ahead");
|
||||
expect(() => checkLimit(null, undefined)).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -1,423 +0,0 @@
|
||||
// New media on HeyGen for `media-use generate`: a still (optionally edited from a reference), a shot (optionally from
|
||||
// a first frame), a person speaking from one photo, a voice reading text, or music. Every call is priced first with
|
||||
// --plan, keyed for idempotency, downloaded into the project and recorded in the media ledger as generated.
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import { findCuts, signatures } from "../measure/cuts.js";
|
||||
import { Fix, probeVideo, round, runFfmpeg, videoClip } from "../measure/media.js";
|
||||
import { idempotencyKey, runJob, type Job } from "./v3.js";
|
||||
|
||||
export type GenerateType = "image" | "video" | "avatar" | "voice" | "music";
|
||||
export const GENERATE_TYPES: readonly GenerateType[] = [
|
||||
"image",
|
||||
"video",
|
||||
"avatar",
|
||||
"voice",
|
||||
"music",
|
||||
];
|
||||
|
||||
export interface GenerateRequest {
|
||||
type: GenerateType;
|
||||
out: string;
|
||||
prompt?: string;
|
||||
/** image: pictures the result is edited from or styled after. video: the person or things to keep in the shot. */
|
||||
references?: string[];
|
||||
/** video: speech the person in the references says, lips in sync (a stretch of the original's voice). */
|
||||
audio?: string;
|
||||
/** video: a clip whose camera and body motion the shot follows (the original's same stretch, up to 15 s). */
|
||||
motion?: string;
|
||||
/** video: regions of the motion clip blanked before it is sent (its drawn text), so the shot does not copy them. */
|
||||
masks?: Mask[];
|
||||
/** video: the shot starts from this picture. */
|
||||
firstFrame?: string;
|
||||
/** avatar: the photo of the person who speaks. */
|
||||
photo?: string;
|
||||
/** avatar and voice: what is said. */
|
||||
script?: string;
|
||||
voice?: string;
|
||||
seconds?: number;
|
||||
aspect?: string;
|
||||
model?: string;
|
||||
instrumental?: boolean;
|
||||
bpm?: number;
|
||||
}
|
||||
|
||||
/** A region of the motion clip in its pixels, and optionally the seconds of the clip it covers. */
|
||||
export interface Mask {
|
||||
box: [number, number, number, number];
|
||||
window?: [number, number];
|
||||
}
|
||||
|
||||
/** `x,y,w,h` or `x,y,w,h,a-b` (seconds of the motion clip). */
|
||||
export function parseMask(text: string): Mask {
|
||||
const m =
|
||||
/^\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*(?:,\s*(\d+(?:\.\d+)?)\s*-\s*(\d+(?:\.\d+)?)\s*)?$/.exec(
|
||||
text,
|
||||
);
|
||||
if (!m || Number(m[3]) <= 0 || Number(m[4]) <= 0)
|
||||
throw new Fix(`--mask ${text} is not x,y,w,h[,a-b]`, "--mask 190,405,700,175,1.0-2.7");
|
||||
const box: Mask["box"] = [Number(m[1]), Number(m[2]), Number(m[3]), Number(m[4])];
|
||||
return m[5] ? { box, window: [Number(m[5]), Number(m[6])] } : { box };
|
||||
}
|
||||
|
||||
/**
|
||||
* Filters that fill each mask's region from the pixels around it (ffmpeg delogo), so no letter survives and the
|
||||
* motion around it stays. delogo needs a pixel of picture on every side, so boxes are kept 1 px inside the frame.
|
||||
*/
|
||||
function maskFilters(masks: Mask[], width: number, height: number): string[] {
|
||||
return masks.flatMap(({ box: [x, y, w, h], window }) => {
|
||||
const [x0, y0] = [Math.max(1, x), Math.max(1, y)];
|
||||
const [x1, y1] = [Math.min(width - 2, x + w), Math.min(height - 2, y + h)];
|
||||
if (x1 - x0 < 1 || y1 - y0 < 1) return [];
|
||||
const when = window ? `:enable='between(t,${window[0]},${window[1]})'` : "";
|
||||
return [`delogo=x=${x0}:y=${y0}:w=${x1 - x0}:h=${y1 - y0}${when}`];
|
||||
});
|
||||
}
|
||||
|
||||
interface Route {
|
||||
path: string;
|
||||
poll: (id: string) => string;
|
||||
body: (r: GenerateRequest, media: Media) => Record<string, unknown>;
|
||||
/** Where the finished job keeps the file's URL. */
|
||||
urls: string[];
|
||||
ledger: "image" | "video" | "voice" | "bgm";
|
||||
ext: string[];
|
||||
}
|
||||
|
||||
type Media = (
|
||||
path: string,
|
||||
kind: "image" | "audio" | "video",
|
||||
) => { type: "base64"; media_type: string; data: string };
|
||||
|
||||
const DEFAULTS = { image: "nano-banana-2", video: "heygen-video-1", music: "lyria-3-pro" } as const;
|
||||
const VIDEO_RESOLUTION = "768p";
|
||||
/** HeyGen follows at most this many seconds of a guide clip; the rest is not sent. */
|
||||
export const GUIDE_MAX_S = 15;
|
||||
const MUSIC_LENGTHS = [30, 60, 120, 180];
|
||||
|
||||
const need = <T>(value: T | undefined, flag: string, example: string): T => {
|
||||
if (value === undefined || value === "")
|
||||
throw new Fix(`${flag} is required for this type`, example);
|
||||
return value;
|
||||
};
|
||||
|
||||
const ROUTES: Record<GenerateType, Route> = {
|
||||
image: {
|
||||
path: "/v3/images",
|
||||
poll: (id) => `/v3/images/${id}`,
|
||||
body: (r, media) => ({
|
||||
model: r.model ?? DEFAULTS.image,
|
||||
prompt: need(r.prompt, "--prompt", '--prompt "the same room, empty, no text"'),
|
||||
num_images: 1,
|
||||
...(r.aspect ? { aspect_ratio: r.aspect } : {}),
|
||||
...(r.references?.length
|
||||
? { reference_images: r.references.map((p) => media(p, "image")) }
|
||||
: {}),
|
||||
}),
|
||||
urls: ["images.0.url", "image_url", "url"],
|
||||
ledger: "image",
|
||||
ext: [".png", ".jpg", ".jpeg", ".webp"],
|
||||
},
|
||||
video: {
|
||||
path: "/v3/models/videos",
|
||||
poll: (id) => `/v3/models/videos/${id}`,
|
||||
// fallow-ignore-next-line complexity
|
||||
body: (r, media) => {
|
||||
// A person speaking given audio is a reference shot: the references show who, the audio drives the lips.
|
||||
const spoken = !!r.audio || !!r.references?.length || !!r.motion;
|
||||
if (spoken && r.firstFrame)
|
||||
throw new Fix(
|
||||
"--first-frame cannot be used with --reference, --audio or --motion",
|
||||
"a speaking shot takes the person as --reference and the voice as --audio",
|
||||
);
|
||||
return {
|
||||
model: r.model ?? DEFAULTS.video,
|
||||
mode: spoken ? "reference_to_video" : r.firstFrame ? "image_to_video" : "text_to_video",
|
||||
prompt: need(r.prompt, "--prompt", '--prompt "slow push in on the desk"'),
|
||||
duration: Math.min(15, Math.max(5, Math.round(r.seconds ?? 5))),
|
||||
// 1080p bills about twice 768p (2026-10-23, 5 s 9:16 jobs: text $0.10 vs $0.20, a reference picture $0.25
|
||||
// vs $0.45, a 1 s guide clip $0.30 vs $0.55), so every shot stays at 768p whatever the source's size.
|
||||
resolution: VIDEO_RESOLUTION,
|
||||
...(r.firstFrame ? { image: media(r.firstFrame, "image") } : {}),
|
||||
...(r.firstFrame ? {} : { aspect_ratio: r.aspect ?? "16:9" }),
|
||||
...(r.references?.length
|
||||
? { reference_images: r.references.map((p) => media(p, "image")) }
|
||||
: {}),
|
||||
...(r.audio ? { reference_audio: [media(r.audio, "audio")] } : {}),
|
||||
// One guide clip: HeyGen follows its camera and body motion, and takes the face from the references.
|
||||
...(r.motion ? { reference_videos: [media(r.motion, "video")] } : {}),
|
||||
};
|
||||
},
|
||||
urls: ["video_url", "output.video_url", "url"],
|
||||
ledger: "video",
|
||||
ext: [".mp4", ".mov", ".webm"],
|
||||
},
|
||||
avatar: {
|
||||
path: "/v3/videos",
|
||||
poll: (id) => `/v3/videos/${id}`,
|
||||
body: (r, media) => ({
|
||||
type: "image",
|
||||
image: media(need(r.photo, "--photo", "--photo assets/presenter.png"), "image"),
|
||||
script: need(r.script, "--script", '--script "What they say"'),
|
||||
voice_id: need(
|
||||
r.voice,
|
||||
"--voice",
|
||||
"--voice <voice id> (npx hyperframes media-use resolve --type voice lists voices)",
|
||||
),
|
||||
...(r.aspect ? { aspect_ratio: r.aspect } : {}),
|
||||
output_format: "mp4",
|
||||
}),
|
||||
urls: ["video_url"],
|
||||
ledger: "video",
|
||||
ext: [".mp4"],
|
||||
},
|
||||
voice: {
|
||||
path: "/v3/voices/speech",
|
||||
poll: (id) => `/v3/voices/speech/${id}`,
|
||||
body: (r) => ({
|
||||
text: need(r.script, "--script", '--script "What the voice says"'),
|
||||
voice_id: need(r.voice, "--voice", "--voice <voice id>"),
|
||||
}),
|
||||
urls: ["audio_url"],
|
||||
ledger: "voice",
|
||||
ext: [".mp3", ".wav", ".m4a"],
|
||||
},
|
||||
music: {
|
||||
path: "/v3/audio/music",
|
||||
poll: (id) => `/v3/audio/music/${id}`,
|
||||
body: (r) => ({
|
||||
model: r.model ?? DEFAULTS.music,
|
||||
prompt: need(r.prompt, "--prompt", '--prompt "warm lo-fi beat, soft keys"'),
|
||||
duration_seconds: MUSIC_LENGTHS.find((s) => s >= (r.seconds ?? 30)) ?? 180,
|
||||
instrumental: r.instrumental ?? true,
|
||||
...(r.bpm ? { bpm: Math.round(r.bpm) } : {}),
|
||||
}),
|
||||
urls: ["audio_url"],
|
||||
ledger: "bgm",
|
||||
ext: [".mp3"],
|
||||
},
|
||||
};
|
||||
|
||||
const isRecord = (v: unknown): v is Record<string, unknown> =>
|
||||
typeof v === "object" && v !== null && !Array.isArray(v);
|
||||
|
||||
/** The first URL at any of these dotted paths of a job. */
|
||||
export function urlOf(job: Job, paths: string[]): string | null {
|
||||
for (const p of paths) {
|
||||
let v: unknown = job;
|
||||
for (const key of p.split("."))
|
||||
v = Array.isArray(v) ? v[Number(key)] : isRecord(v) ? v[key] : undefined;
|
||||
if (typeof v === "string" && /^https?:\/\//.test(v)) return v;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function needFile(path: string): void {
|
||||
if (!existsSync(path))
|
||||
throw new Fix(`no file at ${path}`, "pass a picture or sound from the project");
|
||||
}
|
||||
|
||||
/**
|
||||
* A picture or a sound as the routes take it inline, pictures shrunk to 2048 px so the body stays small, a motion
|
||||
* clip with its masks blanked.
|
||||
*/
|
||||
function inline(work: string, masks: Mask[] = []): Media {
|
||||
return (path, kind) => {
|
||||
needFile(path);
|
||||
const out = join(
|
||||
work,
|
||||
`in-${Math.random().toString(36).slice(2)}.${{ image: "jpg", audio: "mp3", video: "mp4" }[kind]}`,
|
||||
);
|
||||
if (kind === "image")
|
||||
runFfmpeg([
|
||||
"-y",
|
||||
"-i",
|
||||
path,
|
||||
"-vf",
|
||||
"scale='min(2048,iw)':'min(2048,ih)':force_original_aspect_ratio=decrease",
|
||||
"-q:v",
|
||||
"3",
|
||||
"-frames:v",
|
||||
"1",
|
||||
out,
|
||||
]);
|
||||
else if (kind === "audio")
|
||||
runFfmpeg(["-y", "-i", path, "-vn", "-ac", "1", "-c:a", "libmp3lame", "-b:a", "128k", out]);
|
||||
// A guide clip only needs its motion: 720 px on the long side, no sound, at most 15 s.
|
||||
else {
|
||||
const { width, height } = masks.length ? probeVideo(path) : { width: 0, height: 0 };
|
||||
const filters = [
|
||||
...maskFilters(masks, width, height),
|
||||
"scale='if(gt(iw,ih),min(720,iw),-2)':'if(gt(iw,ih),-2,min(720,ih))'",
|
||||
];
|
||||
runFfmpeg([
|
||||
...["-y", "-i", path, "-t", "15", "-an", "-vf", filters.join(",")],
|
||||
...["-c:v", "libx264", "-crf", "28", "-preset", "veryfast", "-pix_fmt", "yuv420p", out],
|
||||
]);
|
||||
}
|
||||
return {
|
||||
type: "base64",
|
||||
media_type: { image: "image/jpeg", audio: "audio/mpeg", video: "video/mp4" }[kind],
|
||||
data: readFileSync(out).toString("base64"),
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The seconds where a new shot starts in a video's first `seconds`, and the length read. A cut within a frame of
|
||||
* either end leaves a stray frame, not a second shot, and does not count.
|
||||
*/
|
||||
function cutsInside(path: string, seconds?: number): { cuts: number[]; length: number } {
|
||||
const info = probeVideo(path);
|
||||
const clip = videoClip(info.path, 0, seconds, info);
|
||||
const { frames } = signatures(clip.input, clip.filters);
|
||||
const cuts = findCuts(frames)
|
||||
.filter((c) => c.frame > 1 && c.frame < frames.length - 1)
|
||||
.map((c) => round(c.frame / info.fps, 3));
|
||||
return { cuts, length: round(frames.length / info.fps, 3) };
|
||||
}
|
||||
|
||||
const COUNT = ["", "one", "two", "three", "four", "five", "six"];
|
||||
|
||||
/**
|
||||
* Refuses a --motion clip holding more than one shot before anything is sent: HeyGen follows the guide's cut into
|
||||
* a cut of its own, so each shot is its own generation.
|
||||
*/
|
||||
function checkGuide(r: Pick<GenerateRequest, "motion">): void {
|
||||
if (!r.motion) return;
|
||||
const { cuts, length } = cutsInside(r.motion, GUIDE_MAX_S);
|
||||
if (!cuts.length) return;
|
||||
const s = (t: number) => t.toFixed(2);
|
||||
const and = (items: string[]) => `${items.slice(0, -1).join(", ")} and ${items.at(-1)}`;
|
||||
const at =
|
||||
cuts.length > 1 ? `cuts at ${and(cuts.map((t) => `${s(t)} s`))}` : `a cut at ${s(cuts[0]!)} s`;
|
||||
const edges = [0, ...cuts, length];
|
||||
const shots = edges.slice(1).map((e, i) => `${i ? s(edges[i]!) : "0"}–${s(e)} s`);
|
||||
throw new Fix(
|
||||
`--motion ${r.motion} has ${at}`,
|
||||
`guide has ${at}: generate ${and(shots)} as ${COUNT[shots.length] ?? shots.length} shots, each with its own --motion clip`,
|
||||
);
|
||||
}
|
||||
|
||||
/** What the real call would refuse, checked without sending anything: `--plan` prices only a request that would run.
|
||||
* The route's body runs with a stand-in for the media, so its required flags and conflicts are checked as sent. */
|
||||
export function checkRequest(r: GenerateRequest): void {
|
||||
const route = ROUTES[r.type];
|
||||
if (!route.ext.includes(extname(r.out).toLowerCase()))
|
||||
throw new Fix(`--out ${r.out} does not fit a ${r.type}`, `--out assets/name${route.ext[0]}`);
|
||||
if (r.masks?.length && !r.motion)
|
||||
throw new Fix("--mask blanks regions of the --motion clip", "pass --motion with --mask");
|
||||
route.body(r, (path) => {
|
||||
needFile(path);
|
||||
return { type: "base64", media_type: "", data: "" };
|
||||
});
|
||||
checkGuide(r);
|
||||
}
|
||||
|
||||
/** The nearest folder at or above `from` holding index.html. */
|
||||
function projectOf(from: string): string | null {
|
||||
for (let dir = resolve(from); ; dir = dirname(dir)) {
|
||||
if (existsSync(join(dir, "index.html"))) return dir;
|
||||
if (dirname(dir) === dir) return null;
|
||||
}
|
||||
}
|
||||
|
||||
export interface Generated {
|
||||
out: string;
|
||||
type: GenerateType;
|
||||
job: string;
|
||||
route: string;
|
||||
ledger?: string;
|
||||
/** video: seconds where HeyGen cut inside the returned shot; null when the file could not be read. */
|
||||
cuts_inside?: number[] | null;
|
||||
}
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
export async function generate(
|
||||
r: GenerateRequest,
|
||||
onWait?: (status: string) => void,
|
||||
credits?: number,
|
||||
): Promise<Generated> {
|
||||
checkRequest(r);
|
||||
const route = ROUTES[r.type];
|
||||
const out = resolve(r.out);
|
||||
mkdirSync(dirname(out), { recursive: true });
|
||||
const project = projectOf(dirname(out));
|
||||
const work = mkdtempSync(join(tmpdir(), "hf-generate-"));
|
||||
try {
|
||||
const body = route.body(r, inline(work, r.masks));
|
||||
const key = idempotencyKey(route.path, [JSON.stringify(body)]);
|
||||
const job = await runJob(route.path, body, key, (id) => route.poll(encodeURIComponent(id)), {
|
||||
log: {
|
||||
projectDir: project ?? undefined,
|
||||
what: `generate ${r.type} ${relative(project ?? ".", out)}`,
|
||||
},
|
||||
onWait: (j) => onWait?.(j.status),
|
||||
timeoutMs: 30 * 60_000,
|
||||
credits,
|
||||
});
|
||||
const url = urlOf(job, route.urls);
|
||||
if (!url)
|
||||
throw new Fix(
|
||||
`HeyGen finished job ${job.id} without a file URL`,
|
||||
"run the same command again; it re-reads the job",
|
||||
);
|
||||
const res = await fetch(url, { signal: AbortSignal.timeout(300_000) });
|
||||
if (!res.ok)
|
||||
throw new Fix(
|
||||
`could not download the result (${res.status})`,
|
||||
"run the same command again; it re-reads the job",
|
||||
);
|
||||
writeFileSync(out, Buffer.from(await res.arrayBuffer()));
|
||||
// HeyGen sometimes cuts inside a shot on its own; the shot is paid for and kept, the cuts are reported.
|
||||
let cuts: number[] | null | undefined;
|
||||
if (r.type === "video")
|
||||
try {
|
||||
cuts = cutsInside(out).cuts;
|
||||
} catch {
|
||||
cuts = null;
|
||||
}
|
||||
let ledger: string | undefined;
|
||||
if (project) {
|
||||
const { recordInPlace } = await import("../media-use/lib/manifest.mjs");
|
||||
const { regenerateIndex } = await import("../media-use/lib/index-gen.mjs");
|
||||
const describe = (p?: string) =>
|
||||
p ? relative(project, resolve(p)).split("\\").join("/") : undefined;
|
||||
ledger = recordInPlace(project, {
|
||||
type: route.ledger,
|
||||
path: relative(project, out).split("\\").join("/"),
|
||||
source: "generated",
|
||||
description: r.prompt ?? r.script ?? r.type,
|
||||
provenance: {
|
||||
provider: "heygen",
|
||||
route: route.path,
|
||||
model: body.model ?? null,
|
||||
job: job.id,
|
||||
prompt: r.prompt ?? null,
|
||||
...(r.references?.length ? { references: r.references.map(describe) } : {}),
|
||||
...(r.firstFrame ? { first_frame: describe(r.firstFrame) } : {}),
|
||||
...(r.photo ? { photo: describe(r.photo) } : {}),
|
||||
...(r.audio ? { audio: describe(r.audio) } : {}),
|
||||
...(r.motion ? { motion: describe(r.motion) } : {}),
|
||||
...(r.masks?.length
|
||||
? { motion_masks: r.masks.map((m) => [...m.box, ...(m.window ?? [])]) }
|
||||
: {}),
|
||||
...(r.voice ? { voice: r.voice } : {}),
|
||||
...(cuts !== undefined ? { cuts_inside: cuts } : {}),
|
||||
},
|
||||
}).id;
|
||||
regenerateIndex(project);
|
||||
}
|
||||
return {
|
||||
out,
|
||||
type: r.type,
|
||||
job: job.id,
|
||||
route: route.path,
|
||||
...(ledger ? { ledger } : {}),
|
||||
...(cuts !== undefined ? { cuts_inside: cuts } : {}),
|
||||
};
|
||||
} finally {
|
||||
rmSync(work, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
@@ -1,52 +0,0 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("../cloud/auth.js", () => ({
|
||||
resolveCloudAuthHeaders: async () => ({ "x-api-key": "k" }),
|
||||
resolveCloudBaseUrl: () => "https://api.dev.heygen.com",
|
||||
}));
|
||||
|
||||
const { pageUrl, readPlan } = await import("./plan.js");
|
||||
|
||||
const asked: string[] = [];
|
||||
const answer = (data: unknown) =>
|
||||
vi.stubGlobal("fetch", async (url: string) => {
|
||||
asked.push(url);
|
||||
return new Response(JSON.stringify({ code: 100, data }));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
describe("readPlan", () => {
|
||||
it("reads the plan the way the subscription route sends it", async () => {
|
||||
answer({
|
||||
subscription: { plan_name: "pro", status: "active" },
|
||||
usage: { remaining_units: 25000 },
|
||||
});
|
||||
expect(await readPlan()).toEqual({ plan: "pro", credits: 25000 });
|
||||
expect(asked.at(-1)).toBe("https://api2.dev.heygen.com/v1/payment/hyperframes/subscription");
|
||||
answer({ subscription: null, usage: { remaining_units: 0 } });
|
||||
expect(await readPlan()).toEqual({ plan: "none", credits: 0 });
|
||||
answer({
|
||||
subscription: { plan_name: "pro", status: "incomplete" },
|
||||
usage: { remaining_units: 0 },
|
||||
});
|
||||
expect((await readPlan())?.plan).toBe("none");
|
||||
});
|
||||
});
|
||||
|
||||
describe("pageUrl", () => {
|
||||
it("links the web app for HYPERFRAMES_ENV and tells the page the buyer came from the CLI", () => {
|
||||
vi.stubEnv("HYPERFRAMES_PLANS_URL", "");
|
||||
vi.stubEnv("HYPERFRAMES_CREDITS_URL", "");
|
||||
vi.stubEnv("HYPERFRAMES_ENV", "prod");
|
||||
expect(pageUrl("plans")).toBe("https://hyperframes.studio/pricing?from=cli");
|
||||
vi.stubEnv("HYPERFRAMES_ENV", "dev");
|
||||
expect(pageUrl("plans")).toBe("https://dev.hyperframes.dev/pricing?from=cli");
|
||||
expect(pageUrl("credits", { needed: 180, tool: "reverse" })).toBe(
|
||||
"https://dev.hyperframes.dev/account/plan?from=cli&needed_credits=180&tool=reverse",
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -1,65 +0,0 @@
|
||||
// The HyperFrames plan and credits the signed-in account holds, and the web pages that change them.
|
||||
import { oauthEnvironment } from "../auth/config.js";
|
||||
import { resolveCloudAuthHeaders, resolveCloudBaseUrl } from "../cloud/auth.js";
|
||||
|
||||
export interface Plan {
|
||||
plan: string;
|
||||
credits: number;
|
||||
mock?: boolean;
|
||||
}
|
||||
|
||||
const UNPAID = new Set(["incomplete", "incomplete_expired"]);
|
||||
|
||||
const isRecord = (v: unknown): v is Record<string, unknown> =>
|
||||
typeof v === "object" && v !== null && !Array.isArray(v);
|
||||
|
||||
// ponytail: HYPERFRAMES_MOCK_PLAN="pro:25000" | "none" stands in when the server will not answer the CLI client
|
||||
// (EF admits only the Desktop client today); drop it once the CLI client is registered.
|
||||
function mockPlan(): Plan | null {
|
||||
const raw = process.env["HYPERFRAMES_MOCK_PLAN"]?.trim();
|
||||
if (!raw) return null;
|
||||
const [plan = "none", credits = "0"] = raw.split(":");
|
||||
return { plan, credits: Number(credits) || 0, mock: true };
|
||||
}
|
||||
|
||||
// Payment routes live on api2, beside the OAuth ones; the v3 generation routes stay on api.
|
||||
const billingBaseUrl = () => resolveCloudBaseUrl().replace("://api.", "://api2.");
|
||||
|
||||
export async function readPlan(): Promise<Plan | null> {
|
||||
try {
|
||||
const res = await fetch(`${billingBaseUrl()}/v1/payment/hyperframes/subscription`, {
|
||||
headers: { ...(await resolveCloudAuthHeaders()), accept: "application/json" },
|
||||
signal: AbortSignal.timeout(15_000),
|
||||
});
|
||||
const body: unknown = await res.json().catch(() => null);
|
||||
const data = isRecord(body) && isRecord(body.data) ? body.data : body;
|
||||
if (!res.ok || !isRecord(data)) return mockPlan();
|
||||
const usage = isRecord(data.usage) ? data.usage : {};
|
||||
// studio-contracts' BillingSubscription: `subscription` is null on Free; an unpaid placeholder still names a plan.
|
||||
const sub = isRecord(data.subscription) ? data.subscription : null;
|
||||
const paid = !!sub && !UNPAID.has(String(sub.status));
|
||||
return {
|
||||
plan: paid ? String(sub.plan_name ?? "paid") : "none",
|
||||
credits: Number(usage.remaining_units ?? 0) || 0,
|
||||
};
|
||||
} catch {
|
||||
return mockPlan();
|
||||
}
|
||||
}
|
||||
|
||||
const webBase = () =>
|
||||
oauthEnvironment() === "dev" ? "https://dev.hyperframes.dev" : "https://hyperframes.studio";
|
||||
const plansUrl = () => process.env["HYPERFRAMES_PLANS_URL"] || `${webBase()}/pricing`;
|
||||
const creditsUrl = () => process.env["HYPERFRAMES_CREDITS_URL"] || `${webBase()}/account/plan`;
|
||||
|
||||
/** A plans or credits page link with what the web side reads: `from=cli`, the credits needed, the waiting step. */
|
||||
export function pageUrl(
|
||||
kind: "plans" | "credits",
|
||||
fields: { needed?: number; tool?: string; base?: string } = {},
|
||||
): string {
|
||||
const url = new URL(fields.base ?? (kind === "plans" ? plansUrl() : creditsUrl()));
|
||||
url.searchParams.set("from", "cli");
|
||||
if (fields.needed) url.searchParams.set("needed_credits", String(fields.needed));
|
||||
if (fields.tool) url.searchParams.set("tool", fields.tool);
|
||||
return url.toString();
|
||||
}
|
||||
@@ -1,120 +0,0 @@
|
||||
// Prices before a paid call, from the HyperFrames app's creation pricing (hyperframes-internal
|
||||
// packages/desktop/shared/creationPricing.mjs, 2026-10-22). HeyGen charges in units of $1/60, rounded up; the label
|
||||
// rounds up to the cent. Routes HeyGen does not price ahead say so instead of guessing.
|
||||
import { Fix, probeVideo, round } from "../measure/media.js";
|
||||
import { GUIDE_MAX_S, type GenerateRequest } from "./generate.js";
|
||||
|
||||
export interface Price {
|
||||
usd: number | null;
|
||||
/** The list price before HeyGen's unit rounding. */
|
||||
list: number | null;
|
||||
/** What a HyperFrames plan charges, before rounding up to a whole credit. */
|
||||
credits: number | null;
|
||||
basis: string;
|
||||
}
|
||||
|
||||
const UNIT = 1 / 60;
|
||||
const charged = (usd: number) => Math.ceil(usd / UNIT - 1e-9) * UNIT;
|
||||
const cents = (usd: number) => Math.ceil(charged(usd) * 100 - 1e-9) / 100;
|
||||
const HF_CREDITS_PER_USD = 1000;
|
||||
const at = (list: number, basis: string, credits = list * HF_CREDITS_PER_USD): Price => ({
|
||||
usd: cents(list),
|
||||
list,
|
||||
credits,
|
||||
basis,
|
||||
});
|
||||
|
||||
const GPT_IMAGE: Record<string, number> = {
|
||||
"1:1": 0.05268,
|
||||
"16:9": 0.0396,
|
||||
"9:16": 0.0396,
|
||||
"4:3": 0.03612,
|
||||
"3:4": 0.03612,
|
||||
};
|
||||
const NANO_IMAGE = 0.067;
|
||||
const VIDEO_768P_PER_S = 0.018;
|
||||
// A reference shot (pictures, voice or guide clip) bills more, and its guide clip's seconds bill too: measured
|
||||
// 2026-10-23 from the wallet, 5 s with a picture $0.25, 10 s $0.50, 5 s with a 1 s guide $0.30, with a 3 s guide $0.40.
|
||||
const VIDEO_REF_768P_PER_S = 0.05;
|
||||
// A reference shot's first two pictures are included; each one after them bills on its own line.
|
||||
const VIDEO_INCLUDED_IMAGES = 2;
|
||||
const VIDEO_EXTRA_IMAGE = 0.02;
|
||||
// The price book's HyperFrames video rows at 768p (experiment-framework price_book rules.json, 2026-10-28).
|
||||
const HF_VIDEO_768P_PER_S = 12;
|
||||
const HF_VIDEO_REF_768P_PER_S = 100 / 3;
|
||||
const HF_VIDEO_EXTRA_IMAGE = 40 / 3;
|
||||
const MUSIC_PER_TRACK = 0.08;
|
||||
const AVATAR_IV_PER_MIN = 4.83;
|
||||
const SCRIPT_WORDS_PER_S = 2;
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
export function priceOf(r: GenerateRequest): Price {
|
||||
switch (r.type) {
|
||||
case "image": {
|
||||
const refs = r.references?.length ?? 0;
|
||||
if ((r.model ?? "nano-banana-2") === "nano-banana-2")
|
||||
return at(NANO_IMAGE, "nano-banana-2, $0.067 an image");
|
||||
const each = GPT_IMAGE[r.aspect ?? "16:9"] ?? GPT_IMAGE["16:9"]!;
|
||||
return at(
|
||||
each + Math.max(0, refs - 1) * 0.0082,
|
||||
`gpt-image-2.5 at ${r.aspect ?? "16:9"}, $0.0082 a reference after the first`,
|
||||
);
|
||||
}
|
||||
case "video": {
|
||||
const seconds = Math.min(15, Math.max(5, Math.round(r.seconds ?? 5)));
|
||||
if (r.audio || r.references?.length || r.motion) {
|
||||
const guide = r.motion ? Math.min(GUIDE_MAX_S, probeVideo(r.motion).pictureDuration) : 0;
|
||||
const extra = Math.max(0, (r.references?.length ?? 0) - VIDEO_INCLUDED_IMAGES);
|
||||
const billed = seconds + guide;
|
||||
return at(
|
||||
billed * VIDEO_REF_768P_PER_S + extra * VIDEO_EXTRA_IMAGE,
|
||||
`heygen-video-1 reference shot at 768p, ${seconds} s${guide ? ` and the ${round(guide, 2)} s guide` : ""} at $0.05 a second${extra ? `, ${extra} extra pictures at $0.02` : ""}`,
|
||||
billed * HF_VIDEO_REF_768P_PER_S + extra * HF_VIDEO_EXTRA_IMAGE,
|
||||
);
|
||||
}
|
||||
return at(
|
||||
seconds * VIDEO_768P_PER_S,
|
||||
`heygen-video-1 at 768p, ${seconds} s at $0.018 a second`,
|
||||
seconds * HF_VIDEO_768P_PER_S,
|
||||
);
|
||||
}
|
||||
case "avatar": {
|
||||
const words = (r.script ?? "").split(/\s+/).filter(Boolean).length;
|
||||
const seconds = Math.max(1, Math.floor(words / SCRIPT_WORDS_PER_S));
|
||||
return at(
|
||||
Math.ceil(seconds * AVATAR_IV_PER_MIN) / 60,
|
||||
`avatar video, about ${seconds} s from ${words} words at $4.83 a minute`,
|
||||
);
|
||||
}
|
||||
case "music":
|
||||
return at(MUSIC_PER_TRACK, "lyria-3-pro, $0.08 a track");
|
||||
case "voice":
|
||||
return {
|
||||
usd: null,
|
||||
list: null,
|
||||
credits: null,
|
||||
basis: "HeyGen does not price catalog speech ahead; it is charged per character",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// No HeyGen unit rounding here: a $0.067 image is charged 67 credits, measured on dev 2026-10-29. Still an estimate.
|
||||
export const creditsOf = (p: Price): number | null =>
|
||||
p.credits === null ? null : Math.ceil(p.credits - 1e-9);
|
||||
|
||||
/** Refuses, before anything is sent, a call that would not fit what is left of the person's credit limit. */
|
||||
export function checkLimit(credits: number | null, limit: number | undefined): void {
|
||||
if (limit === undefined) return;
|
||||
const fix =
|
||||
"tell the person what their limit leaves out, then ask whether to raise it or make the video without this piece";
|
||||
if (credits === null)
|
||||
throw new Fix(
|
||||
`this call cannot be priced ahead, so it cannot be kept under --max-credits ${limit}; nothing was made`,
|
||||
fix,
|
||||
);
|
||||
if (credits > limit)
|
||||
throw new Fix(
|
||||
`this call is about ${credits} credits, over --max-credits ${limit}; nothing was made or charged`,
|
||||
fix,
|
||||
);
|
||||
}
|
||||
@@ -1,72 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { findFFmpeg } from "../browser/ffmpeg.js";
|
||||
|
||||
vi.mock("../cloud/auth.js", () => ({
|
||||
resolveCloudAuthHeaders: async () => ({ "x-api-key": "k" }),
|
||||
refreshIfNeeded: async <T>(credential: T) => credential,
|
||||
resolveCloudBaseUrl: () => "https://api.test",
|
||||
}));
|
||||
vi.stubEnv("HEYGEN_API_KEY", "k");
|
||||
|
||||
const { transcribeWithHeygen, wordsOf } = await import("./transcribe.js");
|
||||
const ffmpeg = findFFmpeg();
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-heygen-asr-test-"));
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
const job = {
|
||||
id: "t1",
|
||||
status: "completed",
|
||||
model: "scribe-v2",
|
||||
language: "en",
|
||||
duration_seconds: 1.5,
|
||||
segments: [
|
||||
{
|
||||
speaker: "speaker_0",
|
||||
start: 0.1,
|
||||
end: 1.2,
|
||||
words: [
|
||||
{ word: "Hello", start: 0.1, end: 0.5 },
|
||||
{ word: "there", start: 0.6, end: 1.2, speaker: "speaker_1" },
|
||||
{ word: " ", start: 1.2, end: 1.3 },
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
describe("HeyGen transcription", () => {
|
||||
it("keeps each word's time and speaker, and drops blanks", () => {
|
||||
expect(wordsOf(job)).toEqual([
|
||||
{ text: "Hello", start: 0.1, end: 0.5, speaker: "speaker_0" },
|
||||
{ text: "there", start: 0.6, end: 1.2, speaker: "speaker_1" },
|
||||
]);
|
||||
});
|
||||
|
||||
it.skipIf(!ffmpeg)("sends the audio once and writes the transcript", async () => {
|
||||
const audio = join(dir, "a.wav");
|
||||
execFileSync(ffmpeg!, ["-v", "error", "-y", "-f", "lavfi", "-i", "sine=f=300:d=1.5", audio]);
|
||||
const bodies: Record<string, unknown>[] = [];
|
||||
vi.stubGlobal("fetch", async (_url: string, init: RequestInit) => {
|
||||
bodies.push(JSON.parse(String(init.body)));
|
||||
return new Response(JSON.stringify({ data: job }), { status: 200 });
|
||||
});
|
||||
const out = join(dir, "transcript.json");
|
||||
const r = await transcribeWithHeygen(audio, out, { diarize: true });
|
||||
expect(bodies).toHaveLength(1);
|
||||
expect(bodies[0]).toMatchObject({
|
||||
model: "scribe-v2",
|
||||
diarize: true,
|
||||
audio: { type: "base64", media_type: "audio/mpeg" },
|
||||
});
|
||||
expect(r).toMatchObject({
|
||||
detectedLanguage: "en",
|
||||
durationSeconds: 1.5,
|
||||
speechOnsetSeconds: 0.1,
|
||||
});
|
||||
expect(JSON.parse(readFileSync(out, "utf8"))).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
@@ -1,97 +0,0 @@
|
||||
// Transcription on HeyGen (`transcribe --engine heygen`): word times from its speech model, with speakers when asked.
|
||||
// The route runs inside the POST and takes an API key; the Idempotency-Key makes a re-run free.
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { runFfmpeg } from "../measure/media.js";
|
||||
import { idempotencyKey, runJob } from "./v3.js";
|
||||
|
||||
const ROUTE = "/v3/audio/transcriptions";
|
||||
const MODEL = "scribe-v2";
|
||||
|
||||
export interface HeygenTranscript {
|
||||
transcriptPath: string;
|
||||
model: string;
|
||||
detectedLanguage: string | null;
|
||||
durationSeconds: number;
|
||||
speechOnsetSeconds: number | null;
|
||||
}
|
||||
|
||||
const isRecord = (v: unknown): v is Record<string, unknown> =>
|
||||
typeof v === "object" && v !== null && !Array.isArray(v);
|
||||
const list = (v: unknown) => (Array.isArray(v) ? v.filter(isRecord) : []);
|
||||
|
||||
/** Words in the transcript file's own shape, `speaker` kept when the model labelled one. */
|
||||
export function wordsOf(job: Record<string, unknown>) {
|
||||
return list(job.segments).flatMap((segment) =>
|
||||
list(segment.words).flatMap((w) => {
|
||||
const text = String(w.word ?? w.text ?? "").trim();
|
||||
if (!text || typeof w.start !== "number" || typeof w.end !== "number") return [];
|
||||
const speaker = w.speaker ?? segment.speaker;
|
||||
return [
|
||||
{
|
||||
text,
|
||||
start: w.start,
|
||||
end: w.end,
|
||||
...(speaker !== undefined ? { speaker: String(speaker) } : {}),
|
||||
},
|
||||
];
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
export async function transcribeWithHeygen(
|
||||
inputPath: string,
|
||||
transcriptPath: string,
|
||||
opts: { diarize?: boolean; projectDir?: string; onStatus?: (message: string) => void } = {},
|
||||
): Promise<HeygenTranscript> {
|
||||
const work = mkdtempSync(join(tmpdir(), "hf-heygen-asr-"));
|
||||
try {
|
||||
opts.onStatus?.("Preparing audio for HeyGen...");
|
||||
const mp3 = join(work, "audio.mp3");
|
||||
runFfmpeg([
|
||||
"-y",
|
||||
"-i",
|
||||
inputPath,
|
||||
"-vn",
|
||||
"-ac",
|
||||
"1",
|
||||
"-ar",
|
||||
"16000",
|
||||
"-c:a",
|
||||
"libmp3lame",
|
||||
"-b:a",
|
||||
"64k",
|
||||
mp3,
|
||||
]);
|
||||
const audio = readFileSync(mp3);
|
||||
const diarize = opts.diarize === true;
|
||||
opts.onStatus?.("Transcribing on HeyGen...");
|
||||
const job = await runJob(
|
||||
ROUTE,
|
||||
{
|
||||
model: MODEL,
|
||||
audio: { type: "base64", media_type: "audio/mpeg", data: audio.toString("base64") },
|
||||
diarize,
|
||||
},
|
||||
idempotencyKey(ROUTE, [audio, String(diarize)]),
|
||||
(id) => `${ROUTE}/${encodeURIComponent(id)}`,
|
||||
{
|
||||
log: { projectDir: opts.projectDir, what: `transcribe ${inputPath}` },
|
||||
timeoutMs: 10 * 60_000,
|
||||
},
|
||||
);
|
||||
const words = wordsOf(job);
|
||||
writeFileSync(transcriptPath, JSON.stringify(words, null, 2));
|
||||
return {
|
||||
transcriptPath,
|
||||
model: typeof job.model === "string" ? job.model : MODEL,
|
||||
detectedLanguage: typeof job.language === "string" ? job.language : null,
|
||||
durationSeconds:
|
||||
typeof job.duration_seconds === "number" ? job.duration_seconds : (words.at(-1)?.end ?? 0),
|
||||
speechOnsetSeconds: words[0]?.start ?? null,
|
||||
};
|
||||
} finally {
|
||||
rmSync(work, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
@@ -1,298 +0,0 @@
|
||||
import { mkdtempSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("../cloud/auth.js", () => ({
|
||||
resolveCloudAuthHeaders: async () => ({ "x-api-key": "k" }),
|
||||
refreshIfNeeded: async <T>(credential: T) => credential,
|
||||
resolveCloudBaseUrl: () => "https://api.test",
|
||||
}));
|
||||
|
||||
const { idempotencyKey, runJob } = await import("./v3.js");
|
||||
|
||||
const reply = (status: number, body: unknown) =>
|
||||
new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } });
|
||||
|
||||
beforeEach(() => {
|
||||
vi.stubEnv("HYPERFRAMES_CONFIG_DIR", mkdtempSync(join(tmpdir(), "hf-v3-test-")));
|
||||
vi.stubEnv("HEYGEN_API_KEY", "payer");
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
describe("runJob", () => {
|
||||
it("sends the key, polls until the job ends and returns it", async () => {
|
||||
const calls: { url: string; init: RequestInit }[] = [];
|
||||
const answers = [
|
||||
reply(200, { data: { id: "j1", status: "pending" } }),
|
||||
reply(200, { data: { id: "j1", status: "processing" } }),
|
||||
reply(200, { data: { id: "j1", status: "completed", output: { json: { ok: 1 } } } }),
|
||||
];
|
||||
vi.stubGlobal("fetch", async (url: string, init: RequestInit) => {
|
||||
calls.push({ url, init });
|
||||
return answers.shift()!;
|
||||
});
|
||||
const job = await runJob("/v3/x", { a: 1 }, "hf-key", (id) => `/v3/x/${id}`, { intervalMs: 1 });
|
||||
expect(job.output).toEqual({ json: { ok: 1 } });
|
||||
expect(calls.map((c) => c.url)).toEqual([
|
||||
"https://api.test/v3/x",
|
||||
"https://api.test/v3/x/j1",
|
||||
"https://api.test/v3/x/j1",
|
||||
]);
|
||||
expect(new Headers(calls[0]!.init.headers).get("Idempotency-Key")).toBe("hf-key");
|
||||
});
|
||||
|
||||
it("turns refusals into what to do next", async () => {
|
||||
vi.stubGlobal("fetch", async () =>
|
||||
reply(402, { error: { code: "insufficient_credit", message: "no" } }),
|
||||
);
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id)).rejects.toMatchObject({
|
||||
fix: expect.stringContaining("top up"),
|
||||
});
|
||||
vi.stubGlobal("fetch", async () =>
|
||||
reply(402, { error: { code: "hf_subscription_required", message: "no plan" } }),
|
||||
);
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id)).rejects.toMatchObject({
|
||||
fix: "Subscribe: https://hyperframes.studio/pricing?from=cli",
|
||||
});
|
||||
vi.stubGlobal("fetch", async () =>
|
||||
reply(402, { code: "hf_usage_exhausted", url: "https://pay.test/credits" }),
|
||||
);
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id)).rejects.toMatchObject({
|
||||
fix: "Add credits: https://pay.test/credits",
|
||||
});
|
||||
vi.stubGlobal("fetch", async () => reply(401, { message: "bad" }));
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id)).rejects.toMatchObject({
|
||||
fix: "npx hyperframes auth login",
|
||||
});
|
||||
vi.stubGlobal("fetch", async () =>
|
||||
reply(200, { data: { id: "j", status: "failed", failure_code: "content_policy_violation" } }),
|
||||
);
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id)).rejects.toMatchObject({
|
||||
fix: expect.stringContaining("reword"),
|
||||
});
|
||||
});
|
||||
|
||||
it("names what an out-of-credits step needs, from the call's own price when HeyGen does not say", async () => {
|
||||
vi.stubGlobal("fetch", async () => reply(402, { error: { code: "hf_usage_exhausted" } }));
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id, { credits: 782 })).rejects.toMatchObject({
|
||||
message: expect.stringContaining("this step needs about 782"),
|
||||
});
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id, { credits: 782 })).rejects.toMatchObject({
|
||||
message: expect.stringContaining("billing top-up --needed 782"),
|
||||
fix: expect.stringContaining("needed_credits=782"),
|
||||
});
|
||||
vi.stubGlobal("fetch", async () =>
|
||||
reply(402, { error: { code: "hf_usage_exhausted", needed_credits: 180, balance: 30 } }),
|
||||
);
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id, { credits: 782 })).rejects.toMatchObject({
|
||||
message: expect.stringContaining("this step needs 180, balance 30"),
|
||||
});
|
||||
vi.stubGlobal("fetch", async () => reply(402, { error: { code: "hf_usage_exhausted" } }));
|
||||
await expect(runJob("/v3/x", {}, "k", (id) => id)).rejects.toMatchObject({
|
||||
message: expect.stringContaining("this step needs more"),
|
||||
});
|
||||
});
|
||||
|
||||
it("sends a failed or refused job again as a new one, and picks a finished one up", async () => {
|
||||
const keys: (string | null)[] = [];
|
||||
const run = (key: string, answers: Response[]) => {
|
||||
vi.stubGlobal("fetch", async (_url: string, init: RequestInit) => {
|
||||
if (init.method === "POST") keys.push(new Headers(init.headers).get("Idempotency-Key"));
|
||||
return answers.shift()!;
|
||||
});
|
||||
return runJob("/v3/x", {}, key, (id) => `/v3/x/${id}`, { intervalMs: 1 });
|
||||
};
|
||||
const failed = () => [
|
||||
reply(200, { data: { id: "j", status: "pending" } }),
|
||||
reply(200, { data: { id: "j", status: "failed", failure_code: "generation_failed" } }),
|
||||
];
|
||||
const done = () => [reply(200, { data: { id: "j", status: "completed" } })];
|
||||
|
||||
await expect(run("hf-fails", failed())).rejects.toMatchObject({
|
||||
fix: "fix the request it names, then run again",
|
||||
});
|
||||
await expect(run("hf-fails", failed())).rejects.toThrow("generation_failed");
|
||||
await run("hf-fails", done());
|
||||
await run("hf-fails", done());
|
||||
expect(keys).toEqual(["hf-fails", "hf-fails-1", "hf-fails-2", "hf-fails-2"]);
|
||||
|
||||
keys.length = 0;
|
||||
await run("hf-works", done());
|
||||
await run("hf-works", done());
|
||||
expect(keys).toEqual(["hf-works", "hf-works"]);
|
||||
|
||||
keys.length = 0;
|
||||
await expect(
|
||||
run("hf-broke", [reply(402, { error: { code: "hf_usage_exhausted" } })]),
|
||||
).rejects.toThrow("out of HyperFrames credits");
|
||||
await run("hf-broke", done());
|
||||
expect(keys).toEqual(["hf-broke", "hf-broke-1"]);
|
||||
});
|
||||
|
||||
it.each([401, 403, 408, 409, 425, 429])(
|
||||
"keeps the key after a %i, which says nothing about the job",
|
||||
async (status) => {
|
||||
const key = `hf-lost-${status}`;
|
||||
const keys: (string | null)[] = [];
|
||||
const answers = [
|
||||
reply(status, { message: "no" }),
|
||||
reply(200, { data: { id: "j", status: "completed" } }),
|
||||
];
|
||||
vi.stubGlobal("fetch", async (_url: string, init: RequestInit) => {
|
||||
keys.push(new Headers(init.headers).get("Idempotency-Key"));
|
||||
return answers.shift()!;
|
||||
});
|
||||
await expect(runJob("/v3/x", {}, key, (id) => id)).rejects.toThrow();
|
||||
await runJob("/v3/x", {}, key, (id) => id);
|
||||
expect(keys).toEqual([key, key]);
|
||||
},
|
||||
);
|
||||
|
||||
it("does not let one account's failure move another account's key", async () => {
|
||||
const keys: (string | null)[] = [];
|
||||
const run = (answers: Response[]) => {
|
||||
vi.stubGlobal("fetch", async (_url: string, init: RequestInit) => {
|
||||
keys.push(new Headers(init.headers).get("Idempotency-Key"));
|
||||
return answers.shift()!;
|
||||
});
|
||||
return runJob("/v3/x", {}, "hf-shared", (id) => id);
|
||||
};
|
||||
vi.stubEnv("HEYGEN_API_KEY", "payer-a");
|
||||
await expect(run([reply(402, { code: "hf_usage_exhausted" })])).rejects.toThrow();
|
||||
vi.stubEnv("HEYGEN_API_KEY", "payer-b");
|
||||
await run([reply(200, { data: { id: "j", status: "completed" } })]);
|
||||
expect(keys).toEqual(["hf-shared", "hf-shared"]);
|
||||
});
|
||||
|
||||
describe("with a host's sign-in token", () => {
|
||||
const signIn = (token: string) => {
|
||||
vi.stubEnv("HEYGEN_API_KEY", "");
|
||||
vi.stubEnv("HEYGEN_ACCESS_TOKEN", token);
|
||||
};
|
||||
const done = () => reply(200, { data: { id: "j", status: "completed" } });
|
||||
const failed = () =>
|
||||
reply(200, { data: { id: "j", status: "failed", failure_code: "generation_failed" } });
|
||||
/** Answers `/v3/users/me` with `accounts`, and each job call with the next of `answers`. */
|
||||
const server = (
|
||||
accounts: Record<string, Response | (() => Response)>,
|
||||
answers: (Response | (() => Response))[],
|
||||
) => {
|
||||
const keys: (string | null)[] = [];
|
||||
vi.stubGlobal("fetch", async (url: string, init: RequestInit) => {
|
||||
if (!url.endsWith("/v3/users/me")) {
|
||||
if (init.method === "POST") keys.push(new Headers(init.headers).get("Idempotency-Key"));
|
||||
const answer = answers.shift()!;
|
||||
return typeof answer === "function" ? answer() : answer;
|
||||
}
|
||||
const token = new Headers(init.headers).get("authorization")?.replace("Bearer ", "");
|
||||
const account = accounts[token!]!;
|
||||
return typeof account === "function" ? account() : account.clone();
|
||||
});
|
||||
return keys;
|
||||
};
|
||||
const accounts = {
|
||||
a: reply(200, { data: { username: "a", email: "same@test" } }),
|
||||
b: reply(200, { data: { username: "b", email: "same@test" } }),
|
||||
};
|
||||
|
||||
it.each([
|
||||
["no local user", null],
|
||||
["someone else's local user", { user: { email: "stale@test" } }],
|
||||
])("gives each account its own failures, with %s", async (_name, stored) => {
|
||||
if (stored)
|
||||
writeFileSync(
|
||||
join(process.env["HYPERFRAMES_CONFIG_DIR"]!, "credentials"),
|
||||
JSON.stringify(stored),
|
||||
);
|
||||
const keys = server(accounts, [
|
||||
done(),
|
||||
reply(402, { error: { code: "hf_usage_exhausted" } }),
|
||||
done(),
|
||||
]);
|
||||
signIn("a");
|
||||
await runJob("/v3/x", {}, "hf-host", (id) => id);
|
||||
signIn("b");
|
||||
await expect(runJob("/v3/x", {}, "hf-host", (id) => id)).rejects.toThrow();
|
||||
signIn("a");
|
||||
await runJob("/v3/x", {}, "hf-host", (id) => id);
|
||||
expect(keys).toEqual(["hf-host", "hf-host", "hf-host"]);
|
||||
});
|
||||
|
||||
it("holds a failure to the account that sent the job, whoever signs in during it", async () => {
|
||||
const keys = server(accounts, [
|
||||
done(),
|
||||
reply(200, { data: { id: "j", status: "pending" } }),
|
||||
() => {
|
||||
signIn("b");
|
||||
return failed();
|
||||
},
|
||||
done(),
|
||||
done(),
|
||||
]);
|
||||
signIn("b");
|
||||
await runJob("/v3/x", {}, "hf-switch", (id) => id);
|
||||
signIn("a");
|
||||
await expect(
|
||||
runJob("/v3/x", {}, "hf-switch", (id) => id, { intervalMs: 1 }),
|
||||
).rejects.toThrow();
|
||||
await runJob("/v3/x", {}, "hf-switch", (id) => id);
|
||||
signIn("a");
|
||||
await runJob("/v3/x", {}, "hf-switch", (id) => id);
|
||||
expect(keys).toEqual(["hf-switch", "hf-switch", "hf-switch", "hf-switch-1"]);
|
||||
});
|
||||
|
||||
it("sends the job with the credential it looked up", async () => {
|
||||
const seen: (string | null)[] = [];
|
||||
const me = accounts.a;
|
||||
vi.stubGlobal("fetch", async (url: string, init: RequestInit) => {
|
||||
if (url.endsWith("/v3/users/me")) {
|
||||
signIn("b");
|
||||
return me.clone();
|
||||
}
|
||||
seen.push(new Headers(init.headers).get("authorization"));
|
||||
return done();
|
||||
});
|
||||
signIn("a");
|
||||
await runJob("/v3/x", {}, "hf-window", (id) => id);
|
||||
expect(seen).toEqual(["Bearer a"]);
|
||||
});
|
||||
|
||||
it("records nothing when the account cannot be told", async () => {
|
||||
const keys = server({ a: () => reply(503, {}) }, [
|
||||
reply(402, { error: { code: "hf_usage_exhausted" } }),
|
||||
done(),
|
||||
]);
|
||||
signIn("a");
|
||||
await expect(runJob("/v3/x", {}, "hf-unknown", (id) => id)).rejects.toThrow();
|
||||
await runJob("/v3/x", {}, "hf-unknown", (id) => id);
|
||||
expect(keys).toEqual(["hf-unknown", "hf-unknown"]);
|
||||
});
|
||||
|
||||
it("sends a job that failed for this account as a new one", async () => {
|
||||
const keys = server(accounts, [failed(), done()]);
|
||||
signIn("a");
|
||||
await expect(runJob("/v3/x", {}, "hf-own", (id) => id)).rejects.toThrow();
|
||||
await runJob("/v3/x", {}, "hf-own", (id) => id);
|
||||
expect(keys).toEqual(["hf-own", "hf-own-1"]);
|
||||
});
|
||||
|
||||
it("keeps the key through a lost answer, then a new sign-in", async () => {
|
||||
const keys = server(accounts, [reply(401, { message: "expired" }), done()]);
|
||||
signIn("a");
|
||||
await expect(runJob("/v3/x", {}, "hf-resigned", (id) => id)).rejects.toThrow();
|
||||
await runJob("/v3/x", {}, "hf-resigned", (id) => id);
|
||||
expect(keys).toEqual(["hf-resigned", "hf-resigned"]);
|
||||
});
|
||||
});
|
||||
|
||||
it("keys the same request the same way", () => {
|
||||
expect(idempotencyKey("/v3/x", ["a", Buffer.from("b")])).toBe(
|
||||
idempotencyKey("/v3/x", ["a", Buffer.from("b")]),
|
||||
);
|
||||
expect(idempotencyKey("/v3/x", ["a"])).not.toBe(idempotencyKey("/v3/y", ["a"]));
|
||||
});
|
||||
});
|
||||
@@ -1,254 +0,0 @@
|
||||
// HeyGen's v3 job routes through the hyperframes credential chain (API key, host token or OAuth, refreshed). A paid
|
||||
// call carries an Idempotency-Key derived from what it asks for, so the same command again picks up the same job
|
||||
// rather than paying twice; a failed job is asked again. Every job is logged to .hyperframes/heygen-jobs.jsonl.
|
||||
import { createHash } from "node:crypto";
|
||||
import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { configDir } from "../auth/paths.js";
|
||||
import { AuthClient, buildAuthHeaders } from "../auth/client.js";
|
||||
import { resolveCredential } from "../auth/resolver.js";
|
||||
import { refreshIfNeeded, resolveCloudAuthHeaders, resolveCloudBaseUrl } from "../cloud/auth.js";
|
||||
import { Fix } from "../measure/media.js";
|
||||
import { pageUrl } from "./plan.js";
|
||||
|
||||
export interface Job {
|
||||
id: string;
|
||||
status: string;
|
||||
failure_code?: string;
|
||||
failure_message?: string;
|
||||
[field: string]: unknown;
|
||||
}
|
||||
|
||||
const RUNNING = new Set(["pending", "processing", "queued", "running", "waiting", "in_progress"]);
|
||||
const isRecord = (v: unknown): v is Record<string, unknown> =>
|
||||
typeof v === "object" && v !== null && !Array.isArray(v);
|
||||
|
||||
/** The same request always gets the same key: a re-run is a resume, never a second charge. */
|
||||
export function idempotencyKey(route: string, parts: readonly (string | Buffer)[]): string {
|
||||
const h = createHash("sha256").update(route);
|
||||
for (const p of parts) h.update(p).update("\0");
|
||||
return `hf-${h.digest("hex").slice(0, 40)}`;
|
||||
}
|
||||
|
||||
class Refused extends Fix {
|
||||
status = 0;
|
||||
}
|
||||
|
||||
/** Refusals that say nothing about the job: a lost answer may have started it, so a retry keeps its key. */
|
||||
const KEEPS_KEY = new Set([401, 403, 408, 409, 425, 429]);
|
||||
|
||||
const failedKeysFile = () => join(configDir(), "heygen-failed-keys");
|
||||
|
||||
const noSignIn = (): never => {
|
||||
throw new Fix("no HeyGen sign-in on this machine", "npx hyperframes auth login");
|
||||
};
|
||||
|
||||
interface Sender {
|
||||
auth: Record<string, string>;
|
||||
/** Who failures belong to: this API and this account; null when the account cannot be told. */
|
||||
scope: string | null;
|
||||
}
|
||||
|
||||
/** Fixed when a job starts: a sign-in during it must not send the job or own its failure. */
|
||||
async function jobSender(): Promise<Sender> {
|
||||
const credential = await resolveCredential().then(refreshIfNeeded).catch(noSignIn);
|
||||
const who =
|
||||
credential.type === "api_key"
|
||||
? credential.key
|
||||
: await new AuthClient().getCurrentUser(credential).then(
|
||||
(user) => user.username ?? user.email,
|
||||
() => undefined,
|
||||
);
|
||||
const scope = who
|
||||
? createHash("sha256").update(`${resolveCloudBaseUrl()}\0${who}`).digest("hex").slice(0, 16)
|
||||
: null;
|
||||
return { auth: buildAuthHeaders(credential), scope };
|
||||
}
|
||||
|
||||
function failedKeys(scope: string): Set<string> {
|
||||
const file = failedKeysFile();
|
||||
const lines = existsSync(file) ? readFileSync(file, "utf8").split("\n") : [];
|
||||
return new Set(
|
||||
lines.filter((l) => l.startsWith(`${scope} `)).map((l) => l.slice(scope.length + 1)),
|
||||
);
|
||||
}
|
||||
|
||||
function markFailed(key: string, scope: string | null): void {
|
||||
if (!scope) return;
|
||||
mkdirSync(configDir(), { recursive: true });
|
||||
appendFileSync(failedKeysFile(), `${scope} ${key}\n`);
|
||||
}
|
||||
|
||||
/** The key to send: `key` itself, or past every key whose job failed or was refused, so a retry is a new job. */
|
||||
function liveKey(key: string, scope: string | null): string {
|
||||
const failed = scope ? failedKeys(scope) : new Set<string>();
|
||||
let sent = key;
|
||||
for (let n = 1; failed.has(sent); n++) sent = `${key}-${n}`;
|
||||
return sent;
|
||||
}
|
||||
|
||||
/** A refused call says what to do: sign in, subscribe, add credits, or change the request. `estimate` is the
|
||||
* caller's own price for the call (what `--plan` prints), named when HeyGen does not say what the step needs. */
|
||||
// fallow-ignore-next-line complexity
|
||||
function refusal(status: number, body: unknown, estimate?: number): Refused {
|
||||
const err = isRecord(body) && isRecord(body.error) ? body.error : isRecord(body) ? body : {};
|
||||
const code = String(err.code ?? err.failure_code ?? "");
|
||||
const message = String(err.message ?? err.failure_message ?? "").slice(0, 300);
|
||||
const url = typeof err.url === "string" ? err.url : undefined;
|
||||
const given = Number(err.needed_credits) || undefined;
|
||||
const needed = given ?? (estimate || undefined);
|
||||
const needs = given ? String(given) : needed ? `about ${needed}` : "more";
|
||||
const balance = err.balance === undefined ? "" : `, balance ${String(err.balance)}`;
|
||||
// The fix is the one line the agent relays as printed; the problem says how to wait for the person.
|
||||
if (/hf_subscription_required/i.test(code))
|
||||
return new Refused(
|
||||
`this is a HyperFrames Pro feature; nothing was charged. Run \`npx hyperframes billing subscribe\`: it opens the page and waits`,
|
||||
`Subscribe: ${url ?? pageUrl("plans", { needed })}`,
|
||||
);
|
||||
if (/hf_usage_exhausted/i.test(code))
|
||||
return new Refused(
|
||||
`out of HyperFrames credits (this step needs ${needs}${balance}); nothing was charged. Run \`npx hyperframes billing top-up${needed ? ` --needed ${needed}` : ""}\`: it opens the page and waits`,
|
||||
`Add credits: ${url ?? pageUrl("credits", { needed })}`,
|
||||
);
|
||||
if (/api_key_required/i.test(code))
|
||||
return new Refused(
|
||||
"this HeyGen feature is not available with this sign-in yet; nothing was charged",
|
||||
"continue without this step",
|
||||
);
|
||||
if (status === 401 || status === 403)
|
||||
return new Refused(
|
||||
`HeyGen refused the credential (${status}${message ? `: ${message}` : ""})`,
|
||||
"npx hyperframes auth login",
|
||||
);
|
||||
if (status === 402 || /insufficient|credit|usage_exhausted|quota/i.test(code))
|
||||
return new Refused(
|
||||
`HeyGen says there is not enough credit (${code || status})`,
|
||||
"top up at https://app.heygen.com, or sign in with npx hyperframes auth login to use the plan's allowance",
|
||||
);
|
||||
if (/content_policy/i.test(code))
|
||||
return new Refused(
|
||||
`HeyGen declined the content: ${message || code}`,
|
||||
"reword the prompt once without the flagged part",
|
||||
);
|
||||
return new Refused(
|
||||
`HeyGen ${status}${code ? ` ${code}` : ""}: ${message || "request refused"}`,
|
||||
"fix the request it names, then run again",
|
||||
);
|
||||
}
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
async function call(
|
||||
method: "GET" | "POST",
|
||||
route: string,
|
||||
body?: unknown,
|
||||
key?: string,
|
||||
estimate?: number,
|
||||
auth?: Record<string, string>,
|
||||
): Promise<Job> {
|
||||
const headers: Record<string, string> = {
|
||||
...(auth ?? (await resolveCloudAuthHeaders().catch(noSignIn))),
|
||||
accept: "application/json",
|
||||
};
|
||||
if (body !== undefined) headers["content-type"] = "application/json";
|
||||
if (key) headers["Idempotency-Key"] = key;
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetch(`${resolveCloudBaseUrl()}${route}`, {
|
||||
method,
|
||||
headers,
|
||||
...(body !== undefined ? { body: JSON.stringify(body) } : {}),
|
||||
signal: AbortSignal.timeout(method === "POST" ? 180_000 : 30_000),
|
||||
});
|
||||
} catch (err) {
|
||||
throw new Fix(
|
||||
`no answer from HeyGen for ${route} (${err instanceof Error ? err.message : String(err)})`,
|
||||
"run the same command again: it resumes the same job",
|
||||
);
|
||||
}
|
||||
const value: unknown = await res.json().catch(() => null);
|
||||
if (res.status >= 500 || value === null)
|
||||
throw new Fix(
|
||||
`HeyGen ${res.status} for ${route} with no readable answer`,
|
||||
"run the same command again: it resumes the same job",
|
||||
);
|
||||
if (!res.ok) {
|
||||
const refused = refusal(res.status, value, estimate);
|
||||
refused.status = res.status;
|
||||
throw refused;
|
||||
}
|
||||
const data = isRecord(value) && isRecord(value.data) ? value.data : value;
|
||||
if (!isRecord(data))
|
||||
throw new Fix(`HeyGen answered ${route} with no job`, "run the same command again");
|
||||
const id = data.id ?? data.job_id ?? data.video_id ?? data.generation_id;
|
||||
return {
|
||||
...data,
|
||||
id: typeof id === "string" ? id : "",
|
||||
status: String(data.status ?? (id ? "pending" : "completed")),
|
||||
};
|
||||
}
|
||||
|
||||
export interface JobLog {
|
||||
/** The project whose .hyperframes/heygen-jobs.jsonl records the job; omitted, nothing is logged. */
|
||||
projectDir?: string;
|
||||
/** What the job is for, in a few words. */
|
||||
what: string;
|
||||
}
|
||||
|
||||
function log(entry: JobLog, record: Record<string, unknown>) {
|
||||
if (!entry.projectDir) return;
|
||||
const dir = join(entry.projectDir, ".hyperframes");
|
||||
mkdirSync(dir, { recursive: true });
|
||||
appendFileSync(
|
||||
join(dir, "heygen-jobs.jsonl"),
|
||||
`${JSON.stringify({ at: new Date().toISOString(), what: entry.what, ...record })}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
export interface RunOptions {
|
||||
/** How long to wait for the job to finish before handing back its id. */
|
||||
timeoutMs?: number;
|
||||
intervalMs?: number;
|
||||
onWait?: (job: Job) => void;
|
||||
log?: JobLog;
|
||||
/** What the call is priced at in credits, named in an out-of-credits refusal that does not say. */
|
||||
credits?: number;
|
||||
}
|
||||
|
||||
/** Starts a job (or picks up the one this key already started) and polls `pollRoute(id)` until it ends. */
|
||||
export async function runJob(
|
||||
route: string,
|
||||
body: unknown,
|
||||
key: string,
|
||||
pollRoute: (id: string) => string,
|
||||
opts: RunOptions = {},
|
||||
): Promise<Job> {
|
||||
const started = Date.now();
|
||||
const { auth, scope } = await jobSender();
|
||||
const sent = liveKey(key, scope);
|
||||
let job: Job;
|
||||
try {
|
||||
job = await call("POST", route, body, sent, opts.credits, auth);
|
||||
} catch (err) {
|
||||
if (err instanceof Refused && !KEEPS_KEY.has(err.status)) markFailed(sent, scope);
|
||||
throw err;
|
||||
}
|
||||
if (opts.log) log(opts.log, { route, id: job.id, key: sent, status: job.status });
|
||||
while (RUNNING.has(job.status.toLowerCase())) {
|
||||
if (Date.now() - started > (opts.timeoutMs ?? 15 * 60_000))
|
||||
throw new Fix(
|
||||
`HeyGen job ${job.id} is still ${job.status}`,
|
||||
"run the same command again later: it picks up this job",
|
||||
);
|
||||
opts.onWait?.(job);
|
||||
await new Promise((r) => setTimeout(r, opts.intervalMs ?? 5000));
|
||||
job = await call("GET", pollRoute(job.id));
|
||||
}
|
||||
if (opts.log) log(opts.log, { route, id: job.id, key: sent, status: job.status });
|
||||
if (!/^(completed|succeeded|success|done)$/i.test(job.status)) {
|
||||
markFailed(sent, scope);
|
||||
throw refusal(422, {
|
||||
error: { code: job.failure_code ?? job.status, message: job.failure_message ?? "" },
|
||||
});
|
||||
}
|
||||
return job;
|
||||
}
|
||||
@@ -1,89 +0,0 @@
|
||||
// What every `hyperframes measure <op>` shares: one JSON object on stdout (summaries, never per-frame arrays;
|
||||
// pictures as file paths), progress on stderr, and an error that says what to run instead.
|
||||
import { c } from "../ui/colors.js";
|
||||
import { failCommand } from "../utils/commandResult.js";
|
||||
import { normalizeErrorMessage } from "../utils/errorMessage.js";
|
||||
import { withMeta } from "../utils/updateCheck.js";
|
||||
import {
|
||||
Fix,
|
||||
formatJson,
|
||||
parseNumbers,
|
||||
parseRoi,
|
||||
probeVideo,
|
||||
type Roi,
|
||||
type VideoInfo,
|
||||
} from "./media.js";
|
||||
|
||||
export async function runMeasure(work: () => object | Promise<object>): Promise<void> {
|
||||
try {
|
||||
console.log(formatJson(withMeta({ ok: true, ...(await work()) })));
|
||||
} catch (err) {
|
||||
const fix = err instanceof Fix ? err.fix : undefined;
|
||||
console.log(
|
||||
formatJson(
|
||||
withMeta({ ok: false, error: normalizeErrorMessage(err), ...(fix ? { fix } : {}) }),
|
||||
),
|
||||
);
|
||||
if (process.stdout.isTTY) console.error(`${c.error("✗")} ${normalizeErrorMessage(err)}`);
|
||||
failCommand();
|
||||
}
|
||||
}
|
||||
|
||||
/** Arguments most ops take. */
|
||||
export const videoArg = {
|
||||
type: "positional",
|
||||
description: "The video file",
|
||||
required: true,
|
||||
} as const;
|
||||
export const roiArg = {
|
||||
type: "string",
|
||||
description: "Region x,y,w,h in the video's pixels (default: whole frame)",
|
||||
} as const;
|
||||
export const outArg = {
|
||||
type: "string",
|
||||
description: "Folder for pictures (default .hyperframes/measure/<op>)",
|
||||
} as const;
|
||||
export const jsonArg = {
|
||||
type: "boolean",
|
||||
description: "Accepted for symmetry; output is always JSON",
|
||||
default: false,
|
||||
} as const;
|
||||
/** Where a second video (your render) is measured, when it is measured the same way. */
|
||||
export const compareWindowArgs = {
|
||||
"compare-start": { type: "string", description: "Its window start (default: --start)" },
|
||||
"compare-roi": { type: "string", description: "Its region (default: --roi)" },
|
||||
} as const;
|
||||
|
||||
/** The video's probe and its ROI (whole frame by default). */
|
||||
export function videoAndRoi(video: unknown, roi: unknown): { info: VideoInfo; roi: Roi } {
|
||||
const info = probeVideo(String(video));
|
||||
return {
|
||||
info,
|
||||
roi:
|
||||
roi === undefined ? [0, 0, info.width, info.height] : parseRoi(roi, info.width, info.height),
|
||||
};
|
||||
}
|
||||
|
||||
/** One number from a flag, or `fallback` when it is absent. */
|
||||
export function num(value: unknown, flag: string, fallback: number): number;
|
||||
export function num(value: unknown, flag: string): number | undefined;
|
||||
export function num(value: unknown, flag: string, fallback?: number): number | undefined {
|
||||
return value === undefined ? fallback : parseNumbers(value, flag)[0];
|
||||
}
|
||||
|
||||
/** start,end seconds from --start/--end (or --from/--to), clamped to the video. */
|
||||
export function window(
|
||||
info: VideoInfo,
|
||||
start: unknown,
|
||||
end: unknown,
|
||||
flags = ["--start", "--end"],
|
||||
) {
|
||||
const a = num(start, flags[0]!, 0);
|
||||
const b = Math.min(info.duration, num(end, flags[1]!, info.duration));
|
||||
if (!(b > a))
|
||||
throw new Fix(
|
||||
`${flags[1]} ${b} is not after ${flags[0]} ${a}`,
|
||||
`${flags[0]} 0 ${flags[1]} ${Math.min(6, info.duration)}`,
|
||||
);
|
||||
return [a, b] as const;
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { compareEnergy } from "./energy.js";
|
||||
import { compareLook, type Look } from "./look.js";
|
||||
|
||||
const look = (over: Partial<Look>): Look => ({
|
||||
frames: 4,
|
||||
luma: 120,
|
||||
contrast: 40,
|
||||
saturation: 0.4,
|
||||
bw_share: 0,
|
||||
warmth: 0,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe("compareLook", () => {
|
||||
it("finds nothing to compare in grayscale or black against itself", () => {
|
||||
const grey = look({ saturation: 0, bw_share: 1 });
|
||||
expect(compareLook(grey, grey).saturation_ratio).toBe(1);
|
||||
expect(compareLook(grey, look({ saturation: 0.002 })).saturation_ratio).toBe(1);
|
||||
const black = look({ luma: 0, contrast: 0, saturation: 0 });
|
||||
expect(compareLook(black, black)).toMatchObject({ luma_ratio: 1, contrast_ratio: 1 });
|
||||
});
|
||||
|
||||
it("still sees colour or light where the reference has none", () => {
|
||||
expect(compareLook(look({ saturation: 0 }), look({})).saturation_ratio).toBeGreaterThan(100);
|
||||
expect(compareLook(look({}), look({ saturation: 0 })).saturation_ratio).toBe(0);
|
||||
expect(compareLook(look({ luma: 0 }), look({})).luma_ratio).toBeGreaterThan(100);
|
||||
});
|
||||
});
|
||||
|
||||
describe("compareEnergy", () => {
|
||||
const still = Array.from({ length: 12 }, () => 0);
|
||||
const moving = Array.from({ length: 12 }, (_, i) => 2 + (i % 3));
|
||||
|
||||
it("calls two still shots the same amount of movement", () => {
|
||||
expect(compareEnergy(still, still)).toMatchObject({ mean_ratio: 1, peak_ratio: 1 });
|
||||
expect(
|
||||
compareEnergy(
|
||||
still,
|
||||
still.map(() => 0.02),
|
||||
),
|
||||
).toMatchObject({ mean_ratio: 1 });
|
||||
});
|
||||
|
||||
it("still sees movement against a still reference, and the reverse", () => {
|
||||
expect(compareEnergy(still, moving).mean_ratio).toBeGreaterThan(100);
|
||||
expect(compareEnergy(moving, still).mean_ratio).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -1,43 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, describe, expect, it } from "vitest";
|
||||
import { findFFmpeg, findFFprobe } from "../browser/ffmpeg.js";
|
||||
import { compareCurves, curves } from "./curve.js";
|
||||
|
||||
const ffmpeg = findFFmpeg();
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-curve-test-"));
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
|
||||
/** A white box sliding 0.5 s with back.out(1.7) from x=20 to x=220, starting at 0.3 s + `shift`. */
|
||||
function slide(name: string, shift: number): string {
|
||||
const q = `min(max((t-0.3)/0.5,0),1)-1`;
|
||||
const x = `20+200*(1+2.7*pow(${q},3)+1.7*pow(${q},2))`.replaceAll("t", `(t-${shift})`);
|
||||
const out = join(dir, name);
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-nostdin", "-v", "error", "-y"],
|
||||
...["-f", "lavfi", "-i", "color=black:s=320x120:d=1.2:r=30"],
|
||||
...["-f", "lavfi", "-i", "color=white:s=40x40:d=1.2:r=30"],
|
||||
...["-filter_complex", `[0][1]overlay=x='${x}':y=40`],
|
||||
...["-pix_fmt", "yuv444p", "-c:v", "libx264", "-crf", "0", out],
|
||||
]);
|
||||
return out;
|
||||
}
|
||||
|
||||
describe.skipIf(!ffmpeg || !findFFprobe())("curves", () => {
|
||||
it("reads a back.out slide's ease, length and overshoot, and pairs it with a later copy", () => {
|
||||
const a = curves(slide("a.mp4", 0), 0, 1.2, [0, 0, 320, 120]);
|
||||
const b = curves(slide("b.mp4", 0.1), 0, 1.2, [0, 0, 320, 120]);
|
||||
expect(a.moves).toHaveLength(1);
|
||||
const x = a.moves[0];
|
||||
if (!x || x.pulse) throw new Error("expected one fitted move");
|
||||
expect(x.prop).toBe("x");
|
||||
expect(x.ease).toMatch(/^back\.out\(1\.[5-9]\)/);
|
||||
expect(Math.abs(x.duration_s - 0.5)).toBeLessThanOrEqual(0.07);
|
||||
expect(Math.abs(x.overshoot_pct - 10)).toBeLessThanOrEqual(3);
|
||||
const c = compareCurves(a, b).find((m) => m.prop === "x")!;
|
||||
expect(Math.abs(c.start_offset_s! - 0.1)).toBeLessThanOrEqual(0.04);
|
||||
expect(c.curve_rmse!).toBeLessThan(0.05);
|
||||
});
|
||||
});
|
||||
@@ -1,303 +0,0 @@
|
||||
// The ease of one graphic move as numbers, ported from the Studio's motion-curve.mjs: inside an ROI, the element's
|
||||
// box (centre, width, height) and its ink (mean difference from the background) on every frame, then for each
|
||||
// property that changes, when the move starts and settles, the overshoot past its end value and the closest GSAP
|
||||
// ease. Compared with a second window, a spring, a back-out or a slow expo is checked rather than eyeballed.
|
||||
import { Fix, probeVideo, readFrames, round, type Roi } from "./media.js";
|
||||
|
||||
const W = 240;
|
||||
export type Prop = "x" | "y" | "w" | "h" | "ink";
|
||||
const PROPS: Prop[] = ["x", "y", "w", "h", "ink"];
|
||||
|
||||
/** One frame of an element: its box centre and size (video pixels) and ink; x and y are null when it is absent. */
|
||||
interface Row {
|
||||
t: number;
|
||||
x: number | null;
|
||||
y: number | null;
|
||||
w: number;
|
||||
h: number;
|
||||
ink: number;
|
||||
}
|
||||
|
||||
type Ease = [name: string, f: (u: number) => number];
|
||||
const pw = (k: number): Ease => [`power${k}.in`, (u) => u ** (k + 1)];
|
||||
const pwo = (k: number): Ease => [`power${k}.out`, (u) => 1 - (1 - u) ** (k + 1)];
|
||||
const pwio = (k: number): Ease => [
|
||||
`power${k}.inOut`,
|
||||
(u) => (u < 0.5 ? 2 ** k * u ** (k + 1) : 1 - (-2 * u + 2) ** (k + 1) / 2),
|
||||
];
|
||||
const back = (s: number): Ease => [
|
||||
`back.out(${s.toFixed(1)})`,
|
||||
(u) => 1 + (s + 1) * (u - 1) ** 3 + s * (u - 1) ** 2,
|
||||
];
|
||||
const elastic = (p: number): Ease => [
|
||||
`elastic.out(1,${p})`,
|
||||
(u) => (u <= 0 ? 0 : u >= 1 ? 1 : 2 ** (-10 * u) * Math.sin(((u - p / 4) * 2 * Math.PI) / p) + 1),
|
||||
];
|
||||
/** GSAP's named eases as functions of normalized time. */
|
||||
const EASES: Ease[] = [
|
||||
["none", (u) => u],
|
||||
...[1, 2, 3, 4].flatMap((k) => [pw(k), pwo(k), pwio(k)]),
|
||||
["sine.out", (u) => Math.sin((u * Math.PI) / 2)],
|
||||
["sine.inOut", (u) => -(Math.cos(Math.PI * u) - 1) / 2],
|
||||
["expo.out", (u) => (u >= 1 ? 1 : 1 - 2 ** (-10 * u))],
|
||||
["circ.out", (u) => Math.sqrt(1 - (u - 1) ** 2)],
|
||||
...Array.from({ length: 26 }, (_, i) => back(0.5 + i * 0.1)),
|
||||
...[0.2, 0.25, 0.3, 0.4, 0.5].map(elastic),
|
||||
];
|
||||
|
||||
export interface FittedMove {
|
||||
prop: Prop;
|
||||
pulse?: undefined;
|
||||
from: number;
|
||||
to: number;
|
||||
start_s: number;
|
||||
duration_s: number;
|
||||
frames: number;
|
||||
overshoot_pct: number;
|
||||
anticipation_pct: number;
|
||||
ease: string;
|
||||
rmse: number;
|
||||
next: string[];
|
||||
/** 21 samples of normalized progress, for comparing two moves' shapes. */
|
||||
curve: number[];
|
||||
}
|
||||
|
||||
/** A stretch that returns to where it began: its peak and length, no ease. */
|
||||
export interface PulseMove {
|
||||
prop: Prop;
|
||||
pulse: true;
|
||||
from: number;
|
||||
peak: number;
|
||||
start_s: number;
|
||||
duration_s: number;
|
||||
frames: number;
|
||||
}
|
||||
|
||||
export type Move = FittedMove | PulseMove;
|
||||
|
||||
/** Box and ink per frame of `roi`, against a flat background (the ROI border's median on the first frame, or `bg`). */
|
||||
// fallow-ignore-next-line complexity
|
||||
function track(
|
||||
path: string,
|
||||
start: number,
|
||||
seconds: number,
|
||||
roi: Roi,
|
||||
{ bg = null, thr = 24 }: { bg?: number[] | null; thr?: number } = {},
|
||||
): { fps: number; bg: number[]; rows: Row[] } {
|
||||
const fps = probeVideo(path).fps;
|
||||
const width = Math.min(W, roi[2]);
|
||||
const { frames: fs, h } = readFrames(path, { start, seconds, roi, width });
|
||||
const first = fs[0];
|
||||
if (!first)
|
||||
throw new Fix(
|
||||
`no frames between ${start}s and ${start + seconds}s`,
|
||||
"pick a window inside the video",
|
||||
);
|
||||
const k = roi[2] / width;
|
||||
let ref = bg;
|
||||
if (!ref) {
|
||||
const border: number[][] = [[], [], []];
|
||||
for (let y = 0; y < h; y++)
|
||||
for (let x = 0; x < width; x++)
|
||||
if (x < 2 || y < 2 || x >= width - 2 || y >= h - 2)
|
||||
for (let c = 0; c < 3; c++) border[c]!.push(first[(y * width + x) * 3 + c]!);
|
||||
ref = border.map((v) => v.sort((a, b) => a - b)[v.length >> 1]!);
|
||||
}
|
||||
const [r0, g0, b0] = [ref[0]!, ref[1]!, ref[2]!];
|
||||
const rows = fs.map((f, i): Row => {
|
||||
const xs: number[] = [];
|
||||
const ys: number[] = [];
|
||||
let ink = 0;
|
||||
for (let y = 0; y < h; y++)
|
||||
for (let x = 0; x < width; x++) {
|
||||
const j = (y * width + x) * 3;
|
||||
const d = Math.max(
|
||||
Math.abs(f[j]! - r0),
|
||||
Math.abs(f[j + 1]! - g0),
|
||||
Math.abs(f[j + 2]! - b0),
|
||||
);
|
||||
ink += d;
|
||||
if (d > thr) {
|
||||
xs.push(x);
|
||||
ys.push(y);
|
||||
}
|
||||
}
|
||||
const t = round(start + i / fps, 3);
|
||||
if (xs.length < 4) return { t, x: null, y: null, w: 0, h: 0, ink: round(ink / (width * h), 2) };
|
||||
xs.sort((a, b) => a - b);
|
||||
ys.sort((a, b) => a - b);
|
||||
const q = (v: number[], p: number) => v[Math.min(v.length - 1, Math.floor(v.length * p))]!;
|
||||
const [x0, x1, y0, y1] = [q(xs, 0.01), q(xs, 0.99), q(ys, 0.01), q(ys, 0.99)];
|
||||
return {
|
||||
t,
|
||||
x: round(roi[0] + ((x0 + x1 + 1) / 2) * k, 1),
|
||||
y: round(roi[1] + ((y0 + y1 + 1) / 2) * k, 1),
|
||||
w: round((x1 - x0 + 1) * k, 1),
|
||||
h: round((y1 - y0 + 1) * k, 1),
|
||||
ink: round(ink / (width * h), 2),
|
||||
};
|
||||
});
|
||||
return { fps, bg: ref, rows };
|
||||
}
|
||||
|
||||
/** Fit one stretch of values from its start value to its settled end value. */
|
||||
function fit(
|
||||
rows: readonly Row[],
|
||||
idx: number[],
|
||||
v: number[],
|
||||
prop: Prop,
|
||||
fps: number,
|
||||
a: number,
|
||||
b: number,
|
||||
): FittedMove {
|
||||
const [v0, v1] = [v[a]!, v[b]!];
|
||||
const span = v1 - v0;
|
||||
const p = v.slice(a, b + 1).map((x) => (x - v0) / span);
|
||||
const u = p.map((_, i) => i / Math.max(1, p.length - 1));
|
||||
const fits = EASES.map(([name, f]) => ({
|
||||
name,
|
||||
rmse: Math.sqrt(p.reduce((s, x, i) => s + (x - f(u[i]!)) ** 2, 0) / p.length),
|
||||
})).sort((m, n) => m.rmse - n.rmse);
|
||||
return {
|
||||
prop,
|
||||
from: round(v0, 1),
|
||||
to: round(v1, 1),
|
||||
start_s: rows[idx[a]!]!.t,
|
||||
duration_s: round((idx[b]! - idx[a]!) / fps, 3),
|
||||
frames: idx[b]! - idx[a]!,
|
||||
overshoot_pct: round(Math.max(0, Math.max(...p) - 1) * 100, 1),
|
||||
anticipation_pct: round(Math.max(0, -Math.min(...p)) * 100, 1),
|
||||
ease: fits[0]!.name,
|
||||
rmse: round(fits[0]!.rmse, 3),
|
||||
next: fits.slice(1, 3).map((f) => `${f.name} ${round(f.rmse, 3)}`),
|
||||
curve: Array.from({ length: 21 }, (_, i) =>
|
||||
round(p[Math.round((i / 20) * (p.length - 1))]!, 3),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/** Every move of one property: stretches of change separated by holds of 0.15 s or more (shorter stills are the
|
||||
* turn at an overshoot's peak). A stretch that returns to where it began is a pulse (peak and length, no ease). */
|
||||
// fallow-ignore-next-line complexity
|
||||
function fitMoves(
|
||||
rows: readonly Row[],
|
||||
prop: Prop,
|
||||
fps: number,
|
||||
minChange: number,
|
||||
quant: number,
|
||||
): Move[] {
|
||||
const idx = rows.flatMap((r, i) => (r[prop] == null ? [] : [i]));
|
||||
const v = idx.map((i) => rows[i]![prop] ?? 0);
|
||||
const out: Move[] = [];
|
||||
for (let i = 0; i < v.length - 1; ) {
|
||||
if (Math.abs(v[i + 1]! - v[i]!) <= quant) {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
const hold = Math.max(3, Math.round(0.15 * fps));
|
||||
let [b, still] = [i + 1, 0];
|
||||
for (let j = i + 1; j < v.length - 1 && still < hold; j++)
|
||||
if (Math.abs(v[j + 1]! - v[j]!) <= quant) still++;
|
||||
else [still, b] = [0, j + 1];
|
||||
const span = v[b]! - v[i]!;
|
||||
const peak = v
|
||||
.slice(i, b + 1)
|
||||
.reduce((m, x) => (Math.abs(x - v[i]!) > Math.abs(m - v[i]!) ? x : m), v[i]!);
|
||||
// a creep of under 3% of the move before it really starts is compression noise, not the move
|
||||
let a = i;
|
||||
while (a < b - 2 && Math.abs(v[a + 1]! - v[a]!) < 0.03 * Math.abs(span)) a++;
|
||||
if (Math.abs(span) >= minChange) out.push(fit(rows, idx, v, prop, fps, a, b));
|
||||
else if (Math.abs(peak - v[i]!) >= minChange)
|
||||
out.push({
|
||||
prop,
|
||||
pulse: true,
|
||||
from: round(v[i]!, 1),
|
||||
peak: round(peak, 1),
|
||||
start_s: rows[idx[i]!]!.t,
|
||||
duration_s: round((idx[b]! - idx[i]!) / fps, 3),
|
||||
frames: idx[b]! - idx[i]!,
|
||||
});
|
||||
i = b;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export interface Curves {
|
||||
start: number;
|
||||
seconds: number;
|
||||
roi: Roi;
|
||||
fps: number;
|
||||
bg: number[];
|
||||
moves: Move[];
|
||||
}
|
||||
|
||||
/** Every move of the element in `roi`, sorted by start. */
|
||||
export function curves(
|
||||
path: string,
|
||||
start: number,
|
||||
seconds: number,
|
||||
roi: Roi,
|
||||
opts: { bg?: number[] | null; thr?: number } = {},
|
||||
): Curves {
|
||||
const { fps, bg, rows } = track(path, start, seconds, roi, opts);
|
||||
const minPx = 0.02 * Math.max(roi[2], roi[3]);
|
||||
const step = roi[2] / Math.min(W, roi[2]);
|
||||
const moves = PROPS.flatMap((k) =>
|
||||
k === "ink" ? fitMoves(rows, k, fps, 2, 0.5) : fitMoves(rows, k, fps, minPx, step),
|
||||
).sort((m, n) => m.start_s - n.start_s);
|
||||
return { start, seconds, roi, fps: round(fps, 2), bg, moves };
|
||||
}
|
||||
|
||||
export interface MoveComparison {
|
||||
prop: Prop;
|
||||
at_s: number;
|
||||
yours?: "no matching move";
|
||||
start_offset_s?: number;
|
||||
duration_frames?: string;
|
||||
pulse?: true;
|
||||
peak?: string;
|
||||
ease?: string;
|
||||
same_family?: boolean;
|
||||
overshoot_pct?: string;
|
||||
curve_rmse?: number;
|
||||
}
|
||||
|
||||
/** Pair each reference move with your move of the same property that starts closest on the window's clock. */
|
||||
export function compareCurves(ref: Curves, yours: Curves): MoveComparison[] {
|
||||
const family = (s: string) => s.replace(/\(.*\)/, "").replace(/\d/, "");
|
||||
return ref.moves.map((m) => {
|
||||
const rel = m.start_s - ref.start;
|
||||
const d = yours.moves
|
||||
.filter((x) => x.prop === m.prop && !!x.pulse === !!m.pulse)
|
||||
.sort(
|
||||
(x, y) => Math.abs(x.start_s - yours.start - rel) - Math.abs(y.start_s - yours.start - rel),
|
||||
)[0];
|
||||
const head = { prop: m.prop, at_s: round(rel, 2) };
|
||||
if (!d) return { ...head, yours: "no matching move" };
|
||||
const common = {
|
||||
...head,
|
||||
start_offset_s: round(d.start_s - yours.start - rel, 3),
|
||||
duration_frames: `${m.frames} → ${d.frames}`,
|
||||
};
|
||||
if (m.pulse || d.pulse)
|
||||
return {
|
||||
...common,
|
||||
pulse: true,
|
||||
peak: `${m.pulse ? m.peak : m.to} → ${d.pulse ? d.peak : d.to}`,
|
||||
};
|
||||
return {
|
||||
...common,
|
||||
ease: `${m.ease} → ${d.ease}`,
|
||||
same_family: family(m.ease) === family(d.ease),
|
||||
overshoot_pct: `${m.overshoot_pct} → ${d.overshoot_pct}`,
|
||||
curve_rmse: round(
|
||||
Math.sqrt(m.curve.reduce((s, x, i) => s + (x - d.curve[i]!) ** 2, 0) / m.curve.length),
|
||||
3,
|
||||
),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** Moves as printed: the shape samples stay internal. */
|
||||
export const printable = (moves: Move[]) =>
|
||||
moves.map((m) => (m.pulse ? m : { ...m, curve: undefined }));
|
||||
@@ -1,52 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { confirmsCut, cutsIn, findCuts, SIG_H, SIG_W } from "./cuts.js";
|
||||
|
||||
/** Frame changes: `base` everywhere, with the given changes at their frames. */
|
||||
const changes = (n: number, base: number, at: Record<number, number>) =>
|
||||
Array.from({ length: n }, (_, i) => (i === 0 ? 0 : (at[i] ?? base)));
|
||||
|
||||
describe("cutsIn", () => {
|
||||
it("finds a hard cut and a two-frame whip as one cut", () => {
|
||||
expect(cutsIn(changes(60, 1, { 20: 40, 40: 30, 41: 25 }))).toEqual([
|
||||
{ frame: 20, frames: 1, change: 40 },
|
||||
{ frame: 40, frames: 2, change: 30 },
|
||||
]);
|
||||
});
|
||||
|
||||
it("finds a soft same-angle punch-in that nothing near it rivals", () => {
|
||||
// dan 9.743 s: 11.3 grey levels against neighbours of about 1.
|
||||
expect(cutsIn(changes(60, 1, { 30: 11.3 }))).toEqual([
|
||||
{ frame: 30, frames: 1, change: 11.3, soft: true },
|
||||
]);
|
||||
});
|
||||
|
||||
it("leaves a smaller change among moving frames or a stepped animation alone", () => {
|
||||
// Hands moving: neighbours up to 5, so 11 is not alone.
|
||||
expect(cutsIn(changes(60, 1, { 25: 5, 30: 11 }))).toEqual([]);
|
||||
// A 15 fps animation in a 30 fps file changes every other frame by about as much.
|
||||
const stepped = Array.from({ length: 60 }, (_, i) => (i % 2 ? 0 : 10.5));
|
||||
expect(cutsIn(stepped)).toEqual([]);
|
||||
expect(cutsIn(changes(60, 1, { 30: 9 }))).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("confirmsCut", () => {
|
||||
it("matches a soft cut with a clear change within a frame on the other side", () => {
|
||||
const cut = { frame: 30, frames: 1, change: 11.3, soft: true as const };
|
||||
expect(confirmsCut(changes(60, 1, { 31: 7 }), cut)).toBe(true);
|
||||
expect(confirmsCut(changes(60, 1, { 32: 7 }), cut)).toBe(false);
|
||||
expect(confirmsCut(changes(60, 1, { 30: 4 }), cut)).toBe(false);
|
||||
expect(confirmsCut(changes(60, 2, { 30: 7 }), cut)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("findCuts", () => {
|
||||
it("reads changes from grey frames", () => {
|
||||
const frame = (v: number) => new Uint8Array(SIG_W * SIG_H).fill(v);
|
||||
const frames = [
|
||||
...Array.from({ length: 20 }, () => frame(100)),
|
||||
...Array.from({ length: 20 }, () => frame(140)),
|
||||
];
|
||||
expect(findCuts(frames)).toEqual([{ frame: 20, frames: 1, change: 40 }]);
|
||||
});
|
||||
});
|
||||
@@ -1,83 +0,0 @@
|
||||
// Cuts from the frame-to-frame difference, ported from the Studio's videoShots.mjs; ffmpeg's scene score misses some.
|
||||
// A punch-in at the same angle changes less than a cut, so a smaller change that stands alone is a soft cut.
|
||||
import { decode, meanAbsDiff, type Frames } from "./media.js";
|
||||
|
||||
export const SIG_W = 64;
|
||||
export const SIG_H = 36;
|
||||
const CUT_LEVEL = 12;
|
||||
const CUT_RATIO = 4;
|
||||
const SOFT_LEVEL = 10;
|
||||
const SOFT_RATIO = 8;
|
||||
const SOFT_ALONE = 4;
|
||||
const CONFIRM_LEVEL = 6;
|
||||
const CONFIRM_SHARE = 0.5;
|
||||
const NEIGHBOURS = 12;
|
||||
|
||||
export interface Cut {
|
||||
frame: number;
|
||||
/** A whip or a two-frame dissolve changes on consecutive frames: one cut this many frames long. */
|
||||
frames: number;
|
||||
/** Its largest change, in mean grey levels. */
|
||||
change: number;
|
||||
soft?: true;
|
||||
}
|
||||
|
||||
/** Every frame of ffmpeg's `input` (after `filters`) as SIG_W x SIG_H grey levels. */
|
||||
export function signatures(input: string[], filters: string[] = []): Frames {
|
||||
return decode(input, [...filters, `scale=${SIG_W}:${SIG_H}:flags=area`], SIG_W, SIG_H, true);
|
||||
}
|
||||
|
||||
const median = (values: number[]) => {
|
||||
const sorted = [...values].sort((x, y) => x - y);
|
||||
return sorted[Math.floor(sorted.length / 2)] ?? 0;
|
||||
};
|
||||
|
||||
export function frameChanges(frames: Uint8Array[]): number[] {
|
||||
return frames.map((f, i) => (i ? meanAbsDiff(frames[i - 1]!, f) : 0));
|
||||
}
|
||||
|
||||
const neighbours = (diffs: number[], i: number) =>
|
||||
diffs.slice(Math.max(1, i - NEIGHBOURS), i).concat(diffs.slice(i + 1, i + NEIGHBOURS + 1));
|
||||
|
||||
export function cutsIn(diffs: number[]): Cut[] {
|
||||
const hits = diffs.flatMap((d, i): { frame: number; change: number; soft?: true }[] => {
|
||||
if (i === 0) return [];
|
||||
const around = neighbours(diffs, i);
|
||||
const typical = Math.max(1, median(around));
|
||||
if (d >= CUT_LEVEL && d >= CUT_RATIO * typical) return [{ frame: i, change: d }];
|
||||
const alone = d >= SOFT_ALONE * Math.max(0, ...around);
|
||||
return d >= SOFT_LEVEL && d >= SOFT_RATIO * typical && alone
|
||||
? [{ frame: i, change: d, soft: true }]
|
||||
: [];
|
||||
});
|
||||
const out: Cut[] = [];
|
||||
for (const { soft, ...h } of hits) {
|
||||
const last = out.at(-1);
|
||||
if (last && h.frame === last.frame + last.frames) {
|
||||
last.frames++;
|
||||
last.change = Math.max(last.change, h.change);
|
||||
if (!soft) delete last.soft;
|
||||
} else out.push({ ...h, frames: 1, ...(soft ? { soft } : {}) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The frame indexes where a new shot starts, each with its change and length. */
|
||||
export function findCuts(frames: Uint8Array[]): Cut[] {
|
||||
return cutsIn(frameChanges(frames));
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether these changes show a clear change within a frame of `frame`, though maybe under a cut's thresholds: a
|
||||
* soft cut found on the other side counts as matched by it, as a soft cut is a change near the line.
|
||||
*/
|
||||
export function confirmsCut(diffs: number[], cut: Cut): boolean {
|
||||
return [cut.frame - 1, cut.frame, cut.frame + 1].some((i) => {
|
||||
const d = diffs[i];
|
||||
if (i < 1 || d === undefined) return false;
|
||||
return (
|
||||
d >= Math.max(CONFIRM_LEVEL, CONFIRM_SHARE * cut.change) &&
|
||||
d >= CUT_RATIO * Math.max(1, median(neighbours(diffs, i)))
|
||||
);
|
||||
});
|
||||
}
|
||||
@@ -1,257 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { soundOffset } from "./diff.js";
|
||||
import {
|
||||
clipFolder,
|
||||
diffVerdicts,
|
||||
ffmpeg,
|
||||
loadOpenCvFirst,
|
||||
noTools,
|
||||
REF_ARGS,
|
||||
} from "./diff.test-helpers.js";
|
||||
import { probeVideo } from "./media.js";
|
||||
import { parseSegments } from "./segments.js";
|
||||
|
||||
const { dir, clip } = clipFolder("hf-diff-person-test-");
|
||||
|
||||
describe.skipIf(noTools)("diffVideos: copies, sound and re-shot people", () => {
|
||||
loadOpenCvFirst();
|
||||
const ref = noTools ? "" : clip("ref.mp4", REF_ARGS);
|
||||
|
||||
// The ledger cuts the original's whole frame for 0–3 s. Two comparisons per test, so none nears the per-test
|
||||
// limit under the flake gate's contention.
|
||||
const ledger = join(dir, "manifest.jsonl");
|
||||
writeFileSync(
|
||||
ledger,
|
||||
JSON.stringify({
|
||||
type: "video",
|
||||
path: "assets/all.mp4",
|
||||
provenance: { provider: "original", window: [0, 3], roi: [0, 0, 320, 180] },
|
||||
}) + "\n",
|
||||
);
|
||||
const titled = '## S01 · 0–3 s · Title\nPixels: rebuilt\nCopy: "TITLE"\n';
|
||||
const reuse = (yours: string, md: string, name: string) =>
|
||||
diffVerdicts({
|
||||
reference: ref,
|
||||
yours,
|
||||
segments: parseSegments(md),
|
||||
ledger,
|
||||
outDir: join(dir, name),
|
||||
});
|
||||
|
||||
it("fails the original's whole frame played where copy is meant to be rebuilt", async () => {
|
||||
const copy = await reuse(ref, titled, "copy");
|
||||
expect(copy.whole_frame_reuse).toMatchObject({ spans: [[0, 3]], drawn: [[0, 3]], pass: false });
|
||||
expect(copy.segments?.[0]?.pass).toBe(false);
|
||||
expect(copy.pass).toBe(false);
|
||||
// Footage reused with its burned-in title is a copy too (Miao, 2026-10-23: no whole-frame cut with drawn things).
|
||||
const burned = await reuse(
|
||||
ref,
|
||||
'## S01 · 0–3 s · Title over footage\nPixels: reused\nCopy: "TITLE"\n',
|
||||
"burned",
|
||||
);
|
||||
expect(burned.whole_frame_reuse).toMatchObject({ share: 1, drawn: [[0, 3]], pass: false });
|
||||
});
|
||||
|
||||
it("passes footage with nothing drawn on it cut whole", async () => {
|
||||
const footage = await reuse(ref, "## S01 · 0–3 s · Footage\nPixels: reused\n", "footage");
|
||||
expect(footage.whole_frame_reuse).toMatchObject({ share: 1, drawn: [], pass: true });
|
||||
expect(footage.pass).toBe(true);
|
||||
});
|
||||
|
||||
it("does not count a cut that only fed a generation as reuse", async () => {
|
||||
// A motion guide is not played: yours differs from it.
|
||||
const flipped = clip("guided.mp4", [...["-i", ref, "-vf"], "hflip"]);
|
||||
const guided = await reuse(flipped, titled, "guided");
|
||||
expect(guided.whole_frame_reuse).toMatchObject({ spans: [], pass: true });
|
||||
// A generated shot that starts from the original's first frame matches it for a few frames only: not reuse.
|
||||
const started = clip("started.mp4", [
|
||||
...["-i", ref, "-filter_complex"],
|
||||
"[0]split[a][b];[b]hflip[f];[a][f]overlay=enable='gte(t\\,0.2)'",
|
||||
]);
|
||||
const start = await reuse(started, titled, "started");
|
||||
expect(start.whole_frame_reuse).toMatchObject({ spans: [], pass: true });
|
||||
});
|
||||
|
||||
it("finds a whole-frame still of the original shown in yours", async () => {
|
||||
const ledger = join(dir, "stills.jsonl");
|
||||
writeFileSync(
|
||||
ledger,
|
||||
[
|
||||
{ type: "image", provenance: { provider: "original", at: 1, roi: [0, 0, 320, 180] } },
|
||||
{ type: "image", provenance: { provider: "original", at: 2.5 } },
|
||||
]
|
||||
.map((e) => JSON.stringify(e))
|
||||
.join("\n"),
|
||||
);
|
||||
// Yours holds the original's frame at 1 s for the whole clip.
|
||||
const held = clip("held.mp4", [
|
||||
...["-i", ref, "-vf"],
|
||||
"select=eq(n\\,30),loop=loop=89:size=1,setpts=N/30/TB",
|
||||
]);
|
||||
const r = await diffVerdicts({
|
||||
reference: ref,
|
||||
yours: held,
|
||||
segments: parseSegments('## S01 · 0–3 s · Collage\nPixels: rebuilt\nCopy: "TITLE"\n'),
|
||||
ledger,
|
||||
outDir: join(dir, "held"),
|
||||
});
|
||||
expect(r.whole_frame_reuse).toMatchObject({ stills: [1], spans: [[0, 3]], pass: false });
|
||||
});
|
||||
|
||||
it("reads sound that starts late as late, and the picture's length as the duration", async () => {
|
||||
// Clicks at uneven times, with the sound running 0.4 s past the picture.
|
||||
const clicks = "aevalsrc=sin(2*PI*660*t)*lt(mod(t*t*3\\,1)\\,0.08):s=48000:d=3.4";
|
||||
const voiced = (name: string, delay: number) =>
|
||||
clip(name, [
|
||||
...["-f", "lavfi", "-i", "testsrc2=d=3:r=30:s=320x180"],
|
||||
...["-itsoffset", String(delay), "-f", "lavfi", "-i", clicks],
|
||||
...["-map", "0:v", "-map", "1:a", "-c:a", "aac"],
|
||||
]);
|
||||
const on = voiced("voiced.mp4", 0);
|
||||
const same = await diffVerdicts({ reference: on, yours: on, outDir: join(dir, "voiced") });
|
||||
expect(same.duration).toMatchObject({ reference: 3, pass: true });
|
||||
expect(same.sound).toMatchObject({ offset_ms: 0, pass: true });
|
||||
const late = await diffVerdicts({
|
||||
reference: on,
|
||||
yours: voiced("late.mp4", 0.2),
|
||||
outDir: join(dir, "late"),
|
||||
});
|
||||
expect(late.sound).toMatchObject({ offset_ms: 200, pass: false });
|
||||
});
|
||||
|
||||
it("places the sound by its decoded timestamps, not the start time the probe reports", () => {
|
||||
// ffmpeg 5.1 drops an HE-AAC edit list's skip twice: the first sample it keeps is timestamped 0.1 s in while
|
||||
// the probe still says the track starts at 0. Here a sound timestamped 0.25 s in, probed as starting at 0,
|
||||
// against the same sound with 0.25 s of silence baked into its samples: in sync.
|
||||
const clicks = "aevalsrc=sin(2*PI*660*t)*lt(mod(t*t*3\\,1)\\,0.08):s=48000:d=3";
|
||||
const picture = ["-f", "lavfi", "-i", "testsrc2=d=3:r=30:s=320x180"];
|
||||
const stamped = clip("stamped.mp4", [
|
||||
...picture,
|
||||
...["-itsoffset", "0.25", "-f", "lavfi", "-i", clicks],
|
||||
...["-map", "0:v", "-map", "1:a", "-c:a", "pcm_s16le", "-f", "mov"],
|
||||
]);
|
||||
const baked = clip("baked.mp4", [
|
||||
...picture,
|
||||
...["-f", "lavfi", "-i", clicks, "-af", "adelay=250:all=1"],
|
||||
...["-map", "0:v", "-map", "1:a", "-c:a", "pcm_s16le", "-f", "mov"],
|
||||
]);
|
||||
const probedAtZero = { ...probeVideo(stamped), audioStart: 0 };
|
||||
expect(soundOffset(stamped, baked, probedAtZero, probeVideo(baked))).toMatchObject({
|
||||
offset_ms: 0,
|
||||
pass: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("passes a re-shot person whose motion is off when the components are right", async () => {
|
||||
// The bottom half (the person) frozen on its first frame: no motion at all. Pixels above match.
|
||||
const frozen = clip("frozen.mp4", [
|
||||
...["-i", ref, "-filter_complex"],
|
||||
"[0]split[a][b];[b]crop=320:90:0:90,trim=end_frame=1,tpad=stop=89:stop_mode=clone[f];[a][f]overlay=0:90",
|
||||
]);
|
||||
const r = await diffVerdicts({
|
||||
reference: ref,
|
||||
yours: frozen,
|
||||
segments: parseSegments(
|
||||
"## S01 · 0–3 s · Board over a presenter\nPixels: rebuilt\nRegenerated: 0, 0.5, 1, 0.5\n",
|
||||
),
|
||||
outDir: join(dir, "frozen"),
|
||||
});
|
||||
const [s] = r.segments!;
|
||||
expect(s!.person?.energy?.mean_ratio).toBeLessThan(0.5);
|
||||
expect(s!.person?.gates).toEqual({ framing: null, text: null, sync: null });
|
||||
expect(s!.pass).toBe(true);
|
||||
expect(r.components).toMatchObject({ passed: 1, total: 1 });
|
||||
expect(r.person?.energy).toBeLessThan(0.5);
|
||||
expect(r.pass).toBe(true);
|
||||
});
|
||||
|
||||
// The original: a white label over a soft lower half (the person). Yours draws the label over a re-shot half.
|
||||
const strokes = (x0: number) =>
|
||||
[0, 14, 28].map((x) => `drawbox=x=${x0 + x}:y=110:w=6:h=24:color=white:t=fill`).join(",");
|
||||
const label = strokes(120);
|
||||
const soft = (delay: number) =>
|
||||
`[0]split[a][b];[b]crop=320:90:0:90,gblur=sigma=10,hue=s=0.4,tpad=start=${delay}:start_mode=clone[f];[a][f]overlay=0:90:shortest=1`;
|
||||
const carried = join(dir, "carried");
|
||||
// The generated clips: one made from a guide with the label on it, one from a masked guide. Yours draws the
|
||||
// label over each; over the first, 8 px off, the letters show doubled.
|
||||
const carriedRef = (): string => {
|
||||
const base = clip("carried-ref.mp4", [
|
||||
...["-i", ref, "-filter_complex"],
|
||||
`${soft(0)},${label}`,
|
||||
]);
|
||||
mkdirSync(join(carried, "assets"), { recursive: true });
|
||||
clip("carried/assets/dirty.mp4", [...["-i", ref, "-filter_complex"], `${soft(5)},${label}`]);
|
||||
clip("carried/assets/clean.mp4", [...["-i", ref, "-filter_complex"], soft(5)]);
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-hide_banner", "-loglevel", "error", "-y", "-i", ref, "-filter_complex"],
|
||||
`${soft(0)},trim=end_frame=1`,
|
||||
...["-frames:v", "1", join(carried, "assets", "still.png")],
|
||||
]);
|
||||
return base;
|
||||
};
|
||||
const carriedBase = noTools ? "" : carriedRef();
|
||||
const judgeCarried = (path: string, x0: number) => {
|
||||
const ledger = join(carried, `${path}.jsonl`);
|
||||
writeFileSync(
|
||||
ledger,
|
||||
JSON.stringify({
|
||||
type: "video",
|
||||
path: `assets/${path}.mp4`,
|
||||
provenance: { provider: "heygen", references: ["assets/still.png"], window: [0, 3] },
|
||||
}) + "\n",
|
||||
);
|
||||
return diffVerdicts({
|
||||
reference: carriedBase,
|
||||
yours: clip(`carried-yours-${path}.mp4`, [
|
||||
...["-i", join(carried, "assets", `${path}.mp4`), "-vf"],
|
||||
strokes(x0),
|
||||
]),
|
||||
segments: parseSegments(
|
||||
"## S01 · 0–3 s · Label over a presenter\nPixels: rebuilt\nRegenerated: 0, 90, 320, 90\nDrawn: 118, 108, 42, 28\n",
|
||||
),
|
||||
ledger,
|
||||
project: carried,
|
||||
outDir: join(dir, `carried-${path}`),
|
||||
});
|
||||
};
|
||||
|
||||
it("fails a re-shot person whose generated clip carries the original's text", async () => {
|
||||
const dirty = await judgeCarried("dirty", 128);
|
||||
expect(dirty.segments?.[0]?.person?.text?.[0]).toMatchObject({
|
||||
drawn: "S01.D1",
|
||||
carried: true,
|
||||
});
|
||||
expect(dirty.segments?.[0]?.pass).toBe(false);
|
||||
expect(dirty.person).toMatchObject({ gates: { text: false }, failed: { text: ["S01"] } });
|
||||
expect(dirty.pass).toBe(false);
|
||||
});
|
||||
|
||||
it("passes a re-shot person whose generated clip came from a masked guide", async () => {
|
||||
const clean = await judgeCarried("clean", 120);
|
||||
expect(clean.segments?.[0]?.drawn?.[0]?.pass).toBe(true);
|
||||
expect(clean.segments?.[0]?.person?.gates.text).toBe(true);
|
||||
expect(clean.segments?.[0]?.pass).toBe(true);
|
||||
expect(clean.person?.shots_at_cuts).toMatchObject({
|
||||
at_cuts: 1,
|
||||
total: 1,
|
||||
cuts_without_shot: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("judges a regenerated segment by motion and look, not pixels", async () => {
|
||||
const shifted = clip("shifted.mp4", [...["-i", ref, "-vf"], "hflip"]);
|
||||
const segments = parseSegments("## S01 · 0–3 s · AI shot\nPixels: regenerated\n");
|
||||
const r = await diffVerdicts({
|
||||
reference: ref,
|
||||
yours: shifted,
|
||||
segments,
|
||||
outDir: join(dir, "regen"),
|
||||
});
|
||||
expect(r.ssim.mean).toBeLessThan(0.9);
|
||||
expect(r.segments?.[0]?.energy?.mean_ratio).toBeCloseTo(1, 1);
|
||||
expect(r.segments?.[0]?.pass).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -1,132 +0,0 @@
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
clipFolder,
|
||||
diffVerdicts,
|
||||
loadOpenCvFirst,
|
||||
noTools,
|
||||
REF_ARGS,
|
||||
} from "./diff.test-helpers.js";
|
||||
import { parseSegments } from "./segments.js";
|
||||
|
||||
const { dir, clip } = clipFolder("hf-diff-seg-test-");
|
||||
|
||||
describe.skipIf(noTools)("diffVideos with segments", () => {
|
||||
loadOpenCvFirst();
|
||||
const ref = noTools ? "" : clip("ref.mp4", REF_ARGS);
|
||||
|
||||
it("judges a rebuilt segment's regenerated region by motion and look, the rest by SSIM", async () => {
|
||||
// The bottom half mirrored, as a re-shot presenter differs in pixels but moves the same.
|
||||
const mixed = clip("mixed.mp4", [
|
||||
...["-i", ref, "-filter_complex"],
|
||||
"[0]split[a][b];[b]crop=320:90:0:90,hflip[f];[a][f]overlay=0:90",
|
||||
]);
|
||||
const md = "## S01 · 0–3 s · Board over a presenter\nPixels: rebuilt\n";
|
||||
const plain = await diffVerdicts({
|
||||
reference: ref,
|
||||
yours: mixed,
|
||||
segments: parseSegments(md),
|
||||
outDir: join(dir, "mixed-plain"),
|
||||
});
|
||||
expect(plain.segments?.[0]?.pass).toBe(false);
|
||||
const r = await diffVerdicts({
|
||||
reference: ref,
|
||||
yours: mixed,
|
||||
segments: parseSegments(`${md}Regenerated: 0, 0.5, 1, 0.5\n`),
|
||||
outDir: join(dir, "mixed"),
|
||||
});
|
||||
const [s] = r.segments!;
|
||||
expect(s!.ssim!.mean).toBeGreaterThan(0.99);
|
||||
expect(s!.regenerated?.roi).toEqual([0, 90, 320, 90]);
|
||||
expect(s!.person?.energy?.mean_ratio).toBeCloseTo(1, 1);
|
||||
expect(s!.pass).toBe(true);
|
||||
// A wrong box outside the region still fails the segment.
|
||||
const boxed = clip("mixed-box.mp4", [
|
||||
...["-i", mixed, "-vf"],
|
||||
"drawbox=x=20:y=10:w=60:h=40:color=red:t=fill",
|
||||
]);
|
||||
const bad = await diffVerdicts({
|
||||
reference: ref,
|
||||
yours: boxed,
|
||||
segments: parseSegments(`${md}Regenerated: 0, 0.5, 1, 0.5\n`),
|
||||
outDir: join(dir, "mixed-box"),
|
||||
});
|
||||
expect(bad.segments?.[0]?.pass).toBe(false);
|
||||
});
|
||||
|
||||
// A label (three white strokes) over a soft bottom half, a regenerated person (here 5 frames late); yours
|
||||
// draws it 24 px off. Two tests of two comparisons each, so neither nears the per-test limit on a busy runner.
|
||||
const strokes = (x0: number) =>
|
||||
[0, 14, 28].map((x) => `drawbox=x=${x0 + x}:y=110:w=6:h=24:color=white:t=fill`).join(",");
|
||||
const soft = (delay: number) =>
|
||||
`[0]split[a][b];[b]crop=320:90:0:90,gblur=sigma=10,hue=s=0.4,tpad=start=${delay}:start_mode=clone[f];[a][f]overlay=0:90:shortest=1`;
|
||||
const labelled = (name: string, delay: number, x0: number) =>
|
||||
noTools ? "" : clip(name, [...["-i", ref, "-filter_complex"], `${soft(delay)},${strokes(x0)}`]);
|
||||
const base = labelled("label-ref.mp4", 0, 120);
|
||||
const right = labelled("label-right.mp4", 5, 120);
|
||||
const off = labelled("label-off.mp4", 5, 144);
|
||||
const md =
|
||||
"## S01 · 0–3 s · Label over a presenter\nPixels: rebuilt\nRegenerated: 0, 90, 320, 90\n";
|
||||
const judge = (yours: string, text: string, name: string) =>
|
||||
diffVerdicts({
|
||||
reference: base,
|
||||
yours,
|
||||
segments: parseSegments(text),
|
||||
outDir: join(dir, name),
|
||||
});
|
||||
|
||||
it("hides a drawn element in a regenerated region without a Drawn line, and flags a box over the whole frame", async () => {
|
||||
expect((await judge(off, md, "drawn-hidden")).segments?.[0]?.pass).toBe(true);
|
||||
const whole = await judge(right, md.replace("0, 90, 320, 90", "0, 0, 320, 180"), "drawn-whole");
|
||||
expect(whole.warnings.some((w) => w.includes("covers the whole frame"))).toBe(true);
|
||||
});
|
||||
|
||||
it("judges each drawn element inside a regenerated region by itself, not by what is behind it", async () => {
|
||||
const drawn = `${md}Drawn: 118, 108, 42, 28\n`;
|
||||
const good = (await judge(right, drawn, "drawn-right")).segments![0]!;
|
||||
expect(good.pass).toBe(true);
|
||||
expect(good.drawn?.[0]).toMatchObject({ id: "S01.D1", box: [118, 108, 42, 28], pass: true });
|
||||
expect(good.scored).toContain("1 drawn elements, each scored by itself");
|
||||
const bad = await judge(off, drawn, "drawn-off");
|
||||
expect(bad.segments?.[0]?.pass).toBe(false);
|
||||
expect(bad.segments?.[0]?.drawn?.[0]?.pass).toBe(false);
|
||||
});
|
||||
|
||||
it("scores a drawn element by itself without segments, and its ink says when yours is thinner", async () => {
|
||||
// The same three-stroke label over two different flat backdrops; yours draws it at half the stroke width.
|
||||
const strokes = (w: number) =>
|
||||
[0, 14, 28].map((x) => `drawbox=x=${120 + x}:y=110:w=${w}:h=24:color=white:t=fill`).join(",");
|
||||
const backdrop = (colour: string) => [
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
`color=c=${colour}:s=320x180:d=3:r=30`,
|
||||
];
|
||||
const base = clip("solo-ref.mp4", [...backdrop("0x305070"), "-vf", strokes(6)]);
|
||||
const right = clip("solo-right.mp4", [...backdrop("0x604030"), "-vf", strokes(6)]);
|
||||
const thin = clip("solo-thin.mp4", [...backdrop("0x604030"), "-vf", strokes(3)]);
|
||||
const box: [number, number, number, number] = [116, 106, 44, 32];
|
||||
const good = await diffVerdicts({
|
||||
reference: base,
|
||||
yours: right,
|
||||
at: [0.5, 1, 1.5, 2],
|
||||
drawn: [{ box }, { box, window: [1, 3] }],
|
||||
outDir: join(dir, "solo-right"),
|
||||
});
|
||||
expect(good.segments).toBeUndefined();
|
||||
expect(good.drawn).toHaveLength(2);
|
||||
expect(good.drawn?.[0]).toMatchObject({ id: "D1", box, window: [0, 3], frames: 4, pass: true });
|
||||
expect(good.drawn?.[1]).toMatchObject({ id: "D2", window: [1, 3], frames: 3, pass: true });
|
||||
const bad = await diffVerdicts({
|
||||
reference: base,
|
||||
yours: thin,
|
||||
at: [0.5, 1, 1.5, 2],
|
||||
drawn: [{ box }],
|
||||
outDir: join(dir, "solo-thin"),
|
||||
});
|
||||
expect(good.drawn?.[0]).toMatchObject({ score: 1, offset: [0, 0], ink: 1 });
|
||||
// Half-weight letters in the right place: the score barely drops, the ink says it.
|
||||
expect(bad.drawn?.[0]?.score).toBeLessThan(0.9);
|
||||
expect(bad.drawn?.[0]?.ink).toBe(0.5);
|
||||
});
|
||||
});
|
||||
@@ -1,63 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { cpSync, existsSync, mkdtempSync, renameSync, rmSync } from "node:fs";
|
||||
import { tmpdir, userInfo } from "node:os";
|
||||
import { basename, join } from "node:path";
|
||||
import { afterAll, beforeAll } from "vitest";
|
||||
import { findFFmpeg, findFFprobe } from "../browser/ffmpeg.js";
|
||||
import { diffVideos, type DiffOptions } from "./diff.js";
|
||||
import { loadOpenCv, OPENCV_DIR } from "./opencv.js";
|
||||
|
||||
export const ffmpeg = findFFmpeg();
|
||||
export const noTools = !ffmpeg || !findFFprobe();
|
||||
|
||||
/** testsrc2, 3 s at 30 fps: the original most of diff's tests compare against. */
|
||||
export const REF_ARGS = ["-f", "lavfi", "-i", "testsrc2=d=3:r=30:s=320x180"];
|
||||
|
||||
/** A temp folder for one test file's clips, removed after its tests, and a maker of lossless synthetic clips in it,
|
||||
* so SSIM differences come from the picture, not the encoder. */
|
||||
export function clipFolder(prefix: string): {
|
||||
dir: string;
|
||||
clip: (name: string, args: string[]) => string;
|
||||
} {
|
||||
const dir = mkdtempSync(join(tmpdir(), prefix));
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
const clip = (name: string, args: string[]): string => {
|
||||
const out = join(dir, name);
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-hide_banner", "-loglevel", "error", "-y", ...args],
|
||||
...["-pix_fmt", "yuv420p", "-c:v", "libx264", "-crf", "0", out],
|
||||
]);
|
||||
return out;
|
||||
};
|
||||
return { dir, clip };
|
||||
}
|
||||
|
||||
/** diffVideos for a test that reads verdicts only: no worst-moment pictures, each of which costs seven ffmpeg runs. */
|
||||
export const diffVerdicts = (options: DiffOptions) => diffVideos({ worst: 0, ...options });
|
||||
|
||||
/** opencv.js loaded before a block's tests, so none pays for its download inside the per-test limit. The files come
|
||||
* from this machine's cache, else a temp copy kept between runs (each has its own home), else the network. */
|
||||
export function loadOpenCvFirst(): void {
|
||||
beforeAll(async () => {
|
||||
const shared = join(tmpdir(), `hf-test-opencv-${basename(OPENCV_DIR)}`);
|
||||
const machine = join(
|
||||
userInfo().homedir,
|
||||
".cache",
|
||||
"hyperframes",
|
||||
"opencv",
|
||||
basename(OPENCV_DIR),
|
||||
);
|
||||
for (const from of [machine, shared])
|
||||
if (!existsSync(join(OPENCV_DIR, "opencv.js")) && existsSync(join(from, "opencv.js")))
|
||||
cpSync(from, OPENCV_DIR, { recursive: true });
|
||||
await loadOpenCv();
|
||||
if (existsSync(join(shared, "opencv.js"))) return;
|
||||
const staged = `${shared}-${process.pid}`;
|
||||
cpSync(OPENCV_DIR, staged, { recursive: true });
|
||||
try {
|
||||
renameSync(staged, shared);
|
||||
} catch {
|
||||
rmSync(staged, { recursive: true, force: true });
|
||||
}
|
||||
}, 90_000);
|
||||
}
|
||||
@@ -1,346 +0,0 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
cellsOutside,
|
||||
diffVideos,
|
||||
longRuns,
|
||||
pickedFrames,
|
||||
regionRoi,
|
||||
selectFilter,
|
||||
wholeFrameCuts,
|
||||
} from "./diff.js";
|
||||
import {
|
||||
clipFolder,
|
||||
diffVerdicts,
|
||||
loadOpenCvFirst,
|
||||
noTools,
|
||||
REF_ARGS,
|
||||
} from "./diff.test-helpers.js";
|
||||
import { carriedFrame, shotFraming, syncPass } from "./person.js";
|
||||
import { parseSegments } from "./segments.js";
|
||||
|
||||
const { dir, clip } = clipFolder("hf-diff-test-");
|
||||
|
||||
describe("parseSegments", () => {
|
||||
it("reads headings and where each window's pixels come from", () => {
|
||||
const md = [
|
||||
"# Film — reverse",
|
||||
"## S01 · 0.00–2.40 s · Title card",
|
||||
"Pixels: rebuilt",
|
||||
"## S02 · 2.40–5.00 s · Talking head",
|
||||
"Pixels: reused",
|
||||
"## S03 · 5.00–6.5 s · Product shot",
|
||||
"Pixels: regenerated",
|
||||
"## S04 · 6.5–8 s",
|
||||
"## Unknowns",
|
||||
].join("\n");
|
||||
expect(parseSegments(md)).toEqual([
|
||||
{ id: "S01", start: 0, end: 2.4, title: "Title card", pixels: "rebuilt" },
|
||||
{ id: "S02", start: 2.4, end: 5, title: "Talking head", pixels: "reused" },
|
||||
{ id: "S03", start: 5, end: 6.5, title: "Product shot", pixels: "regenerated" },
|
||||
{ id: "S04", start: 6.5, end: 8, title: "", pixels: "rebuilt" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("reads a regenerated region inside a rebuilt segment, in pixels or shares", () => {
|
||||
const md = [
|
||||
"## S01 · 0–2 s · Board under the presenter",
|
||||
"Pixels: rebuilt",
|
||||
"Regenerated: 0, 95, 720, 600 (the re-shot presenter)",
|
||||
"## S02 · 2–4 s",
|
||||
"Regenerated: 0 0.5 1 0.5",
|
||||
"## S03 · 4–6 s",
|
||||
"Pixels: regenerated",
|
||||
"Regenerated: 0, 0, 10, 10",
|
||||
].join("\n");
|
||||
const [a, b, c] = parseSegments(md);
|
||||
expect(a!.regenerated).toEqual([0, 95, 720, 600]);
|
||||
expect(b!.regenerated).toEqual([0, 0.5, 1, 0.5]);
|
||||
expect(c!.regenerated).toBeUndefined();
|
||||
});
|
||||
|
||||
it("reads drawn elements over a regenerated person, and whether a segment quotes copy", () => {
|
||||
const md = [
|
||||
"## S01 · 0–2 s · Title over the presenter",
|
||||
"Pixels: rebuilt",
|
||||
"Regenerated: 0, 0, 608, 1080",
|
||||
'Copy: "IS NO" / "LONGER"',
|
||||
"Drawn: 110, 615, 390, 165 (the title)",
|
||||
"Drawn: 40 40 120 120, logo 0.5–1.5 s",
|
||||
"## S02 · 2–4 s",
|
||||
"Copy: none, footage only",
|
||||
].join("\n");
|
||||
const [a, b] = parseSegments(md);
|
||||
expect(a!.drawn).toEqual([
|
||||
{ box: [110, 615, 390, 165] },
|
||||
{ box: [40, 40, 120, 120], window: [0.5, 1.5] },
|
||||
]);
|
||||
expect(a!.copy).toBe(true);
|
||||
expect(b!.drawn).toBeUndefined();
|
||||
expect(b!.copy).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("whole-frame cuts", () => {
|
||||
it("keeps the ledger's cuts and stills from the original that cover the whole frame", () => {
|
||||
const ledger = [
|
||||
{
|
||||
type: "video",
|
||||
provenance: { provider: "original", window: [0, 12.267], roi: [0, 0, 720, 1280] },
|
||||
},
|
||||
{ type: "video", provenance: { provider: "original", window: [1, 2] } },
|
||||
{
|
||||
type: "video",
|
||||
provenance: { provider: "original", window: [3, 4], roi: [0, 0, 720, 704] },
|
||||
},
|
||||
{ type: "image", provenance: { provider: "original", at: 3, roi: [0, 0, 720, 1280] } },
|
||||
{ type: "image", provenance: { provider: "original", at: 5, roi: [60, 260, 480, 560] } },
|
||||
{ type: "video", provenance: { provider: "heygen", window: [0, 5] } },
|
||||
]
|
||||
.map((e) => JSON.stringify(e))
|
||||
.join("\n");
|
||||
expect(wholeFrameCuts(`${ledger}\nnot json`, 720, 1280)).toEqual({
|
||||
windows: [
|
||||
[0, 12.267],
|
||||
[1, 2],
|
||||
],
|
||||
stills: [3],
|
||||
});
|
||||
});
|
||||
|
||||
it("drops runs of frames shorter than the minimum", () => {
|
||||
const on = [true, true, false, true, true, true, true, true];
|
||||
expect(longRuns(on, [0, 1, 2, 3, 4, 5, 9, 10], 3)).toEqual([
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("regenerated regions", () => {
|
||||
it("scales shares to pixels, clips to the frame and leaves the covered cells to the region", () => {
|
||||
expect(regionRoi([0, 0.5, 1, 0.5], 320, 180)).toEqual([0, 90, 320, 90]);
|
||||
expect(regionRoi([600, 1200, 400, 200], 720, 1280)).toEqual([600, 1200, 120, 80]);
|
||||
// 80x44 cells: the bottom half covers rows 2 and 3 (row 2 from y 88, all but 2 px of it).
|
||||
expect(cellsOutside([0, 90, 320, 90], 320, 180)).toEqual([0, 1, 2, 3, 4, 5, 6, 7]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("person gates", () => {
|
||||
const at = (t: number, ...boxes: number[][]) => ({ t, boxes });
|
||||
const face = [0.4, 0.3, 0.2, 0.12];
|
||||
const original = [0, 0.4, 0.8, 1.2].map((t) => at(t, face));
|
||||
|
||||
it("fails a shot's framing only when the face sits clearly off the original's", () => {
|
||||
const moved = (dy: number, grow = 1) =>
|
||||
[0, 0.4, 0.8, 1.2].map((t) => at(t, [0.4, 0.3 + dy, 0.2 * grow, 0.12 * grow]));
|
||||
expect(shotFraming([0, 1.5], original, moved(0.01))).toMatchObject({ pass: true });
|
||||
// Half a face height lower, or 40% bigger.
|
||||
expect(shotFraming([0, 1.5], original, moved(0.06))).toMatchObject({ pass: false });
|
||||
expect(shotFraming([0, 1.5], original, moved(0, 1.4))).toMatchObject({ pass: false });
|
||||
// One stray moment off (a hand over the face, a photo) is not a framing.
|
||||
const stray = original.map((m, i) => (i === 1 ? at(m.t, [0.4, 0.6, 0.2, 0.12]) : m));
|
||||
expect(shotFraming([0, 1.5], original, stray)).toMatchObject({ pass: true });
|
||||
// Another face in yours (a picture on the wall) does not stand in for the person.
|
||||
const crowd = [0, 0.4, 0.8, 1.2].map((t) => at(t, [0.05, 0.05, 0.3, 0.2], face));
|
||||
expect(shotFraming([0, 1.5], original, crowd)).toMatchObject({
|
||||
shift: 0,
|
||||
pass: true,
|
||||
});
|
||||
// Too few faces to judge.
|
||||
expect(shotFraming([0, 0.5], original, original)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("judges lips only on a confident reading, inside +45 / −125 ms", () => {
|
||||
expect(syncPass({ best_lag_ms: 30, confidence: "high" })).toBe(true);
|
||||
expect(syncPass({ best_lag_ms: -120, confidence: "high" })).toBe(true);
|
||||
expect(syncPass({ best_lag_ms: 80, confidence: "high" })).toBe(false);
|
||||
expect(syncPass({ best_lag_ms: -200, confidence: "high" })).toBe(false);
|
||||
expect(syncPass({ best_lag_ms: -300, confidence: "medium" })).toBeNull();
|
||||
});
|
||||
|
||||
it("finds the original's letters in a generated clip, not the place they share", () => {
|
||||
// A textured place (the same in the original, the clean still and the clip) and white letters over it.
|
||||
const [w, h] = [60, 30];
|
||||
const place = new Uint8Array(w * h * 3).map((_, i) => 60 + ((i * 37) % 50));
|
||||
const lettered = (px: Uint8Array) => {
|
||||
const out = new Uint8Array(px);
|
||||
for (let y = 8; y < 22; y++)
|
||||
for (const x0 of [10, 25, 40])
|
||||
for (let x = x0; x < x0 + 4; x++) out.set([255, 255, 255], (y * w + x) * 3);
|
||||
return out;
|
||||
};
|
||||
const original = lettered(place);
|
||||
expect(carriedFrame(original, lettered(place), place, w, h, undefined)).toBeGreaterThan(0.9);
|
||||
expect(carriedFrame(original, place, place, w, h, undefined)).toBeLessThan(0.1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("frame picks", () => {
|
||||
it("keeps a range's frames end-exclusive and renumbers them on the original's clock", () => {
|
||||
expect(pickedFrames({ kind: "range", k0: 3, k1: 6 }, 90)).toEqual([3, 4, 5]);
|
||||
expect(selectFilter({ kind: "range", k0: 3, k1: 6 }, "30000/1001")).toEqual([
|
||||
"select=between(n\\,3\\,5)",
|
||||
"setpts=N/(30000/1001)/TB",
|
||||
]);
|
||||
expect(selectFilter({ kind: "at", ks: [2, 9] }, "30")[0]).toBe("select=eq(n\\,2)+eq(n\\,9)");
|
||||
expect(selectFilter({ kind: "all" }, "30")).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(noTools)("diffVideos", () => {
|
||||
loadOpenCvFirst();
|
||||
const ref = noTools ? "" : clip("ref.mp4", REF_ARGS);
|
||||
|
||||
it("passes a video against itself", async () => {
|
||||
const r = await diffVideos({ reference: ref, yours: ref, outDir: join(dir, "same") });
|
||||
expect(r.pass).toBe(true);
|
||||
expect(r.ssim.frames).toBe(90);
|
||||
expect(r.ssim.mean).toBeGreaterThan(0.999);
|
||||
expect(r.worst).toEqual([]);
|
||||
});
|
||||
|
||||
it("finds a wrong element by its cell when the whole frame still scores high", async () => {
|
||||
// An 80x60 box over a 320x180 frame for one second: about 0.94 whole-frame SSIM.
|
||||
const yours = clip("box.mp4", [
|
||||
...["-i", ref, "-vf"],
|
||||
"drawbox=x=40:y=40:w=80:h=60:color=red:t=fill:enable='between(t,1,2)'",
|
||||
]);
|
||||
const r = await diffVideos({ reference: ref, yours, outDir: join(dir, "box") });
|
||||
expect(r.ssim.mean).toBeGreaterThan(0.92);
|
||||
expect(r.ssim.cell_p05).toBeLessThan(0.8);
|
||||
expect(r.pass).toBe(false);
|
||||
const [first] = r.worst;
|
||||
expect(first!.t).toBeGreaterThanOrEqual(1);
|
||||
expect(first!.t).toBeLessThanOrEqual(2);
|
||||
const [x, y, w, h] = first!.cell.roi;
|
||||
expect(x < 120 && x + w > 40 && y < 100 && y + h > 40).toBe(true);
|
||||
expect(existsSync(r.sheet!)).toBe(true);
|
||||
});
|
||||
|
||||
it("draws the worst cell of a tall frame scaled for the sheet", async () => {
|
||||
// 1080x1920 scales to 960x1706 for the sheet; the bottom row's box rounds a pixel past the edge.
|
||||
const tall = clip("tall.mp4", ["-f", "lavfi", "-i", "testsrc2=d=1:r=30:s=1080x1920"]);
|
||||
const yours = clip("tall-box.mp4", [
|
||||
...["-i", tall, "-vf"],
|
||||
"drawbox=x=0:y=1500:w=1080:h=420:color=red:t=fill",
|
||||
]);
|
||||
const r = await diffVideos({ reference: tall, yours, outDir: join(dir, "tall") });
|
||||
expect(r.worst.length).toBeGreaterThan(0);
|
||||
expect(r.worst[0]!.cell.roi[1]).toBe(1440);
|
||||
});
|
||||
|
||||
it("fails a scene left empty on a plain background", async () => {
|
||||
// A small dark label on white, gone from yours after 1.5 s: still about 0.97 whole-frame SSIM.
|
||||
const label = "drawbox=x=40:y=20:w=36:h=12:color=0x303030:t=fill";
|
||||
const page = clip("page.mp4", [
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=white:s=320x180:r=30:d=3",
|
||||
"-vf",
|
||||
label,
|
||||
]);
|
||||
const yours = clip("emptied.mp4", [
|
||||
...["-f", "lavfi", "-i", "color=c=white:s=320x180:r=30:d=3", "-vf"],
|
||||
`${label}:enable='lt(t,1.5)'`,
|
||||
]);
|
||||
const r = await diffVerdicts({ reference: page, yours, outDir: join(dir, "emptied") });
|
||||
expect(r.missing_content.pass).toBe(false);
|
||||
expect(r.missing_content.share).toBeGreaterThan(0.45);
|
||||
expect(r.missing_content.spans[0]![0]).toBeCloseTo(1.5, 1);
|
||||
expect(r.pass).toBe(false);
|
||||
const same = await diffVerdicts({ reference: page, yours: page, outDir: join(dir, "page") });
|
||||
expect(same.missing_content).toMatchObject({ share: 0, pass: true });
|
||||
// A regenerated segment's picture is its own: what it leaves out is not missing.
|
||||
const regen = await diffVerdicts({
|
||||
reference: page,
|
||||
yours,
|
||||
segments: parseSegments("## S01 · 0–3 s · AI shot\nPixels: regenerated\n"),
|
||||
outDir: join(dir, "emptied-regen"),
|
||||
});
|
||||
expect(regen.missing_content).toMatchObject({ share: 0, pass: true });
|
||||
});
|
||||
|
||||
it("reports missing texture apart from missing shapes, without failing it", async () => {
|
||||
// A fine 4 px checker in the bottom-left cell: flat at an eighth of the width, so texture, not structure.
|
||||
const checker =
|
||||
"if(lt(X\\,80)*gte(Y\\,135)\\,if(mod(floor(X/4)+floor(Y/4)\\,2)\\,60\\,200)\\,235)";
|
||||
const grain = clip("grain.mp4", [
|
||||
...["-f", "lavfi", "-i", "color=c=white:s=320x180:r=30:d=2", "-vf"],
|
||||
`format=gray,geq=lum='${checker}',format=yuv420p`,
|
||||
]);
|
||||
const plain = clip("plain.mp4", [
|
||||
...["-f", "lavfi", "-i", "color=c=white:s=320x180:r=30:d=2", "-vf"],
|
||||
"format=gray,geq=lum=235,format=yuv420p",
|
||||
]);
|
||||
const r = await diffVerdicts({ reference: grain, yours: plain, outDir: join(dir, "grain") });
|
||||
expect(r.missing_content).toMatchObject({ share: 0, pass: true });
|
||||
expect(r.missing_content.texture.share).toBeGreaterThan(0.9);
|
||||
expect(r.warnings.some((w) => w.includes("reported, not failed"))).toBe(true);
|
||||
});
|
||||
|
||||
it("scores only a range, on the original's frame numbers", async () => {
|
||||
const yours = clip("late.mp4", [
|
||||
...["-i", ref, "-vf"],
|
||||
"drawbox=x=0:y=0:w=320:h=180:color=black:t=fill:enable='gte(t,2)'",
|
||||
]);
|
||||
const r = await diffVerdicts({
|
||||
reference: ref,
|
||||
yours,
|
||||
range: [0.5, 1.5],
|
||||
outDir: join(dir, "range"),
|
||||
});
|
||||
expect(r.ssim.frames).toBe(30);
|
||||
expect(r.window).toEqual([0.5, 1.5]);
|
||||
expect(r.pass).toBe(true);
|
||||
});
|
||||
|
||||
it("reports a cut a few frames late as missing and extra", async () => {
|
||||
const two = (name: string, at: number) =>
|
||||
clip(name, [
|
||||
...["-f", "lavfi", "-i", `color=c=0x2050c0:s=320x180:r=30:d=${at}`],
|
||||
...["-f", "lavfi", "-i", `color=c=0xe0c040:s=320x180:r=30:d=${3 - at}`],
|
||||
...["-filter_complex", "[0][1]concat=n=2:v=1[v]", "-map", "[v]"],
|
||||
]);
|
||||
const r = await diffVerdicts({
|
||||
reference: two("cut-ref.mp4", 1),
|
||||
yours: two("cut-late.mp4", 1.1),
|
||||
outDir: join(dir, "cuts"),
|
||||
});
|
||||
expect(r.cuts?.reference).toEqual([1]);
|
||||
expect(r.cuts?.missing).toEqual([1]);
|
||||
expect(r.cuts?.extra).toEqual([1.1]);
|
||||
expect(r.pass).toBe(false);
|
||||
});
|
||||
|
||||
it("matches a soft cut in the original with a hard one in yours", async () => {
|
||||
// Grey 0x80 to 0x8b changes the signature by 10 levels: a soft cut. Yours jumps further on the same frame.
|
||||
const step = (name: string, to: string, at: number) =>
|
||||
clip(name, [
|
||||
...["-f", "lavfi", "-i", `color=c=0x808080:s=320x180:r=30:d=${at}`],
|
||||
...["-f", "lavfi", "-i", `color=c=${to}:s=320x180:r=30:d=${3 - at}`],
|
||||
...["-filter_complex", "[0][1]concat=n=2:v=1[v]", "-map", "[v]"],
|
||||
]);
|
||||
const soft = step("soft-ref.mp4", "0x8b8b8b", 1);
|
||||
const r = await diffVerdicts({
|
||||
reference: soft,
|
||||
yours: step("soft-hard.mp4", "0xa0a0a0", 1),
|
||||
outDir: join(dir, "soft"),
|
||||
});
|
||||
expect(r.cuts).toMatchObject({ reference: [1], missing: [], extra: [], pass: true });
|
||||
const late = await diffVerdicts({
|
||||
reference: soft,
|
||||
yours: step("soft-late.mp4", "0xa0a0a0", 1.2),
|
||||
outDir: join(dir, "soft-late"),
|
||||
});
|
||||
expect(late.cuts).toMatchObject({ missing: [1], extra: [1.2], pass: false });
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,74 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { DRAWN_MIN_MEAN, drawnVerdict, scoreDrawnFrame } from "./drawn.js";
|
||||
|
||||
const [W, H] = [160, 48];
|
||||
|
||||
/** A crop: a background, then bars standing in for letters (x, width) in one colour. */
|
||||
function crop(
|
||||
background: (x: number, y: number) => [number, number, number],
|
||||
bars: [number, number][],
|
||||
ink: [number, number, number] = [255, 255, 255],
|
||||
dy = 0,
|
||||
): Uint8Array {
|
||||
const px = new Uint8Array(W * H * 3);
|
||||
for (let y = 0; y < H; y++)
|
||||
for (let x = 0; x < W; x++) {
|
||||
const on = y >= 10 + dy && y < 38 + dy && bars.some(([a, w]) => x >= a && x < a + w);
|
||||
px.set(on ? ink : background(x, y), (y * W + x) * 3);
|
||||
}
|
||||
return px;
|
||||
}
|
||||
|
||||
// Two different backgrounds behind the same letters: a warm room and a cool, striped one.
|
||||
const room = (x: number, y: number): [number, number, number] => [90 + (x % 7), 60 + (y % 5), 40];
|
||||
const street = (x: number, _y: number): [number, number, number] => [
|
||||
30,
|
||||
70 + ((x >> 3) % 2) * 25,
|
||||
110,
|
||||
];
|
||||
const word: [number, number][] = [10, 30, 50, 70, 90, 110, 130].map((x) => [x, 8]);
|
||||
|
||||
describe("scoreDrawnFrame", () => {
|
||||
it("scores the same letters over a different background as a match", () => {
|
||||
const f = scoreDrawnFrame(crop(room, word), crop(street, word), W, H);
|
||||
expect(f.score).toBeGreaterThan(DRAWN_MIN_MEAN);
|
||||
expect(f.offset).toEqual([0, 0]);
|
||||
});
|
||||
|
||||
it("forgives a pixel or two of offset and says how far", () => {
|
||||
const f = scoreDrawnFrame(crop(room, word), crop(street, word, undefined, 2), W, H);
|
||||
expect(f.score).toBeGreaterThan(DRAWN_MIN_MEAN);
|
||||
expect(f.offset).toEqual([0, 2]);
|
||||
});
|
||||
|
||||
it("fails lighter weight, wider tracking, a doubled letter and another fill", () => {
|
||||
const light = word.map(([x]): [number, number] => [x, 4]);
|
||||
const wide = word.map(([x, w], i): [number, number] => [x + i * 3, w]);
|
||||
const doubled = [...word, ...word.map(([x, w]): [number, number] => [x + 12, w])];
|
||||
for (const yours of [
|
||||
crop(street, light),
|
||||
crop(street, wide),
|
||||
crop(street, doubled),
|
||||
crop(street, word, [250, 210, 0]),
|
||||
])
|
||||
expect(scoreDrawnFrame(crop(room, word), yours, W, H).score).toBeLessThan(DRAWN_MIN_MEAN);
|
||||
});
|
||||
|
||||
it("scores a missing element as nothing matched, and nothing on both sides as no frame", () => {
|
||||
expect(scoreDrawnFrame(crop(room, word), crop(street, []), W, H).score).toBeLessThan(0.5);
|
||||
const flat = () => [128, 128, 128] as [number, number, number];
|
||||
expect(scoreDrawnFrame(crop(flat, []), crop(flat, []), W, H).score).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("drawnVerdict", () => {
|
||||
it("passes on the mean and the 5th percentile of the frames, and reports ink weight", () => {
|
||||
const good = scoreDrawnFrame(crop(room, word), crop(street, word), W, H);
|
||||
const gone = scoreDrawnFrame(crop(room, word), crop(street, []), W, H);
|
||||
const v = drawnVerdict([0, 0, W, H], [0, 1], Array(10).fill(good));
|
||||
expect(v).toMatchObject({ pass: true, frames: 10, ink: 1 });
|
||||
const bad = drawnVerdict([0, 0, W, H], [0, 1], [...Array(9).fill(good), gone]);
|
||||
expect(bad.pass).toBe(false);
|
||||
expect(drawnVerdict([0, 0, W, H], [0, 1], [])).toMatchObject({ pass: null, frames: 0 });
|
||||
});
|
||||
});
|
||||
@@ -1,229 +0,0 @@
|
||||
// A drawn element (text, a badge, a card) over a regenerated person, scored by itself: what is behind it differs on
|
||||
// purpose, so SSIM over its box measures the background. Each frame finds the element's ink on both sides, aligns
|
||||
// them within a few pixels, and asks how much of either side's ink the other matches in colour. Wrong weight,
|
||||
// tracking, slant, size, fill, a doubled letter or a late onset leave ink unmatched; a different background leaves
|
||||
// none. Calibrated on the reverse bench (10-23): correct titles and captions 0.82–0.94, wrong ones 0.40–0.66.
|
||||
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { round, runFfmpeg, type Clip, type Roi } from "./media.js";
|
||||
|
||||
export const DRAWN_MIN_MEAN = 0.8;
|
||||
const DRAWN_MIN_P05 = 0.65;
|
||||
// A pixel is ink when its colour is this far (RGB distance) from its neighbourhood's median.
|
||||
const INK_CONTRAST = 60;
|
||||
// Two pixels match within this RGB distance, one pixel apart at most, after aligning the sides this many pixels.
|
||||
const MATCH_COLOUR = 45;
|
||||
const ALIGN = 3;
|
||||
// Fewer ink pixels than this on both sides: nothing drawn in that frame on either side.
|
||||
const MIN_INK = 30;
|
||||
const MAX_FPS = 10;
|
||||
|
||||
export interface DrawnFrame {
|
||||
/** Share of both sides' ink the other side matches (F1 of matched ink), or null when neither side has ink. */
|
||||
score: number | null;
|
||||
/** Where yours sits against the original, x and y in pixels (within ±3; further lowers the score). */
|
||||
offset: [number, number];
|
||||
inkOriginal: number;
|
||||
inkYours: number;
|
||||
}
|
||||
|
||||
export interface DrawnVerdict {
|
||||
box: Roi;
|
||||
window: [number, number];
|
||||
frames: number;
|
||||
pass: boolean | null;
|
||||
/** Mean and 5th percentile of the frames' scores. */
|
||||
score?: number;
|
||||
p05?: number;
|
||||
/** Median of the frames' offsets. */
|
||||
offset?: [number, number];
|
||||
/** Yours' ink over the original's, median of frames where both have ink: under 1 is lighter or smaller. */
|
||||
ink?: number;
|
||||
}
|
||||
|
||||
/** Each pixel's colour distance from the median colour of the blocks around it. */
|
||||
// fallow-ignore-next-line complexity
|
||||
export function contrast(px: Uint8Array, w: number, h: number): Float32Array {
|
||||
const bs = Math.max(4, Math.floor(Math.max(8, Math.floor(h / 2)) / 2));
|
||||
const [gw, gh] = [Math.ceil(w / bs), Math.ceil(h / bs)];
|
||||
const med = new Float32Array(gw * gh * 3);
|
||||
const hist = new Uint32Array(256);
|
||||
for (let by = 0; by < gh; by++)
|
||||
for (let bx = 0; bx < gw; bx++) {
|
||||
const [x0, x1] = [Math.max(0, (bx - 1) * bs), Math.min(w, (bx + 2) * bs)];
|
||||
const [y0, y1] = [Math.max(0, (by - 1) * bs), Math.min(h, (by + 2) * bs)];
|
||||
const half = ((x1 - x0) * (y1 - y0)) / 2;
|
||||
for (let c = 0; c < 3; c++) {
|
||||
hist.fill(0);
|
||||
for (let y = y0; y < y1; y++)
|
||||
for (let x = x0; x < x1; x++) hist[px[(y * w + x) * 3 + c]!]!++;
|
||||
let [v, seen] = [0, 0];
|
||||
while (v < 255 && seen + hist[v]! < half) seen += hist[v++]!;
|
||||
med[(by * gw + bx) * 3 + c] = v;
|
||||
}
|
||||
}
|
||||
const out = new Float32Array(w * h);
|
||||
for (let y = 0; y < h; y++)
|
||||
for (let x = 0; x < w; x++) {
|
||||
const [o, m] = [(y * w + x) * 3, (Math.floor(y / bs) * gw + Math.floor(x / bs)) * 3];
|
||||
out[y * w + x] = Math.hypot(
|
||||
px[o]! - med[m]!,
|
||||
px[o + 1]! - med[m + 1]!,
|
||||
px[o + 2]! - med[m + 2]!,
|
||||
);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const inkOf = (px: Uint8Array, w: number, h: number) => {
|
||||
const c = contrast(px, w, h);
|
||||
return Uint8Array.from(c, (v) => (v > INK_CONTRAST ? 1 : 0));
|
||||
};
|
||||
|
||||
/** One frame of an element: the original's and yours' RGB crops of the same box. */
|
||||
// fallow-ignore-next-line complexity
|
||||
export function scoreDrawnFrame(a: Uint8Array, b: Uint8Array, w: number, h: number): DrawnFrame {
|
||||
const [ia, ib] = [inkOf(a, w, h), inkOf(b, w, h)];
|
||||
const [na, nb] = [ia.reduce((s, v) => s + v, 0), ib.reduce((s, v) => s + v, 0)];
|
||||
if (na + nb < MIN_INK) return { score: null, offset: [0, 0], inkOriginal: na, inkYours: nb };
|
||||
let mostOverlappingShift = { iou: -1, dx: 0, dy: 0 };
|
||||
for (let dy = -ALIGN; dy <= ALIGN; dy++)
|
||||
for (let dx = -ALIGN; dx <= ALIGN; dx++) {
|
||||
let [both, either] = [0, 0];
|
||||
for (let y = 0; y < h; y++) {
|
||||
const sy = y - dy;
|
||||
for (let x = 0; x < w; x++) {
|
||||
const sx = x - dx;
|
||||
const vb = sy >= 0 && sy < h && sx >= 0 && sx < w ? ib[sy * w + sx]! : 0;
|
||||
const va = ia[y * w + x]!;
|
||||
both += va & vb;
|
||||
either += va | vb;
|
||||
}
|
||||
}
|
||||
const iou = either ? both / either : 0;
|
||||
if (iou > mostOverlappingShift.iou) mostOverlappingShift = { iou, dx, dy };
|
||||
}
|
||||
// Ink on either side counts as matched when the other side has a pixel of its colour within one pixel.
|
||||
const at = (x: number, y: number) =>
|
||||
(Math.min(h - 1, Math.max(0, y)) * w + Math.min(w - 1, Math.max(0, x))) * 3;
|
||||
const near = (p: Uint8Array, o: number, q: Uint8Array, x: number, y: number) => {
|
||||
for (let ry = -1; ry <= 1; ry++)
|
||||
for (let rx = -1; rx <= 1; rx++) {
|
||||
const k = at(x + rx, y + ry);
|
||||
if (Math.hypot(p[o]! - q[k]!, p[o + 1]! - q[k + 1]!, p[o + 2]! - q[k + 2]!) < MATCH_COLOUR)
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
let [hitA, hitB, inkA, inkB] = [0, 0, 0, 0];
|
||||
for (let y = 0; y < h; y++)
|
||||
for (let x = 0; x < w; x++) {
|
||||
const i = y * w + x;
|
||||
const [sx, sy] = [x - mostOverlappingShift.dx, y - mostOverlappingShift.dy];
|
||||
const inB = sy >= 0 && sy < h && sx >= 0 && sx < w && ib[sy * w + sx] === 1;
|
||||
if (ia[i]) {
|
||||
inkA++;
|
||||
if (near(a, i * 3, b, sx, sy)) hitA++;
|
||||
}
|
||||
if (inB) {
|
||||
inkB++;
|
||||
if (near(b, at(sx, sy), a, x, y)) hitB++;
|
||||
}
|
||||
}
|
||||
const [recall, precision] = [inkA ? hitA / inkA : 0, inkB ? hitB / inkB : 0];
|
||||
return {
|
||||
score: recall + precision ? (2 * recall * precision) / (recall + precision) : 0,
|
||||
offset: [0 - mostOverlappingShift.dx, 0 - mostOverlappingShift.dy],
|
||||
inkOriginal: na,
|
||||
inkYours: nb,
|
||||
};
|
||||
}
|
||||
|
||||
const median = (v: number[]) => [...v].sort((x, y) => x - y)[v.length >> 1] ?? 0;
|
||||
|
||||
export function drawnVerdict(
|
||||
box: Roi,
|
||||
window: [number, number],
|
||||
frames: DrawnFrame[],
|
||||
): DrawnVerdict {
|
||||
const head = { box, window, frames: frames.length };
|
||||
const scored = frames.flatMap((f) => (f.score === null ? [] : [f]));
|
||||
if (!scored.length) return { ...head, pass: null };
|
||||
const sorted = scored.map((f) => f.score!).sort((x, y) => x - y);
|
||||
const mean = sorted.reduce((s, v) => s + v, 0) / sorted.length;
|
||||
const p05 = sorted[Math.min(sorted.length - 1, Math.floor(0.05 * sorted.length))]!;
|
||||
const both = scored.filter((f) => f.inkOriginal >= MIN_INK && f.inkYours >= MIN_INK);
|
||||
return {
|
||||
...head,
|
||||
pass: mean >= DRAWN_MIN_MEAN && p05 >= DRAWN_MIN_P05,
|
||||
score: round(mean, 3),
|
||||
p05: round(p05, 3),
|
||||
offset: [median(scored.map((f) => f.offset[0])), median(scored.map((f) => f.offset[1]))],
|
||||
...(both.length ? { ink: round(median(both.map((f) => f.inkYours / f.inkOriginal)), 2) } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
export interface DrawnJob {
|
||||
box: Roi;
|
||||
/** Indexes into the compared frames (the clips' frames after their own filters), in order. */
|
||||
frames: number[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Each job's crops from a clip scaled to `width`x`height` (the original's size), one decode for all of them:
|
||||
* `frames[j][n]` is job j's n-th frame, RGB at the box's size.
|
||||
*/
|
||||
export function readDrawn(
|
||||
clip: Clip,
|
||||
jobs: DrawnJob[],
|
||||
width: number,
|
||||
height: number,
|
||||
): Uint8Array[][] {
|
||||
if (!jobs.length) return [];
|
||||
const work = mkdtempSync(join(tmpdir(), "hf-diff-drawn-"));
|
||||
try {
|
||||
const tags = jobs.map((_, j) => `[d${j}]`).join("");
|
||||
const graph = [
|
||||
`[0:v]${[...clip.filters, "setsar=1", `scale=${width}:${height}:flags=bicubic`].join(",")},split=${jobs.length}${tags}`,
|
||||
...jobs.map(({ box: [x, y, w, h], frames }, j) => {
|
||||
const pick = frames.map((k) => `eq(n\\,${k})`).join("+");
|
||||
return `[d${j}]select=${pick},crop=${w}:${h}:${x}:${y},format=rgb24[e${j}]`;
|
||||
}),
|
||||
].join(";");
|
||||
const outs = jobs.flatMap((_, j) => [
|
||||
...["-map", `[e${j}]`, "-fps_mode", "passthrough", "-f", "rawvideo", `e${j}.rgb`],
|
||||
]);
|
||||
runFfmpeg([...clip.input, "-an", "-filter_complex", graph, ...outs], undefined, work);
|
||||
return jobs.map(({ box: [, , w, h] }, j) => {
|
||||
const raw = readFileSync(join(work, `e${j}.rgb`));
|
||||
const size = w * h * 3;
|
||||
const frames: Uint8Array[] = [];
|
||||
for (let i = 0; i + size <= raw.length; i += size) frames.push(raw.subarray(i, i + size));
|
||||
return frames;
|
||||
});
|
||||
} finally {
|
||||
rmSync(work, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
/** The compared frames (indexes into `ks`) inside [a, b) seconds, thinned to MAX_FPS. */
|
||||
export function drawnFrames(
|
||||
ks: number[],
|
||||
fps: number,
|
||||
a: number,
|
||||
b: number,
|
||||
available: number,
|
||||
): number[] {
|
||||
const stride = Math.max(1, Math.round(fps / MAX_FPS));
|
||||
const inside = ks.flatMap((k, i) =>
|
||||
i < available && k / fps >= a - 1e-6 && k / fps < b - 1e-6 ? [i] : [],
|
||||
);
|
||||
// Every stride-th frame of a run of consecutive frames; moments already that far apart (--at) all count.
|
||||
let last = -Infinity;
|
||||
return inside.filter((i) => {
|
||||
if (ks[i]! - last < stride) return false;
|
||||
last = ks[i]!;
|
||||
return true;
|
||||
});
|
||||
}
|
||||
@@ -1,44 +0,0 @@
|
||||
// Body and camera motion where the pixels differ (a regenerated shot, another person, another room), ported from
|
||||
// the Studio's motion-energy.mjs. Both clips share one 12 fps grid, as mixed 24/30 fps skews per-frame differences.
|
||||
import { meanAbsDiff, pearson, ratioOf, readClip, round, type Clip, type Roi } from "./media.js";
|
||||
|
||||
const ENERGY_RATE = 12;
|
||||
const STILL_SHOT_GREY_LEVELS = 0.1;
|
||||
|
||||
export function energyCurve(clip: Clip, roi?: Roi): number[] {
|
||||
const { frames } = readClip(clip, { fps: ENERGY_RATE, width: 270, gray: true, roi });
|
||||
return frames.slice(1).map((f, i) => meanAbsDiff(f, frames[i]!));
|
||||
}
|
||||
|
||||
export interface EnergyComparison {
|
||||
/** Pearson r of the two curves at the best lag within ±1 s; only meaningful when the action is the same. */
|
||||
r: number;
|
||||
lag_s: number;
|
||||
peak_ratio: number;
|
||||
/** Amount of movement, yours over the reference's. */
|
||||
mean_ratio: number;
|
||||
}
|
||||
|
||||
export function compareEnergy(ref: number[], yours: number[]): EnergyComparison {
|
||||
let best: [number, number] = [-2, 0];
|
||||
for (let lag = -ENERGY_RATE; lag <= ENERGY_RATE; lag++) {
|
||||
const [aa, bb] =
|
||||
lag >= 0
|
||||
? [ref.slice(0, ref.length - lag), yours.slice(lag)]
|
||||
: [ref.slice(-lag), yours.slice(0, yours.length + lag)];
|
||||
if (Math.min(aa.length, bb.length) > 8) {
|
||||
const r = pearson(aa, bb) ?? 0;
|
||||
if (r > best[0]) best = [r, lag];
|
||||
}
|
||||
}
|
||||
const mean = (v: number[]) => v.reduce((s, x) => s + x, 0) / (v.length || 1);
|
||||
return {
|
||||
r: round(best[0] === -2 ? 0 : best[0]),
|
||||
lag_s: round(best[1] / ENERGY_RATE, 2),
|
||||
peak_ratio: round(
|
||||
ratioOf(Math.max(...yours, 0), Math.max(...ref, 0), STILL_SHOT_GREY_LEVELS),
|
||||
2,
|
||||
),
|
||||
mean_ratio: round(ratioOf(mean(yours), mean(ref), STILL_SHOT_GREY_LEVELS), 2),
|
||||
};
|
||||
}
|
||||
@@ -1,107 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, describe, expect, it } from "vitest";
|
||||
import { findFFmpeg, findFFprobe } from "../browser/ffmpeg.js";
|
||||
import { cutClip, cutStill } from "../commands/measure/extract.js";
|
||||
import { detectEvents, lookWindows } from "./events.js";
|
||||
import { probeVideo, readFrames } from "./media.js";
|
||||
|
||||
const ffmpeg = findFFmpeg();
|
||||
const noTools = !ffmpeg || !findFFprobe();
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-events-test-"));
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
|
||||
function clip(name: string, args: string[]): string {
|
||||
const out = join(dir, name);
|
||||
execFileSync(ffmpeg!, ["-hide_banner", "-loglevel", "error", "-y", ...args, out]);
|
||||
return out;
|
||||
}
|
||||
|
||||
describe.skipIf(noTools)("detectEvents", () => {
|
||||
// 25 fps, 3 s: a box appears at 0.52 s, the whole frame flashes white on frame 30 (1.2 s) only, the background
|
||||
// cuts from dark blue to dark green at 2.0 s.
|
||||
const video = noTools
|
||||
? ""
|
||||
: clip("events.mp4", [
|
||||
...["-f", "lavfi", "-i", "color=c=0x101830:s=320x180:r=25:d=2"],
|
||||
...["-f", "lavfi", "-i", "color=c=0x103018:s=320x180:r=25:d=1"],
|
||||
"-filter_complex",
|
||||
"[0][1]concat=n=2:v=1[bg];[bg]" +
|
||||
"drawbox=x=200:y=100:w=60:h=40:color=white:t=fill:enable='gte(n,13)'," +
|
||||
"drawbox=x=0:y=0:w=320:h=180:color=white:t=fill:enable='eq(n,30)'[v]",
|
||||
...["-map", "[v]", "-pix_fmt", "yuv420p", "-c:v", "libx264", "-crf", "0"],
|
||||
]);
|
||||
|
||||
it("tells an appear, a one-frame flash, a cut and a hold apart, each on its frame", () => {
|
||||
const info = probeVideo(video);
|
||||
const found = detectEvents({
|
||||
info,
|
||||
start: 0,
|
||||
end: 3,
|
||||
roi: [0, 0, 320, 180],
|
||||
grid: { cols: 4, rows: 4 },
|
||||
});
|
||||
const of = (type: string) => found.events.filter((e) => e.type === type);
|
||||
expect(of("appear").map((e) => e.frame)).toEqual([13]);
|
||||
expect(of("appear")[0]!.cells).toEqual(["r2c2", "r2c3", "r3c2", "r3c3"]);
|
||||
expect(of("flash").map((e) => [e.frame, e.frames])).toEqual([[30, 1]]);
|
||||
expect(of("cut").map((e) => e.frame)).toEqual([50]);
|
||||
expect(of("hold").length).toBeGreaterThan(0);
|
||||
|
||||
const windows = lookWindows(found.events, info.fps);
|
||||
expect(windows.map((w) => w.what[0]!.split(" ")[0])).toEqual(["appear", "flash", "cut"]);
|
||||
expect(windows[1]!.onset_frames).toEqual([30]);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(noTools)("extract", () => {
|
||||
// Every frame's luma is 40 + 3 × its frame number, with B-frames so the first frame does not start at 0.
|
||||
const video = noTools
|
||||
? ""
|
||||
: clip("counter.mp4", [
|
||||
...[
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"nullsrc=s=64x36:r=25:d=2",
|
||||
"-vf",
|
||||
"geq=lum='40+3*N':cb=128:cr=128",
|
||||
],
|
||||
...["-pix_fmt", "yuv420p", "-c:v", "libx264", "-qp", "0", "-bf", "3"],
|
||||
]);
|
||||
// Grey output is full range; the clip's luma is limited range (16–235).
|
||||
const levelOf = (path: string) => {
|
||||
const { frames } = readFrames(path, { gray: true });
|
||||
return frames.map((f) =>
|
||||
Math.round((((f.reduce((s, v) => s + v, 0) / f.length) * 219) / 255 + 16 - 40) / 3),
|
||||
);
|
||||
};
|
||||
|
||||
it("cuts the still shown at a time", async () => {
|
||||
const out = join(dir, "still.png");
|
||||
const r = await cutStill(probeVideo(video), 0.52, [8, 8, 32, 16], out);
|
||||
expect(r.frame).toBe(13);
|
||||
expect(levelOf(out)).toEqual([13]);
|
||||
});
|
||||
|
||||
it("cuts a clip of exactly the frames in its window at 30 fps too", () => {
|
||||
const thirty = clip("counter30.mp4", [
|
||||
...["-f", "lavfi", "-i", "nullsrc=s=64x36:r=30:d=4", "-vf", "geq=lum='40+3*N':cb=128:cr=128"],
|
||||
...["-pix_fmt", "yuv420p", "-c:v", "libx264", "-qp", "0", "-bf", "3"],
|
||||
]);
|
||||
const out = join(dir, "clip30.mp4");
|
||||
const r = cutClip(probeVideo(thirty), 31 / 30, 37 / 30, [0, 0, 32, 16], out, { crf: 0 });
|
||||
expect(r.frames).toBe(6);
|
||||
expect(levelOf(out)).toEqual([31, 32, 33, 34, 35, 36]);
|
||||
});
|
||||
|
||||
it("cuts a clip of exactly the frames in its window", () => {
|
||||
const out = join(dir, "clip.mp4");
|
||||
const r = cutClip(probeVideo(video), 0.4, 0.8, [0, 0, 33, 17], out, { crf: 0 });
|
||||
expect(r.frames).toBe(10);
|
||||
expect(r.roi).toEqual([0, 0, 32, 16]);
|
||||
expect(levelOf(out)).toEqual([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]);
|
||||
});
|
||||
});
|
||||
@@ -1,436 +0,0 @@
|
||||
// When things change on screen, ported from the Studio's viral-clone events.mjs: from the frame-to-frame grey
|
||||
// difference inside an ROI or each cell of a grid, cuts (most of the frame changes at once), flashes (one or two
|
||||
// frames that return to the picture before), appears and disappears (a region steps from one steady look to
|
||||
// another), holds (still for 0.3 s or more) and other motion, each with its frame number. The map adds the sound's
|
||||
// onsets and beats and groups everything into the windows worth looking at frame by frame.
|
||||
import { findCuts } from "./cuts.js";
|
||||
import { grey, readClip, round, videoClip, writeImage, type Roi, type VideoInfo } from "./media.js";
|
||||
|
||||
// The width motion-rhythm reads motion at, so its still threshold carries over.
|
||||
const EVENT_W = 160;
|
||||
const CUT_SHARE = 0.7;
|
||||
const CUT_GRID = 4;
|
||||
const HOLD_S = 0.3;
|
||||
const STEP_MAX_S = 0.5;
|
||||
const FLASH_FRAMES = 2;
|
||||
const STRIP_MAX = 24;
|
||||
const STRIP_W = 1200;
|
||||
// Events closer than this share one window to look at, up to WINDOW_MAX_S: a talking head changes somewhere all
|
||||
// the time, and one 20 s window shows nothing frame by frame.
|
||||
const WINDOW_GAP_S = 0.25;
|
||||
const WINDOW_MAX_S = 1.5;
|
||||
|
||||
/** A grid cell; rect is x0,y0,x1,y1 in working pixels, end exclusive. */
|
||||
interface Cell {
|
||||
name: string;
|
||||
rect: [number, number, number, number];
|
||||
}
|
||||
|
||||
export type EventType = "cut" | "flash" | "appear" | "disappear" | "change" | "hold";
|
||||
|
||||
export interface ScreenEvent {
|
||||
type: EventType;
|
||||
t: number;
|
||||
frame: number;
|
||||
frames?: number;
|
||||
seconds?: number;
|
||||
cells?: string[];
|
||||
roi?: number[];
|
||||
magnitude?: number;
|
||||
}
|
||||
|
||||
/** An event with the frames and box its strip pair shows (working pixels). */
|
||||
interface Shown extends ScreenEvent {
|
||||
before: number;
|
||||
after: number;
|
||||
rect: number[] | null;
|
||||
}
|
||||
|
||||
function cells(cols: number, rows: number, w: number, h: number): Cell[] {
|
||||
return Array.from({ length: rows * cols }, (_, i) => {
|
||||
const [r, c] = [Math.floor(i / cols), i % cols];
|
||||
return {
|
||||
name: `r${r}c${c}`,
|
||||
rect: [
|
||||
Math.floor((c * w) / cols),
|
||||
Math.floor((r * h) / rows),
|
||||
Math.floor(((c + 1) * w) / cols),
|
||||
Math.floor(((r + 1) * h) / rows),
|
||||
],
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** Mean absolute difference of two frames inside `rect`. */
|
||||
function diff(a: Uint8Array, b: Uint8Array, w: number, [x0, y0, x1, y1]: number[]): number {
|
||||
let s = 0;
|
||||
for (let y = y0!; y < y1!; y++)
|
||||
for (let x = x0!; x < x1!; x++) s += Math.abs(a[y * w + x]! - b[y * w + x]!);
|
||||
return s / Math.max(1, (x1! - x0!) * (y1! - y0!));
|
||||
}
|
||||
|
||||
/** Mean and standard deviation of grey inside `rect`. */
|
||||
function level(f: Uint8Array, w: number, [x0, y0, x1, y1]: number[]) {
|
||||
let [s, s2] = [0, 0];
|
||||
for (let y = y0!; y < y1!; y++)
|
||||
for (let x = x0!; x < x1!; x++) {
|
||||
s += f[y * w + x]!;
|
||||
s2 += f[y * w + x]! ** 2;
|
||||
}
|
||||
const n = Math.max(1, (x1! - x0!) * (y1! - y0!));
|
||||
return { mean: s / n, std: Math.sqrt(Math.max(0, s2 / n - (s / n) ** 2)) };
|
||||
}
|
||||
|
||||
/** Each frame's change from the one before inside `rect` (frame 0: 0). */
|
||||
const changes = (g: Uint8Array[], w: number, rect: number[]) =>
|
||||
g.map((f, i) => (i ? diff(f, g[i - 1]!, w, rect) : 0));
|
||||
|
||||
/** Runs of frames changing by over twice the series' 25th percentile and over 2 grey levels, one quiet frame bridged. */
|
||||
function bursts(d: number[]): [number, number][] {
|
||||
const q = d.slice(1).sort((a, b) => a - b);
|
||||
const act = Math.max(2, 2 * (q[Math.floor(q.length / 4)] ?? 0));
|
||||
const out: [number, number][] = [];
|
||||
for (let i = 1; i < d.length; i++) {
|
||||
if (d[i]! <= act) continue;
|
||||
const last = out.at(-1);
|
||||
if (last && i - last[1] <= 2) last[1] = i;
|
||||
else out.push([i, i]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Still runs of HOLD_S or more by motion-rhythm's still threshold: [first frame of the held picture, its frames]. */
|
||||
function stills(d: number[], fps: number): [number, number][] {
|
||||
const q = d.slice(1).sort((a, b) => a - b);
|
||||
const floor = Math.min(1.5, Math.max(0.5, 2 * (q[Math.floor(q.length / 10)] ?? 0)));
|
||||
const out: [number, number][] = [];
|
||||
for (let i = 1, run = 0; i <= d.length; i++) {
|
||||
if (i < d.length && d[i]! <= floor) run++;
|
||||
else {
|
||||
if (run + 1 >= HOLD_S * fps) out.push([i - run - 1, run + 1]);
|
||||
run = 0;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Joins items whose `near(a, b)` holds, transitively. */
|
||||
function groups<T>(items: T[], near: (a: T, b: T) => boolean): T[][] {
|
||||
const root = items.map((_, i) => i);
|
||||
const find = (i: number): number => (root[i] === i ? i : (root[i] = find(root[i]!)));
|
||||
for (let i = 0; i < items.length; i++)
|
||||
for (let j = i + 1; j < items.length; j++)
|
||||
if (near(items[i]!, items[j]!)) root[find(i)] = find(j);
|
||||
const out = new Map<number, T[]>();
|
||||
items.forEach((it, i) => out.set(find(i), [...(out.get(find(i)) ?? []), it]));
|
||||
return [...out.values()];
|
||||
}
|
||||
|
||||
/**
|
||||
* Frames where CUT_SHARE of the cells change by `thr` or more at once. Two such frames a frame or two apart whose
|
||||
* second returns to the picture before the first are a flash, not two cuts.
|
||||
*/
|
||||
function cutFrames(g: Uint8Array[], w: number, cutCells: Cell[], thr: number) {
|
||||
const d = cutCells.map((c) => changes(g, w, c.rect));
|
||||
const hits = g
|
||||
.map((_, i) => i)
|
||||
.filter((i) => i > 0 && d.filter((x) => x[i]! >= thr).length >= CUT_SHARE * cutCells.length);
|
||||
const cuts: [number, number][] = [];
|
||||
const flashes: [number, number][] = [];
|
||||
for (let k = 0; k < hits.length; k++) {
|
||||
const [i, j] = [hits[k]!, hits[k + 1]];
|
||||
const last = cuts.at(-1);
|
||||
if (
|
||||
j !== undefined &&
|
||||
j - i <= FLASH_FRAMES &&
|
||||
diff(g[j]!, g[i - 1]!, w, [0, 0, w, g[0]!.length / w]) < thr
|
||||
) {
|
||||
flashes.push([i, j - 1]);
|
||||
k++;
|
||||
} else if (last && i === last[1] + 1)
|
||||
last[1] = i; // a whip or a short dissolve: one cut over several frames
|
||||
else cuts.push([i, i]);
|
||||
}
|
||||
return { cuts, flashes, d };
|
||||
}
|
||||
|
||||
interface Burst {
|
||||
ci: number;
|
||||
c: Cell;
|
||||
a: number;
|
||||
b: number;
|
||||
net: number;
|
||||
peak: number;
|
||||
}
|
||||
|
||||
/** Each region's bursts outside the cuts and flashes, split into steps, local flashes and other motion. */
|
||||
// fallow-ignore-next-line complexity
|
||||
function bursting(
|
||||
g: Uint8Array[],
|
||||
w: number,
|
||||
regions: Cell[],
|
||||
skip: (a: number, b: number) => boolean,
|
||||
thr: number,
|
||||
fps: number,
|
||||
) {
|
||||
const settle = Math.round(HOLD_S * fps);
|
||||
const steps: Burst[] = [];
|
||||
const flashes: Burst[] = [];
|
||||
const moves: Burst[] = [];
|
||||
for (const [ci, c] of regions.entries()) {
|
||||
const d = changes(g, w, c.rect);
|
||||
for (const [a, b] of bursts(d)) {
|
||||
if (skip(a, b)) continue;
|
||||
const net = diff(g[b]!, g[a - 1]!, w, c.rect);
|
||||
const peak = Math.max(...d.slice(a, b + 1));
|
||||
const burst = { ci, c, a, b, net, peak };
|
||||
// a flash lights up for a frame or two and the region returns to how it was
|
||||
if (
|
||||
b - a + 1 <= FLASH_FRAMES &&
|
||||
peak >= 2 * thr &&
|
||||
b + 1 < g.length &&
|
||||
diff(g[b + 1]!, g[a - 1]!, w, c.rect) < thr
|
||||
) {
|
||||
flashes.push(burst);
|
||||
continue;
|
||||
}
|
||||
// a step joins two looks that each last: the region moves under a third as much in the 0.3 s either side
|
||||
const was = diff(g[a - 1]!, g[Math.max(0, a - 1 - settle)]!, w, c.rect);
|
||||
const now = diff(g[b]!, g[Math.min(g.length - 1, b + settle)]!, w, c.rect);
|
||||
const step = net >= thr && b - a + 1 <= STEP_MAX_S * fps && Math.max(was, now) <= net / 3;
|
||||
(step ? steps : moves).push(burst);
|
||||
}
|
||||
}
|
||||
return { steps, flashes, moves };
|
||||
}
|
||||
|
||||
export interface DetectOptions {
|
||||
info: VideoInfo;
|
||||
start: number;
|
||||
end: number;
|
||||
/** One region, or (when `grid` is given) the whole frame split into cells. */
|
||||
roi: Roi;
|
||||
grid?: { cols: number; rows: number };
|
||||
thr?: number;
|
||||
}
|
||||
|
||||
export interface Detected {
|
||||
fps: number;
|
||||
frames: number;
|
||||
events: ScreenEvent[];
|
||||
/** Working-size RGB frames and the events a strip should show, for drawing. */
|
||||
rgb: Uint8Array[];
|
||||
w: number;
|
||||
h: number;
|
||||
shown: Shown[];
|
||||
}
|
||||
|
||||
/** Every cut, flash, step, change and hold of a window of a video. */
|
||||
export function detectEvents(opts: DetectOptions): Detected {
|
||||
const { info, start, end, roi, grid } = opts;
|
||||
const thr = opts.thr ?? 5;
|
||||
const fps = info.fps;
|
||||
const whole: Roi = [0, 0, info.width, info.height];
|
||||
const sameAsWhole = roi.every((v, i) => v === whole[i]);
|
||||
const clip = videoClip(info.path, start, end - start, info);
|
||||
const { frames: rgb, w, h } = readClip(clip, { roi, width: EVENT_W });
|
||||
const g = rgb.map(grey);
|
||||
const wholeG = sameAsWhole ? g : readClip(clip, { width: EVENT_W, gray: true }).frames;
|
||||
const wholeH = wholeG[0] ? wholeG[0].length / EVENT_W : h;
|
||||
const f0 = Math.ceil(start * fps - 1e-6);
|
||||
const at = (i: number) => ({ frame: f0 + i, t: round((f0 + i) / fps, 3) });
|
||||
if (g.length < 3) return { fps, frames: g.length, events: [], rgb, w, h, shown: [] };
|
||||
|
||||
// Cuts are read on the whole frame: the grid's own cells, or with an ROI a 4x4 grid of the whole frame.
|
||||
const cutCells = cells(grid?.cols ?? CUT_GRID, grid?.rows ?? CUT_GRID, EVENT_W, wholeH);
|
||||
const cut = cutFrames(wholeG, EVENT_W, cutCells, thr);
|
||||
// A fast handheld move or whip changes most cells on frame after frame: motion, not a run of cuts. A cut is one
|
||||
// the brightness test diff uses also finds (cuts.ts), or a change that stands alone in time.
|
||||
const real = new Set(
|
||||
findCuts(wholeG).flatMap((c) => Array.from({ length: c.frames }, (_, i) => c.frame + i)),
|
||||
);
|
||||
const alone = ([i, j]: [number, number]) =>
|
||||
j === i && cut.cuts.every(([a, b]) => (a === i && b === j) || b < i - 3 || a > j + 3);
|
||||
cut.cuts = cut.cuts.filter((c) => real.has(c[0]) || alone(c));
|
||||
const single: Cell = { name: "roi", rect: [0, 0, w, h] };
|
||||
const regions = grid ? cells(grid.cols, grid.rows, w, h) : [single];
|
||||
const covered = (a: number, b: number) =>
|
||||
[...cut.cuts, ...cut.flashes].some(([x, y]) => x <= b && a <= y);
|
||||
const { steps, flashes, moves } = bursting(g, w, regions, covered, thr, fps);
|
||||
const cols = grid?.cols ?? 1;
|
||||
const touching = (s: Burst, o: Burst) =>
|
||||
Math.abs(Math.floor(s.ci / cols) - Math.floor(o.ci / cols)) <= 1 &&
|
||||
Math.abs((s.ci % cols) - (o.ci % cols)) <= 1;
|
||||
const overlap = (s: Burst, o: Burst) => s.a <= o.b + 2 && o.a <= s.b + 2;
|
||||
const [kx, ky] = [roi[2] / w, roi[3] / h];
|
||||
const event = (grp: Burst[], type: EventType, size: number): Shown => {
|
||||
const [a, b] = [Math.min(...grp.map((s) => s.a)), Math.max(...grp.map((s) => s.b))];
|
||||
const rect = [0, 1, 2, 3].map((k) =>
|
||||
(k < 2 ? Math.min : Math.max)(...grp.map((s) => s.c.rect[k]!)),
|
||||
);
|
||||
const where = grid
|
||||
? {
|
||||
cells: [...new Set(grp.map((s) => s.c.name))].sort(),
|
||||
roi: [
|
||||
rect[0]! * kx + roi[0],
|
||||
rect[1]! * ky + roi[1],
|
||||
(rect[2]! - rect[0]!) * kx,
|
||||
(rect[3]! - rect[1]!) * ky,
|
||||
].map(Math.round),
|
||||
}
|
||||
: {};
|
||||
return {
|
||||
type,
|
||||
...at(a),
|
||||
frames: b - a + 1,
|
||||
...where,
|
||||
magnitude: round(size, 1),
|
||||
before: a - 1,
|
||||
after: b,
|
||||
rect: grid ? rect : null,
|
||||
};
|
||||
};
|
||||
const stepEvents = groups(steps, (s, o) => overlap(s, o) && touching(s, o)).map((grp) => {
|
||||
const e = event(grp, "change", grp.reduce((m, s) => m + s.net, 0) / grp.length);
|
||||
const box = e.rect ?? [0, 0, w, h];
|
||||
const [was, now] = [level(g[e.before]!, w, box), level(g[e.after]!, w, box)];
|
||||
const up = Math.abs(now.std - was.std) >= 2 ? now.std > was.std : now.mean > was.mean;
|
||||
return { ...e, type: up ? ("appear" as const) : ("disappear" as const) };
|
||||
});
|
||||
const flashEvents = [
|
||||
...groups(flashes, (s, o) => overlap(s, o) && touching(s, o)).map((grp) =>
|
||||
event(grp, "flash", Math.max(...grp.map((s) => s.peak))),
|
||||
),
|
||||
...cut.flashes.map(([i, j]) => ({
|
||||
type: "flash" as const,
|
||||
...at(i),
|
||||
frames: j - i + 1,
|
||||
magnitude: round(cut.d.reduce((m, d) => m + d[i]!, 0) / cut.d.length, 1),
|
||||
before: i - 1,
|
||||
after: i,
|
||||
rect: null,
|
||||
})),
|
||||
];
|
||||
const moveEvents = groups(moves, overlap).map((grp) =>
|
||||
event(grp, "change", Math.max(...grp.map((s) => s.peak))),
|
||||
);
|
||||
const cutEvents: Shown[] = cut.cuts.map(([i, j]) => ({
|
||||
type: "cut",
|
||||
...at(i),
|
||||
...(j > i ? { frames: j - i + 1 } : {}),
|
||||
magnitude: round(cut.d.reduce((m, d) => m + d[i]!, 0) / cut.d.length, 1),
|
||||
before: i - 1,
|
||||
after: j,
|
||||
rect: null,
|
||||
}));
|
||||
const still = changes(grid ? wholeG : g, EVENT_W, [0, 0, EVENT_W, grid ? wholeH : h]);
|
||||
const held: ScreenEvent[] = stills(still, fps).map(([i, n]) => ({
|
||||
type: "hold",
|
||||
...at(i),
|
||||
seconds: round(n / fps, 2),
|
||||
}));
|
||||
|
||||
// The strip keeps the biggest: cuts and flashes, then steps by cells × magnitude, then changes by magnitude.
|
||||
const weight = (e: Shown) => (e.cells?.length ?? 1) * (e.magnitude ?? 0);
|
||||
const order = (x: ScreenEvent, y: ScreenEvent) => x.frame - y.frame;
|
||||
const shown = [
|
||||
...cutEvents,
|
||||
...flashEvents,
|
||||
...[...stepEvents].sort((x, y) => weight(y) - weight(x)),
|
||||
...[...moveEvents].sort((x, y) => (y.magnitude ?? 0) - (x.magnitude ?? 0)),
|
||||
]
|
||||
.slice(0, STRIP_MAX)
|
||||
.sort(order);
|
||||
const plain = ({ before: _b, after: _a, rect: _r, ...e }: Shown): ScreenEvent => e;
|
||||
const events = [...cutEvents, ...flashEvents, ...stepEvents, ...moveEvents].map(plain);
|
||||
return { fps, frames: g.length, events: [...events, ...held].sort(order), rgb, w, h, shown };
|
||||
}
|
||||
|
||||
/** One picture of each shown event's before|after pair, its region boxed in red. */
|
||||
export function drawStrip(d: Detected, out: string): string {
|
||||
const { rgb, w, h, shown } = d;
|
||||
const per = Math.max(1, Math.floor(STRIP_W / (2 * w + 12)));
|
||||
const cw = per * (2 * w + 12);
|
||||
const ch = Math.ceil(shown.length / per) * (h + 8);
|
||||
const px = new Uint8Array(cw * ch * 3).fill(255);
|
||||
shown.forEach((e, k) => {
|
||||
const [ox, oy] = [(k % per) * (2 * w + 12), Math.floor(k / per) * (h + 8)];
|
||||
[e.before, e.after].forEach((f, side) => {
|
||||
const x0 = ox + side * (w + 2);
|
||||
for (let y = 0; y < h; y++)
|
||||
px.set(rgb[f]!.subarray(y * w * 3, (y + 1) * w * 3), ((oy + y) * cw + x0) * 3);
|
||||
if (!e.rect) return;
|
||||
const [rx0, ry0, rx1, ry1] = e.rect;
|
||||
for (let y = ry0!; y < ry1!; y++)
|
||||
for (let x = rx0!; x < rx1!; x++)
|
||||
if (y === ry0 || y === ry1! - 1 || x === rx0 || x === rx1! - 1)
|
||||
px.set([255, 0, 0], ((oy + y) * cw + x0 + x) * 3);
|
||||
});
|
||||
});
|
||||
return writeImage(out, px, cw, ch);
|
||||
}
|
||||
|
||||
export interface LookWindow {
|
||||
start: number;
|
||||
end: number;
|
||||
/** What happens, e.g. "cut", "appear r1c2,r1c3", "flash 1f". */
|
||||
what: string[];
|
||||
/** Sound onsets inside the window, seconds. */
|
||||
onsets?: number[];
|
||||
/** The frames its events start on: a strip of the window must show these. */
|
||||
onset_frames: number[];
|
||||
}
|
||||
|
||||
/** Every event but holds, merged into the windows worth looking at frame by frame, two frames padded either side. */
|
||||
export function lookWindows(
|
||||
events: ScreenEvent[],
|
||||
fps: number,
|
||||
onsets: number[] = [],
|
||||
): LookWindow[] {
|
||||
const pad = 2 / fps;
|
||||
const spans = events
|
||||
.filter((e) => e.type !== "hold")
|
||||
.map((e) => {
|
||||
const where = e.cells ? ` ${e.cells.join(",")}` : "";
|
||||
const len = e.type === "flash" ? ` ${e.frames}f` : "";
|
||||
return {
|
||||
a: e.t - pad,
|
||||
b: e.t + ((e.frames ?? 1) - 1) / fps + pad,
|
||||
what: `${e.type}${where}${len}`,
|
||||
frame: e.frame,
|
||||
};
|
||||
})
|
||||
.sort((x, y) => x.a - y.a);
|
||||
const out: Omit<LookWindow, "onsets">[] = [];
|
||||
for (const s of spans) {
|
||||
const last = out.at(-1);
|
||||
if (last && s.a - last.end <= WINDOW_GAP_S && s.b - last.start <= WINDOW_MAX_S) {
|
||||
last.end = Math.max(last.end, s.b);
|
||||
last.what.push(s.what);
|
||||
last.onset_frames.push(s.frame);
|
||||
} else out.push({ start: s.a, end: s.b, what: [s.what], onset_frames: [s.frame] });
|
||||
}
|
||||
return out.map((wnd) => {
|
||||
const hits = onsets.filter((t) => t >= wnd.start && t <= wnd.end);
|
||||
return {
|
||||
start: round(Math.max(0, wnd.start), 3),
|
||||
end: round(wnd.end, 3),
|
||||
what: wnd.what,
|
||||
...(hits.length ? { onsets: hits.map((t) => round(t, 3)) } : {}),
|
||||
onset_frames: [...new Set(wnd.onset_frames)].sort((x, y) => x - y),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** How many of each type, for a summary line. */
|
||||
export function countEvents(events: ScreenEvent[]): Record<EventType, number> {
|
||||
const count = (type: EventType) => events.filter((e) => e.type === type).length;
|
||||
return {
|
||||
cut: count("cut"),
|
||||
flash: count("flash"),
|
||||
appear: count("appear"),
|
||||
disappear: count("disappear"),
|
||||
change: count("change"),
|
||||
hold: count("hold"),
|
||||
};
|
||||
}
|
||||
@@ -1,149 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { cpSync, existsSync, mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir, userInfo } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, describe, expect, it } from "vitest";
|
||||
import { findFFmpeg, findFFprobe } from "../browser/ffmpeg.js";
|
||||
import { againstStart, faceBoxes, faceFit, minWindow, readSamples, sampleFrames } from "./face.js";
|
||||
import { probeVideo, runFfmpeg } from "./media.js";
|
||||
import { OPENCV_DIR } from "./opencv.js";
|
||||
|
||||
// As in opencv.test.ts: borrow this machine's verified opencv.js cache, skip without one.
|
||||
const cached = join(userInfo().homedir, ".cache", "hyperframes", "opencv", "4.10.0");
|
||||
if (existsSync(join(cached, "opencv.js")) && !existsSync(join(OPENCV_DIR, "opencv.js")))
|
||||
cpSync(cached, OPENCV_DIR, { recursive: true });
|
||||
const media = !!findFFmpeg() && !!findFFprobe();
|
||||
const opencv = media && existsSync(join(OPENCV_DIR, "opencv.js"));
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-face-test-"));
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
|
||||
describe("sampleFrames", () => {
|
||||
it("spreads the samples evenly, the last frame included, each once", () => {
|
||||
expect(sampleFrames(30, 0, 1, 4)).toEqual([0, 10, 19, 29]);
|
||||
expect(sampleFrames(30, 2, 2.1, 12)).toEqual([60, 61, 62]);
|
||||
expect(sampleFrames(24, 1, 1.01, 8)).toEqual([24]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("minWindow", () => {
|
||||
it("is a step of the cascades' 20 px x 1.1 ladder, no larger than the share", () => {
|
||||
expect(minWindow(1280, 0)).toBe(20);
|
||||
expect(minWindow(400, 0.04)).toBe(20);
|
||||
// 4% of 1280 is 51.2 px: 20 x 1.1^9 = 47.2 is the last step under it.
|
||||
expect(minWindow(1280, 0.04)).toBe(47);
|
||||
expect(minWindow(1080, 0.04)).toBe(43);
|
||||
});
|
||||
});
|
||||
|
||||
describe("againstStart", () => {
|
||||
it("reads a generated shot from its own start and a full render on film time", () => {
|
||||
// A 5.2 s HeyGen shot for the original's 9.5-12.27 s: its face is at 0-2.77 s of the clip, not 9.5.
|
||||
expect(againstStart(5.17, 12.27, 9.5)).toBe(0);
|
||||
// A shot longer than the window but shorter than the film is still its own clip.
|
||||
expect(againstStart(7.29, 12.27, 2.77)).toBe(0);
|
||||
expect(againstStart(12.27, 12.27, 9.5)).toBe(9.5);
|
||||
expect(againstStart(12.3, 12.27, 9.5)).toBe(9.5);
|
||||
});
|
||||
});
|
||||
|
||||
describe("faceFit", () => {
|
||||
const at = (t: number, ...boxes: number[][]) => ({ t, boxes });
|
||||
const canvas = { width: 720, height: 1280 };
|
||||
|
||||
it("gives the scale about the centre and the x, y that land the clip's face on the original's", () => {
|
||||
// The clip's face, 0.1 high at (0.5, 0.4); the original's twice the size, 10 px right and 20 px up of where
|
||||
// scaling about the centre puts it: centre y 640 + 2 (512 - 640) - 20 = 364.
|
||||
const clip = [0, 0.5, 1, 1.5].map((t) => at(t, [0.45, 0.35, 0.1, 0.1]));
|
||||
const [w, h] = [0.2, 0.2];
|
||||
const [cx, cy] = [(360 + 10) / 720, 364 / 1280];
|
||||
const original = [0, 0.5, 1, 1.5].map((t) =>
|
||||
at(t, [cx - w / 2, cy - h / 2, w, h], [0.05, 0.05, 0.03, 0.03]),
|
||||
);
|
||||
const fit = faceFit(original, clip, canvas, canvas);
|
||||
expect(fit.moments).toBe(4);
|
||||
expect(fit.transform).toEqual({ scale: 2, x: 10, y: -20 });
|
||||
expect(fit.residual).toBe(0);
|
||||
});
|
||||
|
||||
it("fits a clip of another size through object-fit: cover, and shrugs off a stray moment", () => {
|
||||
// 768x1344 under a 720x1280 canvas: cover scales it by 1280/1344 and crops its sides; its face then sits
|
||||
// where the original's is, at the same height, so nothing moves.
|
||||
const s0 = 1280 / 1344;
|
||||
const ox = (720 - 768 * s0) / 2;
|
||||
const [fx, fy, fh] = [360, 500, 200];
|
||||
const clipBox = [
|
||||
(fx - ox - fh / 2) / (768 * s0),
|
||||
(fy - fh / 2) / 1280,
|
||||
fh / (768 * s0),
|
||||
fh / 1280,
|
||||
];
|
||||
const origBox = [(fx - fh / 2) / 720, (fy - fh / 2) / 1280, fh / 720, fh / 1280];
|
||||
const clip = [0, 0.25, 0.5, 0.75, 1].map((t) => at(t, clipBox));
|
||||
const original = [0, 0.25, 0.5, 0.75, 1].map((t, i) =>
|
||||
at(t + 0.02, i === 2 ? [0.1, 0.1, 0.4, 0.4] : origBox),
|
||||
);
|
||||
const fit = faceFit(original, clip, canvas, { width: 768, height: 1344 });
|
||||
expect(fit.transform).toEqual({ scale: 1, x: 0, y: 0 });
|
||||
});
|
||||
|
||||
it("gives no transform without faces on both sides at enough moments", () => {
|
||||
const fit = faceFit(
|
||||
[at(0, [0.4, 0.3, 0.2, 0.1]), at(1)],
|
||||
[at(0, [0.4, 0.3, 0.2, 0.1]), at(1, [0.4, 0.3, 0.2, 0.1])],
|
||||
canvas,
|
||||
canvas,
|
||||
);
|
||||
expect(fit).toEqual({ moments: 1, transform: null });
|
||||
// Moments farther apart than a tenth of a second do not pair.
|
||||
expect(
|
||||
faceFit(
|
||||
[at(0, [0.4, 0.3, 0.2, 0.1]), at(1, [0.4, 0.3, 0.2, 0.1])],
|
||||
[at(0.5, [0.4, 0.3, 0.2, 0.1]), at(1.5, [0.4, 0.3, 0.2, 0.1])],
|
||||
canvas,
|
||||
canvas,
|
||||
).transform,
|
||||
).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!media)("readSamples", () => {
|
||||
// Every frame different, keyframes 2.5 s apart and B-frames: what a seek has to get right.
|
||||
const video = join(dir, "frames.mp4");
|
||||
if (media)
|
||||
execFileSync(findFFmpeg()!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "testsrc2=s=160x96:r=30:d=8"],
|
||||
...["-c:v", "libx264", "-g", "75", "-bf", "2", "-pix_fmt", "yuv420p", video],
|
||||
]);
|
||||
|
||||
it("seeks to the same frames that decoding from the start picks", () => {
|
||||
const info = probeVideo(video);
|
||||
// Nearby samples (one pass) and samples seconds apart (a seek each), off and on keyframes.
|
||||
const ks = [1, 2, 40, 74, 75, 76, 140, 141, 210, 239];
|
||||
const got = readSamples(info, ks, 80, 48);
|
||||
const raw = runFfmpeg([
|
||||
...["-i", video, "-an", "-vf"],
|
||||
`select=${ks.map((k) => `eq(n\\,${k})`).join("+")},scale=80:48,format=gray`,
|
||||
...["-fps_mode", "passthrough", "-f", "rawvideo", "-"],
|
||||
]);
|
||||
expect(got).toHaveLength(ks.length);
|
||||
got.forEach((f, i) =>
|
||||
expect(Buffer.compare(f, raw.subarray(i * 80 * 48, (i + 1) * 80 * 48))).toBe(0),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!opencv)("faceBoxes", () => {
|
||||
const video = join(dir, "blank.mp4");
|
||||
if (opencv)
|
||||
execFileSync(findFFmpeg()!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "color=c=0x406080:s=320x568:r=30:d=4"],
|
||||
...["-c:v", "libx264", "-pix_fmt", "yuv420p", video],
|
||||
]);
|
||||
|
||||
it("labels every sampled frame and finds no face in a blank one", async () => {
|
||||
const r = await faceBoxes(probeVideo(video), 1, 3, 5);
|
||||
expect(r.map((s) => s.frame)).toEqual([30, 45, 60, 74, 89]);
|
||||
expect(r.map((s) => s.t)).toEqual([1, 1.5, 2, 2.467, 2.967]);
|
||||
expect(r.every((s) => s.boxes.length === 0)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -1,255 +0,0 @@
|
||||
// How big the face is and where, ported from hf-media's face_size: OpenCV's frontal and profile Haar cascades on
|
||||
// evenly spaced frames, graded by face height over frame height. A regenerated presenter keeps the likeness from
|
||||
// FACE_HOLDS up; below FACE_FLOOR a video model cannot carry one at all (the harness's face-size eval, 2026-10-08).
|
||||
import { frameAt, round, runFfmpeg, type VideoInfo } from "./media.js";
|
||||
import { greyMat, loadOpenCv } from "./opencv.js";
|
||||
|
||||
const FACE_HOLDS = 0.25;
|
||||
const FACE_SLIDES = 0.15;
|
||||
const FACE_FLOOR = 0.06;
|
||||
/**
|
||||
* The smallest face looked for by default, as a share of the frame's height: well under FACE_FLOOR, and it skips the
|
||||
* cascades' costliest windows. A minimum on their ladder keeps the full search's boxes; scaling the frame moves them.
|
||||
*/
|
||||
const FACE_MIN = 0.04;
|
||||
const WORK_W = 960;
|
||||
const STEP = 1.1;
|
||||
const WINDOW_PX = 20;
|
||||
// Samples closer than this share one decode from the first of them; farther ones get a seek of their own.
|
||||
const RUN_S = 2;
|
||||
|
||||
type Verdict = "pass" | "warn" | "fail" | "unmeasured";
|
||||
const ORDER: Verdict[] = ["fail", "unmeasured", "warn", "pass"];
|
||||
|
||||
export function faceVerdict(height: number | null, cast = false): Verdict {
|
||||
if (height === null) return "unmeasured";
|
||||
if (height >= FACE_HOLDS) return "pass";
|
||||
return height < (cast ? FACE_SLIDES : FACE_FLOOR) ? "fail" : "warn";
|
||||
}
|
||||
|
||||
/** `samples` frame indices spread evenly over [start, end), the last frame included. */
|
||||
export function sampleFrames(fps: number, start: number, end: number, samples: number): number[] {
|
||||
const k0 = frameAt(start, fps);
|
||||
const k1 = Math.max(k0, Math.ceil(end * fps - 1e-6) - 1);
|
||||
const n = Math.min(samples, k1 - k0 + 1);
|
||||
return [
|
||||
...new Set(
|
||||
Array.from({ length: n }, (_, i) => k0 + Math.round((i * (k1 - k0)) / Math.max(1, n - 1))),
|
||||
),
|
||||
];
|
||||
}
|
||||
|
||||
/** The largest window on the cascades' 20 px x 1.1^k ladder that is no more than `share` of a frame `h` high. */
|
||||
export function minWindow(h: number, share: number): number {
|
||||
const steps = Math.floor(Math.log(Math.max(1, (share * h) / WINDOW_PX)) / Math.log(STEP) + 1e-9);
|
||||
return Math.round(WINDOW_PX * STEP ** steps);
|
||||
}
|
||||
|
||||
/**
|
||||
* Frames `ks` (indices from the first frame) as grey bytes at `w` x `h`, which must keep the video's aspect. Each run
|
||||
* of nearby frames is one ffmpeg pass that seeks to its first frame and decodes only to its last; a variable-rate
|
||||
* video, whose indices a seek cannot find, is read from its start.
|
||||
*/
|
||||
export function readSamples(info: VideoInfo, ks: number[], w: number, h: number): Uint8Array[] {
|
||||
const runs: number[][] = [];
|
||||
for (const k of ks) {
|
||||
const run = runs.at(-1);
|
||||
if (run && (info.vfr || (k - run.at(-1)!) / info.fps <= RUN_S)) run.push(k);
|
||||
else runs.push([k]);
|
||||
}
|
||||
const size = w * h;
|
||||
return runs.flatMap((run) => {
|
||||
// Half a frame early: an input seek drops the frames before it, so the first one left is frame run[0].
|
||||
const k0 = info.vfr ? 0 : run[0]!;
|
||||
const seek = k0 > 0 ? ["-ss", ((k0 - 0.5) / info.fps).toFixed(6)] : [];
|
||||
const filters = [
|
||||
`select=${run.map((k) => `eq(n\\,${k - k0})`).join("+")}`,
|
||||
...(w !== info.width || h !== info.height ? [`scale=${w}:${h}`] : []),
|
||||
"format=gray",
|
||||
];
|
||||
const raw = runFfmpeg([
|
||||
...seek,
|
||||
"-i",
|
||||
info.path,
|
||||
"-an",
|
||||
"-vf",
|
||||
filters.join(","),
|
||||
"-fps_mode",
|
||||
"passthrough",
|
||||
"-frames:v",
|
||||
String(run.length),
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-",
|
||||
]);
|
||||
const frames: Uint8Array[] = [];
|
||||
for (let i = 0; i + size <= raw.length; i += size) frames.push(raw.subarray(i, i + size));
|
||||
return frames;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Each sampled frame's faces as [x, y, w, h] shares of the frame. `minFace` is the smallest face looked for as a
|
||||
* share of the frame's height (default FACE_MIN; 0 looks down to the cascades' 20 px, for crowds).
|
||||
*/
|
||||
export async function faceBoxes(
|
||||
info: VideoInfo,
|
||||
start: number,
|
||||
end: number,
|
||||
samples: number,
|
||||
opts: { minFace?: number } = {},
|
||||
) {
|
||||
const cv = await loadOpenCv((m) => console.error(m));
|
||||
const ks = sampleFrames(info.fps, start, end, samples);
|
||||
const w = Math.min(WORK_W, info.width);
|
||||
const h =
|
||||
w === info.width
|
||||
? info.height
|
||||
: Math.max(2, Math.round((info.height * w) / info.width / 2) * 2);
|
||||
const frames = readSamples(info, ks, w, h);
|
||||
const min = minWindow(h, opts.minFace ?? FACE_MIN);
|
||||
const cascades = ["haarcascade_frontalface_alt2.xml", "haarcascade_profileface.xml"].map(
|
||||
(name) => {
|
||||
const c = new cv.CascadeClassifier();
|
||||
c.load(name);
|
||||
return c;
|
||||
},
|
||||
);
|
||||
try {
|
||||
return frames.map((px, i) => {
|
||||
const gray = greyMat(cv, px, w, h);
|
||||
const boxes: number[][] = [];
|
||||
try {
|
||||
for (const c of cascades) {
|
||||
const found = new cv.RectVector();
|
||||
c.detectMultiScale(gray, found, STEP, 5, 0, new cv.Size(min, min));
|
||||
for (let j = 0; j < found.size(); j++) {
|
||||
const r = found.get(j);
|
||||
boxes.push([r.x / w, r.y / h, r.width / w, r.height / h]);
|
||||
}
|
||||
found.delete();
|
||||
}
|
||||
} finally {
|
||||
gray.delete();
|
||||
}
|
||||
return { frame: ks[i]!, t: round(ks[i]! / info.fps, 3), boxes };
|
||||
});
|
||||
} finally {
|
||||
for (const c of cascades) c.delete();
|
||||
}
|
||||
}
|
||||
|
||||
/** The subject's face among a frame's: the largest, or the smallest when the person is cast in a crowd. */
|
||||
function subjectFace(boxes: number[][], cast = false): number[] | null {
|
||||
return boxes.reduce<number[] | null>(
|
||||
(a, b) => (!a || (cast ? b[3]! < a[3]! : b[3]! > a[3]!) ? b : a),
|
||||
null,
|
||||
);
|
||||
}
|
||||
|
||||
/** The subject's face (largest, or the smallest when the person is cast in a crowd), its median and last height. */
|
||||
export function faceSummary(
|
||||
samples: { frame: number; t: number; boxes: number[][] }[],
|
||||
cast = false,
|
||||
) {
|
||||
const pick = (boxes: number[][]) => subjectFace(boxes, cast);
|
||||
const seen = samples.flatMap((s) => {
|
||||
const b = pick(s.boxes);
|
||||
return b ? [{ t: s.t, box: b.map((v) => round(v, 4)) }] : [];
|
||||
});
|
||||
const heights = seen.map((s) => s.box[3]!).sort((a, b) => a - b);
|
||||
const height = heights.length ? heights[heights.length >> 1]! : null;
|
||||
const lastFace = pick(samples.at(-1)?.boxes ?? []);
|
||||
const last = lastFace ? lastFace[3]! : null;
|
||||
const verdicts = [faceVerdict(height, cast), faceVerdict(last, cast)];
|
||||
return {
|
||||
frames_sampled: samples.length,
|
||||
frames_with_face: seen.length,
|
||||
face_height: height === null ? null : round(height, 3),
|
||||
last_face_height: last === null ? null : round(last, 3),
|
||||
verdict: verdicts.reduce((a, b) => (ORDER.indexOf(b) < ORDER.indexOf(a) ? b : a)),
|
||||
bands: { pass: FACE_HOLDS, cast_fail_below: FACE_SLIDES, fail_below: FACE_FLOOR },
|
||||
boxes: seen,
|
||||
...(last === null
|
||||
? {
|
||||
note: "no face at the last sampled frame (turned away, covered or missed): look at the frames",
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
const median = (v: number[]) => [...v].sort((x, y) => x - y)[v.length >> 1]!;
|
||||
|
||||
/** Fewest moments with a face on both sides that a fit is given from. */
|
||||
export const FIT_MIN_FACES = 2;
|
||||
const PAIR_S = 0.1;
|
||||
|
||||
/**
|
||||
* Where --against starts reading the clip: a render as long as the original runs on film time, so at --from; a
|
||||
* shot of its own (a generated clip) starts at its own 0.
|
||||
*/
|
||||
export function againstStart(clipDuration: number, originalDuration: number, from: number): number {
|
||||
return Math.abs(clipDuration - originalDuration) <= 0.5 ? from : 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* The GSAP transform that puts a clip's face on the original's: `scale` about the clip element's centre, then `x`, `y`
|
||||
* in the original's (the canvas's) pixels, for a clip element that fills the canvas with object-fit: cover. Moments
|
||||
* pair by time (`t` from each side's window start, within PAIR_S); each side's subject face is taken per moment and
|
||||
* their centres and heights medianed, so one stray detection does not move the fit. `residual` is the median distance
|
||||
* left between the faces after it, in the original's face heights.
|
||||
*/
|
||||
export function faceFit(
|
||||
original: { t: number; boxes: number[][] }[],
|
||||
clip: { t: number; boxes: number[][] }[],
|
||||
canvas: { width: number; height: number },
|
||||
clipSize: { width: number; height: number },
|
||||
cast = false,
|
||||
) {
|
||||
const pairs = original.flatMap((o) => {
|
||||
const near = clip.reduce<(typeof clip)[number] | null>(
|
||||
(p, q) => (!p || Math.abs(q.t - o.t) < Math.abs(p.t - o.t) ? q : p),
|
||||
null,
|
||||
);
|
||||
if (!near || Math.abs(near.t - o.t) > PAIR_S) return [];
|
||||
const [a, b] = [subjectFace(o.boxes, cast), subjectFace(near.boxes, cast)];
|
||||
return a && b ? [{ a, b }] : [];
|
||||
});
|
||||
if (pairs.length < FIT_MIN_FACES) return { moments: pairs.length, transform: null };
|
||||
const { width: W, height: H } = canvas;
|
||||
// object-fit: cover, centred: the clip's pixel (u, v) lands on the canvas at (ox + u s0, oy + v s0).
|
||||
const s0 = Math.max(W / clipSize.width, H / clipSize.height);
|
||||
const [cw, ch] = [clipSize.width * s0, clipSize.height * s0];
|
||||
const [ox, oy] = [(W - cw) / 2, (H - ch) / 2];
|
||||
const face = (b: number[], w: number, h: number, x0: number, y0: number) => ({
|
||||
x: x0 + (b[0]! + b[2]! / 2) * w,
|
||||
y: y0 + (b[1]! + b[3]! / 2) * h,
|
||||
h: b[3]! * h,
|
||||
});
|
||||
const os = pairs.map((p) => face(p.a, W, H, 0, 0));
|
||||
const cs = pairs.map((p) => face(p.b, cw, ch, ox, oy));
|
||||
const mid = (v: { x: number; y: number; h: number }[]) => ({
|
||||
x: median(v.map((f) => f.x)),
|
||||
y: median(v.map((f) => f.y)),
|
||||
h: median(v.map((f) => f.h)),
|
||||
});
|
||||
const [o, c] = [mid(os), mid(cs)];
|
||||
const scale = o.h / c.h;
|
||||
// About the element's centre (GSAP's default transform origin): p -> centre + scale (p - centre) + (x, y).
|
||||
const x = o.x - (W / 2 + scale * (c.x - W / 2));
|
||||
const y = o.y - (H / 2 + scale * (c.y - H / 2));
|
||||
const residual = median(
|
||||
pairs.map((_, i) => {
|
||||
const [p, q] = [os[i]!, cs[i]!];
|
||||
const [mx, my] = [W / 2 + scale * (q.x - W / 2) + x, H / 2 + scale * (q.y - H / 2) + y];
|
||||
return Math.hypot(mx - p.x, my - p.y) / p.h;
|
||||
}),
|
||||
);
|
||||
return {
|
||||
moments: pairs.length,
|
||||
transform: { scale: round(scale, 3), x: round(x, 1), y: round(y, 1) },
|
||||
original_face: { centre: [round(o.x, 1), round(o.y, 1)], height: round(o.h, 1) },
|
||||
clip_face: { centre: [round(c.x, 1), round(c.y, 1)], height: round(c.h, 1) },
|
||||
residual: round(residual, 3),
|
||||
};
|
||||
}
|
||||
@@ -1,24 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { faceSummary, faceVerdict } from "./face.js";
|
||||
|
||||
describe("faces", () => {
|
||||
it("grades the subject's face by its height", () => {
|
||||
expect(faceVerdict(0.3)).toBe("pass");
|
||||
expect(faceVerdict(0.1)).toBe("warn");
|
||||
expect(faceVerdict(0.1, true)).toBe("fail");
|
||||
expect(faceVerdict(null)).toBe("unmeasured");
|
||||
const r = faceSummary([
|
||||
{
|
||||
frame: 0,
|
||||
t: 0,
|
||||
boxes: [
|
||||
[0.4, 0.2, 0.2, 0.3],
|
||||
[0.1, 0.1, 0.05, 0.08],
|
||||
],
|
||||
},
|
||||
{ frame: 5, t: 0.2, boxes: [] },
|
||||
]);
|
||||
expect(r.face_height).toBe(0.3);
|
||||
expect(r.verdict).toBe("unmeasured");
|
||||
});
|
||||
});
|
||||
@@ -1,87 +0,0 @@
|
||||
// The look of a stretch of film as numbers, ported from the Studio's look-stats.mjs: mean luma and contrast
|
||||
// (0–255), mean saturation (0–1, HSV), the share of black-and-white frames (under 2% of mid-tone pixels carry
|
||||
// colour, so a colour face in a white room is not black and white) and warmth (mean R − B, negative is cool).
|
||||
import { ratioOf, readClip, round, type Clip, type Roi } from "./media.js";
|
||||
|
||||
// Under these a frame has no light, contrast or colour to speak of, on the scales above.
|
||||
const LOOK_FLOOR = { luma: 1, contrast: 1, saturation: 0.01 };
|
||||
|
||||
export interface Look {
|
||||
frames: number;
|
||||
luma: number;
|
||||
contrast: number;
|
||||
saturation: number;
|
||||
bw_share: number;
|
||||
warmth: number;
|
||||
}
|
||||
|
||||
/** Sampled at `fps`, 160 px wide; `roi` measures one region, e.g. a person. */
|
||||
export function look(clip: Clip, fps = 2, roi?: Roi): Look {
|
||||
const { frames } = readClip(clip, { fps, width: 160, roi });
|
||||
const per = frames.map((f) => {
|
||||
let [l, l2, s, warm, mid, tinted] = [0, 0, 0, 0, 0, 0];
|
||||
const n = f.length / 3;
|
||||
for (let i = 0; i < f.length; i += 3) {
|
||||
const [r, g, b] = [f[i]!, f[i + 1]!, f[i + 2]!];
|
||||
const y = 0.299 * r + 0.587 * g + 0.114 * b;
|
||||
const [mx, mn] = [Math.max(r, g, b), Math.min(r, g, b)];
|
||||
l += y;
|
||||
l2 += y * y;
|
||||
s += mx ? (mx - mn) / mx : 0;
|
||||
warm += r - b;
|
||||
if (y > 40 && y < 220) {
|
||||
mid++;
|
||||
if (mx - mn > 25) tinted++;
|
||||
}
|
||||
}
|
||||
const mean = l / n;
|
||||
return {
|
||||
luma: mean,
|
||||
contrast: Math.sqrt(Math.max(0, l2 / n - mean * mean)),
|
||||
sat: s / n,
|
||||
warm: warm / n,
|
||||
bw: tinted < 0.02 * Math.max(1, mid),
|
||||
};
|
||||
});
|
||||
const avg = (k: "luma" | "contrast" | "sat" | "warm") =>
|
||||
per.reduce((a, p) => a + p[k], 0) / (per.length || 1);
|
||||
return {
|
||||
frames: per.length,
|
||||
luma: round(avg("luma"), 1),
|
||||
contrast: round(avg("contrast"), 1),
|
||||
saturation: round(avg("sat"), 3),
|
||||
bw_share: round(per.filter((p) => p.bw).length / (per.length || 1), 2),
|
||||
warmth: round(avg("warm"), 1),
|
||||
};
|
||||
}
|
||||
|
||||
export interface LookComparison {
|
||||
luma_ratio: number;
|
||||
contrast_ratio: number;
|
||||
saturation_ratio: number;
|
||||
bw_share_diff: number;
|
||||
warmth_diff: number;
|
||||
}
|
||||
|
||||
/** Yours over the reference's: ratios near 1 and differences near 0 are the same look. */
|
||||
export function compareLook(ref: Look, yours: Look): LookComparison {
|
||||
const ratio = (k: "luma" | "contrast" | "saturation") =>
|
||||
round(ratioOf(yours[k], ref[k], LOOK_FLOOR[k]), 2);
|
||||
return {
|
||||
luma_ratio: ratio("luma"),
|
||||
contrast_ratio: ratio("contrast"),
|
||||
saturation_ratio: ratio("saturation"),
|
||||
bw_share_diff: round(yours.bw_share - ref.bw_share, 2),
|
||||
warmth_diff: round(yours.warmth - ref.warmth, 1),
|
||||
};
|
||||
}
|
||||
|
||||
/** The same look within a regenerated shot's tolerance (to be calibrated on the bench). */
|
||||
export function sameLook(c: LookComparison): boolean {
|
||||
return (
|
||||
Math.abs(c.luma_ratio - 1) <= 0.15 &&
|
||||
Math.abs(c.contrast_ratio - 1) <= 0.25 &&
|
||||
Math.abs(c.saturation_ratio - 1) <= 0.25 &&
|
||||
Math.abs(c.warmth_diff) <= 12
|
||||
);
|
||||
}
|
||||
@@ -1,359 +0,0 @@
|
||||
// Decode with ffmpeg into raw arrays, so measuring and comparing need nothing but node and ffmpeg.
|
||||
// Ported from the Studio's viral-clone media.mjs and tool.mjs.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, mkdirSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
import { findFFmpeg, findFFprobe, getFFmpegInstallHint } from "../browser/ffmpeg.js";
|
||||
|
||||
const MAX_BUFFER = 1 << 30;
|
||||
/** ffmpeg's own flags on every run. Inline before a spread, the ffprobe `--` contract test reads them as a probe wrapper. */
|
||||
const FFMPEG_QUIET = ["-nostdin", "-v", "error"];
|
||||
|
||||
/** An error whose message says what to run instead; commands print `fix` as its own line. */
|
||||
export class Fix extends Error {
|
||||
constructor(
|
||||
problem: string,
|
||||
readonly fix: string,
|
||||
) {
|
||||
super(problem);
|
||||
this.name = "Fix";
|
||||
}
|
||||
}
|
||||
|
||||
export function ffmpegBin(): string {
|
||||
const bin = findFFmpeg();
|
||||
if (!bin) throw new Fix("ffmpeg is not installed", getFFmpegInstallHint());
|
||||
return bin;
|
||||
}
|
||||
|
||||
function ffprobeBin(): string {
|
||||
const bin = findFFprobe();
|
||||
if (!bin) throw new Fix("ffprobe is not installed", getFFmpegInstallHint());
|
||||
return bin;
|
||||
}
|
||||
|
||||
/** Runs ffmpeg and returns its stdout; `input` is piped to stdin. */
|
||||
export function runFfmpeg(args: string[], input?: Buffer, cwd?: string): Buffer {
|
||||
return execFileSync(ffmpegBin(), [...FFMPEG_QUIET, ...args], {
|
||||
maxBuffer: MAX_BUFFER,
|
||||
windowsHide: true,
|
||||
...(input ? { input } : {}),
|
||||
...(cwd ? { cwd } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
export interface VideoInfo {
|
||||
path: string;
|
||||
width: number;
|
||||
height: number;
|
||||
fps: number;
|
||||
/** The frame rate as ffmpeg writes it, e.g. 30000/1001. */
|
||||
fpsText: string;
|
||||
duration: number;
|
||||
/** The picture's length: the container's can run on with a longer sound track. */
|
||||
pictureDuration: number;
|
||||
frames: number;
|
||||
audio: boolean;
|
||||
/** Seconds the sound starts after the picture (0 when it starts with it or there is none). */
|
||||
audioStart: number;
|
||||
/** Frames over seconds, which a variable-rate video's nominal rate does not tell. */
|
||||
avgFps: number;
|
||||
/** The container's frame rate and the stream's average differ: frames are not evenly spaced. */
|
||||
vfr: boolean;
|
||||
hdr: boolean;
|
||||
codec: string;
|
||||
}
|
||||
|
||||
interface ProbeStream {
|
||||
codec_type?: string;
|
||||
codec_name?: string;
|
||||
width?: number;
|
||||
height?: number;
|
||||
r_frame_rate?: string;
|
||||
avg_frame_rate?: string;
|
||||
nb_frames?: string;
|
||||
duration?: string;
|
||||
start_time?: string;
|
||||
color_transfer?: string;
|
||||
side_data_list?: { rotation?: number }[];
|
||||
tags?: { rotate?: string };
|
||||
}
|
||||
|
||||
const rate = (text: string | undefined): number => {
|
||||
const [a, b] = String(text ?? "0/1")
|
||||
.split("/")
|
||||
.map(Number);
|
||||
return b ? (a ?? 0) / b : (a ?? 0);
|
||||
};
|
||||
|
||||
/** Size (as displayed, rotation applied), clock and sound of a video. */
|
||||
// fallow-ignore-next-line complexity
|
||||
export function probeVideo(path: string): VideoInfo {
|
||||
if (!existsSync(path))
|
||||
throw new Fix(`no file at ${path}`, "pass the video's path from the project folder");
|
||||
let json: { streams?: ProbeStream[]; format?: { duration?: string } };
|
||||
try {
|
||||
json = JSON.parse(
|
||||
execFileSync(
|
||||
ffprobeBin(),
|
||||
["-v", "error", "-show_streams", "-show_format", "-of", "json", "--", path],
|
||||
{ maxBuffer: MAX_BUFFER, windowsHide: true },
|
||||
).toString(),
|
||||
);
|
||||
} catch {
|
||||
throw new Fix(`ffprobe cannot read ${path}`, "pass a video file (mp4, mov, webm)");
|
||||
}
|
||||
const streams = json.streams ?? [];
|
||||
const video = streams.find((s) => s.codec_type === "video");
|
||||
if (!video) throw new Fix(`${path} has no video stream`, "pass a video file, not audio");
|
||||
const rotation = Math.abs(
|
||||
Number(video.side_data_list?.find((d) => d.rotation !== undefined)?.rotation ?? 0) ||
|
||||
Number(video.tags?.rotate ?? 0),
|
||||
);
|
||||
const turned = rotation % 180 === 90;
|
||||
const fpsText = video.r_frame_rate && video.r_frame_rate !== "0/0" ? video.r_frame_rate : "30/1";
|
||||
const fps = rate(fpsText);
|
||||
const avg = rate(video.avg_frame_rate);
|
||||
const duration = Number(json.format?.duration ?? video.duration ?? 0);
|
||||
const counted = Number(video.nb_frames ?? 0);
|
||||
const sound = streams.find((s) => s.codec_type === "audio");
|
||||
const frames = counted > 0 ? counted : Math.round(duration * fps);
|
||||
return {
|
||||
path,
|
||||
width: (turned ? video.height : video.width) ?? 0,
|
||||
height: (turned ? video.width : video.height) ?? 0,
|
||||
fps,
|
||||
fpsText: fpsText.endsWith("/1") ? fpsText.slice(0, -2) : fpsText,
|
||||
duration,
|
||||
pictureDuration: Number(video.duration ?? 0) || frames / (avg || fps),
|
||||
frames,
|
||||
audio: Boolean(sound),
|
||||
audioStart: Math.max(0, Number(sound?.start_time ?? 0) - Number(video.start_time ?? 0) || 0),
|
||||
avgFps: avg || fps,
|
||||
vfr: avg > 0 && Math.abs(avg - fps) / fps > 0.01,
|
||||
hdr: /smpte2084|arib-std-b67/.test(video.color_transfer ?? ""),
|
||||
codec: video.codec_name ?? "",
|
||||
};
|
||||
}
|
||||
|
||||
export type Roi = [x: number, y: number, w: number, h: number];
|
||||
|
||||
export interface ReadOptions {
|
||||
start?: number;
|
||||
/** Seconds to read; omit for the rest of the file. */
|
||||
seconds?: number;
|
||||
roi?: Roi;
|
||||
/** Output width (height follows the ROI's aspect, even); 0 or omitted keeps the ROI's size. */
|
||||
width?: number;
|
||||
gray?: boolean;
|
||||
/** Resample to this rate; 0 or omitted reads every frame. */
|
||||
fps?: number;
|
||||
/** Stop after this many frames. */
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
export interface Frames {
|
||||
frames: Uint8Array[];
|
||||
w: number;
|
||||
h: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Frames from anywhere ffmpeg reads: a window of a video file, or a numbered picture sequence. `filters` run
|
||||
* before any crop or scale, so a clip can pick frames (select, fps) without changing their size.
|
||||
*/
|
||||
export interface Clip {
|
||||
input: string[];
|
||||
filters: string[];
|
||||
width: number;
|
||||
height: number;
|
||||
fps: number;
|
||||
}
|
||||
|
||||
/** A window of a video file as a clip; `info` saves a probe when the caller has one. */
|
||||
export function videoClip(
|
||||
path: string,
|
||||
start = 0,
|
||||
seconds?: number,
|
||||
info = probeVideo(path),
|
||||
): Clip {
|
||||
return {
|
||||
input: [
|
||||
...(start > 0 ? ["-ss", String(start)] : []),
|
||||
"-i",
|
||||
path,
|
||||
...(seconds !== undefined ? ["-t", String(seconds)] : []),
|
||||
],
|
||||
filters: [],
|
||||
width: info.width,
|
||||
height: info.height,
|
||||
fps: info.fps,
|
||||
};
|
||||
}
|
||||
|
||||
/** A clip's frames of one ROI as raw RGB (or grey) bytes. */
|
||||
export function readClip(clip: Clip, opts: Omit<ReadOptions, "start" | "seconds"> = {}): Frames {
|
||||
const { roi, width = 0, gray = false, fps = 0, limit } = opts;
|
||||
const filters = [...clip.filters];
|
||||
if (fps) filters.push(`fps=${fps}`);
|
||||
let [w, h] = [clip.width, clip.height];
|
||||
if (roi) {
|
||||
filters.push(`crop=${roi[2]}:${roi[3]}:${roi[0]}:${roi[1]}`);
|
||||
[w, h] = [roi[2], roi[3]];
|
||||
}
|
||||
if (width && width !== w) {
|
||||
const outH = Math.max(2, Math.round((h * width) / w / 2) * 2);
|
||||
filters.push(`scale=${width}:${outH}`);
|
||||
[w, h] = [width, outH];
|
||||
}
|
||||
return decode(clip.input, filters, w, h, gray, limit);
|
||||
}
|
||||
|
||||
/** The frame shown at `t`, picked by its index: seeking by time can land a frame early or late. */
|
||||
export function readFrameAt(
|
||||
info: VideoInfo,
|
||||
t: number,
|
||||
opts: Omit<ReadOptions, "start" | "seconds" | "fps"> = {},
|
||||
): Frames {
|
||||
const k = Math.min(info.frames - 1, frameAt(t, info.fps));
|
||||
return readClip(
|
||||
{ ...videoClip(info.path, 0, undefined, info), filters: [`select=eq(n\\,${k})`] },
|
||||
{ ...opts, limit: 1 },
|
||||
);
|
||||
}
|
||||
|
||||
/** Frames of one ROI of a video file as raw RGB (or grey) bytes. */
|
||||
export function readFrames(path: string, opts: ReadOptions = {}): Frames {
|
||||
return readClip(videoClip(path, opts.start, opts.seconds), opts);
|
||||
}
|
||||
|
||||
/** Every frame (or the first `limit`) ffmpeg's `input` arguments yield after `filters`, which must end at `w`x`h`. */
|
||||
export function decode(
|
||||
input: string[],
|
||||
filters: string[],
|
||||
w: number,
|
||||
h: number,
|
||||
gray = false,
|
||||
limit?: number,
|
||||
): Frames {
|
||||
const raw = runFfmpeg([
|
||||
...input,
|
||||
"-an",
|
||||
"-vf",
|
||||
[...filters, `format=${gray ? "gray" : "rgb24"}`].join(","),
|
||||
...(limit ? ["-frames:v", String(limit)] : []),
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-",
|
||||
]);
|
||||
const size = w * h * (gray ? 1 : 3);
|
||||
const frames: Uint8Array[] = [];
|
||||
for (let i = 0; i + size <= raw.length; i += size) frames.push(raw.subarray(i, i + size));
|
||||
return { frames, w, h };
|
||||
}
|
||||
|
||||
/** BT.601 luma of RGB bytes. */
|
||||
export function grey(rgb: Uint8Array): Uint8Array {
|
||||
const out = new Uint8Array(rgb.length / 3);
|
||||
for (let i = 0; i < out.length; i++)
|
||||
out[i] = (77 * rgb[3 * i]! + 150 * rgb[3 * i + 1]! + 29 * rgb[3 * i + 2]!) >> 8;
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Mean absolute difference of two equal-length byte arrays. */
|
||||
export function meanAbsDiff(a: Uint8Array, b: Uint8Array): number {
|
||||
let s = 0;
|
||||
for (let i = 0; i < a.length; i++) s += Math.abs(a[i]! - b[i]!);
|
||||
return s / (a.length || 1);
|
||||
}
|
||||
|
||||
/** Writes raw RGB (or grey) bytes as an image; the extension picks the format. */
|
||||
export function writeImage(path: string, px: Uint8Array, w: number, h: number, gray = false) {
|
||||
const header = Buffer.from(`${gray ? "P5" : "P6"}\n${w} ${h}\n255\n`);
|
||||
runFfmpeg(
|
||||
["-y", "-f", gray ? "pgm_pipe" : "ppm_pipe", "-i", "-", "-frames:v", "1", path],
|
||||
Buffer.concat([header, Buffer.from(px.buffer, px.byteOffset, px.byteLength)]),
|
||||
);
|
||||
return path;
|
||||
}
|
||||
|
||||
export const round = (n: number, d = 3) => Math.round(n * 10 ** d) / 10 ** d;
|
||||
|
||||
/** Yours over the reference; two values under `floor` are both nothing, and nothing matches nothing. */
|
||||
export const ratioOf = (yours: number, ref: number, floor: number): number =>
|
||||
yours < floor && ref < floor ? 1 : yours / (ref || 1e-9);
|
||||
|
||||
/** "1,2.5,4" as numbers. */
|
||||
export function parseNumbers(text: unknown, flag: string): number[] {
|
||||
const out = String(text ?? "")
|
||||
.split(",")
|
||||
.filter((s) => s.trim() !== "")
|
||||
.map(Number);
|
||||
if (!out.length || out.some((n) => !Number.isFinite(n)))
|
||||
throw new Fix(
|
||||
`${flag} must be numbers separated by commas, got "${String(text)}"`,
|
||||
`${flag} 1.2,3.4`,
|
||||
);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** "start,end" seconds with 0 <= start < end. */
|
||||
export function parseRange(text: unknown, flag = "--range"): [number, number] {
|
||||
const v = parseNumbers(text, flag);
|
||||
if (v.length !== 2 || v[0]! < 0 || v[1]! <= v[0]!)
|
||||
throw new Fix(
|
||||
`${flag} must be start,end seconds with start < end, got "${String(text)}"`,
|
||||
`${flag} 2.1,4.5`,
|
||||
);
|
||||
return [v[0]!, v[1]!];
|
||||
}
|
||||
|
||||
/** "x,y,w,h" in pixels, inside a `width`x`height` frame when one is given. */
|
||||
export function parseRoi(text: unknown, width?: number, height?: number): Roi {
|
||||
const v = parseNumbers(text, "--roi").map(Math.round);
|
||||
if (v.length !== 4 || v[2]! <= 0 || v[3]! <= 0)
|
||||
throw new Fix(
|
||||
`--roi must be x,y,w,h with a positive size, got "${String(text)}"`,
|
||||
"--roi 100,200,400,300",
|
||||
);
|
||||
const roi: Roi = [v[0]!, v[1]!, v[2]!, v[3]!];
|
||||
if (
|
||||
width &&
|
||||
height &&
|
||||
(roi[0] < 0 || roi[1] < 0 || roi[0] + roi[2] > width || roi[1] + roi[3] > height)
|
||||
)
|
||||
throw new Fix(
|
||||
`--roi ${roi.join(",")} runs past the ${width}x${height} frame`,
|
||||
`keep it inside 0,0,${width},${height}`,
|
||||
);
|
||||
return roi;
|
||||
}
|
||||
|
||||
/** The folder a tool writes its pictures to, made on demand: --out, or .hyperframes/<tool> in the current folder. */
|
||||
export function outDir(out: unknown, tool: string): string {
|
||||
const dir = typeof out === "string" && out ? resolve(out) : resolve(join(".hyperframes", tool));
|
||||
mkdirSync(dir, { recursive: true });
|
||||
return dir;
|
||||
}
|
||||
|
||||
/** The frame shown at time `t`, allowing for times printed to the millisecond (38/24 s printed as 1.583). */
|
||||
export const frameAt = (t: number, fps: number) =>
|
||||
Math.max(0, Math.floor((t + 0.0005) * fps + 1e-6));
|
||||
|
||||
/** JSON with each innermost array or object on one line: readable, and a fraction of the tokens. */
|
||||
export const formatJson = (value: unknown) =>
|
||||
JSON.stringify(value, null, 1).replace(/[[{][^[\]{}]*[\]}]/g, (m) => m.replace(/\n\s*/g, " "));
|
||||
|
||||
/** Pearson correlation of two series over their common length; null when either one is flat. */
|
||||
export function pearson(a: number[], b: number[]): number | null {
|
||||
const n = Math.min(a.length, b.length);
|
||||
const ma = a.slice(0, n).reduce((s, v) => s + v, 0) / n;
|
||||
const mb = b.slice(0, n).reduce((s, v) => s + v, 0) / n;
|
||||
let [num, da, db] = [0, 0, 0];
|
||||
for (let i = 0; i < n; i++) {
|
||||
num += (a[i]! - ma) * (b[i]! - mb);
|
||||
da += (a[i]! - ma) ** 2;
|
||||
db += (b[i]! - mb) ** 2;
|
||||
}
|
||||
return da && db ? num / Math.sqrt(da * db) : null;
|
||||
}
|
||||
@@ -1,175 +0,0 @@
|
||||
// OpenCV for the measuring tools that need it (template tracking, optical flow, faces), as opencv.js: one WebAssembly
|
||||
// file that gives every machine the same numbers. It is not shipped in the CLI; the first use downloads it and the
|
||||
// Haar face models, pinned by size and sha256, into ~/.cache/hyperframes/opencv/<version>/.
|
||||
import { mkdirSync, readFileSync, renameSync, rmSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { downloadToFile } from "../cloud/download.js";
|
||||
import { verifies, type ModelFile } from "../whisper/modelFile.js";
|
||||
import { Fix } from "./media.js";
|
||||
|
||||
const VERSION = "4.10.0";
|
||||
export const OPENCV_DIR = join(homedir(), ".cache", "hyperframes", "opencv", VERSION);
|
||||
|
||||
interface Pinned extends ModelFile {
|
||||
url: string;
|
||||
}
|
||||
|
||||
// opencv.js is @techstark/opencv-js 4.10.0-release.1 (Apache-2.0); the cascades are OpenCV's own (BSD-3-Clause).
|
||||
const CASCADES = "https://raw.githubusercontent.com/opencv/opencv/4.10.0/data/haarcascades";
|
||||
const FILES: readonly Pinned[] = [
|
||||
{
|
||||
name: "opencv.js",
|
||||
bytes: 10_378_215,
|
||||
sha256: "19b46167b59efbef9cc3836264b0b657110833e4b1df84282004b8b3141c048d",
|
||||
url: "https://cdn.jsdelivr.net/npm/@techstark/opencv-js@4.10.0-release.1/dist/opencv.js",
|
||||
},
|
||||
{
|
||||
name: "haarcascade_frontalface_alt2.xml",
|
||||
bytes: 540_616,
|
||||
sha256: "7b0c967d9abbdfbde025eb9c786947d151b6426040d07a8f9562ed8fd90724b4",
|
||||
url: `${CASCADES}/haarcascade_frontalface_alt2.xml`,
|
||||
},
|
||||
{
|
||||
name: "haarcascade_profileface.xml",
|
||||
bytes: 828_514,
|
||||
sha256: "b39a4a3be45539db146a7fc1d3e761a292c196eb88421185e6a615b3055e612d",
|
||||
url: `${CASCADES}/haarcascade_profileface.xml`,
|
||||
},
|
||||
];
|
||||
|
||||
/** The parts of the OpenCV API the measuring tools use. */
|
||||
export interface Mat {
|
||||
rows: number;
|
||||
cols: number;
|
||||
data32F: Float32Array;
|
||||
data64F: Float64Array;
|
||||
data: Uint8Array;
|
||||
delete(): void;
|
||||
}
|
||||
interface Size {
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
export interface Rect {
|
||||
x: number;
|
||||
y: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
export interface RectVector {
|
||||
size(): number;
|
||||
get(i: number): Rect;
|
||||
delete(): void;
|
||||
}
|
||||
export interface Cv {
|
||||
Mat: new () => Mat;
|
||||
RectVector: new () => RectVector;
|
||||
Size: new (w: number, h: number) => Size;
|
||||
CascadeClassifier: new () => {
|
||||
load(path: string): boolean;
|
||||
detectMultiScale(
|
||||
gray: Mat,
|
||||
out: RectVector,
|
||||
scale: number,
|
||||
neighbours: number,
|
||||
flags: number,
|
||||
min: Size,
|
||||
): void;
|
||||
delete(): void;
|
||||
};
|
||||
matFromArray(rows: number, cols: number, type: number, data: ArrayLike<number>): Mat;
|
||||
matchTemplate(image: Mat, templ: Mat, result: Mat, method: number): void;
|
||||
minMaxLoc(src: Mat): {
|
||||
minVal: number;
|
||||
maxVal: number;
|
||||
minLoc: { x: number; y: number };
|
||||
maxLoc: { x: number; y: number };
|
||||
};
|
||||
calcOpticalFlowFarneback(
|
||||
prev: Mat,
|
||||
next: Mat,
|
||||
flow: Mat,
|
||||
pyrScale: number,
|
||||
levels: number,
|
||||
winsize: number,
|
||||
iterations: number,
|
||||
polyN: number,
|
||||
polySigma: number,
|
||||
flags: number,
|
||||
): void;
|
||||
estimateAffine2D(from: Mat, to: Mat): Mat;
|
||||
FS_createDataFile(
|
||||
parent: string,
|
||||
name: string,
|
||||
data: Uint8Array,
|
||||
read: boolean,
|
||||
write: boolean,
|
||||
own: boolean,
|
||||
): void;
|
||||
CV_8UC1: number;
|
||||
CV_32FC2: number;
|
||||
TM_CCOEFF_NORMED: number;
|
||||
}
|
||||
|
||||
const isCv = (v: unknown): v is Cv =>
|
||||
typeof v === "object" && v !== null && "Mat" in v && "matFromArray" in v;
|
||||
|
||||
/** Every pinned file present and verified; the missing ones fetched. */
|
||||
// fallow-ignore-next-line complexity
|
||||
async function ensureOpenCv(onStatus?: (message: string) => void): Promise<void> {
|
||||
const missing: Pinned[] = [];
|
||||
for (const f of FILES) if (!(await verifies(join(OPENCV_DIR, f.name), f))) missing.push(f);
|
||||
if (!missing.length) return;
|
||||
mkdirSync(OPENCV_DIR, { recursive: true });
|
||||
for (const f of missing) {
|
||||
onStatus?.(`Downloading ${f.name} (${Math.round(f.bytes / 1e5) / 10} MB, once)...`);
|
||||
const temp = join(OPENCV_DIR, `${f.name}.part-${process.pid}`);
|
||||
try {
|
||||
await downloadToFile(f.url, temp);
|
||||
if (!(await verifies(temp, f)))
|
||||
throw new Fix(
|
||||
`${f.name} did not match its pinned size and sha256`,
|
||||
"run the command again to retry the download",
|
||||
);
|
||||
renameSync(temp, join(OPENCV_DIR, f.name));
|
||||
} catch (err) {
|
||||
if (err instanceof Fix) throw err;
|
||||
throw new Fix(
|
||||
`could not download ${f.name} (${err instanceof Error ? err.message : String(err)})`,
|
||||
"check the network and run the command again",
|
||||
);
|
||||
} finally {
|
||||
rmSync(temp, { force: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let loaded: Promise<Cv> | null = null;
|
||||
|
||||
/** opencv.js, ready to call, with the face models in its file system. Loaded once per process. */
|
||||
export function loadOpenCv(onStatus?: (message: string) => void): Promise<Cv> {
|
||||
loaded ??= (async () => {
|
||||
await ensureOpenCv(onStatus);
|
||||
const mod: unknown = createRequire(import.meta.url)(join(OPENCV_DIR, "opencv.js"));
|
||||
if (typeof mod !== "object" || mod === null) throw new Error("opencv.js did not load");
|
||||
// Wait on its init hook, never on the module: it is a thenable that does not settle under await.
|
||||
await new Promise<void>((resolve) => {
|
||||
if ("Mat" in mod) resolve();
|
||||
else Object.assign(mod, { onRuntimeInitialized: () => resolve() });
|
||||
});
|
||||
// Ready now; without its `then`, promises can carry it like any other object.
|
||||
Reflect.deleteProperty(mod, "then");
|
||||
if (!isCv(mod)) throw new Error("opencv.js did not initialize");
|
||||
const cv = mod;
|
||||
for (const f of FILES.filter((x) => x.name.endsWith(".xml")))
|
||||
cv.FS_createDataFile("/", f.name, readFileSync(join(OPENCV_DIR, f.name)), true, false, false);
|
||||
return cv;
|
||||
})();
|
||||
return loaded;
|
||||
}
|
||||
|
||||
/** Grey bytes as an 8-bit single-channel Mat. */
|
||||
export const greyMat = (cv: Cv, px: Uint8Array, w: number, h: number) =>
|
||||
cv.matFromArray(h, w, cv.CV_8UC1, px);
|
||||
@@ -1,566 +0,0 @@
|
||||
// A re-shot person never matches the original by pixels, so pixels do not judge them. Three gates do: the framing
|
||||
// is clearly wrong (face centre or height off the original's, per shot), the generated clip carries the original's
|
||||
// words (its motion guide's text came along), or the lips are out of sync. The rest (motion energy, look, whether
|
||||
// each generated clip starts at one of the original's cuts) is reported, not judged.
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { contrast } from "./drawn.js";
|
||||
import { faceBoxes } from "./face.js";
|
||||
import { pearson, probeVideo, round, runFfmpeg, type Roi, type VideoInfo } from "./media.js";
|
||||
import { avSync, type SyncResult } from "./sync.js";
|
||||
|
||||
/**
|
||||
* A shot's framing is clearly wrong when, over the shot, the face centre sits more than FRAMING_SHIFT face heights
|
||||
* from the original's or its height is outside FRAMING_SCALE of it (medians). Calibrated on the reverse bench (10-23,
|
||||
* faces at 4 a second, shots of a second or more): goat round 3's 8.0–10.0 s shot is 0.23 off and ×0.82, dan round 2
|
||||
* at most 0.16 and ×0.93–1.10, cindy round 3 0.02; dan round 3 misses its 35 s close-up at ×0.77 (and is 0.21–0.23
|
||||
* off at 3.6 s and 30 s, which face boxes cannot tell from goat). Shorter shots have too few faces to judge.
|
||||
*/
|
||||
const FRAMING_SHIFT = 0.3;
|
||||
const FRAMING_SCALE: [number, number] = [0.8, 1.25];
|
||||
const FRAMING_MIN_S = 1;
|
||||
const FRAMING_MIN_FACES = 3;
|
||||
const FACE_RATE = 4;
|
||||
/** Picture against sound (positive: the picture is late), the broadcast tolerance; judged on high confidence only. */
|
||||
const SYNC_MS: [number, number] = [-125, 45];
|
||||
const SYNC_MIN_S = 2;
|
||||
const SYNC_MIN_SPEECH = 0.3;
|
||||
/**
|
||||
* A generated clip carries a drawn element's text when it shows the original's letters inside the element's box:
|
||||
* of the original's sharp ink (contrast over TEXT_CONTRAST) where it differs from the clean still the clip was made
|
||||
* from and where yours draws over the clip (each by DRAWN_DIFF: the letters, not the place), the share the clip has
|
||||
* within two pixels as sharp ink of the same colour (TEXT_COLOUR). The clip shares the person and the place with the
|
||||
* original, so only carried letters match there. Calibrated on ig03 round 2 (10-23): t12's "Transparency", "a deep
|
||||
* understanding", "of this market" 0.30, 0.45, 0.56; every clean box 0.24 or less.
|
||||
*/
|
||||
const CARRIED_TEXT = 0.35;
|
||||
const TEXT_CONTRAST = 140;
|
||||
const DRAWN_DIFF = 60;
|
||||
const TEXT_COLOUR = 15;
|
||||
const TEXT_MIN_INK = 50;
|
||||
const CARRIED_SAMPLES = 8;
|
||||
/** A generated clip found by its sound needs this correlation with the original's; it starts at a cut within this. */
|
||||
const LOCATE_MIN = 0.6;
|
||||
export const AT_CUT_S = 0.1;
|
||||
const ENVELOPE_RATE = 100;
|
||||
|
||||
export interface GeneratedClip {
|
||||
/** As the ledger writes it, relative to the project. */
|
||||
path: string;
|
||||
/** The still it was made from (a clean frame of the original), relative to the project. */
|
||||
reference?: string;
|
||||
/** Film time of the clip's first frame, and how well its sound matched the original's there. */
|
||||
offset?: number;
|
||||
match?: number;
|
||||
window?: [number, number];
|
||||
/** Where the film shows it: film window, the clip's time at its start, and its playback rate. */
|
||||
uses?: ClipUse[];
|
||||
/** Whether it starts within AT_CUT_S of the film's start or one of the original's cuts. */
|
||||
at_cut?: boolean;
|
||||
note?: string;
|
||||
}
|
||||
|
||||
export interface ClipUse {
|
||||
window: [number, number];
|
||||
from: number;
|
||||
rate: number;
|
||||
}
|
||||
|
||||
export interface ShotFraming {
|
||||
window: [number, number];
|
||||
/** Medians: face centre distance in the original's face heights, and yours' face height over the original's. */
|
||||
shift: number;
|
||||
scale: number;
|
||||
pass: boolean;
|
||||
}
|
||||
|
||||
export interface CarriedText {
|
||||
drawn: string;
|
||||
clip: string;
|
||||
window: [number, number];
|
||||
/** How well the generated clip's ink inside the box matches the original's text there. */
|
||||
score: number;
|
||||
carried: boolean;
|
||||
}
|
||||
|
||||
export interface PersonVerdict {
|
||||
/** Each gate: false when it fails, null when it could not be measured. */
|
||||
gates: { framing: boolean | null; text: boolean | null; sync: boolean | null };
|
||||
pass: boolean | null;
|
||||
framing?: ShotFraming[];
|
||||
text?: CarriedText[];
|
||||
sync?: {
|
||||
window: [number, number];
|
||||
lag_ms: number | null;
|
||||
original_lag_ms: number | null;
|
||||
confidence: string;
|
||||
pass: boolean | null;
|
||||
}[];
|
||||
}
|
||||
|
||||
const isRecord = (v: unknown): v is Record<string, unknown> => typeof v === "object" && v !== null;
|
||||
|
||||
/** The ledger's generated videos (HeyGen), with the film window it records when it does. */
|
||||
export function generatedClips(ledger: string): GeneratedClip[] {
|
||||
// fallow-ignore-next-line complexity
|
||||
return ledger.split("\n").flatMap((line) => {
|
||||
let entry: unknown;
|
||||
try {
|
||||
entry = JSON.parse(line);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
if (!isRecord(entry) || entry.type !== "video" || typeof entry.path !== "string") return [];
|
||||
const p = entry.provenance;
|
||||
if (!isRecord(p) || p.provider !== "heygen") return [];
|
||||
const w = Array.isArray(p.window) ? p.window.filter((v) => typeof v === "number") : [];
|
||||
const refs = Array.isArray(p.references)
|
||||
? p.references.filter((v) => typeof v === "string")
|
||||
: [];
|
||||
return [
|
||||
{
|
||||
path: entry.path,
|
||||
...(refs[0] ? { reference: refs[0] } : {}),
|
||||
...(w.length === 2 ? { offset: w[0]!, window: [w[0]!, w[1]!] as [number, number] } : {}),
|
||||
},
|
||||
];
|
||||
});
|
||||
}
|
||||
|
||||
/** Loudness every 10 ms of a file's sound. */
|
||||
function envelope(path: string): number[] {
|
||||
const raw = runFfmpeg([...["-i", path, "-vn", "-ac", "1", "-ar", "8000", "-f", "f32le", "-"]]);
|
||||
const s = new Float32Array(
|
||||
raw.buffer.slice(raw.byteOffset, raw.byteOffset + (raw.byteLength & ~3)),
|
||||
);
|
||||
const step = 8000 / ENVELOPE_RATE;
|
||||
const out: number[] = [];
|
||||
for (let i = 0; i + step <= s.length; i += step) {
|
||||
let e = 0;
|
||||
for (let j = i; j < i + step; j++) e += s[j]! ** 2;
|
||||
out.push(Math.sqrt(e / step));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Where `b` sits inside `a` (index of a where b[0] lands) and the correlation there, over lags that keep at least
|
||||
* nine tenths of b inside a.
|
||||
*/
|
||||
function bestLag(a: number[], b: number[]): { lag: number; r: number } {
|
||||
let best = { lag: 0, r: -2 };
|
||||
const keep = Math.ceil(0.9 * b.length);
|
||||
for (let lag = -(b.length - keep); lag <= a.length - keep; lag++) {
|
||||
const [i0, i1] = [Math.max(0, -lag), Math.min(b.length, a.length - lag)];
|
||||
const n = i1 - i0;
|
||||
if (n < Math.max(50, keep)) continue;
|
||||
const r = pearson(a.slice(i0 + lag, i1 + lag), b.slice(i0, i1)) ?? -2;
|
||||
if (r > best.r) best = { lag, r };
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
/**
|
||||
* Each generated clip placed on the film. Its own start (where HeyGen's shot begins) is the ledger's window when it
|
||||
* records one, else where the clip's sound (the original's voice it was made from) lines up with the original's.
|
||||
* Where the film shows it comes from the project's timeline (`uses`, by the clip's path), else its whole window.
|
||||
* Files resolve against the project.
|
||||
*/
|
||||
export function placeClips(
|
||||
clips: GeneratedClip[],
|
||||
project: string | undefined,
|
||||
ref: VideoInfo,
|
||||
cuts: number[],
|
||||
shown: Map<string, ClipUse[]> = new Map(),
|
||||
): (GeneratedClip & { file?: string; info?: VideoInfo; still?: string })[] {
|
||||
let whole: number[] | undefined;
|
||||
// fallow-ignore-next-line complexity
|
||||
return clips.map((c) => {
|
||||
const file = project ? join(project, c.path) : undefined;
|
||||
let info: VideoInfo | undefined;
|
||||
try {
|
||||
info = file ? probeVideo(file) : undefined;
|
||||
} catch {
|
||||
info = undefined;
|
||||
}
|
||||
if (!info || !file) return { ...c, note: "the clip's file is not in the project" };
|
||||
const uses = shown.get(normalize(c.path));
|
||||
let placed: GeneratedClip = c;
|
||||
if (c.offset === undefined && info.audio && ref.audio) {
|
||||
whole ??= envelope(ref.path);
|
||||
const { lag, r } = bestLag(whole, envelope(file));
|
||||
const offset = round(lag / ENVELOPE_RATE + ref.audioStart - info.audioStart, 3);
|
||||
placed =
|
||||
r >= LOCATE_MIN
|
||||
? { ...c, offset, match: round(r, 2) }
|
||||
: { ...c, match: round(r, 2), note: "its sound does not match the original's" };
|
||||
}
|
||||
if (placed.offset === undefined && uses?.length)
|
||||
placed = { ...placed, offset: round(uses[0]!.window[0] - uses[0]!.from / uses[0]!.rate, 3) };
|
||||
if (placed.offset === undefined)
|
||||
return {
|
||||
...placed,
|
||||
file,
|
||||
info,
|
||||
note: placed.note ?? "no window in the ledger, no sound and no place in the project",
|
||||
};
|
||||
const offset = placed.offset;
|
||||
const end = Math.min(ref.pictureDuration, offset + info.pictureDuration);
|
||||
return {
|
||||
...placed,
|
||||
window: placed.window ?? [round(Math.max(0, offset), 3), round(end, 3)],
|
||||
uses: uses ?? [
|
||||
{
|
||||
window: [round(Math.max(0, offset), 3), round(end, 3)],
|
||||
from: Math.max(0, -offset),
|
||||
rate: 1,
|
||||
},
|
||||
],
|
||||
at_cut: [0, ...cuts].some((t) => Math.abs(t - offset) <= AT_CUT_S),
|
||||
file,
|
||||
info,
|
||||
...(c.reference && existsSync(join(project!, c.reference))
|
||||
? { still: join(project!, c.reference) }
|
||||
: {}),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** Where a project's timeline shows each video file, by project-relative path. */
|
||||
export async function clipUses(project: string): Promise<Map<string, ClipUse[]>> {
|
||||
const out = new Map<string, ClipUse[]>();
|
||||
const index = join(project, "index.html");
|
||||
if (!existsSync(index)) return out;
|
||||
const [{ ensureDOMParser }, { describeProject }] = await Promise.all([
|
||||
import("../utils/dom.js"),
|
||||
import("../timeline/describeProject.js"),
|
||||
]);
|
||||
ensureDOMParser();
|
||||
const timeline = await describeProject(index);
|
||||
for (const row of timeline.tracks.find((t) => t.kind === "video")?.rows ?? []) {
|
||||
if (!row.src || row.absEnd <= row.absStart) continue;
|
||||
const key = normalize(row.src);
|
||||
const use = {
|
||||
window: [round(row.absStart, 3), round(row.absEnd, 3)] as [number, number],
|
||||
from: row.mediaStart ?? 0,
|
||||
rate: row.playbackRate ?? 1,
|
||||
};
|
||||
out.set(key, [...(out.get(key) ?? []), use]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** A project-relative path written one way. */
|
||||
const normalize = (p: string) => p.replace(/^\.\//, "").replace(/\\/g, "/");
|
||||
|
||||
/** RGB crops of one box at these times: the picture scaled to cover `w`x`h` (the original's size) first. */
|
||||
function boxAt(
|
||||
info: VideoInfo,
|
||||
times: number[],
|
||||
box: Roi,
|
||||
w: number,
|
||||
h: number,
|
||||
): (Uint8Array | undefined)[] {
|
||||
const ks = times.map((t) => Math.max(0, Math.min(info.frames - 1, Math.round(t * info.fps))));
|
||||
const unique = [...new Set(ks)].sort((x, y) => x - y);
|
||||
const work = mkdtempSync(join(tmpdir(), "hf-person-"));
|
||||
try {
|
||||
const filters = [
|
||||
`select=${unique.map((k) => `eq(n\\,${k})`).join("+")}`,
|
||||
`scale=${w}:${h}:force_original_aspect_ratio=increase:flags=bicubic`,
|
||||
`crop=${w}:${h}`,
|
||||
`crop=${box[2]}:${box[3]}:${box[0]}:${box[1]}`,
|
||||
"format=rgb24",
|
||||
].join(",");
|
||||
runFfmpeg(
|
||||
[
|
||||
...["-i", info.path, "-an", "-vf", filters, "-fps_mode", "passthrough"],
|
||||
...["-f", "rawvideo", "out.rgb"],
|
||||
],
|
||||
undefined,
|
||||
work,
|
||||
);
|
||||
const raw = readFileSync(join(work, "out.rgb"));
|
||||
const size = box[2] * box[3] * 3;
|
||||
return ks.map((k) => {
|
||||
const i = unique.indexOf(k) * size;
|
||||
return i + size <= raw.length ? raw.subarray(i, i + size) : undefined;
|
||||
});
|
||||
} finally {
|
||||
rmSync(work, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
/** Within `r` pixels of (x, y), some pixel passes `test`. */
|
||||
function nearby(
|
||||
w: number,
|
||||
h: number,
|
||||
x: number,
|
||||
y: number,
|
||||
r: number,
|
||||
test: (k: number) => boolean,
|
||||
) {
|
||||
for (let dy = -r; dy <= r; dy++)
|
||||
for (let dx = -r; dx <= r; dx++) {
|
||||
const [xx, yy] = [Math.min(w - 1, Math.max(0, x + dx)), Math.min(h - 1, Math.max(0, y + dy))];
|
||||
if (test(yy * w + xx)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
const dist = (p: Uint8Array, i: number, q: Uint8Array, k: number) =>
|
||||
Math.hypot(p[i * 3]! - q[k * 3]!, p[i * 3 + 1]! - q[k * 3 + 1]!, p[i * 3 + 2]! - q[k * 3 + 2]!);
|
||||
|
||||
/** One frame: the share of the original's letters (apart from the still and from what yours draws) the clip shows. */
|
||||
// fallow-ignore-next-line complexity
|
||||
export function carriedFrame(
|
||||
original: Uint8Array,
|
||||
clip: Uint8Array,
|
||||
still: Uint8Array | undefined,
|
||||
w: number,
|
||||
h: number,
|
||||
yours: Uint8Array | undefined,
|
||||
): number | null {
|
||||
const [sharp, inked] = [contrast(original, w, h), contrast(clip, w, h)];
|
||||
let [n, hit] = [0, 0];
|
||||
for (let y = 0; y < h; y++)
|
||||
for (let x = 0; x < w; x++) {
|
||||
const i = y * w + x;
|
||||
if (sharp[i]! <= TEXT_CONTRAST) continue;
|
||||
if (still && !nearby(w, h, x, y, 2, (k) => dist(original, k, still, k) > DRAWN_DIFF))
|
||||
continue;
|
||||
if (yours && !nearby(w, h, x, y, 2, (k) => dist(yours, k, clip, k) > DRAWN_DIFF)) continue;
|
||||
n++;
|
||||
const same = (k: number) =>
|
||||
inked[k]! > TEXT_CONTRAST && dist(original, i, clip, k) < TEXT_COLOUR;
|
||||
if (nearby(w, h, x, y, 2, same)) hit++;
|
||||
}
|
||||
return n >= TEXT_MIN_INK ? hit / n : null;
|
||||
}
|
||||
|
||||
/** One box of a still picture, scaled to cover `w`x`h` (the original's size) first. */
|
||||
function stillBox(path: string, box: Roi, w: number, h: number): Uint8Array | undefined {
|
||||
try {
|
||||
const raw = runFfmpeg([
|
||||
...["-i", path, "-frames:v", "1", "-vf"],
|
||||
`scale=${w}:${h}:force_original_aspect_ratio=increase:flags=bicubic,crop=${w}:${h},crop=${box[2]}:${box[3]}:${box[0]}:${box[1]},format=rgb24`,
|
||||
...["-f", "rawvideo", "-"],
|
||||
]);
|
||||
return raw.length >= box[2] * box[3] * 3 ? new Uint8Array(raw) : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a generated clip carries the original's text inside a drawn element's box over its window: the clip
|
||||
* alone, against the original there. A clean clip shows the person and the place behind the letters; a clip made
|
||||
* from a motion guide with words on it shows those words.
|
||||
*/
|
||||
function carriedText(
|
||||
ref: VideoInfo,
|
||||
yours: VideoInfo,
|
||||
clip: { path: string; info: VideoInfo; uses: ClipUse[]; still?: string },
|
||||
drawn: { id: string; box: Roi; window: [number, number] },
|
||||
): CarriedText | undefined {
|
||||
// The film times both show, each with the clip's own time there.
|
||||
const spans = clip.uses.flatMap((u) => {
|
||||
const [a, b] = [Math.max(drawn.window[0], u.window[0]), Math.min(drawn.window[1], u.window[1])];
|
||||
return b - a >= 1 / ref.fps ? [{ a, b, u }] : [];
|
||||
});
|
||||
const total = spans.reduce((s, x) => s + x.b - x.a, 0);
|
||||
if (!total) return undefined;
|
||||
// Evenly spread samples over the shown spans.
|
||||
const times = Array.from({ length: CARRIED_SAMPLES }, (_, i) => {
|
||||
let at = ((i + 0.5) * total) / CARRIED_SAMPLES;
|
||||
for (const x of spans) {
|
||||
if (at <= x.b - x.a)
|
||||
return { t: x.a + at, c: x.u.from + (at + x.a - x.u.window[0]) * x.u.rate };
|
||||
at -= x.b - x.a;
|
||||
}
|
||||
const x = spans.at(-1)!;
|
||||
return { t: x.b, c: x.u.from + (x.b - x.u.window[0]) * x.u.rate };
|
||||
});
|
||||
const orig = boxAt(
|
||||
ref,
|
||||
times.map((x) => x.t),
|
||||
drawn.box,
|
||||
ref.width,
|
||||
ref.height,
|
||||
);
|
||||
const generated = boxAt(
|
||||
clip.info,
|
||||
times.map((x) => x.c),
|
||||
drawn.box,
|
||||
ref.width,
|
||||
ref.height,
|
||||
);
|
||||
const mine = boxAt(
|
||||
yours,
|
||||
times.map((x) => x.t),
|
||||
drawn.box,
|
||||
ref.width,
|
||||
ref.height,
|
||||
);
|
||||
const still = clip.still ? stillBox(clip.still, drawn.box, ref.width, ref.height) : undefined;
|
||||
const scores = orig.flatMap((o, i) => {
|
||||
const g = generated[i];
|
||||
const s = o && g ? carriedFrame(o, g, still, drawn.box[2], drawn.box[3], mine[i]) : null;
|
||||
return s === null ? [] : [s];
|
||||
});
|
||||
if (!scores.length) return undefined;
|
||||
const score = scores.reduce((s, v) => s + v, 0) / scores.length;
|
||||
return {
|
||||
drawn: drawn.id,
|
||||
clip: clip.path,
|
||||
window: [round(spans[0]!.a, 3), round(spans.at(-1)!.b, 3)],
|
||||
score: round(score, 3),
|
||||
carried: score >= CARRIED_TEXT,
|
||||
};
|
||||
}
|
||||
|
||||
const median = (v: number[]) => [...v].sort((x, y) => x - y)[v.length >> 1]!;
|
||||
|
||||
const centre = (b: number[]) => [b[0]! + b[2]! / 2, b[1]! + b[3]! / 2] as const;
|
||||
|
||||
/** The original's shots inside a window: split at its cuts. */
|
||||
function shotsIn(window: [number, number], cuts: number[]): [number, number][] {
|
||||
const edges = [window[0], ...cuts.filter((t) => t > window[0] && t < window[1]), window[1]];
|
||||
return edges.slice(1).map((e, i) => [edges[i]!, e]);
|
||||
}
|
||||
|
||||
/**
|
||||
* One shot's framing from both sides' faces, sampled at the same moments (boxes as shares of a frame `aspect` wide
|
||||
* per unit high): per moment, the original's subject (its largest face) against yours' face nearest it; the shot's
|
||||
* medians decide, so one stray detection (a photo, a hand) does not.
|
||||
*/
|
||||
export function shotFraming(
|
||||
window: [number, number],
|
||||
original: { t: number; boxes: number[][] }[],
|
||||
yours: { t: number; boxes: number[][] }[],
|
||||
aspect = 1,
|
||||
): ShotFraming | undefined {
|
||||
const inside = (s: { t: number }) => s.t >= window[0] && s.t < window[1];
|
||||
// Across, a share of the frame is `aspect` times longer than down.
|
||||
const apart = (b: number[], x: number, y: number) =>
|
||||
Math.hypot((centre(b)[0] - x) * aspect, centre(b)[1] - y);
|
||||
const [a, b] = [original.filter(inside), yours.filter(inside)];
|
||||
const moments = a.flatMap((o, i) => {
|
||||
const y = b[i];
|
||||
if (!o.boxes.length || !y?.boxes.length) return [];
|
||||
const subject = o.boxes.reduce((p, q) => (q[3]! > p[3]! ? q : p));
|
||||
const [ox, oy] = centre(subject);
|
||||
const near = y.boxes.reduce((p, q) => (apart(q, ox, oy) < apart(p, ox, oy) ? q : p));
|
||||
return [{ shift: apart(near, ox, oy) / subject[3]!, scale: near[3]! / subject[3]! }];
|
||||
});
|
||||
if (moments.length < FRAMING_MIN_FACES) return undefined;
|
||||
const shift = round(median(moments.map((m) => m.shift)), 3);
|
||||
const scale = round(median(moments.map((m) => m.scale)), 3);
|
||||
return {
|
||||
window: [round(window[0], 3), round(window[1], 3)],
|
||||
shift,
|
||||
scale,
|
||||
pass: shift <= FRAMING_SHIFT && scale >= FRAMING_SCALE[0] && scale <= FRAMING_SCALE[1],
|
||||
};
|
||||
}
|
||||
|
||||
/** Lips in sync: the lag inside SYNC_MS, judged only when the reading is confident (it is a motion proxy). */
|
||||
export function syncPass(r: Pick<SyncResult, "best_lag_ms" | "confidence">): boolean | null {
|
||||
if (r.confidence !== "high" || r.best_lag_ms === null) return null;
|
||||
return r.best_lag_ms >= SYNC_MS[0] && r.best_lag_ms <= SYNC_MS[1];
|
||||
}
|
||||
|
||||
/** A segment's person: framing per shot, carried text per drawn element, lip sync per long speaking shot. */
|
||||
export async function personVerdict(opts: {
|
||||
ref: VideoInfo;
|
||||
yours: VideoInfo;
|
||||
window: [number, number];
|
||||
cuts: number[];
|
||||
clips: (GeneratedClip & { info?: VideoInfo; still?: string })[];
|
||||
drawn: { id: string; box: Roi; window: [number, number] }[];
|
||||
}): Promise<PersonVerdict> {
|
||||
const { ref, yours, window } = opts;
|
||||
const shots = shotsIn(window, opts.cuts);
|
||||
const n = Math.max(2, Math.round((window[1] - window[0]) * FACE_RATE));
|
||||
// A face inside a drawn element (a photo on a card) is not the person.
|
||||
const drawnFace = (t: number, b: number[]) =>
|
||||
opts.drawn.some(({ box: [x, y, w, h], window: [a, z] }) => {
|
||||
const [cx, cy] = [(b[0]! + b[2]! / 2) * ref.width, (b[1]! + b[3]! / 2) * ref.height];
|
||||
return t >= a && t < z && cx >= x && cx < x + w && cy >= y && cy < y + h;
|
||||
});
|
||||
const people = (samples: Awaited<ReturnType<typeof faceBoxes>>) =>
|
||||
samples.map((f) => ({ ...f, boxes: f.boxes.filter((b) => !drawnFace(f.t, b)) }));
|
||||
const [fo, fy] = [
|
||||
people(await faceBoxes(ref, window[0], window[1], n)),
|
||||
people(await faceBoxes(yours, window[0], window[1], n)),
|
||||
];
|
||||
const framing = shots.flatMap((s) =>
|
||||
s[1] - s[0] >= FRAMING_MIN_S
|
||||
? [shotFraming(s, fo, fy, ref.width / ref.height)].filter((f) => f !== undefined)
|
||||
: [],
|
||||
);
|
||||
const text = opts.drawn.flatMap((d) =>
|
||||
opts.clips.flatMap((c) =>
|
||||
c.info && c.uses
|
||||
? [
|
||||
carriedText(
|
||||
ref,
|
||||
yours,
|
||||
{ path: c.path, info: c.info, uses: c.uses, still: c.still },
|
||||
d,
|
||||
),
|
||||
].filter((t) => t !== undefined)
|
||||
: [],
|
||||
),
|
||||
);
|
||||
const sync = shots.flatMap((s) => {
|
||||
if (s[1] - s[0] < SYNC_MIN_S || !yours.audio) return [];
|
||||
const mouth = (
|
||||
info: VideoInfo,
|
||||
samples: { t: number; boxes: number[][] }[],
|
||||
): Roi | undefined => {
|
||||
const boxes = samples
|
||||
.filter((f) => f.t >= s[0] && f.t < s[1] && f.boxes.length)
|
||||
.map((f) => f.boxes.reduce((x, y) => (y[3]! > x[3]! ? y : x)));
|
||||
if (boxes.length < FRAMING_MIN_FACES) return undefined;
|
||||
const [x, y, w, h] = [0, 1, 2, 3].map((i) => median(boxes.map((b) => b[i]!)));
|
||||
const roi: Roi = [
|
||||
Math.round(((x! + 0.2 * w!) * info.width) / 2) * 2,
|
||||
Math.round(((y! + 0.62 * h!) * info.height) / 2) * 2,
|
||||
Math.max(8, Math.round((0.6 * w! * info.width) / 2) * 2),
|
||||
Math.max(8, Math.round((0.38 * h! * info.height) / 2) * 2),
|
||||
];
|
||||
return roi[0] + roi[2] <= info.width && roi[1] + roi[3] <= info.height ? roi : undefined;
|
||||
};
|
||||
const [mo, my] = [mouth(ref, fo), mouth(yours, fy)];
|
||||
if (!my) return [];
|
||||
const mine = avSync(yours, my, s[0], s[1]);
|
||||
if (mine.speech_active_fraction < SYNC_MIN_SPEECH) return [];
|
||||
const theirs = mo && ref.audio ? avSync(ref, mo, s[0], s[1]) : undefined;
|
||||
return [
|
||||
{
|
||||
window: [round(s[0], 3), round(s[1], 3)] as [number, number],
|
||||
lag_ms: mine.best_lag_ms,
|
||||
original_lag_ms: theirs?.best_lag_ms ?? null,
|
||||
confidence: mine.confidence,
|
||||
pass: syncPass(mine),
|
||||
},
|
||||
];
|
||||
});
|
||||
const gate = (v: (boolean | null)[]) =>
|
||||
v.includes(false) ? false : v.some((p) => p === true) ? true : null;
|
||||
const gates = {
|
||||
framing: gate(framing.map((f) => f.pass)),
|
||||
text: gate(text.map((t) => !t.carried)),
|
||||
sync: gate(sync.map((s) => s.pass)),
|
||||
};
|
||||
const all = Object.values(gates);
|
||||
return {
|
||||
gates,
|
||||
pass: all.includes(false) ? false : all.some((g) => g === true) ? true : null,
|
||||
...(framing.length ? { framing } : {}),
|
||||
...(text.length ? { text } : {}),
|
||||
...(sync.length ? { sync } : {}),
|
||||
};
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
// A segments file: `## S01 · 0.00–2.40 s · title` headings, each with optional lines
|
||||
// `Pixels: rebuilt | reused | regenerated`, `Regenerated: x, y, w, h` (source pixels, or shares of the frame when
|
||||
// all are ≤ 1), `Drawn: x, y, w, h` with an optional `a–b s`, and `Copy:` quoting a string; see `Segment`.
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { Fix } from "./media.js";
|
||||
|
||||
/** rebuilt: drawn in HTML; reused: cut from the original; regenerated: new media made for it. */
|
||||
export type PixelSource = "rebuilt" | "reused" | "regenerated";
|
||||
|
||||
export interface Segment {
|
||||
id: string;
|
||||
start: number;
|
||||
end: number;
|
||||
title: string;
|
||||
pixels: PixelSource;
|
||||
/** A regenerated region inside a rebuilt or reused segment: [x, y, w, h], pixels or shares of the frame. */
|
||||
regenerated?: [number, number, number, number];
|
||||
/** Drawn elements over a regenerated person: their boxes, judged by pixels, and when they show. */
|
||||
drawn?: { box: [number, number, number, number]; window?: [number, number] }[];
|
||||
/** The segment quotes copy: it has drawn text. */
|
||||
copy?: true;
|
||||
}
|
||||
|
||||
const HEADING =
|
||||
/^##\s+(S\d+)\s*[·|-]\s*(\d+(?:\.\d+)?)\s*[–—-]\s*(\d+(?:\.\d+)?)\s*s\b\s*(?:[·|-]\s*(.*))?$/gm;
|
||||
const SOURCES: readonly PixelSource[] = ["rebuilt", "reused", "regenerated"];
|
||||
const PIXELS = /^Pixels:\s*(rebuilt|reused|regenerated)\b/im;
|
||||
const NUMBER = String.raw`(\d+(?:\.\d+)?)`;
|
||||
const SEP = String.raw`(?:\s*,\s*|\s+)`;
|
||||
const DRAWN = new RegExp(
|
||||
`^Drawn:\\s*${NUMBER}${SEP}${NUMBER}${SEP}${NUMBER}${SEP}${NUMBER}(?:.*?${NUMBER}\\s*[–—-]\\s*${NUMBER}\\s*s\\b)?`,
|
||||
"gim",
|
||||
);
|
||||
const COPY = /^Copy:.*["“”]\S/im;
|
||||
const REGION =
|
||||
/^Regenerated:\s*(\d+(?:\.\d+)?)(?:\s*,\s*|\s+)(\d+(?:\.\d+)?)(?:\s*,\s*|\s+)(\d+(?:\.\d+)?)(?:\s*,\s*|\s+)(\d+(?:\.\d+)?)/im;
|
||||
|
||||
export function parseSegments(markdown: string): Segment[] {
|
||||
const heads = [...markdown.matchAll(HEADING)];
|
||||
// fallow-ignore-next-line complexity
|
||||
return heads.map((m, i) => {
|
||||
const body = markdown.slice(m.index! + m[0].length, heads[i + 1]?.index ?? markdown.length);
|
||||
const found = PIXELS.exec(body)?.[1]?.toLowerCase();
|
||||
const pixels = SOURCES.find((source) => source === found) ?? "rebuilt";
|
||||
const r = REGION.exec(body);
|
||||
const region: Segment["regenerated"] = r
|
||||
? [Number(r[1]), Number(r[2]), Number(r[3]), Number(r[4])]
|
||||
: undefined;
|
||||
const drawn = [...body.matchAll(DRAWN)].flatMap((d) => {
|
||||
const box: [number, number, number, number] = [
|
||||
Number(d[1]),
|
||||
Number(d[2]),
|
||||
Number(d[3]),
|
||||
Number(d[4]),
|
||||
];
|
||||
if (box[2] <= 0 || box[3] <= 0) return [];
|
||||
return [d[5] ? { box, window: [Number(d[5]), Number(d[6])] as [number, number] } : { box }];
|
||||
});
|
||||
return {
|
||||
id: m[1]!,
|
||||
start: Number(m[2]),
|
||||
end: Number(m[3]),
|
||||
title: (m[4] ?? "").trim(),
|
||||
pixels,
|
||||
...(region && pixels !== "regenerated" && region[2]! > 0 && region[3]! > 0
|
||||
? { regenerated: region }
|
||||
: {}),
|
||||
...(drawn.length ? { drawn } : {}),
|
||||
...(COPY.test(body) ? { copy: true as const } : {}),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export function readSegments(path: string): Segment[] {
|
||||
if (!existsSync(path))
|
||||
throw new Fix(`no segments file at ${path}`, "pass a file of `## S01 · a–b s` segments");
|
||||
const segments = parseSegments(readFileSync(path, "utf8"));
|
||||
if (!segments.length)
|
||||
throw new Fix(
|
||||
`${path} has no "## S01 · 0.00–2.40 s · title" segment headings`,
|
||||
"write one `## S01 · a–b s · title` heading per segment",
|
||||
);
|
||||
return segments;
|
||||
}
|
||||
@@ -1,85 +0,0 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, describe, expect, it } from "vitest";
|
||||
import { findFFmpeg, findFFprobe } from "../browser/ffmpeg.js";
|
||||
import { probeVideo } from "./media.js";
|
||||
import { syncPass } from "./person.js";
|
||||
import { avSync } from "./sync.js";
|
||||
|
||||
const ffmpeg = findFFmpeg();
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-sync-test-"));
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
|
||||
// Uneven bursts so only one lag lines them up.
|
||||
const BURSTS = [
|
||||
[0.3, 0.5],
|
||||
[0.9, 1.0],
|
||||
[1.4, 1.7],
|
||||
[2.1, 2.2],
|
||||
[2.5, 2.8],
|
||||
];
|
||||
const during = (shift: number) =>
|
||||
BURSTS.map(([a, b]) => `between(t,${a! + shift},${b! + shift})`).join("+");
|
||||
|
||||
const MOUTH: [number, number, number, number] = [24, 40, 48, 16];
|
||||
|
||||
function inStep(): string {
|
||||
const out = join(dir, "in-step.mp4");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "color=c=0x404040:s=96x96:r=30:d=3"],
|
||||
...["-f", "lavfi", "-i", "sine=f=220:d=3"],
|
||||
"-filter_complex",
|
||||
`[0]drawbox=x=24:y=40:w=48:h=16:color=white:t=fill:enable='${during(0)}'[v];` +
|
||||
`[1]volume='if(${during(0)},1,0)':eval=frame[a]`,
|
||||
...["-map", "[v]", "-map", "[a]", "-pix_fmt", "yuv420p", "-c:v", "libx264", "-qp", "0", out],
|
||||
]);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The same streams with one of them stamped 0.2 s later, as a real muxer leaves it. */
|
||||
function shifted(source: string, late: "video" | "audio"): string {
|
||||
const out = join(dir, `${late}-late.mp4`);
|
||||
const [first, second] =
|
||||
late === "video" ? [["-itsoffset", "0.2"], []] : [[], ["-itsoffset", "0.2"]];
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", ...first, "-i", source, ...second, "-i", source],
|
||||
...["-map", "0:v", "-map", "1:a", "-c", "copy", out],
|
||||
]);
|
||||
return out;
|
||||
}
|
||||
|
||||
describe.skipIf(!ffmpeg || !findFFprobe())("avSync", () => {
|
||||
it("reads the sound on its own timestamps, not from its first sample", () => {
|
||||
const aligned = inStep();
|
||||
expect(Math.abs(avSync(probeVideo(aligned), MOUTH, 0, 3).best_lag_ms!)).toBeLessThan(20);
|
||||
expect(Math.abs(avSync(probeVideo(aligned), MOUTH, 0.2, 2.9).best_lag_ms!)).toBeLessThan(20);
|
||||
const quiet = avSync(probeVideo(shifted(aligned, "audio")), MOUTH, 0, 3);
|
||||
expect(quiet.best_lag_ms).toBeGreaterThan(-230);
|
||||
expect(quiet.best_lag_ms).toBeLessThan(-170);
|
||||
expect(syncPass(quiet)).toBe(false);
|
||||
const inside = avSync(probeVideo(shifted(aligned, "audio")), MOUTH, 0.2, 2.9);
|
||||
expect(inside.best_lag_ms).toBeLessThan(-170);
|
||||
const late = avSync(probeVideo(shifted(aligned, "video")), MOUTH, 0, 3);
|
||||
expect(late.best_lag_ms).toBeGreaterThan(170);
|
||||
expect(late.best_lag_ms).toBeLessThan(230);
|
||||
});
|
||||
|
||||
it("finds a picture two frames late", () => {
|
||||
const out = join(dir, "late.mp4");
|
||||
execFileSync(ffmpeg!, [
|
||||
...["-v", "error", "-y", "-f", "lavfi", "-i", "color=c=0x404040:s=96x96:r=30:d=3"],
|
||||
...["-f", "lavfi", "-i", "sine=f=220:d=3"],
|
||||
"-filter_complex",
|
||||
`[0]drawbox=x=24:y=40:w=48:h=16:color=white:t=fill:enable='${during(2 / 30)}'[v];` +
|
||||
`[1]volume='if(${during(0)},1,0)':eval=frame[a]`,
|
||||
...["-map", "[v]", "-map", "[a]", "-pix_fmt", "yuv420p", "-c:v", "libx264", "-qp", "0", out],
|
||||
]);
|
||||
const r = avSync(probeVideo(out), [24, 40, 48, 16], 0, 3);
|
||||
expect(r.best_lag_frames).toBeGreaterThan(1.5);
|
||||
expect(r.best_lag_frames).toBeLessThan(2.5);
|
||||
expect(r.best_lag_ms).toBeGreaterThan(0);
|
||||
expect(r.confidence).not.toBe("none");
|
||||
});
|
||||
});
|
||||
@@ -1,139 +0,0 @@
|
||||
// Picture against sound for a person speaking, ported from the Studio's hf-media av_sync: per frame the mouth
|
||||
// region's mean luma change, per frame the change of the sound's loudness, and the lag where the two correlate best.
|
||||
// A motion and energy proxy, not phoneme sync: head motion, gestures, cuts and music in the ROI or mix move it.
|
||||
import {
|
||||
Fix,
|
||||
pearson,
|
||||
readClip,
|
||||
round,
|
||||
runFfmpeg,
|
||||
videoClip,
|
||||
type Roi,
|
||||
type VideoInfo,
|
||||
} from "./media.js";
|
||||
|
||||
const RATE = 16000;
|
||||
|
||||
/** Mono samples of the window at RATE Hz. */
|
||||
function samples(info: VideoInfo, start: number, seconds: number): Float32Array {
|
||||
const raw = runFfmpeg([
|
||||
...["-ss", String(start), "-i", info.path, "-t", String(seconds), "-vn"],
|
||||
...["-af", "aresample=async=1:first_pts=0"],
|
||||
...["-ac", "1", "-ar", String(RATE), "-f", "f32le", "-"],
|
||||
]);
|
||||
return new Float32Array(raw.buffer.slice(raw.byteOffset, raw.byteOffset + (raw.byteLength & ~3)));
|
||||
}
|
||||
|
||||
export interface SyncResult {
|
||||
best_lag_frames: number | null;
|
||||
best_lag_ms: number | null;
|
||||
peak_correlation: number | null;
|
||||
zero_lag_correlation: number | null;
|
||||
margin_over_other_lags: number | null;
|
||||
confidence: "high" | "medium" | "low" | "none";
|
||||
speech_active_fraction: number;
|
||||
mouth_motion_mean: number;
|
||||
note: string | null;
|
||||
}
|
||||
|
||||
// fallow-ignore-next-line complexity
|
||||
export function avSync(
|
||||
info: VideoInfo,
|
||||
mouth: Roi,
|
||||
start: number,
|
||||
end: number,
|
||||
maxLagMs = 400,
|
||||
): SyncResult {
|
||||
if (!info.audio)
|
||||
throw new Fix(
|
||||
`${info.path} has no sound`,
|
||||
"measure sync needs the picture and its sound in one file",
|
||||
);
|
||||
if (!(maxLagMs > 0)) throw new Fix("--max-lag-ms must be positive", "--max-lag-ms 400");
|
||||
const { frames, w, h } = readClip(videoClip(info.path, start, end - start, info), {
|
||||
roi: mouth,
|
||||
width: 160,
|
||||
gray: true,
|
||||
});
|
||||
const motion = frames.map((f, i) => {
|
||||
if (!i) return 0;
|
||||
let s = 0;
|
||||
for (let k = 0; k < f.length; k++) s += Math.abs(f[k]! - frames[i - 1]![k]!);
|
||||
return s / (w * h);
|
||||
});
|
||||
const audio = samples(info, start, end - start);
|
||||
const frame = 1 / info.fps;
|
||||
const envelope = frames.map((_, i) => {
|
||||
const [a, b] = [Math.floor(i * frame * RATE), Math.floor((i + 1) * frame * RATE)];
|
||||
const chunk = audio.subarray(a, Math.max(b, a + 1));
|
||||
let s = 0;
|
||||
for (const v of chunk) s += v * v;
|
||||
return chunk.length ? Math.sqrt(s / chunk.length) : 0;
|
||||
});
|
||||
const speech = envelope.slice(1).map((v, i) => Math.abs(v - envelope[i]!));
|
||||
const mouthMotion = motion.slice(1);
|
||||
const reach = Math.max(1, Math.round(maxLagMs / 1000 / frame));
|
||||
// Positive lag: the mouth moves after the sound, the picture is late.
|
||||
const curve = Array.from({ length: 2 * reach + 1 }, (_, i) => {
|
||||
const lag = i - reach;
|
||||
const [a, b] =
|
||||
lag >= 0
|
||||
? [speech.slice(0, speech.length - lag), mouthMotion.slice(lag)]
|
||||
: [speech.slice(-lag), mouthMotion.slice(0, lag)];
|
||||
return { lag, r: Math.min(a.length, b.length) < 10 ? null : pearson(a, b) };
|
||||
});
|
||||
const top = Math.max(...envelope);
|
||||
const active = top > 0 ? envelope.filter((v) => v > 0.1 * top).length / envelope.length : 0;
|
||||
const base = {
|
||||
speech_active_fraction: round(active, 3),
|
||||
mouth_motion_mean: round(mouthMotion.reduce((s, v) => s + v, 0) / (mouthMotion.length || 1), 3),
|
||||
};
|
||||
const valid = curve.filter((c) => c.r !== null);
|
||||
if (!valid.length)
|
||||
return {
|
||||
best_lag_frames: null,
|
||||
best_lag_ms: null,
|
||||
peak_correlation: null,
|
||||
zero_lag_correlation: null,
|
||||
margin_over_other_lags: null,
|
||||
confidence: "none",
|
||||
...base,
|
||||
note: "no mouth motion or no change in the sound in this window; check the ROI and the window",
|
||||
};
|
||||
const best = curve.reduce(
|
||||
(b, c, i) => ((c.r ?? -Infinity) > (curve[b]!.r ?? -Infinity) ? i : b),
|
||||
0,
|
||||
);
|
||||
const peak = curve[best]!.r!;
|
||||
// A parabola through the peak and its neighbours places it between frames.
|
||||
let shift = 0;
|
||||
const [left, right] = [curve[best - 1]?.r, curve[best + 1]?.r];
|
||||
if (typeof left === "number" && typeof right === "number") {
|
||||
const bend = left - 2 * peak + right;
|
||||
if (bend < 0) shift = (0.5 * (left - right)) / bend;
|
||||
}
|
||||
const lag = curve[best]!.lag;
|
||||
const rest = curve.filter((c) => Math.abs(c.lag - lag) > 2 && c.r !== null).map((c) => c.r!);
|
||||
const margin = rest.length ? peak - Math.max(...rest) : null;
|
||||
const edge = best === 0 || best === curve.length - 1;
|
||||
const confidence =
|
||||
peak >= 0.5 && margin !== null && margin >= 0.1 && !edge && active >= 0.1
|
||||
? "high"
|
||||
: peak >= 0.3 && !edge
|
||||
? "medium"
|
||||
: "low";
|
||||
const notes = [
|
||||
...(edge ? ["the peak is at the search edge; raise --max-lag-ms or check the window"] : []),
|
||||
...(active < 0.1 ? ["little speech in this window; choose a span with continuous speech"] : []),
|
||||
];
|
||||
return {
|
||||
best_lag_frames: round(lag + shift, 2),
|
||||
best_lag_ms: round((lag + shift) * frame * 1000, 1),
|
||||
peak_correlation: round(peak, 3),
|
||||
zero_lag_correlation: round(curve.find((c) => c.lag === 0)?.r ?? 0, 3),
|
||||
margin_over_other_lags: margin === null ? null : round(margin, 3),
|
||||
confidence,
|
||||
...base,
|
||||
note: notes.join("; ") || null,
|
||||
};
|
||||
}
|
||||
@@ -1 +0,0 @@
|
||||
export function regenerateIndex(projectDir: string): void;
|
||||
@@ -18,24 +18,3 @@ export function withReservedFileSync<T>(
|
||||
ext: string,
|
||||
populate: (reservation: Reservation) => T,
|
||||
): T;
|
||||
|
||||
export interface MediaRecord {
|
||||
id: string;
|
||||
type: string;
|
||||
path: string;
|
||||
source: string;
|
||||
description: string;
|
||||
duration?: number;
|
||||
provenance?: Record<string, unknown>;
|
||||
}
|
||||
export function recordInPlace(
|
||||
projectDir: string,
|
||||
record: {
|
||||
type: string;
|
||||
path: string;
|
||||
source: string;
|
||||
description?: string;
|
||||
duration?: number;
|
||||
provenance?: Record<string, unknown>;
|
||||
},
|
||||
): MediaRecord;
|
||||
|
||||
@@ -1660,13 +1660,13 @@ describe("preResolveHostileMediaProxies", () => {
|
||||
expect(mocks.resolveProxy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("pre-resolves an alpha asset through the Chromium-compatible VP8 proxy", async () => {
|
||||
it("pre-resolves an alpha VP9 asset through the Chromium-compatible VP8 proxy", async () => {
|
||||
const projectDir = mkProjectDir();
|
||||
mocks.scanProjectMediaCodecMap.mockResolvedValue({
|
||||
"/alpha.mov": {
|
||||
codecName: "prores",
|
||||
"/alpha.webm": {
|
||||
codecName: "vp9",
|
||||
browserHostile: true,
|
||||
representativeMime: null,
|
||||
representativeMime: 'video/webm; codecs="vp09.00.10.08"',
|
||||
hasAlpha: true,
|
||||
},
|
||||
});
|
||||
@@ -1676,33 +1676,11 @@ describe("preResolveHostileMediaProxies", () => {
|
||||
expect(mocks.resolveProxy).toHaveBeenCalledTimes(1);
|
||||
expect(mocks.resolveProxy).toHaveBeenCalledWith(
|
||||
projectDir,
|
||||
join(projectDir, "alpha.mov"),
|
||||
join(projectDir, "alpha.webm"),
|
||||
"vp8",
|
||||
);
|
||||
});
|
||||
|
||||
it("does not copy VP9 or AV1 videos, which Chrome plays (alpha included)", async () => {
|
||||
const projectDir = mkProjectDir();
|
||||
mocks.scanProjectMediaCodecMap.mockResolvedValue({
|
||||
"/cutout.webm": {
|
||||
codecName: "vp9",
|
||||
browserHostile: true,
|
||||
representativeMime: 'video/webm; codecs="vp09.00.10.08"',
|
||||
hasAlpha: true,
|
||||
},
|
||||
"/clip.mp4": {
|
||||
codecName: "av1",
|
||||
browserHostile: true,
|
||||
representativeMime: 'video/mp4; codecs="av01.0.08M.08"',
|
||||
hasAlpha: false,
|
||||
},
|
||||
});
|
||||
|
||||
await preResolveHostileMediaProxies(projectDir, "<html></html>");
|
||||
|
||||
expect(mocks.resolveProxy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("is a no-op when the codec map has no hostile entries", async () => {
|
||||
const projectDir = mkProjectDir();
|
||||
mocks.scanProjectMediaCodecMap.mockResolvedValue({
|
||||
|
||||
@@ -51,7 +51,6 @@ import {
|
||||
decideMediaProxyEligibility,
|
||||
proxyVariantFor,
|
||||
scanProjectMediaCodecMap,
|
||||
shouldPrewarmProxy,
|
||||
} from "@hyperframes/studio-server/media-codec-map";
|
||||
import {
|
||||
FfmpegUnavailableError,
|
||||
@@ -220,7 +219,7 @@ export async function preResolveHostileMediaProxies(
|
||||
return none;
|
||||
}
|
||||
const hostileEntries = Object.entries(codecMap).filter(
|
||||
([, facts]) => decideMediaProxyEligibility(facts).eligible && shouldPrewarmProxy(facts),
|
||||
([, facts]) => decideMediaProxyEligibility(facts).eligible,
|
||||
);
|
||||
if (hostileEntries.length === 0) return none;
|
||||
|
||||
|
||||
@@ -139,8 +139,6 @@ const PROJECT_FILES = new Set([
|
||||
"REVIEW.md",
|
||||
"STORYLINES.md",
|
||||
"assets/transcript.json",
|
||||
"transcript.json",
|
||||
"REVERSE.md",
|
||||
"asset-descriptions.md",
|
||||
"extracted/page.html",
|
||||
"extracted/visible-text.txt",
|
||||
|
||||
@@ -19,7 +19,6 @@ export const CORE_SKILLS = {
|
||||
figma: 7168,
|
||||
slideshow: 5120,
|
||||
"remotion-to-hyperframes": 5120,
|
||||
"reverse-any-video": 5120,
|
||||
};
|
||||
|
||||
/** @param {string} skill */
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { mkdtempSync, mkdirSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
@@ -15,15 +15,6 @@ test("the repo's core skills pass", () => {
|
||||
assert.deepEqual(checkCoreSkills(), []);
|
||||
});
|
||||
|
||||
test("the skills teach the direct sign-in, not the old key and executable rules", () => {
|
||||
const read = (path) => readFileSync(new URL(`../skills/${path}`, import.meta.url), "utf8");
|
||||
assert.doesNotMatch(read("reverse-any-video/references/people.md"), /Without a HeyGen key/);
|
||||
assert.doesNotMatch(read("media-use/SKILL.md"), /HeyGen calls also need/);
|
||||
const heygen = read("media-use/references/heygen.md");
|
||||
assert.match(heygen, /~\/\.hyperframes\/credentials/);
|
||||
assert.doesNotMatch(heygen, /~\/\.heygen\/credentials/);
|
||||
});
|
||||
|
||||
test("reads the registered commands, quoted or not", () => {
|
||||
const source = `const commandLoaders = {\n init: () => x,\n "catch-up": () => y,\n};`;
|
||||
assert.deepEqual([...registeredCommands(source)], ["init", "catch-up"]);
|
||||
|
||||
@@ -737,6 +737,67 @@ const RETIRED_ROUTING_FILES = [
|
||||
"skills/hyperframes/references/workflow-catalog.md",
|
||||
];
|
||||
|
||||
const REVERT_6754_REASON =
|
||||
"the revert of #6754 keeps the reverse-any-video skill out of main until web subscriptions are live; it re-lands with #6833";
|
||||
|
||||
const REVERT_6754_FILES = [
|
||||
"packages/cli/src/commands/billing.ts",
|
||||
"packages/cli/src/commands/diff.ts",
|
||||
"packages/cli/src/commands/measure.ts",
|
||||
"packages/cli/src/commands/measure/curve.ts",
|
||||
"packages/cli/src/commands/measure/cuts.ts",
|
||||
"packages/cli/src/commands/measure/energy.ts",
|
||||
"packages/cli/src/commands/measure/events.ts",
|
||||
"packages/cli/src/commands/measure/extract.ts",
|
||||
"packages/cli/src/commands/measure/face.ts",
|
||||
"packages/cli/src/commands/measure/frames.ts",
|
||||
"packages/cli/src/commands/measure/look.ts",
|
||||
"packages/cli/src/commands/measure/probe.ts",
|
||||
"packages/cli/src/commands/measure/sync.ts",
|
||||
"packages/cli/src/commands/media-use/generate.ts",
|
||||
"packages/cli/src/heygen/generate.test.ts",
|
||||
"packages/cli/src/heygen/generate.ts",
|
||||
"packages/cli/src/heygen/plan.test.ts",
|
||||
"packages/cli/src/heygen/plan.ts",
|
||||
"packages/cli/src/heygen/pricing.ts",
|
||||
"packages/cli/src/heygen/transcribe.test.ts",
|
||||
"packages/cli/src/heygen/transcribe.ts",
|
||||
"packages/cli/src/heygen/v3.test.ts",
|
||||
"packages/cli/src/heygen/v3.ts",
|
||||
"packages/cli/src/measure/command.ts",
|
||||
"packages/cli/src/measure/compare.test.ts",
|
||||
"packages/cli/src/measure/curve.test.ts",
|
||||
"packages/cli/src/measure/curve.ts",
|
||||
"packages/cli/src/measure/cuts.test.ts",
|
||||
"packages/cli/src/measure/cuts.ts",
|
||||
"packages/cli/src/measure/diff-people.test.ts",
|
||||
"packages/cli/src/measure/diff-segments.test.ts",
|
||||
"packages/cli/src/measure/diff.test-helpers.ts",
|
||||
"packages/cli/src/measure/diff.test.ts",
|
||||
"packages/cli/src/measure/diff.ts",
|
||||
"packages/cli/src/measure/drawn.test.ts",
|
||||
"packages/cli/src/measure/drawn.ts",
|
||||
"packages/cli/src/measure/energy.ts",
|
||||
"packages/cli/src/measure/events.test.ts",
|
||||
"packages/cli/src/measure/events.ts",
|
||||
"packages/cli/src/measure/face.test.ts",
|
||||
"packages/cli/src/measure/face.ts",
|
||||
"packages/cli/src/measure/faceVerdict.test.ts",
|
||||
"packages/cli/src/measure/look.ts",
|
||||
"packages/cli/src/measure/media.ts",
|
||||
"packages/cli/src/measure/opencv.ts",
|
||||
"packages/cli/src/measure/person.ts",
|
||||
"packages/cli/src/measure/segments.ts",
|
||||
"packages/cli/src/measure/sync.test.ts",
|
||||
"packages/cli/src/measure/sync.ts",
|
||||
"packages/cli/src/media-use/lib/index-gen.d.mts",
|
||||
"skills/media-use/references/heygen.md",
|
||||
"skills/reverse-any-video/SKILL.md",
|
||||
"skills/reverse-any-video/references/match.md",
|
||||
"skills/reverse-any-video/references/people.md",
|
||||
"skills/reverse-any-video/references/start.md",
|
||||
];
|
||||
|
||||
const FALLOW_PRUNE_REASON =
|
||||
"fallow 3 reuses one audit snapshot per checkout and the hook prunes with `fallow audit-cache prune`, replacing this script";
|
||||
|
||||
@@ -2066,6 +2127,7 @@ export const ALLOWED_DELETIONS = new Map([
|
||||
...RETIRED_WORKFLOW_FILES.map((path) => [path, RETIRED_WORKFLOW_REASON]),
|
||||
...RETIRED_GUIDE_FILES.map((path) => [path, RETIRED_GUIDE_REASON]),
|
||||
...RETIRED_ROUTING_FILES.map((path) => [path, RETIRED_ROUTING_REASON]),
|
||||
...REVERT_6754_FILES.map((path) => [path, REVERT_6754_REASON]),
|
||||
[
|
||||
"packages/producer/src/services/audioExtractor.ts",
|
||||
"imported by nothing; its own ffmpeg spawn quoted stderr unredacted, outside ManagedChildProcess",
|
||||
@@ -2080,14 +2142,6 @@ export const ALLOWED_DELETIONS = new Map([
|
||||
".github/workflows/lambda-ffmpeg.yml",
|
||||
"the ffmpeg build moves into lambda-arm64-image.yml, which also builds and smokes the image",
|
||||
],
|
||||
[
|
||||
"skills/remotion-to-hyperframes/scripts/frame_strip.sh",
|
||||
"remotion-to-hyperframes now compares with `hyperframes diff` instead of its own shell scripts",
|
||||
],
|
||||
[
|
||||
"skills/remotion-to-hyperframes/scripts/render_diff.sh",
|
||||
"remotion-to-hyperframes now compares with `hyperframes diff` instead of its own shell scripts",
|
||||
],
|
||||
]);
|
||||
|
||||
export function parseBase(argv, fallback = "origin/main") {
|
||||
|
||||
@@ -18,16 +18,12 @@
|
||||
"files": 13
|
||||
},
|
||||
"media-use": {
|
||||
"hash": "96a0143522380926",
|
||||
"files": 117
|
||||
"hash": "f61a33f8b0f8e260",
|
||||
"files": 116
|
||||
},
|
||||
"remotion-to-hyperframes": {
|
||||
"hash": "1775bffb149795f6",
|
||||
"files": 76
|
||||
},
|
||||
"reverse-any-video": {
|
||||
"hash": "a6acb6ab1b62822b",
|
||||
"files": 4
|
||||
"hash": "37ed4536305b3c64",
|
||||
"files": 78
|
||||
},
|
||||
"slideshow": {
|
||||
"hash": "c8a591e2aa7a6d86",
|
||||
|
||||
+21
-20
@@ -15,31 +15,33 @@ npx hyperframes media-use resolve --type <type> --intent "<what you need>" --pro
|
||||
|
||||
It prints one line, `resolved <id> → <path> (<type>, <metadata>)`. List reusable assets first with `--candidates` and judge the fit yourself; reuse, ingest (`--from`) and adopt are in `references/resolve.md`.
|
||||
|
||||
## Where it comes from
|
||||
## Two ways to get it
|
||||
|
||||
| Need | Free | HyperFrames credits | Local or the person's own key (only when they ask) |
|
||||
| --- | --- | --- | --- |
|
||||
| Music (`bgm`) | HeyGen catalog of 10k+ tracks (signed in) | Generated track | Local MusicGen |
|
||||
| Sound effects (`sfx`) | Openverse (credit CC BY sounds); 21 bundled offline | | |
|
||||
| Images, icons | HeyGen asset search, 75k+ (signed in) | Generated images | Local FLUX (mflux), if installed |
|
||||
| Logos (`logo`) | theSVG, GitHub avatar, favicon; never redrawn | | |
|
||||
| Voiceover (`voice`) | HeyGen Voice Turbo (signed in) | Other HeyGen voices, voice cloning | `npx hyperframes tts` (Kokoro on device) |
|
||||
| A presenter speaking | | Avatar video (`--type video`) | Local LTX for generic clips, if installed |
|
||||
| Transcripts | `npx hyperframes transcribe` (Parakeet or whisper) | | |
|
||||
| Cutouts | | `heygen background-removal` | `npx hyperframes remove-background` |
|
||||
| Grades, LUTs, looks | `npx hyperframes media-treatment` | | |
|
||||
| Need | HeyGen (best quality) | Free, no account |
|
||||
| --- | --- | --- |
|
||||
| Music (`bgm`) | Catalog of 10k+ tracks | Local MusicGen (`audio/references/bgm.md`) |
|
||||
| Sound effects (`sfx`) | | Openverse; 21 bundled offline |
|
||||
| Images, icons | Asset search, 75k+ | Local FLUX (mflux), if installed |
|
||||
| Logos (`logo`) | | theSVG, GitHub avatar, favicon; never redrawn |
|
||||
| Voiceover (`voice`) | HeyGen voices, voice cloning | `npx hyperframes tts` (Kokoro on device; Python 3.10+ with `pip install kokoro-onnx soundfile`) |
|
||||
| A presenter speaking | Avatar video (`--type video`) | Local LTX for generic clips, if installed |
|
||||
| Transcripts | | `npx hyperframes transcribe` (Parakeet or whisper) |
|
||||
| Cutouts | `heygen background-removal` | `npx hyperframes remove-background` |
|
||||
| Grades, LUTs, looks | | `npx hyperframes media-treatment` |
|
||||
|
||||
**Sign in.** Every time a request needs a HeyGen model or the catalog, run `npx hyperframes auth status` first. Not signed in: tell the person in one line that sign-in is free and gives Voice Turbo, the music catalog and asset search, ask them to type an exclamation mark and `npx hyperframes auth login` in the prompt, and wait until they say it is done. Use local tools only when the person asks, not as a fallback.
|
||||
**Host apps.** Inside an app that gives you its own media tools, use those first. When `HEYGEN_API_BASE` is set, the host pays for HeyGen: don't ask for the `heygen` CLI or a sign-in, and when a call is refused, relay the host's message as written and stop.
|
||||
|
||||
**Host apps.** Inside an app with its own media tools, use those first. When `HEYGEN_API_BASE` is set, the host pays: don't ask for a sign-in; relay a refused call's message as written and stop.
|
||||
**HeyGen.** Install the [`heygen` CLI](https://developers.heygen.com/cli), then sign in with `npx hyperframes auth login`. HyperFrames supplies its credential to the helper; it does not read the separate HeyGen CLI login. An explicit API key bills API credits, and the server decides OAuth eligibility and charges. Check setup with `npx hyperframes media-use resolve --doctor`. If a required provider is missing or signed out, give the install link and HyperFrames sign-in command, then offer the free path or wait.
|
||||
|
||||
**HeyGen.** `npx hyperframes media-use generate` makes images, shots, talking photos, voices and music on the sign-in alone. Helpers and direct recipes need the [`heygen` CLI](https://developers.heygen.com/cli); if it is missing, give the link and wait until they say it is installed. Direct `heygen` recipes bill that CLI's own login, not HyperFrames: use them only when that separate payer is selected.
|
||||
**Direct HeyGen recipes.** Commands such as `heygen video-agent create`, `heygen video create`, `heygen voice clone create`, `heygen lipsync`, `heygen video-translate` and `heygen ai-clipping` in `references/operations.md` use the HeyGen CLI's own credential and HeyGen/API billing. They do not inherit HyperFrames OAuth or its subscription. Use only when that separate payer is selected.
|
||||
|
||||
**Credits.** Plan for the best result: don't swap in cheaper or free options, or drop items, unless the person asks to save cost. Never tell the person prices, estimates or what a run used; `--plan` is for you. A credit limit the person sets: pass what is left of it as `--max-credits` on every `generate`, minus each call's `credits`, and never go past it. When a step won't fit (a `--plan` over what is left, or a refused call), say what doesn't fit and ask before dropping or shrinking it. A refused call names a `billing` command: tell the person why, ask them to run it the same way, then retry the step.
|
||||
**Free.** Every tool in the free column needs no account; local models are opt-in installs (`references/setup-providers.md`). Without `heygen`, `resolve` for music, images and icons fails and says so; credit CC BY sounds (`audio/references/sfx.md`). `--local-only` skips every network provider.
|
||||
|
||||
**Cost.** Confirm before a paid call you chose to make (an avatar video counts); one the person asked for just runs. `--provider <name>` pins one generator when the person names it.
|
||||
|
||||
## Looks on real footage
|
||||
|
||||
Feedback that footage is dark, flat or dull, should feel retro or printed, needs a face hidden, or needs a reveal is media intent. Inspect the real `<img>`/`<video>`, pick one intent, and follow `references/media-treatments.md`; persist with `npx hyperframes media-treatment`, never a CSS, SVG or `.cube` stand-in.
|
||||
Feedback that footage is dark, flat or dull, should feel retro or printed, needs a face hidden, or needs a reveal is media intent: read `references/media-treatments.md`. Inspect the real `<img>`/`<video>`, pick one intent, start from `npx hyperframes media-treatment --capabilities --json`, then `--capability <id>`; never `--all`. Persist the final treatment with `npx hyperframes media-treatment`; no CSS or SVG stand-ins for supported effects, and no generated `.cube` just to encode exposure or warmth.
|
||||
|
||||
## Offer once, never add silently
|
||||
|
||||
@@ -52,10 +54,9 @@ When you build or review a piece, scan once for concrete gaps (on-screen text wi
|
||||
| Resolve, reuse, adopt, ingest, flags | `references/resolve.md` |
|
||||
| Grading, LUTs, `grade-compare` | `references/grading.md` |
|
||||
| Voiceover, music, sound effects, captions, transcription | `references/audio.md` |
|
||||
| Cut, reframe or transform media; direct `heygen` recipes | `references/operations.md` |
|
||||
| Cut, reframe or transform media; HeyGen recipes | `references/operations.md` |
|
||||
| Treatments, realtime effects, overlays, reveals | `references/media-treatments.md` |
|
||||
| Install, sign-in, credentials, `--provider`, `--local-only`, RAM ladders | `references/setup-providers.md` |
|
||||
| `media-use generate`, HeyGen transcription, prices | `references/heygen.md` |
|
||||
| Install, sign-in, providers, RAM ladders | `references/setup-providers.md` |
|
||||
| Fades, gain, ducking, automation, effects, buses on placed tracks | `references/mixing.md` |
|
||||
| Local commands: grade, cutout, loudness, beats | `references/local.md` |
|
||||
| Remembered preferences | `references/memory.md` |
|
||||
|
||||
@@ -214,33 +214,8 @@ export async function heygenJSON(path, { method = "GET", headers = {}, body } =
|
||||
return text ? JSON.parse(text) : {};
|
||||
}
|
||||
|
||||
function errorCode(body) {
|
||||
try {
|
||||
const parsed = JSON.parse(body);
|
||||
return parsed?.error?.code ?? parsed?.code ?? null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// One line the agent relays as written when the HyperFrames plan can't pay: no plan, out of credits, or a
|
||||
// sign-in the plan doesn't cover yet. Null for every other failure.
|
||||
export function billingRefusal(e) {
|
||||
const code = e?.body ? errorCode(e.body) : null;
|
||||
if (code === "hf_subscription_required")
|
||||
return "This needs a HyperFrames plan. Run `npx hyperframes billing subscribe`: it opens the page and waits.";
|
||||
if (code === "hf_usage_exhausted")
|
||||
return "Out of HyperFrames credits. Run `npx hyperframes billing top-up`: it opens the page and waits.";
|
||||
if (code === "api_key_required")
|
||||
return "This HeyGen feature isn't available with this sign-in yet. Nothing was charged. Use a HeyGen API key, or wait for HyperFrames billing in the CLI.";
|
||||
return null;
|
||||
}
|
||||
|
||||
// HeyGen's own words for a failed call: a billing refusal line first, then its {"error":{"message"}}, else the
|
||||
// raw body, else the error's message.
|
||||
// HeyGen's own words for a failed call: its {"error":{"message"}}, else the raw body, else the error's message.
|
||||
export function heygenMessage(e) {
|
||||
const refusal = billingRefusal(e);
|
||||
if (refusal) return refusal;
|
||||
if (!e?.body) return e?.message ? String(e.message) : String(e);
|
||||
try {
|
||||
return JSON.parse(e.body).error?.message ?? e.body;
|
||||
|
||||
@@ -5,13 +5,11 @@ import { createServer } from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
billingRefusal,
|
||||
heygenAuthHeaders,
|
||||
heygenAuthMethod,
|
||||
heygenBase,
|
||||
heygenCredential,
|
||||
heygenJSON,
|
||||
heygenMessage,
|
||||
loadEnvFromDir,
|
||||
} from "./heygen.mjs";
|
||||
|
||||
@@ -350,34 +348,3 @@ test("a base that isn't HeyGen's gets no stored or host OAuth credential, only a
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
test("a refused HyperFrames-plan call becomes one line with the billing command", () => {
|
||||
const refused = (code) => ({
|
||||
status: 402,
|
||||
body: JSON.stringify({ error: { code, message: "raw" } }),
|
||||
});
|
||||
assert.equal(
|
||||
heygenMessage(refused("hf_subscription_required")),
|
||||
"This needs a HyperFrames plan. Run `npx hyperframes billing subscribe`: it opens the page and waits.",
|
||||
);
|
||||
assert.equal(
|
||||
heygenMessage(refused("hf_usage_exhausted")),
|
||||
"Out of HyperFrames credits. Run `npx hyperframes billing top-up`: it opens the page and waits.",
|
||||
);
|
||||
assert.match(
|
||||
heygenMessage({ status: 403, body: JSON.stringify({ error: { code: "api_key_required" } }) }),
|
||||
/Nothing was charged/,
|
||||
);
|
||||
// Every other failure keeps HeyGen's own words.
|
||||
assert.equal(
|
||||
billingRefusal({
|
||||
status: 400,
|
||||
body: JSON.stringify({ error: { code: "invalid_parameter", message: "bad voice" } }),
|
||||
}),
|
||||
null,
|
||||
);
|
||||
assert.equal(
|
||||
heygenMessage({ status: 400, body: JSON.stringify({ error: { message: "bad voice" } }) }),
|
||||
"bad voice",
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
# HeyGen from the HyperFrames CLI
|
||||
|
||||
`media-use generate` and `transcribe --engine heygen` call HeyGen directly with the HyperFrames sign-in; they need no `heygen` CLI. Every paid call carries a key derived from what it asks, so running the same command again picks up the same job instead of paying twice (a failed or refused one is sent as a new job), and each job is logged in the project's `.hyperframes/heygen-jobs.jsonl`.
|
||||
|
||||
## Signing in
|
||||
|
||||
`npx hyperframes auth status` says what is signed in. The credential is read in this order: `HEYGEN_API_KEY`, `HYPERFRAMES_API_KEY`, a host app's `HEYGEN_ACCESS_TOKEN`, then `~/.hyperframes/credentials`, which `npx hyperframes auth login` writes. `heygen auth login` is a separate sign-in that only the `heygen` CLI reads.
|
||||
|
||||
| Credential | Pays with | Covers |
|
||||
| -------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------- |
|
||||
| API key | API credits | everything below |
|
||||
| Sign-in (OAuth) | the plan's allowance on subscription credits | avatar videos, video shots, images and voices; music and transcription take an API key |
|
||||
| Inside the HyperFrames app | the app | whatever the app allows; nothing to set up |
|
||||
|
||||
A refusal names its fix: `api_key_required` needs an API key, `402` or `insufficient_credit` needs a top-up, `hf_usage_exhausted` means the plan's allowance is used up.
|
||||
|
||||
## Making media: `media-use generate`
|
||||
|
||||
Make it without quoting a price or asking for a budget; `--plan` only prints what a call would cost (`credits` is an estimate in HyperFrames credits), for your own planning; never pass it on. When the person names a credit limit, keep under it with `--max-credits` (SKILL.md, Credits). Make a piece again only for a fault you can name (a cut inside it, text in it, the wrong framing), once, never to chase a motion or look score; a `content_policy_violation` gets one reworded try. A failed or refused job also gets one more try, test calls included; then stop and ask the person.
|
||||
|
||||
```bash
|
||||
# a still, edited from a reference picture: a clean plate with the words removed
|
||||
npx hyperframes media-use generate --type image --reference assets/frame.png --prompt "the same shot with no text or logos" --out assets/plate.png --plan
|
||||
# a shot of 5–15 s, starting from a first frame; longer shots are shots chained from each one's last frame
|
||||
npx hyperframes media-use generate --type video --first-frame assets/s03-first.png --prompt "slow push in" --seconds 8 --out assets/s03.mp4
|
||||
# a person re-shot from a still over one shot of the original (never across a cut), saying that shot's voice with lips
|
||||
# in sync and moving as the guide (the original shot itself) does; --mask x,y,w,h[,a-b] (repeatable, the guide's
|
||||
# pixels and seconds) blanks text in the guide, which HeyGen would copy
|
||||
npx hyperframes media-use generate --type video --reference assets/person.png --audio assets/s02-voice.m4a --motion assets/s02-original.mp4 --mask 110,615,390,165,0.4-2.0 --prompt "talks to camera, seated" --seconds 8 --aspect 9:16 --out assets/s02-person.mp4
|
||||
# the same for a person who does not speak: no --audio, the guide clip carries the motion
|
||||
npx hyperframes media-use generate --type video --reference assets/person.png --motion assets/s05-original.mp4 --prompt "holds up a jar, smiles" --seconds 6 --aspect 9:16 --out assets/s05-person.mp4
|
||||
# a person in a photo speaking a script in a HeyGen voice
|
||||
npx hyperframes media-use generate --type avatar --photo assets/person.png --script "…" --voice <voice-id> --out assets/person.mp4
|
||||
# a voice reading text
|
||||
npx hyperframes media-use generate --type voice --script "…" --voice <voice-id> --out assets/vo.mp3
|
||||
# music, 30, 60, 120 or 180 s (instrumental unless --vocals), trimmed to length in the composition
|
||||
npx hyperframes media-use generate --type music --prompt "warm lo-fi, soft keys" --seconds 30 --bpm 90 --out assets/music.mp3
|
||||
```
|
||||
|
||||
The file lands at `--out` and the media ledger records it as generated, with the route, model, job, prompt and the files it was made from. Pictures and shots carry no text or logos: ask for clean plates and draw words in HyperFrames.
|
||||
|
||||
| Type | Model | Price |
|
||||
| ------ | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| image | nano-banana-2 (default), or gpt-image-2.5 with `--model` | $0.067 an image; gpt-image-2.5 $0.036–0.053 by aspect, plus $0.0082 a reference after the first |
|
||||
| video | heygen-video-1, 768p | $0.02 a second from a prompt or first frame; $0.05 a second with a reference picture, voice or guide clip, and the guide's seconds count too (1080p costs about twice) |
|
||||
| avatar | talking photo | $4.83 a minute of speech |
|
||||
| music | lyria-3-pro | $0.08 a track |
|
||||
| voice | the voice's engine | charged per character, not priced ahead |
|
||||
|
||||
`--plan`'s `usd` is what an API key is charged (HeyGen rounds up to units of $1/60); its `credits` is what a HyperFrames plan charges, on the plan's own rates: a video shot's seconds and guide seconds, plus each picture past the first two. Voice ids come from `heygen voice list` or `npx hyperframes media-use resolve --type voice`; a cloned voice from `heygen voice clone create` (`operations.md`).
|
||||
|
||||
## Transcription: `transcribe --engine heygen`
|
||||
|
||||
```bash
|
||||
npx hyperframes transcribe interview.mp4 --engine heygen --speakers
|
||||
```
|
||||
|
||||
Word times from HeyGen's speech model, each word with its speaker under `--speakers`, in the same `transcript.json` local engines write. It takes an API key. The local engines (`--engine auto`) are free and good for one clear voice; HeyGen is the better choice for several speakers, noise or music under the voice.
|
||||
@@ -52,15 +52,12 @@ These have a local option AND a higher-quality HeyGen-CLI option. Run the local
|
||||
one for free/offline; use the HeyGen CLI when quality matters. Showing the user
|
||||
a **side-by-side** (local vs HeyGen) is the honest way to let them choose.
|
||||
|
||||
| Op | Local (free) | HeyGen CLI (quality) |
|
||||
| ----------------------- | -------------------------------------------------- | --------------------------- |
|
||||
| Background removal | `hyperframes remove-background in.png` (u2net) | `heygen background-removal` |
|
||||
| Upscale | `realesrgan-ncnn-vulkan -i in.png -o out.png -s 4` | n/a |
|
||||
| Lipsync (dub) | n/a | `heygen lipsync` |
|
||||
| Translate | n/a | `heygen video-translate` |
|
||||
| Clone a voice | n/a | `heygen voice clone create` |
|
||||
| Text to video | n/a | `heygen video-agent create` |
|
||||
| Clips from a long video | n/a | `heygen ai-clipping` |
|
||||
| Op | Local (free) | HeyGen CLI (quality) |
|
||||
| ------------------ | -------------------------------------------------- | --------------------------- |
|
||||
| Background removal | `hyperframes remove-background in.png` (u2net) | `heygen background-removal` |
|
||||
| Upscale | `realesrgan-ncnn-vulkan -i in.png -o out.png -s 4` | n/a |
|
||||
| Lipsync (dub) | n/a | `heygen lipsync` |
|
||||
| Translate | n/a | `heygen video-translate` |
|
||||
|
||||
After any op: `resolve --from out.ext --type <type>` to register the derived
|
||||
asset (it records provenance and auto-promotes to the global cache).
|
||||
|
||||
@@ -52,8 +52,9 @@ in `references/operations.md`).
|
||||
|
||||
Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free,
|
||||
private, offline once cached). The `codex` CLI remains the ChatGPT-sub image
|
||||
upsell. Paid calls follow the Credits rule in `SKILL.md`: no prices unasked,
|
||||
and a question only when a step won't fit.
|
||||
upsell. Cost rule (X4): the agent confirms before an agent-initiated paid call;
|
||||
a user-requested one just runs — `heygen.video` is flagged paid (metered free
|
||||
allowance) so an agent-initiated `resolve --type video` confirms first.
|
||||
|
||||
To force a specific generator (e.g. a user says "make this image with codex"),
|
||||
pass `--provider codex`: it pins resolution to that provider and skips the
|
||||
@@ -73,15 +74,15 @@ tools are OPT-IN alternatives where they exist; install one to unlock its free,
|
||||
private, on-device path instead of or ahead of HeyGen for that type. Only
|
||||
`ffmpeg`/`ffprobe` are strictly required for the tool to run at all.
|
||||
|
||||
| Tool | Serves | Install |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `ffmpeg`/`ffprobe` | adopt probing, smart-grade signalstats, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
|
||||
| `heygen` | catalog (bgm/image/icon) + TTS (voice) + avatar video — the HyperFrames-authenticated path | install through [verified HeyGen release instructions](https://developers.heygen.com/cli), then `npx hyperframes auth login` (helper needs >= v0.3.0) |
|
||||
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
|
||||
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
|
||||
| `parakeet-mlx` | local transcription (default ASR, best) on Apple Silicon | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
|
||||
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
|
||||
| `npx hyperframes` | Kokoro TTS (voice), Parakeet or whisper.cpp (transcribe), remove-background | via the hyperframes CLI; Kokoro needs Python 3.10+ with `pip install kokoro-onnx soundfile`; Parakeet on macOS, Windows and glibc 2.32+ Linux: `npx hyperframes models install parakeet`; whisper.cpp is built on first use (Homebrew on macOS, else git+cmake) |
|
||||
| Tool | Serves | Install |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `ffmpeg`/`ffprobe` | adopt probing, smart-grade signalstats, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
|
||||
| `heygen` | catalog (bgm/image/icon) + TTS (voice) + avatar video — the HyperFrames-authenticated path | install through [verified HeyGen release instructions](https://developers.heygen.com/cli), then `npx hyperframes auth login` (helper needs >= v0.3.0) |
|
||||
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
|
||||
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
|
||||
| `parakeet-mlx` | local transcription (default ASR, best) on Apple Silicon | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
|
||||
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
|
||||
| `npx hyperframes` | Kokoro TTS (voice), Parakeet or whisper.cpp (transcribe), remove-background | via the hyperframes CLI; Parakeet on macOS, Windows and glibc 2.32+ Linux: `npx hyperframes models install parakeet`; whisper.cpp is built on first use (Homebrew on macOS, else git+cmake) |
|
||||
|
||||
The RAM-graded local-model shortlist + exact per-tier install/invoke lives in
|
||||
the CLI's bundled media-use engine, not in the standalone skill. Without a tool on PATH, its provider
|
||||
|
||||
@@ -22,7 +22,7 @@ Code that depends on wall-clock time, live physics or the network can only come
|
||||
1. For Remotion, run `python3 <SKILL_DIR>/scripts/lint_source.py <source dir>` first. Blockers (`useState`, `useReducer`, `useEffect` with dependencies, async `calculateMetadata`, React UI kits such as MUI, Chakra, Mantine, antd, shadcn, Radix, NextUI) don't translate: host them or refactor them out, never approximate (`references/escape-hatch.md`). Warnings (`@remotion/lambda` config, `delayRender`, `useMemo`, custom hooks) are dropped and noted.
|
||||
2. Translate to the `/hyperframes` contract: one sub-composition per scene, times in seconds (`frame / fps`), asset paths root-relative. Each Remotion API's equivalent is in `references/api-map.md`. Put Remotion's render-page rule `* { box-sizing: border-box; }` once in the root `index.html` `<style>`: without it every bordered or padded element grows by its border and padding.
|
||||
3. Keep it editable: the source's props become `data-composition-variables` bound with `data-var-text`, `data-var-src` or `var(--id)`; each `Sequence` becomes a timed clip or a sub-composition; a reused component becomes one sub-composition mounted several times with `data-variable-values`.
|
||||
4. Compare frame by frame: `npx hyperframes diff <original.mp4> . --at <t,…>` scores your project against the original at those times without rendering; sample every scene and both sides of every cut. Then render and `npx hyperframes diff <original.mp4> <hf.mp4>` scores every frame: SSIM, the worst cell of each frame (one late or recoloured element), cuts, and a sheet of the worst moments (thresholds per complexity tier in `references/eval.md`). Render the Remotion side with `Config.setVideoImageFormat("png")` and `Config.setColorSpace("bt709")`, or the diff measures the encoders (about 0.05 SSIM).
|
||||
4. Compare frame by frame: `npx hyperframes snapshot --at <t,…> --against <original.mp4>` saves the original's frame at each time next to yours. Sample every scene and both sides of every cut; look for one late or recoloured element, not only the overall picture. For a scored check, `scripts/render_diff.sh <original.mp4> <hf.mp4> <out>` gives SSIM per frame and `scripts/frame_strip.sh` shows which frames diverged (thresholds per complexity tier in `references/eval.md`). Render the Remotion side with `Config.setVideoImageFormat("png")` and `Config.setColorSpace("bt709")`, or the diff measures the encoders (about 0.05 SSIM).
|
||||
5. Write what didn't translate cleanly (dropped volume ramps, approximated presentations, substituted fonts) to `TRANSLATION_NOTES.md`.
|
||||
|
||||
| Remotion | Read |
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
#
|
||||
# Output:
|
||||
# <fixture>/diff/summary.json per-fixture SSIM summary
|
||||
# <fixture>/diff/worst.jpg per-fixture sheet of the worst moments
|
||||
# <fixture>/strip/strip.png per-fixture comparison strip (only on fail)
|
||||
# ./run-report.json aggregate report
|
||||
|
||||
set -euo pipefail
|
||||
@@ -29,6 +29,8 @@ SKILL_DIR="$(cd "$THIS_DIR/../.." && pwd)"
|
||||
REPO_ROOT="$(cd "$SKILL_DIR/../.." && pwd)"
|
||||
|
||||
LINT="$SKILL_DIR/scripts/lint_source.py"
|
||||
DIFF="$SKILL_DIR/scripts/render_diff.sh"
|
||||
STRIP="$SKILL_DIR/scripts/frame_strip.sh"
|
||||
HF_CLI="$REPO_ROOT/packages/cli/dist/cli.js"
|
||||
REPORT="$THIS_DIR/run-report.json"
|
||||
|
||||
@@ -144,21 +146,22 @@ run_render_tier() {
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Gate on the tier's mean threshold only, as the calibration runs did; the
|
||||
# worst-moment sheet lands in diff/worst.jpg either way.
|
||||
mkdir -p "$fixture_dir/diff"
|
||||
local mean
|
||||
if node "$HF_CLI" diff \
|
||||
if R2HF_SSIM_THRESHOLD="$threshold" "$DIFF" \
|
||||
"$fixture_dir/remotion-src/out/baseline.mp4" \
|
||||
"$fixture_dir/hf.mp4" \
|
||||
--min-mean "$threshold" --min-p05 0 --min-cell 0 \
|
||||
--out "$fixture_dir/diff" --json >"$fixture_dir/diff/summary.json" 2>/dev/null; then
|
||||
mean=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['ssim']['mean'])" "$fixture_dir/diff/summary.json")
|
||||
"$fixture_dir/diff" >/dev/null; then
|
||||
local mean
|
||||
mean=$(read_json_value "$fixture_dir/diff/summary.json" "mean")
|
||||
echo " ✓ pass (mean SSIM $mean, threshold $threshold)"
|
||||
write_result "$fixture_name" "pass" mean_ssim "$mean" threshold "$threshold"
|
||||
else
|
||||
mean=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1])).get('ssim', {}).get('mean', ''))" "$fixture_dir/diff/summary.json")
|
||||
echo " ✗ fail (mean SSIM $mean, threshold $threshold; sheet $fixture_dir/diff/worst.jpg)"
|
||||
local mean
|
||||
mean=$(read_json_value "$fixture_dir/diff/summary.json" "mean")
|
||||
echo " ✗ fail (mean SSIM $mean, threshold $threshold)"
|
||||
"$STRIP" \
|
||||
"$fixture_dir/remotion-src/out/baseline.mp4" \
|
||||
"$fixture_dir/hf.mp4" \
|
||||
"$fixture_dir/strip" 8 >/dev/null
|
||||
write_result "$fixture_name" "fail" stage "ssim" mean_ssim "$mean" threshold "$threshold"
|
||||
fi
|
||||
}
|
||||
|
||||
@@ -33,8 +33,8 @@ cd remotion-src && npm install && npm run render
|
||||
# Render HyperFrames translation
|
||||
cd ../hf-src && npx hyperframes render --output ../hf.mp4
|
||||
|
||||
# Compare with the HF CLI
|
||||
npx hyperframes diff ../remotion-src/out/baseline.mp4 ../hf.mp4 --out ../diff
|
||||
# Compare with the eval harness (from skill scripts/)
|
||||
../../../scripts/render_diff.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./diff
|
||||
```
|
||||
|
||||
`expected.json` documents the SSIM threshold (0.95) for this fixture; the
|
||||
|
||||
@@ -43,7 +43,7 @@ cd remotion-src && npm install && npm run render
|
||||
cd ../hf-src && npx hyperframes render --output ../hf.mp4
|
||||
|
||||
# 4. Compare
|
||||
npx hyperframes diff ../remotion-src/out/baseline.mp4 ../hf.mp4 --out ../diff
|
||||
../../../scripts/render_diff.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./diff
|
||||
```
|
||||
|
||||
## Why threshold 0.95?
|
||||
|
||||
@@ -82,5 +82,5 @@ cd remotion-src && npm install && npm run render
|
||||
cd ../hf-src && npx hyperframes render --output ../hf.mp4
|
||||
|
||||
# Compare
|
||||
npx hyperframes diff ../remotion-src/out/baseline.mp4 ../hf.mp4 --out ../diff
|
||||
../../../scripts/render_diff.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./diff
|
||||
```
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
# Eval: how to validate a translation end-to-end
|
||||
|
||||
Every translation should be measured. The skill ships a source linter and
|
||||
a tiered test corpus; the CLI's `diff` command scores the result. Together
|
||||
they gate translation quality.
|
||||
Every translation should be measured. The skill ships three scripts and
|
||||
a tiered test corpus that, together, gate translation quality.
|
||||
|
||||
## The tools
|
||||
## The three scripts
|
||||
|
||||
| Tool | Input | Output |
|
||||
| -------------------------- | --------------------------- | ----------------------------------------------------------------------------- |
|
||||
| `scripts/lint_source.py` | Remotion source dir or file | JSON findings + exit code (0 clean, 1 has blockers) |
|
||||
| `npx hyperframes diff A B` | original MP4, HF MP4 | per-frame SSIM, worst cell per frame, cuts, `worst.jpg` sheet; exit 1 on fail |
|
||||
| Script | Input | Output |
|
||||
| ------------------------ | --------------------------- | ------------------------------------------------------------------- |
|
||||
| `scripts/lint_source.py` | Remotion source dir or file | JSON findings + exit code (0 clean, 1 has blockers) |
|
||||
| `scripts/render_diff.sh` | two MP4 paths | per-frame SSIM + JSON summary (`mean`, `min`, `p05`, `p95`, `pass`) |
|
||||
| `scripts/frame_strip.sh` | two MP4 paths | side-by-side comparison strip PNG for visual debugging |
|
||||
|
||||
Run them in this order: **lint → render → diff → (if fail) read the sheet**.
|
||||
Run them in this order: **lint → render → diff → (if fail) strip**.
|
||||
|
||||
## Per-fixture flow
|
||||
|
||||
@@ -30,38 +30,83 @@ cd remotion-src && npm install && npm run render
|
||||
cd .. && node ../../../packages/cli/dist/cli.js render hf-src/ --output hf.mp4
|
||||
# -> hf.mp4
|
||||
|
||||
# 5. SSIM diff, gated on the tier's mean threshold only
|
||||
mkdir -p diff && node ../../../packages/cli/dist/cli.js diff ./remotion-src/out/baseline.mp4 ./hf.mp4 \
|
||||
--min-mean 0.95 --min-p05 0 --min-cell 0 --out ./diff --json > diff/summary.json
|
||||
# -> diff/summary.json, diff/worst.jpg, diff/ssim.log
|
||||
```
|
||||
# 5. SSIM diff
|
||||
../../scripts/render_diff.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./diff
|
||||
# -> diff/summary.json
|
||||
|
||||
Before rendering, `npx hyperframes diff <baseline.mp4> hf-src --at 0.5,2,4.1`
|
||||
scores the project at chosen times with no render.
|
||||
# 6. If diff fails, generate frame strip for visual inspection
|
||||
../../scripts/frame_strip.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./strip 8
|
||||
# -> strip/strip.png
|
||||
```
|
||||
|
||||
## Reading `diff/summary.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"pass": true,
|
||||
"ssim": { "frames": 90, "mean": 0.974, "p05": 0.972, "min": 0.971, "cell_p05": 0.93 },
|
||||
"worst": [
|
||||
{ "t": 1.2, "frame": 36, "ssim": 0.971, "cell": { "roi": [480, 270, 480, 270], "ssim": 0.88 } }
|
||||
],
|
||||
"cuts": { "reference": [1.5], "yours": [1.5], "missing": [], "extra": [], "pass": true },
|
||||
"duration": { "reference": 3, "yours": 3, "pass": true },
|
||||
"sheet": "diff/worst.jpg"
|
||||
"frame_count": 90,
|
||||
"mean": 0.974,
|
||||
"min": 0.972,
|
||||
"max": 0.999,
|
||||
"p05": 0.972,
|
||||
"p95": 0.983,
|
||||
"threshold": 0.95,
|
||||
"pass": true
|
||||
}
|
||||
```
|
||||
|
||||
| Field | What it tells you |
|
||||
| --------------- | -------------------------------------------------------------------------------- |
|
||||
| `ssim.mean` | average SSIM across all frames; the headline number |
|
||||
| `ssim.p05` | 5th percentile: most frames sit above it |
|
||||
| `ssim.cell_p05` | each frame's worst cell of a 4x4 grid; a wrong element shows here, not in `mean` |
|
||||
| `worst` | the lowest-scoring moments, each with the cell (in original pixels) that failed |
|
||||
| `cuts` | hard cuts on both sides; a missing or extra one is a sequencing problem |
|
||||
| `pass` | every gate held: `--min-mean`, `--min-p05`, `--min-cell`, cuts and duration |
|
||||
| Field | What it tells you |
|
||||
| ------------- | --------------------------------------------------------------------------- |
|
||||
| `mean` | average SSIM across all frames; the headline number |
|
||||
| `min` | worst frame; below threshold means at least one frame is structurally wrong |
|
||||
| `p05` / `p95` | 5th / 95th percentile — most frames sit between these |
|
||||
| `threshold` | from `R2HF_SSIM_THRESHOLD` env var (default 0.85) |
|
||||
| `pass` | whether `mean >= threshold` |
|
||||
|
||||
## Validated tier thresholds
|
||||
|
||||
Calibrated against actual Remotion + HF renders:
|
||||
|
||||
| Tier | Composition shape | Mean | Threshold | Margin |
|
||||
| ---- | ------------------------------------------- | ----- | --------- | ------ |
|
||||
| T1 | single-element fade-in | 0.974 | 0.95 | +0.022 |
|
||||
| T2 | multi-scene + spring + audio + image | 0.985 | 0.95 | +0.016 |
|
||||
| T3 | data-driven, custom subcomponents, count-up | 0.953 | 0.90 | +0.038 |
|
||||
|
||||
Each fixture's `expected.json` carries:
|
||||
|
||||
- `ssim_threshold` — the gate for `pass`
|
||||
- `validation` — the actual measured numbers from the calibration run
|
||||
- `translation_notes` — what's lossy and why
|
||||
|
||||
## Critical: encoder config
|
||||
|
||||
Both Remotion and HF must output the same pixel format for SSIM to be
|
||||
meaningful. Remotion's default JPEG output writes `yuvj420p` (full-range);
|
||||
HF outputs `yuv420p` (limited-range). The mismatch costs ~0.05 SSIM.
|
||||
|
||||
Every fixture's `remotion.config.ts` sets:
|
||||
|
||||
```ts
|
||||
Config.setVideoImageFormat("png");
|
||||
Config.setColorSpace("bt709");
|
||||
```
|
||||
|
||||
If the user's source doesn't have these, add them in the translation
|
||||
step — otherwise the diff measures encoder differences, not translation
|
||||
fidelity.
|
||||
|
||||
## What the noise floor looks like
|
||||
|
||||
The dominant non-translation noise is **system font fallback divergence**.
|
||||
Remotion's bundled Chromium and HF's `chrome-headless-shell` interpret
|
||||
`font-weight: 800` differently when there's no real font installed:
|
||||
|
||||
- Remotion HELLO at 160px: medium-weight stroke
|
||||
- HF HELLO at 160px: heavy-weight stroke
|
||||
|
||||
This costs ~0.025 mean SSIM. Visible in T1's frame strip.
|
||||
[fonts.md](fonts.md) covers how to mitigate (use Inter, load explicit
|
||||
Google Fonts).
|
||||
|
||||
## Threshold rule of thumb
|
||||
|
||||
@@ -76,8 +121,8 @@ re-rendered on different hardware will drift.
|
||||
|
||||
## When the diff fails
|
||||
|
||||
1. **Look at `diff/worst.jpg` first.** The worst moments side by side, each
|
||||
with its failing cell boxed, show whether the failure is structural
|
||||
1. **Look at `frame_strip.sh` output first.** A side-by-side strip at 6–10
|
||||
evenly-spaced timestamps shows whether the failure is structural
|
||||
(wrong scene durations, missing element) or cosmetic (different font
|
||||
weight, slight timing skew).
|
||||
2. **Check `diff/ssim.log`.** Per-frame SSIM tells you _which_ frames
|
||||
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env bash
|
||||
# frame_strip.sh — produce a side-by-side comparison strip from two videos.
|
||||
#
|
||||
# Used to debug failing render_diff.sh runs visually: pick a sample timestamp
|
||||
# range, extract frames from both videos, lay them out as a grid for review.
|
||||
#
|
||||
# Usage:
|
||||
# frame_strip.sh <baseline.mp4> <translated.mp4> [output-dir] [samples]
|
||||
#
|
||||
# Defaults: output-dir=./strip-out, samples=8 (evenly spaced across duration).
|
||||
# Output:
|
||||
# strip.png — single PNG with `samples` rows, each row is
|
||||
# (baseline frame | translated frame) at one timestamp
|
||||
# timestamps.txt — the timestamps sampled
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
if [[ $# -lt 2 || $# -gt 4 ]]; then
|
||||
echo "usage: $0 <baseline.mp4> <translated.mp4> [output-dir] [samples]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
BASELINE="$1"
|
||||
TRANSLATED="$2"
|
||||
OUTDIR="${3:-./strip-out}"
|
||||
SAMPLES="${4:-8}"
|
||||
|
||||
if ! command -v ffmpeg >/dev/null 2>&1 || ! command -v ffprobe >/dev/null 2>&1; then
|
||||
echo "error: ffmpeg/ffprobe not on PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
mkdir -p "$OUTDIR"
|
||||
|
||||
# Build the strip in a single ffmpeg invocation. Two inputs (baseline and
|
||||
# translated) are sampled at N evenly-spaced timestamps via the `select`
|
||||
# filter, then assembled with hstack (per-row pairs) + vstack (rows).
|
||||
# Earlier versions spawned 3 ffmpeg calls per timestamp + a final vstack;
|
||||
# this is one call regardless of N.
|
||||
python3 - "$BASELINE" "$TRANSLATED" "$OUTDIR" "$SAMPLES" <<'PY'
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
baseline, translated, outdir, samples = sys.argv[1], sys.argv[2], Path(sys.argv[3]), int(sys.argv[4])
|
||||
|
||||
# Read fps + duration from the baseline so we can map timestamps to frame
|
||||
# indexes for the `select` filter (frame-accurate, doesn't depend on
|
||||
# keyframe alignment).
|
||||
probe = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=r_frame_rate,nb_read_frames,duration",
|
||||
"-show_entries", "format=duration",
|
||||
"-of", "json", "-count_frames", "--", baseline],
|
||||
check=True, capture_output=True, text=True,
|
||||
)
|
||||
data = json.loads(probe.stdout)
|
||||
stream = data["streams"][0]
|
||||
num, den = stream["r_frame_rate"].split("/")
|
||||
fps = float(num) / float(den)
|
||||
nb_frames = int(stream.get("nb_read_frames") or 0)
|
||||
if nb_frames <= 0:
|
||||
duration = float(stream.get("duration") or data["format"]["duration"])
|
||||
nb_frames = int(duration * fps)
|
||||
|
||||
# Even-spaced sample frames in the 5%-95% window (skip fade-in/out noise).
|
||||
start = max(0, int(nb_frames * 0.05))
|
||||
end = max(start, int(nb_frames * 0.95) - 1)
|
||||
if samples == 1:
|
||||
frames = [start]
|
||||
else:
|
||||
step = (end - start) / (samples - 1)
|
||||
frames = [int(start + i * step) for i in range(samples)]
|
||||
|
||||
(outdir / "timestamps.txt").write_text(
|
||||
"\n".join(f"{f / fps:.3f}" for f in frames) + "\n"
|
||||
)
|
||||
|
||||
# select='eq(n,F1)+eq(n,F2)+...' picks exactly the listed frames from each
|
||||
# input. We then hstack per-frame pairs and vstack the result.
|
||||
select_expr = "+".join(f"eq(n,{f})" for f in frames)
|
||||
n = len(frames)
|
||||
filter_parts = [
|
||||
f"[0:v]select='{select_expr}',setpts=N/FRAME_RATE/TB,split={n}"
|
||||
+ "".join(f"[b{i}]" for i in range(n)),
|
||||
f"[1:v]select='{select_expr}',setpts=N/FRAME_RATE/TB,split={n}"
|
||||
+ "".join(f"[t{i}]" for i in range(n)),
|
||||
]
|
||||
for i in range(n):
|
||||
filter_parts.append(f"[b{i}][t{i}]hstack=inputs=2[row{i}]")
|
||||
filter_parts.append(
|
||||
"".join(f"[row{i}]" for i in range(n)) + f"vstack=inputs={n}[out]"
|
||||
)
|
||||
filter_graph = ";".join(filter_parts)
|
||||
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-i", baseline, "-i", translated,
|
||||
"-filter_complex", filter_graph,
|
||||
"-map", "[out]", "-frames:v", "1",
|
||||
str(outdir / "strip.png"),
|
||||
]
|
||||
subprocess.run(cmd, check=True)
|
||||
print(f"wrote {outdir / 'strip.png'} ({n} samples)")
|
||||
PY
|
||||
+103
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env bash
|
||||
# render_diff.sh — compute per-frame SSIM between two video files.
|
||||
#
|
||||
# The eval primitive for the remotion-to-hyperframes skill: given a Remotion
|
||||
# render and a HyperFrames render of the same composition, report whether the
|
||||
# translation is visually equivalent.
|
||||
#
|
||||
# Usage:
|
||||
# render_diff.sh <baseline.mp4> <translated.mp4> [output-dir]
|
||||
#
|
||||
# Output (in output-dir, defaults to ./diff-out):
|
||||
# ssim.log — per-frame SSIM lines from ffmpeg
|
||||
# summary.json — { mean, min, p05, p95, frame_count, pass, threshold }
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — pass (mean SSIM >= threshold)
|
||||
# 1 — fail (mean SSIM < threshold)
|
||||
# 2 — usage / setup error
|
||||
#
|
||||
# Threshold defaults to 0.85 (loose; tier-specific thresholds are applied by
|
||||
# the orchestrator). Override with R2HF_SSIM_THRESHOLD=0.95 in the environment.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
THRESHOLD="${R2HF_SSIM_THRESHOLD:-0.85}"
|
||||
|
||||
if [[ $# -lt 2 || $# -gt 3 ]]; then
|
||||
echo "usage: $0 <baseline.mp4> <translated.mp4> [output-dir]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
BASELINE="$1"
|
||||
TRANSLATED="$2"
|
||||
OUTDIR="${3:-./diff-out}"
|
||||
|
||||
if [[ ! -f "$BASELINE" ]]; then
|
||||
echo "error: baseline not found: $BASELINE" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [[ ! -f "$TRANSLATED" ]]; then
|
||||
echo "error: translated not found: $TRANSLATED" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ! command -v ffmpeg >/dev/null 2>&1; then
|
||||
echo "error: ffmpeg not on PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
mkdir -p "$OUTDIR"
|
||||
SSIM_LOG="$OUTDIR/ssim.log"
|
||||
SUMMARY="$OUTDIR/summary.json"
|
||||
|
||||
# ffmpeg's ssim filter writes one line per frame to stats_file and a single
|
||||
# Mean SSIM line to stderr. We capture both — per-frame for distribution
|
||||
# stats, and the mean for the headline number.
|
||||
ffmpeg -hide_banner -nostats -loglevel info \
|
||||
-i "$BASELINE" -i "$TRANSLATED" \
|
||||
-lavfi "[0:v]scale=iw:ih[ref];[1:v]scale=iw:ih[main];[main][ref]ssim=stats_file=$SSIM_LOG" \
|
||||
-f null - 2>"$OUTDIR/ffmpeg.stderr"
|
||||
|
||||
# Parse: each line in ssim.log looks like
|
||||
# n:1 Y:0.987655 U:0.992345 V:0.991234 All:0.989012 (19.512345)
|
||||
# We want the All:N column.
|
||||
python3 - "$SSIM_LOG" "$SUMMARY" "$THRESHOLD" <<'PY'
|
||||
import json, math, re, sys
|
||||
from pathlib import Path
|
||||
|
||||
log_path = Path(sys.argv[1])
|
||||
out_path = Path(sys.argv[2])
|
||||
threshold = float(sys.argv[3])
|
||||
|
||||
values = []
|
||||
pattern = re.compile(r"All:([\d.]+)")
|
||||
for line in log_path.read_text().splitlines():
|
||||
m = pattern.search(line)
|
||||
if m:
|
||||
try:
|
||||
values.append(float(m.group(1)))
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
if not values:
|
||||
print(f"error: no SSIM samples parsed from {log_path}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
values.sort()
|
||||
n = len(values)
|
||||
mean = sum(values) / n
|
||||
p_idx = lambda p: min(n - 1, max(0, int(math.floor(p * n))))
|
||||
summary = {
|
||||
"frame_count": n,
|
||||
"mean": round(mean, 6),
|
||||
"min": round(values[0], 6),
|
||||
"max": round(values[-1], 6),
|
||||
"p05": round(values[p_idx(0.05)], 6),
|
||||
"p95": round(values[p_idx(0.95)], 6),
|
||||
"threshold": threshold,
|
||||
"pass": bool(mean >= threshold),
|
||||
}
|
||||
out_path.write_text(json.dumps(summary, indent=2) + "\n")
|
||||
print(json.dumps(summary, indent=2))
|
||||
sys.exit(0 if summary["pass"] else 1)
|
||||
PY
|
||||
@@ -1,7 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# smoke.sh — exercise the skill's source linter against fixture .tsx files.
|
||||
# smoke.sh — exercise the eval harness scripts against synthetic inputs.
|
||||
#
|
||||
# Scoring a translation is the CLI's `hyperframes diff`, tested with the CLI.
|
||||
# Generates two synthetic videos with ffmpeg's testsrc filter, runs render_diff
|
||||
# and frame_strip against them, and runs lint_source against fixture .tsx files.
|
||||
# Asserts the harness produces sensible output without depending on a real
|
||||
# Remotion or HyperFrames render pipeline being installed.
|
||||
#
|
||||
# Usage: ./smoke.sh
|
||||
# Exit 0 on pass.
|
||||
@@ -13,6 +16,50 @@ SCRIPTS_DIR="$(cd "$THIS_DIR/.." && pwd)"
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
|
||||
echo "==> smoke: render_diff.sh against identical inputs"
|
||||
# Generate the same test pattern twice. Identical inputs → SSIM should be ~1.0.
|
||||
ffmpeg -y -hide_banner -loglevel error \
|
||||
-f lavfi -i "testsrc=duration=2:size=320x240:rate=30" \
|
||||
-pix_fmt yuv420p "$WORK/baseline.mp4"
|
||||
cp "$WORK/baseline.mp4" "$WORK/translated.mp4"
|
||||
|
||||
R2HF_SSIM_THRESHOLD=0.99 "$SCRIPTS_DIR/render_diff.sh" \
|
||||
"$WORK/baseline.mp4" "$WORK/translated.mp4" "$WORK/diff" >/dev/null
|
||||
|
||||
MEAN=$(python3 -c "import json,sys; print(json.load(open('$WORK/diff/summary.json'))['mean'])")
|
||||
PASS=$(python3 -c "import json,sys; print(json.load(open('$WORK/diff/summary.json'))['pass'])")
|
||||
if [[ "$PASS" != "True" ]]; then
|
||||
echo "FAIL: identical inputs failed pass check (mean=$MEAN)"
|
||||
exit 1
|
||||
fi
|
||||
echo " identical inputs → mean SSIM=$MEAN (pass=True)"
|
||||
|
||||
echo "==> smoke: render_diff.sh against different inputs"
|
||||
# Different test pattern → SSIM should be lower. With a high threshold it should fail.
|
||||
ffmpeg -y -hide_banner -loglevel error \
|
||||
-f lavfi -i "testsrc2=duration=2:size=320x240:rate=30" \
|
||||
-pix_fmt yuv420p "$WORK/different.mp4"
|
||||
|
||||
set +e
|
||||
R2HF_SSIM_THRESHOLD=0.99 "$SCRIPTS_DIR/render_diff.sh" \
|
||||
"$WORK/baseline.mp4" "$WORK/different.mp4" "$WORK/diff2" >/dev/null
|
||||
RC=$?
|
||||
set -e
|
||||
if [[ "$RC" -eq 0 ]]; then
|
||||
echo "FAIL: different inputs unexpectedly passed at threshold 0.99"
|
||||
exit 1
|
||||
fi
|
||||
DIFF_MEAN=$(python3 -c "import json; print(json.load(open('$WORK/diff2/summary.json'))['mean'])")
|
||||
echo " different inputs → mean SSIM=$DIFF_MEAN (correctly failed at 0.99)"
|
||||
|
||||
echo "==> smoke: frame_strip.sh produces a strip"
|
||||
"$SCRIPTS_DIR/frame_strip.sh" "$WORK/baseline.mp4" "$WORK/different.mp4" "$WORK/strip" 4 >/dev/null
|
||||
if [[ ! -f "$WORK/strip/strip.png" ]]; then
|
||||
echo "FAIL: frame_strip.sh did not produce strip.png"
|
||||
exit 1
|
||||
fi
|
||||
echo " strip.png written ($(stat -c%s "$WORK/strip/strip.png" 2>/dev/null || stat -f%z "$WORK/strip/strip.png") bytes)"
|
||||
|
||||
echo "==> smoke: lint_source.py on clean fixture (expect exit 0)"
|
||||
set +e
|
||||
python3 "$SCRIPTS_DIR/lint_source.py" "$THIS_DIR/fixtures/clean.tsx" --json >"$WORK/clean.json"
|
||||
|
||||
@@ -1,38 +0,0 @@
|
||||
---
|
||||
name: reverse-any-video
|
||||
description: Rebuild any video file as an editable HyperFrames project - motion graphics, talking heads, live action or AI-generated footage. Measure and watch it, rebuild every layer, then check the render against the original frame by frame. Use to reverse, recreate, replicate or copy a video.
|
||||
metadata:
|
||||
internal: true
|
||||
---
|
||||
|
||||
**Plugin installs:** Before setup or freshness commands, follow [plugin execution rules](../hyperframes/references/plugin-installation.md) when this skill is inside a HyperFrames plugin.
|
||||
|
||||
# Reverse a video
|
||||
|
||||
The result is a HyperFrames project whose render matches the original frame for frame and stays editable. Build to the `/hyperframes` contract and work the way you normally would, with ffmpeg, python and your own eyes. This skill holds only what earlier rebuilds got wrong or spent hours on. Reverse videos the person has the right to use.
|
||||
|
||||
Reverse is a HyperFrames Pro feature: start only when `npx hyperframes auth status` shows a sign-in on a paid plan, not Free. Open with `references/start.md`. Resuming (a `REVERSE.md` or half-built project is there) is a start too: run `auth status` again and follow `references/start.md` before acting. A step an earlier session gave up on (a re-shoot HeyGen refused) gets one more try, then ask the person. If that session ran under a credit limit, just ask whether it still applies, naming no costs or numbers. Rebuild the video as it is; remix it only when the person asks.
|
||||
|
||||
## The original decides the frame
|
||||
|
||||
1. `npx hyperframes measure probe <video>`: the root composition is that size exactly, `data-fps` is its rate (NTSC stays a fraction: `30000/1001`) and its duration is the video's frames ÷ fps, not the audio's length. `init`'s default canvas is wrong for most videos.
|
||||
2. `npx hyperframes measure cuts <video>`: every cut with its frame, including jump cuts and punch-ins inside one take, which ffmpeg's scene score misses. Every shot starts at its listed time. Write times as frame ÷ fps rounded down (frame 83 at 30 fps is `2.766666`): `2.767` lands a frame late.
|
||||
3. The sound is the original's: copy the video into the project and play it as one `<audio src="original.mp4">` over the whole film (no extracted or re-encoded track); every clip is muted.
|
||||
4. Read consecutive frames wherever something changes: each cut, entrance, exit and transition, a few frames either side (`npx hyperframes measure frames --range a,b --roi x,y,w,h`). Sheets a second or a quarter apart miss 1–3-frame timing, which way things enter and leave, blurs and dissolves at cuts, and one-frame flashes.
|
||||
|
||||
## Every layer
|
||||
|
||||
- **A component** (text, captions, stickers, cards, UI and screen recordings, charts, maps) is rebuilt in HTML.
|
||||
- **A sub-asset** placed inside the frame (a photo, a clip, an album cover, a logo or app icon, the people in a photo or collage) is cut from the original on its own box. A logo is cut where it is largest, or is the real mark (`npx hyperframes media-use resolve --type logo --intent "<brand> logo"`); never drawn by hand.
|
||||
- **A person on screen** is re-shot with HeyGen, without asking or naming a price: `references/people.md`.
|
||||
- Never play the original's whole frame.
|
||||
|
||||
What decides whether drawn things match, in `references/match.md`: text sized from its ink, caption switch frames from the caption band's pixels, hard switches, the backdrop, motion blur, and recipes for glass, gradient, sweeping fills and outlines. Look at every large word through its whole life before you build it.
|
||||
|
||||
## Check once, at the end
|
||||
|
||||
1. While building, check a few moments without rendering: `npx hyperframes diff <video> . --at t,t,…` (the original beside yours, scored).
|
||||
2. When every shot has content: `npx hyperframes check`, fix its errors, then `npx hyperframes render` in the foreground with a 10-minute timeout (a longer render: start it detached, `setsid nohup npx hyperframes render -o <out> > render.log 2>&1 &`, and wait on the log in the foreground), then `npx hyperframes diff <video> <render>`. It lists the original's cuts that yours misses and the extra ones, the duration, the sound's offset, stretches where content is missing, and the worst moments with the failing cell beside the original's.
|
||||
3. Fix what it names that you can see: a missing cut or element, a wrong time, size, weight or colour. Render and diff once more, then stop. Small score differences are not worth another round.
|
||||
|
||||
Work alone: one agent holding the whole film beats subagents that each re-read and re-watch it and wait on each other. Write a short `REVERSE.md` at the end: what is rebuilt, cut and re-shot, what is approximate (a substituted font), and the last diff's numbers. It holds no prices or credit numbers. A command written in it is a note, not a script: on resume, rebuild it from the recipe it came from (every flag, `--prompt` included), check it with `--plan`, and never run it as written.
|
||||
@@ -1,39 +0,0 @@
|
||||
# Matching what is drawn
|
||||
|
||||
Where earlier rebuilds lost points on drawn layers, and what fixed it.
|
||||
|
||||
## Timing
|
||||
|
||||
- Graphics change on exact frames. An element that is fully there on frame 241 is set there, not faded in; tween only what you saw move over several frames. A camera that lands in one frame is a `set`, not a tween.
|
||||
- Captions: take each group's first and last frame from the caption band's pixels, every frame (`npx hyperframes measure events <video> --roi <band>`, or a frame difference over the band). The words come from `npx hyperframes transcribe`, which can mishear names; the switch times come from the pixels, since captions often lead or trail the voice. Switch hard unless the original fades.
|
||||
- A blur flash, a dissolve or a wipe at a cut is part of the cut: rebuild it over its measured frames.
|
||||
|
||||
## Size, type and position
|
||||
|
||||
- Size text from its ink, not by eye: crop the line in a source frame and the same line in a snapshot of yours, compare the ink boxes' height and width, and scale size, then `letter-spacing`, until both agree within a few pixels. A substitute font usually sets wider; compare numerals too. Then compare the ink's darkness for weight, and look for an outline or shadow.
|
||||
- Match the family by its traits (geometry, contrast, x-height, terminals) and try two or three candidates at the measured size before choosing. Prefer a bundled font, then a Google Fonts family in `assets/fonts/`. Say in the report which fonts are substitutes.
|
||||
- Positions come from measured boxes in source pixels; the canvas is the source's size, so they go in as measured.
|
||||
|
||||
## Look
|
||||
|
||||
- Motion graphics: the backdrop covers most of the frame. Sample a grid of the empty backdrop at several moments and match its gradient and brightness over time.
|
||||
- Fast moves in the original are motion-blurred; sharp tiles at speed score as wrong. Blur along the move while it is fast (below).
|
||||
- Dots and pixel textures: measure the lattice (pitch, offset, colour) from a frame and render from that grid.
|
||||
- Flat graphics that move like footage (a map under a camera, a diagram that scrolls): trace each plate into flat-colour shapes and fit the camera move; it matches far better than redrawing from outside data.
|
||||
|
||||
## Every large word, through its life
|
||||
|
||||
Look at a strip of consecutive frames over its box from the frame it appears to the frame it leaves (`measure frames --range a,b --roi <its box> --max 16`). Read how it arrives (pop, scale, fade, type-on, word by word, wipe), how it changes while on screen (a fill sweeping across, a sheen, a counter, a colour change: direction and duration) and how it leaves. Then zoom in at full size and name what the letters are made of: solid, gradient, glass, outline, glow, knockout.
|
||||
|
||||
## Recipes a seeked timeline reproduces
|
||||
|
||||
- **Fill that sweeps across the letters:** two copies of the word; the lit one on top, `clip-path: inset(0 100% 0 0)` tweened to `inset(0 0 0 0)` over the measured time and direction.
|
||||
- **Gradient type:** `background: linear-gradient(<direction>, <both end colours>)` with `background-clip: text` and a transparent fill.
|
||||
- **Glass type over footage** (the letters show the footage behind them bent, with a dark rim and a light edge): a second copy of the shot inside the letters, clipped by an SVG `<clipPath>` holding the `<text>`, bent by `feTurbulence` (`baseFrequency` about 0.02, fixed seed) into `feDisplacementMap` (scale 40–60); a darker copy of the text multiplied over it at 15–35%, a 2–3 px dark stroke, a light top edge. It sits in the shot's camera group.
|
||||
- **Outline type:** `-webkit-text-stroke` at the measured width, transparent fill.
|
||||
- **Directional motion blur:** an SVG `<feGaussianBlur stdDeviation="0 0">` on the moving wrapper, its `stdDeviation` tweened (`"24 0"` smears horizontally); CSS `blur()` is symmetric and wrong.
|
||||
- **Ghost trail:** 2–4 clones behind the element along the move, opacity about 0.5, 0.3, 0.15.
|
||||
- **Glow:** stacked `text-shadow` or `drop-shadow`, tweening radius and alpha.
|
||||
- **Masks and wipes:** `clip-path` or `mask-image` with a tweened stop.
|
||||
- **Flash:** a full-frame layer for exactly the measured frames.
|
||||
- **Grain:** SVG `feTurbulence` with a fixed seed, stepped per frame for moving grain. Grain lowers SSIM whatever you do: match its look, not its pixels.
|
||||
@@ -1,25 +0,0 @@
|
||||
# Re-shooting a person
|
||||
|
||||
Every person on screen, speaking or not, is re-shot with HeyGen so the original's face and the text burned over it never ship. What makes a re-shot person match is the original's own clip driving the motion, and one generated shot per original shot. The HyperFrames sign-in is enough; no API key is needed. Only when the person says not to use HeyGen, cut them on their own box instead and say the burned-in text came along.
|
||||
|
||||
## The recipe
|
||||
|
||||
1. **One shot per original shot.** From `measure cuts`: a generated shot never spans a cut, jump cuts and punch-ins included. One take across a jump cut deletes the cut; a motion guide that crosses a cut makes HeyGen cut somewhere of its own. HeyGen makes 5–15 s: a shorter shot is asked for at 5 s and trimmed; a longer one is split where the original reframes, else chained: the next part's reference is the last frame of the one before.
|
||||
2. **A plate per shot:** a real frame of that shot at the source's aspect, with every drawn thing and every other face (a photo on the wall, a sticker) painted out, since HeyGen may animate the wrong face (`cv2.inpaint` for small marks, a temporal median where the person moves); it keeps the room sharp and true. Where drawn things are large or cross the person, an image edit cleans better (`npx hyperframes media-use generate --type image --reference <frame>`, asked to remove the text and keep everything else); check it kept the room sharp.
|
||||
3. **Generate every shot at once,** in the background, and build the graphics while they render (each comes back in about a minute):
|
||||
|
||||
```bash
|
||||
npx hyperframes media-use generate --type video --reference <plate> --motion <the original shot> \
|
||||
--mask <x,y,w,h,a-b> --audio <the original voice over that shot> --aspect <the source's> --seconds <n> \
|
||||
--prompt "<who, where, what they do, the camera>" --out <shot>.mp4
|
||||
```
|
||||
|
||||
`--motion` is the original shot itself, cut at its cut times: HeyGen follows its body and camera motion and takes the face from the plate. It also copies text burned into the guide, so give one `--mask` per drawn box in that shot (the clip's pixels and seconds). Mask the text's own box, never a full-width band: a gesture under a mask is lost. A person who does not speak gets no `--audio`.
|
||||
|
||||
A credit limit the person named: price every shot first with `--plan`, keep the people that fit, pass what is left as `--max-credits`, and ask before cutting a person to stay under it.
|
||||
|
||||
4. **Check each shot as it arrives:** no cut inside it (generate prints `cuts_inside`, and refuses a guide with a cut), none of the original's text, the face box within 5% of the original's (`npx hyperframes measure face`). Regenerate a failing shot once; then trim it to its good part. A shot HeyGen refuses or fails (`generation_failed`) gets one more try, test calls included, in this session or a resumed one; then stop and ask the person whether to keep the original's footage for now or drop the shot.
|
||||
5. **Frame it:** scale and move each shot so its face box sits on the original's at the shot's start and end (`npx hyperframes measure face <shot> --against <video> --from a --to b` prints the `scale`, `x`, `y` to set), tweening between them only where the original's camera moves. The framing jumps at cuts and nowhere else.
|
||||
6. **Text behind the person:** where the original's text passes behind them, cut them out of the shot (`npx hyperframes remove-background <shot> -o <cutout>.webm`) and stack shot, text, cutout.
|
||||
|
||||
Keep the original's voice as the soundtrack and the shots muted. Do not clone voices or make avatars for a re-shoot: the original voice drives the lips. The people inside a photo, a collage or a screenshot are sub-assets, cut with it.
|
||||
@@ -1,63 +0,0 @@
|
||||
# Opening a Reverse
|
||||
|
||||
What the person reads at the start, and when credits run out mid-run. Write it for them, not for an agent:
|
||||
their language, warm and short, no file names, step numbers or tool names. Show any command they type as a code
|
||||
block they can copy.
|
||||
|
||||
## Not signed in yet
|
||||
|
||||
Say why HeyGen matters before asking: signing in is what gets the best remake. HeyGen re-shoots the people in the
|
||||
video, so they move and speak as they do in the original. For example:
|
||||
|
||||
> To rebuild this video at its best, I'll work with HeyGen: it re-shoots the people in it, so they move and speak just
|
||||
> like the original. Sign in once by typing this in the prompt:
|
||||
>
|
||||
> ```
|
||||
> ! npx hyperframes auth login
|
||||
> ```
|
||||
>
|
||||
> Your browser opens the HeyGen sign-in. Once you're in, reply "done".
|
||||
>
|
||||
> Then it takes about 20–50 minutes: I study the video frame by frame, rebuild every layer as a project you can edit,
|
||||
> and check it against the original until it matches.
|
||||
|
||||
## Signed in on Free
|
||||
|
||||
Reverse is part of Pro. Say so once, give the upgrade step, and start nothing until they are on Pro. For example:
|
||||
|
||||
> Reverse is part of HyperFrames Pro: it re-shoots the people in your video with HeyGen. Upgrade by typing this in the
|
||||
> prompt:
|
||||
>
|
||||
> ```
|
||||
> ! npx hyperframes billing subscribe
|
||||
> ```
|
||||
>
|
||||
> It opens the plans page and finishes once you're on Pro. Reply "done" and I'll start.
|
||||
|
||||
## Signed in on Pro
|
||||
|
||||
One line, then start.
|
||||
|
||||
> You're signed in. This takes about 20–50 minutes; I'll show you the project when it matches.
|
||||
|
||||
Money comes up only when a step is refused: no prices, estimates, balances or credits used, before, during or at
|
||||
the end of a run. `auth status` is for the sign-in: never repeat its plan or credits line to the person.
|
||||
|
||||
## Out of credits mid-run
|
||||
|
||||
A refused step prints what it needs and the command that fixes it (top up, or pick a plan). Ask the person to run that
|
||||
command, then run the same step again once they reply. Never trim the plan to fit.
|
||||
|
||||
> This step needs 180 credits and you have 30. Add credits by typing this in the prompt:
|
||||
>
|
||||
> ```
|
||||
> ! npx hyperframes billing top-up --needed 180
|
||||
> ```
|
||||
>
|
||||
> It opens the top-up page and finishes once your balance covers the step. Reply "done" and I'll pick up right
|
||||
> where I stopped.
|
||||
|
||||
## Time
|
||||
|
||||
Say 20–50 minutes for a clip under a minute; never invent a longer figure. A longer film is worked in sections, so
|
||||
say so and give the number of sections.
|
||||
Reference in New Issue
Block a user