mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-11 06:30:03 +00:00
feat(media-use): color grading — grade/lut resolve, smart-grade, grade-compare + compare (#2041)
* feat(media-use): color grading — grade/lut resolve, smart-grade, grade-compare CLI Add color grading to media-use as first-class resolve types plus a faithful comparison command. All local, offline, deterministic — no model, no GPU. - resolve -t grade / -t lut: produce a data-color-grading block (or a frozen .cube). Look cascade: core preset (no file) -> bundled .cube library -> parametric buildCube. Emitted .cube is Rec.709 and validated against core's colorLuts constraints (LUT_3D_SIZE <= 64) before it is frozen. - smart grade (grade --for <media>): ffmpeg signalstats -> adjust suggestion (exposure / contrast / white balance), surfaced with the measured evidence on stderr as a starting point; never auto-applied. - hyperframes grade-compare: renders N candidate grades onto a reference frame through the real runtime shader into one labeled comparison PNG, so an agent picks a look without opening Studio. Prepends an "original" baseline cell by default (--no-baseline to omit). Shares the headless-capture pipeline with snapshot via capture/captureCompositionFrame. - media-use SKILL: proactive "media opportunity pass" guidance (grounded signal -> offer, ask once, surface don't mutate). Verified: media-use 116/116, grade-compare 7/7, snapshot 9/9, lint + format clean, full build green, comparison renders end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * test(cli): narrow grade-compare baseline assertion off unknown-typed grading Assert the whole cell via toEqual instead of reaching into .grading.preset / .grading.lut on the unknown-typed field, keeping the test typecheck-clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): agent-authored LUTs via --params + validate --from cube; never-read-.cube guardrail - resolve -t lut / -t grade --params '<json>': build a parametric .cube from explicit params (bypassing the intent cascade), validate, and freeze in one step. --intent becomes the optional description. Lets an agent commit a look it computed itself. - --from <file.cube> now validates the ingested LUT for lut/grade types and rejects an invalid/oversized cube (no partial write) — the escape hatch for a LUT the agent generated with its own code. - SKILL.md: hard rule to never read a .cube body into context (~size^3 lines, zero legible signal) — inspect via grade-compare (see it) or cube-validate (ok/size), read the manifest description for meaning; plus both authoring paths and the parametric-vs-film-stock ceiling note. Verified: media-use 116/116, lint + format clean; smokes — --params builds a valid frozen cube, grade --params returns a lut block, bad JSON and an oversized --from cube are both rejected with no stray file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): grade-compare validates referenced LUTs, warns on no-op cells, caps candidates Bug-bash follow-ups — grade-compare silently accepted bad input: - Validate LUT *content*, not just existence: each referenced .cube is parsed with core's parseCubeLut (now exported from @hyperframes/core) and rejected with a per-cell error ("LUT for \"<label>\" is not a valid .cube: ..."). A file that exists but isn't a valid cube no longer renders a silent no-op cell. - Warn on inactive cells: a grading that normalizes to inactive (e.g. a malformed {lut:12345}) emits a stderr warning naming the cell; the auto-prepended "original" baseline is intentionally inactive and stays silent. stdout remains valid JSON. - Cap candidates at 16 (excluding baseline): over-cap input renders the first N and reports {truncated:true, total:M} on stdout + a stderr note — no silent drop, no unbounded giant sheet. Verified: grade-compare 10/10; non-cube LUT → clear error; {lut:12345} → warning + ok; 20 cells → cells=17 truncated total=20; valid runs unchanged. Lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(cli): general `hyperframes compare` visual-variant primitive Generalize grade-compare's "render N variants → one labeled sheet → the agent looks and picks" loop into a standalone command that works on ANY variation (font, layout, motion, grade, whole compositions) — the tool never needs to know what differs. - `hyperframes compare <path...> [--at <sec>] [--labels a,b,c] [--out] [--cols] [--json]`: renders each agent-authored composition variant through the real runtime (captureCompositionFrame) and stitches one labeled comparison sheet + JSON ({ok, sheet, rendered, variants, truncated?/total?}). 2+ paths required; caps at 16 with loud truncation. It presents, it does not judge — choosing is the caller's job. - Factored the shared "render a labeled set → contact sheet" path so compare, grade-compare, and snapshot all sit on it (no duplication). grade-compare is now the first color-specific specialization of this primitive. - New pathArgs util + contactSheet test; hyperframes-cli SKILL documents compare as the agent's "see your own renders and choose" primitive. Verified: 26/26 across compare + grade-compare + snapshot + contactSheet (no regressions); compare renders 3 variants into one visibly-distinct labeled sheet; 2+-path error path clean; lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): green the skills CI — skip ffmpeg tests when absent, oxfmt markdown The "Test: skills" CI job runs bare `node --test` with no ffmpeg on PATH (by design — skills tests are meant to be node-builtin-only). The grade-analyzer + smart-grade tests shell to ffmpeg and were failing there with ENOENT. Guard them to skip when ffmpeg isn't on PATH; they still run locally / where it is. Also oxfmt README.md + hyperframes/media-use SKILL.md (the whole-repo `oxfmt --check .` Format job caught markdown left unformatted by the rebase conflict resolution). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): skip core-conformance test when tsx is unavailable The "Test: skills" CI job installs no deps, so the normalizeHfColorGrading conformance test (which imports core's TS via `node --import tsx`) failed there. Guard it to skip when tsx can't resolve; runs locally / in the deps-installed Test job. Completes the skills-CI greening (the ffmpeg guards handled the rest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): escape grade-compare src double-quotes (CodeQL XSS) + Windows-safe compare test - grade-compare built `<img src="...">` (double-quoted) with the single-quote escaper, leaving `"` unescaped — a `"` in the frame path could break out (CodeQL: incomplete HTML attribute sanitization). Use escapeXml for src. - compare label test hard-coded POSIX paths that can't match on Windows; assert the derived labels (the subject); path resolution is covered elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * refactor(media-use): generate LUT library from params (drop committed .cube files) The 3 bundled .cube files were 733 lines each (2,199 total) and were themselves buildCube output — pure repo bloat. Replace with compact per-look params in luts/index.json, generated on resolve; add an optional `url` for future scanned LUTs to be CDN-hosted + downloaded on demand (freezeUrl) instead of committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): serve library LUTs from CDN on-demand (static.heygen.ai/luts), params fallback Looks now carry a CDN `url` (hosted at s3://heygen-public/luts → static.heygen.ai/luts/<id>.cube); resolve downloads + validates + freezes on demand, like bgm/image. `params` stays as the deterministic offline fallback (--local-only, or if the download fails), so resolution is never blocked on the network. Provider prefers url, falls back to params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(media-use): address #2041 review — atomic LUT writes, compare telemetry, follow-ups - Atomic .cube writes: library provider (url + params) and the parametric generator now write to a .tmp path, validate, then rename, so a crash can never orphan an invalid .cube at the final path (was validate-after-write). - track("media_use_resolve") now emits provenance.via (url/params-fallback/params). - grade-compare + compare: --timeout flag (was hardcoded 5000) and a media_use_compare event (cells, truncated, total, render_ready_timed_out); openSettledCompositionPage now surfaces the render-ready timeout. - compare staging skips node_modules/.git; --for gets an upfront existence check. - Rec.709 luma comment; HYPERFRAMES_ANALYZE_TIMEOUT_MS override; measured note uses basename; LUT s3 hosting moved from index.json into luts/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Miguel Ángel
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
7face26f04
commit
57b3c78987
@@ -1,12 +1,15 @@
|
||||
// fallow-ignore-file complexity
|
||||
import { spawn } from "node:child_process";
|
||||
import { defineCommand } from "citty";
|
||||
import { existsSync, mkdtempSync, readFileSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { resolve, join, relative, isAbsolute, basename } from "node:path";
|
||||
import {
|
||||
openSettledCompositionPage,
|
||||
runFfmpegOnce,
|
||||
seekCompositionTimeline,
|
||||
} from "../capture/captureCompositionFrame.js";
|
||||
import { resolveProject } from "../utils/project.js";
|
||||
import { normalizeErrorMessage } from "../utils/errorMessage.js";
|
||||
import { resolveCompositionViewportFromHtml } from "../utils/compositionViewport.js";
|
||||
import { serveStaticProjectHtml } from "../utils/staticProjectServer.js";
|
||||
import { c } from "../ui/colors.js";
|
||||
import { findFFmpeg } from "../browser/ffmpeg.js";
|
||||
@@ -67,46 +70,25 @@ async function extractVideoFrameToBuffer(
|
||||
try {
|
||||
const ffmpegPath = findFFmpeg();
|
||||
if (!ffmpegPath) return null;
|
||||
const result = await new Promise<{ code: number | null; stderr: string; timedOut: boolean }>(
|
||||
(resolvePromise) => {
|
||||
// `-ss` before `-i` performs a fast keyframe seek; adequate for snapshot accuracy
|
||||
// (±1 frame) and orders of magnitude faster than the decode-and-scan alternative.
|
||||
const args = ["-hide_banner", "-loglevel", "error"];
|
||||
if (useVp9AlphaDecoder) {
|
||||
args.push("-c:v", "libvpx-vp9");
|
||||
}
|
||||
args.push(
|
||||
"-ss",
|
||||
String(Math.max(0, timeSeconds)),
|
||||
"-i",
|
||||
videoPath,
|
||||
"-frames:v",
|
||||
"1",
|
||||
"-q:v",
|
||||
"2",
|
||||
"-y",
|
||||
outPath,
|
||||
);
|
||||
const ff = spawn(ffmpegPath, args);
|
||||
let stderr = "";
|
||||
let timedOut = false;
|
||||
const timer = setTimeout(() => {
|
||||
timedOut = true;
|
||||
ff.kill("SIGTERM");
|
||||
}, FFMPEG_EXTRACT_TIMEOUT_MS);
|
||||
ff.stderr.on("data", (d: Buffer) => {
|
||||
stderr += d.toString();
|
||||
});
|
||||
ff.on("close", (code) => {
|
||||
clearTimeout(timer);
|
||||
resolvePromise({ code, stderr, timedOut });
|
||||
});
|
||||
ff.on("error", () => {
|
||||
clearTimeout(timer);
|
||||
resolvePromise({ code: null, stderr: "ffmpeg spawn failed", timedOut });
|
||||
});
|
||||
},
|
||||
// `-ss` before `-i` performs a fast keyframe seek; adequate for snapshot accuracy
|
||||
// (±1 frame) and orders of magnitude faster than the decode-and-scan alternative.
|
||||
const args = ["-hide_banner", "-loglevel", "error"];
|
||||
if (useVp9AlphaDecoder) {
|
||||
args.push("-c:v", "libvpx-vp9");
|
||||
}
|
||||
args.push(
|
||||
"-ss",
|
||||
String(Math.max(0, timeSeconds)),
|
||||
"-i",
|
||||
videoPath,
|
||||
"-frames:v",
|
||||
"1",
|
||||
"-q:v",
|
||||
"2",
|
||||
"-y",
|
||||
outPath,
|
||||
);
|
||||
const result = await runFfmpegOnce(ffmpegPath, args, FFMPEG_EXTRACT_TIMEOUT_MS);
|
||||
if (result.code !== 0 || result.timedOut || !existsSync(outPath)) return null;
|
||||
return readFileSync(outPath);
|
||||
} finally {
|
||||
@@ -193,7 +175,6 @@ async function captureSnapshots(
|
||||
},
|
||||
): Promise<string[]> {
|
||||
const { bundleToSingleHtml } = await import("@hyperframes/core/compiler");
|
||||
const { ensureBrowser } = await import("../browser/manager.js");
|
||||
|
||||
const numFrames = opts.frames ?? 5;
|
||||
|
||||
@@ -203,75 +184,12 @@ async function captureSnapshots(
|
||||
const savedPaths: string[] = [];
|
||||
|
||||
try {
|
||||
const browser = await ensureBrowser();
|
||||
const puppeteer = await import("puppeteer-core");
|
||||
const chromeBrowser = await puppeteer.default.launch({
|
||||
headless: true,
|
||||
executablePath: browser.executablePath,
|
||||
args: [
|
||||
"--no-sandbox",
|
||||
"--disable-gpu",
|
||||
"--disable-dev-shm-usage",
|
||||
"--enable-webgl",
|
||||
"--use-gl=angle",
|
||||
"--use-angle=swiftshader",
|
||||
],
|
||||
const { browser: chromeBrowser, page } = await openSettledCompositionPage(html, server.url, {
|
||||
renderReadyTimeoutMs: opts.timeout ?? 5000,
|
||||
renderReadyWarningSuffix: "snapshots may be inaccurate",
|
||||
});
|
||||
|
||||
try {
|
||||
const page = await chromeBrowser.newPage();
|
||||
await page.setViewport(resolveCompositionViewportFromHtml(html));
|
||||
|
||||
await page.goto(server.url, {
|
||||
waitUntil: "domcontentloaded",
|
||||
timeout: 10000,
|
||||
});
|
||||
|
||||
// __renderReady is set after the player is constructed AND the root
|
||||
// timeline is bound — waiting for it guarantees renderSeek will work.
|
||||
const timeoutMs = opts.timeout ?? 5000;
|
||||
const runtimeReady = await page
|
||||
.waitForFunction(() => !!(window as any).__renderReady, { timeout: timeoutMs })
|
||||
.then(() => true)
|
||||
.catch(() => false);
|
||||
|
||||
if (!runtimeReady) {
|
||||
console.warn(
|
||||
`\n ${c.warn("⚠")} Runtime did not become render-ready within ${timeoutMs}ms — snapshots may be inaccurate`,
|
||||
);
|
||||
}
|
||||
|
||||
// Wait for shader transition pre-rendering (HyperShader IndexedDB hydration).
|
||||
// Uses the ready state flag as primary signal, with the loading overlay
|
||||
// display:none as a fallback for older builds.
|
||||
await page
|
||||
.waitForFunction(
|
||||
() => {
|
||||
const win = window as unknown as {
|
||||
__hf?: { shaderTransitions?: Record<string, { ready?: boolean }> };
|
||||
};
|
||||
const shaderTransitions = win.__hf?.shaderTransitions;
|
||||
if (shaderTransitions !== undefined) {
|
||||
return Object.values(shaderTransitions).every((s) => s.ready === true);
|
||||
}
|
||||
const overlay = document.querySelector(
|
||||
"[data-hyper-shader-loading]",
|
||||
) as HTMLElement | null;
|
||||
if (!overlay) return true;
|
||||
return window.getComputedStyle(overlay).display === "none";
|
||||
},
|
||||
{ timeout: 90_000 },
|
||||
)
|
||||
.catch(() => {
|
||||
console.warn(` ${c.warn("⚠")} Shader transitions did not finish pre-rendering`);
|
||||
});
|
||||
|
||||
// Wait for fonts to finish loading before capturing
|
||||
await page.evaluate(() => document.fonts.ready).catch(() => {});
|
||||
|
||||
// Extra settle time for media and animations to initialize
|
||||
await new Promise((r) => setTimeout(r, 1500));
|
||||
|
||||
// Font verification — split into loaded / errored / unused. Only status
|
||||
// "error" is a real failure; a face still "unloaded"/"loading" after
|
||||
// document.fonts.ready + the settle wait was simply never requested by any
|
||||
@@ -406,26 +324,7 @@ async function captureSnapshots(
|
||||
for (let i = 0; i < positions.length; i++) {
|
||||
const time = positions[i]!;
|
||||
|
||||
await page.evaluate((t: number) => {
|
||||
const player = (window as any).__player;
|
||||
if (!player) return;
|
||||
const safe = Math.max(0, Number(t) || 0);
|
||||
if (typeof player.renderSeek === "function") {
|
||||
player.renderSeek(safe);
|
||||
} else if (typeof player.seek === "function") {
|
||||
player.seek(safe);
|
||||
}
|
||||
if ((window as any).gsap?.ticker?.tick) {
|
||||
(window as any).gsap.ticker.tick();
|
||||
}
|
||||
}, time);
|
||||
|
||||
await page.evaluate(`new Promise(function(r) {
|
||||
var settled = false;
|
||||
function finish() { if (settled) return; settled = true; r(); }
|
||||
window.setTimeout(finish, 100);
|
||||
requestAnimationFrame(function() { requestAnimationFrame(finish); });
|
||||
})`);
|
||||
await seekCompositionTimeline(page, time);
|
||||
|
||||
if (cameraExpr) await page.evaluate(cameraExpr);
|
||||
|
||||
|
||||
Reference in New Issue
Block a user