mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-07 10:06:21 +00:00
* feat(media-use): color grading — grade/lut resolve, smart-grade, grade-compare CLI Add color grading to media-use as first-class resolve types plus a faithful comparison command. All local, offline, deterministic — no model, no GPU. - resolve -t grade / -t lut: produce a data-color-grading block (or a frozen .cube). Look cascade: core preset (no file) -> bundled .cube library -> parametric buildCube. Emitted .cube is Rec.709 and validated against core's colorLuts constraints (LUT_3D_SIZE <= 64) before it is frozen. - smart grade (grade --for <media>): ffmpeg signalstats -> adjust suggestion (exposure / contrast / white balance), surfaced with the measured evidence on stderr as a starting point; never auto-applied. - hyperframes grade-compare: renders N candidate grades onto a reference frame through the real runtime shader into one labeled comparison PNG, so an agent picks a look without opening Studio. Prepends an "original" baseline cell by default (--no-baseline to omit). Shares the headless-capture pipeline with snapshot via capture/captureCompositionFrame. - media-use SKILL: proactive "media opportunity pass" guidance (grounded signal -> offer, ask once, surface don't mutate). Verified: media-use 116/116, grade-compare 7/7, snapshot 9/9, lint + format clean, full build green, comparison renders end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * test(cli): narrow grade-compare baseline assertion off unknown-typed grading Assert the whole cell via toEqual instead of reaching into .grading.preset / .grading.lut on the unknown-typed field, keeping the test typecheck-clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): agent-authored LUTs via --params + validate --from cube; never-read-.cube guardrail - resolve -t lut / -t grade --params '<json>': build a parametric .cube from explicit params (bypassing the intent cascade), validate, and freeze in one step. --intent becomes the optional description. Lets an agent commit a look it computed itself. - --from <file.cube> now validates the ingested LUT for lut/grade types and rejects an invalid/oversized cube (no partial write) — the escape hatch for a LUT the agent generated with its own code. - SKILL.md: hard rule to never read a .cube body into context (~size^3 lines, zero legible signal) — inspect via grade-compare (see it) or cube-validate (ok/size), read the manifest description for meaning; plus both authoring paths and the parametric-vs-film-stock ceiling note. Verified: media-use 116/116, lint + format clean; smokes — --params builds a valid frozen cube, grade --params returns a lut block, bad JSON and an oversized --from cube are both rejected with no stray file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): grade-compare validates referenced LUTs, warns on no-op cells, caps candidates Bug-bash follow-ups — grade-compare silently accepted bad input: - Validate LUT *content*, not just existence: each referenced .cube is parsed with core's parseCubeLut (now exported from @hyperframes/core) and rejected with a per-cell error ("LUT for \"<label>\" is not a valid .cube: ..."). A file that exists but isn't a valid cube no longer renders a silent no-op cell. - Warn on inactive cells: a grading that normalizes to inactive (e.g. a malformed {lut:12345}) emits a stderr warning naming the cell; the auto-prepended "original" baseline is intentionally inactive and stays silent. stdout remains valid JSON. - Cap candidates at 16 (excluding baseline): over-cap input renders the first N and reports {truncated:true, total:M} on stdout + a stderr note — no silent drop, no unbounded giant sheet. Verified: grade-compare 10/10; non-cube LUT → clear error; {lut:12345} → warning + ok; 20 cells → cells=17 truncated total=20; valid runs unchanged. Lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(cli): general `hyperframes compare` visual-variant primitive Generalize grade-compare's "render N variants → one labeled sheet → the agent looks and picks" loop into a standalone command that works on ANY variation (font, layout, motion, grade, whole compositions) — the tool never needs to know what differs. - `hyperframes compare <path...> [--at <sec>] [--labels a,b,c] [--out] [--cols] [--json]`: renders each agent-authored composition variant through the real runtime (captureCompositionFrame) and stitches one labeled comparison sheet + JSON ({ok, sheet, rendered, variants, truncated?/total?}). 2+ paths required; caps at 16 with loud truncation. It presents, it does not judge — choosing is the caller's job. - Factored the shared "render a labeled set → contact sheet" path so compare, grade-compare, and snapshot all sit on it (no duplication). grade-compare is now the first color-specific specialization of this primitive. - New pathArgs util + contactSheet test; hyperframes-cli SKILL documents compare as the agent's "see your own renders and choose" primitive. Verified: 26/26 across compare + grade-compare + snapshot + contactSheet (no regressions); compare renders 3 variants into one visibly-distinct labeled sheet; 2+-path error path clean; lint/format clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): green the skills CI — skip ffmpeg tests when absent, oxfmt markdown The "Test: skills" CI job runs bare `node --test` with no ffmpeg on PATH (by design — skills tests are meant to be node-builtin-only). The grade-analyzer + smart-grade tests shell to ffmpeg and were failing there with ENOENT. Guard them to skip when ffmpeg isn't on PATH; they still run locally / where it is. Also oxfmt README.md + hyperframes/media-use SKILL.md (the whole-repo `oxfmt --check .` Format job caught markdown left unformatted by the rebase conflict resolution). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(ci): skip core-conformance test when tsx is unavailable The "Test: skills" CI job installs no deps, so the normalizeHfColorGrading conformance test (which imports core's TS via `node --import tsx`) failed there. Guard it to skip when tsx can't resolve; runs locally / in the deps-installed Test job. Completes the skills-CI greening (the ffmpeg guards handled the rest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(cli): escape grade-compare src double-quotes (CodeQL XSS) + Windows-safe compare test - grade-compare built `<img src="...">` (double-quoted) with the single-quote escaper, leaving `"` unescaped — a `"` in the frame path could break out (CodeQL: incomplete HTML attribute sanitization). Use escapeXml for src. - compare label test hard-coded POSIX paths that can't match on Windows; assert the derived labels (the subject); path resolution is covered elsewhere. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * refactor(media-use): generate LUT library from params (drop committed .cube files) The 3 bundled .cube files were 733 lines each (2,199 total) and were themselves buildCube output — pure repo bloat. Replace with compact per-look params in luts/index.json, generated on resolve; add an optional `url` for future scanned LUTs to be CDN-hosted + downloaded on demand (freezeUrl) instead of committed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * feat(media-use): serve library LUTs from CDN on-demand (static.heygen.ai/luts), params fallback Looks now carry a CDN `url` (hosted at s3://heygen-public/luts → static.heygen.ai/luts/<id>.cube); resolve downloads + validates + freezes on demand, like bgm/image. `params` stays as the deterministic offline fallback (--local-only, or if the download fails), so resolution is never blocked on the network. Provider prefers url, falls back to params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv * fix(media-use): address #2041 review — atomic LUT writes, compare telemetry, follow-ups - Atomic .cube writes: library provider (url + params) and the parametric generator now write to a .tmp path, validate, then rename, so a crash can never orphan an invalid .cube at the final path (was validate-after-write). - track("media_use_resolve") now emits provenance.via (url/params-fallback/params). - grade-compare + compare: --timeout flag (was hardcoded 5000) and a media_use_compare event (cells, truncated, total, render_ready_timed_out); openSettledCompositionPage now surfaces the render-ready timeout. - compare staging skips node_modules/.git; --for gets an upfront existence check. - Rec.709 luma comment; HYPERFRAMES_ANALYZE_TIMEOUT_MS override; measured note uses basename; LUT s3 hosting moved from index.json into luts/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5k87mPZ4d6yiFwcWSb8Vv --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
353 lines
10 KiB
TypeScript
353 lines
10 KiB
TypeScript
/**
|
||
* Generate labeled contact sheet grids from images.
|
||
*
|
||
* Stitches images into a numbered grid with cell labels.
|
||
* Saves 50-65% tokens vs. AI agents reading images individually.
|
||
*/
|
||
|
||
import sharp from "sharp";
|
||
import { readdirSync, readFileSync, writeFileSync, unlinkSync, existsSync } from "node:fs";
|
||
import { join, extname, basename, dirname } from "node:path";
|
||
|
||
interface ContactSheetOptions {
|
||
cols?: number;
|
||
maxImages?: number;
|
||
padding?: number;
|
||
labelMode?: "index" | "filename" | "custom";
|
||
labels?: string[];
|
||
quality?: number;
|
||
/** Target width per cell in pixels (default: 600) */
|
||
cellWidth?: number;
|
||
}
|
||
|
||
/**
|
||
* Create a contact sheet from a list of image paths.
|
||
* Returns the output file path, or null if no images.
|
||
*/
|
||
export async function createContactSheet(
|
||
imagePaths: string[],
|
||
outputPath: string,
|
||
opts: ContactSheetOptions = {},
|
||
): Promise<string | null> {
|
||
const {
|
||
cols = 3,
|
||
maxImages = 16,
|
||
padding = 4,
|
||
labelMode = "index",
|
||
labels,
|
||
quality = 88,
|
||
cellWidth = 600,
|
||
} = opts;
|
||
|
||
const files = imagePaths.slice(0, maxImages);
|
||
if (files.length === 0) return null;
|
||
|
||
// Read first image to determine aspect ratio
|
||
const firstMeta = await sharp(files[0]!).metadata();
|
||
const srcW = firstMeta.width || 1920;
|
||
const srcH = firstMeta.height || 1080;
|
||
|
||
// Scale to target cell width, maintain aspect ratio
|
||
const scale = cellWidth / srcW;
|
||
const cellW = cellWidth;
|
||
const cellH = Math.round(srcH * scale);
|
||
|
||
const rows = Math.ceil(files.length / cols);
|
||
const labelH = 26;
|
||
const totalW = cols * cellW + (cols + 1) * padding;
|
||
const totalH = rows * (cellH + labelH) + (rows + 1) * padding;
|
||
|
||
const overlays: sharp.OverlayOptions[] = [];
|
||
|
||
for (let i = 0; i < files.length; i++) {
|
||
const col = i % cols;
|
||
const row = Math.floor(i / cols);
|
||
const x = padding + col * (cellW + padding);
|
||
const y = padding + row * (cellH + labelH + padding);
|
||
|
||
// Resize image to cell size — contain keeps full image visible (no cropping)
|
||
const resized = await sharp(files[i]!)
|
||
.resize(cellW, cellH, { fit: "contain", background: { r: 26, g: 26, b: 26 } })
|
||
.toBuffer();
|
||
|
||
overlays.push({ input: resized, left: x, top: y + labelH });
|
||
|
||
// Label text
|
||
let labelText = `${i + 1}`;
|
||
if (labelMode === "filename") {
|
||
labelText = `${i + 1}. ${basename(files[i]!).replace(extname(files[i]!), "")}`;
|
||
} else if (labelMode === "custom" && labels?.[i]) {
|
||
labelText = `${i + 1}. ${labels[i]}`;
|
||
}
|
||
|
||
// Truncate label to fit cell
|
||
if (labelText.length > 60) labelText = labelText.slice(0, 57) + "...";
|
||
|
||
const labelSvg = Buffer.from(
|
||
`<svg width="${cellW}" height="${labelH}">` +
|
||
`<rect width="${cellW}" height="${labelH}" fill="#1a1a1a"/>` +
|
||
`<text x="8" y="18" font-family="Arial,Helvetica,sans-serif" font-size="13" font-weight="bold" fill="#ffffff">${escapeXml(labelText)}</text>` +
|
||
`</svg>`,
|
||
);
|
||
|
||
overlays.push({ input: labelSvg, left: x, top: y });
|
||
}
|
||
|
||
const sheet = sharp({
|
||
create: {
|
||
width: totalW,
|
||
height: totalH,
|
||
channels: 3,
|
||
background: { r: 26, g: 26, b: 26 },
|
||
},
|
||
}).composite(overlays);
|
||
|
||
if (extname(outputPath).toLowerCase() === ".png") {
|
||
await sheet.png().toFile(outputPath);
|
||
} else {
|
||
await sheet.jpeg({ quality }).toFile(outputPath);
|
||
}
|
||
|
||
return outputPath;
|
||
}
|
||
|
||
function escapeXml(s: string): string {
|
||
return s
|
||
.replace(/&/g, "&")
|
||
.replace(/</g, "<")
|
||
.replace(/>/g, ">")
|
||
.replace(/"/g, """)
|
||
.replace(/'/g, "'");
|
||
}
|
||
|
||
/**
|
||
* Split imagePaths into pages of `pageSize`, write one contact sheet per page.
|
||
* Output files: basePath → base-1.jpg, base-2.jpg, ...
|
||
* Returns the list of written file paths (empty if no images).
|
||
*/
|
||
async function createContactSheetPages(
|
||
imagePaths: string[],
|
||
outputBasePath: string,
|
||
opts: ContactSheetOptions & { pageSize?: number } = {},
|
||
labelOffset = 0,
|
||
customLabels?: string[],
|
||
): Promise<string[]> {
|
||
if (imagePaths.length === 0) return [];
|
||
const { pageSize = imagePaths.length, ...sheetOpts } = opts;
|
||
const ext = outputBasePath.match(/\.[^.]+$/)?.[0] ?? ".jpg";
|
||
const base = outputBasePath.slice(0, -ext.length);
|
||
|
||
const pages = Math.ceil(imagePaths.length / pageSize);
|
||
const results: string[] = [];
|
||
|
||
for (let p = 0; p < pages; p++) {
|
||
const chunk = imagePaths.slice(p * pageSize, (p + 1) * pageSize);
|
||
const chunkLabels = customLabels?.slice(p * pageSize, (p + 1) * pageSize);
|
||
const outPath = pages === 1 ? outputBasePath : `${base}-${p + 1}${ext}`;
|
||
|
||
const labelsForChunk = chunkLabels
|
||
? { labelMode: "custom" as const, labels: chunkLabels }
|
||
: sheetOpts.labelMode === "filename"
|
||
? { labelMode: "filename" as const }
|
||
: { labelMode: "index" as const };
|
||
|
||
const written = await createContactSheet(chunk, outPath, {
|
||
...sheetOpts,
|
||
...labelsForChunk,
|
||
maxImages: chunk.length,
|
||
});
|
||
if (written) results.push(written);
|
||
void labelOffset; // used by callers that pre-compute labels
|
||
}
|
||
return results;
|
||
}
|
||
|
||
/**
|
||
* Contact sheet for scroll screenshots. Paginated — all screenshots covered.
|
||
* Labels: "1. 0% scroll", "2. 23% scroll", etc.
|
||
* Returns array of written file paths.
|
||
*/
|
||
export async function createScrollContactSheet(
|
||
screenshotsDir: string,
|
||
outputPath: string,
|
||
): Promise<string[]> {
|
||
if (!existsSync(screenshotsDir)) return [];
|
||
|
||
const scrollFiles = readdirSync(screenshotsDir)
|
||
.filter((f) => f.startsWith("scroll-") && f.endsWith(".png"))
|
||
.sort();
|
||
|
||
if (scrollFiles.length === 0) return [];
|
||
|
||
const paths = scrollFiles.map((f) => join(screenshotsDir, f));
|
||
const labels = scrollFiles.map((f) => {
|
||
const m = f.match(/scroll-(\d+)\.png/);
|
||
return m ? `${m[1]}% scroll` : f;
|
||
});
|
||
|
||
// 3 cols max for readability; 9 per page (3×3) so cells stay large enough to read
|
||
return createContactSheetPages(
|
||
paths,
|
||
outputPath,
|
||
{ cols: 3, cellWidth: 600, pageSize: 9 },
|
||
0,
|
||
labels,
|
||
);
|
||
}
|
||
|
||
/**
|
||
* Contact sheet for snapshot frames. All frames covered across pages.
|
||
* Labels: "1. 1.0s", "2. 3.0s", etc.
|
||
* Returns array of written file paths.
|
||
*/
|
||
export async function createSnapshotContactSheet(
|
||
snapshotsDir: string,
|
||
outputPath: string,
|
||
): Promise<string[]> {
|
||
if (!existsSync(snapshotsDir)) return [];
|
||
|
||
const snapshotFiles = readdirSync(snapshotsDir)
|
||
.filter((f) => f.startsWith("frame-") && f.endsWith(".png"))
|
||
.sort();
|
||
|
||
if (snapshotFiles.length === 0) return [];
|
||
|
||
const paths = snapshotFiles.map((f) => join(snapshotsDir, f));
|
||
const labels = snapshotFiles.map((f) => {
|
||
const m = f.match(/at-([\d.]+)s/);
|
||
return m ? `${m[1]}s` : f;
|
||
});
|
||
|
||
// 3 cols, 9 per page (3×3)
|
||
return createContactSheetPages(
|
||
paths,
|
||
outputPath,
|
||
{ cols: 3, cellWidth: 600, pageSize: 9 },
|
||
0,
|
||
labels,
|
||
);
|
||
}
|
||
|
||
/**
|
||
* Contact sheet for captured assets. Paginated — all assets covered.
|
||
* Labels: "1. filename"
|
||
* Returns array of written file paths.
|
||
*/
|
||
export async function createAssetContactSheet(
|
||
assetsDir: string,
|
||
outputPath: string,
|
||
): Promise<string[]> {
|
||
if (!existsSync(assetsDir)) return [];
|
||
|
||
const imageExts = new Set([".png", ".jpg", ".jpeg", ".webp"]);
|
||
const assetFiles = readdirSync(assetsDir)
|
||
.filter((f) => imageExts.has(extname(f).toLowerCase()) && !f.includes("contact-sheet"))
|
||
.sort();
|
||
|
||
if (assetFiles.length === 0) return [];
|
||
|
||
const paths = assetFiles.map((f) => join(assetsDir, f));
|
||
|
||
// 4 cols, 12 per page (4×3) — covers all assets across as many pages as needed
|
||
return createContactSheetPages(paths, outputPath, {
|
||
cols: 4,
|
||
cellWidth: 480,
|
||
labelMode: "filename",
|
||
pageSize: 12,
|
||
});
|
||
}
|
||
|
||
/**
|
||
* Contact sheet for SVGs — renders each SVG to a thumbnail PNG, then grids them.
|
||
* Sharp supports SVG input natively, so no browser needed.
|
||
* Labels: "1. filename"
|
||
*
|
||
* Accepts one or two directories: the primary svgs/ subdir and optionally the
|
||
* parent assets/ root (for external SVGs downloaded as <img src="*.svg">).
|
||
* Files are deduplicated by basename so duplicates across dirs are collapsed.
|
||
*/
|
||
// fallow-ignore-next-line complexity
|
||
export async function createSvgContactSheet(
|
||
svgsDir: string,
|
||
outputPath: string,
|
||
assetsRootDir?: string,
|
||
): Promise<string[]> {
|
||
const dirsToScan = [svgsDir, assetsRootDir].filter(
|
||
(d): d is string => d !== undefined && existsSync(d),
|
||
);
|
||
if (dirsToScan.length === 0) return [];
|
||
|
||
const seen = new Set<string>();
|
||
const svgPaths: string[] = [];
|
||
|
||
for (const dir of dirsToScan) {
|
||
for (const f of readdirSync(dir)
|
||
.filter((f) => f.endsWith(".svg"))
|
||
.sort()) {
|
||
if (!seen.has(f)) {
|
||
seen.add(f);
|
||
svgPaths.push(join(dir, f));
|
||
}
|
||
}
|
||
}
|
||
|
||
if (svgPaths.length === 0) return [];
|
||
|
||
const svgFileNames = svgPaths.map((p) => p.split("/").pop()!);
|
||
|
||
// Render ALL SVGs to PNG thumbnails first, then paginate the sheets
|
||
const thumbSize = 200;
|
||
const tmpDir = dirname(outputPath);
|
||
const tmpPaths: string[] = [];
|
||
const labels: string[] = [];
|
||
|
||
for (let i = 0; i < svgPaths.length; i++) {
|
||
const svgPath = svgPaths[i]!;
|
||
const tmpPath = join(tmpDir, `.thumb-${i}.png`);
|
||
try {
|
||
const svgBuf = readFileSync(svgPath);
|
||
const thumb = await sharp(svgBuf)
|
||
.resize(thumbSize, thumbSize, {
|
||
fit: "contain",
|
||
background: { r: 245, g: 245, b: 245, alpha: 1 },
|
||
})
|
||
.flatten({ background: { r: 245, g: 245, b: 245 } })
|
||
.png()
|
||
.toBuffer();
|
||
writeFileSync(tmpPath, thumb);
|
||
tmpPaths.push(tmpPath);
|
||
labels.push(svgFileNames[i]!.replace(".svg", ""));
|
||
} catch {
|
||
// SVG might be malformed — skip
|
||
}
|
||
}
|
||
|
||
if (tmpPaths.length === 0) return [];
|
||
|
||
// 5 cols, 15 per page (5×3) — all SVGs covered across pages
|
||
let results: string[] = [];
|
||
try {
|
||
results = await createContactSheetPages(
|
||
tmpPaths,
|
||
outputPath,
|
||
{
|
||
cols: 5,
|
||
cellWidth: thumbSize,
|
||
pageSize: 15,
|
||
},
|
||
0,
|
||
labels,
|
||
);
|
||
} finally {
|
||
for (const tmp of tmpPaths) {
|
||
try {
|
||
unlinkSync(tmp);
|
||
} catch {
|
||
/* best effort */
|
||
}
|
||
}
|
||
}
|
||
|
||
return results;
|
||
}
|