mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-12 20:47:30 +00:00
fix(capture): content-hash SVG slugs + render SVG→PNG for Gemini captions
Two distinct bugs in the capture pipeline surfaced when AI agents ran on heygen.com and huly.io captures. Verified visually against real captured SVGs before fixing. BUG 1 — SVG slug scramble (assetDownloader.ts): Label-derived slugging in the downloaded-image flow assigned the wrong filename to several captured SVGs. Verified by rasterizing the actual bytes via sharp: heygen-logo.svg → contains the GOOGLE wordmark hubspot-logo.svg → contains the TRIVAGO logo logo.svg → contains the real HeyGen wordmark (correct) workday.svg → contains workday (correct) Root cause: catalog → URL alignment drift in label inference (aria-label / nearest-heading / sectionClasses propagating across partner-logo carousels). Hard to reliably fix in the inference layer. Fix: switch both SVG-naming flows (inline DOM extraction + downloaded URLs) to content-addressable filenames. Slug becomes `svg-<8char-sha1>.svg` or `logo-<hash>.svg` when the DOM marks it as a logo (the isLogo flag is more reliable than label inference). Filename becomes a function of the bytes — can never mismatch the content. Rasters keep human-readable catalog slugs; only SVGs change. BUG 2 — Gemini hallucinating on SVG content (contentExtractor.ts): Previous code sent raw SVG path markup as TEXT to gemini-3.1-flash-lite- preview and asked it to interpret what the paths would render. Real descriptions in asset-descriptions.md: hubspot-logo.svg → "stylized text 'VIVIENNE'" (pure hallucination) workday.svg → "text 'wrestling'" (pure hallucination) huly-logo.svg → "text 'Kube'" (pure hallucination) heygen-logo.svg → "text 'oogo'" (partial-misread of Google) Fix: rasterize each SVG to a 256×256 PNG via sharp first, then send the PNG bytes to the Vision API — same call shape we already use for raster images. Vision models can actually see PNGs reliably. Adds a polarity check: SVGs with white-glyph fills (e.g. huly-logo.svg uses fill="#fff") flatten against dark slate; dark-glyph SVGs flatten against white. Without polarity detection, white-on-white renders as a blank PNG and Gemini sees nothing. Smoke-verified: huly's "✕ huly" wordmark + Google wordmark + HeyGen wordmark + trivago logo all render correctly against contrasting backgrounds. Prompt updated to add: "If you see a wordmark, READ THE LETTERS LITERALLY — do not guess a brand from context." Guards against the remaining failure mode where Gemini sees the pixels correctly but extrapolates a wrong brand from contextual cues. Failure modes: sharp's SVG rasterizer can fail on exotic features (external fonts, foreignObject, missing-primitive filters); those SVGs fall back to no-caption rather than blocking the capture. The contact sheet (Chrome-rendered) is still the agent's authoritative source for SVG content.
This commit is contained in:
@@ -7,9 +7,22 @@
|
||||
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
import { join, extname } from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import type { DesignTokens, DownloadedAsset } from "./types.js";
|
||||
import type { CatalogedAsset } from "./assetCataloger.js";
|
||||
|
||||
/**
|
||||
* Content-hash slug for SVGs — `svg-<8-char-sha1>` for icons / `logo-<hash>`
|
||||
* when DOM evidence says it's a logo. Replaces label-derived slugging which
|
||||
* mis-assigned brand names to the wrong SVG bodies (e.g. `heygen-logo.svg`
|
||||
* landing on the Google partner-logo SVG). The hash is a function of the
|
||||
* bytes, so the filename can never mismatch the content.
|
||||
*/
|
||||
function svgContentHashSlug(svgSource: string | Buffer, isLogo: boolean): string {
|
||||
const hash = createHash("sha1").update(svgSource).digest("hex").slice(0, 8);
|
||||
return isLogo ? `logo-${hash}` : `svg-${hash}`;
|
||||
}
|
||||
|
||||
export async function downloadAssets(
|
||||
tokens: DesignTokens,
|
||||
outputDir: string,
|
||||
@@ -22,15 +35,20 @@ export async function downloadAssets(
|
||||
const assets: DownloadedAsset[] = [];
|
||||
const downloadedUrls = new Set<string>();
|
||||
|
||||
// 1. ALL inline SVGs — save as files (logos get priority naming)
|
||||
// 1. ALL inline SVGs — save as files. Names are content-hash based
|
||||
// (`svg-<hash>.svg` or `logo-<hash>.svg`) so the filename can never
|
||||
// drift from the SVG body. The DOM-derived `label` is unreliable —
|
||||
// it has misassigned `heygen-logo.svg` to the Google partner SVG in
|
||||
// past captures because aria-label / nearest-heading inference can
|
||||
// pick up text from the wrong ancestor. Content-hash is invariant.
|
||||
mkdirSync(join(outputDir, "assets", "svgs"), { recursive: true });
|
||||
const usedSvgNames = new Set<string>();
|
||||
for (let i = 0; i < tokens.svgs.length && i < 30; i++) {
|
||||
const svg = tokens.svgs[i]!;
|
||||
if (!svg.outerHTML || svg.outerHTML.length < 50) continue;
|
||||
const label = svg.label?.replace(/[^a-zA-Z0-9-_ ]/g, "").trim();
|
||||
let slug = label ? slugify(label) : svg.isLogo ? `logo-${i}` : `icon-${i}`;
|
||||
// Deduplicate — two SVGs with same aria-label get suffixed
|
||||
const slug = svgContentHashSlug(svg.outerHTML, !!svg.isLogo);
|
||||
// Hash collisions are negligible for 8-char sha1 prefix over <30 SVGs,
|
||||
// but suffix-dedupe anyway for safety + idempotent re-runs.
|
||||
let finalSlug = slug;
|
||||
let suffix = 2;
|
||||
while (usedSvgNames.has(finalSlug)) {
|
||||
@@ -135,8 +153,18 @@ export async function downloadAssets(
|
||||
if (result.status !== "fulfilled" || !result.value) continue;
|
||||
const { url, isPoster, parsedUrl, ext, buffer, catalog } = result.value;
|
||||
try {
|
||||
// Generate human-readable name from catalog context
|
||||
const slug = deriveAssetName(parsedUrl, catalog, isPoster, imgIdx, usedNames);
|
||||
// SVGs use content-hash names because catalog-derived slugs
|
||||
// mis-assigned brand names to the wrong SVG bodies (the same
|
||||
// alignment failure that produced `heygen-logo.svg` containing
|
||||
// the Google wordmark). Rasters keep the catalog-derived
|
||||
// human-readable slug — they were not affected by the bug.
|
||||
let slug: string;
|
||||
if (ext === ".svg") {
|
||||
const isLogo = !!catalog?.contexts?.some((c) => /logo|brand|wordmark/i.test(c));
|
||||
slug = svgContentHashSlug(buffer, isLogo);
|
||||
} else {
|
||||
slug = deriveAssetName(parsedUrl, catalog, isPoster, imgIdx, usedNames);
|
||||
}
|
||||
const name = `${slug}${ext}`;
|
||||
usedNames.add(slug);
|
||||
const localPath = `assets/${name}`;
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
import type { Page } from "puppeteer-core";
|
||||
import { existsSync, readdirSync, statSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import sharp from "sharp";
|
||||
import type { CatalogedAsset } from "./assetCataloger.js";
|
||||
import type { DesignTokens } from "./types.js";
|
||||
|
||||
@@ -232,7 +233,12 @@ export async function captionImagesWithGemini(
|
||||
}
|
||||
progress("design", `${Object.keys(geminiCaptions).length} images captioned with Gemini`);
|
||||
|
||||
// Caption SVGs by sending source code as text (vision API rejects image/svg+xml).
|
||||
// Caption SVGs by RENDERING each to PNG via sharp first, then sending the
|
||||
// PNG bytes to the Vision API — same call shape as raster images.
|
||||
// Previous implementation sent SVG path markup as TEXT, which produced
|
||||
// pure hallucinations on wordmarks (`hubspot-logo.svg` → "VIVIENNE",
|
||||
// `huly-logo.svg` → "Kube", `workday.svg` → "wrestling"). Vision models
|
||||
// can't reliably mental-render path commands; they need actual pixels.
|
||||
const svgFiles: Array<{ file: string; relPath: string }> = [];
|
||||
const assetsDir = join(outputDir, "assets");
|
||||
for (const f of readdirSync(assetsDir)) {
|
||||
@@ -246,17 +252,39 @@ export async function captionImagesWithGemini(
|
||||
}
|
||||
|
||||
if (svgFiles.length > 0) {
|
||||
progress("design", `Captioning ${svgFiles.length} SVGs via code analysis...`);
|
||||
progress("design", `Rasterizing + captioning ${svgFiles.length} SVGs via vision API...`);
|
||||
const SVG_BATCH = 20;
|
||||
const MAX_SVG_CHARS = 10_000;
|
||||
const SVG_RENDER_SIZE = 256; // px — enough resolution for Gemini to read wordmarks, small enough to keep payload sub-MB
|
||||
for (let i = 0; i < svgFiles.length; i += SVG_BATCH) {
|
||||
const batch = svgFiles.slice(i, i + SVG_BATCH);
|
||||
const results = await Promise.allSettled(
|
||||
batch.map(async ({ relPath }) => {
|
||||
const filePath = join(assetsDir, relPath);
|
||||
let svgText = readFileSync(filePath, "utf-8");
|
||||
if (svgText.length > MAX_SVG_CHARS) {
|
||||
svgText = svgText.slice(0, MAX_SVG_CHARS) + "\n<!-- truncated -->";
|
||||
let pngBase64: string;
|
||||
try {
|
||||
// Detect SVG fill polarity so we can pick a contrasting flatten
|
||||
// background. White-glyph SVGs (huly's "✕ huly" wordmark uses
|
||||
// fill="#fff") render invisible against white; dark-glyph SVGs
|
||||
// render invisible against black. Choosing the background by
|
||||
// dominant fill keeps both polarities readable for the vision API.
|
||||
const svgSource = readFileSync(filePath, "utf-8");
|
||||
const lightFillHits = (svgSource.match(/fill\s*=\s*["'](#fff(fff)?|white|#f[ef][ef])["']/gi) || []).length;
|
||||
const darkFillHits = (svgSource.match(/fill\s*=\s*["'](#000(000)?|black|#[0-3]{6}|#[0-3]{3})["']/gi) || []).length;
|
||||
const bg = lightFillHits > darkFillHits
|
||||
? { r: 32, g: 32, b: 32 } // dark slate behind light glyphs
|
||||
: { r: 255, g: 255, b: 255 }; // white behind dark glyphs (default)
|
||||
// sharp rasterizes SVG → PNG natively.
|
||||
const pngBuffer = await sharp(filePath)
|
||||
.resize({ width: SVG_RENDER_SIZE, height: SVG_RENDER_SIZE, fit: "inside", withoutEnlargement: false })
|
||||
.flatten({ background: bg })
|
||||
.png()
|
||||
.toBuffer();
|
||||
pngBase64 = pngBuffer.toString("base64");
|
||||
} catch (err) {
|
||||
// SVG rasterization can fail on exotic features (external fonts,
|
||||
// foreignObject, filters with missing primitives). Skip caption
|
||||
// rather than block — agent will fall back to contact-sheet view.
|
||||
return { file: relPath, caption: "" };
|
||||
}
|
||||
const response = await ai.models.generateContent({
|
||||
model,
|
||||
@@ -264,12 +292,13 @@ export async function captionImagesWithGemini(
|
||||
{
|
||||
role: "user",
|
||||
parts: [
|
||||
{ inlineData: { mimeType: "image/png", data: pngBase64 } },
|
||||
{
|
||||
text:
|
||||
"This SVG code is from a website. Describe what it renders in ONE short sentence " +
|
||||
"for a video storyboard. Focus on: what shape/icon/illustration it is, its colors. " +
|
||||
"Be factual.\n\n" +
|
||||
svgText,
|
||||
"Describe this SVG asset rendered from a website in ONE short sentence for a video storyboard. " +
|
||||
"Focus on: what shape/icon/illustration/wordmark it is, its colors, any text it contains. " +
|
||||
"If you see a wordmark, READ THE LETTERS LITERALLY — do not guess a brand from context. " +
|
||||
"Be factual.",
|
||||
},
|
||||
],
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user