fix(capture): content-hash SVG slugs + render SVG→PNG for Gemini captions

Two distinct bugs in the capture pipeline surfaced when AI agents
ran on heygen.com and huly.io captures. Verified visually against
real captured SVGs before fixing.

BUG 1 — SVG slug scramble (assetDownloader.ts):

Label-derived slugging in the downloaded-image flow assigned the wrong
filename to several captured SVGs. Verified by rasterizing the actual
bytes via sharp:
  heygen-logo.svg → contains the GOOGLE wordmark
  hubspot-logo.svg → contains the TRIVAGO logo
  logo.svg → contains the real HeyGen wordmark (correct)
  workday.svg → contains workday (correct)

Root cause: catalog → URL alignment drift in label inference (aria-label
/ nearest-heading / sectionClasses propagating across partner-logo
carousels). Hard to reliably fix in the inference layer.

Fix: switch both SVG-naming flows (inline DOM extraction + downloaded
URLs) to content-addressable filenames. Slug becomes
`svg-<8char-sha1>.svg` or `logo-<hash>.svg` when the DOM marks it as a
logo (the isLogo flag is more reliable than label inference). Filename
becomes a function of the bytes — can never mismatch the content.
Rasters keep human-readable catalog slugs; only SVGs change.

BUG 2 — Gemini hallucinating on SVG content (contentExtractor.ts):

Previous code sent raw SVG path markup as TEXT to gemini-3.1-flash-lite-
preview and asked it to interpret what the paths would render. Real
descriptions in asset-descriptions.md:
  hubspot-logo.svg → "stylized text 'VIVIENNE'" (pure hallucination)
  workday.svg → "text 'wrestling'" (pure hallucination)
  huly-logo.svg → "text 'Kube'" (pure hallucination)
  heygen-logo.svg → "text 'oogo'" (partial-misread of Google)

Fix: rasterize each SVG to a 256×256 PNG via sharp first, then send
the PNG bytes to the Vision API — same call shape we already use for
raster images. Vision models can actually see PNGs reliably.

Adds a polarity check: SVGs with white-glyph fills (e.g. huly-logo.svg
uses fill="#fff") flatten against dark slate; dark-glyph SVGs flatten
against white. Without polarity detection, white-on-white renders as a
blank PNG and Gemini sees nothing. Smoke-verified: huly's "✕ huly"
wordmark + Google wordmark + HeyGen wordmark + trivago logo all render
correctly against contrasting backgrounds.

Prompt updated to add: "If you see a wordmark, READ THE LETTERS
LITERALLY — do not guess a brand from context." Guards against the
remaining failure mode where Gemini sees the pixels correctly but
extrapolates a wrong brand from contextual cues.

Failure modes: sharp's SVG rasterizer can fail on exotic features
(external fonts, foreignObject, missing-primitive filters); those SVGs
fall back to no-caption rather than blocking the capture. The contact
sheet (Chrome-rendered) is still the agent's authoritative source for
SVG content.
This commit is contained in:
ukimsanov
2026-06-11 16:27:36 -07:00
parent 6bdd975ec5
commit 5f290345d1
2 changed files with 73 additions and 16 deletions
+34 -6
View File
@@ -7,9 +7,22 @@
import { writeFileSync, mkdirSync } from "node:fs";
import { join, extname } from "node:path";
import { createHash } from "node:crypto";
import type { DesignTokens, DownloadedAsset } from "./types.js";
import type { CatalogedAsset } from "./assetCataloger.js";
/**
* Content-hash slug for SVGs — `svg-<8-char-sha1>` for icons / `logo-<hash>`
* when DOM evidence says it's a logo. Replaces label-derived slugging which
* mis-assigned brand names to the wrong SVG bodies (e.g. `heygen-logo.svg`
* landing on the Google partner-logo SVG). The hash is a function of the
* bytes, so the filename can never mismatch the content.
*/
function svgContentHashSlug(svgSource: string | Buffer, isLogo: boolean): string {
const hash = createHash("sha1").update(svgSource).digest("hex").slice(0, 8);
return isLogo ? `logo-${hash}` : `svg-${hash}`;
}
export async function downloadAssets(
tokens: DesignTokens,
outputDir: string,
@@ -22,15 +35,20 @@ export async function downloadAssets(
const assets: DownloadedAsset[] = [];
const downloadedUrls = new Set<string>();
// 1. ALL inline SVGs — save as files (logos get priority naming)
// 1. ALL inline SVGs — save as files. Names are content-hash based
// (`svg-<hash>.svg` or `logo-<hash>.svg`) so the filename can never
// drift from the SVG body. The DOM-derived `label` is unreliable —
// it has misassigned `heygen-logo.svg` to the Google partner SVG in
// past captures because aria-label / nearest-heading inference can
// pick up text from the wrong ancestor. Content-hash is invariant.
mkdirSync(join(outputDir, "assets", "svgs"), { recursive: true });
const usedSvgNames = new Set<string>();
for (let i = 0; i < tokens.svgs.length && i < 30; i++) {
const svg = tokens.svgs[i]!;
if (!svg.outerHTML || svg.outerHTML.length < 50) continue;
const label = svg.label?.replace(/[^a-zA-Z0-9-_ ]/g, "").trim();
let slug = label ? slugify(label) : svg.isLogo ? `logo-${i}` : `icon-${i}`;
// Deduplicate — two SVGs with same aria-label get suffixed
const slug = svgContentHashSlug(svg.outerHTML, !!svg.isLogo);
// Hash collisions are negligible for 8-char sha1 prefix over <30 SVGs,
// but suffix-dedupe anyway for safety + idempotent re-runs.
let finalSlug = slug;
let suffix = 2;
while (usedSvgNames.has(finalSlug)) {
@@ -135,8 +153,18 @@ export async function downloadAssets(
if (result.status !== "fulfilled" || !result.value) continue;
const { url, isPoster, parsedUrl, ext, buffer, catalog } = result.value;
try {
// Generate human-readable name from catalog context
const slug = deriveAssetName(parsedUrl, catalog, isPoster, imgIdx, usedNames);
// SVGs use content-hash names because catalog-derived slugs
// mis-assigned brand names to the wrong SVG bodies (the same
// alignment failure that produced `heygen-logo.svg` containing
// the Google wordmark). Rasters keep the catalog-derived
// human-readable slug — they were not affected by the bug.
let slug: string;
if (ext === ".svg") {
const isLogo = !!catalog?.contexts?.some((c) => /logo|brand|wordmark/i.test(c));
slug = svgContentHashSlug(buffer, isLogo);
} else {
slug = deriveAssetName(parsedUrl, catalog, isPoster, imgIdx, usedNames);
}
const name = `${slug}${ext}`;
usedNames.add(slug);
const localPath = `assets/${name}`;
+39 -10
View File
@@ -11,6 +11,7 @@
import type { Page } from "puppeteer-core";
import { existsSync, readdirSync, statSync, readFileSync } from "node:fs";
import { join } from "node:path";
import sharp from "sharp";
import type { CatalogedAsset } from "./assetCataloger.js";
import type { DesignTokens } from "./types.js";
@@ -232,7 +233,12 @@ export async function captionImagesWithGemini(
}
progress("design", `${Object.keys(geminiCaptions).length} images captioned with Gemini`);
// Caption SVGs by sending source code as text (vision API rejects image/svg+xml).
// Caption SVGs by RENDERING each to PNG via sharp first, then sending the
// PNG bytes to the Vision API — same call shape as raster images.
// Previous implementation sent SVG path markup as TEXT, which produced
// pure hallucinations on wordmarks (`hubspot-logo.svg` → "VIVIENNE",
// `huly-logo.svg` → "Kube", `workday.svg` → "wrestling"). Vision models
// can't reliably mental-render path commands; they need actual pixels.
const svgFiles: Array<{ file: string; relPath: string }> = [];
const assetsDir = join(outputDir, "assets");
for (const f of readdirSync(assetsDir)) {
@@ -246,17 +252,39 @@ export async function captionImagesWithGemini(
}
if (svgFiles.length > 0) {
progress("design", `Captioning ${svgFiles.length} SVGs via code analysis...`);
progress("design", `Rasterizing + captioning ${svgFiles.length} SVGs via vision API...`);
const SVG_BATCH = 20;
const MAX_SVG_CHARS = 10_000;
const SVG_RENDER_SIZE = 256; // px — enough resolution for Gemini to read wordmarks, small enough to keep payload sub-MB
for (let i = 0; i < svgFiles.length; i += SVG_BATCH) {
const batch = svgFiles.slice(i, i + SVG_BATCH);
const results = await Promise.allSettled(
batch.map(async ({ relPath }) => {
const filePath = join(assetsDir, relPath);
let svgText = readFileSync(filePath, "utf-8");
if (svgText.length > MAX_SVG_CHARS) {
svgText = svgText.slice(0, MAX_SVG_CHARS) + "\n<!-- truncated -->";
let pngBase64: string;
try {
// Detect SVG fill polarity so we can pick a contrasting flatten
// background. White-glyph SVGs (huly's "✕ huly" wordmark uses
// fill="#fff") render invisible against white; dark-glyph SVGs
// render invisible against black. Choosing the background by
// dominant fill keeps both polarities readable for the vision API.
const svgSource = readFileSync(filePath, "utf-8");
const lightFillHits = (svgSource.match(/fill\s*=\s*["'](#fff(fff)?|white|#f[ef][ef])["']/gi) || []).length;
const darkFillHits = (svgSource.match(/fill\s*=\s*["'](#000(000)?|black|#[0-3]{6}|#[0-3]{3})["']/gi) || []).length;
const bg = lightFillHits > darkFillHits
? { r: 32, g: 32, b: 32 } // dark slate behind light glyphs
: { r: 255, g: 255, b: 255 }; // white behind dark glyphs (default)
// sharp rasterizes SVG → PNG natively.
const pngBuffer = await sharp(filePath)
.resize({ width: SVG_RENDER_SIZE, height: SVG_RENDER_SIZE, fit: "inside", withoutEnlargement: false })
.flatten({ background: bg })
.png()
.toBuffer();
pngBase64 = pngBuffer.toString("base64");
} catch (err) {
// SVG rasterization can fail on exotic features (external fonts,
// foreignObject, filters with missing primitives). Skip caption
// rather than block — agent will fall back to contact-sheet view.
return { file: relPath, caption: "" };
}
const response = await ai.models.generateContent({
model,
@@ -264,12 +292,13 @@ export async function captionImagesWithGemini(
{
role: "user",
parts: [
{ inlineData: { mimeType: "image/png", data: pngBase64 } },
{
text:
"This SVG code is from a website. Describe what it renders in ONE short sentence " +
"for a video storyboard. Focus on: what shape/icon/illustration it is, its colors. " +
"Be factual.\n\n" +
svgText,
"Describe this SVG asset rendered from a website in ONE short sentence for a video storyboard. " +
"Focus on: what shape/icon/illustration/wordmark it is, its colors, any text it contains. " +
"If you see a wordmark, READ THE LETTERS LITERALLY — do not guess a brand from context. " +
"Be factual.",
},
],
},