mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-07 10:06:21 +00:00
fix(capture): let vision captioning authenticate the way a server can (#3561)
* fix(capture): let vision captioning authenticate the way a server can Three defects in one phase, all of which end with a capture that reports "Captioned N/N images" and then "0 images captioned with Gemini" — a successful-looking run that hands the agent nothing to see by. 1. Credential. The captioner only accepted an API key. A server deployment holds a service account, not a key, and a rejected key is indistinguishable from an unset one here: every request returns empty text and no error. Vertex is now a first-class provider, ranked above the bare key and below an explicit OPENROUTER_API_KEY opt-in, configured by HYPERFRAMES_VERTEX_SERVICE_ACCOUNT + HYPERFRAMES_VERTEX_PROJECT_ID (region via HYPERFRAMES_VERTEX_LOCATION). It carries its own model default because the Gemini API's flash-lite preview id is not resolvable on Vertex. 2. Empty captions. Thinking tokens are drawn from maxOutputTokens, so a model left free to think can spend the whole budget and return no text — a successful request with no caption. thinkingBudget is pinned to 0; a one-line factual caption needs no reasoning. 3. Native abort. Rasterizing a batch of SVGs concurrently drove up to SVG_BATCH simultaneous librsvg renders through libvips and corrupted the heap: `free(): unaligned chunk detected in tcache 2` (SIGABRT) during this phase, twice in fourteen days, losing the whole capture each time. A native abort cannot be caught, so the concurrency is removed rather than handled — rasterization is serialized and libvips' worker pool is bounded, while the vision requests, which are the slow leg, stay parallel. Throughput barely moves: 225 captions across three real captures, 0 failures, 13-25s each. * test(capture): pin the rasterization loop to one render at a time The serialization fix shipped without a regression test on the grounds that native heap corruption is not unit-testable. The corruption is not, but the property that prevents it is: `sharp` is mocked to record how many renders are in flight, and a six-SVG batch must never reach two. A deliberately slow caption stub makes overlapping renders the faster path, so a future refactor that "optimises" the loop back to `Promise.all` fails here instead of aborting in production. Also covered: `sharp.concurrency(1)` is applied — serializing the loop while leaving libvips' pool at the host core count still fans one render across every core — and an unrasterizable SVG is skipped without breaking serialization for its siblings. Verified as a real guard: reverting only contentExtractor.ts to origin/main fails 7 of the 22 cases in this file. * fix(capture): tell the truth in the asset-descriptions header when Vertex captioned The provider gate in `contentExtractor` accepts Vertex when a project and a service account are both set -- which is the configuration a server deployment actually has. The header written next to the captions still tested only for an API key, so a capture whose captions Vertex had just generated was labelled "GEMINI_API_KEY not set -- descriptions below are catalog-derived". That header is not cosmetic: it travels into the context the template editor reads, telling it to distrust captions that are real. Mirror the same two variables here, and name every provider in the fallback text instead of only the API key. * fix(capture): hand libvips' worker pool back after the renders `sharp.concurrency(1)` is process-global and was set once, for the whole life of the process. The bound is right for the rasterize loop -- a native abort in libvips cannot be caught, so the renders must not overlap -- but its scope was every later sharp caller in the process, none of which asked for captioning, all of them pinned to one thread from then on. Now the host's value is read first and restored in a `finally` around the rasterize loop, so a skipped SVG cannot cost the process its threads either. The vision requests below are network work and gain nothing from a pinned pool. The mock had to grow the getter half of sharp's API -- `concurrency()` with no argument reports the current value -- since save-and-restore is untestable without it. Verified as a real guard: dropping only the restore fails both new cases. Raised by Rames Jusso in review of #3561 and concurred by Magi.
This commit is contained in:
WaterrrForever
@@ -1,4 +1,4 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
@@ -8,13 +8,64 @@ import {
|
||||
type VisionCaptionOutcome,
|
||||
} from "./contentExtractor.js";
|
||||
|
||||
const { generateContentMock } = vi.hoisted(() => ({
|
||||
const { generateContentMock, clientOptions, sharpState } = vi.hoisted(() => ({
|
||||
generateContentMock: vi.fn(),
|
||||
// How the SDK client was constructed is the whole difference between the Vertex and API-key
|
||||
// paths, so the Vertex cases assert on it rather than on the request.
|
||||
clientOptions: [] as Record<string, unknown>[],
|
||||
// A native abort inside libvips cannot be caught, so the only defence is never running two
|
||||
// renders at once. That is a property of the loop, and this records it.
|
||||
sharpState: {
|
||||
inFlight: 0,
|
||||
maxInFlight: 0,
|
||||
concurrencyCalls: [] as number[],
|
||||
// Stands in for the host core count sharp reports before anything touches it, so a failure
|
||||
// to restore shows up as a wrong value rather than as a coincidental match with 1.
|
||||
HOST_CONCURRENCY: 8,
|
||||
concurrency: 8,
|
||||
renders: [] as string[],
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock("sharp", () => {
|
||||
const pipeline = (filePath: string) => {
|
||||
const chain = {
|
||||
resize: () => chain,
|
||||
flatten: () => chain,
|
||||
png: () => chain,
|
||||
toBuffer: async () => {
|
||||
sharpState.inFlight += 1;
|
||||
sharpState.maxInFlight = Math.max(sharpState.maxInFlight, sharpState.inFlight);
|
||||
sharpState.renders.push(filePath);
|
||||
// Yield, so an overlapping caller would be observed rather than serialized by luck.
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
sharpState.inFlight -= 1;
|
||||
return Buffer.from([0x89, 0x50, 0x4e, 0x47]);
|
||||
},
|
||||
};
|
||||
return chain;
|
||||
};
|
||||
const sharp = Object.assign(pipeline, {
|
||||
// Real `sharp.concurrency()` is a getter when called with no argument and a process-global
|
||||
// setter otherwise. The mock has to be both, or code that saves and restores the host value
|
||||
// cannot be tested at all.
|
||||
concurrency: (value?: number) => {
|
||||
if (typeof value === "number") {
|
||||
sharpState.concurrencyCalls.push(value);
|
||||
sharpState.concurrency = value;
|
||||
}
|
||||
return sharpState.concurrency;
|
||||
},
|
||||
});
|
||||
return { default: sharp };
|
||||
});
|
||||
|
||||
vi.mock("@google/genai", () => ({
|
||||
GoogleGenAI: class {
|
||||
models = { generateContent: generateContentMock };
|
||||
constructor(options: Record<string, unknown>) {
|
||||
clientOptions.push(options);
|
||||
}
|
||||
},
|
||||
}));
|
||||
|
||||
@@ -414,3 +465,302 @@ describe("captionImagesWithGemini — Gemini provider", () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("captionImagesWithGemini — Vertex AI provider", () => {
|
||||
const dirs: string[] = [];
|
||||
const SERVICE_ACCOUNT = JSON.stringify({
|
||||
type: "service_account",
|
||||
project_id: "prefab-kit-000000",
|
||||
private_key: "-----BEGIN PRIVATE KEY-----super-secret-material-----END PRIVATE KEY-----",
|
||||
client_email: "capture@prefab-kit-000000.iam.gserviceaccount.com",
|
||||
});
|
||||
|
||||
// Earlier describe blocks construct the SDK client too, so the record is cleared going in
|
||||
// rather than only on the way out.
|
||||
beforeEach(() => {
|
||||
clientOptions.length = 0;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
generateContentMock.mockReset();
|
||||
clientOptions.length = 0;
|
||||
vi.unstubAllEnvs();
|
||||
for (const d of dirs) rmSync(d, { recursive: true, force: true });
|
||||
dirs.length = 0;
|
||||
});
|
||||
|
||||
function vertexEnv(): void {
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "");
|
||||
vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000");
|
||||
vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", SERVICE_ACCOUNT);
|
||||
}
|
||||
|
||||
it("prefers a service account over a bare API key, and authenticates against the project", async () => {
|
||||
// A deployment can hold a Gemini key that is present but rejected. Every request then
|
||||
// returns empty text, so the capture reports "Captioned N/N" followed by "0 images
|
||||
// captioned" and no error — which is exactly what production was doing.
|
||||
const dir = makeProjectWithImages();
|
||||
dirs.push(dir);
|
||||
vertexEnv();
|
||||
vi.stubEnv("GEMINI_API_KEY", "a-key-that-the-server-would-be-rejected-for");
|
||||
generateContentMock.mockResolvedValue({ text: "A dark blue product screenshot." });
|
||||
|
||||
const stages: string[] = [];
|
||||
const captions = await captionImagesWithGemini(
|
||||
dir,
|
||||
(stage, detail) => {
|
||||
stages.push(detail ?? stage);
|
||||
},
|
||||
[],
|
||||
);
|
||||
|
||||
expect(captions).toEqual({ "hero.png": "A dark blue product screenshot." });
|
||||
expect(stages.join(" ")).toContain("Vertex AI");
|
||||
expect(clientOptions).toHaveLength(1);
|
||||
expect(clientOptions[0]).toMatchObject({
|
||||
vertexai: true,
|
||||
project: "prefab-kit-000000",
|
||||
location: "us-central1",
|
||||
googleAuthOptions: { credentials: JSON.parse(SERVICE_ACCOUNT) },
|
||||
});
|
||||
expect(clientOptions[0]).not.toHaveProperty("apiKey");
|
||||
});
|
||||
|
||||
it("honours an explicit region", async () => {
|
||||
const dir = makeProjectWithImages();
|
||||
dirs.push(dir);
|
||||
vertexEnv();
|
||||
vi.stubEnv("HYPERFRAMES_VERTEX_LOCATION", "europe-west4");
|
||||
generateContentMock.mockResolvedValue({ text: "A caption." });
|
||||
|
||||
await captionImagesWithGemini(dir, () => {}, []);
|
||||
|
||||
expect(clientOptions[0]).toMatchObject({ location: "europe-west4" });
|
||||
});
|
||||
|
||||
it("spends no output budget on thinking", async () => {
|
||||
// Thinking tokens come out of maxOutputTokens, so a model left free to think can consume
|
||||
// the whole budget and return empty text: a successful request that produces no caption.
|
||||
const dir = makeProjectWithImages();
|
||||
dirs.push(dir);
|
||||
vertexEnv();
|
||||
// Capture the request inside the mock, where the argument is well-typed — avoids
|
||||
// indexing `mock.calls` (and the repo's ban on `as` assertions).
|
||||
let model: unknown;
|
||||
let thinkingConfig: unknown;
|
||||
generateContentMock.mockImplementation(
|
||||
async (request: { model: string; config?: { thinkingConfig?: unknown } }) => {
|
||||
model = request.model;
|
||||
thinkingConfig = request.config?.thinkingConfig;
|
||||
return { text: "A caption." };
|
||||
},
|
||||
);
|
||||
|
||||
await captionImagesWithGemini(dir, () => {}, []);
|
||||
|
||||
expect(generateContentMock).toHaveBeenCalledTimes(1);
|
||||
expect(thinkingConfig).toEqual({ thinkingBudget: 0 });
|
||||
// The API's flash-lite preview id is not resolvable on Vertex, so the default differs.
|
||||
expect(model).toBe("gemini-2.5-flash");
|
||||
});
|
||||
|
||||
it("skips captioning when the service account is unparseable, without echoing it", async () => {
|
||||
const dir = makeProjectWithImages();
|
||||
dirs.push(dir);
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "");
|
||||
vi.stubEnv("GEMINI_API_KEY", "");
|
||||
vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000");
|
||||
vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", "{not-json super-secret-material");
|
||||
|
||||
const warnings: string[] = [];
|
||||
let outcome: VisionCaptionOutcome | undefined;
|
||||
const captions = await captionImagesWithGemini(dir, () => {}, warnings, {
|
||||
onOutcome: (value) => {
|
||||
outcome = value;
|
||||
},
|
||||
});
|
||||
|
||||
expect(captions).toEqual({});
|
||||
expect(generateContentMock).not.toHaveBeenCalled();
|
||||
expect(warnings.join(" ")).toContain("not valid JSON");
|
||||
expect(warnings.join(" ")).not.toContain("super-secret-material");
|
||||
if (!outcome) throw new Error("Expected vision caption outcome");
|
||||
expect(outcome.internalError).toBe(true);
|
||||
expect(resolveVisionPhaseCompletion(outcome, 10_000)).toEqual({
|
||||
status: "degraded",
|
||||
reason: "internal-error",
|
||||
});
|
||||
});
|
||||
|
||||
it("stays on OpenRouter when the user has opted into it explicitly", async () => {
|
||||
const dir = makeProjectWithImages();
|
||||
dirs.push(dir);
|
||||
vertexEnv();
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "or-key");
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ choices: [{ message: { content: "A caption." } }] }),
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
await captionImagesWithGemini(dir, () => {}, []);
|
||||
|
||||
expect(fetchMock).toHaveBeenCalled();
|
||||
expect(clientOptions).toHaveLength(0);
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
it("needs both halves of the credential before it will use Vertex", async () => {
|
||||
const dir = makeProjectWithImages();
|
||||
dirs.push(dir);
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "");
|
||||
vi.stubEnv("GEMINI_API_KEY", "");
|
||||
vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000");
|
||||
vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", "");
|
||||
|
||||
const captions = await captionImagesWithGemini(dir, () => {}, []);
|
||||
|
||||
expect(captions).toEqual({});
|
||||
expect(clientOptions).toHaveLength(0);
|
||||
expect(generateContentMock).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe("captionImagesWithGemini — SVG rasterization", () => {
|
||||
const dirs: string[] = [];
|
||||
|
||||
beforeEach(() => {
|
||||
sharpState.inFlight = 0;
|
||||
sharpState.maxInFlight = 0;
|
||||
sharpState.concurrencyCalls.length = 0;
|
||||
sharpState.concurrency = sharpState.HOST_CONCURRENCY;
|
||||
sharpState.renders.length = 0;
|
||||
clientOptions.length = 0;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
generateContentMock.mockReset();
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
for (const d of dirs) rmSync(d, { recursive: true, force: true });
|
||||
dirs.length = 0;
|
||||
});
|
||||
|
||||
function makeProjectWithSvgs(count: number): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-svg-"));
|
||||
mkdirSync(join(dir, "assets", "svgs"), { recursive: true });
|
||||
for (let i = 0; i < count; i++) {
|
||||
writeFileSync(
|
||||
join(dir, "assets", "svgs", `logo-${i}.svg`),
|
||||
'<svg xmlns="http://www.w3.org/2000/svg"><path fill="#000" d="M0 0h8v8H0z"/></svg>',
|
||||
);
|
||||
}
|
||||
return dir;
|
||||
}
|
||||
|
||||
it("never runs two libvips renders at once", async () => {
|
||||
// Production aborted here with `free(): unaligned chunk detected in tcache 2` (SIGABRT),
|
||||
// twice in fourteen days, losing the whole capture. A native abort cannot be caught by the
|
||||
// surrounding try/catch, so the concurrency has to be absent rather than handled — which
|
||||
// makes "one at a time" the assertion, not "errors are reported".
|
||||
const dir = makeProjectWithSvgs(6);
|
||||
dirs.push(dir);
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "or-key");
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => {
|
||||
// Slower than a render, so overlapping renders would be the easy way to go faster —
|
||||
// the point is that the loop does not take it.
|
||||
await new Promise((resolve) => setTimeout(resolve, 15));
|
||||
return new Response(
|
||||
JSON.stringify({ choices: [{ message: { content: "A dark glyph." } }] }),
|
||||
{
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
},
|
||||
);
|
||||
}),
|
||||
);
|
||||
|
||||
const captions = await captionImagesWithGemini(dir, () => {}, []);
|
||||
|
||||
expect(sharpState.renders).toHaveLength(6);
|
||||
expect(sharpState.maxInFlight).toBe(1);
|
||||
expect(Object.keys(captions)).toHaveLength(6);
|
||||
});
|
||||
|
||||
it("bounds libvips' own worker pool for the renders, then hands it back", async () => {
|
||||
// Left at its default the pool sizes itself to the host's core count, so serializing the
|
||||
// loop alone still leaves one render fanning out across every core. But `sharp.concurrency`
|
||||
// is process-global: leaving it at 1 pins every later sharp caller in this process, none of
|
||||
// which asked for captioning, so the bound has to be given back when the renders are done.
|
||||
const dir = makeProjectWithSvgs(1);
|
||||
dirs.push(dir);
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "or-key");
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(
|
||||
async () =>
|
||||
new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
await captionImagesWithGemini(dir, () => {}, []);
|
||||
|
||||
expect(sharpState.concurrencyCalls).toEqual([1, sharpState.HOST_CONCURRENCY]);
|
||||
expect(sharpState.concurrency).toBe(sharpState.HOST_CONCURRENCY);
|
||||
});
|
||||
|
||||
it("hands the worker pool back even when a rasterize throws", async () => {
|
||||
// The restore is in a `finally`, because a skipped SVG must not cost the rest of the process
|
||||
// its threads.
|
||||
const dir = makeProjectWithSvgs(2);
|
||||
dirs.push(dir);
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "or-key");
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(
|
||||
async () =>
|
||||
new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
),
|
||||
);
|
||||
// Unreadable, which is how an exotic SVG behaves through sharp.
|
||||
rmSync(join(dir, "assets", "svgs", "logo-0.svg"));
|
||||
mkdirSync(join(dir, "assets", "svgs", "logo-0.svg"));
|
||||
|
||||
await captionImagesWithGemini(dir, () => {}, []);
|
||||
|
||||
expect(sharpState.concurrency).toBe(sharpState.HOST_CONCURRENCY);
|
||||
});
|
||||
|
||||
it("keeps captioning the rest when one SVG cannot be rasterized", async () => {
|
||||
const dir = makeProjectWithSvgs(3);
|
||||
dirs.push(dir);
|
||||
vi.stubEnv("OPENROUTER_API_KEY", "or-key");
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(
|
||||
async () =>
|
||||
new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
),
|
||||
);
|
||||
// One file is unreadable, which is how an exotic SVG behaves through sharp.
|
||||
rmSync(join(dir, "assets", "svgs", "logo-1.svg"));
|
||||
mkdirSync(join(dir, "assets", "svgs", "logo-1.svg"));
|
||||
|
||||
const warnings: string[] = [];
|
||||
const captions = await captionImagesWithGemini(dir, () => {}, warnings);
|
||||
|
||||
expect(Object.keys(captions).sort()).toEqual(["svgs/logo-0.svg", "svgs/logo-2.svg"]);
|
||||
expect(sharpState.maxInFlight).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -266,22 +266,38 @@ export async function captionImagesWithGemini(
|
||||
}
|
||||
const openRouterKey = process.env.OPENROUTER_API_KEY;
|
||||
const geminiKey = process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY;
|
||||
if (!openRouterKey && !geminiKey) {
|
||||
// Vertex authenticates with a service account and a project instead of an API key. Server
|
||||
// deployments have those; they often do not have a working Gemini API key, and a rejected
|
||||
// key is indistinguishable from an unset one in the output — every request simply returns
|
||||
// nothing and the capture reports "0 images captioned".
|
||||
const vertexProject = process.env.HYPERFRAMES_VERTEX_PROJECT_ID;
|
||||
const vertexServiceAccount = process.env.HYPERFRAMES_VERTEX_SERVICE_ACCOUNT;
|
||||
const useVertex = Boolean(vertexProject && vertexServiceAccount);
|
||||
if (!openRouterKey && !useVertex && !geminiKey) {
|
||||
reportOutcome();
|
||||
return geminiCaptions;
|
||||
}
|
||||
|
||||
// OpenRouter takes priority when both keys are set — it's the explicit opt-in
|
||||
// for users without Google access. Both providers satisfy the same
|
||||
// single-image → one-line-caption contract (`captionOne`), so the batching and
|
||||
// SVG-rasterization loops below stay provider-agnostic.
|
||||
const useOpenRouter = Boolean(openRouterKey);
|
||||
const providerName = useOpenRouter ? "OpenRouter" : "Gemini";
|
||||
// Default mirrors the Gemini path's tier (3.x flash-lite). Override per
|
||||
// provider via HYPERFRAMES_OPENROUTER_MODEL / HYPERFRAMES_GEMINI_MODEL.
|
||||
const model = useOpenRouter
|
||||
? process.env.HYPERFRAMES_OPENROUTER_MODEL || "google/gemini-3.1-flash-lite"
|
||||
: process.env.HYPERFRAMES_GEMINI_MODEL || "gemini-3.1-flash-lite-preview";
|
||||
// OpenRouter takes priority — it's the explicit opt-in for users without Google access.
|
||||
// Vertex outranks the bare API key because it is the credential a deployment actually
|
||||
// holds. All three satisfy the same single-image → one-line-caption contract
|
||||
// (`captionOne`), so the batching and SVG-rasterization loops stay provider-agnostic.
|
||||
const provider: "openrouter" | "vertex" | "gemini" = openRouterKey
|
||||
? "openrouter"
|
||||
: useVertex
|
||||
? "vertex"
|
||||
: "gemini";
|
||||
const providerName = { openrouter: "OpenRouter", vertex: "Vertex AI", gemini: "Gemini" }[
|
||||
provider
|
||||
];
|
||||
// Override per provider via HYPERFRAMES_OPENROUTER_MODEL / HYPERFRAMES_VERTEX_MODEL /
|
||||
// HYPERFRAMES_GEMINI_MODEL. Vertex publishes a different model set than the Gemini API —
|
||||
// the API's flash-lite preview id is not resolvable there — so it carries its own default.
|
||||
const model = {
|
||||
openrouter: process.env.HYPERFRAMES_OPENROUTER_MODEL || "google/gemini-3.1-flash-lite",
|
||||
vertex: process.env.HYPERFRAMES_VERTEX_MODEL || "gemini-2.5-flash",
|
||||
gemini: process.env.HYPERFRAMES_GEMINI_MODEL || "gemini-3.1-flash-lite-preview",
|
||||
}[provider];
|
||||
const requestTimeoutMs = resolveVisionRequestTimeoutMs();
|
||||
|
||||
progress("design", `Captioning images with ${providerName} vision...`);
|
||||
@@ -297,7 +313,7 @@ export async function captionImagesWithGemini(
|
||||
}) => Promise<string>;
|
||||
|
||||
let captionOne: CaptionOne;
|
||||
if (openRouterKey) {
|
||||
if (provider === "openrouter") {
|
||||
captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => {
|
||||
return runBoundedVisionRequest(async (signal) => {
|
||||
const res = await fetch("https://openrouter.ai/api/v1/chat/completions", {
|
||||
@@ -335,10 +351,33 @@ export async function captionImagesWithGemini(
|
||||
}, timeoutMs);
|
||||
};
|
||||
} else {
|
||||
// Unreachable when geminiKey is unset (guarded above); re-narrow for TS.
|
||||
if (!geminiKey) return geminiCaptions;
|
||||
const { GoogleGenAI } = await import("@google/genai");
|
||||
const ai = new GoogleGenAI({ apiKey: geminiKey });
|
||||
let ai: InstanceType<typeof GoogleGenAI>;
|
||||
if (provider === "vertex") {
|
||||
// Re-narrow for TS; `useVertex` already guaranteed both are set.
|
||||
if (!vertexProject || !vertexServiceAccount) return geminiCaptions;
|
||||
let credentials: Record<string, unknown>;
|
||||
try {
|
||||
credentials = JSON.parse(vertexServiceAccount) as Record<string, unknown>;
|
||||
} catch {
|
||||
warnings.push(
|
||||
"HYPERFRAMES_VERTEX_SERVICE_ACCOUNT is not valid JSON; skipped vision captioning.",
|
||||
);
|
||||
internalError = true;
|
||||
reportOutcome();
|
||||
return geminiCaptions;
|
||||
}
|
||||
ai = new GoogleGenAI({
|
||||
vertexai: true,
|
||||
project: vertexProject,
|
||||
location: process.env.HYPERFRAMES_VERTEX_LOCATION || "us-central1",
|
||||
googleAuthOptions: { credentials },
|
||||
});
|
||||
} else {
|
||||
// Unreachable when geminiKey is unset (guarded above); re-narrow for TS.
|
||||
if (!geminiKey) return geminiCaptions;
|
||||
ai = new GoogleGenAI({ apiKey: geminiKey });
|
||||
}
|
||||
captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => {
|
||||
const response = await runBoundedVisionRequest(
|
||||
(signal) =>
|
||||
@@ -352,6 +391,12 @@ export async function captionImagesWithGemini(
|
||||
],
|
||||
config: {
|
||||
maxOutputTokens: maxTokens,
|
||||
// A one-line factual caption needs no reasoning, and leaving thinking on is
|
||||
// not merely wasteful: thinking tokens are drawn from maxOutputTokens, so the
|
||||
// model can spend the whole budget and return empty text. That surfaces as a
|
||||
// successful request with no caption — the capture then logs "Captioned N/N
|
||||
// images" followed by "0 images captioned", which is what production shows.
|
||||
thinkingConfig: { thinkingBudget: 0 },
|
||||
abortSignal: signal,
|
||||
httpOptions: { timeout: timeoutMs },
|
||||
},
|
||||
@@ -464,6 +509,13 @@ export async function captionImagesWithGemini(
|
||||
reportOutcome();
|
||||
return geminiCaptions;
|
||||
}
|
||||
// libvips' worker pool sizes itself to the host's core count, and several concurrent SVG
|
||||
// renders then multiply that — the combination corrupted the heap in production (see the
|
||||
// serialized rasterize loop below). `sharp.concurrency` is process-global and outlives this
|
||||
// function, so remember the host's value and bound the pool only around the renders that
|
||||
// need it: captioning must not leave every later sharp caller in this process — none of
|
||||
// which asked for captioning — pinned to a single thread for the rest of its life.
|
||||
const hostConcurrency = sharp.concurrency();
|
||||
progress("design", `Rasterizing + captioning ${svgFiles.length} SVGs via vision API...`);
|
||||
const SVG_BATCH = 20;
|
||||
const SVG_RENDER_SIZE = 256; // px — enough resolution for Gemini to read wordmarks, small enough to keep payload sub-MB
|
||||
@@ -476,10 +528,19 @@ export async function captionImagesWithGemini(
|
||||
}
|
||||
const timeoutMs = Math.max(1, Math.min(requestTimeoutMs, remainingMs));
|
||||
const batch = svgFiles.slice(i, i + SVG_BATCH);
|
||||
const results = await Promise.allSettled(
|
||||
batch.map(async ({ relPath }) => {
|
||||
// Rasterize the batch one file at a time, then caption it in parallel.
|
||||
//
|
||||
// Rasterizing the whole batch concurrently drove up to SVG_BATCH simultaneous librsvg
|
||||
// renders through libvips, and that corrupts the heap: production captures aborted
|
||||
// with `free(): unaligned chunk detected in tcache 2` (SIGABRT) during this phase and
|
||||
// lost the entire capture. A native abort cannot be caught by the try/catch below, so
|
||||
// the concurrency has to go rather than be handled. Rasterizing is local and cheap;
|
||||
// the vision request is the slow leg and stays parallel, so throughput barely moves.
|
||||
const rasterized: { relPath: string; pngBase64: string | null }[] = [];
|
||||
sharp.concurrency(1);
|
||||
try {
|
||||
for (const { relPath } of batch) {
|
||||
const filePath = join(assetsDir, relPath);
|
||||
let pngBase64: string;
|
||||
try {
|
||||
// Flatten against a contrasting background — white-on-white SVGs render invisible to Vision.
|
||||
const svgSource = readFileSync(filePath, "utf-8");
|
||||
@@ -504,12 +565,21 @@ export async function captionImagesWithGemini(
|
||||
.flatten({ background: bg })
|
||||
.png()
|
||||
.toBuffer();
|
||||
pngBase64 = pngBuffer.toString("base64");
|
||||
rasterized.push({ relPath, pngBase64: pngBuffer.toString("base64") });
|
||||
} catch {
|
||||
// exotic SVG features may break sharp; skip caption rather than block
|
||||
svgsSkipped++;
|
||||
return { file: relPath, caption: "" };
|
||||
rasterized.push({ relPath, pngBase64: null });
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
// Hand the pool back even if a rasterize threw: the vision requests below are network
|
||||
// work that gains nothing from a pinned pool, and the process outlives this capture.
|
||||
sharp.concurrency(hostConcurrency);
|
||||
}
|
||||
const results = await Promise.allSettled(
|
||||
rasterized.map(async ({ relPath, pngBase64 }) => {
|
||||
if (pngBase64 === null) return { file: relPath, caption: "" };
|
||||
const caption = await captionOne({
|
||||
mimeType: "image/png",
|
||||
base64: pngBase64,
|
||||
|
||||
@@ -757,15 +757,21 @@ export async function captureWebsite(
|
||||
const lines = generateAssetDescriptions(outputDir, tokens, catalogedAssets, geminiCaptions);
|
||||
|
||||
if (lines.length > 0) {
|
||||
// Mirrors the provider gate in contentExtractor: Vertex needs a project AND a service
|
||||
// account, and is the configuration a server deployment actually has. Without it here the
|
||||
// header claimed "GEMINI_API_KEY not set — descriptions are catalog-derived" on a capture
|
||||
// whose captions Vertex had just generated, and that header is read downstream.
|
||||
const hasVisionKey = !!(
|
||||
!skipVision &&
|
||||
(process.env.OPENROUTER_API_KEY ||
|
||||
process.env.GEMINI_API_KEY ||
|
||||
process.env.GOOGLE_API_KEY)
|
||||
process.env.GOOGLE_API_KEY ||
|
||||
(process.env.HYPERFRAMES_VERTEX_PROJECT_ID &&
|
||||
process.env.HYPERFRAMES_VERTEX_SERVICE_ACCOUNT))
|
||||
);
|
||||
const header = hasVisionKey
|
||||
? "# Asset Descriptions\n\nOne line per file. Read this instead of opening every image individually.\n\nTo find a specific brand or icon, **grep this file for the brand name in the description text** (e.g. `grep -i 'autodesk' asset-descriptions.md`). The Gemini Vision captions identify what's actually in each file — that's the agent's selector.\n\nThe `logo-<hash>.svg` filename prefix is a cheap structural hint (DOM said this SVG was inside a `<header>`, home-link `<a>`, or had an aria-label matching the page brand). It is NOT a content claim — many `logo-*` files are nav icons or decorative shapes. Trust the captions, not the filename prefix.\n\n"
|
||||
: "# Asset Descriptions\n\n⚠️ GEMINI_API_KEY not set — descriptions below are catalog-derived (alt text, headings, section context, filename) instead of Vision-generated. To get richer Vision descriptions on the next capture, set GEMINI_API_KEY (or GOOGLE_API_KEY) and re-run.\n\nThe `logo-<hash>.svg` filename prefix is a structural hint (DOM said this SVG was inside a `<header>`, home-link `<a>`, or had an aria-label matching the page brand). To pick the actual brand logo without Vision, open the `logo-*` candidates in a previewer or rasterize them with `sharp` before referencing — composing a fake logo ships off-brand in the final video.\n\n";
|
||||
: "# Asset Descriptions\n\n⚠️ No vision credentials — descriptions below are catalog-derived (alt text, headings, section context, filename) instead of Vision-generated. To get richer Vision descriptions on the next capture, set GEMINI_API_KEY (or GOOGLE_API_KEY), or HYPERFRAMES_VERTEX_PROJECT_ID plus HYPERFRAMES_VERTEX_SERVICE_ACCOUNT for Vertex service-account auth, and re-run.\n\nThe `logo-<hash>.svg` filename prefix is a structural hint (DOM said this SVG was inside a `<header>`, home-link `<a>`, or had an aria-label matching the page brand). To pick the actual brand logo without Vision, open the `logo-*` candidates in a previewer or rasterize them with `sharp` before referencing — composing a fake logo ships off-brand in the final video.\n\n";
|
||||
writeFileSync(
|
||||
join(outputDir, "extracted", "asset-descriptions.md"),
|
||||
header + lines.map((l) => "- " + l).join("\n") + "\n",
|
||||
|
||||
Reference in New Issue
Block a user