diff --git a/docs/guides/authentication.mdx b/docs/guides/authentication.mdx index bc849912f9..2b2f38706f 100644 --- a/docs/guides/authentication.mdx +++ b/docs/guides/authentication.mdx @@ -75,7 +75,7 @@ vision key is available. | **Voice (TTS)** | HeyGen → ElevenLabs → Kokoro | HeyGen credential (above) · then `ELEVENLABS_API_KEY` | Kokoro: `pip install kokoro-onnx soundfile` | | **Music (BGM)** | HeyGen library → Lyria → MusicGen | HeyGen credential (above) · then `GEMINI_API_KEY` → `GOOGLE_API_KEY` | MusicGen: `pip install transformers torch soundfile numpy` | | **Sound effects** | HeyGen library → bundled library | HeyGen credential (above) | bundled — no deps | -| **Capture descriptions** | OpenRouter → Gemini | `OPENROUTER_API_KEY` → `GEMINI_API_KEY` | None; optional for [website capture](/guides/product-launch-video) | +| **Capture descriptions** | Gemini | `GEMINI_API_KEY` | None; optional for [website capture](/guides/product-launch-video) | Run `npx hyperframes doctor` to check which local dependencies are installed. The media workflows run `hyperframes auth status` before generation and tell @@ -154,7 +154,6 @@ to a shared space with `--space`. | `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Explicit Gemini TTS selection and Lyria music generation; capture descriptions use `GEMINI_API_KEY`. | | `GOOGLE_APPLICATION_CREDENTIALS` / `GCS_CREDS` | Gemini TTS service-account JSON file path / injected JSON. Used when neither API key is set. | | `GOOGLE_CLOUD_PROJECT` / `GCLOUD_PROJECT_ID` | Gemini OAuth quota project; first set value wins, then the service account project. | -| `OPENROUTER_API_KEY` | Capture descriptions; takes priority over Gemini for that step. | ## Related topics diff --git a/docs/guides/deploy.mdx b/docs/guides/deploy.mdx index 74219913db..ff2ecfe6ff 100644 --- a/docs/guides/deploy.mdx +++ b/docs/guides/deploy.mdx @@ -46,7 +46,7 @@ platform limits. The Cloudflare template is a TanStack Start application on Workers. A `RenderContainer` Durable Object runs the renderer and streams the result to R2. The template also includes optional prompt-based generation with a user's -own OpenRouter credential. +own model-provider credential. [Deploy the Cloudflare template](https://deploy.workers.cloudflare.com/?url=https://github.com/heygen-com/hyperframes-cloudflare-template) diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx index 88e68350ff..1b9e93dd10 100644 --- a/docs/packages/cli.mdx +++ b/docs/packages/cli.mdx @@ -387,9 +387,7 @@ to those local files. Distinct URLs such as `demo.woff2?v=1` and URL unchanged. For AI image descriptions, set `GEMINI_API_KEY` in a `.env` file -(~$0.001/image), or `OPENROUTER_API_KEY` to route any vision model through -[OpenRouter](https://openrouter.ai) — it wins if both are set, and -`HYPERFRAMES_OPENROUTER_MODEL` overrides the model. +(~$0.001/image). #### Local HTML files @@ -2515,7 +2513,7 @@ hyperframes auth logout --yes # no prompt | `HYPERFRAMES_OAUTH_CLIENT_ID` | Override the selected public OAuth client. | For the keys other capabilities use — ElevenLabs and Gemini for voice and music -fallback, OpenRouter and Gemini for capture — and how the skills prioritize +fallback, Gemini for capture — and how the skills prioritize them, see [Authentication and API keys](/guides/authentication). ## hyperframes cloud diff --git a/packages/cli/src/capture/contentExtractor.file-race.test.ts b/packages/cli/src/capture/contentExtractor.file-race.test.ts index e0e974f154..8edf8afa90 100644 --- a/packages/cli/src/capture/contentExtractor.file-race.test.ts +++ b/packages/cli/src/capture/contentExtractor.file-race.test.ts @@ -4,6 +4,17 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { captionImagesWithGemini } from "./contentExtractor.js"; +const { generateContentMock } = vi.hoisted(() => ({ generateContentMock: vi.fn() })); + +vi.mock("../utils/optionalPackages.js", () => ({ + loadSharp: async () => (await import("sharp")).default, + loadOptionalPackage: vi.fn(async () => ({ + GoogleGenAI: class { + models = { generateContent: generateContentMock }; + }, + })), +})); + const hooks = vi.hoisted(() => ({ afterStat: () => {}, beforeRead: () => {}, @@ -41,10 +52,6 @@ describe("caption image file reads", () => { let dir: string; let file: string; let sentBody: string; - const fetchMock = vi.fn(async (_url: string, init?: RequestInit) => { - sentBody = typeof init?.body === "string" ? init.body : ""; - return new Response(JSON.stringify({ choices: [{ message: { content: "A hero." } }] })); - }); beforeEach(() => { dir = fs.mkdtempSync(join(tmpdir(), "hf-caption-read-")); @@ -56,15 +63,19 @@ describe("caption image file reads", () => { hooks.descriptor = -1; hooks.closed = []; sentBody = ""; - fetchMock.mockClear(); - vi.stubEnv("OPENROUTER_API_KEY", "test-key"); - vi.stubGlobal("fetch", fetchMock); + generateContentMock.mockReset(); + generateContentMock.mockImplementation(async (request: unknown) => { + sentBody = JSON.stringify(request); + return { text: "A hero." }; + }); + vi.stubEnv("GEMINI_API_KEY", "test-key"); + vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); }); afterEach(() => { hooks.afterStat = () => {}; hooks.beforeRead = () => {}; - vi.unstubAllGlobals(); vi.unstubAllEnvs(); fs.rmSync(dir, { recursive: true, force: true }); }); @@ -92,7 +103,7 @@ describe("caption image file reads", () => { hooks.beforeRead = read; const captions = await captionImagesWithGemini(dir, () => {}, []); expect(captions).toEqual(size === 4_000_000 ? { "hero.png": "A hero." } : {}); - expect(fetchMock).toHaveBeenCalledTimes(size === 4_000_000 ? 1 : 0); + expect(generateContentMock).toHaveBeenCalledTimes(size === 4_000_000 ? 1 : 0); expect(read).toHaveBeenCalledTimes(size === 4_000_000 ? 1 : 0); expectClosed(); }); @@ -105,8 +116,8 @@ describe("caption image file reads", () => { else hooks.beforeRead = fail; const warnings: string[] = []; expect(await captionImagesWithGemini(dir, () => {}, warnings)).toEqual({}); - expect(warnings).toEqual(["OpenRouter vision failed for 1 asset(s); captions omitted."]); - expect(fetchMock).not.toHaveBeenCalled(); + expect(warnings).toEqual(["Gemini vision failed for 1 asset(s); captions omitted."]); + expect(generateContentMock).not.toHaveBeenCalled(); expectClosed(); }); }); diff --git a/packages/cli/src/capture/contentExtractor.test.ts b/packages/cli/src/capture/contentExtractor.test.ts index 10d7c5e75f..ad6d165ccc 100644 --- a/packages/cli/src/capture/contentExtractor.test.ts +++ b/packages/cli/src/capture/contentExtractor.test.ts @@ -73,15 +73,10 @@ vi.mock("../utils/optionalPackages.js", () => ({ })), })); -// These tests exercise the OpenRouter provider path only — it makes a plain -// `fetch` call we can stub, with no native (`sharp`) or `@google/genai` -// dependency. OpenRouter wins over Gemini when OPENROUTER_API_KEY is set, so we -// don't need to clear the Gemini keys for the OpenRouter cases. - function makeProjectWithImages(files = ["hero.png"]): string { const dir = mkdtempSync(join(tmpdir(), "hf-caption-")); mkdirSync(join(dir, "assets"), { recursive: true }); - // Contents are irrelevant to the OpenRouter path (it just base64-encodes the + // Contents are irrelevant to the provider path (it just base64-encodes the // bytes); only the .png extension matters for the image filter. for (const file of files) { writeFileSync(join(dir, "assets", file), Buffer.from([0x89, 0x50, 0x4e, 0x47])); @@ -89,184 +84,55 @@ function makeProjectWithImages(files = ["hero.png"]): string { return dir; } -describe("captionImagesWithGemini — OpenRouter provider", () => { +describe("captionImagesWithGemini — request and outcome handling", () => { const dirs: string[] = []; afterEach(() => { - vi.unstubAllGlobals(); + generateContentMock.mockReset(); vi.unstubAllEnvs(); for (const d of dirs) rmSync(d, { recursive: true, force: true }); dirs.length = 0; }); - it("captions via OpenRouter when OPENROUTER_API_KEY is set", async () => { + function geminiOnly(): void { + vi.stubEnv("GEMINI_API_KEY", "gemini-test-key"); + vi.stubEnv("GOOGLE_API_KEY", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); + } + + it("captions the image through Gemini with the configured model", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-test-key"); - vi.stubEnv("HYPERFRAMES_OPENROUTER_MODEL", "google/gemini-3.1-flash-lite"); - - // Capture the request inside the mock, where the args are well-typed — - // avoids casting `mock.calls` (and the repo's ban on `as` assertions). - let capturedUrl: string | undefined; - let capturedInit: RequestInit | undefined; - const fetchMock = vi.fn(async (url: string, init?: RequestInit) => { - capturedUrl = url; - capturedInit = init; - return new Response( - JSON.stringify({ choices: [{ message: { content: "A dark hero with blue accents." } }] }), - { status: 200, headers: { "content-type": "application/json" } }, - ); + geminiOnly(); + vi.stubEnv("HYPERFRAMES_GEMINI_MODEL", "gemini-test-model"); + let request: + | { model: string; contents: Array<{ parts: Array<{ inlineData?: unknown }> }> } + | undefined; + generateContentMock.mockImplementation(async (value: typeof request) => { + request = value; + return { text: "A dark hero with blue accents." }; }); - vi.stubGlobal("fetch", fetchMock); const warnings: string[] = []; const captions = await captionImagesWithGemini(dir, () => {}, warnings); expect(captions).toEqual({ "hero.png": "A dark hero with blue accents." }); expect(warnings).toEqual([]); - expect(fetchMock).toHaveBeenCalledTimes(1); - - expect(capturedUrl).toBe("https://openrouter.ai/api/v1/chat/completions"); - expect(new Headers(capturedInit?.headers).get("authorization")).toBe("Bearer or-test-key"); - const body = JSON.parse(typeof capturedInit?.body === "string" ? capturedInit.body : "{}"); - expect(body.model).toBe("google/gemini-3.1-flash-lite"); - const image = body.messages[0].content.find((p: { type: string }) => p.type === "image_url"); - expect(image?.image_url?.url).toMatch(/^data:image\/png;base64,/); - }); - - it("degrades gracefully (no throw, no captions) when OpenRouter returns a non-OK status", async () => { - const dir = makeProjectWithImages(); - dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-bad-key"); - - vi.stubGlobal( - "fetch", - vi.fn( - async () => - new Response("provider detail containing or-bad-key", { - status: 401, - statusText: "Unauthorized", - }), - ), - ); - - const warnings: string[] = []; - let outcome: VisionCaptionOutcome | undefined; - const captions = await captionImagesWithGemini(dir, () => {}, warnings, { - onOutcome: (value) => { - outcome = value; - }, - }); - - expect(captions).toEqual({}); - expect(warnings).toEqual(["OpenRouter vision failed for 1 asset(s); captions omitted."]); - expect(warnings.join(" ")).not.toContain("or-bad-key"); - expect(outcome).toEqual({ - timedOutRequests: 0, - failedRequests: 1, - budgetExhausted: false, - }); - if (!outcome) throw new Error("Expected vision caption outcome"); - expect(resolveVisionPhaseCompletion(outcome, 10_000)).toEqual({ - status: "degraded", - reason: "provider-error", + expect(generateContentMock).toHaveBeenCalledTimes(1); + expect(request?.model).toBe("gemini-test-model"); + expect(request?.contents[0]?.parts[0]?.inlineData).toEqual({ + mimeType: "image/png", + data: Buffer.from([0x89, 0x50, 0x4e, 0x47]).toString("base64"), }); }); - it("terminates captioning when the vision provider never responds", async () => { - const dir = makeProjectWithImages(); - dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-hanging-key"); - vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); - - vi.stubGlobal( - "fetch", - vi.fn((_url: string, init?: RequestInit) => { - return new Promise((_resolve, reject) => { - init?.signal?.addEventListener("abort", () => reject(init.signal?.reason), { - once: true, - }); - }); - }), - ); - - const warnings: string[] = []; - const result = await Promise.race([ - captionImagesWithGemini(dir, () => {}, warnings), - new Promise<"hung">((resolve) => setTimeout(() => resolve("hung"), 250)), - ]); - - expect(result).toEqual({}); - }); - - it.each([ - { label: "success JSON", status: 200, partialBody: '{"choices":[' }, - { label: "failure text", status: 500, partialBody: "upstream failed: " }, - ])( - "terminates captioning when OpenRouter $label body never ends", - async ({ status, partialBody }) => { - const dir = makeProjectWithImages(); - dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-stalled-body-key"); - vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); - - vi.stubGlobal( - "fetch", - vi.fn(async () => { - const body = new ReadableStream({ - start(controller) { - controller.enqueue(new TextEncoder().encode(partialBody)); - }, - }); - return new Response(body, { - status, - headers: { "content-type": "application/json" }, - }); - }), - ); - - const warnings: string[] = []; - let outcome: VisionCaptionOutcome | undefined; - const result = await Promise.race([ - captionImagesWithGemini(dir, () => {}, warnings, { - onOutcome: (value) => { - outcome = value; - }, - }), - new Promise<"hung">((resolve) => setTimeout(() => resolve("hung"), 250)), - ]); - - expect(result).toEqual({}); - expect(warnings).toEqual(["OpenRouter vision timed out for 1 asset(s); captions omitted."]); - expect(outcome).toEqual({ - timedOutRequests: 1, - failedRequests: 0, - budgetExhausted: false, - }); - }, - ); - it("keeps successful captions while reporting a failed sibling request", async () => { const dir = makeProjectWithImages(["failed.png", "success.png"]); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-mixed-failure-key"); - - let requestCount = 0; - vi.stubGlobal( - "fetch", - vi.fn(async () => { - requestCount++; - if (requestCount === 1) { - return new Response("upstream exploded with or-mixed-failure-key", { - status: 500, - statusText: "Internal Server Error", - }); - } - return new Response( - JSON.stringify({ choices: [{ message: { content: "Successful image." } }] }), - { status: 200, headers: { "content-type": "application/json" } }, - ); - }), - ); + geminiOnly(); + generateContentMock + .mockRejectedValueOnce(new Error("upstream exploded with gemini-test-key")) + .mockResolvedValue({ text: "Successful image." }); const warnings: string[] = []; let outcome: VisionCaptionOutcome | undefined; @@ -277,8 +143,8 @@ describe("captionImagesWithGemini — OpenRouter provider", () => { }); expect(captions).toEqual({ "success.png": "Successful image." }); - expect(warnings).toEqual(["OpenRouter vision failed for 1 asset(s); captions omitted."]); - expect(warnings.join(" ")).not.toContain("or-mixed-failure-key"); + expect(warnings).toEqual(["Gemini vision failed for 1 asset(s); captions omitted."]); + expect(warnings.join(" ")).not.toContain("gemini-test-key"); expect(outcome).toEqual({ timedOutRequests: 0, failedRequests: 1, @@ -289,35 +155,15 @@ describe("captionImagesWithGemini — OpenRouter provider", () => { it("keeps a successful sibling caption when another request times out", async () => { const dir = makeProjectWithImages(["hang.png", "success.png"]); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-mixed-key"); + geminiOnly(); vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); + // Both fixtures contain the same bytes, so use call order: the first request hangs. + generateContentMock + .mockImplementationOnce(() => new Promise(() => {})) + .mockResolvedValue({ text: "Successful image." }); - vi.stubGlobal( - "fetch", - vi.fn((_url: string, init?: RequestInit) => { - const body = JSON.parse(typeof init?.body === "string" ? init.body : "{}"); - const imageUrl = body.messages?.[0]?.content?.[1]?.image_url?.url; - if (typeof imageUrl === "string" && imageUrl.includes("iVBORw==")) { - // Both fixtures contain the same bytes, so use call order: the first - // request hangs while the second succeeds independently. - const fetchMock = vi.mocked(fetch); - if (fetchMock.mock.calls.length === 1) return new Promise(() => {}); - } - return Promise.resolve( - new Response( - JSON.stringify({ choices: [{ message: { content: "Successful image." } }] }), - { - status: 200, - headers: { "content-type": "application/json" }, - }, - ), - ); - }), - ); - - const warnings: string[] = []; let outcome: VisionCaptionOutcome | undefined; - const captions = await captionImagesWithGemini(dir, () => {}, warnings, { + const captions = await captionImagesWithGemini(dir, () => {}, [], { onOutcome: (value) => { outcome = value; }, @@ -339,35 +185,31 @@ describe("captionImagesWithGemini — OpenRouter provider", () => { it("does not call a configured provider when vision is skipped", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-unused-key"); - const fetchMock = vi.fn(); - vi.stubGlobal("fetch", fetchMock); + geminiOnly(); const captions = await captionImagesWithGemini(dir, () => {}, [], { skipVision: true }); expect(captions).toEqual({}); - expect(fetchMock).not.toHaveBeenCalled(); + expect(generateContentMock).not.toHaveBeenCalled(); }); it("does not start a vision batch after the capture budget is exhausted", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-budget-key"); - const fetchMock = vi.fn(); - vi.stubGlobal("fetch", fetchMock); + geminiOnly(); const captions = await captionImagesWithGemini(dir, () => {}, [], { remainingMs: () => 0, }); expect(captions).toEqual({}); - expect(fetchMock).not.toHaveBeenCalled(); + expect(generateContentMock).not.toHaveBeenCalled(); }); it("classifies non-provider pipeline failures without leaking the local path", async () => { const dir = mkdtempSync(join(tmpdir(), "hf-caption-no-assets-")); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-unused-key"); + geminiOnly(); const warnings: string[] = []; let outcome: VisionCaptionOutcome | undefined; @@ -378,13 +220,13 @@ describe("captionImagesWithGemini — OpenRouter provider", () => { }); expect(captions).toEqual({}); - expect(warnings).toEqual(["OpenRouter captioning failed internally; captions omitted."]); + expect(warnings).toEqual(["Gemini captioning failed internally; captions omitted."]); expect(warnings.join(" ")).not.toContain(dir); expect(outcome).toEqual({ timedOutRequests: 0, failedRequests: 0, - budgetExhausted: false, internalError: true, + budgetExhausted: false, }); if (!outcome) throw new Error("Expected vision caption outcome"); expect(resolveVisionPhaseCompletion(outcome, 10_000)).toEqual({ @@ -396,18 +238,15 @@ describe("captionImagesWithGemini — OpenRouter provider", () => { it("skips captioning entirely when no provider key is present", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", ""); vi.stubEnv("GOOGLE_API_KEY", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); - const fetchMock = vi.fn(); - vi.stubGlobal("fetch", fetchMock); - - const warnings: string[] = []; - const captions = await captionImagesWithGemini(dir, () => {}, warnings); + const captions = await captionImagesWithGemini(dir, () => {}, []); expect(captions).toEqual({}); - expect(fetchMock).not.toHaveBeenCalled(); + expect(generateContentMock).not.toHaveBeenCalled(); }); }); @@ -424,7 +263,6 @@ describe("captionImagesWithGemini — Gemini provider", () => { it("terminates captioning when Gemini never settles even if the SDK ignores abort", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", "gemini-hanging-key"); vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); generateContentMock.mockImplementation(() => new Promise(() => {})); @@ -440,7 +278,6 @@ describe("captionImagesWithGemini — Gemini provider", () => { it("reports a rejected Gemini request without leaking its error detail", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", "gemini-secret-key"); generateContentMock.mockRejectedValue( new Error("provider echoed gemini-secret-key in its response"), @@ -494,7 +331,6 @@ describe("captionImagesWithGemini — Vertex AI provider", () => { }); function vertexEnv(): void { - vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000"); vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", SERVICE_ACCOUNT); } @@ -585,7 +421,6 @@ describe("captionImagesWithGemini — Vertex AI provider", () => { it("skips captioning when the service account is unparseable, without echoing it", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", ""); vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000"); vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", "{not-json super-secret-material"); @@ -610,28 +445,9 @@ describe("captionImagesWithGemini — Vertex AI provider", () => { }); }); - it("stays on OpenRouter when the user has opted into it explicitly", async () => { - const dir = makeProjectWithImages(); - dirs.push(dir); - vertexEnv(); - vi.stubEnv("OPENROUTER_API_KEY", "or-key"); - const fetchMock = vi.fn().mockResolvedValue({ - ok: true, - json: async () => ({ choices: [{ message: { content: "A caption." } }] }), - }); - vi.stubGlobal("fetch", fetchMock); - - await captionImagesWithGemini(dir, () => {}, []); - - expect(fetchMock).toHaveBeenCalled(); - expect(clientOptions).toHaveLength(0); - vi.unstubAllGlobals(); - }); - it("needs both halves of the credential before it will use Vertex", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", ""); vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000"); vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); @@ -683,22 +499,15 @@ describe("captionImagesWithGemini — SVG rasterization", () => { // makes "one at a time" the assertion, not "errors are reported". const dir = makeProjectWithSvgs(6); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-key"); - vi.stubGlobal( - "fetch", - vi.fn(async () => { - // Slower than a render, so overlapping renders would be the easy way to go faster — - // the point is that the loop does not take it. - await new Promise((resolve) => setTimeout(resolve, 15)); - return new Response( - JSON.stringify({ choices: [{ message: { content: "A dark glyph." } }] }), - { - status: 200, - headers: { "content-type": "application/json" }, - }, - ); - }), - ); + vi.stubEnv("GEMINI_API_KEY", "gemini-key"); + vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); + generateContentMock.mockImplementation(async () => { + // Slower than a render, so overlapping renders would be the easy way to go faster — + // the point is that the loop does not take it. + await new Promise((resolve) => setTimeout(resolve, 15)); + return { text: "A dark glyph." }; + }); const captions = await captionImagesWithGemini(dir, () => {}, []); @@ -714,17 +523,10 @@ describe("captionImagesWithGemini — SVG rasterization", () => { // which asked for captioning, so the bound has to be given back when the renders are done. const dir = makeProjectWithSvgs(1); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-key"); - vi.stubGlobal( - "fetch", - vi.fn( - async () => - new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), { - status: 200, - headers: { "content-type": "application/json" }, - }), - ), - ); + vi.stubEnv("GEMINI_API_KEY", "gemini-key"); + vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); + generateContentMock.mockResolvedValue({ text: "A glyph." }); await captionImagesWithGemini(dir, () => {}, []); @@ -737,17 +539,10 @@ describe("captionImagesWithGemini — SVG rasterization", () => { // its threads. const dir = makeProjectWithSvgs(2); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-key"); - vi.stubGlobal( - "fetch", - vi.fn( - async () => - new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), { - status: 200, - headers: { "content-type": "application/json" }, - }), - ), - ); + vi.stubEnv("GEMINI_API_KEY", "gemini-key"); + vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); + generateContentMock.mockResolvedValue({ text: "A glyph." }); // Unreadable, which is how an exotic SVG behaves through sharp. rmSync(join(dir, "assets", "svgs", "logo-0.svg")); mkdirSync(join(dir, "assets", "svgs", "logo-0.svg")); @@ -760,17 +555,10 @@ describe("captionImagesWithGemini — SVG rasterization", () => { it("keeps captioning the rest when one SVG cannot be rasterized", async () => { const dir = makeProjectWithSvgs(3); dirs.push(dir); - vi.stubEnv("OPENROUTER_API_KEY", "or-key"); - vi.stubGlobal( - "fetch", - vi.fn( - async () => - new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), { - status: 200, - headers: { "content-type": "application/json" }, - }), - ), - ); + vi.stubEnv("GEMINI_API_KEY", "gemini-key"); + vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); + vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); + generateContentMock.mockResolvedValue({ text: "A glyph." }); // One file is unreadable, which is how an exotic SVG behaves through sharp. rmSync(join(dir, "assets", "svgs", "logo-1.svg")); mkdirSync(join(dir, "assets", "svgs", "logo-1.svg")); diff --git a/packages/cli/src/capture/contentExtractor.ts b/packages/cli/src/capture/contentExtractor.ts index a951f43177..baf2472eef 100644 --- a/packages/cli/src/capture/contentExtractor.ts +++ b/packages/cli/src/capture/contentExtractor.ts @@ -245,9 +245,8 @@ export async function extractVisibleText(page: Page): Promise { /** * Caption downloaded images using a vision model. * - * Provider is chosen by which API key is present: OPENROUTER_API_KEY → OpenRouter - * (any vision model via its OpenAI-style API), else GEMINI_API_KEY/GOOGLE_API_KEY - * → Google Gemini, else no captioning. OpenRouter wins if both are set. + * Provider is chosen by which credential is present: Vertex service account, else + * GEMINI_API_KEY/GOOGLE_API_KEY → Google Gemini, else no captioning. * * Batches requests to stay under free-tier rate limits. * Returns a map of filename -> caption string. @@ -277,7 +276,6 @@ export async function captionImagesWithGemini( reportOutcome(); return geminiCaptions; } - const openRouterKey = process.env.OPENROUTER_API_KEY; const geminiKey = process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY; // Vertex authenticates with a service account and a project instead of an API key. Server // deployments have those; they often do not have a working Gemini API key, and a rejected @@ -286,28 +284,20 @@ export async function captionImagesWithGemini( const vertexProject = process.env.HYPERFRAMES_VERTEX_PROJECT_ID; const vertexServiceAccount = process.env.HYPERFRAMES_VERTEX_SERVICE_ACCOUNT; const useVertex = Boolean(vertexProject && vertexServiceAccount); - if (!openRouterKey && !useVertex && !geminiKey) { + if (!useVertex && !geminiKey) { reportOutcome(); return geminiCaptions; } - // OpenRouter takes priority — it's the explicit opt-in for users without Google access. // Vertex outranks the bare API key because it is the credential a deployment actually - // holds. All three satisfy the same single-image → one-line-caption contract + // holds. Both satisfy the same single-image → one-line-caption contract // (`captionOne`), so the batching and SVG-rasterization loops stay provider-agnostic. - const provider: "openrouter" | "vertex" | "gemini" = openRouterKey - ? "openrouter" - : useVertex - ? "vertex" - : "gemini"; - const providerName = { openrouter: "OpenRouter", vertex: "Vertex AI", gemini: "Gemini" }[ - provider - ]; - // Override per provider via HYPERFRAMES_OPENROUTER_MODEL / HYPERFRAMES_VERTEX_MODEL / - // HYPERFRAMES_GEMINI_MODEL. Vertex publishes a different model set than the Gemini API — - // the API's flash-lite preview id is not resolvable there — so it carries its own default. + const provider: "vertex" | "gemini" = useVertex ? "vertex" : "gemini"; + const providerName = { vertex: "Vertex AI", gemini: "Gemini" }[provider]; + // Override per provider via HYPERFRAMES_VERTEX_MODEL / HYPERFRAMES_GEMINI_MODEL. Vertex + // publishes a different model set than the Gemini API (the API's flash-lite preview id is + // not resolvable there), so it carries its own default. const model = { - openrouter: process.env.HYPERFRAMES_OPENROUTER_MODEL || "google/gemini-3.1-flash-lite", vertex: process.env.HYPERFRAMES_VERTEX_MODEL || "gemini-2.5-flash", gemini: process.env.HYPERFRAMES_GEMINI_MODEL || "gemini-3.1-flash-lite-preview", }[provider]; @@ -326,114 +316,72 @@ export async function captionImagesWithGemini( }) => Promise; let captionOne: CaptionOne; - if (provider === "openrouter") { - captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => { - return runBoundedVisionRequest(async (signal) => { - const res = await fetch("https://openrouter.ai/api/v1/chat/completions", { - method: "POST", - headers: { - Authorization: `Bearer ${openRouterKey}`, - "Content-Type": "application/json", - }, - signal, - body: JSON.stringify({ - model, - messages: [ - { - role: "user", - content: [ - { type: "text", text: prompt }, - { - type: "image_url", - image_url: { url: `data:${mimeType};base64,${base64}` }, - }, - ], - }, - ], - max_tokens: maxTokens, - }), - }); - if (!res.ok) { - await res.text(); - throw new Error(`OpenRouter request failed with HTTP ${res.status}`); - } - const data = (await res.json()) as { - choices?: Array<{ message?: { content?: string } }>; - }; - return data.choices?.[0]?.message?.content?.trim() || ""; - }, timeoutMs); - }; - } else { - let GoogleGenAI: OptionalPackageModules["@google/genai"]["GoogleGenAI"]; + let GoogleGenAI: OptionalPackageModules["@google/genai"]["GoogleGenAI"]; + try { + ({ GoogleGenAI } = await loadOptionalPackage("@google/genai", "vision captioning")); + } catch (err) { + warnings.push(`Skipped vision captioning: ${(err as Error).message}`); + internalError = true; + reportOutcome(); + return geminiCaptions; + } + let ai: InstanceType; + if (provider === "vertex") { + // Re-narrow for TS; `useVertex` already guaranteed both are set. + if (!vertexProject || !vertexServiceAccount) return geminiCaptions; + let credentials: Record; try { - ({ GoogleGenAI } = await loadOptionalPackage("@google/genai", "vision captioning")); - } catch (err) { - warnings.push(`Skipped vision captioning: ${(err as Error).message}`); + credentials = JSON.parse(vertexServiceAccount) as Record; + } catch { + warnings.push( + "HYPERFRAMES_VERTEX_SERVICE_ACCOUNT is not valid JSON; skipped vision captioning.", + ); internalError = true; reportOutcome(); return geminiCaptions; } - let ai: InstanceType; - if (provider === "vertex") { - // Re-narrow for TS; `useVertex` already guaranteed both are set. - if (!vertexProject || !vertexServiceAccount) return geminiCaptions; - let credentials: Record; - try { - credentials = JSON.parse(vertexServiceAccount) as Record; - } catch { - warnings.push( - "HYPERFRAMES_VERTEX_SERVICE_ACCOUNT is not valid JSON; skipped vision captioning.", - ); - internalError = true; - reportOutcome(); - return geminiCaptions; - } - ai = new GoogleGenAI({ - vertexai: true, - project: vertexProject, - location: process.env.HYPERFRAMES_VERTEX_LOCATION || "us-central1", - googleAuthOptions: { credentials }, - }); - } else { - // Unreachable when geminiKey is unset (guarded above); re-narrow for TS. - if (!geminiKey) return geminiCaptions; - ai = new GoogleGenAI({ apiKey: geminiKey }); - } - captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => { - const response = await runBoundedVisionRequest( - (signal) => - ai.models.generateContent({ - model, - contents: [ - { - role: "user", - parts: [{ inlineData: { mimeType, data: base64 } }, { text: prompt }], - }, - ], - config: { - maxOutputTokens: maxTokens, - // A one-line factual caption needs no reasoning, and leaving thinking on is - // not merely wasteful: thinking tokens are drawn from maxOutputTokens, so the - // model can spend the whole budget and return empty text. That surfaces as a - // successful request with no caption — the capture then logs "Captioned N/N - // images" followed by "0 images captioned", which is what production shows. - thinkingConfig: { thinkingBudget: 0 }, - abortSignal: signal, - httpOptions: { timeout: timeoutMs }, - }, - }), - timeoutMs, - ); - return response.text?.trim() || ""; - }; + ai = new GoogleGenAI({ + vertexai: true, + project: vertexProject, + location: process.env.HYPERFRAMES_VERTEX_LOCATION || "us-central1", + googleAuthOptions: { credentials }, + }); + } else { + // Unreachable when geminiKey is unset (guarded above); re-narrow for TS. + if (!geminiKey) return geminiCaptions; + ai = new GoogleGenAI({ apiKey: geminiKey }); } + captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => { + const response = await runBoundedVisionRequest( + (signal) => + ai.models.generateContent({ + model, + contents: [ + { + role: "user", + parts: [{ inlineData: { mimeType, data: base64 } }, { text: prompt }], + }, + ], + config: { + maxOutputTokens: maxTokens, + // Thinking tokens come out of maxOutputTokens, so a thinking model can spend the + // whole budget and return empty text: a successful request with no caption. + thinkingConfig: { thinkingBudget: 0 }, + abortSignal: signal, + httpOptions: { timeout: timeoutMs }, + }, + }), + timeoutMs, + ); + return response.text?.trim() || ""; + }; const imageFiles = readdirSync(join(outputDir, "assets")).filter((f: string) => /\.(png|jpg|jpeg|webp|gif)$/i.test(f), ); // Caption in parallel batches. Gemini free tier is ~5 RPM (slow but $0), - // paid/OpenRouter ~2000 RPM. We batch 20 with a 2s inter-batch pause and rely + // paid ~2000 RPM. We batch 20 with a 2s inter-batch pause and rely // on Promise.allSettled so one rate-limited image does not fail its siblings; // rejected requests are aggregated into a sanitized warning below. const BATCH_SIZE = 20; diff --git a/packages/cli/src/capture/postExtractionPhase.ts b/packages/cli/src/capture/postExtractionPhase.ts index 8733462c53..26b7fb9e6c 100644 --- a/packages/cli/src/capture/postExtractionPhase.ts +++ b/packages/cli/src/capture/postExtractionPhase.ts @@ -34,8 +34,7 @@ import type { VisionCaptionOutcome } from "./contentExtractor.js"; function hasVisionCredentials(skipVision: boolean): boolean { return Boolean( !skipVision && - (process.env.OPENROUTER_API_KEY || - process.env.GEMINI_API_KEY || + (process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY || (process.env.HYPERFRAMES_VERTEX_PROJECT_ID && process.env.HYPERFRAMES_VERTEX_SERVICE_ACCOUNT)), diff --git a/packages/cli/src/capture/scaffolding.test.ts b/packages/cli/src/capture/scaffolding.test.ts index 3aaf99341d..f6f68ce63f 100644 --- a/packages/cli/src/capture/scaffolding.test.ts +++ b/packages/cli/src/capture/scaffolding.test.ts @@ -206,13 +206,11 @@ describe("loadEnvFile", () => { it("loads supported vision provider settings without replacing inherited credentials", () => { const settings = { - OPENROUTER_API_KEY: "router-key", GOOGLE_API_KEY: "google-key", HYPERFRAMES_VERTEX_PROJECT_ID: "example-project", HYPERFRAMES_VERTEX_SERVICE_ACCOUNT: '{"client_email":"test@example.com"}', HYPERFRAMES_VERTEX_LOCATION: "us-central1", HYPERFRAMES_VERTEX_MODEL: "vertex-model", - HYPERFRAMES_OPENROUTER_MODEL: "router-model", HYPERFRAMES_GEMINI_MODEL: "gemini-model", HYPERFRAMES_VISION_TIMEOUT_MS: "2000", }; diff --git a/packages/cli/src/utils/dotEnv.test.ts b/packages/cli/src/utils/dotEnv.test.ts index 259d0c0c5d..27e0203744 100644 --- a/packages/cli/src/utils/dotEnv.test.ts +++ b/packages/cli/src/utils/dotEnv.test.ts @@ -13,11 +13,8 @@ describe("applyDotEnv", () => { it("reads a comment after an empty value as no value", () => { const env: NodeJS.ProcessEnv = {}; - applyDotEnv( - "GEMINI_API_KEY= # paste your key here\nOPENROUTER_API_KEY=\t# later\nHASH=#abc", - env, - ); - expect(env).toEqual({ GEMINI_API_KEY: "", OPENROUTER_API_KEY: "", HASH: "#abc" }); + applyDotEnv("GEMINI_API_KEY= # paste your key here\nGOOGLE_API_KEY=\t# later\nHASH=#abc", env); + expect(env).toEqual({ GEMINI_API_KEY: "", GOOGLE_API_KEY: "", HASH: "#abc" }); }); it("never takes the feedback email from a project file", () => { @@ -79,7 +76,7 @@ describe("applyDotEnv quoted multiline values", () => { it("a commented-out example with an unclosed quote does not swallow the real keys below it", () => { const env: NodeJS.ProcessEnv = {}; - applyDotEnv('# OPENROUTER_API_KEY="sk-old\nGEMINI_API_KEY=real1\nGOOGLE_API_KEY=real2', env); + applyDotEnv('# FIGMA_TOKEN="sk-old\nGEMINI_API_KEY=real1\nGOOGLE_API_KEY=real2', env); expect(env).toEqual({ GEMINI_API_KEY: "real1", GOOGLE_API_KEY: "real2" }); }); @@ -133,7 +130,7 @@ describe("loadProjectDotEnv", () => { }); it("says nothing when every key is one the CLI reads, or there is no .env", () => { - const dir = projectWith("OPENROUTER_API_KEY=k\n"); + const dir = projectWith("GOOGLE_API_KEY=k\n"); try { const warn = vi.fn(); loadProjectDotEnv(dir, {}, warn); diff --git a/packages/cli/src/utils/dotEnv.ts b/packages/cli/src/utils/dotEnv.ts index 9ea33e5cdb..c4d10c065f 100644 --- a/packages/cli/src/utils/dotEnv.ts +++ b/packages/cli/src/utils/dotEnv.ts @@ -7,14 +7,12 @@ export const PROJECT_ENV_KEYS = [ "HYPERFRAMES_API_KEY", "ELEVENLABS_API_KEY", "FIGMA_TOKEN", - "OPENROUTER_API_KEY", "GEMINI_API_KEY", "GOOGLE_API_KEY", "HYPERFRAMES_VERTEX_PROJECT_ID", "HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", "HYPERFRAMES_VERTEX_LOCATION", "HYPERFRAMES_VERTEX_MODEL", - "HYPERFRAMES_OPENROUTER_MODEL", "HYPERFRAMES_GEMINI_MODEL", "HYPERFRAMES_VISION_TIMEOUT_MS", // Telemetry opt-outs only turn sending off, and a project .env could already set them. diff --git a/skills-manifest.json b/skills-manifest.json index b17fc312f5..5d5d7e72c8 100644 --- a/skills-manifest.json +++ b/skills-manifest.json @@ -6,7 +6,7 @@ "files": 2 }, "hyperframes": { - "hash": "ceaef1d1899f6fb7", + "hash": "8e36e506a04c1bac", "files": 77 }, "hyperframes-cli": { diff --git a/skills/hyperframes/references/film.md b/skills/hyperframes/references/film.md index ae6d515ebe..97e4bc8ed5 100644 --- a/skills/hyperframes/references/film.md +++ b/skills/hyperframes/references/film.md @@ -13,7 +13,7 @@ What is specific to HyperFrames, or was measured on shipped films, when one vide - `npx hyperframes capture --json` writes `screenshots/full-page.png`: a 1x plate of the whole page, taken after the scroll pass with sticky and fixed elements made static, pixel-exact for a 1920-wide viewport travelling down it. It is skipped above 16384 px tall (use `scroll-*.png`); pushing in past 1:1 needs its own 2x capture of that region. - To show the site as it is, use its screenshots. Rebuilding the page in HTML drifts from the real layout; rebuild only the part that moves, or overlay captured assets at measured positions. -- `--capture-budget` (default 120000 ms) bounds work after navigation; `--timeout` bounds navigation only. Videos are not downloaded: `capture --video --list`, then `--index N`. `asset-descriptions.md` has vision captions only when `OPENROUTER_API_KEY`, `GEMINI_API_KEY` or `GOOGLE_API_KEY` is set. +- `--capture-budget` (default 120000 ms) bounds work after navigation; `--timeout` bounds navigation only. Videos are not downloaded: `capture --video --list`, then `--index N`. `asset-descriptions.md` has vision captions only when `GEMINI_API_KEY` or `GOOGLE_API_KEY` is set. - "very little text content" with an empty asset catalog means the capture is unusable even when it says `ok: true`. A `logo-*.svg` name is a guess from page structure, not proof. - Brand tokens: link blue `#0000EE` and status reds and greens are never the accent; the accent is the colour that recurs across the most roles. Icon fonts (Font Awesome, icomoon, webflow-icons) are never the display or body face; a captured monospace goes to the mono role. Use only weights the brand font ships, or 600/700 render as fake bold. Every preset has a light ground: for a dark brand, swap ground and ink. - Capture keeps each `@font-face` rule, `unicode-range` subsets included, in `extracted/page.html`; copying the font files alone loses which characters each covers. Name copied fonts by family, weight and style, or an upright and an italic "Regular" collide.