diff --git a/docs/guides/authentication.mdx b/docs/guides/authentication.mdx index 2b2f38706f..bc849912f9 100644 --- a/docs/guides/authentication.mdx +++ b/docs/guides/authentication.mdx @@ -75,7 +75,7 @@ vision key is available. | **Voice (TTS)** | HeyGen → ElevenLabs → Kokoro | HeyGen credential (above) · then `ELEVENLABS_API_KEY` | Kokoro: `pip install kokoro-onnx soundfile` | | **Music (BGM)** | HeyGen library → Lyria → MusicGen | HeyGen credential (above) · then `GEMINI_API_KEY` → `GOOGLE_API_KEY` | MusicGen: `pip install transformers torch soundfile numpy` | | **Sound effects** | HeyGen library → bundled library | HeyGen credential (above) | bundled — no deps | -| **Capture descriptions** | Gemini | `GEMINI_API_KEY` | None; optional for [website capture](/guides/product-launch-video) | +| **Capture descriptions** | OpenRouter → Gemini | `OPENROUTER_API_KEY` → `GEMINI_API_KEY` | None; optional for [website capture](/guides/product-launch-video) | Run `npx hyperframes doctor` to check which local dependencies are installed. The media workflows run `hyperframes auth status` before generation and tell @@ -154,6 +154,7 @@ to a shared space with `--space`. | `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Explicit Gemini TTS selection and Lyria music generation; capture descriptions use `GEMINI_API_KEY`. | | `GOOGLE_APPLICATION_CREDENTIALS` / `GCS_CREDS` | Gemini TTS service-account JSON file path / injected JSON. Used when neither API key is set. | | `GOOGLE_CLOUD_PROJECT` / `GCLOUD_PROJECT_ID` | Gemini OAuth quota project; first set value wins, then the service account project. | +| `OPENROUTER_API_KEY` | Capture descriptions; takes priority over Gemini for that step. | ## Related topics diff --git a/docs/guides/deploy.mdx b/docs/guides/deploy.mdx index ff2ecfe6ff..74219913db 100644 --- a/docs/guides/deploy.mdx +++ b/docs/guides/deploy.mdx @@ -46,7 +46,7 @@ platform limits. The Cloudflare template is a TanStack Start application on Workers. A `RenderContainer` Durable Object runs the renderer and streams the result to R2. The template also includes optional prompt-based generation with a user's -own model-provider credential. +own OpenRouter credential. [Deploy the Cloudflare template](https://deploy.workers.cloudflare.com/?url=https://github.com/heygen-com/hyperframes-cloudflare-template) diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx index 1b9e93dd10..88e68350ff 100644 --- a/docs/packages/cli.mdx +++ b/docs/packages/cli.mdx @@ -387,7 +387,9 @@ to those local files. Distinct URLs such as `demo.woff2?v=1` and URL unchanged. For AI image descriptions, set `GEMINI_API_KEY` in a `.env` file -(~$0.001/image). +(~$0.001/image), or `OPENROUTER_API_KEY` to route any vision model through +[OpenRouter](https://openrouter.ai) — it wins if both are set, and +`HYPERFRAMES_OPENROUTER_MODEL` overrides the model. #### Local HTML files @@ -2513,7 +2515,7 @@ hyperframes auth logout --yes # no prompt | `HYPERFRAMES_OAUTH_CLIENT_ID` | Override the selected public OAuth client. | For the keys other capabilities use — ElevenLabs and Gemini for voice and music -fallback, Gemini for capture — and how the skills prioritize +fallback, OpenRouter and Gemini for capture — and how the skills prioritize them, see [Authentication and API keys](/guides/authentication). ## hyperframes cloud diff --git a/packages/cli/src/capture/contentExtractor.file-race.test.ts b/packages/cli/src/capture/contentExtractor.file-race.test.ts index 8edf8afa90..e0e974f154 100644 --- a/packages/cli/src/capture/contentExtractor.file-race.test.ts +++ b/packages/cli/src/capture/contentExtractor.file-race.test.ts @@ -4,17 +4,6 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { captionImagesWithGemini } from "./contentExtractor.js"; -const { generateContentMock } = vi.hoisted(() => ({ generateContentMock: vi.fn() })); - -vi.mock("../utils/optionalPackages.js", () => ({ - loadSharp: async () => (await import("sharp")).default, - loadOptionalPackage: vi.fn(async () => ({ - GoogleGenAI: class { - models = { generateContent: generateContentMock }; - }, - })), -})); - const hooks = vi.hoisted(() => ({ afterStat: () => {}, beforeRead: () => {}, @@ -52,6 +41,10 @@ describe("caption image file reads", () => { let dir: string; let file: string; let sentBody: string; + const fetchMock = vi.fn(async (_url: string, init?: RequestInit) => { + sentBody = typeof init?.body === "string" ? init.body : ""; + return new Response(JSON.stringify({ choices: [{ message: { content: "A hero." } }] })); + }); beforeEach(() => { dir = fs.mkdtempSync(join(tmpdir(), "hf-caption-read-")); @@ -63,19 +56,15 @@ describe("caption image file reads", () => { hooks.descriptor = -1; hooks.closed = []; sentBody = ""; - generateContentMock.mockReset(); - generateContentMock.mockImplementation(async (request: unknown) => { - sentBody = JSON.stringify(request); - return { text: "A hero." }; - }); - vi.stubEnv("GEMINI_API_KEY", "test-key"); - vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); + fetchMock.mockClear(); + vi.stubEnv("OPENROUTER_API_KEY", "test-key"); + vi.stubGlobal("fetch", fetchMock); }); afterEach(() => { hooks.afterStat = () => {}; hooks.beforeRead = () => {}; + vi.unstubAllGlobals(); vi.unstubAllEnvs(); fs.rmSync(dir, { recursive: true, force: true }); }); @@ -103,7 +92,7 @@ describe("caption image file reads", () => { hooks.beforeRead = read; const captions = await captionImagesWithGemini(dir, () => {}, []); expect(captions).toEqual(size === 4_000_000 ? { "hero.png": "A hero." } : {}); - expect(generateContentMock).toHaveBeenCalledTimes(size === 4_000_000 ? 1 : 0); + expect(fetchMock).toHaveBeenCalledTimes(size === 4_000_000 ? 1 : 0); expect(read).toHaveBeenCalledTimes(size === 4_000_000 ? 1 : 0); expectClosed(); }); @@ -116,8 +105,8 @@ describe("caption image file reads", () => { else hooks.beforeRead = fail; const warnings: string[] = []; expect(await captionImagesWithGemini(dir, () => {}, warnings)).toEqual({}); - expect(warnings).toEqual(["Gemini vision failed for 1 asset(s); captions omitted."]); - expect(generateContentMock).not.toHaveBeenCalled(); + expect(warnings).toEqual(["OpenRouter vision failed for 1 asset(s); captions omitted."]); + expect(fetchMock).not.toHaveBeenCalled(); expectClosed(); }); }); diff --git a/packages/cli/src/capture/contentExtractor.test.ts b/packages/cli/src/capture/contentExtractor.test.ts index ad6d165ccc..10d7c5e75f 100644 --- a/packages/cli/src/capture/contentExtractor.test.ts +++ b/packages/cli/src/capture/contentExtractor.test.ts @@ -73,10 +73,15 @@ vi.mock("../utils/optionalPackages.js", () => ({ })), })); +// These tests exercise the OpenRouter provider path only — it makes a plain +// `fetch` call we can stub, with no native (`sharp`) or `@google/genai` +// dependency. OpenRouter wins over Gemini when OPENROUTER_API_KEY is set, so we +// don't need to clear the Gemini keys for the OpenRouter cases. + function makeProjectWithImages(files = ["hero.png"]): string { const dir = mkdtempSync(join(tmpdir(), "hf-caption-")); mkdirSync(join(dir, "assets"), { recursive: true }); - // Contents are irrelevant to the provider path (it just base64-encodes the + // Contents are irrelevant to the OpenRouter path (it just base64-encodes the // bytes); only the .png extension matters for the image filter. for (const file of files) { writeFileSync(join(dir, "assets", file), Buffer.from([0x89, 0x50, 0x4e, 0x47])); @@ -84,55 +89,184 @@ function makeProjectWithImages(files = ["hero.png"]): string { return dir; } -describe("captionImagesWithGemini — request and outcome handling", () => { +describe("captionImagesWithGemini — OpenRouter provider", () => { const dirs: string[] = []; afterEach(() => { - generateContentMock.mockReset(); + vi.unstubAllGlobals(); vi.unstubAllEnvs(); for (const d of dirs) rmSync(d, { recursive: true, force: true }); dirs.length = 0; }); - function geminiOnly(): void { - vi.stubEnv("GEMINI_API_KEY", "gemini-test-key"); - vi.stubEnv("GOOGLE_API_KEY", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); - } - - it("captions the image through Gemini with the configured model", async () => { + it("captions via OpenRouter when OPENROUTER_API_KEY is set", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - geminiOnly(); - vi.stubEnv("HYPERFRAMES_GEMINI_MODEL", "gemini-test-model"); - let request: - | { model: string; contents: Array<{ parts: Array<{ inlineData?: unknown }> }> } - | undefined; - generateContentMock.mockImplementation(async (value: typeof request) => { - request = value; - return { text: "A dark hero with blue accents." }; + vi.stubEnv("OPENROUTER_API_KEY", "or-test-key"); + vi.stubEnv("HYPERFRAMES_OPENROUTER_MODEL", "google/gemini-3.1-flash-lite"); + + // Capture the request inside the mock, where the args are well-typed — + // avoids casting `mock.calls` (and the repo's ban on `as` assertions). + let capturedUrl: string | undefined; + let capturedInit: RequestInit | undefined; + const fetchMock = vi.fn(async (url: string, init?: RequestInit) => { + capturedUrl = url; + capturedInit = init; + return new Response( + JSON.stringify({ choices: [{ message: { content: "A dark hero with blue accents." } }] }), + { status: 200, headers: { "content-type": "application/json" } }, + ); }); + vi.stubGlobal("fetch", fetchMock); const warnings: string[] = []; const captions = await captionImagesWithGemini(dir, () => {}, warnings); expect(captions).toEqual({ "hero.png": "A dark hero with blue accents." }); expect(warnings).toEqual([]); - expect(generateContentMock).toHaveBeenCalledTimes(1); - expect(request?.model).toBe("gemini-test-model"); - expect(request?.contents[0]?.parts[0]?.inlineData).toEqual({ - mimeType: "image/png", - data: Buffer.from([0x89, 0x50, 0x4e, 0x47]).toString("base64"), + expect(fetchMock).toHaveBeenCalledTimes(1); + + expect(capturedUrl).toBe("https://openrouter.ai/api/v1/chat/completions"); + expect(new Headers(capturedInit?.headers).get("authorization")).toBe("Bearer or-test-key"); + const body = JSON.parse(typeof capturedInit?.body === "string" ? capturedInit.body : "{}"); + expect(body.model).toBe("google/gemini-3.1-flash-lite"); + const image = body.messages[0].content.find((p: { type: string }) => p.type === "image_url"); + expect(image?.image_url?.url).toMatch(/^data:image\/png;base64,/); + }); + + it("degrades gracefully (no throw, no captions) when OpenRouter returns a non-OK status", async () => { + const dir = makeProjectWithImages(); + dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", "or-bad-key"); + + vi.stubGlobal( + "fetch", + vi.fn( + async () => + new Response("provider detail containing or-bad-key", { + status: 401, + statusText: "Unauthorized", + }), + ), + ); + + const warnings: string[] = []; + let outcome: VisionCaptionOutcome | undefined; + const captions = await captionImagesWithGemini(dir, () => {}, warnings, { + onOutcome: (value) => { + outcome = value; + }, + }); + + expect(captions).toEqual({}); + expect(warnings).toEqual(["OpenRouter vision failed for 1 asset(s); captions omitted."]); + expect(warnings.join(" ")).not.toContain("or-bad-key"); + expect(outcome).toEqual({ + timedOutRequests: 0, + failedRequests: 1, + budgetExhausted: false, + }); + if (!outcome) throw new Error("Expected vision caption outcome"); + expect(resolveVisionPhaseCompletion(outcome, 10_000)).toEqual({ + status: "degraded", + reason: "provider-error", }); }); + it("terminates captioning when the vision provider never responds", async () => { + const dir = makeProjectWithImages(); + dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", "or-hanging-key"); + vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); + + vi.stubGlobal( + "fetch", + vi.fn((_url: string, init?: RequestInit) => { + return new Promise((_resolve, reject) => { + init?.signal?.addEventListener("abort", () => reject(init.signal?.reason), { + once: true, + }); + }); + }), + ); + + const warnings: string[] = []; + const result = await Promise.race([ + captionImagesWithGemini(dir, () => {}, warnings), + new Promise<"hung">((resolve) => setTimeout(() => resolve("hung"), 250)), + ]); + + expect(result).toEqual({}); + }); + + it.each([ + { label: "success JSON", status: 200, partialBody: '{"choices":[' }, + { label: "failure text", status: 500, partialBody: "upstream failed: " }, + ])( + "terminates captioning when OpenRouter $label body never ends", + async ({ status, partialBody }) => { + const dir = makeProjectWithImages(); + dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", "or-stalled-body-key"); + vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); + + vi.stubGlobal( + "fetch", + vi.fn(async () => { + const body = new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode(partialBody)); + }, + }); + return new Response(body, { + status, + headers: { "content-type": "application/json" }, + }); + }), + ); + + const warnings: string[] = []; + let outcome: VisionCaptionOutcome | undefined; + const result = await Promise.race([ + captionImagesWithGemini(dir, () => {}, warnings, { + onOutcome: (value) => { + outcome = value; + }, + }), + new Promise<"hung">((resolve) => setTimeout(() => resolve("hung"), 250)), + ]); + + expect(result).toEqual({}); + expect(warnings).toEqual(["OpenRouter vision timed out for 1 asset(s); captions omitted."]); + expect(outcome).toEqual({ + timedOutRequests: 1, + failedRequests: 0, + budgetExhausted: false, + }); + }, + ); + it("keeps successful captions while reporting a failed sibling request", async () => { const dir = makeProjectWithImages(["failed.png", "success.png"]); dirs.push(dir); - geminiOnly(); - generateContentMock - .mockRejectedValueOnce(new Error("upstream exploded with gemini-test-key")) - .mockResolvedValue({ text: "Successful image." }); + vi.stubEnv("OPENROUTER_API_KEY", "or-mixed-failure-key"); + + let requestCount = 0; + vi.stubGlobal( + "fetch", + vi.fn(async () => { + requestCount++; + if (requestCount === 1) { + return new Response("upstream exploded with or-mixed-failure-key", { + status: 500, + statusText: "Internal Server Error", + }); + } + return new Response( + JSON.stringify({ choices: [{ message: { content: "Successful image." } }] }), + { status: 200, headers: { "content-type": "application/json" } }, + ); + }), + ); const warnings: string[] = []; let outcome: VisionCaptionOutcome | undefined; @@ -143,8 +277,8 @@ describe("captionImagesWithGemini — request and outcome handling", () => { }); expect(captions).toEqual({ "success.png": "Successful image." }); - expect(warnings).toEqual(["Gemini vision failed for 1 asset(s); captions omitted."]); - expect(warnings.join(" ")).not.toContain("gemini-test-key"); + expect(warnings).toEqual(["OpenRouter vision failed for 1 asset(s); captions omitted."]); + expect(warnings.join(" ")).not.toContain("or-mixed-failure-key"); expect(outcome).toEqual({ timedOutRequests: 0, failedRequests: 1, @@ -155,15 +289,35 @@ describe("captionImagesWithGemini — request and outcome handling", () => { it("keeps a successful sibling caption when another request times out", async () => { const dir = makeProjectWithImages(["hang.png", "success.png"]); dirs.push(dir); - geminiOnly(); + vi.stubEnv("OPENROUTER_API_KEY", "or-mixed-key"); vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); - // Both fixtures contain the same bytes, so use call order: the first request hangs. - generateContentMock - .mockImplementationOnce(() => new Promise(() => {})) - .mockResolvedValue({ text: "Successful image." }); + vi.stubGlobal( + "fetch", + vi.fn((_url: string, init?: RequestInit) => { + const body = JSON.parse(typeof init?.body === "string" ? init.body : "{}"); + const imageUrl = body.messages?.[0]?.content?.[1]?.image_url?.url; + if (typeof imageUrl === "string" && imageUrl.includes("iVBORw==")) { + // Both fixtures contain the same bytes, so use call order: the first + // request hangs while the second succeeds independently. + const fetchMock = vi.mocked(fetch); + if (fetchMock.mock.calls.length === 1) return new Promise(() => {}); + } + return Promise.resolve( + new Response( + JSON.stringify({ choices: [{ message: { content: "Successful image." } }] }), + { + status: 200, + headers: { "content-type": "application/json" }, + }, + ), + ); + }), + ); + + const warnings: string[] = []; let outcome: VisionCaptionOutcome | undefined; - const captions = await captionImagesWithGemini(dir, () => {}, [], { + const captions = await captionImagesWithGemini(dir, () => {}, warnings, { onOutcome: (value) => { outcome = value; }, @@ -185,31 +339,35 @@ describe("captionImagesWithGemini — request and outcome handling", () => { it("does not call a configured provider when vision is skipped", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - geminiOnly(); + vi.stubEnv("OPENROUTER_API_KEY", "or-unused-key"); + const fetchMock = vi.fn(); + vi.stubGlobal("fetch", fetchMock); const captions = await captionImagesWithGemini(dir, () => {}, [], { skipVision: true }); expect(captions).toEqual({}); - expect(generateContentMock).not.toHaveBeenCalled(); + expect(fetchMock).not.toHaveBeenCalled(); }); it("does not start a vision batch after the capture budget is exhausted", async () => { const dir = makeProjectWithImages(); dirs.push(dir); - geminiOnly(); + vi.stubEnv("OPENROUTER_API_KEY", "or-budget-key"); + const fetchMock = vi.fn(); + vi.stubGlobal("fetch", fetchMock); const captions = await captionImagesWithGemini(dir, () => {}, [], { remainingMs: () => 0, }); expect(captions).toEqual({}); - expect(generateContentMock).not.toHaveBeenCalled(); + expect(fetchMock).not.toHaveBeenCalled(); }); it("classifies non-provider pipeline failures without leaking the local path", async () => { const dir = mkdtempSync(join(tmpdir(), "hf-caption-no-assets-")); dirs.push(dir); - geminiOnly(); + vi.stubEnv("OPENROUTER_API_KEY", "or-unused-key"); const warnings: string[] = []; let outcome: VisionCaptionOutcome | undefined; @@ -220,13 +378,13 @@ describe("captionImagesWithGemini — request and outcome handling", () => { }); expect(captions).toEqual({}); - expect(warnings).toEqual(["Gemini captioning failed internally; captions omitted."]); + expect(warnings).toEqual(["OpenRouter captioning failed internally; captions omitted."]); expect(warnings.join(" ")).not.toContain(dir); expect(outcome).toEqual({ timedOutRequests: 0, failedRequests: 0, - internalError: true, budgetExhausted: false, + internalError: true, }); if (!outcome) throw new Error("Expected vision caption outcome"); expect(resolveVisionPhaseCompletion(outcome, 10_000)).toEqual({ @@ -238,15 +396,18 @@ describe("captionImagesWithGemini — request and outcome handling", () => { it("skips captioning entirely when no provider key is present", async () => { const dir = makeProjectWithImages(); dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", ""); vi.stubEnv("GOOGLE_API_KEY", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); - const captions = await captionImagesWithGemini(dir, () => {}, []); + const fetchMock = vi.fn(); + vi.stubGlobal("fetch", fetchMock); + + const warnings: string[] = []; + const captions = await captionImagesWithGemini(dir, () => {}, warnings); expect(captions).toEqual({}); - expect(generateContentMock).not.toHaveBeenCalled(); + expect(fetchMock).not.toHaveBeenCalled(); }); }); @@ -263,6 +424,7 @@ describe("captionImagesWithGemini — Gemini provider", () => { it("terminates captioning when Gemini never settles even if the SDK ignores abort", async () => { const dir = makeProjectWithImages(); dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", "gemini-hanging-key"); vi.stubEnv("HYPERFRAMES_VISION_TIMEOUT_MS", "20"); generateContentMock.mockImplementation(() => new Promise(() => {})); @@ -278,6 +440,7 @@ describe("captionImagesWithGemini — Gemini provider", () => { it("reports a rejected Gemini request without leaking its error detail", async () => { const dir = makeProjectWithImages(); dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", "gemini-secret-key"); generateContentMock.mockRejectedValue( new Error("provider echoed gemini-secret-key in its response"), @@ -331,6 +494,7 @@ describe("captionImagesWithGemini — Vertex AI provider", () => { }); function vertexEnv(): void { + vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000"); vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", SERVICE_ACCOUNT); } @@ -421,6 +585,7 @@ describe("captionImagesWithGemini — Vertex AI provider", () => { it("skips captioning when the service account is unparseable, without echoing it", async () => { const dir = makeProjectWithImages(); dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", ""); vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000"); vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", "{not-json super-secret-material"); @@ -445,9 +610,28 @@ describe("captionImagesWithGemini — Vertex AI provider", () => { }); }); + it("stays on OpenRouter when the user has opted into it explicitly", async () => { + const dir = makeProjectWithImages(); + dirs.push(dir); + vertexEnv(); + vi.stubEnv("OPENROUTER_API_KEY", "or-key"); + const fetchMock = vi.fn().mockResolvedValue({ + ok: true, + json: async () => ({ choices: [{ message: { content: "A caption." } }] }), + }); + vi.stubGlobal("fetch", fetchMock); + + await captionImagesWithGemini(dir, () => {}, []); + + expect(fetchMock).toHaveBeenCalled(); + expect(clientOptions).toHaveLength(0); + vi.unstubAllGlobals(); + }); + it("needs both halves of the credential before it will use Vertex", async () => { const dir = makeProjectWithImages(); dirs.push(dir); + vi.stubEnv("OPENROUTER_API_KEY", ""); vi.stubEnv("GEMINI_API_KEY", ""); vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", "prefab-kit-000000"); vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); @@ -499,15 +683,22 @@ describe("captionImagesWithGemini — SVG rasterization", () => { // makes "one at a time" the assertion, not "errors are reported". const dir = makeProjectWithSvgs(6); dirs.push(dir); - vi.stubEnv("GEMINI_API_KEY", "gemini-key"); - vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); - generateContentMock.mockImplementation(async () => { - // Slower than a render, so overlapping renders would be the easy way to go faster — - // the point is that the loop does not take it. - await new Promise((resolve) => setTimeout(resolve, 15)); - return { text: "A dark glyph." }; - }); + vi.stubEnv("OPENROUTER_API_KEY", "or-key"); + vi.stubGlobal( + "fetch", + vi.fn(async () => { + // Slower than a render, so overlapping renders would be the easy way to go faster — + // the point is that the loop does not take it. + await new Promise((resolve) => setTimeout(resolve, 15)); + return new Response( + JSON.stringify({ choices: [{ message: { content: "A dark glyph." } }] }), + { + status: 200, + headers: { "content-type": "application/json" }, + }, + ); + }), + ); const captions = await captionImagesWithGemini(dir, () => {}, []); @@ -523,10 +714,17 @@ describe("captionImagesWithGemini — SVG rasterization", () => { // which asked for captioning, so the bound has to be given back when the renders are done. const dir = makeProjectWithSvgs(1); dirs.push(dir); - vi.stubEnv("GEMINI_API_KEY", "gemini-key"); - vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); - generateContentMock.mockResolvedValue({ text: "A glyph." }); + vi.stubEnv("OPENROUTER_API_KEY", "or-key"); + vi.stubGlobal( + "fetch", + vi.fn( + async () => + new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }), + ), + ); await captionImagesWithGemini(dir, () => {}, []); @@ -539,10 +737,17 @@ describe("captionImagesWithGemini — SVG rasterization", () => { // its threads. const dir = makeProjectWithSvgs(2); dirs.push(dir); - vi.stubEnv("GEMINI_API_KEY", "gemini-key"); - vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); - generateContentMock.mockResolvedValue({ text: "A glyph." }); + vi.stubEnv("OPENROUTER_API_KEY", "or-key"); + vi.stubGlobal( + "fetch", + vi.fn( + async () => + new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }), + ), + ); // Unreadable, which is how an exotic SVG behaves through sharp. rmSync(join(dir, "assets", "svgs", "logo-0.svg")); mkdirSync(join(dir, "assets", "svgs", "logo-0.svg")); @@ -555,10 +760,17 @@ describe("captionImagesWithGemini — SVG rasterization", () => { it("keeps captioning the rest when one SVG cannot be rasterized", async () => { const dir = makeProjectWithSvgs(3); dirs.push(dir); - vi.stubEnv("GEMINI_API_KEY", "gemini-key"); - vi.stubEnv("HYPERFRAMES_VERTEX_PROJECT_ID", ""); - vi.stubEnv("HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", ""); - generateContentMock.mockResolvedValue({ text: "A glyph." }); + vi.stubEnv("OPENROUTER_API_KEY", "or-key"); + vi.stubGlobal( + "fetch", + vi.fn( + async () => + new Response(JSON.stringify({ choices: [{ message: { content: "A glyph." } }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }), + ), + ); // One file is unreadable, which is how an exotic SVG behaves through sharp. rmSync(join(dir, "assets", "svgs", "logo-1.svg")); mkdirSync(join(dir, "assets", "svgs", "logo-1.svg")); diff --git a/packages/cli/src/capture/contentExtractor.ts b/packages/cli/src/capture/contentExtractor.ts index baf2472eef..10b5512cc1 100644 --- a/packages/cli/src/capture/contentExtractor.ts +++ b/packages/cli/src/capture/contentExtractor.ts @@ -245,8 +245,9 @@ export async function extractVisibleText(page: Page): Promise { /** * Caption downloaded images using a vision model. * - * Provider is chosen by which credential is present: Vertex service account, else - * GEMINI_API_KEY/GOOGLE_API_KEY → Google Gemini, else no captioning. + * Provider is chosen by which API key is present: OPENROUTER_API_KEY → OpenRouter + * (any vision model via its OpenAI-style API), else GEMINI_API_KEY/GOOGLE_API_KEY + * → Google Gemini, else no captioning. OpenRouter wins if both are set. * * Batches requests to stay under free-tier rate limits. * Returns a map of filename -> caption string. @@ -276,6 +277,7 @@ export async function captionImagesWithGemini( reportOutcome(); return geminiCaptions; } + const openRouterKey = process.env.OPENROUTER_API_KEY; const geminiKey = process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY; // Vertex authenticates with a service account and a project instead of an API key. Server // deployments have those; they often do not have a working Gemini API key, and a rejected @@ -284,20 +286,28 @@ export async function captionImagesWithGemini( const vertexProject = process.env.HYPERFRAMES_VERTEX_PROJECT_ID; const vertexServiceAccount = process.env.HYPERFRAMES_VERTEX_SERVICE_ACCOUNT; const useVertex = Boolean(vertexProject && vertexServiceAccount); - if (!useVertex && !geminiKey) { + if (!openRouterKey && !useVertex && !geminiKey) { reportOutcome(); return geminiCaptions; } + // OpenRouter takes priority — it's the explicit opt-in for users without Google access. // Vertex outranks the bare API key because it is the credential a deployment actually - // holds. Both satisfy the same single-image → one-line-caption contract + // holds. All three satisfy the same single-image → one-line-caption contract // (`captionOne`), so the batching and SVG-rasterization loops stay provider-agnostic. - const provider: "vertex" | "gemini" = useVertex ? "vertex" : "gemini"; - const providerName = { vertex: "Vertex AI", gemini: "Gemini" }[provider]; - // Override per provider via HYPERFRAMES_VERTEX_MODEL / HYPERFRAMES_GEMINI_MODEL. Vertex - // publishes a different model set than the Gemini API (the API's flash-lite preview id is - // not resolvable there), so it carries its own default. + const provider: "openrouter" | "vertex" | "gemini" = openRouterKey + ? "openrouter" + : useVertex + ? "vertex" + : "gemini"; + const providerName = { openrouter: "OpenRouter", vertex: "Vertex AI", gemini: "Gemini" }[ + provider + ]; + // Override per provider via HYPERFRAMES_OPENROUTER_MODEL / HYPERFRAMES_VERTEX_MODEL / + // HYPERFRAMES_GEMINI_MODEL. Vertex publishes a different model set than the Gemini API — + // the API's flash-lite preview id is not resolvable there — so it carries its own default. const model = { + openrouter: process.env.HYPERFRAMES_OPENROUTER_MODEL || "google/gemini-3.1-flash-lite", vertex: process.env.HYPERFRAMES_VERTEX_MODEL || "gemini-2.5-flash", gemini: process.env.HYPERFRAMES_GEMINI_MODEL || "gemini-3.1-flash-lite-preview", }[provider]; @@ -316,72 +326,111 @@ export async function captionImagesWithGemini( }) => Promise; let captionOne: CaptionOne; - let GoogleGenAI: OptionalPackageModules["@google/genai"]["GoogleGenAI"]; - try { - ({ GoogleGenAI } = await loadOptionalPackage("@google/genai", "vision captioning")); - } catch (err) { - warnings.push(`Skipped vision captioning: ${(err as Error).message}`); - internalError = true; - reportOutcome(); - return geminiCaptions; - } - let ai: InstanceType; - if (provider === "vertex") { - // Re-narrow for TS; `useVertex` already guaranteed both are set. - if (!vertexProject || !vertexServiceAccount) return geminiCaptions; - let credentials: Record; + if (provider === "openrouter") { + captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => { + return runBoundedVisionRequest(async (signal) => { + const res = await fetch("https://openrouter.ai/api/v1/chat/completions", { + method: "POST", + headers: { + Authorization: `Bearer ${openRouterKey}`, + "Content-Type": "application/json", + }, + signal, + body: JSON.stringify({ + model, + messages: [ + { + role: "user", + content: [ + { type: "text", text: prompt }, + { + type: "image_url", + image_url: { url: `data:${mimeType};base64,${base64}` }, + }, + ], + }, + ], + max_tokens: maxTokens, + }), + }); + if (!res.ok) { + await res.text(); + throw new Error(`OpenRouter request failed with HTTP ${res.status}`); + } + const data = (await res.json()) as { + choices?: Array<{ message?: { content?: string } }>; + }; + return data.choices?.[0]?.message?.content?.trim() || ""; + }, timeoutMs); + }; + } else { + let GoogleGenAI: OptionalPackageModules["@google/genai"]["GoogleGenAI"]; try { - credentials = JSON.parse(vertexServiceAccount) as Record; - } catch { - warnings.push( - "HYPERFRAMES_VERTEX_SERVICE_ACCOUNT is not valid JSON; skipped vision captioning.", - ); + ({ GoogleGenAI } = await loadOptionalPackage("@google/genai", "vision captioning")); + } catch (err) { + warnings.push(`Skipped vision captioning: ${(err as Error).message}`); internalError = true; reportOutcome(); return geminiCaptions; } - ai = new GoogleGenAI({ - vertexai: true, - project: vertexProject, - location: process.env.HYPERFRAMES_VERTEX_LOCATION || "us-central1", - googleAuthOptions: { credentials }, - }); - } else { - // Unreachable when geminiKey is unset (guarded above); re-narrow for TS. - if (!geminiKey) return geminiCaptions; - ai = new GoogleGenAI({ apiKey: geminiKey }); - } - captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => { - const response = await runBoundedVisionRequest( - (signal) => - ai.models.generateContent({ - model, - contents: [ - { - role: "user", - parts: [{ inlineData: { mimeType, data: base64 } }, { text: prompt }], + let ai: InstanceType; + if (provider === "vertex") { + // Re-narrow for TS; `useVertex` already guaranteed both are set. + if (!vertexProject || !vertexServiceAccount) return geminiCaptions; + let credentials: Record; + try { + credentials = JSON.parse(vertexServiceAccount) as Record; + } catch { + warnings.push( + "HYPERFRAMES_VERTEX_SERVICE_ACCOUNT is not valid JSON; skipped vision captioning.", + ); + internalError = true; + reportOutcome(); + return geminiCaptions; + } + ai = new GoogleGenAI({ + vertexai: true, + project: vertexProject, + location: process.env.HYPERFRAMES_VERTEX_LOCATION || "us-central1", + googleAuthOptions: { credentials }, + }); + } else { + // Unreachable when geminiKey is unset (guarded above); re-narrow for TS. + if (!geminiKey) return geminiCaptions; + ai = new GoogleGenAI({ apiKey: geminiKey }); + } + captionOne = async ({ mimeType, base64, prompt, maxTokens, timeoutMs }) => { + const response = await runBoundedVisionRequest( + (signal) => + ai.models.generateContent({ + model, + contents: [ + { + role: "user", + parts: [{ inlineData: { mimeType, data: base64 } }, { text: prompt }], + }, + ], + config: { + maxOutputTokens: maxTokens, + // Thinking tokens are drawn from maxOutputTokens, so the model can spend the + // whole budget and return a successful request with no caption text. + thinkingConfig: { thinkingBudget: 0 }, + abortSignal: signal, + httpOptions: { timeout: timeoutMs }, }, - ], - config: { - maxOutputTokens: maxTokens, - // Thinking tokens come out of maxOutputTokens, so a thinking model can spend the - // whole budget and return empty text: a successful request with no caption. - thinkingConfig: { thinkingBudget: 0 }, - abortSignal: signal, - httpOptions: { timeout: timeoutMs }, - }, - }), - timeoutMs, - ); - return response.text?.trim() || ""; - }; + }), + timeoutMs, + ); + return response.text?.trim() || ""; + }; + } const imageFiles = readdirSync(join(outputDir, "assets")).filter((f: string) => /\.(png|jpg|jpeg|webp|gif)$/i.test(f), ); // Caption in parallel batches. Gemini free tier is ~5 RPM (slow but $0), - // paid ~2000 RPM. We batch 20 with a 2s inter-batch pause and rely + // paid/OpenRouter ~2000 RPM. We batch 20 with a 2s inter-batch pause and rely // on Promise.allSettled so one rate-limited image does not fail its siblings; // rejected requests are aggregated into a sanitized warning below. const BATCH_SIZE = 20; diff --git a/packages/cli/src/capture/postExtractionPhase.ts b/packages/cli/src/capture/postExtractionPhase.ts index 26b7fb9e6c..8733462c53 100644 --- a/packages/cli/src/capture/postExtractionPhase.ts +++ b/packages/cli/src/capture/postExtractionPhase.ts @@ -34,7 +34,8 @@ import type { VisionCaptionOutcome } from "./contentExtractor.js"; function hasVisionCredentials(skipVision: boolean): boolean { return Boolean( !skipVision && - (process.env.GEMINI_API_KEY || + (process.env.OPENROUTER_API_KEY || + process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY || (process.env.HYPERFRAMES_VERTEX_PROJECT_ID && process.env.HYPERFRAMES_VERTEX_SERVICE_ACCOUNT)), diff --git a/packages/cli/src/capture/scaffolding.test.ts b/packages/cli/src/capture/scaffolding.test.ts index f6f68ce63f..3aaf99341d 100644 --- a/packages/cli/src/capture/scaffolding.test.ts +++ b/packages/cli/src/capture/scaffolding.test.ts @@ -206,11 +206,13 @@ describe("loadEnvFile", () => { it("loads supported vision provider settings without replacing inherited credentials", () => { const settings = { + OPENROUTER_API_KEY: "router-key", GOOGLE_API_KEY: "google-key", HYPERFRAMES_VERTEX_PROJECT_ID: "example-project", HYPERFRAMES_VERTEX_SERVICE_ACCOUNT: '{"client_email":"test@example.com"}', HYPERFRAMES_VERTEX_LOCATION: "us-central1", HYPERFRAMES_VERTEX_MODEL: "vertex-model", + HYPERFRAMES_OPENROUTER_MODEL: "router-model", HYPERFRAMES_GEMINI_MODEL: "gemini-model", HYPERFRAMES_VISION_TIMEOUT_MS: "2000", }; diff --git a/packages/cli/src/utils/dotEnv.test.ts b/packages/cli/src/utils/dotEnv.test.ts index 27e0203744..259d0c0c5d 100644 --- a/packages/cli/src/utils/dotEnv.test.ts +++ b/packages/cli/src/utils/dotEnv.test.ts @@ -13,8 +13,11 @@ describe("applyDotEnv", () => { it("reads a comment after an empty value as no value", () => { const env: NodeJS.ProcessEnv = {}; - applyDotEnv("GEMINI_API_KEY= # paste your key here\nGOOGLE_API_KEY=\t# later\nHASH=#abc", env); - expect(env).toEqual({ GEMINI_API_KEY: "", GOOGLE_API_KEY: "", HASH: "#abc" }); + applyDotEnv( + "GEMINI_API_KEY= # paste your key here\nOPENROUTER_API_KEY=\t# later\nHASH=#abc", + env, + ); + expect(env).toEqual({ GEMINI_API_KEY: "", OPENROUTER_API_KEY: "", HASH: "#abc" }); }); it("never takes the feedback email from a project file", () => { @@ -76,7 +79,7 @@ describe("applyDotEnv quoted multiline values", () => { it("a commented-out example with an unclosed quote does not swallow the real keys below it", () => { const env: NodeJS.ProcessEnv = {}; - applyDotEnv('# FIGMA_TOKEN="sk-old\nGEMINI_API_KEY=real1\nGOOGLE_API_KEY=real2', env); + applyDotEnv('# OPENROUTER_API_KEY="sk-old\nGEMINI_API_KEY=real1\nGOOGLE_API_KEY=real2', env); expect(env).toEqual({ GEMINI_API_KEY: "real1", GOOGLE_API_KEY: "real2" }); }); @@ -130,7 +133,7 @@ describe("loadProjectDotEnv", () => { }); it("says nothing when every key is one the CLI reads, or there is no .env", () => { - const dir = projectWith("GOOGLE_API_KEY=k\n"); + const dir = projectWith("OPENROUTER_API_KEY=k\n"); try { const warn = vi.fn(); loadProjectDotEnv(dir, {}, warn); diff --git a/packages/cli/src/utils/dotEnv.ts b/packages/cli/src/utils/dotEnv.ts index c4d10c065f..9ea33e5cdb 100644 --- a/packages/cli/src/utils/dotEnv.ts +++ b/packages/cli/src/utils/dotEnv.ts @@ -7,12 +7,14 @@ export const PROJECT_ENV_KEYS = [ "HYPERFRAMES_API_KEY", "ELEVENLABS_API_KEY", "FIGMA_TOKEN", + "OPENROUTER_API_KEY", "GEMINI_API_KEY", "GOOGLE_API_KEY", "HYPERFRAMES_VERTEX_PROJECT_ID", "HYPERFRAMES_VERTEX_SERVICE_ACCOUNT", "HYPERFRAMES_VERTEX_LOCATION", "HYPERFRAMES_VERTEX_MODEL", + "HYPERFRAMES_OPENROUTER_MODEL", "HYPERFRAMES_GEMINI_MODEL", "HYPERFRAMES_VISION_TIMEOUT_MS", // Telemetry opt-outs only turn sending off, and a project .env could already set them. diff --git a/skills-manifest.json b/skills-manifest.json index 5d5d7e72c8..b17fc312f5 100644 --- a/skills-manifest.json +++ b/skills-manifest.json @@ -6,7 +6,7 @@ "files": 2 }, "hyperframes": { - "hash": "8e36e506a04c1bac", + "hash": "ceaef1d1899f6fb7", "files": 77 }, "hyperframes-cli": { diff --git a/skills/hyperframes/references/film.md b/skills/hyperframes/references/film.md index 97e4bc8ed5..ae6d515ebe 100644 --- a/skills/hyperframes/references/film.md +++ b/skills/hyperframes/references/film.md @@ -13,7 +13,7 @@ What is specific to HyperFrames, or was measured on shipped films, when one vide - `npx hyperframes capture --json` writes `screenshots/full-page.png`: a 1x plate of the whole page, taken after the scroll pass with sticky and fixed elements made static, pixel-exact for a 1920-wide viewport travelling down it. It is skipped above 16384 px tall (use `scroll-*.png`); pushing in past 1:1 needs its own 2x capture of that region. - To show the site as it is, use its screenshots. Rebuilding the page in HTML drifts from the real layout; rebuild only the part that moves, or overlay captured assets at measured positions. -- `--capture-budget` (default 120000 ms) bounds work after navigation; `--timeout` bounds navigation only. Videos are not downloaded: `capture --video --list`, then `--index N`. `asset-descriptions.md` has vision captions only when `GEMINI_API_KEY` or `GOOGLE_API_KEY` is set. +- `--capture-budget` (default 120000 ms) bounds work after navigation; `--timeout` bounds navigation only. Videos are not downloaded: `capture --video --list`, then `--index N`. `asset-descriptions.md` has vision captions only when `OPENROUTER_API_KEY`, `GEMINI_API_KEY` or `GOOGLE_API_KEY` is set. - "very little text content" with an empty asset catalog means the capture is unusable even when it says `ok: true`. A `logo-*.svg` name is a guess from page structure, not proof. - Brand tokens: link blue `#0000EE` and status reds and greens are never the accent; the accent is the colour that recurs across the most roles. Icon fonts (Font Awesome, icomoon, webflow-icons) are never the display or body face; a captured monospace goes to the mono role. Use only weights the brand font ships, or 600/700 render as fake bold. Every preset has a light ground: for a dark brand, swap ground and ink. - Capture keeps each `@font-face` rule, `unicode-range` subsets included, in `extracted/page.html`; copying the font files alone loses which characters each covers. Name copied fonts by family, weight and style, or an upright and an italic "Regular" collide.