mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
fix(producer): localize remote <img> sources + await image readiness (#1197)
* fix(producer): localize remote <img> sources + await image readiness Producer's frame-capture has `pollVideosReady` (waits readyState >= 2 for every <video>) but no equivalent for <img>. Combined with htmlCompiler's `collectExternalAssets` explicitly skipping http(s) URLs (line 805-806), agent-pipeline-generated compositions (astral / daphne / hyperion multi-v2 outputs with raw S3 <img src>) reach Chrome with a network dependency that races the readiness gate AND can be evicted mid-render. Either path produces blank-frame flicker. Reproduction (02_kobe agent output, 42s render @ 30fps): scene_02's remote S3 background-image painted from t=7.0s, vanished at t=10.5s (frame size 139KB vs 700-940KB neighbors), back at t=11.0s. GSAP timeline said opacity:1 throughout — Chrome simply didn't have the pixels. Two-layer fix: 1. **Producer** — `localizeRemoteImageSources` in `htmlCompiler.ts` mirrors the existing `localizeRemoteMediaSources` (video/audio) + `localizeRemoteFontFaces` pattern, reusing `downloadAndRewriteUrls` and the `_remote_media/` subdir. Wired into `compileForRender` between the media and font localize steps. Once the file is local, Chrome's image cache is bounded by disk reads, not S3 latency. 2. **Engine** — `pollImagesReady` + `decodeAllImages` helpers in `frameCapture.ts` parallel to `pollVideosReady`. Waits for every `<img>` (skipping data: URIs) to have `complete && naturalWidth > 0`, then forces GPU upload via `img.decode()`. Called from both the classic-xvfb path and the BeginFrame path after their respective video readiness checks. Defense-in-depth — Layer 1 closes the symptom for current+future agent-pipeline outputs; Layer 2 protects any future code path that leaves a remote URL in place. Tests: 7 new cases in `htmlCompiler.test.ts` covering happy-path rewrite, 404 fallback, dedup of duplicate URLs, non-HTTP and data: URI passthrough, both quote styles, and the agent-pipeline shape where `src` is not the first attribute. All pass alongside the existing 56 htmlCompiler tests. * fix(producer): scope remote-img regex to real src; correct stale comments Review follow-ups on the remote-<img> localization fix: - Tighten REMOTE_IMG_TAG_RE with a (?<![\w-]) lookbehind so it matches a real `src` attribute only. The previous `\bsrc` also matched `data-src` (and `data-*-src`) lazy-loader placeholders, which would download/rewrite a URL the render never paints. Added a regression test; `srcset` stays excluded by the `\s*=` requirement. - Fix comments that claimed frameCapture has "no pollImagesReady analog" — this PR adds exactly that, so the docstrings were self-contradictory. Reframed localization as the primary fix and pollImagesReady as the defense-in-depth layer, and documented the <img src>-only scope (srcset / <picture> / SVG <image> / CSS background-image are follow-ups). Verified locally end-to-end on the 02_kobe repro: all 4 remote S3 <img> URLs localize to _remote_media/, the render completes, and the frame at t~10.5s that was a 139KB blank in the broken render now paints the trophy background in every native-fps frame. htmlCompiler.test.ts 64 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(engine): pollImagesReady broken-image escape + skip decode on in-flight Addresses two real bugs Magi caught in review on hf#1197: 1. pollImagesReady would spin the full pageReadyTimeout (45s default) for any <img> that settled with an error — Chrome marks 404 / decode failure / CORS rejection with (complete=true, naturalWidth=0), and the previous predicate `complete && naturalWidth > 0` returned false for those, so the poll ran to timeout. This is the HTMLImageElement equivalent of pollVideosReady's `ve.error` early-exit. Add a `complete && naturalWidth === 0` branch that treats settled-with- error as done — waiting won't make it load. Particularly relevant because localizeRemoteImageSources falls back to the original URL on download failure; that failed URL is now hit by a 45s stall instead of the broken-image marker rendering immediately. 2. decodeAllImages called img.decode() on every image, including those still in flight after pollImagesReady timed out. Per the WHATWG spec, decode() on a loading image awaits the fetch — never resolving until the network completes or puppeteer's evaluate timeout fires and throws an uncaught error that aborts the render. Pre-filter to only call decode() on images that successfully loaded. Test coverage: new frameCapture-pollImagesReady.test.ts with 8 cases covering empty docs, all-loaded, broken (complete + naturalWidth=0), data: URI, empty src, in-flight → resolves, in-flight → timeout, and the mixed batch. The broken-image test explicitly asserts elapsed < 500ms on a 1000ms timeout — guards against the regression Magi flagged. * docs(engine): clarify decodeAllImages prevents init race, not eviction Vai correctly noted that decode() forces initial GPU upload but does not prevent Chrome from evicting decoded pixels mid-render. The producer-side localizeRemoteImageSources is what bounds the eviction risk (local file-server paging vs S3 re-fetch). Comment updated to reflect that split of responsibilities. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
2be41937a9
commit
72c461d86a
@@ -0,0 +1,153 @@
|
|||||||
|
/**
|
||||||
|
* Tests for `pollImagesReady` — the image-side analog of `pollVideosReady`.
|
||||||
|
*
|
||||||
|
* Critical contract:
|
||||||
|
* - Successfully loaded image (complete=true, naturalWidth>0) → settled.
|
||||||
|
* - Broken / 404 image (complete=true, naturalWidth=0) → settled.
|
||||||
|
* This mirrors `pollVideosReady`'s `ve.error` early-exit. Without it,
|
||||||
|
* the htmlCompiler 404-fallback path (where a remote <img> URL failed
|
||||||
|
* to download and the original URL is preserved) would silently spin
|
||||||
|
* the full `pageReadyTimeout` budget waiting for an image that will
|
||||||
|
* never load — a 45 s regression vs the pre-PR behavior.
|
||||||
|
* - In-flight image (complete=false) → still waiting.
|
||||||
|
* - data: URI src → settled (no network fetch).
|
||||||
|
* - Empty src → settled (nothing to load).
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import type { Page } from "puppeteer-core";
|
||||||
|
import { pollImagesReady } from "./frameCapture.js";
|
||||||
|
|
||||||
|
interface ImageSpec {
|
||||||
|
src: string;
|
||||||
|
complete: boolean;
|
||||||
|
naturalWidth: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mock `page` whose `evaluate(fn)` invokes `fn` with a Node-side `document`
|
||||||
|
// mock that returns synthetic image objects matching the spec. Snapshots the
|
||||||
|
// image state at evaluate-time, so callers can mutate `imgs` between polls
|
||||||
|
// to simulate progressive load completion.
|
||||||
|
function makeMockPage(imgs: () => ImageSpec[]): Page {
|
||||||
|
return {
|
||||||
|
evaluate: async (fn: () => unknown) => {
|
||||||
|
const snapshot = imgs();
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
const prevDoc = (globalThis as any).document;
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).document = {
|
||||||
|
querySelectorAll: () =>
|
||||||
|
snapshot.map((spec) => ({
|
||||||
|
getAttribute: (attr: string) => (attr === "src" ? spec.src : null),
|
||||||
|
complete: spec.complete,
|
||||||
|
naturalWidth: spec.naturalWidth,
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
return await fn();
|
||||||
|
} finally {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).document = prevDoc;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
} as any as Page;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("pollImagesReady", () => {
|
||||||
|
it("resolves immediately when there are no <img> elements", async () => {
|
||||||
|
const page = makeMockPage(() => []);
|
||||||
|
const result = await pollImagesReady(page, 1000, 10);
|
||||||
|
expect(result).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("resolves immediately when every image has loaded successfully", async () => {
|
||||||
|
const page = makeMockPage(() => [
|
||||||
|
{ src: "/a.png", complete: true, naturalWidth: 100 },
|
||||||
|
{ src: "/b.png", complete: true, naturalWidth: 200 },
|
||||||
|
]);
|
||||||
|
const result = await pollImagesReady(page, 1000, 10);
|
||||||
|
expect(result).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats a broken image (complete=true, naturalWidth=0) as settled — does NOT wait for timeout", async () => {
|
||||||
|
// This is the bug Magi flagged. Without the broken-image escape, this
|
||||||
|
// test would block the full 1000ms timeout and return false.
|
||||||
|
const page = makeMockPage(() => [
|
||||||
|
{ src: "/a.png", complete: true, naturalWidth: 100 },
|
||||||
|
{ src: "https://broken.example.com/404.png", complete: true, naturalWidth: 0 },
|
||||||
|
]);
|
||||||
|
const t0 = Date.now();
|
||||||
|
const result = await pollImagesReady(page, 1000, 10);
|
||||||
|
const elapsed = Date.now() - t0;
|
||||||
|
expect(result).toBe(true);
|
||||||
|
// Must resolve fast — well under the 1000ms timeout.
|
||||||
|
expect(elapsed).toBeLessThan(500);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats a data: URI src as settled regardless of complete/naturalWidth", async () => {
|
||||||
|
const page = makeMockPage(() => [
|
||||||
|
{ src: "data:image/svg+xml,%3Csvg/%3E", complete: false, naturalWidth: 0 },
|
||||||
|
]);
|
||||||
|
const result = await pollImagesReady(page, 1000, 10);
|
||||||
|
expect(result).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats an empty src as settled (nothing to load)", async () => {
|
||||||
|
const page = makeMockPage(() => [{ src: "", complete: false, naturalWidth: 0 }]);
|
||||||
|
const result = await pollImagesReady(page, 1000, 10);
|
||||||
|
expect(result).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("waits for an in-flight image and resolves once it completes", async () => {
|
||||||
|
// Image starts in-flight, then completes after ~50ms.
|
||||||
|
let started = false;
|
||||||
|
const startTime = { value: 0 };
|
||||||
|
const page = makeMockPage(() => {
|
||||||
|
if (!started) {
|
||||||
|
started = true;
|
||||||
|
startTime.value = Date.now();
|
||||||
|
}
|
||||||
|
const elapsed = Date.now() - startTime.value;
|
||||||
|
const loaded = elapsed >= 50;
|
||||||
|
return [{ src: "/slow.png", complete: loaded, naturalWidth: loaded ? 100 : 0 }];
|
||||||
|
});
|
||||||
|
const result = await pollImagesReady(page, 1000, 10);
|
||||||
|
expect(result).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("times out and returns false when an in-flight image never resolves", async () => {
|
||||||
|
// Image stays in-flight (complete=false) for the full timeout.
|
||||||
|
const page = makeMockPage(() => [
|
||||||
|
{ src: "/never-loads.png", complete: false, naturalWidth: 0 },
|
||||||
|
]);
|
||||||
|
const result = await pollImagesReady(page, 100, 10);
|
||||||
|
expect(result).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("mixed batch: loaded + broken + data: + in-flight → waits only on the in-flight image", async () => {
|
||||||
|
let resolved = false;
|
||||||
|
const start = Date.now();
|
||||||
|
const page = makeMockPage(() => {
|
||||||
|
const elapsed = Date.now() - start;
|
||||||
|
if (elapsed >= 30) resolved = true;
|
||||||
|
return [
|
||||||
|
{ src: "/loaded.png", complete: true, naturalWidth: 800 },
|
||||||
|
{ src: "https://broken.example.com/404.jpg", complete: true, naturalWidth: 0 },
|
||||||
|
{ src: "data:image/svg+xml,abc", complete: false, naturalWidth: 0 },
|
||||||
|
{
|
||||||
|
src: "/in-flight.png",
|
||||||
|
complete: resolved,
|
||||||
|
naturalWidth: resolved ? 200 : 0,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
});
|
||||||
|
const t0 = Date.now();
|
||||||
|
const result = await pollImagesReady(page, 1000, 10);
|
||||||
|
const elapsed = Date.now() - t0;
|
||||||
|
expect(result).toBe(true);
|
||||||
|
// Should wait roughly for the in-flight image to settle (~30ms) — not the
|
||||||
|
// full timeout. Allow generous slack for CI scheduler jitter.
|
||||||
|
expect(elapsed).toBeLessThan(500);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -550,6 +550,89 @@ async function pollVideosReady(
|
|||||||
return check();
|
return check();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Wait for every `<img>` with a non-`data:` src to have settled — either
|
||||||
|
// successfully loaded (`complete && naturalWidth > 0`) or failed with a
|
||||||
|
// broken-image marker (`complete && naturalWidth === 0`, the HTMLImageElement
|
||||||
|
// equivalent of HTMLMediaElement.error). htmlCompiler localises remote `<img>`
|
||||||
|
// URLs to the local file server before this point, so in practice this polls
|
||||||
|
// for the local fetch to land — but the guard is a defensive net so that any
|
||||||
|
// future composition path that leaves a remote URL in place won't capture
|
||||||
|
// frames before the pixels arrive. Mirrors `pollVideosReady` for parity with
|
||||||
|
// the video-side readiness contract (videos exit-early on `ve.error`; images
|
||||||
|
// exit-early on `complete && naturalWidth === 0`).
|
||||||
|
/** @internal exported for unit testing only */
|
||||||
|
export async function pollImagesReady(
|
||||||
|
page: Page,
|
||||||
|
timeoutMs: number,
|
||||||
|
intervalMs: number = 100,
|
||||||
|
): Promise<boolean> {
|
||||||
|
const check = async (): Promise<boolean> => {
|
||||||
|
return Boolean(
|
||||||
|
await page.evaluate(() => {
|
||||||
|
const imgs = Array.from(document.querySelectorAll("img"));
|
||||||
|
return (
|
||||||
|
imgs.length === 0 ||
|
||||||
|
imgs.every((img) => {
|
||||||
|
const ie = img as HTMLImageElement;
|
||||||
|
const src = ie.getAttribute("src") || "";
|
||||||
|
if (!src || src.startsWith("data:")) return true;
|
||||||
|
// A `complete` image with zero naturalWidth has settled with an
|
||||||
|
// error (404 / decode failure / CORS rejection / blocked). Treat
|
||||||
|
// as done — waiting won't make it load — and let the render
|
||||||
|
// continue with the broken-image marker visible. Mirrors how
|
||||||
|
// pollVideosReady treats `ve.error`.
|
||||||
|
if (ie.complete && ie.naturalWidth === 0) return true;
|
||||||
|
if (ie.complete && ie.naturalWidth > 0) return true;
|
||||||
|
return false;
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
};
|
||||||
|
const deadline = Date.now() + timeoutMs;
|
||||||
|
while (Date.now() < deadline) {
|
||||||
|
if (await check()) return true;
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, intervalMs));
|
||||||
|
}
|
||||||
|
return check();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Force every successfully-loaded `<img>` to be GPU-uploaded before the first
|
||||||
|
// frame capture. `naturalWidth > 0` means the bitmap has been decoded into
|
||||||
|
// CPU memory, but compositor-side GPU upload can still happen lazily on first
|
||||||
|
// paint. Calling `img.decode()` returns a Promise that resolves once the image
|
||||||
|
// is ready for synchronous painting — eliminating the small first-frame race
|
||||||
|
// between "image is technically loaded" and "the rasterized texture is on the
|
||||||
|
// GPU and ready to composite".
|
||||||
|
//
|
||||||
|
// Note this is purely an init-time guard; it doesn't prevent Chrome from
|
||||||
|
// evicting decoded pixels mid-render. The producer-side `localizeRemoteImageSources`
|
||||||
|
// is what bounds the eviction risk (a re-fetch hits the local file server's
|
||||||
|
// disk-backed paging, not S3 over the network).
|
||||||
|
//
|
||||||
|
// Critical: `decode()` on an in-flight image waits for the fetch to resolve.
|
||||||
|
// If `pollImagesReady` timed out with some images still loading (`!complete`),
|
||||||
|
// calling `decode()` on them would block here until the network finally
|
||||||
|
// completes — or until puppeteer's evaluate timeout fires and throws an
|
||||||
|
// uncaught error that aborts the render. Skip in-flight and broken images;
|
||||||
|
// only force GPU upload for images that successfully loaded.
|
||||||
|
async function decodeAllImages(page: Page): Promise<void> {
|
||||||
|
await page.evaluate(async () => {
|
||||||
|
const imgs = Array.from(document.querySelectorAll("img"));
|
||||||
|
await Promise.all(
|
||||||
|
imgs.map((img) => {
|
||||||
|
const ie = img as HTMLImageElement;
|
||||||
|
if (typeof ie.decode !== "function") return Promise.resolve();
|
||||||
|
// Skip still-loading images (in-flight decode() would hang) and
|
||||||
|
// broken images (decode() rejects, but pre-filtering is clearer
|
||||||
|
// than relying on the .catch).
|
||||||
|
if (!ie.complete || ie.naturalWidth === 0) return Promise.resolve();
|
||||||
|
return ie.decode().catch(() => undefined);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
async function applyVideoMetadataHints(
|
async function applyVideoMetadataHints(
|
||||||
page: Page,
|
page: Page,
|
||||||
hints: readonly CaptureVideoMetadataHint[] | undefined,
|
hints: readonly CaptureVideoMetadataHint[] | undefined,
|
||||||
@@ -707,6 +790,26 @@ export async function initializeSession(session: CaptureSession): Promise<void>
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const imagesReady = await pollImagesReady(page, pageReadyTimeout);
|
||||||
|
if (!imagesReady) {
|
||||||
|
const failedImages = await page.evaluate(() => {
|
||||||
|
return Array.from(document.querySelectorAll("img"))
|
||||||
|
.filter((img) => {
|
||||||
|
const ie = img as HTMLImageElement;
|
||||||
|
const src = ie.getAttribute("src") || "";
|
||||||
|
if (!src || src.startsWith("data:")) return false;
|
||||||
|
return !(ie.complete && ie.naturalWidth > 0);
|
||||||
|
})
|
||||||
|
.map((img) => (img as HTMLImageElement).src || img.getAttribute("src") || "(no src)")
|
||||||
|
.join(", ");
|
||||||
|
});
|
||||||
|
console.warn(
|
||||||
|
`[FrameCapture] Some image elements did not load within ${pageReadyTimeout}ms: ${failedImages}. ` +
|
||||||
|
`Continuing render — affected images may appear blank/missing in early frames.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
await decodeAllImages(page);
|
||||||
|
|
||||||
await page.evaluate(`document.fonts?.ready`);
|
await page.evaluate(`document.fonts?.ready`);
|
||||||
await waitForOptionalTailwindReady(page, pageReadyTimeout);
|
await waitForOptionalTailwindReady(page, pageReadyTimeout);
|
||||||
|
|
||||||
@@ -812,6 +915,28 @@ export async function initializeSession(session: CaptureSession): Promise<void>
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Image readiness — parity with pollVideosReady. Defense against remote
|
||||||
|
// <img> URLs that bypass the htmlCompiler localize step.
|
||||||
|
const bfImagesReady = await pollImagesReady(page, pageReadyTimeout);
|
||||||
|
if (!bfImagesReady) {
|
||||||
|
const failedImages = await page.evaluate(() => {
|
||||||
|
return Array.from(document.querySelectorAll("img"))
|
||||||
|
.filter((img) => {
|
||||||
|
const ie = img as HTMLImageElement;
|
||||||
|
const src = ie.getAttribute("src") || "";
|
||||||
|
if (!src || src.startsWith("data:")) return false;
|
||||||
|
return !(ie.complete && ie.naturalWidth > 0);
|
||||||
|
})
|
||||||
|
.map((img) => (img as HTMLImageElement).src || img.getAttribute("src") || "(no src)")
|
||||||
|
.join(", ");
|
||||||
|
});
|
||||||
|
console.warn(
|
||||||
|
`[FrameCapture] Some image elements did not load within ${pageReadyTimeout}ms: ${failedImages}. ` +
|
||||||
|
`Continuing render — affected images may appear blank/missing in early frames.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
await decodeAllImages(page);
|
||||||
|
|
||||||
// Font check (no rAF dependency — uses fonts.ready API directly)
|
// Font check (no rAF dependency — uses fonts.ready API directly)
|
||||||
await page.evaluate(`document.fonts?.ready`);
|
await page.evaluate(`document.fonts?.ready`);
|
||||||
await waitForOptionalTailwindReady(page, pageReadyTimeout);
|
await waitForOptionalTailwindReady(page, pageReadyTimeout);
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import {
|
|||||||
discoverAudioVolumeAutomationFromTimeline,
|
discoverAudioVolumeAutomationFromTimeline,
|
||||||
inlineExternalScripts,
|
inlineExternalScripts,
|
||||||
localizeRemoteMediaSources,
|
localizeRemoteMediaSources,
|
||||||
|
localizeRemoteImageSources,
|
||||||
localizeRemoteFontFaces,
|
localizeRemoteFontFaces,
|
||||||
recompileWithResolutions,
|
recompileWithResolutions,
|
||||||
} from "./htmlCompiler.js";
|
} from "./htmlCompiler.js";
|
||||||
@@ -950,6 +951,146 @@ describe("localizeRemoteMediaSources", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ── localizeRemoteImageSources ───────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// Regression coverage for the agent-pipeline `<img>` flicker bug: producer's
|
||||||
|
// frame-capture has no `pollImagesReady` analog of `pollVideosReady`, so a
|
||||||
|
// composition with raw S3 `<img src="https://...">` URLs (astral / daphne /
|
||||||
|
// hyperion multi-v2 outputs) reaches Chrome with a network dependency that
|
||||||
|
// races the readiness gate AND can be evicted mid-render. Localising before
|
||||||
|
// render is the architectural fix; `pollImagesReady` in frameCapture is the
|
||||||
|
// defense-in-depth layer.
|
||||||
|
//
|
||||||
|
// Mirrors the localizeRemoteMediaSources test shape; fetch is patched in
|
||||||
|
// for success cases and a real 404 covers the fallback path.
|
||||||
|
|
||||||
|
describe("localizeRemoteImageSources", () => {
|
||||||
|
it("rewrites remote <img> src to _remote_media path when download succeeds", async () => {
|
||||||
|
const orig = globalThis.fetch;
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = async () => new Response(new Uint8Array(100), { status: 200 });
|
||||||
|
try {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-ok-"));
|
||||||
|
const html = `<img class="hero" src="https://img-ok.example.com/photo.png" />`;
|
||||||
|
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||||
|
expect(result).not.toContain("https://img-ok.example.com/");
|
||||||
|
expect(result).toContain("_remote_media/");
|
||||||
|
expect(remoteMediaAssets.size).toBe(1);
|
||||||
|
} finally {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = orig;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("preserves original URL on download failure without throwing", async () => {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-fail-"));
|
||||||
|
const url = "https://example.com/will-404-image-localize-test.png";
|
||||||
|
const html = `<img src="${url}" />`;
|
||||||
|
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||||
|
expect(result).toContain(url);
|
||||||
|
expect(remoteMediaAssets.size).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("deduplicates: two <img> tags with the same src URL → one download", async () => {
|
||||||
|
const orig = globalThis.fetch;
|
||||||
|
let fetchCount = 0;
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = async () => {
|
||||||
|
fetchCount++;
|
||||||
|
return new Response(new Uint8Array(100), { status: 200 });
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-dedup-"));
|
||||||
|
const html = `<img src="https://dedup-img.example.com/hero.jpg" />
|
||||||
|
<img src="https://dedup-img.example.com/hero.jpg" />`;
|
||||||
|
await localizeRemoteImageSources(html, dl);
|
||||||
|
expect(fetchCount).toBe(1);
|
||||||
|
} finally {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = orig;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not rewrite local (non-HTTP) src paths", async () => {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-local-"));
|
||||||
|
const html = `<img src="assets/hero.png" />`;
|
||||||
|
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||||
|
expect(result).toContain("assets/hero.png");
|
||||||
|
expect(result).not.toContain("_remote_media/");
|
||||||
|
expect(remoteMediaAssets.size).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not rewrite data: URI src", async () => {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-data-"));
|
||||||
|
const html = `<img src="data:image/svg+xml,%3Csvg/%3E" />`;
|
||||||
|
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||||
|
expect(result).toContain("data:image/svg+xml");
|
||||||
|
expect(remoteMediaAssets.size).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rewrites both double-quoted and single-quoted src attributes", async () => {
|
||||||
|
const orig = globalThis.fetch;
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = async () => new Response(new Uint8Array(100), { status: 200 });
|
||||||
|
try {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-quotes-"));
|
||||||
|
const html = `<img src="https://q-img.example.com/dq.png" />
|
||||||
|
<img src='https://q-img.example.com/sq.jpg' />`;
|
||||||
|
const { html: result } = await localizeRemoteImageSources(html, dl);
|
||||||
|
expect(result).not.toContain("https://q-img.example.com/");
|
||||||
|
expect(result.match(/_remote_media\//g)?.length).toBe(2);
|
||||||
|
} finally {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = orig;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not match data-src (lazy-loader placeholder), only the real src attribute", async () => {
|
||||||
|
// A lazy-loader emits the real asset in `data-src` and a placeholder in
|
||||||
|
// `src`. We must localise what Chrome actually paints (the real `src`),
|
||||||
|
// not the `data-src` URL — matching `data-src` would download an asset the
|
||||||
|
// render never shows and could break the loader's runtime swap.
|
||||||
|
const orig = globalThis.fetch;
|
||||||
|
let fetchCount = 0;
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = async () => {
|
||||||
|
fetchCount++;
|
||||||
|
return new Response(new Uint8Array(100), { status: 200 });
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-datasrc-"));
|
||||||
|
const html = `<img data-src="https://lazy.example.com/real.png" src="https://cdn.example.com/placeholder.png" />`;
|
||||||
|
const { html: result } = await localizeRemoteImageSources(html, dl);
|
||||||
|
// The real src is localised; the data-src URL is left untouched.
|
||||||
|
expect(result).toContain("https://lazy.example.com/real.png");
|
||||||
|
expect(result).not.toContain("https://cdn.example.com/placeholder.png");
|
||||||
|
expect(fetchCount).toBe(1);
|
||||||
|
} finally {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = orig;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("handles src attribute not as the first attribute (agent-pipeline shape)", async () => {
|
||||||
|
// The 02_kobe astral-pipeline composition that surfaced this bug emits
|
||||||
|
// <img> tags with `class` before `src`. Regex must not assume src position.
|
||||||
|
const orig = globalThis.fetch;
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = async () => new Response(new Uint8Array(100), { status: 200 });
|
||||||
|
try {
|
||||||
|
const dl = mkdtempSync(join(tmpdir(), "hf-img-attr-order-"));
|
||||||
|
const html = `<img class="kobe-cutout" alt="kobe" src="https://astral.example.com/d828bca.png" />`;
|
||||||
|
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||||
|
expect(result).not.toContain("https://astral.example.com/");
|
||||||
|
expect(result).toContain("_remote_media/");
|
||||||
|
expect(remoteMediaAssets.size).toBe(1);
|
||||||
|
} finally {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
(globalThis as any).fetch = orig;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
// ── localizeRemoteFontFaces ──────────────────────────────────────────────────
|
// ── localizeRemoteFontFaces ──────────────────────────────────────────────────
|
||||||
|
|
||||||
describe("localizeRemoteFontFaces", () => {
|
describe("localizeRemoteFontFaces", () => {
|
||||||
|
|||||||
@@ -877,6 +877,14 @@ const REMOTE_MEDIA_SUBDIR = "_remote_media";
|
|||||||
// have `>` inside quoted attribute values (data-title etc.).
|
// have `>` inside quoted attribute values (data-title etc.).
|
||||||
const REMOTE_MEDIA_TAG_RE =
|
const REMOTE_MEDIA_TAG_RE =
|
||||||
/<(?:video|audio)\b[^>]*?\bsrc\s*=\s*["'](https?:\/\/[^"']+)["'][^>]*>/gi;
|
/<(?:video|audio)\b[^>]*?\bsrc\s*=\s*["'](https?:\/\/[^"']+)["'][^>]*>/gi;
|
||||||
|
// Match <img> tags (including agent-pipeline-emitted variants where `src` is
|
||||||
|
// not the first attribute). Producer-side localisation is the primary fix for
|
||||||
|
// the remote-<img> flicker; frameCapture's `pollImagesReady`/`decodeAllImages`
|
||||||
|
// are the defense-in-depth layer for any remote URL that bypasses this step.
|
||||||
|
// The `(?<![\w-])` lookbehind pins the match to a real `src` attribute so we
|
||||||
|
// don't rewrite `data-src` / `data-*-src` (lazy-loader placeholders whose URL
|
||||||
|
// is not what Chrome actually paints). `srcset` is excluded by the `\s*=`.
|
||||||
|
const REMOTE_IMG_TAG_RE = /<img\b[^>]*?(?<![\w-])src\s*=\s*["'](https?:\/\/[^"']+)["'][^>]*>/gi;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Download a set of remote URLs in parallel into `remoteDir`, build the
|
* Download a set of remote URLs in parallel into `remoteDir`, build the
|
||||||
@@ -971,6 +979,49 @@ export async function localizeRemoteMediaSources(
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Download any remote `src` URLs on `<img>` elements into a local subdirectory
|
||||||
|
* of `downloadDir`, rewrite the HTML src attributes to relative paths, and
|
||||||
|
* return a `{ relativePath → absoluteLocalPath }` map for the orchestrator.
|
||||||
|
*
|
||||||
|
* Why: a composition with remote S3 `<img src>` URLs reaches Chrome unchanged;
|
||||||
|
* the readiness check can pass before the image is fully decoded, *and* Chrome
|
||||||
|
* may evict decoded pixels mid-render under memory pressure and re-fetch from
|
||||||
|
* the remote origin. Either path produces blank-frame flicker. Localising the
|
||||||
|
* sources before render eliminates both races — once the file is local,
|
||||||
|
* Chrome's image cache is bounded by fast disk reads, not S3 latency, so a
|
||||||
|
* mid-render re-fetch lands within a frame instead of flickering. This is the
|
||||||
|
* primary fix; frameCapture's `pollImagesReady` is the defense-in-depth layer.
|
||||||
|
*
|
||||||
|
* Scope: only `<img src>` is localised here. Remote `srcset`,
|
||||||
|
* `<picture><source>`, SVG `<image href>`, and CSS `background-image: url()`
|
||||||
|
* outside `@font-face` are NOT covered — agent-pipeline compositions emit
|
||||||
|
* plain `<img src>`, but those are open follow-ups if other shapes appear.
|
||||||
|
*
|
||||||
|
* This bites agent-pipeline-generated compositions (astral / daphne /
|
||||||
|
* hyperion `multi-v2` outputs) which render directly without going through
|
||||||
|
* `hyperframes publish`'s archive-time localize step.
|
||||||
|
*/
|
||||||
|
/** @internal exported for unit testing only */
|
||||||
|
export async function localizeRemoteImageSources(
|
||||||
|
html: string,
|
||||||
|
downloadDir: string,
|
||||||
|
): Promise<{ html: string; remoteMediaAssets: Map<string, string> }> {
|
||||||
|
const urlSet = new Set<string>();
|
||||||
|
const re = new RegExp(REMOTE_IMG_TAG_RE.source, REMOTE_IMG_TAG_RE.flags);
|
||||||
|
let m: RegExpExecArray | null;
|
||||||
|
while ((m = re.exec(html)) !== null) {
|
||||||
|
if (m[1]) urlSet.add(m[1]);
|
||||||
|
}
|
||||||
|
return downloadAndRewriteUrls(
|
||||||
|
urlSet,
|
||||||
|
html,
|
||||||
|
join(downloadDir, REMOTE_MEDIA_SUBDIR),
|
||||||
|
"Remote image download failed for",
|
||||||
|
"Localized remote image source(s)",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// Match url("https://...") or url('https://...') inside @font-face blocks.
|
// Match url("https://...") or url('https://...') inside @font-face blocks.
|
||||||
// We scan the full HTML (which includes <style> blocks) — matching against
|
// We scan the full HTML (which includes <style> blocks) — matching against
|
||||||
// @font-face context precisely would require a CSS parser; instead we match
|
// @font-face context precisely would require a CSS parser; instead we match
|
||||||
@@ -1159,12 +1210,22 @@ export async function compileForRender(
|
|||||||
externalAssets.set(relPath, absPath);
|
externalAssets.set(relPath, absPath);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Download remote <img> sources. Same race shape as video/audio: the
|
||||||
|
// readiness gate can pass before Chrome decodes the pixels, and Chrome can
|
||||||
|
// evict decoded pixels mid-render and re-fetch, producing intermittent
|
||||||
|
// blank-frame flicker. Localising to disk removes both races.
|
||||||
|
const { html: htmlWithLocalImages, remoteMediaAssets: remoteImageAssets } =
|
||||||
|
await localizeRemoteImageSources(htmlWithLocalMedia, downloadDir);
|
||||||
|
for (const [relPath, absPath] of remoteImageAssets) {
|
||||||
|
externalAssets.set(relPath, absPath);
|
||||||
|
}
|
||||||
|
|
||||||
// Download remote @font-face src URLs and rewrite to local paths.
|
// Download remote @font-face src URLs and rewrite to local paths.
|
||||||
// Remote font URLs fail with a CORS rejection at render time (S3 does not
|
// Remote font URLs fail with a CORS rejection at render time (S3 does not
|
||||||
// allow http://localhost:PORT as origin), causing Chrome to silently fall
|
// allow http://localhost:PORT as origin), causing Chrome to silently fall
|
||||||
// back to the next font in the stack.
|
// back to the next font in the stack.
|
||||||
const { html, remoteMediaAssets: remoteFontAssets } = await localizeRemoteFontFaces(
|
const { html, remoteMediaAssets: remoteFontAssets } = await localizeRemoteFontFaces(
|
||||||
htmlWithLocalMedia,
|
htmlWithLocalImages,
|
||||||
downloadDir,
|
downloadDir,
|
||||||
);
|
);
|
||||||
for (const [relPath, absPath] of remoteFontAssets) {
|
for (const [relPath, absPath] of remoteFontAssets) {
|
||||||
|
|||||||
Reference in New Issue
Block a user