mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
fix(producer): localize remote <img> sources + await image readiness (#1197)
* fix(producer): localize remote <img> sources + await image readiness Producer's frame-capture has `pollVideosReady` (waits readyState >= 2 for every <video>) but no equivalent for <img>. Combined with htmlCompiler's `collectExternalAssets` explicitly skipping http(s) URLs (line 805-806), agent-pipeline-generated compositions (astral / daphne / hyperion multi-v2 outputs with raw S3 <img src>) reach Chrome with a network dependency that races the readiness gate AND can be evicted mid-render. Either path produces blank-frame flicker. Reproduction (02_kobe agent output, 42s render @ 30fps): scene_02's remote S3 background-image painted from t=7.0s, vanished at t=10.5s (frame size 139KB vs 700-940KB neighbors), back at t=11.0s. GSAP timeline said opacity:1 throughout — Chrome simply didn't have the pixels. Two-layer fix: 1. **Producer** — `localizeRemoteImageSources` in `htmlCompiler.ts` mirrors the existing `localizeRemoteMediaSources` (video/audio) + `localizeRemoteFontFaces` pattern, reusing `downloadAndRewriteUrls` and the `_remote_media/` subdir. Wired into `compileForRender` between the media and font localize steps. Once the file is local, Chrome's image cache is bounded by disk reads, not S3 latency. 2. **Engine** — `pollImagesReady` + `decodeAllImages` helpers in `frameCapture.ts` parallel to `pollVideosReady`. Waits for every `<img>` (skipping data: URIs) to have `complete && naturalWidth > 0`, then forces GPU upload via `img.decode()`. Called from both the classic-xvfb path and the BeginFrame path after their respective video readiness checks. Defense-in-depth — Layer 1 closes the symptom for current+future agent-pipeline outputs; Layer 2 protects any future code path that leaves a remote URL in place. Tests: 7 new cases in `htmlCompiler.test.ts` covering happy-path rewrite, 404 fallback, dedup of duplicate URLs, non-HTTP and data: URI passthrough, both quote styles, and the agent-pipeline shape where `src` is not the first attribute. All pass alongside the existing 56 htmlCompiler tests. * fix(producer): scope remote-img regex to real src; correct stale comments Review follow-ups on the remote-<img> localization fix: - Tighten REMOTE_IMG_TAG_RE with a (?<![\w-]) lookbehind so it matches a real `src` attribute only. The previous `\bsrc` also matched `data-src` (and `data-*-src`) lazy-loader placeholders, which would download/rewrite a URL the render never paints. Added a regression test; `srcset` stays excluded by the `\s*=` requirement. - Fix comments that claimed frameCapture has "no pollImagesReady analog" — this PR adds exactly that, so the docstrings were self-contradictory. Reframed localization as the primary fix and pollImagesReady as the defense-in-depth layer, and documented the <img src>-only scope (srcset / <picture> / SVG <image> / CSS background-image are follow-ups). Verified locally end-to-end on the 02_kobe repro: all 4 remote S3 <img> URLs localize to _remote_media/, the render completes, and the frame at t~10.5s that was a 139KB blank in the broken render now paints the trophy background in every native-fps frame. htmlCompiler.test.ts 64 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(engine): pollImagesReady broken-image escape + skip decode on in-flight Addresses two real bugs Magi caught in review on hf#1197: 1. pollImagesReady would spin the full pageReadyTimeout (45s default) for any <img> that settled with an error — Chrome marks 404 / decode failure / CORS rejection with (complete=true, naturalWidth=0), and the previous predicate `complete && naturalWidth > 0` returned false for those, so the poll ran to timeout. This is the HTMLImageElement equivalent of pollVideosReady's `ve.error` early-exit. Add a `complete && naturalWidth === 0` branch that treats settled-with- error as done — waiting won't make it load. Particularly relevant because localizeRemoteImageSources falls back to the original URL on download failure; that failed URL is now hit by a 45s stall instead of the broken-image marker rendering immediately. 2. decodeAllImages called img.decode() on every image, including those still in flight after pollImagesReady timed out. Per the WHATWG spec, decode() on a loading image awaits the fetch — never resolving until the network completes or puppeteer's evaluate timeout fires and throws an uncaught error that aborts the render. Pre-filter to only call decode() on images that successfully loaded. Test coverage: new frameCapture-pollImagesReady.test.ts with 8 cases covering empty docs, all-loaded, broken (complete + naturalWidth=0), data: URI, empty src, in-flight → resolves, in-flight → timeout, and the mixed batch. The broken-image test explicitly asserts elapsed < 500ms on a 1000ms timeout — guards against the regression Magi flagged. * docs(engine): clarify decodeAllImages prevents init race, not eviction Vai correctly noted that decode() forces initial GPU upload but does not prevent Chrome from evicting decoded pixels mid-render. The producer-side localizeRemoteImageSources is what bounds the eviction risk (local file-server paging vs S3 re-fetch). Comment updated to reflect that split of responsibilities. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
2be41937a9
commit
72c461d86a
@@ -0,0 +1,153 @@
|
||||
/**
|
||||
* Tests for `pollImagesReady` — the image-side analog of `pollVideosReady`.
|
||||
*
|
||||
* Critical contract:
|
||||
* - Successfully loaded image (complete=true, naturalWidth>0) → settled.
|
||||
* - Broken / 404 image (complete=true, naturalWidth=0) → settled.
|
||||
* This mirrors `pollVideosReady`'s `ve.error` early-exit. Without it,
|
||||
* the htmlCompiler 404-fallback path (where a remote <img> URL failed
|
||||
* to download and the original URL is preserved) would silently spin
|
||||
* the full `pageReadyTimeout` budget waiting for an image that will
|
||||
* never load — a 45 s regression vs the pre-PR behavior.
|
||||
* - In-flight image (complete=false) → still waiting.
|
||||
* - data: URI src → settled (no network fetch).
|
||||
* - Empty src → settled (nothing to load).
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { Page } from "puppeteer-core";
|
||||
import { pollImagesReady } from "./frameCapture.js";
|
||||
|
||||
interface ImageSpec {
|
||||
src: string;
|
||||
complete: boolean;
|
||||
naturalWidth: number;
|
||||
}
|
||||
|
||||
// Mock `page` whose `evaluate(fn)` invokes `fn` with a Node-side `document`
|
||||
// mock that returns synthetic image objects matching the spec. Snapshots the
|
||||
// image state at evaluate-time, so callers can mutate `imgs` between polls
|
||||
// to simulate progressive load completion.
|
||||
function makeMockPage(imgs: () => ImageSpec[]): Page {
|
||||
return {
|
||||
evaluate: async (fn: () => unknown) => {
|
||||
const snapshot = imgs();
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const prevDoc = (globalThis as any).document;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).document = {
|
||||
querySelectorAll: () =>
|
||||
snapshot.map((spec) => ({
|
||||
getAttribute: (attr: string) => (attr === "src" ? spec.src : null),
|
||||
complete: spec.complete,
|
||||
naturalWidth: spec.naturalWidth,
|
||||
})),
|
||||
};
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).document = prevDoc;
|
||||
}
|
||||
},
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
} as any as Page;
|
||||
}
|
||||
|
||||
describe("pollImagesReady", () => {
|
||||
it("resolves immediately when there are no <img> elements", async () => {
|
||||
const page = makeMockPage(() => []);
|
||||
const result = await pollImagesReady(page, 1000, 10);
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
|
||||
it("resolves immediately when every image has loaded successfully", async () => {
|
||||
const page = makeMockPage(() => [
|
||||
{ src: "/a.png", complete: true, naturalWidth: 100 },
|
||||
{ src: "/b.png", complete: true, naturalWidth: 200 },
|
||||
]);
|
||||
const result = await pollImagesReady(page, 1000, 10);
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
|
||||
it("treats a broken image (complete=true, naturalWidth=0) as settled — does NOT wait for timeout", async () => {
|
||||
// This is the bug Magi flagged. Without the broken-image escape, this
|
||||
// test would block the full 1000ms timeout and return false.
|
||||
const page = makeMockPage(() => [
|
||||
{ src: "/a.png", complete: true, naturalWidth: 100 },
|
||||
{ src: "https://broken.example.com/404.png", complete: true, naturalWidth: 0 },
|
||||
]);
|
||||
const t0 = Date.now();
|
||||
const result = await pollImagesReady(page, 1000, 10);
|
||||
const elapsed = Date.now() - t0;
|
||||
expect(result).toBe(true);
|
||||
// Must resolve fast — well under the 1000ms timeout.
|
||||
expect(elapsed).toBeLessThan(500);
|
||||
});
|
||||
|
||||
it("treats a data: URI src as settled regardless of complete/naturalWidth", async () => {
|
||||
const page = makeMockPage(() => [
|
||||
{ src: "data:image/svg+xml,%3Csvg/%3E", complete: false, naturalWidth: 0 },
|
||||
]);
|
||||
const result = await pollImagesReady(page, 1000, 10);
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
|
||||
it("treats an empty src as settled (nothing to load)", async () => {
|
||||
const page = makeMockPage(() => [{ src: "", complete: false, naturalWidth: 0 }]);
|
||||
const result = await pollImagesReady(page, 1000, 10);
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
|
||||
it("waits for an in-flight image and resolves once it completes", async () => {
|
||||
// Image starts in-flight, then completes after ~50ms.
|
||||
let started = false;
|
||||
const startTime = { value: 0 };
|
||||
const page = makeMockPage(() => {
|
||||
if (!started) {
|
||||
started = true;
|
||||
startTime.value = Date.now();
|
||||
}
|
||||
const elapsed = Date.now() - startTime.value;
|
||||
const loaded = elapsed >= 50;
|
||||
return [{ src: "/slow.png", complete: loaded, naturalWidth: loaded ? 100 : 0 }];
|
||||
});
|
||||
const result = await pollImagesReady(page, 1000, 10);
|
||||
expect(result).toBe(true);
|
||||
});
|
||||
|
||||
it("times out and returns false when an in-flight image never resolves", async () => {
|
||||
// Image stays in-flight (complete=false) for the full timeout.
|
||||
const page = makeMockPage(() => [
|
||||
{ src: "/never-loads.png", complete: false, naturalWidth: 0 },
|
||||
]);
|
||||
const result = await pollImagesReady(page, 100, 10);
|
||||
expect(result).toBe(false);
|
||||
});
|
||||
|
||||
it("mixed batch: loaded + broken + data: + in-flight → waits only on the in-flight image", async () => {
|
||||
let resolved = false;
|
||||
const start = Date.now();
|
||||
const page = makeMockPage(() => {
|
||||
const elapsed = Date.now() - start;
|
||||
if (elapsed >= 30) resolved = true;
|
||||
return [
|
||||
{ src: "/loaded.png", complete: true, naturalWidth: 800 },
|
||||
{ src: "https://broken.example.com/404.jpg", complete: true, naturalWidth: 0 },
|
||||
{ src: "data:image/svg+xml,abc", complete: false, naturalWidth: 0 },
|
||||
{
|
||||
src: "/in-flight.png",
|
||||
complete: resolved,
|
||||
naturalWidth: resolved ? 200 : 0,
|
||||
},
|
||||
];
|
||||
});
|
||||
const t0 = Date.now();
|
||||
const result = await pollImagesReady(page, 1000, 10);
|
||||
const elapsed = Date.now() - t0;
|
||||
expect(result).toBe(true);
|
||||
// Should wait roughly for the in-flight image to settle (~30ms) — not the
|
||||
// full timeout. Allow generous slack for CI scheduler jitter.
|
||||
expect(elapsed).toBeLessThan(500);
|
||||
});
|
||||
});
|
||||
@@ -550,6 +550,89 @@ async function pollVideosReady(
|
||||
return check();
|
||||
}
|
||||
|
||||
// Wait for every `<img>` with a non-`data:` src to have settled — either
|
||||
// successfully loaded (`complete && naturalWidth > 0`) or failed with a
|
||||
// broken-image marker (`complete && naturalWidth === 0`, the HTMLImageElement
|
||||
// equivalent of HTMLMediaElement.error). htmlCompiler localises remote `<img>`
|
||||
// URLs to the local file server before this point, so in practice this polls
|
||||
// for the local fetch to land — but the guard is a defensive net so that any
|
||||
// future composition path that leaves a remote URL in place won't capture
|
||||
// frames before the pixels arrive. Mirrors `pollVideosReady` for parity with
|
||||
// the video-side readiness contract (videos exit-early on `ve.error`; images
|
||||
// exit-early on `complete && naturalWidth === 0`).
|
||||
/** @internal exported for unit testing only */
|
||||
export async function pollImagesReady(
|
||||
page: Page,
|
||||
timeoutMs: number,
|
||||
intervalMs: number = 100,
|
||||
): Promise<boolean> {
|
||||
const check = async (): Promise<boolean> => {
|
||||
return Boolean(
|
||||
await page.evaluate(() => {
|
||||
const imgs = Array.from(document.querySelectorAll("img"));
|
||||
return (
|
||||
imgs.length === 0 ||
|
||||
imgs.every((img) => {
|
||||
const ie = img as HTMLImageElement;
|
||||
const src = ie.getAttribute("src") || "";
|
||||
if (!src || src.startsWith("data:")) return true;
|
||||
// A `complete` image with zero naturalWidth has settled with an
|
||||
// error (404 / decode failure / CORS rejection / blocked). Treat
|
||||
// as done — waiting won't make it load — and let the render
|
||||
// continue with the broken-image marker visible. Mirrors how
|
||||
// pollVideosReady treats `ve.error`.
|
||||
if (ie.complete && ie.naturalWidth === 0) return true;
|
||||
if (ie.complete && ie.naturalWidth > 0) return true;
|
||||
return false;
|
||||
})
|
||||
);
|
||||
}),
|
||||
);
|
||||
};
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
if (await check()) return true;
|
||||
await new Promise((resolve) => setTimeout(resolve, intervalMs));
|
||||
}
|
||||
return check();
|
||||
}
|
||||
|
||||
// Force every successfully-loaded `<img>` to be GPU-uploaded before the first
|
||||
// frame capture. `naturalWidth > 0` means the bitmap has been decoded into
|
||||
// CPU memory, but compositor-side GPU upload can still happen lazily on first
|
||||
// paint. Calling `img.decode()` returns a Promise that resolves once the image
|
||||
// is ready for synchronous painting — eliminating the small first-frame race
|
||||
// between "image is technically loaded" and "the rasterized texture is on the
|
||||
// GPU and ready to composite".
|
||||
//
|
||||
// Note this is purely an init-time guard; it doesn't prevent Chrome from
|
||||
// evicting decoded pixels mid-render. The producer-side `localizeRemoteImageSources`
|
||||
// is what bounds the eviction risk (a re-fetch hits the local file server's
|
||||
// disk-backed paging, not S3 over the network).
|
||||
//
|
||||
// Critical: `decode()` on an in-flight image waits for the fetch to resolve.
|
||||
// If `pollImagesReady` timed out with some images still loading (`!complete`),
|
||||
// calling `decode()` on them would block here until the network finally
|
||||
// completes — or until puppeteer's evaluate timeout fires and throws an
|
||||
// uncaught error that aborts the render. Skip in-flight and broken images;
|
||||
// only force GPU upload for images that successfully loaded.
|
||||
async function decodeAllImages(page: Page): Promise<void> {
|
||||
await page.evaluate(async () => {
|
||||
const imgs = Array.from(document.querySelectorAll("img"));
|
||||
await Promise.all(
|
||||
imgs.map((img) => {
|
||||
const ie = img as HTMLImageElement;
|
||||
if (typeof ie.decode !== "function") return Promise.resolve();
|
||||
// Skip still-loading images (in-flight decode() would hang) and
|
||||
// broken images (decode() rejects, but pre-filtering is clearer
|
||||
// than relying on the .catch).
|
||||
if (!ie.complete || ie.naturalWidth === 0) return Promise.resolve();
|
||||
return ie.decode().catch(() => undefined);
|
||||
}),
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
async function applyVideoMetadataHints(
|
||||
page: Page,
|
||||
hints: readonly CaptureVideoMetadataHint[] | undefined,
|
||||
@@ -707,6 +790,26 @@ export async function initializeSession(session: CaptureSession): Promise<void>
|
||||
);
|
||||
}
|
||||
|
||||
const imagesReady = await pollImagesReady(page, pageReadyTimeout);
|
||||
if (!imagesReady) {
|
||||
const failedImages = await page.evaluate(() => {
|
||||
return Array.from(document.querySelectorAll("img"))
|
||||
.filter((img) => {
|
||||
const ie = img as HTMLImageElement;
|
||||
const src = ie.getAttribute("src") || "";
|
||||
if (!src || src.startsWith("data:")) return false;
|
||||
return !(ie.complete && ie.naturalWidth > 0);
|
||||
})
|
||||
.map((img) => (img as HTMLImageElement).src || img.getAttribute("src") || "(no src)")
|
||||
.join(", ");
|
||||
});
|
||||
console.warn(
|
||||
`[FrameCapture] Some image elements did not load within ${pageReadyTimeout}ms: ${failedImages}. ` +
|
||||
`Continuing render — affected images may appear blank/missing in early frames.`,
|
||||
);
|
||||
}
|
||||
await decodeAllImages(page);
|
||||
|
||||
await page.evaluate(`document.fonts?.ready`);
|
||||
await waitForOptionalTailwindReady(page, pageReadyTimeout);
|
||||
|
||||
@@ -812,6 +915,28 @@ export async function initializeSession(session: CaptureSession): Promise<void>
|
||||
);
|
||||
}
|
||||
|
||||
// Image readiness — parity with pollVideosReady. Defense against remote
|
||||
// <img> URLs that bypass the htmlCompiler localize step.
|
||||
const bfImagesReady = await pollImagesReady(page, pageReadyTimeout);
|
||||
if (!bfImagesReady) {
|
||||
const failedImages = await page.evaluate(() => {
|
||||
return Array.from(document.querySelectorAll("img"))
|
||||
.filter((img) => {
|
||||
const ie = img as HTMLImageElement;
|
||||
const src = ie.getAttribute("src") || "";
|
||||
if (!src || src.startsWith("data:")) return false;
|
||||
return !(ie.complete && ie.naturalWidth > 0);
|
||||
})
|
||||
.map((img) => (img as HTMLImageElement).src || img.getAttribute("src") || "(no src)")
|
||||
.join(", ");
|
||||
});
|
||||
console.warn(
|
||||
`[FrameCapture] Some image elements did not load within ${pageReadyTimeout}ms: ${failedImages}. ` +
|
||||
`Continuing render — affected images may appear blank/missing in early frames.`,
|
||||
);
|
||||
}
|
||||
await decodeAllImages(page);
|
||||
|
||||
// Font check (no rAF dependency — uses fonts.ready API directly)
|
||||
await page.evaluate(`document.fonts?.ready`);
|
||||
await waitForOptionalTailwindReady(page, pageReadyTimeout);
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
discoverAudioVolumeAutomationFromTimeline,
|
||||
inlineExternalScripts,
|
||||
localizeRemoteMediaSources,
|
||||
localizeRemoteImageSources,
|
||||
localizeRemoteFontFaces,
|
||||
recompileWithResolutions,
|
||||
} from "./htmlCompiler.js";
|
||||
@@ -950,6 +951,146 @@ describe("localizeRemoteMediaSources", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ── localizeRemoteImageSources ───────────────────────────────────────────────
|
||||
//
|
||||
// Regression coverage for the agent-pipeline `<img>` flicker bug: producer's
|
||||
// frame-capture has no `pollImagesReady` analog of `pollVideosReady`, so a
|
||||
// composition with raw S3 `<img src="https://...">` URLs (astral / daphne /
|
||||
// hyperion multi-v2 outputs) reaches Chrome with a network dependency that
|
||||
// races the readiness gate AND can be evicted mid-render. Localising before
|
||||
// render is the architectural fix; `pollImagesReady` in frameCapture is the
|
||||
// defense-in-depth layer.
|
||||
//
|
||||
// Mirrors the localizeRemoteMediaSources test shape; fetch is patched in
|
||||
// for success cases and a real 404 covers the fallback path.
|
||||
|
||||
describe("localizeRemoteImageSources", () => {
|
||||
it("rewrites remote <img> src to _remote_media path when download succeeds", async () => {
|
||||
const orig = globalThis.fetch;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = async () => new Response(new Uint8Array(100), { status: 200 });
|
||||
try {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-ok-"));
|
||||
const html = `<img class="hero" src="https://img-ok.example.com/photo.png" />`;
|
||||
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||
expect(result).not.toContain("https://img-ok.example.com/");
|
||||
expect(result).toContain("_remote_media/");
|
||||
expect(remoteMediaAssets.size).toBe(1);
|
||||
} finally {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = orig;
|
||||
}
|
||||
});
|
||||
|
||||
it("preserves original URL on download failure without throwing", async () => {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-fail-"));
|
||||
const url = "https://example.com/will-404-image-localize-test.png";
|
||||
const html = `<img src="${url}" />`;
|
||||
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||
expect(result).toContain(url);
|
||||
expect(remoteMediaAssets.size).toBe(0);
|
||||
});
|
||||
|
||||
it("deduplicates: two <img> tags with the same src URL → one download", async () => {
|
||||
const orig = globalThis.fetch;
|
||||
let fetchCount = 0;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = async () => {
|
||||
fetchCount++;
|
||||
return new Response(new Uint8Array(100), { status: 200 });
|
||||
};
|
||||
try {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-dedup-"));
|
||||
const html = `<img src="https://dedup-img.example.com/hero.jpg" />
|
||||
<img src="https://dedup-img.example.com/hero.jpg" />`;
|
||||
await localizeRemoteImageSources(html, dl);
|
||||
expect(fetchCount).toBe(1);
|
||||
} finally {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = orig;
|
||||
}
|
||||
});
|
||||
|
||||
it("does not rewrite local (non-HTTP) src paths", async () => {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-local-"));
|
||||
const html = `<img src="assets/hero.png" />`;
|
||||
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||
expect(result).toContain("assets/hero.png");
|
||||
expect(result).not.toContain("_remote_media/");
|
||||
expect(remoteMediaAssets.size).toBe(0);
|
||||
});
|
||||
|
||||
it("does not rewrite data: URI src", async () => {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-data-"));
|
||||
const html = `<img src="data:image/svg+xml,%3Csvg/%3E" />`;
|
||||
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||
expect(result).toContain("data:image/svg+xml");
|
||||
expect(remoteMediaAssets.size).toBe(0);
|
||||
});
|
||||
|
||||
it("rewrites both double-quoted and single-quoted src attributes", async () => {
|
||||
const orig = globalThis.fetch;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = async () => new Response(new Uint8Array(100), { status: 200 });
|
||||
try {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-quotes-"));
|
||||
const html = `<img src="https://q-img.example.com/dq.png" />
|
||||
<img src='https://q-img.example.com/sq.jpg' />`;
|
||||
const { html: result } = await localizeRemoteImageSources(html, dl);
|
||||
expect(result).not.toContain("https://q-img.example.com/");
|
||||
expect(result.match(/_remote_media\//g)?.length).toBe(2);
|
||||
} finally {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = orig;
|
||||
}
|
||||
});
|
||||
|
||||
it("does not match data-src (lazy-loader placeholder), only the real src attribute", async () => {
|
||||
// A lazy-loader emits the real asset in `data-src` and a placeholder in
|
||||
// `src`. We must localise what Chrome actually paints (the real `src`),
|
||||
// not the `data-src` URL — matching `data-src` would download an asset the
|
||||
// render never shows and could break the loader's runtime swap.
|
||||
const orig = globalThis.fetch;
|
||||
let fetchCount = 0;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = async () => {
|
||||
fetchCount++;
|
||||
return new Response(new Uint8Array(100), { status: 200 });
|
||||
};
|
||||
try {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-datasrc-"));
|
||||
const html = `<img data-src="https://lazy.example.com/real.png" src="https://cdn.example.com/placeholder.png" />`;
|
||||
const { html: result } = await localizeRemoteImageSources(html, dl);
|
||||
// The real src is localised; the data-src URL is left untouched.
|
||||
expect(result).toContain("https://lazy.example.com/real.png");
|
||||
expect(result).not.toContain("https://cdn.example.com/placeholder.png");
|
||||
expect(fetchCount).toBe(1);
|
||||
} finally {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = orig;
|
||||
}
|
||||
});
|
||||
|
||||
it("handles src attribute not as the first attribute (agent-pipeline shape)", async () => {
|
||||
// The 02_kobe astral-pipeline composition that surfaced this bug emits
|
||||
// <img> tags with `class` before `src`. Regex must not assume src position.
|
||||
const orig = globalThis.fetch;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = async () => new Response(new Uint8Array(100), { status: 200 });
|
||||
try {
|
||||
const dl = mkdtempSync(join(tmpdir(), "hf-img-attr-order-"));
|
||||
const html = `<img class="kobe-cutout" alt="kobe" src="https://astral.example.com/d828bca.png" />`;
|
||||
const { html: result, remoteMediaAssets } = await localizeRemoteImageSources(html, dl);
|
||||
expect(result).not.toContain("https://astral.example.com/");
|
||||
expect(result).toContain("_remote_media/");
|
||||
expect(remoteMediaAssets.size).toBe(1);
|
||||
} finally {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(globalThis as any).fetch = orig;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── localizeRemoteFontFaces ──────────────────────────────────────────────────
|
||||
|
||||
describe("localizeRemoteFontFaces", () => {
|
||||
|
||||
@@ -877,6 +877,14 @@ const REMOTE_MEDIA_SUBDIR = "_remote_media";
|
||||
// have `>` inside quoted attribute values (data-title etc.).
|
||||
const REMOTE_MEDIA_TAG_RE =
|
||||
/<(?:video|audio)\b[^>]*?\bsrc\s*=\s*["'](https?:\/\/[^"']+)["'][^>]*>/gi;
|
||||
// Match <img> tags (including agent-pipeline-emitted variants where `src` is
|
||||
// not the first attribute). Producer-side localisation is the primary fix for
|
||||
// the remote-<img> flicker; frameCapture's `pollImagesReady`/`decodeAllImages`
|
||||
// are the defense-in-depth layer for any remote URL that bypasses this step.
|
||||
// The `(?<![\w-])` lookbehind pins the match to a real `src` attribute so we
|
||||
// don't rewrite `data-src` / `data-*-src` (lazy-loader placeholders whose URL
|
||||
// is not what Chrome actually paints). `srcset` is excluded by the `\s*=`.
|
||||
const REMOTE_IMG_TAG_RE = /<img\b[^>]*?(?<![\w-])src\s*=\s*["'](https?:\/\/[^"']+)["'][^>]*>/gi;
|
||||
|
||||
/**
|
||||
* Download a set of remote URLs in parallel into `remoteDir`, build the
|
||||
@@ -971,6 +979,49 @@ export async function localizeRemoteMediaSources(
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Download any remote `src` URLs on `<img>` elements into a local subdirectory
|
||||
* of `downloadDir`, rewrite the HTML src attributes to relative paths, and
|
||||
* return a `{ relativePath → absoluteLocalPath }` map for the orchestrator.
|
||||
*
|
||||
* Why: a composition with remote S3 `<img src>` URLs reaches Chrome unchanged;
|
||||
* the readiness check can pass before the image is fully decoded, *and* Chrome
|
||||
* may evict decoded pixels mid-render under memory pressure and re-fetch from
|
||||
* the remote origin. Either path produces blank-frame flicker. Localising the
|
||||
* sources before render eliminates both races — once the file is local,
|
||||
* Chrome's image cache is bounded by fast disk reads, not S3 latency, so a
|
||||
* mid-render re-fetch lands within a frame instead of flickering. This is the
|
||||
* primary fix; frameCapture's `pollImagesReady` is the defense-in-depth layer.
|
||||
*
|
||||
* Scope: only `<img src>` is localised here. Remote `srcset`,
|
||||
* `<picture><source>`, SVG `<image href>`, and CSS `background-image: url()`
|
||||
* outside `@font-face` are NOT covered — agent-pipeline compositions emit
|
||||
* plain `<img src>`, but those are open follow-ups if other shapes appear.
|
||||
*
|
||||
* This bites agent-pipeline-generated compositions (astral / daphne /
|
||||
* hyperion `multi-v2` outputs) which render directly without going through
|
||||
* `hyperframes publish`'s archive-time localize step.
|
||||
*/
|
||||
/** @internal exported for unit testing only */
|
||||
export async function localizeRemoteImageSources(
|
||||
html: string,
|
||||
downloadDir: string,
|
||||
): Promise<{ html: string; remoteMediaAssets: Map<string, string> }> {
|
||||
const urlSet = new Set<string>();
|
||||
const re = new RegExp(REMOTE_IMG_TAG_RE.source, REMOTE_IMG_TAG_RE.flags);
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = re.exec(html)) !== null) {
|
||||
if (m[1]) urlSet.add(m[1]);
|
||||
}
|
||||
return downloadAndRewriteUrls(
|
||||
urlSet,
|
||||
html,
|
||||
join(downloadDir, REMOTE_MEDIA_SUBDIR),
|
||||
"Remote image download failed for",
|
||||
"Localized remote image source(s)",
|
||||
);
|
||||
}
|
||||
|
||||
// Match url("https://...") or url('https://...') inside @font-face blocks.
|
||||
// We scan the full HTML (which includes <style> blocks) — matching against
|
||||
// @font-face context precisely would require a CSS parser; instead we match
|
||||
@@ -1159,12 +1210,22 @@ export async function compileForRender(
|
||||
externalAssets.set(relPath, absPath);
|
||||
}
|
||||
|
||||
// Download remote <img> sources. Same race shape as video/audio: the
|
||||
// readiness gate can pass before Chrome decodes the pixels, and Chrome can
|
||||
// evict decoded pixels mid-render and re-fetch, producing intermittent
|
||||
// blank-frame flicker. Localising to disk removes both races.
|
||||
const { html: htmlWithLocalImages, remoteMediaAssets: remoteImageAssets } =
|
||||
await localizeRemoteImageSources(htmlWithLocalMedia, downloadDir);
|
||||
for (const [relPath, absPath] of remoteImageAssets) {
|
||||
externalAssets.set(relPath, absPath);
|
||||
}
|
||||
|
||||
// Download remote @font-face src URLs and rewrite to local paths.
|
||||
// Remote font URLs fail with a CORS rejection at render time (S3 does not
|
||||
// allow http://localhost:PORT as origin), causing Chrome to silently fall
|
||||
// back to the next font in the stack.
|
||||
const { html, remoteMediaAssets: remoteFontAssets } = await localizeRemoteFontFaces(
|
||||
htmlWithLocalMedia,
|
||||
htmlWithLocalImages,
|
||||
downloadDir,
|
||||
);
|
||||
for (const [relPath, absPath] of remoteFontAssets) {
|
||||
|
||||
Reference in New Issue
Block a user