Files
hyperframes/packages/cli/src/capture/userAgent.ts
Miguel Ángel 7a07ea9ac3 fix(cli): fetch a page's assets as the same agent that loaded the page (#3726)
A capture is one session with two halves: Chrome navigates the page with a
browser User-Agent, then Node fetches the assets that page referenced. Those
halves sent three different identities — "HyperFrames/1.0" from the asset and
media downloaders, a bare "Mozilla/5.0" from the stylesheet inliner, and the
real Chrome UA from the navigation itself.

An origin is free to answer those differently, and anti-bot edges do. Capturing
one large site, GET /favicon.svg answers 403 text/html to "HyperFrames/1.0" and
200 image/svg+xml to the UA the very same capture had just navigated with. The
favicon ranker had already picked that SVG as the best declared icon; the 403
discarded it and the downloader fell through to the next candidate, so the icon
written to assets/ was chosen by the CDN's bot rules rather than by the ranker.
The capture reported it as one "unavailable" drop and carried on.

Hoist the navigation UA into CAPTURE_USER_AGENT and use it for every
out-of-band fetch the capture makes: favicons, images, og:image, fonts,
stylesheets, Lottie JSON and videos. One constant is what stops the two halves
drifting apart again.

Verified end to end against that site: before, assets/favicon.png (the
apple-touch icon) plus one unavailable drop; after, assets/favicon.svg, byte
identical to the file the site itself serves.
2026-09-05 22:20:15 -04:00

18 lines
1.0 KiB
TypeScript

/**
* The single identity a capture presents to the origin it is capturing.
*
* A capture is one session with two halves: Chrome navigates the page, then Node fetches the
* assets that page referenced. When those halves send different `User-Agent` headers the origin
* is free to answer them differently, and an anti-bot edge does exactly that — it serves the
* document to the browser and refuses the out-of-band asset fetch.
*
* Measured against openai.com: `GET /favicon.svg` answers `403 text/html` for `HyperFrames/1.0`
* and `200 image/svg+xml` for this string. The favicon ranker had already chosen that SVG as the
* best declared icon; the 403 discarded it and the capture silently fell through to the next
* candidate, so the icon on disk was decided by the CDN's bot rules rather than by the ranker.
*
* One constant, used by both halves, is what keeps that from being reintroduced.
*/
export const CAPTURE_USER_AGENT =
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";