mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-19 19:36:04 +00:00
A capture is one session with two halves: Chrome navigates the page with a browser User-Agent, then Node fetches the assets that page referenced. Those halves sent three different identities — "HyperFrames/1.0" from the asset and media downloaders, a bare "Mozilla/5.0" from the stylesheet inliner, and the real Chrome UA from the navigation itself. An origin is free to answer those differently, and anti-bot edges do. Capturing one large site, GET /favicon.svg answers 403 text/html to "HyperFrames/1.0" and 200 image/svg+xml to the UA the very same capture had just navigated with. The favicon ranker had already picked that SVG as the best declared icon; the 403 discarded it and the downloader fell through to the next candidate, so the icon written to assets/ was chosen by the CDN's bot rules rather than by the ranker. The capture reported it as one "unavailable" drop and carried on. Hoist the navigation UA into CAPTURE_USER_AGENT and use it for every out-of-band fetch the capture makes: favicons, images, og:image, fonts, stylesheets, Lottie JSON and videos. One constant is what stops the two halves drifting apart again. Verified end to end against that site: before, assets/favicon.png (the apple-touch icon) plus one unavailable drop; after, assets/favicon.svg, byte identical to the file the site itself serves.
18 lines
1.0 KiB
TypeScript
18 lines
1.0 KiB
TypeScript
/**
|
|
* The single identity a capture presents to the origin it is capturing.
|
|
*
|
|
* A capture is one session with two halves: Chrome navigates the page, then Node fetches the
|
|
* assets that page referenced. When those halves send different `User-Agent` headers the origin
|
|
* is free to answer them differently, and an anti-bot edge does exactly that — it serves the
|
|
* document to the browser and refuses the out-of-band asset fetch.
|
|
*
|
|
* Measured against openai.com: `GET /favicon.svg` answers `403 text/html` for `HyperFrames/1.0`
|
|
* and `200 image/svg+xml` for this string. The favicon ranker had already chosen that SVG as the
|
|
* best declared icon; the 403 discarded it and the capture silently fell through to the next
|
|
* candidate, so the icon on disk was decided by the CDN's bot rules rather than by the ranker.
|
|
*
|
|
* One constant, used by both halves, is what keeps that from being reintroduced.
|
|
*/
|
|
export const CAPTURE_USER_AGENT =
|
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";
|