mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-18 16:08:56 +00:00
524ea46bf9d3fbf965fdb129506a830dbacd98df
26
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
39347da9de |
fix(cli): leave files from outside the project out of Lambda and Cloud Run site uploads (#8060)
* fix(cli): leave files from outside the project out of Lambda and Cloud Run site uploads A site upload for `lambda` or `cloudrun` followed every symlink in the project and shipped what it pointed at, wherever that was on the machine. `publish` and `cloud render` already ask first. Site uploads now leave a file or folder linked in from outside the project out and name each one; `--include-outside-files` (or `includeOutsideFiles` in the SDK's deploySite, renderToLambda and renderToCloudRun) uploads them. publishProjectArchive called without a prepared archive now leaves outside files out too, since nothing approved them. — Rames Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(producer): drop the new option comments the comment ratchet refuses Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7aaeb93716 |
fix(producer): the plan copy checks every project entry, whatever its name (#7906)
* fix(producer): the plan copy checks every project entry, whatever its name A top-level name that starts with two dots, such as ..assets, is an ordinary project entry. The plan copy treated it as outside the project and skipped the link-resolving containment check, so a link with such a name had its target copied into compiled/. Only the project folder itself now skips the check, and the outside test compares whole path segments. * fix(aws-lambda,gcp-cloud-run): untar leaves out symbolic links and special files A project archive can come from another person. A symbolic link in it could point anywhere on the render host, and the planner would copy its target. Untar now unpacks only files, folders and hard links. node-tar already keeps a hard link's target inside the destination folder. The CLI's site upload follows symbolic links when it packs, so its archives hold none. * fix(aws-lambda,gcp-cloud-run): untar turns hard links into separate plain files The site upload packs two hard-linked project files as one file and a hard link. Untar still unpacks both, then gives each hard-linked file its own copy, so the extracted project holds no links of any kind and loses no file. * fix(aws-lambda,gcp-cloud-run): copy hard-linked files without mkdtemp; untar test in its own file * fix(cloud): keep the hard-link copy name short so long file names still unpack |
||
|
|
014326b40b |
fix: every composition file server refuses credential and tool-state files (#7889)
* wip: paused before review * test: cover credential files on the preview routes and the deploy notice * refactor: name the inspector's file lookup for what it serves * fix(cli): a link to a private file outside the project keeps its own left-out reason * fix: escape the upload notice and skip private files from old manifests |
||
|
|
55b5147098 |
fix: Cloud Run renders reach only public addresses; frame cache folder is private to its user (#7777)
* fix(gcp-cloud-run): render pages cannot reach the service's own dispatch endpoint The Cloud Run image now sets PRODUCER_RESTRICT_PAGE_NETWORK=true, so a rendered page reaches only its own project server and public addresses, not loopback ports or the metadata server. The dispatch endpoint also refuses any POST that carries an Origin header: browsers send one on every POST and Cloud Workflows never does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(engine): the default frame cache folder is private to the user who renders The default extraction cache sits in the shared temp folder. It is now created with mode 0700, an existing one owned by the same user is closed to others (its entries stay, so cache hits carry over), and one that is a link or belongs to another user is not used: that render runs without the cache and prints a warning. The animated GIF cache under the same folder follows the same rule. A folder set with HYPERFRAMES_EXTRACT_CACHE_DIR is left as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(gcp-cloud-run): a Terraform variable turns the page network restriction off restrict_page_network (default true) sets PRODUCER_RESTRICT_PAGE_NETWORK on the service. The package README and docs say what the restriction refuses and how to turn it off for trusted compositions that need private hosts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(engine): restore an unset TMPDIR as unset after the cache-root tests Assigning undefined to process.env stores the string "undefined", so on a runner with no TMPDIR every later test made its temp folder under ./undefined. * test(engine): the frame-extractor link test also restores an unset TMPDIR as unset * chore(engine): raise the packed size budget to 500000 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1a9fb8acb6 |
feat: opt-in 4:4:4 chroma for SDR MP4 masters (#5820)
* feat(engine): decouple transparent background from png capture format PNG capture used to imply a cleared page background. A new CaptureOptions.transparentBackground lets lossless opaque PNG capture (4:4:4 MP4 output) keep the page background. Absent, it still follows format === "png", so alpha output and motion blur are unchanged. Opaque PNG skips drawElement, whose verification and worker-encode paths are JPEG-only. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(engine): add 4:4:4 chroma option to getEncoderPreset getEncoderPreset takes an optional chromaSubsampling ("420" by default). "444" returns yuv444p for SDR MP4, so libx264 encodes the High 4:4:4 Predictive profile and odd sizes need no even padding. WebM, MOV and HDR presets ignore it. The ChromaSubsampling type is exported for the producer. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(producer): chromaSubsampling render option with lossless png capture RenderConfig.chromaSubsampling ("420" by default) opts an SDR MP4 render into 4:4:4. "444" encodes libx264 yuv444p and captures every frame as opaque PNG: Chrome's quality-100 JPEG is already 4:2:0 on beginFrame and screenshot capture, and drawElement can fall back to screenshot mid-render, so PNG is the only capture that keeps the colour. drawElement is turned off for these renders. The probe and main capture options share resolveCaptureSurface, so they cannot disagree on format, quality or background. Formats other than mp4, GPU encoding and HDR output are rejected before any work starts, auto-detected HDR sources fail before capture, and distributed requests refuse 444. The default render is unchanged. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(producer,studio-server): accept chromaSubsampling on render routes The shared parseEncoderOptions validates chromaSubsampling and answers 400 for "444" with a non-mp4 format, gpu, or force-hdr, so a request that would come back 4:2:0 never starts a render. The Studio render route, both Studio adapters and the producer HTTP server forward it. The producer server also refuses "444" with outputDynamicRange "hdr", which parseEncoderOptions cannot see. No Studio UI control; the option is route-level like crf and hdrMode. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cli): add --chroma-subsampling flag `hyperframes render --chroma-subsampling 444` opts an SDR MP4 render into full-resolution chroma (yuv444p with lossless PNG capture). The default stays 420. The plan rejects 444 with a non-mp4 format, --gpu or --hdr before anything renders, with a usage error that names the conflicting flag. The value reaches the local render request and, only when it is 444, the Docker argv, so default Docker invocations are unchanged. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: document 4:4:4 chroma subsampling Add a "4:4:4 masters" section to the rendering guide: when to use --chroma-subsampling 444, why it captures PNG, the measured chroma gain (large for text and UI, none for footage and gradients), its limits, and a player compatibility table to be filled in from testing. Document the flag in the CLI reference, the built-in rendering docs and the hyperframes-cli skill reference (with the regenerated skills manifest), and the chromaSubsampling option in the producer docs. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add 4:4:4 player compatibility results Fill the 4:4:4 compatibility table from testing H.264 High 4:4:4 Predictive output. Chrome 155, Edge 155, VLC 3.0, Safari on iOS 26.6 and Safari 26.6 on macOS play it, and so does AVFoundation on macOS 26.6, the framework QuickTime Player plays through. Firefox 157 on Windows and both Windows media players do not. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: note the measured render cost of 4:4:4 output Replace the vague "slower capture" note with measured numbers: a text and UI composition rendered in about twice the default time, full-frame footage in about three and a half times, because PNG encoding cost grows with image detail. Refs #5555 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(producer): reject chromaSubsampling 444 in distributed render config Distributed configs have no chroma field and validateDistributedRenderConfig ignored unknown keys, so a JS caller passing chromaSubsampling: "444" to the Lambda or Cloud Run SDK got a 4:2:0 file with no error. Reject it, and make the rendering guide say so precisely. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(producer): check a 444 render reaches the encoder as yuv444p Every helper had tests, but nothing checked the orchestrator passes chromaSubsampling through, so dropping it from the encoder preset or capture surface would turn 444 renders back into 4:2:0 with all tests green. Render a small fixture with "444" and with the default and check pix_fmt with ffprobe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(producer): move the 444 drawElement gate into a helper Keeps runCompileStage under the fallow complexity threshold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: keep comment share at or below the merge-base comment-ratchet failed on 11 files this PR touched. Keep the 4:4:4 rationale once in chromaSubsampling.ts, move the distributed-render reason into its error message, and cut the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(producer): check a 444 render captures full-resolution chroma The encoder check alone passes when chromaSubsampling is dropped from the capture surface: frames are captured as JPEG and encoded yuv444p with 4:2:0 colour. Put a red/blue edge at an odd x, inside a 4:2:0 chroma pair, and require the two pixels to differ in Cb. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(producer): trim the chromaSubsampling.ts header to the package comment share Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: finish switching to capturesAlpha from #6262 The merge that adopted main's transparent / capturesAlpha left these files uncommitted, so resolveCaptureSurface still returned transparentBackground. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(engine): use transparent / capturesAlpha from #6262 main added CaptureOptions.transparent and capturesAlpha (#6262), which do what this PR's transparentBackground did. Use them, keep the PNG check in drawElementWorkerEncodeEligible, and re-pin test-classification.mjs and the skills manifest against current main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(producer): cut the 4:4:4 comment blocks to one to three lines The reasoning lives in the PR description with the measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: reconcile 4:4:4 chroma with latest main Keep main's video-layer capture overrides, template entry resolution, ProRes profile parsing and blur-differential tests alongside the chroma option, and fold the encoder sub-parsers into one loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: keep main's compile-stage tests and regenerate the skills manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a9a8d89377 |
fix(engine): a render no longer finishes without an image that loaded late (#7304)
* fix(engine): a render no longer finishes without an image that loaded late * style(engine): format the frame gate * fix(engine): an image outside the frame no longer stops a render * fix(engine): a render stops and warns when a frame's image never arrives, with no cloud retries * fix(producer): a cloud chunk keeps the missing image's error name so retries stop |
||
|
|
7f48a6894c |
feat!: remove Plan v1 distributed transport (#6970)
* feat!: remove Plan v1 distributed transport BREAKING CHANGE: Cloud adapters accept only Plan v2; producer v1 exports are removed. * ci: authorize Plan v1 cleanup deletions * docs: avoid citing generated smoke result file * fix(gcp): reject unsupported plan protocols in workflow * ci: preserve gcp package size reduction * fix: reject stale Plan v1 SDK options * chore: remove retired Plan parity scripts |
||
|
|
54d0af57eb |
feat(engine): render sets a keyframe interval and a peak bitrate cap (#6444)
* feat(engine): encoders cap peak bitrate and place keyframes on a set interval * style(engine): shorten the rate-cap comment * style(engine): drop the doc lines the comment ratchet flagged * feat(producer): mp4 renders honour a keyframe interval and a peak bitrate cap on every encoder path * style(producer): shorten the delivery encoder file headers * style(producer): drop a comment line from the segment manifest * style(producer): shorten the integration test header * test: delivery encoder tests follow the ffprobe argv contract and run in the integration lane * feat: the render request, server and Lambda plan carry the keyframe interval and peak bitrate cap * fix(studio): the Studio render route passes the keyframe interval and bitrate cap to the encoder * chore(studio-server): drop a comment the helper's name already says * feat(cli): render and lambda render take --keyframe-interval, --max-bitrate and --buffer-size |
||
|
|
992df8b203 |
perf(cloud): move project bytes once through shared content storage (#5772)
* perf(producer): distributed chunks download only the videos they show A project video under compiled/ that chunks extract their own frames from now goes only to the chunks that show it, like video-sources/ files. A chunk skips the readiness wait for a deferred video it does not show, so it no longer logs media_load_failed for a file it was never given. * fix(producer): cloud site uploads pack the files their id hashes, symlinks included * perf(aws-lambda): chunks read only the byte ranges of a video they need A chunk that extracts its own frames from a deferred project video now reads it from S3 by byte range through a short-lived pre-signed URL, instead of downloading and hashing the whole object first. S3 attests the object's stored SHA-256 before the URL is signed; every other artifact is still downloaded and verified as before. * perf(aws-lambda): upload project files once to a shared content-addressed area * chore(producer): keep planV2 comment share at the base level * fix(engine): verify the TLS certificate of https video inputs and never surface their signed query * fix(aws-lambda): renders re-upload a site's expired files instead of failing * perf(aws-lambda): the plan step probes project videos in place instead of downloading them * fix(engine): https video reads reconnect and time out, and redaction covers wrapped causes Signed-URL reads now pass -rw_timeout, -reconnect and -reconnect_delay_max (both Lambda builds accept them) and ffmpeg also -reconnect_on_network_error, so a transient S3 error or a stalled read fails fast or recovers instead of waiting out the process timeout. Query redaction now descends into error.cause. * fix(aws-lambda): share entry bundles and ratchet package budget * fix(aws-lambda): site ids uploaded before files.json still render A site id from an earlier sites create has only sites/<id>/project.tar.gz. renderToLambda now falls back to that tarball, which the plan step still reads, and when neither key exists it fails before starting with both keys named and the fix (run sites create again). * fix(aws-lambda): renders fail before upload on a stack too old to read sites A stack deployed before files.json sites untars the site file in its plan step and fails after four retries. renderToLambda now reads the state machine definition first and stops with STACK_TOO_OLD and the redeploy command when the plan task does not list SITE_MANIFEST_INVALID, which new SAM and CDK stacks now treat as terminal. The README's upgrade section drops the planProtocol v1 fallback, which hits the same failure. * docs(aws-lambda): document shared content storage and the upgrade path The deploy guide still described the tarball HeadObject short-circuit. It now covers the cas/sha256/ store and its 30-day lifecycle rule, the STACK_TOO_OLD, SITE_FILES_MISSING and SITE_NOT_FOUND errors, that old site ids keep working, and that every project uploads once more after the upgrade. A test keeps a troubleshooting entry for each pre-submit error. * fix(producer): chunks log each video they read by URL, without its query A chunk reading a video straight from the bucket left no trace in its log. It now logs one line per remote source with the path and the URL minus its query string, so signatures never reach the log. * fix(engine): ffmpeg errors never keep part of a signed URL's query A process's stderr tail was trimmed before its URLs were redacted, so a cut inside a signed URL left a query fragment the URL pattern no longer matched, and the fragment reached the error. The stderr tail is now redacted before every trim, and the redactor also strips a signing parameter cut off from its URL. * fix(aws-lambda): comments name the site manifest instead of a file path * fix(engine): stderr redaction survives chunk splits, and every ffmpeg spawn uses it The stderr tail was redacted on each chunk and then cut, so once a cut severed a long URL's scheme, the rest of its signature in the next chunk matched neither pattern. The tail is now kept raw with 2x slack and redacted only when the text is produced, then cut to size. The unused producer audioExtractor (its own spawn, unredacted stderr) is deleted and the parity harness's PCM decode goes through ManagedChildProcess; a test fails on any ffmpeg/ffprobe spawn in engine, producer or aws-lambda that does not. * chore(producer): record the audioExtractor deletion as intended * fix(engine): drop a signature fragment the stderr tail cut from its URL The raw tail is kept at twice the cap, but that cut can itself land inside a signature, and when redaction shrinks the rest below the cap the fragment survived. After a cut, the severed token is dropped before redaction. * refactor(engine): name the severed-token drop instead of commenting it * test(engine): a cut-off URL leaves nothing behind; aws-lambda size budget to measured * fix(producer): chunk URL-read tests use main's frame list; producer size budget to measured * chore(producer): size budget at the measured size, no headroom * chore(producer): size budget back to measured x1.05, the check's own rule |
||
|
|
c69fd2c6ec |
perf(producer): size distributed chunks to the worker limit and concurrency (#5765)
* perf(producer): size distributed chunks to the worker limit and concurrency * chore(cli): drop a comment the field name already says |
||
|
|
d5162beb50 |
refactor(core): one owner for the distributed fps set (#5756)
The integer rates Lambda and Cloud Run accept were written out by hand in about 25 places across core, engine, producer, the adapters and the CLI. They now come from DISTRIBUTED_FPS / DistributedFps / isDistributedFps in @hyperframes/core. Engine config snapshots accept 25 like the rest. |
||
|
|
cc9e1bb33a |
feat(producer): distributed renders accept 25 fps (#5423)
Lambda and Cloud Run renders refused 25 fps although it is an integer
rate like 24, 30 and 60, and PAL footage is 25 fps. The distributed fps
set is now {24, 25, 30, 60} in the producer's config validation, the
plan v2 manifest, the render request mapping, the Lambda and Cloud Run
event types, the CLI's lambda and cloudrun commands and the regression
harnesses.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
|
17a2a00ed5 |
feat(cloud): default distributed plans to v2 (#3311)
* feat(cloud): default distributed plans to v2 * fix(cloud): address plan v2 review feedback * fix(examples): document explicit v2 samples |
||
|
|
dc4383113c |
fix(producer): mix audio into a container that can record encoder delay (#3200)
* fix(producer): mix audio into a container that can record encoder delay Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz) after its authored `data-start`, against a frame-accurate video track. The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix was written to a raw ADTS `.aac` file, which has nowhere to record that delay, so it decoded as real leading silence and every stage downstream preserved it faithfully. Measuring each intermediate localises it precisely: the source WAV is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and mux stages inherit it unchanged. The filter graph itself is correct - run by hand to PCM it lands on the authored start. Switch the artifact to an MP4-family container, which stores the delay as an edit list that decoders strip. Same codec, same bitrate, so no size or quality change. The filename is a contract shared by three consumers - the mux input, the distributed plan artifact, and the PNG-sequence sidecar handed to users for NLE ingest - and its extension is what selects the muxer. Give it one owner in the engine rather than five literals, so those consumers cannot drift onto different containers. Note for reviewers: this renames the distributed plan's audio artifact, which is an on-disk contract between the plan writer and the assembler. Both move together here, but a plan written by an older build would not be found by a newer assembler. Flagging in case that mixed-version window matters for how these are deployed. * fix(cloud): read the plan audio artifact name from the producer contract The aws-lambda and gcp-cloud-run adapters each restated the plan's audio filename in five places, so renaming it in the producer left them looking for a file that is no longer written. CI caught it: the gcp dispatch test asserting a plan has no audio artifact started seeing one. Export the name from `@hyperframes/producer/distributed` and consume it in both adapters. This is the same failure the constant exists to prevent, one package boundary further out: a literal that drifts from the writer's is a silently missing audio track rather than a loud error, because both call sites only ever ask whether the file exists. * fix(cloud): accept a legacy plan's audio artifact name for one release Review raised a rolling-deploy window I had flagged but left undecided: `plan` and `assemble` are separate invocations bridged by object storage, so a pre-rollout planner can be paired with a post-rollout assembler. Both readers locate the artifact by existence alone, which makes that pairing a silently muted video rather than an error. That is reachable enough to be worth two lines, so reads now accept the old name while writes only ever emit the new one. Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`) rather than four call sites, marked for deletion one release out. Also fixes a hole in the first pass of this: the plan-v2 materializer matched either name but then joined the CURRENT one, so a legacy plan resolved to a path that was never written. It now joins the artifact's own name. Review nits in the same pass: correct the pad-branch docstring, which still described a concat-copy shape the pad branch stopped using when it moved to apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so it cannot model a shape that reintroduces the priming delay. * test(producer): rebake the missing-host-comp-id golden without the audio delay The pinned reference was rendered before this branch, so it carries the 1024 sample encoder-priming delay in its audio. With the delay gone the correct audio now sits ahead of the reference and the harness's envelope correlation drops below its floor. Cross-correlating the old and new references at native 48 kHz gives a lag of exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved by exactly the amount this branch removes. Regenerated inside the CI container (Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches the encoder CI will compare against - the container reproduced CI's failure to the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and passes at correlation 1.0 after it. Note for archaeology: the new reference is also 3 dB louder than the old one. That gap is not from this branch - `main` and this branch render the fixture at the same level - it is pre-existing drift the reference had accumulated, which a scale-invariant correlator could never see. The rebake absorbs it. Only output.mp4 is updated. `--update` also rewrites compiled.html, but that diff is embedded-font churn with no bearing on the comparison, which reports "Failed at compilation: 0" either way. * test(producer): rebake the variables-prod golden without the audio delay Same cause as the missing-host-comp-id rebake, caught by shard-8 once the earlier shard stopped failing and the rest of the matrix could run: this reference also carries the encoder-priming delay this branch removes. Reproduced in the CI container to the digit (correlation 0.42704173048439215, lagWindows -12), rebaked there, and it now passes at correlation 1.0. Worth recording: the shift here is 2048 samples (42.67 ms) at correlation 0.99983, exactly twice the 1024 of the other fixture. The delay compounds once per un-compensated AAC generation, and this fixture's audio needs its duration normalized, so it takes the pad/trim branch's re-encode and picks up a second frame of priming on top of the mixer's. So the pre-fix error was not a fixed 21 ms - it grew with the number of times the audio was re-encoded. All nine shards ran in that CI round with only this one failing, so the matrix has now covered every fixture against this change. |
||
|
|
349c066a83 |
fix(producer): classify JSON error bodies as non-media sources too
A source that answers with a JSON error body still reached ffprobe and
produced `moov atom not found`. Replicate returns
`{"detail": "requested file not found"}` for a dead asset, and a gateway
in front of it can relay that body with a success status.
The sniff now treats `<`, `{`, or `[` as the opening byte of a text
document. No supported container starts with any of them, so this is the
same trade as before: three bytes instead of an allowlist that grows one
entry per payload shape observed in production.
Renamed accordingly, since the class now covers JSON as well as markup:
MARKUP_NOT_MEDIA -> NOT_MEDIA_PAYLOAD, MarkupNotMediaError ->
NotMediaPayloadError, markupPayload.ts -> notMediaPayload.ts. Registry
entries in the Lambda name map, the CDK and SAM plan lists, the Cloud Run
set, and SAFE_RENDER_ERROR_CODES move with it.
Also documents the reachability boundary on the error class: only a 2xx
response gets here. `downloadToTemp` rejects 404/410 as `http_not_found`
before writing a byte, and every ffprobe input is local because
videoFrameExtractor downloads http srcs first. So the shapes this
classifies are soft-404 and interstitial HTML, S3/CloudFront error
documents, and JSON API error bodies -- each served with a success
status. A genuine 404 surfaces as a download failure, not as this error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
f3689c1481 |
fix(producer): scope and harden the markup-payload sniff
Review follow-up on the STUDIO-5433 defense.
Correctness
- The sniff ran above the documented video/audio failure split, so an
<audio> src that resolved to an HTML payload aborted the whole render
instead of degrading to duration 0. It now runs inside the same try, so
video surfaces the typed error while audio still drops out, with a
warning naming the element.
- Raw fs errors (EISDIR on a directory src, EACCES, the existsSync->open
ENOENT race, EMFILE) escaped and failed the compile with an unclassified
error carrying an unredacted temp path. The sniff is now a classifier that
never throws: an unreadable file reports "not markup" and the real probe
produces the real error.
- Elements whose duration the compiler never resolves (a data-end video, a
looping audio) skipped the sniff entirely, so the original ffprobe error
still escaped, and looping audio was reported as owner "system" after
every frame had been captured. Video is now caught in the asset preflight,
which sees every local src regardless of authored timing; audio is
classified per-element in audioMixer as source/invalid_media/owner "user",
keeping audio failures non-fatal as they already were.
- Detection is a byte-level check for a leading "<" (BOM-, whitespace- and
NUL-tolerant, looped read) instead of a <!doctype|<html|<?xml string
prefix, which missed a NUL-prefixed payload, >256B of leading whitespace,
UTF-16-encoded HTML, and a prolog-less <svg. No supported container starts
with "<", so the allowlist no longer grows per payload shape.
- finally { await fh.close() } could replace the in-flight typed error with
the close error.
Routing and privacy
- MARKUP_NOT_MEDIA is now in SAFE_RENDER_ERROR_CODES, the Lambda terminal
name map, the CDK and SAM non-retryable plan lists, and the Cloud Run
non-retryable set, and the class carries owner/retryable. Previously the
API emitted errorCode: undefined and a deterministic authoring bug burned
the full distributed retry budget.
- The message no longer carries 32 raw payload bytes or the src.
redactTelemetryString preserves host and path for HTTP srcs, so
per-tenant CDN paths reached a message the server forwards to clients.
Correlation is a sha256 element fingerprint, matching
AssetMediaTypeMismatchError.
- The message names both causes (unresolved nested-composition URL, or an
HTML/XML error page served as 200) rather than misdiagnosing an S3 403
body as an authoring bug.
Tests
- Byte-level detection is unit-tested in engine: markup shapes, BOMs,
UTF-16, nine container signatures, unreadable inputs.
- Replaced the tautological assertions. The old checks for "html" in and
"moov" absent from a fixed message template could not fail for any input.
- New coverage for audio degradation, the audioMixer classification, the
preflight video/image/audio split, and the API error metadata.
- The sibling htmlCompiler.mediaType failure was a vitest-under-bun runner
mismatch, not a missing ffmpeg binary. It passes, including the 4-wide
probe-semaphore invariant the sniff now runs inside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
1d636f603c |
refactor(producer): share plan execution builder (#2906)
## What Refactor distributed planning around one shared local execution-plan builder: - `buildLocalExecutionPlan()` now owns compile/probe/extract/audio/freeze. - Legacy `plan()` remains a deprecated v1 transport wrapper. - Plan v2 calls the shared builder directly and publishes through the existing manifest/CAS contract. - Add neutral `createPlanV2FromExecutionPlan()`, `publishPlanV2FromExecutionPlan()`, `getPlanV2ExecutionPlanHash()`, and `PLAN_PROTOCOL_V1` names. - Retain deprecated v1-named exports and wire aliases. - Recommend explicit Plan v2 opt-in for new producer, Lambda, and Cloud Run integrations. ## Why Plan v2 previously looked like it invoked a v1 planner even though v1 and v2 share the same frozen local execution representation. This removes that migration-era coupling while preserving the public minor-version compatibility contract. ## How The shared builder returns neutral internal execution-plan fields. The v1 wrapper maps those fields back to the existing `PlanResult`; the v2 publisher consumes them directly. Compatibility is intentional and covered by exact shape tests: - omitted `planProtocol` still serializes/selects `"v1"`; - v1 layouts, descriptor-less decoding, event unions, workflow branches, and exports remain; - the v1 descriptor JSON is byte-identical and `CURRENT_PLAN_PROTOCOL` is an identity-preserving alias; - v2 manifest bytes, key order, hash framing, and `sourcePlanV1Hash` wire key remain unchanged; - no enumerable neutral hash field was added to manifests or returned result objects; - v1/v2 result objects, cloud event payloads, and SDK handle key sets remain unchanged. ## Test plan - Focused Plan v1/v2/protocol/export/size compatibility: 141 passed - `@hyperframes/core`: 1,419 passed - `@hyperframes/producer` unit lane: 990 passed - `@hyperframes/aws-lambda`: 140 passed - `@hyperframes/gcp-cloud-run`: 101 passed - Producer, Lambda, and Cloud Run typechecks - Repository-wide lint, format check, workspace/package-subpath checks - Full workspace build - `git diff --check` - [x] Unit tests added/updated - [ ] Manual testing performed - [x] Documentation updated (if applicable) |
||
|
|
20f4f8f49c |
fix(producer): retry transient deterministic font fetches (#2865)
* fix(producer): retry transient deterministic font fetches * fix(producer): secure Lambda font cache directory |
||
|
|
557d82b6a9 |
fix(producer): validate distributed video metadata (#2839)
## What - enforce a finite, validated `meta/videos.json` contract shared by Plan v1 and Plan v2 - preserve authored finite ends and source-derived trim-aware ends; bound any still-open end at the validated composition end - fail distributed planning when any declared video source did not extract instead of publishing a blank-capable plan - make the v1 chunk reader reject malformed/null video timing before frame injection - route deterministic video-source/metadata failures as non-retryable in AWS and GCP while retaining retries for transient extraction failures ## Why An open-ended video whose remote source could not be resolved retained `Infinity` through planning. Plan v2 correctly rejected that value, while Plan v1 serialized it as `null`; the v1 frame lookup could then suppress injected frames and silently produce incorrect output. The invariant belongs at the shared metadata boundary. Both protocols must receive identical finite timing, and unavailable sources must fail closed before plan publication. ## Test plan - [x] producer distributed planning, metadata, v1 chunk boundary, Plan v2 conversion/materialization, and public exports - [x] core runtime media semantics (authored slots, natural duration, looping, non-looping hold) - [x] engine video extraction and frame lookup - [x] AWS Lambda/CDK/SAM and GCP Cloud Run error normalization/retry classification - [x] producer, core, engine, AWS, and GCP typechecks/builds - [x] formatting, oxlint, tracked-artifact, fallow, and commit hooks - [x] exact incident composition replayed through the AWS Lambda handler's Lambda-local path in a Lambda-like container; Plan v1 and Plan v2 both fail closed as `VIDEO_SOURCE_UNRENDERABLE` during planning, before plan publication - [x] full PR CI, including all nine regression shards and Windows render/tests No production flags or deployment/release workflows are changed. |
||
|
|
96cafb47c6 | fix(producer): fallback distributed capture safely | ||
|
|
2a284a8e3a | fix(gcp): enforce effective BeginFrame capture | ||
|
|
74d7bfde48 | feat(gcp-cloud-run): publish plan v2 directly to GCS | ||
|
|
0499a5cbcb | fix(gcp-cloud-run): normalize v2 integrity codes (#2790) | ||
|
|
f9f00b0efc | feat(producer): version distributed plan protocol (#2777) | ||
|
|
942db16fcd | fix(cloudrun): publish adapter contract | ||
|
|
4da567df22 |
feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda (issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble) are unchanged; this package is the storage/compute/orchestration glue. Package: Cloud Run handler (one image, three actions), runs under bun; GCS transport; in-image chrome-headless-shell resolver; client SDK (renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile; Cloud Workflows definition; Terraform module; CLI cloudrun deploy|sites|render|render-batch|progress|destroy with --output-resolution and --strict-variables; 62 unit tests + docs + live smoke script. Shared extraction (removes ~640 lines of adapter duplication): move the cloud-agnostic config validator + content-hash into producer/distributed; both adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`, failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run to the root `build` filter so its dist exists for publish + runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install The regression test image runs `bun install --frozen-lockfile` after copying each workspace package.json individually. The CLI now depends on @hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to resolve it unless its manifest is present. Add the COPY line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): add machine-sizing flags to `cloudrun deploy` Closes the parity gap with `lambda deploy` (which exposes --memory etc.). `cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout into the Terraform apply; omitted flags keep the module defaults (4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address PR review (security, waste, limits, alerts) - server.ts: bucket-allowlist guard no longer fails open silently. Unset env logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces. - server.ts: stop double-shipping audio.aac. It already rides in the plan tarball every consumer downloads, so drop the redundant standalone upload (plan) + re-download/overwrite (assemble); assemble reads it from the untar, falling back to a supplied AudioGcsUri for compat. - server.ts: chunk extension via path.extname() instead of slice(lastIndexOf). - workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20) — Cloud Workflows hard-caps concurrent iterations at 20. - Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break the image rebuild. - terraform: add min_instances var (default 0); add a workflow-failure alert (finished_execution_count status=FAILED) alongside the request-count one. - costAccounting: document that displayCost excludes GCS storage/egress. Verified against the actual APIs: @google-cloud/workflows@4.4.0 ICreateExecutionRequest has no executionId (so the idempotency-token suggestion isn't available in this client); Workflows concurrency cap is 20; failure metric is workflows.googleapis.com/finished_execution_count (status label). 174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding - workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE → PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the opposite cause), misleading anyone triaging the alert. - workflow.yaml: forward Config.cfr to the assemble step (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler but never sent, so exact-CFR was silently off for every Cloud Run render. Uses the same `in`-operator guard already proven in the retryable predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(release): include gcp-cloud-run in set-version PACKAGES list set-version.ts (driven by release:prepare) bumps an explicit package list to the shared version on each release. gcp-cloud-run was wired into the build + publish.yml but missing here, so a release would leave it at a stale version and publish.yml would push the wrong version. Add it so the new package version-bumps + publishes in lockstep with the others. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |