Commit Graph
26 Commits
Author SHA1 Message Date
James RussoandClaude Opus 5.5 39347da9de fix(cli): leave files from outside the project out of Lambda and Cloud Run site uploads (#8060)
* fix(cli): leave files from outside the project out of Lambda and Cloud Run site uploads

A site upload for `lambda` or `cloudrun` followed every symlink in the project and
shipped what it pointed at, wherever that was on the machine. `publish` and `cloud
render` already ask first. Site uploads now leave a file or folder linked in from
outside the project out and name each one; `--include-outside-files` (or
`includeOutsideFiles` in the SDK's deploySite, renderToLambda and renderToCloudRun)
uploads them. publishProjectArchive called without a prepared archive now leaves
outside files out too, since nothing approved them.

— Rames

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(producer): drop the new option comments the comment ratchet refuses

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 09:25:25 +00:00
Michael Wang 7aaeb93716 fix(producer): the plan copy checks every project entry, whatever its name (#7906)
* fix(producer): the plan copy checks every project entry, whatever its name

A top-level name that starts with two dots, such as ..assets, is an ordinary
project entry. The plan copy treated it as outside the project and skipped the
link-resolving containment check, so a link with such a name had its target
copied into compiled/. Only the project folder itself now skips the check, and
the outside test compares whole path segments.

* fix(aws-lambda,gcp-cloud-run): untar leaves out symbolic links and special files

A project archive can come from another person. A symbolic link in it could
point anywhere on the render host, and the planner would copy its target. Untar
now unpacks only files, folders and hard links. node-tar already keeps a hard
link's target inside the destination folder. The CLI's site upload follows
symbolic links when it packs, so its archives hold none.

* fix(aws-lambda,gcp-cloud-run): untar turns hard links into separate plain files

The site upload packs two hard-linked project files as one file and a hard
link. Untar still unpacks both, then gives each hard-linked file its own copy,
so the extracted project holds no links of any kind and loses no file.

* fix(aws-lambda,gcp-cloud-run): copy hard-linked files without mkdtemp; untar test in its own file

* fix(cloud): keep the hard-link copy name short so long file names still unpack
2026-11-16 06:38:08 +00:00
Michael Wang 014326b40b fix: every composition file server refuses credential and tool-state files (#7889)
* wip: paused before review

* test: cover credential files on the preview routes and the deploy notice

* refactor: name the inspector's file lookup for what it serves

* fix(cli): a link to a private file outside the project keeps its own left-out reason

* fix: escape the upload notice and skip private files from old manifests
2026-11-16 01:42:08 +00:00
Michael WangandClaude Opus 5.5 55b5147098 fix: Cloud Run renders reach only public addresses; frame cache folder is private to its user (#7777)
* fix(gcp-cloud-run): render pages cannot reach the service's own dispatch endpoint

The Cloud Run image now sets PRODUCER_RESTRICT_PAGE_NETWORK=true, so a
rendered page reaches only its own project server and public addresses,
not loopback ports or the metadata server. The dispatch endpoint also
refuses any POST that carries an Origin header: browsers send one on every
POST and Cloud Workflows never does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(engine): the default frame cache folder is private to the user who renders

The default extraction cache sits in the shared temp folder. It is now
created with mode 0700, an existing one owned by the same user is closed
to others (its entries stay, so cache hits carry over), and one that is a
link or belongs to another user is not used: that render runs without the
cache and prints a warning. The animated GIF cache under the same folder
follows the same rule. A folder set with HYPERFRAMES_EXTRACT_CACHE_DIR is
left as it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(gcp-cloud-run): a Terraform variable turns the page network restriction off

restrict_page_network (default true) sets PRODUCER_RESTRICT_PAGE_NETWORK on
the service. The package README and docs say what the restriction refuses
and how to turn it off for trusted compositions that need private hosts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(engine): restore an unset TMPDIR as unset after the cache-root tests

Assigning undefined to process.env stores the string "undefined", so on a
runner with no TMPDIR every later test made its temp folder under ./undefined.

* test(engine): the frame-extractor link test also restores an unset TMPDIR as unset

* chore(engine): raise the packed size budget to 500000

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:34:11 +00:00
PaprikaChenandClaude Opus 5.5 1a9fb8acb6 feat: opt-in 4:4:4 chroma for SDR MP4 masters (#5820)
* feat(engine): decouple transparent background from png capture format

PNG capture used to imply a cleared page background. A new
CaptureOptions.transparentBackground lets lossless opaque PNG capture
(4:4:4 MP4 output) keep the page background. Absent, it still follows
format === "png", so alpha output and motion blur are unchanged.

Opaque PNG skips drawElement, whose verification and worker-encode
paths are JPEG-only.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(engine): add 4:4:4 chroma option to getEncoderPreset

getEncoderPreset takes an optional chromaSubsampling ("420" by
default). "444" returns yuv444p for SDR MP4, so libx264 encodes the
High 4:4:4 Predictive profile and odd sizes need no even padding.
WebM, MOV and HDR presets ignore it. The ChromaSubsampling type is
exported for the producer.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(producer): chromaSubsampling render option with lossless png capture

RenderConfig.chromaSubsampling ("420" by default) opts an SDR MP4
render into 4:4:4. "444" encodes libx264 yuv444p and captures every
frame as opaque PNG: Chrome's quality-100 JPEG is already 4:2:0 on
beginFrame and screenshot capture, and drawElement can fall back to
screenshot mid-render, so PNG is the only capture that keeps the
colour. drawElement is turned off for these renders.

The probe and main capture options share resolveCaptureSurface, so
they cannot disagree on format, quality or background.

Formats other than mp4, GPU encoding and HDR output are rejected
before any work starts, auto-detected HDR sources fail before capture,
and distributed requests refuse 444. The default render is unchanged.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(producer,studio-server): accept chromaSubsampling on render routes

The shared parseEncoderOptions validates chromaSubsampling and answers
400 for "444" with a non-mp4 format, gpu, or force-hdr, so a request
that would come back 4:2:0 never starts a render. The Studio render
route, both Studio adapters and the producer HTTP server forward it.
The producer server also refuses "444" with outputDynamicRange "hdr",
which parseEncoderOptions cannot see.

No Studio UI control; the option is route-level like crf and hdrMode.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(cli): add --chroma-subsampling flag

`hyperframes render --chroma-subsampling 444` opts an SDR MP4 render
into full-resolution chroma (yuv444p with lossless PNG capture). The
default stays 420. The plan rejects 444 with a non-mp4 format, --gpu
or --hdr before anything renders, with a usage error that names the
conflicting flag.

The value reaches the local render request and, only when it is 444,
the Docker argv, so default Docker invocations are unchanged.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: document 4:4:4 chroma subsampling

Add a "4:4:4 masters" section to the rendering guide: when to use
--chroma-subsampling 444, why it captures PNG, the measured chroma gain
(large for text and UI, none for footage and gradients), its limits,
and a player compatibility table to be filled in from testing.

Document the flag in the CLI reference, the built-in rendering docs and
the hyperframes-cli skill reference (with the regenerated skills
manifest), and the chromaSubsampling option in the producer docs.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: add 4:4:4 player compatibility results

Fill the 4:4:4 compatibility table from testing H.264 High 4:4:4
Predictive output. Chrome 155, Edge 155, VLC 3.0, Safari on iOS 26.6
and Safari 26.6 on macOS play it, and so does AVFoundation on macOS
26.6, the framework QuickTime Player plays through. Firefox 157 on
Windows and both Windows media players do not.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: note the measured render cost of 4:4:4 output

Replace the vague "slower capture" note with measured numbers: a text
and UI composition rendered in about twice the default time, full-frame
footage in about three and a half times, because PNG encoding cost grows
with image detail.

Refs #5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(producer): reject chromaSubsampling 444 in distributed render config

Distributed configs have no chroma field and validateDistributedRenderConfig
ignored unknown keys, so a JS caller passing chromaSubsampling: "444" to the
Lambda or Cloud Run SDK got a 4:2:0 file with no error. Reject it, and make
the rendering guide say so precisely.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(producer): check a 444 render reaches the encoder as yuv444p

Every helper had tests, but nothing checked the orchestrator passes
chromaSubsampling through, so dropping it from the encoder preset or capture
surface would turn 444 renders back into 4:2:0 with all tests green. Render a
small fixture with "444" and with the default and check pix_fmt with ffprobe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(producer): move the 444 drawElement gate into a helper

Keeps runCompileStage under the fallow complexity threshold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: keep comment share at or below the merge-base

comment-ratchet failed on 11 files this PR touched. Keep the 4:4:4
rationale once in chromaSubsampling.ts, move the distributed-render
reason into its error message, and cut the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(producer): check a 444 render captures full-resolution chroma

The encoder check alone passes when chromaSubsampling is dropped from the
capture surface: frames are captured as JPEG and encoded yuv444p with
4:2:0 colour. Put a red/blue edge at an odd x, inside a 4:2:0 chroma
pair, and require the two pixels to differ in Cb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(producer): trim the chromaSubsampling.ts header to the package comment share

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: finish switching to capturesAlpha from #6262

The merge that adopted main's transparent / capturesAlpha left these files
uncommitted, so resolveCaptureSurface still returned transparentBackground.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(engine): use transparent / capturesAlpha from #6262

main added CaptureOptions.transparent and capturesAlpha (#6262), which do
what this PR's transparentBackground did. Use them, keep the PNG check in
drawElementWorkerEncodeEligible, and re-pin test-classification.mjs and
the skills manifest against current main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(producer): cut the 4:4:4 comment blocks to one to three lines

The reasoning lives in the PR description with the measurements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: reconcile 4:4:4 chroma with latest main

Keep main's video-layer capture overrides, template entry resolution,
ProRes profile parsing and blur-differential tests alongside the chroma
option, and fold the encoder sub-parsers into one loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: keep main's compile-stage tests and regenerate the skills manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:20:55 +00:00
Miguel Ángel a9a8d89377 fix(engine): a render no longer finishes without an image that loaded late (#7304)
* fix(engine): a render no longer finishes without an image that loaded late

* style(engine): format the frame gate

* fix(engine): an image outside the frame no longer stops a render

* fix(engine): a render stops and warns when a frame's image never arrives, with no cloud retries

* fix(producer): a cloud chunk keeps the missing image's error name so retries stop
2026-11-06 20:11:01 +00:00
James Russo 7f48a6894c feat!: remove Plan v1 distributed transport (#6970)
* feat!: remove Plan v1 distributed transport

BREAKING CHANGE: Cloud adapters accept only Plan v2; producer v1 exports are removed.

* ci: authorize Plan v1 cleanup deletions

* docs: avoid citing generated smoke result file

* fix(gcp): reject unsupported plan protocols in workflow

* ci: preserve gcp package size reduction

* fix: reject stale Plan v1 SDK options

* chore: remove retired Plan parity scripts
2026-11-02 04:13:57 +00:00
Miguel Ángel 54d0af57eb feat(engine): render sets a keyframe interval and a peak bitrate cap (#6444)
* feat(engine): encoders cap peak bitrate and place keyframes on a set interval

* style(engine): shorten the rate-cap comment

* style(engine): drop the doc lines the comment ratchet flagged

* feat(producer): mp4 renders honour a keyframe interval and a peak bitrate cap on every encoder path

* style(producer): shorten the delivery encoder file headers

* style(producer): drop a comment line from the segment manifest

* style(producer): shorten the integration test header

* test: delivery encoder tests follow the ffprobe argv contract and run in the integration lane

* feat: the render request, server and Lambda plan carry the keyframe interval and peak bitrate cap

* fix(studio): the Studio render route passes the keyframe interval and bitrate cap to the encoder

* chore(studio-server): drop a comment the helper's name already says

* feat(cli): render and lambda render take --keyframe-interval, --max-bitrate and --buffer-size
2026-10-27 00:29:06 +00:00
Miguel Ángel 992df8b203 perf(cloud): move project bytes once through shared content storage (#5772)
* perf(producer): distributed chunks download only the videos they show

A project video under compiled/ that chunks extract their own frames from
now goes only to the chunks that show it, like video-sources/ files. A
chunk skips the readiness wait for a deferred video it does not show, so
it no longer logs media_load_failed for a file it was never given.

* fix(producer): cloud site uploads pack the files their id hashes, symlinks included

* perf(aws-lambda): chunks read only the byte ranges of a video they need

A chunk that extracts its own frames from a deferred project video now
reads it from S3 by byte range through a short-lived pre-signed URL,
instead of downloading and hashing the whole object first. S3 attests the
object's stored SHA-256 before the URL is signed; every other artifact is
still downloaded and verified as before.

* perf(aws-lambda): upload project files once to a shared content-addressed area

* chore(producer): keep planV2 comment share at the base level

* fix(engine): verify the TLS certificate of https video inputs and never surface their signed query

* fix(aws-lambda): renders re-upload a site's expired files instead of failing

* perf(aws-lambda): the plan step probes project videos in place instead of downloading them

* fix(engine): https video reads reconnect and time out, and redaction covers wrapped causes

Signed-URL reads now pass -rw_timeout, -reconnect and -reconnect_delay_max (both Lambda builds accept them) and ffmpeg also -reconnect_on_network_error, so a transient S3 error or a stalled read fails fast or recovers instead of waiting out the process timeout. Query redaction now descends into error.cause.

* fix(aws-lambda): share entry bundles and ratchet package budget

* fix(aws-lambda): site ids uploaded before files.json still render

A site id from an earlier sites create has only sites/<id>/project.tar.gz.
renderToLambda now falls back to that tarball, which the plan step still
reads, and when neither key exists it fails before starting with both keys
named and the fix (run sites create again).

* fix(aws-lambda): renders fail before upload on a stack too old to read sites

A stack deployed before files.json sites untars the site file in its plan
step and fails after four retries. renderToLambda now reads the state
machine definition first and stops with STACK_TOO_OLD and the redeploy
command when the plan task does not list SITE_MANIFEST_INVALID, which new
SAM and CDK stacks now treat as terminal. The README's upgrade section
drops the planProtocol v1 fallback, which hits the same failure.

* docs(aws-lambda): document shared content storage and the upgrade path

The deploy guide still described the tarball HeadObject short-circuit. It
now covers the cas/sha256/ store and its 30-day lifecycle rule, the
STACK_TOO_OLD, SITE_FILES_MISSING and SITE_NOT_FOUND errors, that old site
ids keep working, and that every project uploads once more after the
upgrade. A test keeps a troubleshooting entry for each pre-submit error.

* fix(producer): chunks log each video they read by URL, without its query

A chunk reading a video straight from the bucket left no trace in its log.
It now logs one line per remote source with the path and the URL minus its
query string, so signatures never reach the log.

* fix(engine): ffmpeg errors never keep part of a signed URL's query

A process's stderr tail was trimmed before its URLs were redacted, so a
cut inside a signed URL left a query fragment the URL pattern no longer
matched, and the fragment reached the error. The stderr tail is now
redacted before every trim, and the redactor also strips a signing
parameter cut off from its URL.

* fix(aws-lambda): comments name the site manifest instead of a file path

* fix(engine): stderr redaction survives chunk splits, and every ffmpeg spawn uses it

The stderr tail was redacted on each chunk and then cut, so once a cut
severed a long URL's scheme, the rest of its signature in the next chunk
matched neither pattern. The tail is now kept raw with 2x slack and
redacted only when the text is produced, then cut to size. The unused
producer audioExtractor (its own spawn, unredacted stderr) is deleted and
the parity harness's PCM decode goes through ManagedChildProcess; a test
fails on any ffmpeg/ffprobe spawn in engine, producer or aws-lambda that
does not.

* chore(producer): record the audioExtractor deletion as intended

* fix(engine): drop a signature fragment the stderr tail cut from its URL

The raw tail is kept at twice the cap, but that cut can itself land inside a
signature, and when redaction shrinks the rest below the cap the fragment
survived. After a cut, the severed token is dropped before redaction.

* refactor(engine): name the severed-token drop instead of commenting it

* test(engine): a cut-off URL leaves nothing behind; aws-lambda size budget to measured

* fix(producer): chunk URL-read tests use main's frame list; producer size budget to measured

* chore(producer): size budget at the measured size, no headroom

* chore(producer): size budget back to measured x1.05, the check's own rule
2026-10-19 19:23:51 +00:00
Miguel Ángel c69fd2c6ec perf(producer): size distributed chunks to the worker limit and concurrency (#5765)
* perf(producer): size distributed chunks to the worker limit and concurrency

* chore(cli): drop a comment the field name already says
2026-10-17 08:45:47 +00:00
Miguel Ángel d5162beb50 refactor(core): one owner for the distributed fps set (#5756)
The integer rates Lambda and Cloud Run accept were written out by hand in
about 25 places across core, engine, producer, the adapters and the CLI.
They now come from DISTRIBUTED_FPS / DistributedFps / isDistributedFps in
@hyperframes/core. Engine config snapshots accept 25 like the rest.
2026-10-17 08:45:41 +00:00
Robin EbersandCursor cc9e1bb33a feat(producer): distributed renders accept 25 fps (#5423)
Lambda and Cloud Run renders refused 25 fps although it is an integer
rate like 24, 30 and 60, and PAL footage is 25 fps. The distributed fps
set is now {24, 25, 30, 60} in the producer's config validation, the
plan v2 manifest, the render request mapping, the Lambda and Cloud Run
event types, the CLI's lambda and cloudrun commands and the regression
harnesses.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 22:00:32 -07:00
James Russo 17a2a00ed5 feat(cloud): default distributed plans to v2 (#3311)
* feat(cloud): default distributed plans to v2

* fix(cloud): address plan v2 review feedback

* fix(examples): document explicit v2 samples
2026-08-17 17:24:31 -04:00
Miguel Ángel dc4383113c fix(producer): mix audio into a container that can record encoder delay (#3200)
* fix(producer): mix audio into a container that can record encoder delay

Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz)
after its authored `data-start`, against a frame-accurate video track.

The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix
was written to a raw ADTS `.aac` file, which has nowhere to record that delay,
so it decoded as real leading silence and every stage downstream preserved it
faithfully. Measuring each intermediate localises it precisely: the source WAV
is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and
mux stages inherit it unchanged. The filter graph itself is correct - run by
hand to PCM it lands on the authored start.

Switch the artifact to an MP4-family container, which stores the delay as an
edit list that decoders strip. Same codec, same bitrate, so no size or quality
change.

The filename is a contract shared by three consumers - the mux input, the
distributed plan artifact, and the PNG-sequence sidecar handed to users for
NLE ingest - and its extension is what selects the muxer. Give it one owner in
the engine rather than five literals, so those consumers cannot drift onto
different containers.

Note for reviewers: this renames the distributed plan's audio artifact, which
is an on-disk contract between the plan writer and the assembler. Both move
together here, but a plan written by an older build would not be found by a
newer assembler. Flagging in case that mixed-version window matters for how
these are deployed.

* fix(cloud): read the plan audio artifact name from the producer contract

The aws-lambda and gcp-cloud-run adapters each restated the plan's audio
filename in five places, so renaming it in the producer left them looking for a
file that is no longer written. CI caught it: the gcp dispatch test asserting a
plan has no audio artifact started seeing one.

Export the name from `@hyperframes/producer/distributed` and consume it in both
adapters. This is the same failure the constant exists to prevent, one package
boundary further out: a literal that drifts from the writer's is a silently
missing audio track rather than a loud error, because both call sites only ever
ask whether the file exists.

* fix(cloud): accept a legacy plan's audio artifact name for one release

Review raised a rolling-deploy window I had flagged but left undecided: `plan`
and `assemble` are separate invocations bridged by object storage, so a
pre-rollout planner can be paired with a post-rollout assembler. Both readers
locate the artifact by existence alone, which makes that pairing a silently
muted video rather than an error. That is reachable enough to be worth two
lines, so reads now accept the old name while writes only ever emit the new one.

Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`)
rather than four call sites, marked for deletion one release out.

Also fixes a hole in the first pass of this: the plan-v2 materializer matched
either name but then joined the CURRENT one, so a legacy plan resolved to a path
that was never written. It now joins the artifact's own name.

Review nits in the same pass: correct the pad-branch docstring, which still
described a concat-copy shape the pad branch stopped using when it moved to
apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so
it cannot model a shape that reintroduces the priming delay.

* test(producer): rebake the missing-host-comp-id golden without the audio delay

The pinned reference was rendered before this branch, so it carries the 1024
sample encoder-priming delay in its audio. With the delay gone the correct audio
now sits ahead of the reference and the harness's envelope correlation drops
below its floor.

Cross-correlating the old and new references at native 48 kHz gives a lag of
exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved
by exactly the amount this branch removes. Regenerated inside the CI container
(Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches
the encoder CI will compare against - the container reproduced CI's failure to
the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and
passes at correlation 1.0 after it.

Note for archaeology: the new reference is also 3 dB louder than the old one.
That gap is not from this branch - `main` and this branch render the fixture at
the same level - it is pre-existing drift the reference had accumulated, which a
scale-invariant correlator could never see. The rebake absorbs it.

Only output.mp4 is updated. `--update` also rewrites compiled.html, but that
diff is embedded-font churn with no bearing on the comparison, which reports
"Failed at compilation: 0" either way.

* test(producer): rebake the variables-prod golden without the audio delay

Same cause as the missing-host-comp-id rebake, caught by shard-8 once the
earlier shard stopped failing and the rest of the matrix could run: this
reference also carries the encoder-priming delay this branch removes.

Reproduced in the CI container to the digit (correlation 0.42704173048439215,
lagWindows -12), rebaked there, and it now passes at correlation 1.0.

Worth recording: the shift here is 2048 samples (42.67 ms) at correlation
0.99983, exactly twice the 1024 of the other fixture. The delay compounds once
per un-compensated AAC generation, and this fixture's audio needs its duration
normalized, so it takes the pad/trim branch's re-encode and picks up a second
frame of priming on top of the mixer's. So the pre-fix error was not a fixed
21 ms - it grew with the number of times the audio was re-encoded.

All nine shards ran in that CI round with only this one failing, so the matrix
has now covered every fixture against this change.
2026-08-11 00:12:43 -04:00
Vance IngallsandClaude Opus 5 349c066a83 fix(producer): classify JSON error bodies as non-media sources too
A source that answers with a JSON error body still reached ffprobe and
produced `moov atom not found`. Replicate returns
`{"detail": "requested file not found"}` for a dead asset, and a gateway
in front of it can relay that body with a success status.

The sniff now treats `<`, `{`, or `[` as the opening byte of a text
document. No supported container starts with any of them, so this is the
same trade as before: three bytes instead of an allowlist that grows one
entry per payload shape observed in production.

Renamed accordingly, since the class now covers JSON as well as markup:
MARKUP_NOT_MEDIA -> NOT_MEDIA_PAYLOAD, MarkupNotMediaError ->
NotMediaPayloadError, markupPayload.ts -> notMediaPayload.ts. Registry
entries in the Lambda name map, the CDK and SAM plan lists, the Cloud Run
set, and SAFE_RENDER_ERROR_CODES move with it.

Also documents the reachability boundary on the error class: only a 2xx
response gets here. `downloadToTemp` rejects 404/410 as `http_not_found`
before writing a byte, and every ffprobe input is local because
videoFrameExtractor downloads http srcs first. So the shapes this
classifies are soft-404 and interstitial HTML, S3/CloudFront error
documents, and JSON API error bodies -- each served with a success
status. A genuine 404 surfaces as a download failure, not as this error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 03:16:11 -07:00
Vance IngallsandClaude Opus 5 f3689c1481 fix(producer): scope and harden the markup-payload sniff
Review follow-up on the STUDIO-5433 defense.

Correctness

- The sniff ran above the documented video/audio failure split, so an
  <audio> src that resolved to an HTML payload aborted the whole render
  instead of degrading to duration 0. It now runs inside the same try, so
  video surfaces the typed error while audio still drops out, with a
  warning naming the element.
- Raw fs errors (EISDIR on a directory src, EACCES, the existsSync->open
  ENOENT race, EMFILE) escaped and failed the compile with an unclassified
  error carrying an unredacted temp path. The sniff is now a classifier that
  never throws: an unreadable file reports "not markup" and the real probe
  produces the real error.
- Elements whose duration the compiler never resolves (a data-end video, a
  looping audio) skipped the sniff entirely, so the original ffprobe error
  still escaped, and looping audio was reported as owner "system" after
  every frame had been captured. Video is now caught in the asset preflight,
  which sees every local src regardless of authored timing; audio is
  classified per-element in audioMixer as source/invalid_media/owner "user",
  keeping audio failures non-fatal as they already were.
- Detection is a byte-level check for a leading "<" (BOM-, whitespace- and
  NUL-tolerant, looped read) instead of a <!doctype|<html|<?xml string
  prefix, which missed a NUL-prefixed payload, >256B of leading whitespace,
  UTF-16-encoded HTML, and a prolog-less <svg. No supported container starts
  with "<", so the allowlist no longer grows per payload shape.
- finally { await fh.close() } could replace the in-flight typed error with
  the close error.

Routing and privacy

- MARKUP_NOT_MEDIA is now in SAFE_RENDER_ERROR_CODES, the Lambda terminal
  name map, the CDK and SAM non-retryable plan lists, and the Cloud Run
  non-retryable set, and the class carries owner/retryable. Previously the
  API emitted errorCode: undefined and a deterministic authoring bug burned
  the full distributed retry budget.
- The message no longer carries 32 raw payload bytes or the src.
  redactTelemetryString preserves host and path for HTTP srcs, so
  per-tenant CDN paths reached a message the server forwards to clients.
  Correlation is a sha256 element fingerprint, matching
  AssetMediaTypeMismatchError.
- The message names both causes (unresolved nested-composition URL, or an
  HTML/XML error page served as 200) rather than misdiagnosing an S3 403
  body as an authoring bug.

Tests

- Byte-level detection is unit-tested in engine: markup shapes, BOMs,
  UTF-16, nine container signatures, unreadable inputs.
- Replaced the tautological assertions. The old checks for "html" in and
  "moov" absent from a fixed message template could not fail for any input.
- New coverage for audio degradation, the audioMixer classification, the
  preflight video/image/audio split, and the API error metadata.
- The sibling htmlCompiler.mediaType failure was a vitest-under-bun runner
  mismatch, not a missing ffmpeg binary. It passes, including the 4-wide
  probe-semaphore invariant the sniff now runs inside.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 17:38:51 -07:00
James Russo 1d636f603c refactor(producer): share plan execution builder (#2906)
## What

Refactor distributed planning around one shared local execution-plan builder:

- `buildLocalExecutionPlan()` now owns compile/probe/extract/audio/freeze.
- Legacy `plan()` remains a deprecated v1 transport wrapper.
- Plan v2 calls the shared builder directly and publishes through the existing manifest/CAS contract.
- Add neutral `createPlanV2FromExecutionPlan()`, `publishPlanV2FromExecutionPlan()`, `getPlanV2ExecutionPlanHash()`, and `PLAN_PROTOCOL_V1` names.
- Retain deprecated v1-named exports and wire aliases.
- Recommend explicit Plan v2 opt-in for new producer, Lambda, and Cloud Run integrations.

## Why

Plan v2 previously looked like it invoked a v1 planner even though v1 and v2 share the same frozen local execution representation. This removes that migration-era coupling while preserving the public minor-version compatibility contract.

## How

The shared builder returns neutral internal execution-plan fields. The v1 wrapper maps those fields back to the existing `PlanResult`; the v2 publisher consumes them directly.

Compatibility is intentional and covered by exact shape tests:

- omitted `planProtocol` still serializes/selects `"v1"`;
- v1 layouts, descriptor-less decoding, event unions, workflow branches, and exports remain;
- the v1 descriptor JSON is byte-identical and `CURRENT_PLAN_PROTOCOL` is an identity-preserving alias;
- v2 manifest bytes, key order, hash framing, and `sourcePlanV1Hash` wire key remain unchanged;
- no enumerable neutral hash field was added to manifests or returned result objects;
- v1/v2 result objects, cloud event payloads, and SDK handle key sets remain unchanged.

## Test plan

- Focused Plan v1/v2/protocol/export/size compatibility: 141 passed
- `@hyperframes/core`: 1,419 passed
- `@hyperframes/producer` unit lane: 990 passed
- `@hyperframes/aws-lambda`: 140 passed
- `@hyperframes/gcp-cloud-run`: 101 passed
- Producer, Lambda, and Cloud Run typechecks
- Repository-wide lint, format check, workspace/package-subpath checks
- Full workspace build
- `git diff --check`

- [x] Unit tests added/updated
- [ ] Manual testing performed
- [x] Documentation updated (if applicable)
2026-07-30 17:41:02 -07:00
James Russo 20f4f8f49c fix(producer): retry transient deterministic font fetches (#2865)
* fix(producer): retry transient deterministic font fetches

* fix(producer): secure Lambda font cache directory
2026-07-28 22:49:03 -07:00
James Russo 557d82b6a9 fix(producer): validate distributed video metadata (#2839)
## What

- enforce a finite, validated `meta/videos.json` contract shared by Plan v1 and Plan v2
- preserve authored finite ends and source-derived trim-aware ends; bound any still-open end at the validated composition end
- fail distributed planning when any declared video source did not extract instead of publishing a blank-capable plan
- make the v1 chunk reader reject malformed/null video timing before frame injection
- route deterministic video-source/metadata failures as non-retryable in AWS and GCP while retaining retries for transient extraction failures

## Why

An open-ended video whose remote source could not be resolved retained `Infinity` through planning. Plan v2 correctly rejected that value, while Plan v1 serialized it as `null`; the v1 frame lookup could then suppress injected frames and silently produce incorrect output.

The invariant belongs at the shared metadata boundary. Both protocols must receive identical finite timing, and unavailable sources must fail closed before plan publication.

## Test plan

- [x] producer distributed planning, metadata, v1 chunk boundary, Plan v2 conversion/materialization, and public exports
- [x] core runtime media semantics (authored slots, natural duration, looping, non-looping hold)
- [x] engine video extraction and frame lookup
- [x] AWS Lambda/CDK/SAM and GCP Cloud Run error normalization/retry classification
- [x] producer, core, engine, AWS, and GCP typechecks/builds
- [x] formatting, oxlint, tracked-artifact, fallow, and commit hooks
- [x] exact incident composition replayed through the AWS Lambda handler's Lambda-local path in a Lambda-like container; Plan v1 and Plan v2 both fail closed as `VIDEO_SOURCE_UNRENDERABLE` during planning, before plan publication
- [x] full PR CI, including all nine regression shards and Windows render/tests

No production flags or deployment/release workflows are changed.
2026-07-28 00:42:36 -07:00
James 96cafb47c6 fix(producer): fallback distributed capture safely 2026-07-27 03:08:40 +00:00
James 2a284a8e3a fix(gcp): enforce effective BeginFrame capture 2026-07-27 00:02:37 +00:00
James 74d7bfde48 feat(gcp-cloud-run): publish plan v2 directly to GCS 2026-07-26 05:48:38 +00:00
James Russo 0499a5cbcb fix(gcp-cloud-run): normalize v2 integrity codes (#2790) 2026-07-25 23:59:36 -04:00
James Russo f9f00b0efc feat(producer): version distributed plan protocol (#2777) 2026-07-25 19:17:07 -04:00
James 942db16fcd fix(cloudrun): publish adapter contract 2026-07-18 20:17:01 -04:00
James RussoandClaude Opus 4.8 4da567df22 feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253)
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter

Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda
(issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble)
are unchanged; this package is the storage/compute/orchestration glue.

Package: Cloud Run handler (one image, three actions), runs under bun; GCS
transport; in-image chrome-headless-shell resolver; client SDK
(renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile;
Cloud Workflows definition; Terraform module; CLI cloudrun
deploy|sites|render|render-batch|progress|destroy with --output-resolution and
--strict-variables; 62 unit tests + docs + live smoke script.

Shared extraction (removes ~640 lines of adapter duplication): move the
cloud-agnostic config validator + content-hash into producer/distributed; both
adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build

The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`,
failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that
build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk
subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run
to the root `build` filter so its dist exists for publish + runtime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install

The regression test image runs `bun install --frozen-lockfile` after copying
each workspace package.json individually. The CLI now depends on
@hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to
resolve it unless its manifest is present. Add the COPY line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(cli): add machine-sizing flags to `cloudrun deploy`

Closes the parity gap with `lambda deploy` (which exposes --memory etc.).
`cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout
into the Terraform apply; omitted flags keep the module defaults
(4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module
directly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(gcp-cloud-run): address PR review (security, waste, limits, alerts)

- server.ts: bucket-allowlist guard no longer fails open silently. Unset env
  logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces.
- server.ts: stop double-shipping audio.aac. It already rides in the plan
  tarball every consumer downloads, so drop the redundant standalone upload
  (plan) + re-download/overwrite (assemble); assemble reads it from the untar,
  falling back to a supplied AudioGcsUri for compat.
- server.ts: chunk extension via path.extname() instead of slice(lastIndexOf).
- workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20)
  — Cloud Workflows hard-caps concurrent iterations at 20.
- Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break
  the image rebuild.
- terraform: add min_instances var (default 0); add a workflow-failure alert
  (finished_execution_count status=FAILED) alongside the request-count one.
- costAccounting: document that displayCost excludes GCS storage/egress.

Verified against the actual APIs: @google-cloud/workflows@4.4.0
ICreateExecutionRequest has no executionId (so the idempotency-token suggestion
isn't available in this client); Workflows concurrency cap is 20; failure
metric is workflows.googleapis.com/finished_execution_count (status label).
174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding

- workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE →
  PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the
  opposite cause), misleading anyone triaging the alert.
- workflow.yaml: forward Config.cfr to the assemble step
  (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler
  but never sent, so exact-CFR was silently off for every Cloud Run render.
  Uses the same `in`-operator guard already proven in the retryable predicate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(release): include gcp-cloud-run in set-version PACKAGES list

set-version.ts (driven by release:prepare) bumps an explicit package list to
the shared version on each release. gcp-cloud-run was wired into the build +
publish.yml but missing here, so a release would leave it at a stale version
and publish.yml would push the wrong version. Add it so the new package
version-bumps + publishes in lockstep with the others.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 14:43:38 -07:00