Commit Graph
454 Commits
Author SHA1 Message Date
Miguel Ángel dee8380d22 feat(aws-lambda): a cloud render mixes its audio beside its chunks (#5997)
* feat(aws-lambda): a cloud render mixes its audio beside its chunks

* fix(aws-lambda): the audio task keeps a worker free and runs only when the plan left audio out

* test(aws-lambda): the audio task's free worker holds for a reserved pool of 1 and 2

* refactor(aws-lambda): one helper points every handler at the bundled Chrome
2026-10-21 05:26:58 +00:00
Miguel Ángel c8f322c259 fix(aws-lambda): lambda sites create uploads source files over 2 GiB and 5 GB (#5964)
* fix(aws-lambda): lambda sites create uploads source files over 2 GiB and 5 GB

Site ids are hashed in fixed-size reads instead of whole-file reads, so files
over 2 GiB no longer fail; every existing site keeps its id.

Files over S3's 5 GB single-PUT limit upload in parts that S3 checks one by
one, and the object is created only after the whole file's SHA-256 matches.
Content-store refresh and render-time verification handle those objects too.

* fix(aws-lambda): keep small-site hashing as fast as before

Reads use a 256 KiB buffer: the 8 MiB one made hashing small projects slower,
and big files hash no slower with the smaller buffer.

* perf(aws-lambda): stream big uploads from disk with 8 parts in flight

Each part is hashed from disk and then streamed from disk instead of held in
memory, so a 5 GB upload peaks near 150 MB instead of 500 MB. Eight parts go
up at once, and a part whose connection drops is resent from a fresh read.
2026-10-20 17:52:18 +00:00
Miguel Ángel ae02e5acef chore: release v0.8.181 (#5928) 2026-10-20 03:13:32 +00:00
e45832f452 chore: release v0.8.180 (#5920)
* chore: release v0.8.180

* docs(release): describe chunk sub-ranges as groundwork for split retries

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* chore: release v0.8.180 notes list the empty-frame fix

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
2026-10-20 01:05:20 +00:00
Miguel Ángel 40e8cccf8e chore: release v0.8.179 (#5914) 2026-10-19 18:23:52 -04:00
Miguel Ángel b501dc4e1b feat(cli): lambda deploy works from the npm install, no repo checkout (#5866)
* feat(cli): lambda deploy works from the npm install, no repo checkout

Each release builds the Lambda handler zip on Linux, attaches it to the
GitHub release, and pins its URL and sha256 (plus the SAM template) in
@hyperframes/aws-lambda before npm publish. Outside a checkout, lambda
deploy downloads that zip, refuses it on a sha256 mismatch, and deploys
it with the packaged template. With neither a checkout nor a pinned zip
it fails up front with one message naming every way out.

* fix(cli): lambda deploy test mocks the pinned-handler reader, comments fit the checks

* test(cli): sam tests deploy from a SAM directory, not the repo root

* chore(aws-lambda): raise the package size budget for the pinned handler template

Measured with pnpm@10.17.1 pack on Linux. Release state (prebuilt/ written by pin-handler-zip): 77,307 packed / 285,654 unpacked. CI state (no prebuilt/): 70,104 / 253,390. Packed is the check's 5% headroom over the release size. Unpacked is 292,000 (2.2% over release) because a 5% budget would put the CI pack under the 85% ratchet.

* test(cli): anchor the docs link check in the lambda deploy fallback test

* fix(cli): lambda deploy logs the verified handler sha256 and keeps a presigned query out of errors

* fix(cli): a redirect-limit download error keeps a signed query out of its message

* test(cli): the redirect-limit leak test types its rejection

* docs(aws-lambda): header comments cut to what the code cannot say

* chore(aws-lambda): size budget fits both the CI and the release package
2026-10-19 21:41:50 +00:00
Miguel Ángel 6fe20f764d fix(aws-lambda): old stacks are refused before any upload; stale local copies never re-upload (#5909)
* fix(aws-lambda): old stacks are refused before any upload; stale local copies never re-upload

* docs(aws-lambda): where the CLI keeps a deployed stack's state
2026-10-19 21:41:37 +00:00
Miguel Ángel 051175719e chore: release v0.8.178 (#5910) 2026-10-19 16:54:25 -04:00
Miguel Ángel 992df8b203 perf(cloud): move project bytes once through shared content storage (#5772)
* perf(producer): distributed chunks download only the videos they show

A project video under compiled/ that chunks extract their own frames from
now goes only to the chunks that show it, like video-sources/ files. A
chunk skips the readiness wait for a deferred video it does not show, so
it no longer logs media_load_failed for a file it was never given.

* fix(producer): cloud site uploads pack the files their id hashes, symlinks included

* perf(aws-lambda): chunks read only the byte ranges of a video they need

A chunk that extracts its own frames from a deferred project video now
reads it from S3 by byte range through a short-lived pre-signed URL,
instead of downloading and hashing the whole object first. S3 attests the
object's stored SHA-256 before the URL is signed; every other artifact is
still downloaded and verified as before.

* perf(aws-lambda): upload project files once to a shared content-addressed area

* chore(producer): keep planV2 comment share at the base level

* fix(engine): verify the TLS certificate of https video inputs and never surface their signed query

* fix(aws-lambda): renders re-upload a site's expired files instead of failing

* perf(aws-lambda): the plan step probes project videos in place instead of downloading them

* fix(engine): https video reads reconnect and time out, and redaction covers wrapped causes

Signed-URL reads now pass -rw_timeout, -reconnect and -reconnect_delay_max (both Lambda builds accept them) and ffmpeg also -reconnect_on_network_error, so a transient S3 error or a stalled read fails fast or recovers instead of waiting out the process timeout. Query redaction now descends into error.cause.

* fix(aws-lambda): share entry bundles and ratchet package budget

* fix(aws-lambda): site ids uploaded before files.json still render

A site id from an earlier sites create has only sites/<id>/project.tar.gz.
renderToLambda now falls back to that tarball, which the plan step still
reads, and when neither key exists it fails before starting with both keys
named and the fix (run sites create again).

* fix(aws-lambda): renders fail before upload on a stack too old to read sites

A stack deployed before files.json sites untars the site file in its plan
step and fails after four retries. renderToLambda now reads the state
machine definition first and stops with STACK_TOO_OLD and the redeploy
command when the plan task does not list SITE_MANIFEST_INVALID, which new
SAM and CDK stacks now treat as terminal. The README's upgrade section
drops the planProtocol v1 fallback, which hits the same failure.

* docs(aws-lambda): document shared content storage and the upgrade path

The deploy guide still described the tarball HeadObject short-circuit. It
now covers the cas/sha256/ store and its 30-day lifecycle rule, the
STACK_TOO_OLD, SITE_FILES_MISSING and SITE_NOT_FOUND errors, that old site
ids keep working, and that every project uploads once more after the
upgrade. A test keeps a troubleshooting entry for each pre-submit error.

* fix(producer): chunks log each video they read by URL, without its query

A chunk reading a video straight from the bucket left no trace in its log.
It now logs one line per remote source with the path and the URL minus its
query string, so signatures never reach the log.

* fix(engine): ffmpeg errors never keep part of a signed URL's query

A process's stderr tail was trimmed before its URLs were redacted, so a
cut inside a signed URL left a query fragment the URL pattern no longer
matched, and the fragment reached the error. The stderr tail is now
redacted before every trim, and the redactor also strips a signing
parameter cut off from its URL.

* fix(aws-lambda): comments name the site manifest instead of a file path

* fix(engine): stderr redaction survives chunk splits, and every ffmpeg spawn uses it

The stderr tail was redacted on each chunk and then cut, so once a cut
severed a long URL's scheme, the rest of its signature in the next chunk
matched neither pattern. The tail is now kept raw with 2x slack and
redacted only when the text is produced, then cut to size. The unused
producer audioExtractor (its own spawn, unredacted stderr) is deleted and
the parity harness's PCM decode goes through ManagedChildProcess; a test
fails on any ffmpeg/ffprobe spawn in engine, producer or aws-lambda that
does not.

* chore(producer): record the audioExtractor deletion as intended

* fix(engine): drop a signature fragment the stderr tail cut from its URL

The raw tail is kept at twice the cap, but that cut can itself land inside a
signature, and when redaction shrinks the rest below the cap the fragment
survived. After a cut, the severed token is dropped before redaction.

* refactor(engine): name the severed-token drop instead of commenting it

* test(engine): a cut-off URL leaves nothing behind; aws-lambda size budget to measured

* fix(producer): chunk URL-read tests use main's frame list; producer size budget to measured

* chore(producer): size budget at the measured size, no headroom

* chore(producer): size budget back to measured x1.05, the check's own rule
2026-10-19 19:23:51 +00:00
Miguel Ángel 2a2a1d2d90 chore: release v0.8.177 (#5897) 2026-10-19 14:03:31 -04:00
Miguel Ángel 5f619ef6ca chore: release v0.8.176 (#5891) 2026-10-19 12:14:57 -04:00
Miguel Ángel c7ad19be17 test(aws-lambda): handler tests never call real AWS (#5884)
Plan reads the account's Lambda concurrency when a render sets none, and
the handler tests left that client to its default, so a machine with AWS
credentials made a real GetAccountSettings call. Every handler call in the
file now gets a stub that refuses, and a guard fails any test that sends
through a real Lambda or S3 client.
2026-10-19 14:31:06 +00:00
Miguel Ángel ed8e2d2002 chore: release v0.8.175 (#5886) 2026-10-19 10:51:06 -04:00
Miguel Ángel 1d631bf98a chore: release v0.8.174 (#5869) 2026-10-18 22:21:10 -04:00
Miguel Ángel 2a14cbc924 perf(aws-lambda): size distributed renders from account concurrency limits (#5813)
* perf(aws-lambda): run chunks through a Distributed Map to lift the 40-chunk cap

The inline Map runs at most 40 chunk renders at once. Both state machine
definitions (CDK and SAM) now use a Distributed Map with STANDARD child
executions, capped at the function's reserved concurrency so chunks past it
queue instead of being throttled. Progress reads each chunk's child
execution history so frames, cost and chunk errors still show up.

* fix(aws-lambda): retry Lambda throttles for minutes and stop on oversized chunks

A throttled task used to burn its four catch-all retries in about 30 s.
It now gets its own jittered backoff of several minutes, because another
render on the same function can still hold the reserved slots. Plan
treats CHUNK_EXCEEDS_WORKER_TIMEOUT as terminal in both definitions.

* chore(aws-lambda): trim stack comments to keep the comment share flat

* fix(aws-lambda): keep render --wait working at hundreds of chunks

Progress polls read the chunk Map Run's counts (one DescribeMapRun) instead of
every child execution's history, take exact chunk frames and cost from the Map
output once it exits, and read failed children only when the counts show a
failure. The SDK client retries throttled calls up to 8 times with backoff.

* feat(aws-lambda): size each render's concurrency to the account instead of a fixed 8

* fix(lambda): keep the concurrency SDK external in published modules

* fix(cli): keep an existing stack's reserved concurrency on redeploy

Redeploying without --concurrency sent ReservedConcurrency=-1, which
removed the cap of 8 that earlier CLI versions reserved by default and
stopped recording it in the state file. The override is now left out
when the flag is omitted, so SAM keeps the stack's current value (a new
stack still gets the template's unreserved default). The state file
records the deployed value, and deploy prints the kept reservation.

* fix(aws-lambda): keep progress working for callers without DescribeMapRun

Callers whose IAM policy came from an earlier `lambda policies user`
lack states:DescribeMapRun, so reading a running Distributed Map threw
AccessDeniedException out of `lambda progress` and `render --wait`
while the render itself went on to succeed. getRenderProgress now
catches only that error around the map-run read, returns the parent
snapshot (status, plan and assemble progress) and adds a warning to
re-apply the policy, which both commands print. Other errors still
throw.

* fix(cli): let the documented deploy policy drop a reserved concurrency

When a redeploy removes a function's reserved concurrency, CloudFormation
calls lambda:DeleteFunctionConcurrency. The policy printed by
`lambda policies role|user` granted only PutFunctionConcurrency, so the
update failed and rolled back. Grant it, and pin it with
states:DescribeMapRun in the required-actions test.

* fix(aws-lambda): read smoke chunk hashes from the Distributed Map output

With chunks in a Distributed Map, their task results live in child
executions, so the smoke gate's chunk-hash check found 0 chunks in the
parent history and failed every v1/v2 comparison. Read Chunks[] from the
RenderChunks / RenderChunksV2 MapStateExited output instead, as
getRenderProgress does. The stack this script deploys only has the
Distributed Map, so the inline-Map path is gone.

* fix(producer): cap a pinned chunk size at the worker concurrency, not 16

With --chunk-size or --target-chunk-frames and no --max-parallel-chunks,
the plan still capped the render at 16 chunks however many workers the
account allowed, so --chunk-size 30 on a 60 s render gave 16 chunks of
113 frames. The cap is now one wave across the worker concurrency, as
the --max-parallel-chunks docs say; 16 remains when the concurrency is
unknown.

* feat(cli): remove a stack's reserved concurrency with --concurrency=none

Since an omitted --concurrency now keeps the stack's reservation, there
was no CLI way back to the unreserved default. --concurrency=none sends
ReservedConcurrency=-1, the state file records the stack as unreserved
and deploy says the reservation was removed. Positive N and an omitted
flag behave as before; any other value is rejected instead of being
silently ignored.

* chore(aws-lambda): raise the package size budget for account-sized concurrency

Measured with pnpm@10.17.1 pack after rebasing on main with the S3 assembly change: 67,763 packed / 244,685 unpacked; the budget is the check's own 5% headroom over that.
2026-10-19 00:57:30 +00:00
Miguel Ángel 70fbe32f8d perf(cloud): assemble chunk output with a single faststart index (#5812)
* perf(producer): assemble chunks that carry audio under one faststart index

Reads each chunk's moov, merges the sample tables into one moov at the front, and writes
the chunks' mdat payloads after it unchanged: the same bytes storage can copy into place.
The assembler no longer needs the plan audio for such plans.

* perf(aws-lambda): build the final mp4 inside S3 with its index up front

The Lambda plan asks chunks to carry their audio. Assemble writes the faststart header as
part 1 of a multipart upload and copies each chunk's mdat payload with UploadPartCopy, so
the video bytes never pass through the function; any failure aborts the upload.

* fix(engine): a chunk's first frame shows its own time, not the composition's opening frame

* fix(producer): stitch silent AAC tails and clean failed outputs

* fix(producer): open stitched chunks read-only with an explicit mode

* chore(aws-lambda): raise the package size budget for S3-side MP4 assembly

handler.js and index.js each bundle the new S3 stitch code (+5,550 bytes apiece) and s3Stitch.d.ts adds 970: +3,479 packed and +12,070 unpacked over main, measured with the CI pack. The budget rises by exactly that delta.

* fix(aws-lambda): v1 plans give chunks their audio too, so both protocols match

* perf(producer): plans whose chunks carry audio encode it once, at the frames' exact length

* test(producer): terminate the plan audio ffprobe argv like every other probe
2026-10-19 00:13:02 +00:00
Miguel Ángel 9566f96f92 chore: release v0.8.173 (#5857) 2026-10-18 17:37:10 -04:00
Miguel Ángel c8288eb9b6 chore: release v0.8.172 (#5848) 2026-10-18 15:53:44 +00:00
Miguel Ángel 76ba031dcc chore: release v0.8.171 (#5838) 2026-10-18 09:34:30 +00:00
Miguel Ángel b74686ef8e chore: release v0.8.170 (#5822) 2026-10-18 05:26:26 +00:00
Miguel Ángel 7735338b49 chore: release v0.8.169 (#5804) 2026-10-17 16:55:51 +00:00
Miguel Ángel 6e1f9ee11f chore: release v0.8.168 (#5780) 2026-10-17 09:54:30 +00:00
Miguel Ángel fb17f41196 perf(producer): overlap the plan step's audio work and hash files once (#5777)
* perf(producer): overlap the plan step's audio work and hash files once

* refactor(producer): simplify the plan overlap and publish path
2026-10-17 09:52:44 +00:00
Miguel Ángel a498cf2032 perf(producer): assemble in one pass and keep its disk use under one output copy (#5768)
* perf(producer): assemble in one pass and keep its disk use under one output copy

The distributed assembler joined the chunks into a full-size file, muxed the
audio into a second one and rewrote that again for faststart, so local disk
peaked near four copies of the video and a 30-minute 1080p render came close
to Lambda's 10 GB /tmp. The join, audio mux and faststart now run as one
ffmpeg pass over the concat demuxer; the audio pad/trim takes its frame count
from the chunks. Video and audio packets are unchanged. Assemble and the
Lambda assemble handler log per-stage timings.

* refactor(producer): split assemble into plan check, cfr re-encode and final write

Keeps assemble under the complexity gate and drops an orphaned doc comment in chunkEncoder. No behaviour change.
2026-10-17 08:45:53 +00:00
Miguel Ángel c69fd2c6ec perf(producer): size distributed chunks to the worker limit and concurrency (#5765)
* perf(producer): size distributed chunks to the worker limit and concurrency

* chore(cli): drop a comment the field name already says
2026-10-17 08:45:47 +00:00
Miguel Ángel d5162beb50 refactor(core): one owner for the distributed fps set (#5756)
The integer rates Lambda and Cloud Run accept were written out by hand in
about 25 places across core, engine, producer, the adapters and the CLI.
They now come from DISTRIBUTED_FPS / DistributedFps / isDistributedFps in
@hyperframes/core. Engine config snapshots accept 25 like the rest.
2026-10-17 08:45:41 +00:00
Robin EbersandCursor cc9e1bb33a feat(producer): distributed renders accept 25 fps (#5423)
Lambda and Cloud Run renders refused 25 fps although it is an integer
rate like 24, 30 and 60, and PAL footage is 25 fps. The distributed fps
set is now {24, 25, 30, 60} in the producer's config validation, the
plan v2 manifest, the render request mapping, the Lambda and Cloud Run
event types, the CLI's lambda and cloudrun commands and the regression
harnesses.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-16 22:00:32 -07:00
Miguel Ángel 35b17c3fe6 chore: release v0.8.167 (#5730) 2026-10-17 00:44:30 +00:00
Miguel Ángel a43cdd93bf chore: release v0.8.166 (#5709) 2026-10-16 18:16:28 +00:00
Miguel Ángel 1714f7c7b4 chore: release v0.8.165 (#5703) 2026-10-16 17:30:38 +00:00
Miguel Ángel 4d6b5be33b chore: release v0.8.164 (#5679) 2026-10-16 08:42:19 +00:00
Miguel Ángel 490ecafbc4 chore: release v0.8.163 (#5676) 2026-10-16 01:52:29 +00:00
Miguel Ángel 76222e41d7 chore: release v0.8.162 (#5649) 2026-10-15 11:54:09 +00:00
Xuanru Li ce060c7acc chore: release v0.8.161 (#5643) 2026-10-15 08:49:20 +00:00
James RussoandClaude Opus 5.5 f55b366175 fix(build): ship NodeNext-resolvable declarations from every package (#5532)
* fix(build): ship NodeNext-resolvable declarations from every package

tsc copies relative specifiers verbatim, so packages whose sources use
bundler-style extensionless imports (core, inspector) published .d.ts files
that a module/moduleResolution nodenext consumer cannot resolve. Core already
rewrote its emitted JavaScript for this, but not its declarations.

Move that rewrite to scripts/rewrite-relative-import-extensions.mjs, make it
cover .d.ts too (adding .js or /index.js, and failing the build when a
declaration import has no declaration), and run it after every tsc emit:
core, engine, sdk, producer, aws-lambda, gcp-cloud-run and inspector.

verify:packed-manifests now typechecks the clean packed consumer with
skipLibCheck false and type-only imports of every typed subpath, so an
unresolvable import inside a published declaration fails the gate and is
attributed to its package.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* chore(ci): allow the moved core rewrite-esm-extensions script deletion

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci: rebuild the edit-accuracy bench when the import-extension rewriter changes

The package builds now run scripts/rewrite-relative-import-extensions.mjs, so main's
edit_accuracy paths filter has to include it (check-edit-accuracy-filter test).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 05:59:41 +00:00
Xuanru Li ec97816a92 chore: release v0.8.160 (#5636) 2026-10-15 05:43:55 +00:00
Miguel Ángel f35b05fba0 chore: release v0.8.159 (#5621)
* chore: release v0.8.159

* chore: v0.8.159 notes list the first-frame fix

* chore: v0.8.159 notes list the sandboxed GPU check fix

* chore: v0.8.159 notes list the hidden-scene fix and the offline font test
2026-10-15 00:27:52 +00:00
Miguel Ángel fa99eec9d0 chore: release v0.8.158 (#5617) 2026-10-14 19:19:18 +00:00
Miguel Ángel b72c3fb25d chore: release v0.8.157 (#5596) 2026-10-14 09:19:15 +00:00
James RussoandClaude Opus 5.5 74e17fa9cf chore: release v0.8.156 (#5577)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 04:11:14 +00:00
James Russo 5a6c0575f8 chore: release v0.8.155 (#5567) 2026-10-13 23:07:51 +00:00
Xuanru Li 8147f7b940 chore: release v0.8.154 (#5545)
* chore: release v0.8.154

* ci: give the studio test job fifteen minutes

The queue run installs and builds inside the same ten minute cap.
It was still inside the studio tests when that cap cancelled the job.
2026-10-13 19:23:45 +00:00
Miguel Ángel 05786f5558 chore: release v0.8.153 (#5541) 2026-10-13 15:19:09 +00:00
Xuanru Li e8eb358fe2 chore: release v0.8.152 (#5534)
* chore: release v0.8.152

* docs(changelog): correct the v0.8.152 release notes

The summary overstated snapshot retention, history turns, and video state.
It also left out the new artifact paths and nullable keyframe times.
2026-10-13 07:37:26 +00:00
Miguel Ángel 945c61ebb6 chore: release v0.8.151 (#5529) 2026-10-13 04:13:03 +00:00
Miguel Ángel 8b27935e8c chore: release v0.8.150 (#5495) 2026-10-12 13:51:49 +00:00
James RussoandClaude Opus 5.5 bf881bf79d chore: release v0.8.149 (#5472)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 18:52:16 +00:00
Miguel Ángel 43be84d82c chore: release v0.8.148 (#5470) 2026-10-11 17:34:38 +00:00
Miguel Ángel d4dd6c097d chore: release v0.8.147 (#5456) 2026-10-11 08:55:42 +00:00
Miguel Ángel 8bca7c4417 chore: release v0.8.146 (#5448) 2026-10-11 04:25:16 +00:00