ext/hyperframes
2
0
Fork 0
mirror of https://github.com/heygen-com/hyperframes.git synced 2026-09-01 19:42:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,754 Commits 848 Branches 400 Tags
3f3293da8693f71aae5bd867fcff70080aab2454
Commit Graph
7 Commits
Author SHA1 Message Date
James Russo 9b18fadccd feat(producer): optional targetChunkFrames to bound per-chunk frames (#1332) 
* feat(producer): optional targetChunkFrames to bound per-chunk frames

* feat(cli): expose --target-chunk-frames on lambda + cloudrun render; document it
 2026-06-10 18:09:59 -07:00
James RussoandClaude Opus 4.8 4da567df22 feat(gcp-cloud-run): Google Cloud Run + Workflows distributed render adapter (#1253) 
* feat(gcp-cloud-run): add Google Cloud Run + Workflows distributed render adapter

Adds @hyperframes/gcp-cloud-run, the GCP counterpart to @hyperframes/aws-lambda
(issue #932). The OSS distributed primitives (plan, renderChunk x N, assemble)
are unchanged; this package is the storage/compute/orchestration glue.

Package: Cloud Run handler (one image, three actions), runs under bun; GCS
transport; in-image chrome-headless-shell resolver; client SDK
(renderToCloudRun, getRenderProgress, deploySite, computeRenderCost); Dockerfile;
Cloud Workflows definition; Terraform module; CLI cloudrun
deploy|sites|render|render-batch|progress|destroy with --output-resolution and
--strict-variables; 62 unit tests + docs + live smoke script.

Shared extraction (removes ~640 lines of adapter duplication): move the
cloud-agnostic config validator + content-hash into producer/distributed; both
adapters import them. Validated end-to-end on GCP at 37.4 dB PSNR vs baseline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): resolve @hyperframes/gcp-cloud-run in the CLI build + root build

The CLI bundle (esbuild) couldn't resolve `@hyperframes/gcp-cloud-run/sdk`,
failing Build/Typecheck/CLI-smoke (and the perf/windows/regression jobs that
build first). Mirror the aws-lambda handling: mark the gcp adapter + its /sdk
subpath external in tsup.config.ts with a source alias, and add gcp-cloud-run
to the root `build` filter so its dist exists for publish + runtime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): copy gcp-cloud-run manifest in Dockerfile.test for frozen install

The regression test image runs `bun install --frozen-lockfile` after copying
each workspace package.json individually. The CLI now depends on
@hyperframes/gcp-cloud-run (workspace:*), so the frozen install fails to
resolve it unless its manifest is present. Add the COPY line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(cli): add machine-sizing flags to `cloudrun deploy`

Closes the parity gap with `lambda deploy` (which exposes --memory etc.).
`cloudrun deploy` now threads --cpu, --memory, --max-instances, and --timeout
into the Terraform apply; omitted flags keep the module defaults
(4 vCPU / 16Gi / 100 instances / 3600s). For finer control, apply the module
directly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(gcp-cloud-run): address PR review (security, waste, limits, alerts)

- server.ts: bucket-allowlist guard no longer fails open silently. Unset env
  logs a one-time WARNING; "*" is an explicit opt-out; otherwise it enforces.
- server.ts: stop double-shipping audio.aac. It already rides in the plan
  tarball every consumer downloads, so drop the redundant standalone upload
  (plan) + re-download/overwrite (assemble); assemble reads it from the untar,
  falling back to a supplied AudioGcsUri for compat.
- server.ts: chunk extension via path.extname() instead of slice(lastIndexOf).
- workflow.yaml: clamp parallel concurrency_limit to math.min(chunkCount, 20)
  — Cloud Workflows hard-caps concurrent iterations at 20.
- Dockerfile: pin bun (bun-v1.3.9) so an interop change can't silently break
  the image rebuild.
- terraform: add min_instances var (default 0); add a workflow-failure alert
  (finished_execution_count status=FAILED) alongside the request-count one.
- costAccounting: document that displayCost excludes GCS storage/egress.

Verified against the actual APIs: @google-cloud/workflows@4.4.0
ICreateExecutionRequest has no executionId (so the idempotency-token suggestion
isn't available in this client); Workflows concurrency cap is 20; failure
metric is workflows.googleapis.com/finished_execution_count (status label).
174 adapter tests pass, fallow/oxlint/oxfmt/terraform clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(gcp-cloud-run): address round-2 review — error code + CFR forwarding

- workflow.yaml: rename the zero-chunk failure code PLAN_TOO_LARGE →
  PLAN_PRODUCED_ZERO_CHUNKS. The old code implied a size-ceiling breach (the
  opposite cause), misleading anyone triaging the alert.
- workflow.yaml: forward Config.cfr to the assemble step
  (`Cfr: ${("cfr" in config) and config.cfr}`). It was read by the handler
  but never sent, so exact-CFR was silently off for every Cloud Run render.
  Uses the same `in`-operator guard already proven in the retryable predicate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(release): include gcp-cloud-run in set-version PACKAGES list

set-version.ts (driven by release:prepare) bumps an explicit package list to
the shared version on each release. gcp-cloud-run was wired into the build +
publish.yml but missing here, so a release would leave it at a stale version
and publish.yml would push the wrong version. Add it so the new package
version-bumps + publishes in lockstep with the others.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-07 14:43:38 -07:00
James d28c082416 docs(concepts): clarify what can and can't be a variable 2026-05-19 20:47:46 -04:00
James 480d0cfa5a docs(deploy): templates-on-lambda guide for personalised video at scale 
User-facing guide for the automated template-rendering pipeline now
shippable end-to-end after PRs 9.1-9.4:

  - What a template is (composition + data-composition-variables)
  - Declaring variables (syntax, types, defaults, getVariables())
  - Local iteration loop (hyperframes render --variables / --variables-file
    / --strict-variables)
  - Deploying to Lambda (pointers to deploy guide + sites create)
  - Single personalised render (lambda render --variables)
  - Batch pipeline (lambda render-batch --batch users.jsonl, with a worked
    5-row example, manifest output, progress polling, --dry-run)
  - Programmatic via SDK (TypeScript example with deploySite +
    Promise.all(renderToLambda))
  - Working with large variables (the 256 KiB Step Functions ceiling,
    URL-your-assets convention, the one-line escape note for genuine
    >256 KiB cases)
  - Cost + scale considerations (Lambda concurrency, max-parallel-chunks
    vs max-concurrent, in-process vs distributed crossover)
  - Migrating from @remotion/lambda inputProps (side-by-side table; same
    256 KiB cap and same URL-your-assets convention, so migration is
    mechanical)

Includes a Mermaid architecture diagram for the site-upload-once +
N-execution fan-out flow at the top.

Adds the guide to the Deploy navigation group in docs.json (between
the existing aws-lambda and migrating-to-hyperframes-lambda pages).

Phase 9 PR 9.5 of the distributed rendering plan — the load-bearing
artifact for the user-facing pitch.
 2026-05-19 19:54:30 -04:00
James RussoandClaude Opus 4.7 5d264e146c docs(lambda): document webm support + simplify-review fixes (#953) 
* docs(lambda): document webm support in distributed mode

PR 8.4 of the WebM distributed-rendering plan (v1.5 backlog #1; see
DISTRIBUTED-RENDERING-PLAN.md §7.2). User-facing docs catch up with the
shipped capability.

Updates docs/deploy/migrating-to-hyperframes-lambda.mdx:

- "Output format" row in the migration table now lists `webm` alongside
  mp4 / mov / png-sequence with a note that webm uses libvpx-vp9 +
  closed-GOP concat-copy. HDR mp4 remains the only refused format.

- "No webm distributed" caveat replaced with "webm uses closed-GOP VP9"
  explainer covering the encoder args (`-g <chunkSize>`,
  `-keyint_min <chunkSize>`, `-auto-alt-ref 0`, `-cpu-used 2`), why
  alt-ref disable is load-bearing, and that the output preserves alpha
  via yuva420p with Opus audio.

- Migration checklist no longer asks adopters to filter out webm
  compositions; only HDR-dependent renders need to stay on the previous
  framework.

aws-lambda.mdx doesn't currently call out webm as unsupported (only HDR
in the v1 surface list), so it gets no copy edits beyond the migration
guide.

The internal planning doc (DISTRIBUTED-RENDERING-PLAN.md §7.2, §8,
§12 — kept outside the repo) gets matching updates: format support
matrix flipped ✓, v1.5 backlog #1 marked shipped, HDR promoted to the
new top item, and the rev-12 → rev-13 status line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: address simplify-review findings on webm stack

Folds in cleanups identified by a multi-agent code-review pass over the
4-PR webm-distributed stack:

- plan.ts: `resolveEncoderTriple()` webm case now calls
  `getEncoderPreset(quality, "webm")` for its preset string instead of
  hardcoding "good". The hardcode was wrong for `quality: "draft"`
  (`getEncoderPreset` returns "realtime" for that tier) — would have
  silently overridden the draft → realtime mapping for distributed webm
  renders.
- chunkEncoder.ts: trim the new VP9 closed-GOP comment block from ~18
  lines of WHY narration down to the 6 lines that actually explain why
  (alt-ref + cpu-used drift). Match the alpha branch's idempotent-push
  comment to the same standard.
- chunkEncoder.test.ts: drop the duplicate WHY comment that restated
  the implementation comment in plain words.
- webm-concat-copy.test.ts: rewrite the file-header docstring to
  describe the contract being tested instead of the PR-8.1-gating
  history; strip "PR 8.2 / Path A / Path B" references from error
  messages (they belong in PR bodies, not in test output). Consolidate
  the yuva420p alpha smoke into a single `it()` block (was a full
  4-test describe with duplicated setup) — the yuv420p block already
  covers the probe/decode/frame-count contract; the alpha smoke only
  needs to prove the alpha args don't break concat-copy.
- plan.test.ts: drop the "PR 8.1 proved the contract" comment.
- webm-vp9 fixture: drop the aspirational "Other webm-with-audio
  fixtures cover the mux path separately when added" sentence (no
  other fixtures exist). Regenerated the baseline via
  `docker:test:update webm-vp9` to reflect the updated comment.
- migrating-to-hyperframes-lambda.mdx: add a paragraph about
  distributed webm's perf cost — ~10-25% larger files at constant CRF
  due to forced keyframes, and slower per-chunk encode due to
  `-cpu-used 2` being more conservative than the libvpx default.

All unit tests + the webm-vp9 distributed-simulated regression still
pass after these changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cli): accept --format=webm in `hyperframes lambda render`

The CLI's `lambda render` subcommand's FORMATS allowlist and the
`RenderArgs.format` type still narrowed to `mp4 | mov | png-sequence`,
so even though the producer + aws-lambda packages now support webm
end-to-end, the CLI surface rejected it with `--format must be mp4|mov|
png-sequence`. Add webm to both spots and update the --help description.

Surfaced during real-AWS deploy prep — the local lambda-local /
distributed-simulated tests didn't go through the CLI so the gap went
unnoticed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(producer): font cache writes to /tmp on Lambda (read-only \$HOME)

The deterministic Google Fonts cache was rooted at
`\$HOME/.cache/hyperframes/fonts`, which fails on AWS Lambda — the
runtime's `\$HOME` resolves to a `/home/sbx_*` directory tree that's
read-only. `mkdirSync(..., { recursive: true })` can't create that
path and the plan stage trips with `ENOENT: no such file or directory,
mkdir '/home/sbx_user1051/.cache/hyperframes/fonts/space-mono'` on
every Lambda render that pulls a Google Font (i.e. every distributed
fixture using `@import url("https://fonts.googleapis.com/...")`).

Detect Lambda via `\$AWS_LAMBDA_FUNCTION_NAME` and route the cache to
`tmpdir()/hyperframes/fonts` in that case. Lambda's `/tmp` survives
across invocations on a warm container, so cache hit rate is the same
as non-Lambda runs. Also honor an explicit
`\$HYPERFRAMES_FONT_CACHE_DIR` override for adopters who want a
different location regardless of the runtime.

Surfaced while verifying webm distributed end-to-end on real AWS — the
same bug affects mp4 fixtures using Google Fonts; webm just happened to
be the one I tried first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: extract DistributedFormat type + trim font-cache resolver

Second simplify-review pass on the webm stack flagged two cleanups:

1. **`DistributedFormat` type duplicated 10 times.** Every file in the
   distributed pipeline carried its own copy of
   `"mp4" | "mov" | "png-sequence" | "webm"` — adding a new format
   meant a 10-place edit with no compile-time guarantee they stayed in
   sync. Extract a single source of truth in
   `packages/producer/src/services/distributed/shared.ts`, re-export
   from `@hyperframes/producer/distributed` and
   `@hyperframes/aws-lambda/sdk`, and have all callers pull from
   there. The aws-lambda `ALLOWED_FORMATS` runtime tuple and the CLI's
   `FORMATS` tuple now both use `satisfies readonly DistributedFormat[]`
   so the compiler enforces the runtime allowlist stays in sync with
   the type.

2. **`deterministicFonts.ts` font-cache resolver was over-commented.**
   Trim the 7-line block to 4 lines (drop the aspirational
   "and other read-only-FS execution environments" — only Lambda is
   detected — and the warm-container `/tmp` persistence narration —
   anyone reading already knows Lambda /tmp semantics). Collapse the
   two-step `if (explicit && explicit.length > 0)` into a single
   nullish-coalesce expression now that the empty-string defensive
   check is gone (`process.env.X` is `string | undefined`, no third
   shape to guard against).

Out-of-scope skips (called out by the agents, deferred):
- In-process `RenderConfig.format` and the in-process CLI's
  `render.ts` format union still carry their own inline copies. The
  union happens to coincide today but they're separate concerns —
  leaving them alone limits this PR's blast radius.
- `fontCacheDir(slug)` / `resolveFontCacheRoot()` naming asymmetry
  flagged as taste; skipping.
- Pre-existing redundant `existsSync` before `mkdirSync({ recursive:
  true })` in `fontCacheDir` — out of scope.

All tests + typecheck still pass. Lambda render still works
end-to-end (no functional changes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(lambda): drop plan-doc reference from migration checklist

PR review feedback: source/docs should not mention the
distributed-rendering planning doc. Tighten the migration checklist
sentence to describe the webm path directly rather than referencing
the doc's version label.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(producer): split resolveEncoderTriple into mp4 + non-mp4 helpers

CI Fallow audit on PR #953 flagged `resolveEncoderTriple` at CRAP 31.6 —
the function interleaved (a) mp4 codec validation + dispatch, (b) the
non-mp4 codec-rejection throw, and (c) per-format dispatch. Splitting
into `resolveMp4EncoderTriple` + `resolveNonMp4EncoderTriple` drops the
top-level function's cyclomatic complexity below the threshold while
preserving every error message and code path. Behavior unchanged.

Also extracts an `EncoderTriple` type alias so the three functions
share the return shape declaratively rather than repeating it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-19 04:11:26 -04:00
James Russo b6b6b8ed3b docs(lambda): add migration guide + non-Lambda Dockerfile example (#915) 
Two adopter-facing artifacts that close out Phase 6b's user-facing
surface:

  - docs/deploy/migrating-to-hyperframes-lambda.mdx — side-by-side
    concept mapping for users coming from another one-command-deploy
    video renderer. Covers the verb mapping (deploy/render/progress/
    destroy/sites/policies), composition format (plain HTML vs JSX),
    render config, and a handful of intentional differences (no HDR
    in distributed mode, no webm, gpu-mode=software requirement,
    fail-closed font fetch, local stack-state files, narrow-after-
    first-deploy IAM pattern). Closes with a migration checklist.
    Per repo convention, no competitor framework is named anywhere
    in the source — adopters self-identify.

  - examples/k8s-jobs/Dockerfile.example + README.md — reference
    Dockerfile for adopters who want to run distributed renders
    outside AWS Lambda. Bakes Node 22 + chrome-headless-shell +
    ffmpeg + the producer source. Deliberately not published to
    a registry; adopters build it themselves so Chrome / ffmpeg /
    producer versions stay pinned to the checkout they audited.
    The README documents the typical K8s Jobs orchestration shape
    that points adopters at packages/aws-lambda/src/handler.ts as
    the reference adapter.

Migration guide registered under the existing Deploy group in
docs.json. .gitignore extended to negate the new examples/k8s-jobs/
path the same way examples/aws-lambda/ is negated.

No source code changes.
 2026-05-17 14:12:22 -04:00
James Russo a1a8c7790f docs(lambda): add docs/deploy/aws-lambda.mdx deployment guide (#914) 
* docs(lambda): add docs/deploy/aws-lambda.mdx deployment guide

End-to-end deploy guide for the AWS Lambda surface. Covers:

  - Architecture diagram (Step Functions Plan → Map(N) → Assemble +
    the single Lambda function dispatching by Action; pulled from
    the distributed rendering plan §15.2).
  - Prerequisites table (AWS creds, SAM CLI, bun, repo checkout).
  - Three deployment paths: hyperframes lambda CLI (recommended),
    direct sam deploy against examples/aws-lambda/template.yaml,
    and HyperframesRenderStack CDK construct.
  - IAM bootstrap via hyperframes lambda policies user/role/validate.
  - Cost shape — how Lambda GB-seconds + SFN transitions roll up
    into the displayCost the progress verb prints.
  - Troubleshooting block with the typed error names operators
    actually hit (PLAN_HASH_MISMATCH, BROWSER_GPU_NOT_SOFTWARE,
    iam:CreateRole denial, stuck RUNNING, S3 Retain semantics).
  - "What's NOT in v1" callout so adopters don't burn time looking
    for webhooks / compositions verb / HDR support.

Registered under a new "Deploy" group in docs.json's Documentation
tab, sitting after Packages so the conceptual flow is "what you
can build" → "how to ship it."

No code changes.

* docs(lambda): address PR review on AWS Lambda deployment guide

One blocker + two important items from Vai's review:

  - The BROWSER_GPU_NOT_SOFTWARE troubleshooting entry pointed
    adopters at a non-existent `data-gpu-mode` composition attribute.
    Replaced with the actual root cause (Chrome launch flags +
    @sparticuz/chromium libs in the handler ZIP) and the actual
    remediation: rebuild + redeploy via `lambda deploy` (which
    always rebuilds the ZIP). The composition-attribute story
    would have sent users editing the wrong file entirely.

  - Added a `sites create` subsection under Path 1 so adopters
    running tight inner loops know how to reuse a project upload
    across many renders instead of re-tarring + re-uploading on
    each call. The CLI surface was first-class but the doc had
    been silent.

  - Added a Warning callout under Path 2 explaining that the SAM
    template's own ReservedConcurrency default is `-1` (unreserved)
    — a reader simplifying the Path 2 example by dropping the
    --parameter-overrides flag would silently switch to unreserved
    concurrency and pay the runaway-Map cost. The warning mirrors
    the cost-shape callout earlier in the page.
 2026-05-17 14:08:22 -04:00