mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-06 06:37:29 +00:00
* perf(aws-lambda): run chunks through a Distributed Map to lift the 40-chunk cap The inline Map runs at most 40 chunk renders at once. Both state machine definitions (CDK and SAM) now use a Distributed Map with STANDARD child executions, capped at the function's reserved concurrency so chunks past it queue instead of being throttled. Progress reads each chunk's child execution history so frames, cost and chunk errors still show up. * fix(aws-lambda): retry Lambda throttles for minutes and stop on oversized chunks A throttled task used to burn its four catch-all retries in about 30 s. It now gets its own jittered backoff of several minutes, because another render on the same function can still hold the reserved slots. Plan treats CHUNK_EXCEEDS_WORKER_TIMEOUT as terminal in both definitions. * chore(aws-lambda): trim stack comments to keep the comment share flat * fix(aws-lambda): keep render --wait working at hundreds of chunks Progress polls read the chunk Map Run's counts (one DescribeMapRun) instead of every child execution's history, take exact chunk frames and cost from the Map output once it exits, and read failed children only when the counts show a failure. The SDK client retries throttled calls up to 8 times with backoff. * feat(aws-lambda): size each render's concurrency to the account instead of a fixed 8 * fix(lambda): keep the concurrency SDK external in published modules * fix(cli): keep an existing stack's reserved concurrency on redeploy Redeploying without --concurrency sent ReservedConcurrency=-1, which removed the cap of 8 that earlier CLI versions reserved by default and stopped recording it in the state file. The override is now left out when the flag is omitted, so SAM keeps the stack's current value (a new stack still gets the template's unreserved default). The state file records the deployed value, and deploy prints the kept reservation. * fix(aws-lambda): keep progress working for callers without DescribeMapRun Callers whose IAM policy came from an earlier `lambda policies user` lack states:DescribeMapRun, so reading a running Distributed Map threw AccessDeniedException out of `lambda progress` and `render --wait` while the render itself went on to succeed. getRenderProgress now catches only that error around the map-run read, returns the parent snapshot (status, plan and assemble progress) and adds a warning to re-apply the policy, which both commands print. Other errors still throw. * fix(cli): let the documented deploy policy drop a reserved concurrency When a redeploy removes a function's reserved concurrency, CloudFormation calls lambda:DeleteFunctionConcurrency. The policy printed by `lambda policies role|user` granted only PutFunctionConcurrency, so the update failed and rolled back. Grant it, and pin it with states:DescribeMapRun in the required-actions test. * fix(aws-lambda): read smoke chunk hashes from the Distributed Map output With chunks in a Distributed Map, their task results live in child executions, so the smoke gate's chunk-hash check found 0 chunks in the parent history and failed every v1/v2 comparison. Read Chunks[] from the RenderChunks / RenderChunksV2 MapStateExited output instead, as getRenderProgress does. The stack this script deploys only has the Distributed Map, so the inline-Map path is gone. * fix(producer): cap a pinned chunk size at the worker concurrency, not 16 With --chunk-size or --target-chunk-frames and no --max-parallel-chunks, the plan still capped the render at 16 chunks however many workers the account allowed, so --chunk-size 30 on a 60 s render gave 16 chunks of 113 frames. The cap is now one wave across the worker concurrency, as the --max-parallel-chunks docs say; 16 remains when the concurrency is unknown. * feat(cli): remove a stack's reserved concurrency with --concurrency=none Since an omitted --concurrency now keeps the stack's reservation, there was no CLI way back to the unreserved default. --concurrency=none sends ReservedConcurrency=-1, the state file records the stack as unreserved and deploy says the reservation was removed. Positive N and an omitted flag behave as before; any other value is rejected instead of being silently ignored. * chore(aws-lambda): raise the package size budget for account-sized concurrency Measured with pnpm@10.17.1 pack after rebasing on main with the S3 assembly change: 67,763 packed / 244,685 unpacked; the budget is the check's own 5% headroom over that.
219 lines
7.4 KiB
TypeScript
219 lines
7.4 KiB
TypeScript
/**
|
|
* Lambda event + result types for the HyperFrames distributed render handler.
|
|
*
|
|
* The Step Functions state machine in `examples/aws-lambda/template.yaml`
|
|
* dispatches on the `Action` field. Each action maps 1:1 onto one of the
|
|
* three OSS distributed primitives:
|
|
*
|
|
* "plan" → `plan(projectDir, config, planDir)` (Activity A)
|
|
* "renderChunk" → `renderChunk(planDir, chunkIndex, output)` (Activity B)
|
|
* "assemble" → `assemble(planDir, chunkPaths, audio, out)` (Activity C)
|
|
*
|
|
* All file I/O is mediated by S3 — the handler downloads inputs into
|
|
* `/tmp` (Lambda's only writable filesystem path), invokes the primitive,
|
|
* uploads outputs back to S3, and returns a small JSON payload that fits
|
|
* inside Step Functions' history budget (under 200 bytes for chunk
|
|
* results per §2.4).
|
|
*/
|
|
|
|
import type {
|
|
DistributedFormat,
|
|
SerializableDistributedRenderConfig,
|
|
} from "@hyperframes/producer/distributed";
|
|
|
|
export type { SerializableDistributedRenderConfig } from "@hyperframes/producer/distributed";
|
|
|
|
/** Discriminator for the three roles the one Lambda image fulfills. */
|
|
export type LambdaAction = "plan" | "renderChunk" | "assemble";
|
|
/** Transport protocol selected for one complete distributed render. */
|
|
export type LambdaPlanProtocol = "v1" | "v2";
|
|
|
|
/**
|
|
* Top-level shape of any event the handler may receive.
|
|
*
|
|
* Step Functions can also invoke with a wrapped payload (e.g. when a Map
|
|
* state's `ItemSelector` passes through `$$.Map.Item.Value`), so the
|
|
* handler unwraps both `event.Payload` and `event.Input` before
|
|
* dispatching.
|
|
*/
|
|
export type LambdaEvent =
|
|
| PlanEvent
|
|
| RenderChunkEvent
|
|
| AssembleEvent
|
|
| { Payload: LambdaEvent }
|
|
| { Input: LambdaEvent };
|
|
|
|
/** Activity A: produce a planDir, upload to S3. */
|
|
interface PlanEventBase {
|
|
Action: "plan";
|
|
/** S3 URI pointing at a `tar -czf`-archived project directory (`s3://bucket/key.tar.gz`). */
|
|
ProjectS3Uri: string;
|
|
/** S3 URI prefix where the planDir tar should be uploaded (`s3://bucket/{prefix}/`). */
|
|
PlanOutputS3Prefix: string;
|
|
/** `DistributedRenderConfig` minus runtime-only fields (logger, abortSignal). */
|
|
Config: SerializableDistributedRenderConfig;
|
|
}
|
|
|
|
/**
|
|
* Legacy plan transport. Callers must select it explicitly.
|
|
*
|
|
* @deprecated Use {@link PlanV2Event} for new integrations.
|
|
*/
|
|
export interface PlanV1Event extends PlanEventBase {
|
|
PlanProtocol: "v1";
|
|
}
|
|
|
|
/** Default content-addressed v2 plan transport. */
|
|
export interface PlanV2Event extends PlanEventBase {
|
|
PlanProtocol?: "v2";
|
|
}
|
|
|
|
export type PlanEvent = PlanV1Event | PlanV2Event;
|
|
|
|
/** Activity B: fetch planDir, render one chunk, upload result. */
|
|
interface RenderChunkEventBase {
|
|
Action: "renderChunk";
|
|
/**
|
|
* `PlanResult.planHash` from the Plan invocation. For v1, the handler
|
|
* verifies it against the untarred planDir's `plan.json`; for v2, it
|
|
* verifies it against the content-addressed manifest before invoking the
|
|
* producer. Divergence throws a typed `PLAN_HASH_MISMATCH` so the state
|
|
* machine routes it as non-retryable.
|
|
*/
|
|
PlanHash: string;
|
|
/** 0-based chunk index this invocation should render. */
|
|
ChunkIndex: number;
|
|
/** S3 URI prefix where the chunk output should be uploaded (`s3://bucket/{prefix}/`). */
|
|
ChunkOutputS3Prefix: string;
|
|
/** Output container format from the plan's encoder.json; drives file vs frame-dir handling. */
|
|
Format: DistributedFormat;
|
|
}
|
|
|
|
/**
|
|
* Legacy chunk event. Callers must select it explicitly.
|
|
*
|
|
* @deprecated Use {@link RenderChunkV2Event} for new integrations.
|
|
*/
|
|
export interface RenderChunkV1Event extends RenderChunkEventBase {
|
|
PlanProtocol: "v1";
|
|
/** S3 URI of the v1 plan tar produced by a PlanEvent invocation. */
|
|
PlanS3Uri: string;
|
|
}
|
|
|
|
/**
|
|
* V2 chunk event. It intentionally cannot carry `PlanS3Uri`: the manifest
|
|
* describes the exact content-addressed artifacts needed by this chunk.
|
|
*/
|
|
export interface RenderChunkV2Event extends RenderChunkEventBase {
|
|
PlanProtocol?: "v2";
|
|
PlanV2ManifestS3Uri: string;
|
|
PlanV2ArtifactS3Prefix: string;
|
|
}
|
|
|
|
export type RenderChunkEvent = RenderChunkV1Event | RenderChunkV2Event;
|
|
|
|
/** Activity C: fetch planDir + all chunks + audio, assemble, upload final. */
|
|
interface AssembleEventBase {
|
|
Action: "assemble";
|
|
/** S3 URIs of every chunk, ordered by chunk index. Length must equal `chunkCount`. */
|
|
ChunkS3Uris: string[];
|
|
/** S3 URI of the planDir's audio artifact if the composition has audio; `null` otherwise. */
|
|
AudioS3Uri: string | null;
|
|
/** Final output S3 URI (`s3://bucket/key.mp4`). */
|
|
OutputS3Uri: string;
|
|
/** Output container format; drives file vs frame-dir handling. */
|
|
Format: DistributedFormat;
|
|
/**
|
|
* Optional exact-CFR re-encode at assemble time. When `true`, the final
|
|
* assembled video is re-encoded with `-fps_mode cfr -r <fps>` so the
|
|
* stream's `avg_frame_rate` matches the container's `r_frame_rate`
|
|
* exactly (and the file's duration is exact, not PTS-derived). Trade-off
|
|
* is ~2-5x the assemble wall-clock. mp4 only — webm / mov stream-copy
|
|
* paths already produce exact avg_frame_rate. Default `false` /
|
|
* unset preserves current `-c copy` behavior.
|
|
*/
|
|
Cfr?: boolean;
|
|
}
|
|
|
|
/**
|
|
* Legacy assemble event. Callers must select it explicitly.
|
|
*
|
|
* @deprecated Use {@link AssembleV2Event} for new integrations.
|
|
*/
|
|
export interface AssembleV1Event extends AssembleEventBase {
|
|
PlanProtocol: "v1";
|
|
/** S3 URI of the v1 plan tar produced by a PlanEvent invocation. */
|
|
PlanS3Uri: string;
|
|
}
|
|
|
|
/** V2 assemble event, scoped to manifest-declared assembler artifacts. */
|
|
export interface AssembleV2Event extends AssembleEventBase {
|
|
PlanProtocol?: "v2";
|
|
PlanV2ManifestS3Uri: string;
|
|
PlanV2ArtifactS3Prefix: string;
|
|
PlanHash: string;
|
|
}
|
|
|
|
export type AssembleEvent = AssembleV1Event | AssembleV2Event;
|
|
|
|
// ── Result types — kept small to fit Step Functions history budgets ─────────
|
|
|
|
/** Result of a `plan` invocation. Carries enough to size the Map(N) state. */
|
|
interface PlanLambdaResultBase {
|
|
Action: "plan";
|
|
PlanHash: string;
|
|
ChunkCount: number;
|
|
MaxChunkConcurrency: number;
|
|
TotalFrames: number;
|
|
Fps: SerializableDistributedRenderConfig["fps"];
|
|
Width: number;
|
|
Height: number;
|
|
Format: DistributedFormat;
|
|
HasAudio: boolean;
|
|
AudioS3Uri: string | null;
|
|
FfmpegVersion: string;
|
|
ProducerVersion: string;
|
|
DurationMs: number;
|
|
}
|
|
|
|
/**
|
|
* Existing v1 result. Kept unchanged for wire compatibility.
|
|
*
|
|
* @deprecated New integrations should consume {@link PlanV2LambdaResult}.
|
|
*/
|
|
export interface PlanV1LambdaResult extends PlanLambdaResultBase {
|
|
PlanS3Uri: string;
|
|
}
|
|
|
|
/** V2 result. The two v2 locators are never aliases for `PlanS3Uri`. */
|
|
export interface PlanV2LambdaResult extends PlanLambdaResultBase {
|
|
PlanProtocol: "v2";
|
|
PlanV2ManifestS3Uri: string;
|
|
PlanV2ArtifactS3Prefix: string;
|
|
}
|
|
|
|
export type PlanLambdaResult = PlanV1LambdaResult | PlanV2LambdaResult;
|
|
|
|
/** Result of a `renderChunk` invocation. Sized ≤200 bytes per §2.4. */
|
|
export interface RenderChunkLambdaResult {
|
|
Action: "renderChunk";
|
|
ChunkS3Uri: string;
|
|
ChunkIndex: number;
|
|
Sha256: string;
|
|
FramesEncoded: number;
|
|
/** Effective engine mode after browser probing. Emitted by current handlers. */
|
|
CaptureMode?: "beginframe" | "screenshot" | "drawelement";
|
|
DurationMs: number;
|
|
}
|
|
|
|
/** Result of an `assemble` invocation. */
|
|
export interface AssembleLambdaResult {
|
|
Action: "assemble";
|
|
OutputS3Uri: string;
|
|
FramesEncoded: number;
|
|
FileSize: number;
|
|
DurationMs: number;
|
|
}
|
|
|
|
export type LambdaResult = PlanLambdaResult | RenderChunkLambdaResult | AssembleLambdaResult;
|