mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-16 14:27:26 +00:00
* fix(producer): mix audio into a container that can record encoder delay Every rendered composition's audio landed 1024 samples (21.33 ms at 48 kHz) after its authored `data-start`, against a frame-accurate video track. The mix is AAC-encoded, and AAC encoders emit ~1024 priming samples. The mix was written to a raw ADTS `.aac` file, which has nowhere to record that delay, so it decoded as real leading silence and every stage downstream preserved it faithfully. Measuring each intermediate localises it precisely: the source WAV is exact, the mixer's own output is already 21.33 ms late, and the pad/trim and mux stages inherit it unchanged. The filter graph itself is correct - run by hand to PCM it lands on the authored start. Switch the artifact to an MP4-family container, which stores the delay as an edit list that decoders strip. Same codec, same bitrate, so no size or quality change. The filename is a contract shared by three consumers - the mux input, the distributed plan artifact, and the PNG-sequence sidecar handed to users for NLE ingest - and its extension is what selects the muxer. Give it one owner in the engine rather than five literals, so those consumers cannot drift onto different containers. Note for reviewers: this renames the distributed plan's audio artifact, which is an on-disk contract between the plan writer and the assembler. Both move together here, but a plan written by an older build would not be found by a newer assembler. Flagging in case that mixed-version window matters for how these are deployed. * fix(cloud): read the plan audio artifact name from the producer contract The aws-lambda and gcp-cloud-run adapters each restated the plan's audio filename in five places, so renaming it in the producer left them looking for a file that is no longer written. CI caught it: the gcp dispatch test asserting a plan has no audio artifact started seeing one. Export the name from `@hyperframes/producer/distributed` and consume it in both adapters. This is the same failure the constant exists to prevent, one package boundary further out: a literal that drifts from the writer's is a silently missing audio track rather than a loud error, because both call sites only ever ask whether the file exists. * fix(cloud): accept a legacy plan's audio artifact name for one release Review raised a rolling-deploy window I had flagged but left undecided: `plan` and `assemble` are separate invocations bridged by object storage, so a pre-rollout planner can be paired with a post-rollout assembler. Both readers locate the artifact by existence alone, which makes that pairing a silently muted video rather than an error. That is reachable enough to be worth two lines, so reads now accept the old name while writes only ever emit the new one. Give the fallback one owner (`resolvePlanAudioPath` / `isPlanAudioArtifactPath`) rather than four call sites, marked for deletion one release out. Also fixes a hole in the first pass of this: the plan-v2 materializer matched either name but then joined the CURRENT one, so a legacy plan resolved to a path that was never written. It now joins the artifact's own name. Review nits in the same pass: correct the pad-branch docstring, which still described a concat-copy shape the pad branch stopped using when it moved to apad + re-encode, and fix the Windows fixture's stale `.aac` output extension so it cannot model a shape that reintroduces the priming delay. * test(producer): rebake the missing-host-comp-id golden without the audio delay The pinned reference was rendered before this branch, so it carries the 1024 sample encoder-priming delay in its audio. With the delay gone the correct audio now sits ahead of the reference and the harness's envelope correlation drops below its floor. Cross-correlating the old and new references at native 48 kHz gives a lag of exactly 1024 samples (21.33 ms) at a correlation of 0.99985: same audio, moved by exactly the amount this branch removes. Regenerated inside the CI container (Dockerfile.test, ffmpeg 5.1.9) rather than natively, so the reference matches the encoder CI will compare against - the container reproduced CI's failure to the digit (correlation 0.3938764027803616, lagWindows -12) before the rebake and passes at correlation 1.0 after it. Note for archaeology: the new reference is also 3 dB louder than the old one. That gap is not from this branch - `main` and this branch render the fixture at the same level - it is pre-existing drift the reference had accumulated, which a scale-invariant correlator could never see. The rebake absorbs it. Only output.mp4 is updated. `--update` also rewrites compiled.html, but that diff is embedded-font churn with no bearing on the comparison, which reports "Failed at compilation: 0" either way. * test(producer): rebake the variables-prod golden without the audio delay Same cause as the missing-host-comp-id rebake, caught by shard-8 once the earlier shard stopped failing and the rest of the matrix could run: this reference also carries the encoder-priming delay this branch removes. Reproduced in the CI container to the digit (correlation 0.42704173048439215, lagWindows -12), rebaked there, and it now passes at correlation 1.0. Worth recording: the shift here is 2048 samples (42.67 ms) at correlation 0.99983, exactly twice the 1024 of the other fixture. The delay compounds once per un-compensated AAC generation, and this fixture's audio needs its duration normalized, so it takes the pad/trim branch's re-encode and picks up a second frame of priming on top of the mixer's. So the pre-fix error was not a fixed 21 ms - it grew with the number of times the audio was re-encoded. All nine shards ran in that CI round with only this one failing, so the matrix has now covered every fixture against this change.
218 lines
7.4 KiB
TypeScript
218 lines
7.4 KiB
TypeScript
/**
|
|
* Lambda event + result types for the HyperFrames distributed render handler.
|
|
*
|
|
* The Step Functions state machine in `examples/aws-lambda/template.yaml`
|
|
* dispatches on the `Action` field. Each action maps 1:1 onto one of the
|
|
* three OSS distributed primitives:
|
|
*
|
|
* "plan" → `plan(projectDir, config, planDir)` (Activity A)
|
|
* "renderChunk" → `renderChunk(planDir, chunkIndex, output)` (Activity B)
|
|
* "assemble" → `assemble(planDir, chunkPaths, audio, out)` (Activity C)
|
|
*
|
|
* All file I/O is mediated by S3 — the handler downloads inputs into
|
|
* `/tmp` (Lambda's only writable filesystem path), invokes the primitive,
|
|
* uploads outputs back to S3, and returns a small JSON payload that fits
|
|
* inside Step Functions' history budget (under 200 bytes for chunk
|
|
* results per §2.4).
|
|
*/
|
|
|
|
import type {
|
|
DistributedFormat,
|
|
SerializableDistributedRenderConfig,
|
|
} from "@hyperframes/producer/distributed";
|
|
|
|
export type { SerializableDistributedRenderConfig } from "@hyperframes/producer/distributed";
|
|
|
|
/** Discriminator for the three roles the one Lambda image fulfills. */
|
|
export type LambdaAction = "plan" | "renderChunk" | "assemble";
|
|
/** Transport protocol selected for one complete distributed render. */
|
|
export type LambdaPlanProtocol = "v1" | "v2";
|
|
|
|
/**
|
|
* Top-level shape of any event the handler may receive.
|
|
*
|
|
* Step Functions can also invoke with a wrapped payload (e.g. when a Map
|
|
* state's `ItemSelector` passes through `$$.Map.Item.Value`), so the
|
|
* handler unwraps both `event.Payload` and `event.Input` before
|
|
* dispatching.
|
|
*/
|
|
export type LambdaEvent =
|
|
| PlanEvent
|
|
| RenderChunkEvent
|
|
| AssembleEvent
|
|
| { Payload: LambdaEvent }
|
|
| { Input: LambdaEvent };
|
|
|
|
/** Activity A: produce a planDir, upload to S3. */
|
|
interface PlanEventBase {
|
|
Action: "plan";
|
|
/** S3 URI pointing at a `tar -czf`-archived project directory (`s3://bucket/key.tar.gz`). */
|
|
ProjectS3Uri: string;
|
|
/** S3 URI prefix where the planDir tar should be uploaded (`s3://bucket/{prefix}/`). */
|
|
PlanOutputS3Prefix: string;
|
|
/** `DistributedRenderConfig` minus runtime-only fields (logger, abortSignal). */
|
|
Config: SerializableDistributedRenderConfig;
|
|
}
|
|
|
|
/**
|
|
* Legacy/default plan transport. Absence is deliberately interpreted as v1.
|
|
*
|
|
* @deprecated Use {@link PlanV2Event} for new integrations.
|
|
*/
|
|
export interface PlanV1Event extends PlanEventBase {
|
|
PlanProtocol?: "v1";
|
|
}
|
|
|
|
/** Explicit opt-in to the content-addressed v2 plan transport. */
|
|
export interface PlanV2Event extends PlanEventBase {
|
|
PlanProtocol: "v2";
|
|
}
|
|
|
|
export type PlanEvent = PlanV1Event | PlanV2Event;
|
|
|
|
/** Activity B: fetch planDir, render one chunk, upload result. */
|
|
interface RenderChunkEventBase {
|
|
Action: "renderChunk";
|
|
/**
|
|
* `PlanResult.planHash` from the Plan invocation. For v1, the handler
|
|
* verifies it against the untarred planDir's `plan.json`; for v2, it
|
|
* verifies it against the content-addressed manifest before invoking the
|
|
* producer. Divergence throws a typed `PLAN_HASH_MISMATCH` so the state
|
|
* machine routes it as non-retryable.
|
|
*/
|
|
PlanHash: string;
|
|
/** 0-based chunk index this invocation should render. */
|
|
ChunkIndex: number;
|
|
/** S3 URI prefix where the chunk output should be uploaded (`s3://bucket/{prefix}/`). */
|
|
ChunkOutputS3Prefix: string;
|
|
/** Output container format from the plan's encoder.json; drives file vs frame-dir handling. */
|
|
Format: DistributedFormat;
|
|
}
|
|
|
|
/**
|
|
* Legacy/default chunk event.
|
|
*
|
|
* @deprecated Use {@link RenderChunkV2Event} for new integrations.
|
|
*/
|
|
export interface RenderChunkV1Event extends RenderChunkEventBase {
|
|
PlanProtocol?: "v1";
|
|
/** S3 URI of the v1 plan tar produced by a PlanEvent invocation. */
|
|
PlanS3Uri: string;
|
|
}
|
|
|
|
/**
|
|
* V2 chunk event. It intentionally cannot carry `PlanS3Uri`: the manifest
|
|
* describes the exact content-addressed artifacts needed by this chunk.
|
|
*/
|
|
export interface RenderChunkV2Event extends RenderChunkEventBase {
|
|
PlanProtocol: "v2";
|
|
PlanV2ManifestS3Uri: string;
|
|
PlanV2ArtifactS3Prefix: string;
|
|
}
|
|
|
|
export type RenderChunkEvent = RenderChunkV1Event | RenderChunkV2Event;
|
|
|
|
/** Activity C: fetch planDir + all chunks + audio, assemble, upload final. */
|
|
interface AssembleEventBase {
|
|
Action: "assemble";
|
|
/** S3 URIs of every chunk, ordered by chunk index. Length must equal `chunkCount`. */
|
|
ChunkS3Uris: string[];
|
|
/** S3 URI of the planDir's audio artifact if the composition has audio; `null` otherwise. */
|
|
AudioS3Uri: string | null;
|
|
/** Final output S3 URI (`s3://bucket/key.mp4`). */
|
|
OutputS3Uri: string;
|
|
/** Output container format; drives file vs frame-dir handling. */
|
|
Format: DistributedFormat;
|
|
/**
|
|
* Optional exact-CFR re-encode at assemble time. When `true`, the final
|
|
* assembled video is re-encoded with `-fps_mode cfr -r <fps>` so the
|
|
* stream's `avg_frame_rate` matches the container's `r_frame_rate`
|
|
* exactly (and the file's duration is exact, not PTS-derived). Trade-off
|
|
* is ~2-5x the assemble wall-clock. mp4 only — webm / mov stream-copy
|
|
* paths already produce exact avg_frame_rate. Default `false` /
|
|
* unset preserves current `-c copy` behavior.
|
|
*/
|
|
Cfr?: boolean;
|
|
}
|
|
|
|
/**
|
|
* Legacy/default assemble event.
|
|
*
|
|
* @deprecated Use {@link AssembleV2Event} for new integrations.
|
|
*/
|
|
export interface AssembleV1Event extends AssembleEventBase {
|
|
PlanProtocol?: "v1";
|
|
/** S3 URI of the v1 plan tar produced by a PlanEvent invocation. */
|
|
PlanS3Uri: string;
|
|
}
|
|
|
|
/** V2 assemble event, scoped to manifest-declared assembler artifacts. */
|
|
export interface AssembleV2Event extends AssembleEventBase {
|
|
PlanProtocol: "v2";
|
|
PlanV2ManifestS3Uri: string;
|
|
PlanV2ArtifactS3Prefix: string;
|
|
PlanHash: string;
|
|
}
|
|
|
|
export type AssembleEvent = AssembleV1Event | AssembleV2Event;
|
|
|
|
// ── Result types — kept small to fit Step Functions history budgets ─────────
|
|
|
|
/** Result of a `plan` invocation. Carries enough to size the Map(N) state. */
|
|
interface PlanLambdaResultBase {
|
|
Action: "plan";
|
|
PlanHash: string;
|
|
ChunkCount: number;
|
|
TotalFrames: number;
|
|
Fps: 24 | 30 | 60;
|
|
Width: number;
|
|
Height: number;
|
|
Format: DistributedFormat;
|
|
HasAudio: boolean;
|
|
AudioS3Uri: string | null;
|
|
FfmpegVersion: string;
|
|
ProducerVersion: string;
|
|
DurationMs: number;
|
|
}
|
|
|
|
/**
|
|
* Existing v1 result. Kept unchanged for wire compatibility.
|
|
*
|
|
* @deprecated New integrations should consume {@link PlanV2LambdaResult}.
|
|
*/
|
|
export interface PlanV1LambdaResult extends PlanLambdaResultBase {
|
|
PlanS3Uri: string;
|
|
}
|
|
|
|
/** V2 result. The two v2 locators are never aliases for `PlanS3Uri`. */
|
|
export interface PlanV2LambdaResult extends PlanLambdaResultBase {
|
|
PlanProtocol: "v2";
|
|
PlanV2ManifestS3Uri: string;
|
|
PlanV2ArtifactS3Prefix: string;
|
|
}
|
|
|
|
export type PlanLambdaResult = PlanV1LambdaResult | PlanV2LambdaResult;
|
|
|
|
/** Result of a `renderChunk` invocation. Sized ≤200 bytes per §2.4. */
|
|
export interface RenderChunkLambdaResult {
|
|
Action: "renderChunk";
|
|
ChunkS3Uri: string;
|
|
ChunkIndex: number;
|
|
Sha256: string;
|
|
FramesEncoded: number;
|
|
/** Effective engine mode after browser probing. Emitted by current handlers. */
|
|
CaptureMode?: "beginframe" | "screenshot" | "drawelement";
|
|
DurationMs: number;
|
|
}
|
|
|
|
/** Result of an `assemble` invocation. */
|
|
export interface AssembleLambdaResult {
|
|
Action: "assemble";
|
|
OutputS3Uri: string;
|
|
FramesEncoded: number;
|
|
FileSize: number;
|
|
DurationMs: number;
|
|
}
|
|
|
|
export type LambdaResult = PlanLambdaResult | RenderChunkLambdaResult | AssembleLambdaResult;
|