perf(aws-lambda): size distributed renders from account concurrency limits (#5813)

* perf(aws-lambda): run chunks through a Distributed Map to lift the 40-chunk cap The inline Map runs at most 40 chunk renders at once. Both state machine definitions (CDK and SAM) now use a Distributed Map with STANDARD child executions, capped at the function's reserved concurrency so chunks past it queue instead of being throttled. Progress reads each chunk's child execution history so frames, cost and chunk errors still show up. * fix(aws-lambda): retry Lambda throttles for minutes and stop on oversized chunks A throttled task used to burn its four catch-all retries in about 30 s. It now gets its own jittered backoff of several minutes, because another render on the same function can still hold the reserved slots. Plan treats CHUNK_EXCEEDS_WORKER_TIMEOUT as terminal in both definitions. * chore(aws-lambda): trim stack comments to keep the comment share flat * fix(aws-lambda): keep render --wait working at hundreds of chunks Progress polls read the chunk Map Run's counts (one DescribeMapRun) instead of every child execution's history, take exact chunk frames and cost from the Map output once it exits, and read failed children only when the counts show a failure. The SDK client retries throttled calls up to 8 times with backoff. * feat(aws-lambda): size each render's concurrency to the account instead of a fixed 8 * fix(lambda): keep the concurrency SDK external in published modules * fix(cli): keep an existing stack's reserved concurrency on redeploy Redeploying without --concurrency sent ReservedConcurrency=-1, which removed the cap of 8 that earlier CLI versions reserved by default and stopped recording it in the state file. The override is now left out when the flag is omitted, so SAM keeps the stack's current value (a new stack still gets the template's unreserved default). The state file records the deployed value, and deploy prints the kept reservation. * fix(aws-lambda): keep progress working for callers without DescribeMapRun Callers whose IAM policy came from an earlier `lambda policies user` lack states:DescribeMapRun, so reading a running Distributed Map threw AccessDeniedException out of `lambda progress` and `render --wait` while the render itself went on to succeed. getRenderProgress now catches only that error around the map-run read, returns the parent snapshot (status, plan and assemble progress) and adds a warning to re-apply the policy, which both commands print. Other errors still throw. * fix(cli): let the documented deploy policy drop a reserved concurrency When a redeploy removes a function's reserved concurrency, CloudFormation calls lambda:DeleteFunctionConcurrency. The policy printed by `lambda policies role|user` granted only PutFunctionConcurrency, so the update failed and rolled back. Grant it, and pin it with states:DescribeMapRun in the required-actions test. * fix(aws-lambda): read smoke chunk hashes from the Distributed Map output With chunks in a Distributed Map, their task results live in child executions, so the smoke gate's chunk-hash check found 0 chunks in the parent history and failed every v1/v2 comparison. Read Chunks[] from the RenderChunks / RenderChunksV2 MapStateExited output instead, as getRenderProgress does. The stack this script deploys only has the Distributed Map, so the inline-Map path is gone. * fix(producer): cap a pinned chunk size at the worker concurrency, not 16 With --chunk-size or --target-chunk-frames and no --max-parallel-chunks, the plan still capped the render at 16 chunks however many workers the account allowed, so --chunk-size 30 on a 60 s render gave 16 chunks of 113 frames. The cap is now one wave across the worker concurrency, as the --max-parallel-chunks docs say; 16 remains when the concurrency is unknown. * feat(cli): remove a stack's reserved concurrency with --concurrency=none Since an omitted --concurrency now keeps the stack's reservation, there was no CLI way back to the unreserved default. --concurrency=none sends ReservedConcurrency=-1, the state file records the stack as unreserved and deploy says the reservation was removed. Positive N and an omitted flag behave as before; any other value is rejected instead of being silently ignored. * chore(aws-lambda): raise the package size budget for account-sized concurrency Measured with pnpm@10.17.1 pack after rebasing on main with the S3 assembly change: 67,763 packed / 244,685 unpacked; the budget is the check's own 5% headroom over that.
2026-10-30 09:16:55 +00:00 · 2026-10-19 00:57:30 +00:00
parent 5658f70065
commit 2a14cbc924
28 changed files with 982 additions and 127 deletions
@@ -57,15 +57,15 @@ hf_decode_pcm() {
    -ac 2 -ar 48000 -c:a pcm_s16le -f s16le "$2"
 }

+# Chunks run in Distributed Map child executions; the Map's exit output carries every result.
 hf_extract_chunk_hashes() {
  local history="$1" output="$2"
  jq '[
    .events[] |
-    .taskSucceededEventDetails.output? |
-    select(type == "string") |
-    (try fromjson catch empty) |
-    .Payload |
-    select(type == "object" and .Action == "renderChunk") |
+    select(.type == "MapStateExited") |
+    .stateExitedEventDetails |
+    select(.name == "RenderChunks" or .name == "RenderChunksV2") |
+    (.output | fromjson | .Chunks // [])[] |
    {ChunkIndex, Sha256, FramesEncoded}
  ] | sort_by(.ChunkIndex)' "$history" > "$output"
 }
@@ -39,28 +39,25 @@ jq -e '
  .encoded.equal == false
 ' "$WORK/equal.json" >/dev/null

-cat > "$WORK/v1-history.json" <<'JSON'
-{"events":[
-  {"taskSucceededEventDetails":{"output":"{\"Payload\":{\"Action\":\"renderChunk\",\"ChunkIndex\":1,\"Sha256\":\"bbb\",\"FramesEncoded\":12}}"}},
-  {"taskSucceededEventDetails":{"output":"{\"Payload\":{\"Action\":\"plan\",\"ChunkCount\":2}}"}},
-  {"taskSucceededEventDetails":{"output":"{\"Payload\":{\"Action\":\"renderChunk\",\"ChunkIndex\":0,\"Sha256\":\"aaa\",\"FramesEncoded\":12}}"}}
-]}
-JSON
-cat > "$WORK/v2-history.json" <<'JSON'
-{"events":[
-  {"taskSucceededEventDetails":{"output":"{\"Payload\":{\"Action\":\"renderChunk\",\"ChunkIndex\":0,\"Sha256\":\"aaa\",\"FramesEncoded\":12}}"}},
-  {"taskSucceededEventDetails":{"output":"{\"Payload\":{\"Action\":\"renderChunk\",\"ChunkIndex\":1,\"Sha256\":\"bbb\",\"FramesEncoded\":12}}"}}
-]}
-JSON
+# Map exit output as Step Functions records it: the whole state, chunk results under Chunks.
+map_exit() {
+  jq -nc --arg name "$1" --argjson chunks "$2" \
+    '{type: "MapStateExited", stateExitedEventDetails: {name: $name, output: ({Plan: {ChunkCount: 2}, Chunks: $chunks} | tojson)}}'
+}
+CHUNKS='[{"ChunkIndex":1,"Sha256":"bbb","FramesEncoded":12,"DurationMs":9},{"ChunkIndex":0,"Sha256":"aaa","FramesEncoded":12,"DurationMs":8}]'
+jq -n --argjson exit "$(map_exit RenderChunks "$CHUNKS")" \
+  '{events: [{type: "TaskSucceeded", taskSucceededEventDetails: {output: "{}"}}, $exit]}' \
+  > "$WORK/v1-history.json"
+jq -n --argjson exit "$(map_exit RenderChunksV2 "$CHUNKS")" '{events: [$exit]}' \
+  > "$WORK/v2-history.json"
 hf_compare_render_semantics \
  "$WORK/original.mp4" "$WORK/rewrapped.mp4" "$WORK/chunks-equal" \
  "$WORK/v1-history.json" "$WORK/v2-history.json"
 jq -e '.semanticEqual == true and .chunks == {checked:true,equal:true,v1Count:2,v2Count:2}' \
  "$WORK/chunks-equal.json" >/dev/null

-jq '(.events[0].taskSucceededEventDetails.output) =
-  "{\"Payload\":{\"Action\":\"renderChunk\",\"ChunkIndex\":0,\"Sha256\":\"different\",\"FramesEncoded\":12}}"' \
-  "$WORK/v2-history.json" > "$WORK/v2-history-mismatch.json"
+jq -n --argjson exit "$(map_exit RenderChunksV2 "$(jq -c '.[1].Sha256 = "different"' <<<"$CHUNKS")")" \
+  '{events: [$exit]}' > "$WORK/v2-history-mismatch.json"
 if hf_compare_render_semantics \
  "$WORK/original.mp4" "$WORK/rewrapped.mp4" "$WORK/chunks-mismatch" \
  "$WORK/v1-history.json" "$WORK/v2-history-mismatch.json"; then
@@ -43,7 +43,8 @@ Parameters:
    Description: >-
      Lambda reserved concurrency cap. Set to a positive integer to bound
      simultaneous chunk renders (e.g. 50 to limit cost). -1 means
-      unreserved (account default).
+      unreserved: each render runs up to 90% of the account's unreserved
+      concurrency.

  ChromeSource:
    Type: String
@@ -159,6 +160,7 @@ Resources:
          # clarity + so users can override during local SAM invoke.
          TMPDIR: /tmp
          HYPERFRAMES_RENDER_BUCKET: !Ref RenderBucket
+          HYPERFRAMES_LAMBDA_RESERVED_CONCURRENCY: !Ref ReservedConcurrency
      Policies:
        - S3CrudPolicy:
            BucketName: !Ref RenderBucket
@@ -166,6 +168,11 @@ Resources:
            - Effect: Allow
              Action: s3:AbortMultipartUpload
              Resource: !Sub "arn:${AWS::Partition}:s3:::${RenderBucket}/*"
+        # Plan sizes an unreserved render to the account's concurrency.
+        - Statement:
+            - Effect: Allow
+              Action: lambda:GetAccountSettings
+              Resource: "*"
      # CloudWatch Logs perms are covered by SAM's default
      # AWSLambdaBasicExecutionRole — explicit `CloudWatchLogsFullAccess`
      # would be overscope (`logs:*` on `*`, including DeleteLogGroup +
@@ -249,6 +256,7 @@ Resources:
              PlanS3Uri.$: "$.Payload.PlanS3Uri"
              PlanHash.$: "$.Payload.PlanHash"
              ChunkCount.$: "$.Payload.ChunkCount"
+              MaxChunkConcurrency.$: "$.Payload.MaxChunkConcurrency"
              Format.$: "$.Payload.Format"
              HasAudio.$: "$.Payload.HasAudio"
              AudioS3Uri.$: "$.Payload.AudioS3Uri"
@@ -274,7 +282,15 @@ Resources:
                  - NotMediaPayloadError
                  - PLAN_ARTIFACT_DIGEST_MISMATCH
                  - FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED
+                  - CHUNK_EXCEEDS_WORKER_TIMEOUT
                MaxAttempts: 0
+              # Another render on the same function can still take the reserved slots; wait minutes.
+              - ErrorEquals: [Lambda.TooManyRequestsException]
+                IntervalSeconds: 5
+                MaxAttempts: 10
+                BackoffRate: 2
+                MaxDelaySeconds: 60
+                JitterStrategy: FULL
              - ErrorEquals: [States.ALL]
                IntervalSeconds: 2
                MaxAttempts: 4
@@ -299,6 +315,7 @@ Resources:
              PlanV2ArtifactS3Prefix.$: "$.Payload.PlanV2ArtifactS3Prefix"
              PlanHash.$: "$.Payload.PlanHash"
              ChunkCount.$: "$.Payload.ChunkCount"
+              MaxChunkConcurrency.$: "$.Payload.MaxChunkConcurrency"
              Format.$: "$.Payload.Format"
              HasAudio.$: "$.Payload.HasAudio"
            ResultPath: $.Plan
@@ -321,8 +338,15 @@ Resources:
                  - PlanV2IntegrityError
                  - PLAN_ARTIFACT_DIGEST_MISMATCH
                  - FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED
+                  - CHUNK_EXCEEDS_WORKER_TIMEOUT
                  - ChromeBinaryUnavailableError
                MaxAttempts: 0
+              - ErrorEquals: [Lambda.TooManyRequestsException]
+                IntervalSeconds: 5
+                MaxAttempts: 10
+                BackoffRate: 2
+                MaxDelaySeconds: 60
+                JitterStrategy: FULL
              - ErrorEquals: [States.ALL]
                IntervalSeconds: 2
                MaxAttempts: 4
@@ -369,6 +393,7 @@ Resources:

          RenderChunks:
            Type: Map
+            Label: RenderChunks
            ItemsPath: $.Iterator.ChunkIndexes
            ItemSelector:
              ChunkIndex.$: "$$.Map.Item.Value"
@@ -376,17 +401,15 @@ Resources:
              PlanHash.$: "$.Plan.PlanHash"
              ChunkOutputS3Prefix.$: "$.PlanOutputS3Prefix"
              Format.$: "$.Plan.Format"
-            # Map fan-out cap derives from the Plan's chunkCount so
-            # caller-supplied `Config.maxParallelChunks` (which
-            # `plan()` honours when sizing the chunk list) is the
-            # single source of truth. A hardcoded value here would
-            # silently throttle adopters who scale up the chunk count
-            # in their event payload.
-            MaxConcurrencyPath: $.Plan.ChunkCount
+            # Chunks past the plan's concurrency queue here instead of being throttled by Lambda.
+            MaxConcurrencyPath: $.Plan.MaxChunkConcurrency
            ResultPath: $.Chunks
            ItemProcessor:
              ProcessorConfig:
-                Mode: INLINE
+                # Inline Map runs at most 40 iterations at once. STANDARD children
+                # because EXPRESS ones stop at 5 minutes and a chunk may run 15.
+                Mode: DISTRIBUTED
+                ExecutionType: STANDARD
              StartAt: RenderChunk
              States:
                RenderChunk:
@@ -406,6 +429,8 @@ Resources:
                    ChunkS3Uri.$: "$.Payload.ChunkS3Uri"
                    ChunkIndex.$: "$.Payload.ChunkIndex"
                    Sha256.$: "$.Payload.Sha256"
+                    FramesEncoded.$: "$.Payload.FramesEncoded"
+                    DurationMs.$: "$.Payload.DurationMs"
                  Retry:
                    - ErrorEquals:
                        - FFMPEG_VERSION_MISMATCH
@@ -418,6 +443,12 @@ Resources:
                        - INVALID_VIDEO_METADATA
                        - PLAN_ARTIFACT_DIGEST_MISMATCH
                      MaxAttempts: 0
+                    - ErrorEquals: [Lambda.TooManyRequestsException]
+                      IntervalSeconds: 5
+                      MaxAttempts: 10
+                      BackoffRate: 2
+                      MaxDelaySeconds: 60
+                      JitterStrategy: FULL
                    - ErrorEquals: [States.ALL]
                      IntervalSeconds: 2
                      MaxAttempts: 4
@@ -460,6 +491,12 @@ Resources:
                  - PlanProtocolUnsupportedError
                  - PLAN_ARTIFACT_DIGEST_MISMATCH
                MaxAttempts: 0
+              - ErrorEquals: [Lambda.TooManyRequestsException]
+                IntervalSeconds: 5
+                MaxAttempts: 10
+                BackoffRate: 2
+                MaxDelaySeconds: 60
+                JitterStrategy: FULL
              - ErrorEquals: [States.ALL]
                IntervalSeconds: 2
                MaxAttempts: 4
@@ -469,6 +506,7 @@ Resources:

          RenderChunksV2:
            Type: Map
+            Label: RenderChunksV2
            ItemsPath: $.Iterator.ChunkIndexes
            ItemSelector:
              ChunkIndex.$: "$$.Map.Item.Value"
@@ -477,11 +515,13 @@ Resources:
              PlanHash.$: "$.Plan.PlanHash"
              ChunkOutputS3Prefix.$: "$.PlanOutputS3Prefix"
              Format.$: "$.Plan.Format"
-            MaxConcurrencyPath: $.Plan.ChunkCount
+            # Chunks past the plan's concurrency queue here instead of being throttled by Lambda.
+            MaxConcurrencyPath: $.Plan.MaxChunkConcurrency
            ResultPath: $.Chunks
            ItemProcessor:
              ProcessorConfig:
-                Mode: INLINE
+                Mode: DISTRIBUTED
+                ExecutionType: STANDARD
              StartAt: RenderChunkV2
              States:
                RenderChunkV2:
@@ -502,6 +542,8 @@ Resources:
                    ChunkS3Uri.$: "$.Payload.ChunkS3Uri"
                    ChunkIndex.$: "$.Payload.ChunkIndex"
                    Sha256.$: "$.Payload.Sha256"
+                    FramesEncoded.$: "$.Payload.FramesEncoded"
+                    DurationMs.$: "$.Payload.DurationMs"
                  Retry:
                    - ErrorEquals:
                        - FFMPEG_VERSION_MISMATCH
@@ -518,6 +560,12 @@ Resources:
                        - PLAN_ARTIFACT_DIGEST_MISMATCH
                        - ChromeBinaryUnavailableError
                      MaxAttempts: 0
+                    - ErrorEquals: [Lambda.TooManyRequestsException]
+                      IntervalSeconds: 5
+                      MaxAttempts: 10
+                      BackoffRate: 2
+                      MaxDelaySeconds: 60
+                      JitterStrategy: FULL
                    - ErrorEquals: [States.ALL]
                      IntervalSeconds: 2
                      MaxAttempts: 4
@@ -561,6 +609,12 @@ Resources:
                  - PLAN_ARTIFACT_DIGEST_MISMATCH
                  - ChromeBinaryUnavailableError
                MaxAttempts: 0
+              - ErrorEquals: [Lambda.TooManyRequestsException]
+                IntervalSeconds: 5
+                MaxAttempts: 10
+                BackoffRate: 2
+                MaxDelaySeconds: 60
+                JitterStrategy: FULL
              - ErrorEquals: [States.ALL]
                IntervalSeconds: 2
                MaxAttempts: 4
@@ -587,6 +641,20 @@ Resources:
              - Effect: Allow
                Action: lambda:InvokeFunction
                Resource: !GetAtt RenderFunction.Arn
+        # A Distributed Map runs each chunk as a child execution of this state machine.
+        # ARNs built by name: !Ref RenderStateMachine here would be a dependency cycle.
+        - PolicyName: RunDistributedMapChildren
+          PolicyDocument:
+            Version: "2012-10-17"
+            Statement:
+              - Effect: Allow
+                Action: states:StartExecution
+                Resource: !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${ProjectName}-render"
+              - Effect: Allow
+                Action:
+                  - states:DescribeExecution
+                  - states:StopExecution
+                Resource: !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:execution:${ProjectName}-render:*"
        - PolicyName: WriteCloudwatchLogs
          PolicyDocument:
            Version: "2012-10-17"