Files
hyperframes/.github/workflows/windows-render.yml
T
Miguel Ángel 6a59ef6106 fix: skip metadata waits for injected video frames (#575)
## Problem

Closes #574.

On Windows with cached headless-shell Chrome, a composition that reuses the same video file in three timeline clips can fail before frame capture starts:

```html
<video id="video1" src="1.mp4" data-start="0" muted data-duration="4" data-track-index="0" data-media-start="0"></video>
<video id="video2" src="1.mp4" data-start="4" muted data-duration="4" data-track-index="0" data-media-start="4"></video>
<video id="video3" src="1.mp4" data-start="8" muted data-duration="4" data-track-index="0" data-media-start="8"></video>
```

The reported render reaches video frame extraction, then dies at frame-capture initialization with:

```text
[FrameCapture] video metadata not ready after 45000ms. Video elements must load metadata before capture starts.
```

The important detail is that by this stage HyperFrames has already extracted video pixels through FFmpeg. Native Chromium video metadata is only being waited on for DOM layout stability, not because Chromium is the source of rendered pixels.

## Root Cause

The render pipeline has two separate media responsibilities:

- FFmpeg extracts video frames and audio from declared media.
- Chromium owns DOM layout and capture, while injected FFmpeg frames supply the video pixels before each captured frame.

Before this PR, every capture session still waited for every DOM `<video>` to reach `readyState >= 1` unless the element was a native HDR exception. That made native browser media metadata a hard render prerequisite even when the browser would not decode or provide the final video pixels.

That is why the issue fails at `25% Starting frame capture`: FFmpeg extraction has already succeeded, but capture initialization blocks on repeated native `<video src="1.mp4">` metadata loading in cached Windows headless-shell Chrome.

There was a second constraint: the readiness wait also prevents first-frame layout bugs. If a skipped `<video>` has no native metadata, Chromium can use the default `300x150` intrinsic video size, which breaks layouts such as `width: 100%; height: auto` before the first injected frame. The fix therefore must not simply skip all video readiness waits; it must provide dimensions for any skipped videos.

## What This Fixes

- Treats videos with successfully extracted FFmpeg frames and usable dimensions as out-of-band rendered video sources.
- Skips native browser metadata readiness waits for those extracted videos because Chromium is not responsible for their pixels.
- Passes FFmpeg-probed dimensions into capture as `videoMetadataHints`.
- Applies those hints before the readiness wait in both screenshot and BeginFrame initialization paths.
- Sets missing `width` / `height` attributes and an explicit `aspect-ratio` only when the element does not already provide one, preserving author styles where present.
- Keeps native HDR video IDs in the skip list, preserving the existing HEVC/HDR behavior where Chrome may not decode the source but FFmpeg/native HDR compositing can still render it.
- Uses one `buildCaptureOptions()` helper so calibration, HDR DOM capture, streaming capture, parallel capture, and sequential capture receive the same skip IDs and metadata hints.
- Adds tests for the skip-list and metadata-hint contract.
- Adds a Windows CI regression that reproduces the issue shape after the canary render warms the cached-browser path.

## Reviewer Map

Primary files:

- `packages/producer/src/services/renderOrchestrator.ts`
  - `collectVideoReadinessSkipIds()` includes native HDR IDs plus extracted videos that have finite positive FFmpeg dimensions.
  - `collectVideoMetadataHints()` converts extracted FFmpeg metadata into capture hints.
  - `buildCaptureOptions()` threads `skipReadinessVideoIds` and `videoMetadataHints` into every capture path.
- `packages/engine/src/services/frameCapture.ts`
  - `applyVideoMetadataHints()` runs in the page before video readiness polling.
  - Both screenshot and BeginFrame initialization call it before checking non-skipped videos for `readyState >= 1`.
- `packages/engine/src/types.ts`
  - Adds `CaptureVideoMetadataHint` and documents that readiness skips should be paired with metadata hints when layout may depend on intrinsic dimensions.
- `packages/producer/src/services/renderOrchestrator.test.ts`
  - Covers that extracted videos with dimensions are skipped, invalid dimensions are not, native HDR IDs are preserved, and hints are stable/sorted.
- `.github/workflows/windows-render.yml`
  - Adds the issue #574 Windows regression with the exact three-clip markup and a generated deterministic `1.mp4`.

## Why This Is Safe

The skip is intentionally gated:

- A standard video is skipped only after `extractAllVideoFrames()` succeeded for that video and returned usable dimensions.
- Videos with invalid dimensions are not skipped, so the old browser readiness guard still applies.
- DOM videos are still present for layout and element bounds; only the native metadata wait is skipped for sources whose pixels come from FFmpeg injection.
- Metadata hints are applied conservatively: existing `width`, `height`, and explicit `aspect-ratio` are not overwritten.
- Non-extracted videos, images, fonts, page readiness, and `window.__hf` readiness keep the existing waits.
- The fix is not limited to the sequential path from the issue; it is threaded through calibration, HDR DOM capture, streaming encode, parallel capture, and sequential capture.

A first local revision skipped readiness too broadly and caused `overlay-montage-prod` first-frame layout shrinkage. The current version fixes that by pairing skips with FFmpeg metadata hints; `overlay-montage-prod` now passes and is listed in verification below.

## Verification

### Root-Cause Reproduction Before Fix

The reporter did not attach the actual `1.mp4`, so the regression uses the exact issue markup and a deterministic generated 12s H.264 file named `1.mp4`.

I reproduced the failure in GitHub Actions by running this branch's new Windows workflow against unpatched `main`:

```bash
gh workflow run windows-render.yml --repo heygen-com/hyperframes --ref fix/reused-video-metadata -f ref=main
```

That means the workflow contains the new issue #574 regression, but the code under test is `main` without this fix.

Baseline failure:

- Run: https://github.com/heygen-com/hyperframes/actions/runs/25174603730
- Failed job: https://github.com/heygen-com/hyperframes/actions/runs/25174603730/job/73803179086
- Checkout proof: `ref: main`, `origin/main`, commit `8662598a3ac64018a2999d189ffb369e6d46b53a`.
- Failure proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, then `25% Starting frame capture` -> `[FrameCapture] video metadata not ready after 15000ms`.

This is the same failure class as the issue, on Windows, in cached-browser mode, before the fix.

### Fixed Windows Regression

The same regression passes on this PR branch:

- Run: https://github.com/heygen-com/hyperframes/actions/runs/25175048215
- Passing job: https://github.com/heygen-com/hyperframes/actions/runs/25175048215/job/73804781017
- Checkout proof: PR merge contains `79d6b41c9f2ba137cbfb9301678e0815b16c4f5a` merged into `8662598a3ac64018a2999d189ffb369e6d46b53a`.
- Passing proof: `Browser: cache`, `staticDuration:12`, `videoCount:3`, `25% Starting frame capture`, captures `360/360` frames, renders `issue-574.mp4`, and `ffprobe` verifies `1920x1080 @ 30/1, 12s`.

### Local Checks

- `bun run build:hyperframes-runtime`
- `bunx vitest run packages/producer/src/services/renderOrchestrator.test.ts`
- `bun run --filter @hyperframes/producer typecheck`
- `bun run --filter @hyperframes/engine typecheck`
- `bunx oxlint packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `bunx oxfmt --check .github/workflows/windows-render.yml packages/engine/src/services/frameCapture.ts packages/engine/src/types.ts packages/engine/src/index.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts`
- `git diff --check`
- Lefthook pre-commit: lint, format, typecheck where applicable
- Lefthook commit-msg: commitlint

### Local Render Checks

- Created `/tmp/hf-issue-574-repro` with the issue shape: three clips using the same `1.mp4`, `data-media-start=0/4/8`, 12s total.
- `PRODUCER_PLAYER_READY_TIMEOUT_MS=5000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-repro --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-h264-fixed-v2.mp4` -> completed.
- Created `/tmp/hf-issue-574-prores` with the same three-clip shape using one FFmpeg-readable ProRes `.mov`, which exercises the browser-metadata failure class because Chromium should not be needed to decode the source.
- `PRODUCER_PLAYER_READY_TIMEOUT_MS=3000 bun packages/cli/src/cli.ts render /tmp/hf-issue-574-prores --workers 1 --quality draft --fps 30 --output /tmp/hf-issue-574-prores-fixed-v2.mp4` -> completed.
- `bun run --filter @hyperframes/producer test --sequential --keep-temp overlay-montage-prod` -> passed; this guards against skipped metadata shrinking `height:auto` video layout before the first injected frame.
- `ffmpeg -v error -i /tmp/hf-issue-574-prores-fixed-v2.mp4 -f null -`
- `ffmpeg -v error -i /tmp/hf-issue-574-h264-fixed-v2.mp4 -f null -`
- `ffprobe -v error -show_entries format=duration:stream=codec_name,width,height,r_frame_rate -of json /tmp/hf-issue-574-h264-fixed-v2.mp4` -> H.264, 320x180, 30fps, 12.0s.

### Current PR Checks

- Windows render verification: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215.
- Windows tests: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048215.
- Main CI build/lint/typecheck/test/smoke jobs: pass on https://github.com/heygen-com/hyperframes/actions/runs/25175048175.
- Regression shards observed passing include HDR, render-compat, styles A-G, and `overlay-montage-prod`. At the time this body was updated, the `fast` regression shard was still in progress in run https://github.com/heygen-com/hyperframes/actions/runs/25174515546.

### Browser Verification

- Used `agent-browser` to open `file:///tmp/hf-issue-574-h264-fixed-v2.mp4` and verify the rendered output displays in Chromium.
- Screenshot: `.debug/issue-574/h264-output-page.png`
- Agent-browser recording: `.debug/issue-574/h264-output-playback.webm`

## Notes / Caveats

- The reporter's exact `1.mp4` was not attached to #574. The committed Windows regression uses a generated deterministic H.264 file with the same filename and exact markup from the issue.
- The exact H.264 issue shape did not reproduce the timeout on this macOS/system-Chrome machine before the fix; it rendered successfully locally. The GitHub Actions baseline above reproduces it on Windows/cache without the fix.
- The Windows fixture intentionally runs after the existing canary render so the browser path is `Browser: cache`, matching the reporter's environment.
- The generated fixture emits sparse-keyframe warnings. Those warnings are expected and are not the failure being fixed; the baseline failure occurs before any frame capture because native browser video metadata never becomes ready.
- Browser proof artifacts are local-only under `.debug/issue-574/` and intentionally not committed.
2026-04-30 18:50:57 +02:00

376 lines
15 KiB
YAML

name: Windows render verification
# Manually triggered smoke test that renders a HyperFrames composition on a
# real Windows runner. Proves the PR #336 `where ffmpeg` fix actually works
# end-to-end: FFmpeg is discovered natively on Windows, Chrome is installed
# and launched, frames are captured, and an MP4 is produced — without Docker
# or WSL.
on:
pull_request:
# `edited` is required so the workflow re-fires when a PR's base ref is
# set back to `main` after a Graphite stack restack momentarily flips
# the base off of `main`. Without it, `pull_request` triggers are not
# re-evaluated on `base_ref_changed`, leaving required checks skipped
# for that head SHA forever.
types: [opened, synchronize, reopened, edited]
branches: [main]
push:
branches: [main]
workflow_dispatch:
inputs:
ref:
description: "Git ref to render (branch / tag / SHA)."
required: false
default: "main"
concurrency:
group: windows-render-${{ github.ref }}
cancel-in-progress: true
jobs:
changes:
name: Detect changes
runs-on: ubuntu-latest
timeout-minutes: 2
outputs:
code: ${{ steps.filter.outputs.code }}
steps:
# Force git-based change detection instead of the pull_request REST API.
# The API path can fail the workflow on transient listFiles timeouts
# before the Windows render jobs even start.
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
fetch-depth: 0
- uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4
id: filter
with:
token: ""
filters: |
code:
- "packages/**"
- "scripts/**"
- "package.json"
- "bun.lock"
- ".github/workflows/windows-render.yml"
render-windows:
name: Render on windows-latest
needs: changes
if: needs.changes.outputs.code == 'true' || github.event_name == 'workflow_dispatch'
runs-on: windows-latest
timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
ref: ${{ github.event.inputs.ref }}
lfs: true
- name: Show platform info
shell: pwsh
run: |
Write-Host "OS: $([System.Environment]::OSVersion.VersionString)"
Write-Host "PowerShell: $($PSVersionTable.PSVersion)"
Write-Host "Runner: windows-latest"
# -----------------------------------------------------------------
# Install FFmpeg via the shared composite action so the install logic
# stays identical between this job and `test-windows` below. See
# .github/actions/install-ffmpeg-windows for why we bypass Chocolatey.
# -----------------------------------------------------------------
- name: Install FFmpeg
uses: ./.github/actions/install-ffmpeg-windows
# -----------------------------------------------------------------
# Verify FFmpeg feature inventory.
#
# The engine shells out to a fixed set of encoders (libx264 for MP4,
# libx265 for HEVC, libvpx-vp9 for WebM, prores_ks for transparent
# MOV, aac for audio), muxers (mp4 / mov / webm), and demuxers
# (image2pipe for streaming RGBA frames, rawvideo for HDR PQ frames,
# mov,mp4 for video frame extraction). Some of these are GPL-only,
# so a future build swap could silently drop one and break a code
# path the canary render doesn't exercise. Fail fast here instead.
# -----------------------------------------------------------------
- name: Verify FFmpeg feature inventory
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
function Assert-FfmpegFeature {
param(
[Parameter(Mandatory)] [string] $Listing,
[Parameter(Mandatory)] [string] $Name,
[Parameter(Mandatory)] [string] $Kind
)
# `ffmpeg -encoders` etc. emit one feature per line as
# `<flags> <name> <description>`, so a whitespace boundary on
# each side is enough to disambiguate (e.g. `mov` vs `movflags`).
$pattern = "(^|\s)$([regex]::Escape($Name))(\s|$)"
if ($Listing -notmatch $pattern) {
throw "Required FFmpeg $Kind '$Name' not present in this build"
}
Write-Host " ok: $Kind $Name"
}
Write-Host "--- encoders ---"
$encoders = (& ffmpeg -hide_banner -encoders 2>&1) -join "`n"
foreach ($enc in @('libx264', 'libx265', 'libvpx-vp9', 'prores_ks', 'aac')) {
Assert-FfmpegFeature -Listing $encoders -Name $enc -Kind 'encoder'
}
Write-Host "--- muxers ---"
$muxers = (& ffmpeg -hide_banner -muxers 2>&1) -join "`n"
foreach ($mux in @('mp4', 'mov', 'webm')) {
Assert-FfmpegFeature -Listing $muxers -Name $mux -Kind 'muxer'
}
Write-Host "--- demuxers ---"
$demuxers = (& ffmpeg -hide_banner -demuxers 2>&1) -join "`n"
foreach ($dem in @('image2pipe', 'rawvideo', 'mov,mp4,m4a,3gp,3g2,mj2')) {
Assert-FfmpegFeature -Listing $demuxers -Name $dem -Kind 'demuxer'
}
- name: Install Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
- name: Install Node
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 22
- name: Install dependencies
shell: pwsh
run: bun install --frozen-lockfile
- name: Build all packages
shell: pwsh
run: bun run build
# -----------------------------------------------------------------
# Prove the PR #336 fix: hyperframes doctor exercises findFFmpeg()
# and whichBinary() — both must pass on Windows without workarounds.
# -----------------------------------------------------------------
- name: hyperframes doctor (verifies `where ffmpeg` fix)
shell: pwsh
run: node packages/cli/dist/cli.js doctor
- name: Scaffold canary composition
shell: pwsh
run: |
New-Item -ItemType Directory -Force -Path "$env:RUNNER_TEMP\windows-canary" | Out-Null
cd "$env:RUNNER_TEMP\windows-canary"
node "$env:GITHUB_WORKSPACE\packages\cli\dist\cli.js" init canary --example blank --non-interactive --skip-skills
$fixtures = "$env:GITHUB_WORKSPACE\.github\workflows\fixtures"
Copy-Item "$fixtures\windows-canary.html" "canary\index.html" -Force
- name: Render canary composition
shell: pwsh
run: |
cd "$env:RUNNER_TEMP\windows-canary\canary"
node "$env:GITHUB_WORKSPACE\packages\cli\dist\cli.js" render `
--fps 30 `
--quality draft `
--workers 2 `
--output renders\canary.mp4
- name: Verify rendered MP4
shell: pwsh
run: |
$mp4 = "$env:RUNNER_TEMP\windows-canary\canary\renders\canary.mp4"
if (-not (Test-Path $mp4)) { throw "canary.mp4 not produced" }
$probe = ffprobe -v error -select_streams v:0 `
-show_entries stream=width,height,r_frame_rate -show_entries format=duration `
-of default=noprint_wrappers=1 $mp4
Write-Host $probe
# Parse probe output
$width = ($probe | Select-String '^width=(.+)$').Matches.Groups[1].Value
$height = ($probe | Select-String '^height=(.+)$').Matches.Groups[1].Value
$fps = ($probe | Select-String '^r_frame_rate=(.+)$').Matches.Groups[1].Value
$duration = [double]($probe | Select-String '^duration=(.+)$').Matches.Groups[1].Value
if ([int]$width -ne 1920) { throw "expected 1920 width, got $width" }
if ([int]$height -ne 1080) { throw "expected 1080 height, got $height" }
if ($fps -ne "30/1") { throw "expected 30fps, got $fps" }
if ($duration -lt 7.5 -or $duration -gt 8.5) { throw "expected ~8s duration, got $duration" }
Write-Host "canary.mp4 ok: ${width}x${height} @ $fps, ${duration}s"
- name: Scaffold issue #574 reused-video regression
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
$project = "$env:RUNNER_TEMP\issue-574-reused-video"
New-Item -ItemType Directory -Force -Path $project | Out-Null
cd $project
ffmpeg -y `
-f lavfi -i "testsrc2=size=1920x1080:rate=30:duration=12" `
-f lavfi -i "sine=frequency=880:sample_rate=48000:duration=12" `
-c:v libx264 `
-pix_fmt yuv420p `
-r 30 `
-g 250 `
-keyint_min 250 `
-c:a aac `
-shortest `
1.mp4
@'
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Issue 574 reused video regression</title>
<style>
html,
body {
margin: 0;
padding: 0;
background: #000;
}
#root {
position: relative;
width: 1920px;
height: 1080px;
overflow: hidden;
background: #000;
}
video {
position: absolute;
inset: 0;
width: 100%;
height: 100%;
object-fit: cover;
}
</style>
</head>
<body>
<div
id="root"
data-composition-id="root"
data-start="0"
data-duration="12"
data-width="1920"
data-height="1080"
>
<video id="video1" src="1.mp4" data-start="0" muted data-duration="4" data-track-index="0" data-media-start="0"></video>
<video id="video2" src="1.mp4" data-start="4" muted data-duration="4" data-track-index="0" data-media-start="4"></video>
<video id="video3" src="1.mp4" data-start="8" muted data-duration="4" data-track-index="0" data-media-start="8"></video>
</div>
<script>
window.__timelines = window.__timelines || {};
</script>
</body>
</html>
'@ | Set-Content -Path index.html -Encoding utf8
- name: Render issue #574 reused-video regression
shell: pwsh
env:
PRODUCER_PLAYER_READY_TIMEOUT_MS: "15000"
run: |
cd "$env:RUNNER_TEMP\issue-574-reused-video"
node "$env:GITHUB_WORKSPACE\packages\cli\dist\cli.js" render `
--fps 30 `
--quality standard `
--workers 1 `
--output renders\issue-574.mp4
- name: Verify issue #574 rendered MP4
shell: pwsh
run: |
$mp4 = "$env:RUNNER_TEMP\issue-574-reused-video\renders\issue-574.mp4"
if (-not (Test-Path $mp4)) { throw "issue-574.mp4 not produced" }
$probe = ffprobe -v error -select_streams v:0 `
-show_entries stream=width,height,r_frame_rate -show_entries format=duration `
-of default=noprint_wrappers=1 $mp4
Write-Host $probe
$width = ($probe | Select-String '^width=(.+)$').Matches.Groups[1].Value
$height = ($probe | Select-String '^height=(.+)$').Matches.Groups[1].Value
$fps = ($probe | Select-String '^r_frame_rate=(.+)$').Matches.Groups[1].Value
$duration = [double]($probe | Select-String '^duration=(.+)$').Matches.Groups[1].Value
if ([int]$width -ne 1920) { throw "expected 1920 width, got $width" }
if ([int]$height -ne 1080) { throw "expected 1080 height, got $height" }
if ($fps -ne "30/1") { throw "expected 30fps, got $fps" }
if ($duration -lt 11.5 -or $duration -gt 12.5) { throw "expected ~12s duration, got $duration" }
Write-Host "issue-574.mp4 ok: ${width}x${height} @ $fps, ${duration}s"
- name: Upload rendered MP4 artifact
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: windows-render-${{ github.run_id }}
path: |
${{ runner.temp }}/windows-canary/canary/renders/canary.mp4
${{ runner.temp }}/issue-574-reused-video/renders/issue-574.mp4
if-no-files-found: error
retention-days: 7
# -------------------------------------------------------------------
# Unit-test suites on Windows. Mirrors the Linux `test` job in ci.yml
# so we catch Windows-specific regressions (path separators, shell
# invocations, CRLF, file URLs, etc.) in existing vitest suites.
# The producer package is skipped because its tests require Docker /
# Linux-only tooling (Dockerfile.test, LFS golden MP4 baselines).
# -------------------------------------------------------------------
test-windows:
name: Tests on windows-latest
needs: changes
if: needs.changes.outputs.code == 'true' || github.event_name == 'workflow_dispatch'
runs-on: windows-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
ref: ${{ github.event.inputs.ref }}
lfs: true
# -----------------------------------------------------------------
# Install FFmpeg so vitest suites that gate on `HAS_FFMPEG`
# (e.g. packages/engine videoFrameExtractor.test.ts) actually run on
# Windows. Without it those suites `describe.skipIf(!HAS_FFMPEG)`
# themselves silently and any Windows-specific regression in the
# FFmpeg-driven code paths would not be caught here.
# -----------------------------------------------------------------
- name: Install FFmpeg
uses: ./.github/actions/install-ffmpeg-windows
- name: Install Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
- name: Install Node
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 22
- name: Install dependencies
shell: pwsh
run: bun install --frozen-lockfile
- name: Build
shell: pwsh
run: bun run build
- name: Run tests (all packages except producer)
shell: pwsh
run: bun run --filter "!@hyperframes/producer" test
- name: Run runtime contract test
shell: pwsh
run: bun run --filter "@hyperframes/core" test:hyperframe-runtime-ci