mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 12:54:29 +00:00
## Problem Template-wrapped sub-compositions could still lose correct parent timing during render in more than one place. In the validated repros, a host sub-composition starting after the intro (and in one follow-up repro, starting at `20s` after earlier compositions) contained scene-local media inside it. On the broken paths: - template-wrapped media could be missed during compile and scheduled at raw scene-local time - already-correct first-pass offsets could be clobbered during `recompileWithResolutions()` - even after those two fixes, the browser-metadata reconcile step in `executeRenderJob()` could still overwrite a compiled global `end` with a scene-local `data-end` from the inlined DOM, clipping the tail off late-start sub-composition media ## What this fixes ### Template-wrapped media discovery - `parseVideoElements`, `parseImageElements`, and `parseAudioElements` now unwrap a single top-level `<template>` wrapper before scraping media - the unwrap helper is DOM-based, not regex-based, so it avoids the CodeQL backtracking warning and only unwraps the exact single-wrapper shape we want - multiple sibling templates or other top-level content are left untouched instead of being rewritten heuristically ### Offset preservation after duration resolution - `recompileWithResolutions()` now preserves the first-pass sub-composition media arrays when the already-inlined HTML no longer contains `[data-composition-src]` hosts - that prevents correctly offset media metadata from being overwritten by scene-local media parsed from the merged DOM ### Browser metadata reconciliation in the compiled time origin - browser-discovered media can still report scene-local `data-start` / `data-end` from the merged DOM after inlining - the producer now reprojects browser `end` values into the compiled element's time origin before reconciling them back into `composition.videos` / `composition.audios` - this prevents late-start sub-composition media from getting truncated back to a scene-local end during the probe phase ### Regression coverage - adds focused engine tests for the template unwrap helper - adds producer regression coverage for both the initial compile path and the post-inline `recompileWithResolutions()` path - adds producer regression coverage for late-start host compositions (`t≈20`) with scene-local media inside them - adds producer unit coverage for the browser-end reprojection helper used by the reconcile path ## Root cause There were three distinct renderer failures behind the bug: ### 1. Template contents were invisible to the media scrapers `parseSubCompositions()` reads raw sub-composition HTML and applies the host offset to discovered media. But the engine media helpers were querying the parsed document directly, and linkedom follows browser semantics here: top-level `<template>` contents live in a `DocumentFragment`, so `querySelectorAll()` never saw those `<video>` / `<audio>` / `<img>` nodes. That meant template-wrapped sub-compositions could silently produce zero discovered media during the first pass. ### 2. The duration-resolution recompile could clobber already-correct offsets After the browser resolves composition durations, `recompileWithResolutions()` reparses the already-inlined HTML. By that point the original `[data-composition-src]` hosts are gone, so `parseSubCompositions()` legitimately returns no nested media. The old code still rebuilt the deduped media arrays from the merged DOM, which let scene-local media parsed from the inlined HTML overwrite the correctly offset first-pass metadata. ### 3. The browser probe reconcile path mixed two timing coordinate systems `discoverMediaFromBrowser()` reads `data-start` / `data-end` directly from the live DOM after sub-compositions are already inlined. For nested media, those attributes can still be scene-local even though the compiled metadata has already been offset into the parent host timeline. The old reconcile path compared those values directly and overwrote `existing.end` whenever the numbers differed. For a late-start sub-composition, that could replace a correct global end like `25.5` with a scene-local end like `5.5`, cutting the clip off during render. ## Verification ### Local checks - `bun test packages/engine/src/utils/htmlTemplate.test.ts` - `bun test packages/producer/src/services/htmlCompiler.test.ts` - `bunx vitest run packages/producer/src/services/renderOrchestrator.test.ts` - `bun run --filter @hyperframes/engine test` - `bun run --filter @hyperframes/engine typecheck` - `bun run --filter @hyperframes/producer typecheck` - `bunx oxlint packages/engine/src/utils/htmlTemplate.ts packages/engine/src/utils/htmlTemplate.test.ts packages/producer/src/services/renderOrchestrator.ts packages/producer/src/services/renderOrchestrator.test.ts packages/producer/src/services/htmlCompiler.test.ts` - `bunx oxfmt --check packages/engine/src/utils/htmlTemplate.ts packages/engine/src/utils/htmlTemplate.test.ts` - `bun run build:producer` ### Render / browser verification Verified against two local repros: 1. **Early offset repro** - host starts at `2s` - child media is scene-local `0-4s` - compiled render summary keeps the child video/audio at `start: 2` - browser verification via `agent-browser` confirmed the `2.2s` frame still shows the child clip active in the host timeline 2. **Late offset repro** - earlier compositions run first, then the target host starts at `20s` - child media starts scene-local at `1.5s` and should remain visible through `24.5s` - compiled render summary keeps the child video/audio at `start: 21.5`, `end: 25.5` - browser verification via `agent-browser` confirmed the `24.5s` frame still shows the late clip visible, which is the exact tail-clipping case the old reconcile path could break ## Notes - the `/tmp/hf-pr475-repro` and `/tmp/hf-pr476-late-offset-repro` projects plus their browser-proof artifacts are verification-only and are not part of this PR - this PR stays narrowly scoped to sub-composition media timing across compile, recompile, and browser probe reconciliation; it does not broaden into general sub-composition HTML normalization beyond the single-wrapper case
267 lines
10 KiB
YAML
267 lines
10 KiB
YAML
name: Windows render verification
|
|
|
|
# Manually triggered smoke test that renders a HyperFrames composition on a
|
|
# real Windows runner. Proves the PR #336 `where ffmpeg` fix actually works
|
|
# end-to-end: FFmpeg is discovered natively on Windows, Chrome is installed
|
|
# and launched, frames are captured, and an MP4 is produced — without Docker
|
|
# or WSL.
|
|
|
|
on:
|
|
pull_request:
|
|
# `edited` is required so the workflow re-fires when a PR's base ref is
|
|
# set back to `main` after a Graphite stack restack momentarily flips
|
|
# the base off of `main`. Without it, `pull_request` triggers are not
|
|
# re-evaluated on `base_ref_changed`, leaving required checks skipped
|
|
# for that head SHA forever.
|
|
types: [opened, synchronize, reopened, edited]
|
|
branches: [main]
|
|
push:
|
|
branches: [main]
|
|
workflow_dispatch:
|
|
inputs:
|
|
ref:
|
|
description: "Git ref to render (branch / tag / SHA)."
|
|
required: false
|
|
default: "main"
|
|
|
|
concurrency:
|
|
group: windows-render-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
changes:
|
|
name: Detect changes
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 2
|
|
outputs:
|
|
code: ${{ steps.filter.outputs.code }}
|
|
steps:
|
|
# Force git-based change detection instead of the pull_request REST API.
|
|
# The API path can fail the workflow on transient listFiles timeouts
|
|
# before the Windows render jobs even start.
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
- uses: dorny/paths-filter@v4
|
|
id: filter
|
|
with:
|
|
token: ""
|
|
filters: |
|
|
code:
|
|
- "packages/**"
|
|
- "scripts/**"
|
|
- "package.json"
|
|
- "bun.lock"
|
|
- ".github/workflows/windows-render.yml"
|
|
|
|
render-windows:
|
|
name: Render on windows-latest
|
|
needs: changes
|
|
if: needs.changes.outputs.code == 'true' || github.event_name == 'workflow_dispatch'
|
|
runs-on: windows-latest
|
|
timeout-minutes: 30
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ github.event.inputs.ref }}
|
|
lfs: true
|
|
|
|
- name: Show platform info
|
|
shell: pwsh
|
|
run: |
|
|
Write-Host "OS: $([System.Environment]::OSVersion.VersionString)"
|
|
Write-Host "PowerShell: $($PSVersionTable.PSVersion)"
|
|
Write-Host "Runner: windows-latest"
|
|
|
|
# -----------------------------------------------------------------
|
|
# Install FFmpeg via the shared composite action so the install logic
|
|
# stays identical between this job and `test-windows` below. See
|
|
# .github/actions/install-ffmpeg-windows for why we bypass Chocolatey.
|
|
# -----------------------------------------------------------------
|
|
- name: Install FFmpeg
|
|
uses: ./.github/actions/install-ffmpeg-windows
|
|
|
|
# -----------------------------------------------------------------
|
|
# Verify FFmpeg feature inventory.
|
|
#
|
|
# The engine shells out to a fixed set of encoders (libx264 for MP4,
|
|
# libx265 for HEVC, libvpx-vp9 for WebM, prores_ks for transparent
|
|
# MOV, aac for audio), muxers (mp4 / mov / webm), and demuxers
|
|
# (image2pipe for streaming RGBA frames, rawvideo for HDR PQ frames,
|
|
# mov,mp4 for video frame extraction). Some of these are GPL-only,
|
|
# so a future build swap could silently drop one and break a code
|
|
# path the canary render doesn't exercise. Fail fast here instead.
|
|
# -----------------------------------------------------------------
|
|
- name: Verify FFmpeg feature inventory
|
|
shell: pwsh
|
|
run: |
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
function Assert-FfmpegFeature {
|
|
param(
|
|
[Parameter(Mandatory)] [string] $Listing,
|
|
[Parameter(Mandatory)] [string] $Name,
|
|
[Parameter(Mandatory)] [string] $Kind
|
|
)
|
|
# `ffmpeg -encoders` etc. emit one feature per line as
|
|
# `<flags> <name> <description>`, so a whitespace boundary on
|
|
# each side is enough to disambiguate (e.g. `mov` vs `movflags`).
|
|
$pattern = "(^|\s)$([regex]::Escape($Name))(\s|$)"
|
|
if ($Listing -notmatch $pattern) {
|
|
throw "Required FFmpeg $Kind '$Name' not present in this build"
|
|
}
|
|
Write-Host " ok: $Kind $Name"
|
|
}
|
|
|
|
Write-Host "--- encoders ---"
|
|
$encoders = (& ffmpeg -hide_banner -encoders 2>&1) -join "`n"
|
|
foreach ($enc in @('libx264', 'libx265', 'libvpx-vp9', 'prores_ks', 'aac')) {
|
|
Assert-FfmpegFeature -Listing $encoders -Name $enc -Kind 'encoder'
|
|
}
|
|
|
|
Write-Host "--- muxers ---"
|
|
$muxers = (& ffmpeg -hide_banner -muxers 2>&1) -join "`n"
|
|
foreach ($mux in @('mp4', 'mov', 'webm')) {
|
|
Assert-FfmpegFeature -Listing $muxers -Name $mux -Kind 'muxer'
|
|
}
|
|
|
|
Write-Host "--- demuxers ---"
|
|
$demuxers = (& ffmpeg -hide_banner -demuxers 2>&1) -join "`n"
|
|
foreach ($dem in @('image2pipe', 'rawvideo', 'mov,mp4,m4a,3gp,3g2,mj2')) {
|
|
Assert-FfmpegFeature -Listing $demuxers -Name $dem -Kind 'demuxer'
|
|
}
|
|
|
|
- name: Install Bun
|
|
uses: oven-sh/setup-bun@v2
|
|
|
|
- name: Install Node
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: 22
|
|
|
|
- name: Install dependencies
|
|
shell: pwsh
|
|
run: bun install --frozen-lockfile
|
|
|
|
- name: Build all packages
|
|
shell: pwsh
|
|
run: bun run build
|
|
|
|
# -----------------------------------------------------------------
|
|
# Prove the PR #336 fix: hyperframes doctor exercises findFFmpeg()
|
|
# and whichBinary() — both must pass on Windows without workarounds.
|
|
# -----------------------------------------------------------------
|
|
- name: hyperframes doctor (verifies `where ffmpeg` fix)
|
|
shell: pwsh
|
|
run: node packages/cli/dist/cli.js doctor
|
|
|
|
- name: Scaffold canary composition
|
|
shell: pwsh
|
|
run: |
|
|
New-Item -ItemType Directory -Force -Path "$env:RUNNER_TEMP\windows-canary" | Out-Null
|
|
cd "$env:RUNNER_TEMP\windows-canary"
|
|
node "$env:GITHUB_WORKSPACE\packages\cli\dist\cli.js" init canary --example blank --non-interactive --skip-skills
|
|
|
|
$fixtures = "$env:GITHUB_WORKSPACE\.github\workflows\fixtures"
|
|
Copy-Item "$fixtures\windows-canary.html" "canary\index.html" -Force
|
|
|
|
- name: Render canary composition
|
|
shell: pwsh
|
|
run: |
|
|
cd "$env:RUNNER_TEMP\windows-canary\canary"
|
|
node "$env:GITHUB_WORKSPACE\packages\cli\dist\cli.js" render `
|
|
--fps 30 `
|
|
--quality draft `
|
|
--workers 2 `
|
|
--output renders\canary.mp4
|
|
|
|
- name: Verify rendered MP4
|
|
shell: pwsh
|
|
run: |
|
|
$mp4 = "$env:RUNNER_TEMP\windows-canary\canary\renders\canary.mp4"
|
|
if (-not (Test-Path $mp4)) { throw "canary.mp4 not produced" }
|
|
|
|
$probe = ffprobe -v error -select_streams v:0 `
|
|
-show_entries stream=width,height,r_frame_rate -show_entries format=duration `
|
|
-of default=noprint_wrappers=1 $mp4
|
|
Write-Host $probe
|
|
|
|
# Parse probe output
|
|
$width = ($probe | Select-String '^width=(.+)$').Matches.Groups[1].Value
|
|
$height = ($probe | Select-String '^height=(.+)$').Matches.Groups[1].Value
|
|
$fps = ($probe | Select-String '^r_frame_rate=(.+)$').Matches.Groups[1].Value
|
|
$duration = [double]($probe | Select-String '^duration=(.+)$').Matches.Groups[1].Value
|
|
|
|
if ([int]$width -ne 1920) { throw "expected 1920 width, got $width" }
|
|
if ([int]$height -ne 1080) { throw "expected 1080 height, got $height" }
|
|
if ($fps -ne "30/1") { throw "expected 30fps, got $fps" }
|
|
if ($duration -lt 7.5 -or $duration -gt 8.5) { throw "expected ~8s duration, got $duration" }
|
|
|
|
Write-Host "canary.mp4 ok: ${width}x${height} @ $fps, ${duration}s"
|
|
|
|
- name: Upload rendered MP4 artifact
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: windows-render-${{ github.run_id }}
|
|
path: ${{ runner.temp }}/windows-canary/canary/renders/canary.mp4
|
|
if-no-files-found: error
|
|
retention-days: 7
|
|
|
|
# -------------------------------------------------------------------
|
|
# Unit-test suites on Windows. Mirrors the Linux `test` job in ci.yml
|
|
# so we catch Windows-specific regressions (path separators, shell
|
|
# invocations, CRLF, file URLs, etc.) in existing vitest suites.
|
|
# The producer package is skipped because its tests require Docker /
|
|
# Linux-only tooling (Dockerfile.test, LFS golden MP4 baselines).
|
|
# -------------------------------------------------------------------
|
|
test-windows:
|
|
name: Tests on windows-latest
|
|
needs: changes
|
|
if: needs.changes.outputs.code == 'true' || github.event_name == 'workflow_dispatch'
|
|
runs-on: windows-latest
|
|
timeout-minutes: 20
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ github.event.inputs.ref }}
|
|
lfs: true
|
|
|
|
# -----------------------------------------------------------------
|
|
# Install FFmpeg so vitest suites that gate on `HAS_FFMPEG`
|
|
# (e.g. packages/engine videoFrameExtractor.test.ts) actually run on
|
|
# Windows. Without it those suites `describe.skipIf(!HAS_FFMPEG)`
|
|
# themselves silently and any Windows-specific regression in the
|
|
# FFmpeg-driven code paths would not be caught here.
|
|
# -----------------------------------------------------------------
|
|
- name: Install FFmpeg
|
|
uses: ./.github/actions/install-ffmpeg-windows
|
|
|
|
- name: Install Bun
|
|
uses: oven-sh/setup-bun@v2
|
|
|
|
- name: Install Node
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: 22
|
|
|
|
- name: Install dependencies
|
|
shell: pwsh
|
|
run: bun install --frozen-lockfile
|
|
|
|
- name: Build
|
|
shell: pwsh
|
|
run: bun run build
|
|
|
|
- name: Run tests (all packages except producer)
|
|
shell: pwsh
|
|
run: bun run --filter "!@hyperframes/producer" test
|
|
|
|
- name: Run runtime contract test
|
|
shell: pwsh
|
|
run: bun run --filter "@hyperframes/core" test:hyperframe-runtime-ci
|