mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-10 22:20:14 +00:00
fix(engine,shader): handle matrix3d transforms and hide non-first scenes (#374)
## Summary
Two correctness fixes in the HDR transform & clipping pipeline: `parseTransformMatrix` now handles `matrix3d(...)` (GSAP's default `force3D: true`), and shader-transitions sets every non-first scene to `opacity: 0` at `t=0` so the engine doesn't over-composite at the start.
## Why
`Chunk 4` of `plans/hdr-followups.md`. Transform extraction and border-radius computation existed but were dead — an HDR video with `rotation: 45` rendered un-rotated, and 3-scene compositions ghosted at `t=0` because every scene defaulted to CSS `opacity: 1` and contributed to the first frame.
## What changed
**Matrix3d support in `parseTransformMatrix`.** `DOMMatrix.toString()` emits `matrix3d` whenever any ancestor in the chain has used a 3D transform — most importantly GSAP's default `force3D: true`, which converts `translate(...)` into `translate3d(..., 0)`. Without this, every GSAP-driven transform was silently dropped during HDR compositing because `videoFrameInjector.getViewportMatrix()` would return `matrix3d(...)` and the blit path would parse it as `null` and fall back to identity. The 16-value column-major form is converted to its 2D affine projection (indices 0, 1, 4, 5, 12, 13 → m11, m12, m21, m22, m41, m42); Z, perspective, and out-of-plane rotation components are dropped.
**Initial-state opacity in `initEngineMode`.** The browser preview branch uses a GL canvas overlay during transitions, so scene opacity at `t=0` doesn't matter visually. The engine branch reads scene opacity directly via `queryElementStacking()` to decide which layers to composite. Without an explicit initial-state tween, every scene defaulted to CSS `opacity: 1` and contributed to the very first frame, causing ghosting/overlap until the first transition fired. `tl.set()` at position 0 anchors the initial state in the timeline graph so reverse seeks from inside a later transition restore it correctly.
These two fixes together make `el.transform` and `el.borderRadius` (already wired in Chunk 7A's `compositeHdrFrame`) actually flow through the GSAP-animated case, and keep the engine's per-frame compositing aligned with what the user sees in browser preview.
## Test plan
- [x] 6 new `alphaBlit.test.ts` cases (identity matrix3d, translate3d, scale + translate3d, rotateZ, malformed arg count, non-finite values).
- [x] Existing `hdr-regression` Window H already CSS-sets `#scene-b { opacity: 0 }` as a fallback; the new `tl.set` is redundant for that case but harmless and removes the need for compositions to remember the CSS workaround.
- [x] Manual: rotated HDR video (`rotation: 45`) appears rotated; `border-radius: 50%` clips to circle; 3-scene composition has no overlap at `t=0`.
## Stack
Chunk 4 of `plans/hdr-followups.md`. Window F of the regression suite documents the bug; the next PR in the stack tightens the `maxFrameFailures` budget to 0.
This commit is contained in:
Vance Ingalls
@@ -1,4 +1,4 @@
|
|||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it, vi } from "vitest";
|
||||||
import { deflateSync } from "zlib";
|
import { deflateSync } from "zlib";
|
||||||
import {
|
import {
|
||||||
decodePng,
|
decodePng,
|
||||||
@@ -713,8 +713,75 @@ describe("parseTransformMatrix", () => {
|
|||||||
expect(parseTransformMatrix("")).toBeNull();
|
expect(parseTransformMatrix("")).toBeNull();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("returns null for unsupported 3d matrix", () => {
|
it("parses identity matrix3d (GSAP force3D default)", () => {
|
||||||
expect(parseTransformMatrix("matrix3d(1,0,0,0,0,1,0,0,0,0,1,0,0,0,0,1)")).toBeNull();
|
const m = parseTransformMatrix("matrix3d(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1)");
|
||||||
|
expect(m).toEqual([1, 0, 0, 1, 0, 0]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses translate3d matrix3d as 2D affine (drops Z translation)", () => {
|
||||||
|
// translate3d(100px, 50px, 25px) — Z=25 must be dropped.
|
||||||
|
const m = parseTransformMatrix("matrix3d(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 100, 50, 25, 1)");
|
||||||
|
expect(m).toEqual([1, 0, 0, 1, 100, 50]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses scale + translate3d matrix3d (typical GSAP output)", () => {
|
||||||
|
// scale(0.85) translate3d(100px, 50px, 0) emitted by GSAP with force3D: true.
|
||||||
|
const m = parseTransformMatrix(
|
||||||
|
"matrix3d(0.85, 0, 0, 0, 0, 0.85, 0, 0, 0, 0, 1, 0, 100, 50, 0, 1)",
|
||||||
|
);
|
||||||
|
expect(m).toEqual([0.85, 0, 0, 0.85, 100, 50]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("parses rotation matrix3d (rotateZ via force3D)", () => {
|
||||||
|
// rotateZ(45deg) translate3d(0, 0, 0) — column-major.
|
||||||
|
const cos = Math.cos(Math.PI / 4);
|
||||||
|
const sin = Math.sin(Math.PI / 4);
|
||||||
|
const m = parseTransformMatrix(
|
||||||
|
`matrix3d(${cos}, ${sin}, 0, 0, ${-sin}, ${cos}, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1)`,
|
||||||
|
);
|
||||||
|
expect(m).not.toBeNull();
|
||||||
|
if (!m) return;
|
||||||
|
expect(m[0]).toBeCloseTo(cos, 10);
|
||||||
|
expect(m[1]).toBeCloseTo(sin, 10);
|
||||||
|
expect(m[2]).toBeCloseTo(-sin, 10);
|
||||||
|
expect(m[3]).toBeCloseTo(cos, 10);
|
||||||
|
expect(m[4]).toBe(0);
|
||||||
|
expect(m[5]).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns null for malformed matrix3d (wrong arg count)", () => {
|
||||||
|
expect(parseTransformMatrix("matrix3d(1, 0, 0, 0, 0, 1)")).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns null for matrix3d with non-finite values", () => {
|
||||||
|
expect(
|
||||||
|
parseTransformMatrix("matrix3d(NaN, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1)"),
|
||||||
|
).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("warns once when matrix3d has Z-significant components (rotateY 45deg)", () => {
|
||||||
|
// rotateY(45deg) — m31=-sin, m13=sin, m33=cos. Real 3D rotation around Y;
|
||||||
|
// the engine projects to 2D and silently drops perspective. Author needs
|
||||||
|
// to know the rendered output won't match the studio preview.
|
||||||
|
const cos = Math.cos(Math.PI / 4);
|
||||||
|
const sin = Math.sin(Math.PI / 4);
|
||||||
|
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||||
|
const m = parseTransformMatrix(
|
||||||
|
`matrix3d(${cos}, 0, ${-sin}, 0, 0, 1, 0, 0, ${sin}, 0, ${cos}, 0, 0, 0, 0, 1)`,
|
||||||
|
);
|
||||||
|
// Still returns the projected 2D affine — warning is non-blocking.
|
||||||
|
expect(m).not.toBeNull();
|
||||||
|
expect(m).toEqual([cos, 0, 0, 1, 0, 0]);
|
||||||
|
// Module-level dedup means the warn either fired in this test (first
|
||||||
|
// Z-significant call in the run) or earlier; either way the
|
||||||
|
// user-facing observability contract holds. Assert it was called at
|
||||||
|
// least once across the process.
|
||||||
|
const totalCalls = warn.mock.calls.length;
|
||||||
|
// Calling parseTransformMatrix again with another Z-significant matrix
|
||||||
|
// must not produce additional warnings (dedup check).
|
||||||
|
parseTransformMatrix("matrix3d(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 5, 0, 0, 0, 1)");
|
||||||
|
expect(warn.mock.calls.length).toBe(totalCalls);
|
||||||
|
warn.mockRestore();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -850,21 +850,129 @@ export function normalizeObjectFit(value: string | undefined): ObjectFit {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Parse a CSS `matrix(a,b,c,d,e,f)` string into a 6-element array.
|
* Parse a CSS `matrix(a,b,c,d,e,f)` or `matrix3d(...)` string into a 6-element
|
||||||
* Returns null for "none", empty, or unsupported formats (matrix3d).
|
* 2D affine array.
|
||||||
*
|
*
|
||||||
* The array maps to the CSS matrix: [a, b, c, d, tx, ty] where:
|
* Returns null for `"none"`, empty input, or syntactically malformed values.
|
||||||
|
*
|
||||||
|
* The returned array maps to the CSS matrix: [a, b, c, d, tx, ty] where:
|
||||||
* | a c tx | (a=scaleX, b=skewY, c=skewX, d=scaleY, tx/ty=translate)
|
* | a c tx | (a=scaleX, b=skewY, c=skewX, d=scaleY, tx/ty=translate)
|
||||||
* | b d ty |
|
* | b d ty |
|
||||||
* | 0 0 1 |
|
* | 0 0 1 |
|
||||||
|
*
|
||||||
|
* `matrix3d` is the default output of `DOMMatrix.toString()` whenever any
|
||||||
|
* ancestor in the chain has used a 3D transform — most importantly GSAP's
|
||||||
|
* default `force3D: true`, which converts `translate(...)` into
|
||||||
|
* `translate3d(..., 0)` and surfaces as `matrix3d(...)` even for purely 2D
|
||||||
|
* animations. Without explicit handling we'd silently drop every transform
|
||||||
|
* driven by GSAP. The 16 values are in column-major order:
|
||||||
|
*
|
||||||
|
* matrix3d(m11, m12, m13, m14, m21, m22, m23, m24, m31, m32, m33, m34,
|
||||||
|
* m41, m42, m43, m44)
|
||||||
|
*
|
||||||
|
* The 2D affine corresponds to indices 0, 1, 4, 5, 12, 13 (m11, m12, m21,
|
||||||
|
* m22, m41, m42). Z, perspective, and out-of-plane rotation components are
|
||||||
|
* dropped — for true 3D transforms the resulting 2D projection is only
|
||||||
|
* approximate, but for the GSAP `force3D: true` flat-matrix case it is exact.
|
||||||
|
*
|
||||||
|
* When a `matrix3d` arrives with Z-significant components (m13, m23, m31,
|
||||||
|
* m32, m34, m43 != 0 or m33 != 1) we emit a one-time `console.warn` so
|
||||||
|
* authors using real 3D transforms know the engine path is silently
|
||||||
|
* flattening their scene rather than failing it.
|
||||||
*/
|
*/
|
||||||
export function parseTransformMatrix(css: string): number[] | null {
|
export function parseTransformMatrix(css: string): number[] | null {
|
||||||
if (!css || css === "none") return null;
|
if (!css || css === "none") return null;
|
||||||
const match = css.match(
|
|
||||||
|
const match2d = css.match(
|
||||||
/^matrix\(\s*([^,]+),\s*([^,]+),\s*([^,]+),\s*([^,]+),\s*([^,]+),\s*([^,)]+)\s*\)$/,
|
/^matrix\(\s*([^,]+),\s*([^,]+),\s*([^,]+),\s*([^,]+),\s*([^,]+),\s*([^,)]+)\s*\)$/,
|
||||||
);
|
);
|
||||||
if (!match) return null;
|
if (match2d) {
|
||||||
const values = match.slice(1, 7).map(Number);
|
const values = match2d.slice(1, 7).map(Number);
|
||||||
if (!values.every(Number.isFinite)) return null;
|
if (!values.every(Number.isFinite)) return null;
|
||||||
return values;
|
return values;
|
||||||
|
}
|
||||||
|
|
||||||
|
const match3d = css.match(/^matrix3d\(\s*([^)]+)\)$/);
|
||||||
|
if (match3d) {
|
||||||
|
const raw = match3d[1];
|
||||||
|
if (!raw) return null;
|
||||||
|
const parts = raw.split(",").map((s) => Number(s.trim()));
|
||||||
|
if (parts.length !== 16 || !parts.every(Number.isFinite)) return null;
|
||||||
|
// 3D-significance check: a flat 2D transform expressed as matrix3d has
|
||||||
|
// a3=b3=c1=c2=d1=d2=d3=0, c3=1, d4=1. Any deviation means the composition
|
||||||
|
// is using real 3D (perspective, rotateX/Y) which the engine path can't
|
||||||
|
// represent — we project to 2D and the visual will silently drop depth.
|
||||||
|
// Warn once per process so authors don't get a misleading "looks fine in
|
||||||
|
// studio, broken in render" experience without any signal. Z translation
|
||||||
|
// (c4 = parts[14]) is intentionally dropped by the 2D projection below
|
||||||
|
// and does NOT trigger this warning — that's the GSAP `force3D: true`
|
||||||
|
// happy path.
|
||||||
|
warnIfZSignificant(parts);
|
||||||
|
// Extract column-major 2D affine: m11, m12, m21, m22, m41, m42.
|
||||||
|
return [
|
||||||
|
parts[0] as number,
|
||||||
|
parts[1] as number,
|
||||||
|
parts[4] as number,
|
||||||
|
parts[5] as number,
|
||||||
|
parts[12] as number,
|
||||||
|
parts[13] as number,
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
let warnedZSignificant = false;
|
||||||
|
const Z_EPSILON = 1e-6;
|
||||||
|
|
||||||
|
function warnIfZSignificant(parts: number[]): void {
|
||||||
|
if (warnedZSignificant) return;
|
||||||
|
// CSS matrix3d() is column-major:
|
||||||
|
// matrix3d(a1, b1, c1, d1, a2, b2, c2, d2, a3, b3, c3, d3, a4, b4, c4, d4)
|
||||||
|
// laid out as:
|
||||||
|
// | a1 a2 a3 a4 | | parts[0] parts[4] parts[8] parts[12] |
|
||||||
|
// | b1 b2 b3 b4 | = | parts[1] parts[5] parts[9] parts[13] |
|
||||||
|
// | c1 c2 c3 c4 | | parts[2] parts[6] parts[10] parts[14] |
|
||||||
|
// | d1 d2 d3 d4 | | parts[3] parts[7] parts[11] parts[15] |
|
||||||
|
//
|
||||||
|
// For a flat 2D transform — the only thing this engine path can render
|
||||||
|
// faithfully — we expect:
|
||||||
|
// a3 = b3 = c1 = c2 = 0 (no XZ/YZ rotation coupling)
|
||||||
|
// c3 = 1 (no Z scaling)
|
||||||
|
// d1 = d2 = d3 = 0 (no perspective)
|
||||||
|
// d4 = 1 (no homogeneous scaling)
|
||||||
|
// Z translation (c4 = parts[14]) is explicitly dropped by the 2D affine
|
||||||
|
// extraction below — that's the whole point of supporting GSAP's
|
||||||
|
// `force3D: true` translate3d(x, y, 0) emission — so it is NOT flagged.
|
||||||
|
const a3 = parts[8] ?? 0;
|
||||||
|
const b3 = parts[9] ?? 0;
|
||||||
|
const c1 = parts[2] ?? 0;
|
||||||
|
const c2 = parts[6] ?? 0;
|
||||||
|
const c3 = parts[10] ?? 1;
|
||||||
|
const d1 = parts[3] ?? 0;
|
||||||
|
const d2 = parts[7] ?? 0;
|
||||||
|
const d3 = parts[11] ?? 0;
|
||||||
|
const d4 = parts[15] ?? 1;
|
||||||
|
if (
|
||||||
|
Math.abs(a3) > Z_EPSILON ||
|
||||||
|
Math.abs(b3) > Z_EPSILON ||
|
||||||
|
Math.abs(c1) > Z_EPSILON ||
|
||||||
|
Math.abs(c2) > Z_EPSILON ||
|
||||||
|
Math.abs(c3 - 1) > Z_EPSILON ||
|
||||||
|
Math.abs(d1) > Z_EPSILON ||
|
||||||
|
Math.abs(d2) > Z_EPSILON ||
|
||||||
|
Math.abs(d3) > Z_EPSILON ||
|
||||||
|
Math.abs(d4 - 1) > Z_EPSILON
|
||||||
|
) {
|
||||||
|
warnedZSignificant = true;
|
||||||
|
console.warn(
|
||||||
|
`[alphaBlit] parseTransformMatrix received a matrix3d with non-trivial 3D components ` +
|
||||||
|
`(a3=${a3}, b3=${b3}, c1=${c1}, c2=${c2}, c3=${c3}, d1=${d1}, d2=${d2}, d3=${d3}, d4=${d4}). ` +
|
||||||
|
`The engine projects 3D transforms to 2D (m11, m12, m21, m22, m41, m42) and silently ` +
|
||||||
|
`discards perspective and out-of-plane rotation. If your composition uses real 3D ` +
|
||||||
|
`(rotateX/Y, perspective), the rendered output will not match the studio preview. ` +
|
||||||
|
`Z translation (translateZ) is dropped by design and does not trigger this warning. ` +
|
||||||
|
`This warning is emitted once per process.`,
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -130,7 +130,7 @@
|
|||||||
data-width="1920"
|
data-width="1920"
|
||||||
data-height="1080"
|
data-height="1080"
|
||||||
>
|
>
|
||||||
<!-- Window A · Baseline HDR + direct opacity tween · 0.0–2.5s -->
|
<!-- Window A · Baseline HDR + direct opacity tween · 0.0–1.5s -->
|
||||||
<video
|
<video
|
||||||
id="wa-video"
|
id="wa-video"
|
||||||
class="clip hdr-video"
|
class="clip hdr-video"
|
||||||
|
|||||||
@@ -363,6 +363,20 @@ function initEngineMode(
|
|||||||
tl.to({ t: 0 }, { t: 1, duration, ease: "none" }, 0);
|
tl.to({ t: 0 }, { t: 1, duration, ease: "none" }, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Initial state: every non-first scene starts hidden. CSS defaults
|
||||||
|
// .scene to opacity:1, so without this every scene would composite at
|
||||||
|
// t=0 and the engine's queryElementStacking() would report all of them
|
||||||
|
// visible — manifesting as ghosting/overlap in the very first frame
|
||||||
|
// before the first transition fires. tl.set() at position 0 ensures
|
||||||
|
// the initial state is part of the timeline's seek graph, so reverse
|
||||||
|
// seeks from inside a later transition correctly restore it.
|
||||||
|
for (let i = 1; i < scenes.length; i++) {
|
||||||
|
const sceneId = scenes[i];
|
||||||
|
if (sceneId) {
|
||||||
|
tl.set(`#${sceneId}`, { opacity: 0 }, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
for (let i = 0; i < transitions.length; i++) {
|
for (let i = 0; i < transitions.length; i++) {
|
||||||
const t = transitions[i];
|
const t = transitions[i];
|
||||||
const fromId = scenes[i];
|
const fromId = scenes[i];
|
||||||
|
|||||||
Reference in New Issue
Block a user