feat(engine): adaptive per-frame sample count for motion blur (#4031)

* feat(engine): adaptive per-frame sample count for motion blur

Measured against the accuracy bar set in #4010 (72.4dB mean PSNR vs the
960fps shutter integral), the fixed default of 16 samples misses it for
hard-edge content while paying nothing extra for static/slow content is
still important. This picks a sample count per frame from measured
motion, floor 16 ceiling 64 (issue #4029).

Two probe captures at the true shutter-window edges (never a sample that
enters the average) measure motion magnitude, mapped through a monotone
step table to a sample count. Frames confirmed to move only non-spatial
properties (opacity, colour) skip the probe and use the floor directly,
since a fade's shutter integral is a smooth blend that does not need many
samples. MotionBlurPlan's window (from shutter angle/phase alone) is now
independent of how many samples fill it, which is what lets the count
vary per frame without moving the window; motionBlurWindowIsStatic is
bounded by the widest sample inset any chosen K could produce so the
static-dedup safety guarantee holds for every K in [16, 64].

* refactor(engine): simplify the adaptive motion-blur sample selection

Two findings from an independent code-reuse/quality/efficiency review of
the previous commit:

- armStaticDedup ran the GSAP-timeline walk (computeStaticFrameSet)
  twice on the same page when adaptive motion blur and static dedup
  both apply, once for each purpose. Shares the one result instead.
- The non-spatial floor path read the sample count from
  adaptiveSampleCount(0), which only worked because the step table
  happens to be sorted with the first bucket covering zero. Reads
  the now-exported DEFAULT_SAMPLES_PER_FRAME directly instead.

* fix(engine): read raw options for the adaptive motion-blur gate

Independent review caught it: every armStaticDedup call site runs
before finalizeSessionInit resolves session.motionBlur, so the
adaptive-classification gate's `session.motionBlur &&
session.motionBlur.fixedSamplesPerFrame === null` was always false at
runtime. The non-spatial probe-skip described in the PR never engaged
outside tests that hand-set motionBlurNonSpatialFrames directly,
bypassing real init order.

Reads session.options.motionBlur (set unconditionally at session
creation) instead, which carries the same "adaptive requested"
information the resolved plan would have. Adds a source-order lock
test plus direct coverage of computeStaticFrameSet's
nonSpatialOnlyFrameSet, mutation-proved against the exact regression.

* fix(engine): add xPercent/yPercent to the spatial-property list

Round 2 of the independent review: the spatial-property list deciding
whether a frame's motion needs the adaptive probe omitted GSAP's
percentage-based position properties (xPercent/yPercent) and two 3D
transform properties (perspective/transformPerspective) that two other
lists already in this repo (packages/parsers/src/gsapConstants.ts,
packages/studio/.../gsapAnimatesProperty.ts) both carry. A composition
positioned via xPercent/yPercent would have been silently floored to
K=16 with no probe and no log line, the same failure shape as the
round-1 bug through a data gap instead of a dead gate.

Moves the list to an exported SPATIAL_TWEEN_PROPERTIES array in
motionBlur.ts, passed into computeStaticFrameSet's page.evaluate as
data rather than duplicated inline, so it stays a single list and is
directly testable. Adds tests that execute the real page-side closure
(the existing tests all mocked the whole result and never exercised
the matching code), mutation-proved against the exact gap found.

* fix(engine): add rotate to the spatial-property list

Round 3 of the independent review, comparing SPATIAL_TWEEN_PROPERTIES
side by side against the two sibling lists rather than trusting the
prior commit's summary: GSAP's individual "rotate" transform property
(distinct from "rotation", carried in
packages/studio/.../gsapAnimatesProperty.ts's GSAP_TRANSFORM_PROPS)
was still missing, same failure shape as the two prior rounds.

The same pass also raised transformOrigin, which is deliberately NOT
added: gsapConstants.ts's own classifyTweenPropertyGroup excludes it
as a pivot-point modifier with no independent visual effect, and a
test now locks that reasoning in rather than leaving it to be
"fixed" again by a future pass that does not read that far.
This commit is contained in:
Miguel Ángel
2026-09-17 01:09:12 -04:00
committed by GitHub
parent 484a06fb65
commit 2ab7b86868
4 changed files with 611 additions and 83 deletions
@@ -2,6 +2,7 @@ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import {
captureFrame,
captureFrameToBuffer,
computeStaticFrameSet,
resolveSessionMotionBlur,
type CaptureSession,
} from "./frameCapture.js";
@@ -73,7 +74,10 @@ function makeSession(overrides: Partial<CaptureSession> = {}): CaptureSession {
totalMs: 0,
frameMs: [],
},
motionBlur: resolveMotionBlurPlan({}) ?? undefined,
// Fixed K=16 by default so the tests in this file that predate adaptive sampling
// (accumulation, event suppression, dedup composition) keep testing exactly what
// they did before — adaptive selection gets its own describe block below.
motionBlur: resolveMotionBlurPlan({ samplesPerFrame: 16 }) ?? undefined,
...overrides,
} as unknown as CaptureSession;
}
@@ -174,21 +178,30 @@ describe("sub-frame accumulation reaches the page with distinct sample times", (
describe("accumulation composes with static-frame dedup", () => {
// 720 degrees is the widest shutter After Effects offers, and the AD5 reference export
// uses it, so it is the case a reader is most likely to check the frame ranges against.
const wideShutter = resolveMotionBlurPlan({ shutterAngle: 720, shutterPhase: -360 }) ?? undefined;
// Fixed K=16, same reason as makeSession's default above.
const wideShutter =
resolveMotionBlurPlan({ shutterAngle: 720, shutterPhase: -360, samplesPerFrame: 16 }) ??
undefined;
it("pays nothing on a static frame instead of capturing K samples of the same instant", async () => {
/** Reuse claims frame 11's blurred output equals frame 10's, so the whole window
* either side of it must be static for that claim to be true. */
async function expectFrame11Reused(overrides: Partial<CaptureSession>) {
const anchor = solidPng(42);
const session = makeSession({
// The whole window either side of frame 11 is static, which is what reuse claims.
staticFrames: new Set([9, 10, 11, 12]),
lastFrameBuffer: anchor,
lastFrameAbsoluteIndex: 10,
...overrides,
});
const result = await captureFrameToBuffer(session, 11, 11 / 30);
expect(result.buffer).toBe(anchor);
expect(vi.mocked(pageScreenshotCapture)).not.toHaveBeenCalled();
return session;
}
it("pays nothing on a static frame instead of capturing K samples of the same instant", async () => {
const session = await expectFrame11Reused({});
expect(seeks).toHaveLength(0);
expect(session.staticDedupCount).toBe(1);
});
@@ -225,18 +238,7 @@ describe("accumulation composes with static-frame dedup", () => {
});
it("still reuses at 720 degrees when the whole window is static", async () => {
const anchor = solidPng(42);
const session = makeSession({
motionBlur: wideShutter,
staticFrames: new Set([9, 10, 11, 12]),
lastFrameBuffer: anchor,
lastFrameAbsoluteIndex: 10,
});
const result = await captureFrameToBuffer(session, 11, 11 / 30);
expect(result.buffer).toBe(anchor);
expect(vi.mocked(pageScreenshotCapture)).not.toHaveBeenCalled();
await expectFrame11Reused({ motionBlur: wideShutter });
});
});
@@ -279,7 +281,10 @@ describe("resolveSessionMotionBlur rejects what accumulation cannot render", ()
});
it("resolves a plan for a screenshot-mode PNG session", () => {
expect(resolveSessionMotionBlur(withOptions({}))?.sampleTickOffsets).toHaveLength(16);
const plan = resolveSessionMotionBlur(withOptions({}));
expect(plan).not.toBeUndefined();
// `options.motionBlur: {}` sets no explicit samplesPerFrame, so the plan is adaptive.
expect(plan?.fixedSamplesPerFrame).toBeNull();
});
it("rejects a capture mode whose sampling is not implemented yet", () => {
@@ -300,3 +305,222 @@ describe("resolveSessionMotionBlur rejects what accumulation cannot render", ()
).toThrow(/video/);
});
});
describe("adaptive sample count (issue #4029)", () => {
// No explicit samplesPerFrame → adaptive.
const adaptivePlan = resolveMotionBlurPlan({}) ?? undefined;
it("probes the true window edges, escalates K on a large diff, and never lets a probe into the average", async () => {
// First two screenshots are the probes (0, 255 — a large diff); everything after is
// a constant 100, so if a probe leaked into the accumulator the average would move.
let call = 0;
vi.mocked(pageScreenshotCapture).mockImplementation(async () =>
solidPng(call++ < 2 ? [0, 255][call - 1] : 100),
);
const result = await captureFrameToBuffer(
makeSession({ motionBlur: adaptivePlan }),
10,
10 / 30,
);
// A 0/255 probe diff is well past the top step, so K=64.
expect(vi.mocked(pageScreenshotCapture)).toHaveBeenCalledTimes(2 + 64);
expect([...decodePng(result.buffer).data.slice(0, 4)]).toEqual([100, 100, 100, 255]);
// seeks[0] is the one eventful seek; the two probe seeks come right after it, both
// suppressed, and distinct from each other (the true window edges, not one point).
const [probeA, probeB] = seeks.slice(1, 3);
expect(probeA?.suppressEvents).toBe(true);
expect(probeB?.suppressEvents).toBe(true);
expect(probeA?.time).not.toBe(probeB?.time);
});
it("skips the probe and uses the floor for a frame confirmed non-spatial", async () => {
let calls = 0;
vi.mocked(pageScreenshotCapture).mockImplementation(async () => {
calls++;
return solidPng(50);
});
await captureFrameToBuffer(
makeSession({ motionBlur: adaptivePlan, motionBlurNonSpatialFrames: new Set([10]) }),
10,
10 / 30,
);
expect(calls).toBe(16); // floor only, no probe pair
});
it("probes (rather than assuming the floor) for a frame outside the confirmed non-spatial set", async () => {
let calls = 0;
vi.mocked(pageScreenshotCapture).mockImplementation(async () => {
calls++;
return solidPng(50); // identical probes -> zero diff -> floor, but via the probe path
});
await captureFrameToBuffer(
makeSession({ motionBlur: adaptivePlan, motionBlurNonSpatialFrames: new Set([999]) }),
10,
10 / 30,
);
expect(calls).toBe(2 + 16); // probed, diff was zero, landed on the floor anyway
});
});
describe("computeStaticFrameSet classifies spatial vs non-spatial intervals", () => {
function makePage(intervals: Array<{ start: number; end: number; spatial: boolean }>) {
return {
evaluate: vi
.fn()
.mockResolvedValueOnce({
intervals,
tweenCount: intervals.length,
duration: 10,
hasVideo: false,
hasCanvas: false,
hasNonGsapAnim: false,
hasUnresolvableClipStart: false,
hasTimelineCall: false,
})
.mockResolvedValueOnce([]), // computeClipBoundaryFrames' own [data-start] scan
} as unknown as Parameters<typeof computeStaticFrameSet>[0];
}
it("marks a frame touched only by a non-spatial interval (e.g. opacity) as safe to skip the probe", async () => {
const page = makePage([{ start: 0, end: 1, spatial: false }]);
const result = await computeStaticFrameSet(page, 30);
expect(result.eligible).toBe(true);
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(true);
});
it("does not mark a frame touched by any spatial interval (e.g. x), even alongside a non-spatial one", async () => {
const page = makePage([
{ start: 0, end: 1, spatial: false },
{ start: 0, end: 1, spatial: true },
]);
const result = await computeStaticFrameSet(page, 30);
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(false);
});
it("reports an empty set when the analysis is ineligible, never a false 'confirmed non-spatial'", async () => {
const page = {
evaluate: vi
.fn()
.mockResolvedValueOnce({
intervals: [{ start: 0, end: 1, spatial: false }],
tweenCount: 0, // no GSAP tweens -> ineligible, per the existing "reasons" gate
duration: 10,
hasVideo: false,
hasCanvas: false,
hasNonGsapAnim: false,
hasUnresolvableClipStart: false,
hasTimelineCall: false,
})
.mockResolvedValueOnce([]), // computeClipBoundaryFrames' own [data-start] scan
} as unknown as Parameters<typeof computeStaticFrameSet>[0];
const result = await computeStaticFrameSet(page, 30);
expect(result.eligible).toBe(false);
expect(result.nonSpatialOnlyFrameSet.size).toBe(0);
});
});
describe("armStaticDedup reads the raw caller options for the adaptive gate, not the resolved plan", () => {
// armStaticDedup always runs before finalizeSessionInit sets session.motionBlur, so a
// gate reading that field is always false; it must read session.options.motionBlur.
const source = readFileSync(new URL("./frameCapture.ts", import.meta.url), "utf8");
it("gates the adaptive classification block on session.options.motionBlur", () => {
// Anchored on the `let` declaration, AFTER the explanatory comment above it, so this
// reads only the functional `if (...)` condition — not prose that could name either
// symbol without the code actually using it.
const start = source.indexOf("let sharedStaticFrameStats");
if (start < 0) throw new Error("sharedStaticFrameStats declaration not found");
const conditionEnd = source.indexOf(") {", start);
const condition = source.slice(start, conditionEnd);
expect(condition).toContain("session.options.motionBlur");
// Bare "session.motionBlur" (the resolved plan) not followed by a letter, so this
// does not false-positive on the unrelated "session.motionBlurNonSpatialFrames".
expect(condition).not.toMatch(/session\.motionBlur(?![A-Za-z])/);
});
it("every armStaticDedup call site runs before finalizeSessionInit in the same init path", () => {
// Confirms the premise: if a future refactor made these coincide, the raw-options
// gate above would still be correct (session.options.motionBlur is set unconditionally
// at session creation), but this pins today's actual ordering as documented context.
const finalizeCalls = [...source.matchAll(/finalizeSessionInit\(session\)/g)].map(
(m) => m.index as number,
);
const armCalls = [...source.matchAll(/armStaticDedup\(session,/g)].map(
(m) => m.index as number,
);
expect(finalizeCalls.length).toBeGreaterThan(0);
for (const finalizeAt of finalizeCalls) {
const precedingArm = armCalls.filter((a) => a < finalizeAt).at(-1);
expect(precedingArm).toBeDefined();
}
});
});
describe("computeStaticFrameSet's real page-side property matching (not a canned mock)", () => {
// Runs the ACTUAL closure passed to page.evaluate, against a stubbed window/document —
// the other computeStaticFrameSet tests above mock the whole result and never exercise
// isSpatial()/SPATIAL_PROPS at all, which is how xPercent/yPercent went unclassified.
function makeTimeline(vars: Record<string, unknown>) {
const child = { startTime: () => 0, duration: () => 1, totalDuration: () => 1, vars };
return { getChildren: () => [child], duration: () => 1 };
}
function makePage(timelines: Record<string, unknown>) {
const root = globalThis as Record<string, unknown>;
root.window = { __timelines: timelines, __hf: { duration: 1 } };
root.document = {
querySelector: () => null,
querySelectorAll: () => [],
getAnimations: undefined,
};
return {
evaluate: async (fn: unknown, ...args: unknown[]) =>
typeof fn === "function" ? (fn as (...a: unknown[]) => unknown)(...args) : undefined,
} as unknown as Parameters<typeof computeStaticFrameSet>[0];
}
afterEach(() => {
const root = globalThis as Record<string, unknown>;
delete root.window;
delete root.document;
});
it.each(["xPercent", "yPercent", "perspective", "transformPerspective", "rotate"])(
"classifies a tween on %s as spatial",
async (prop) => {
const page = makePage({ main: makeTimeline({ [prop]: 50 }) });
const result = await computeStaticFrameSet(page, 30);
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(false);
},
);
it("does not classify transformOrigin alone as spatial (a pivot-point modifier, not independent motion)", async () => {
const page = makePage({ main: makeTimeline({ transformOrigin: "center" }) });
const result = await computeStaticFrameSet(page, 30);
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(true);
});
it("classifies a tween on opacity as non-spatial", async () => {
const page = makePage({ main: makeTimeline({ opacity: 0.5 }) });
const result = await computeStaticFrameSet(page, 30);
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(true);
});
});
+122 -15
View File
@@ -22,9 +22,14 @@ import {
import { DrawElementCaptureError } from "./drawElementCaptureError.js";
import { encodePng } from "../utils/alphaBlit.js";
import {
DEFAULT_SAMPLES_PER_FRAME,
MotionBlurAccumulator,
SPATIAL_TWEEN_PROPERTIES,
adaptiveSampleCount,
motionBlurProbeTimes,
motionBlurSampleTimes,
motionBlurWindowIsStatic,
probeDiffMagnitude,
resolveMotionBlurPlan,
type MotionBlurPlan,
} from "./motionBlur.js";
@@ -114,6 +119,9 @@ export interface CaptureSession {
* silently rendering unblurred.
*/
motionBlur?: MotionBlurPlan;
/** Frames confirmed to move only non-spatial properties (opacity, colour); set for
* adaptive motion blur. Absent from this set means "probe" — see computeStaticFrameSet. */
motionBlurNonSpatialFrames?: Set<number>;
// ── Static-dedup observability (set by armStaticDedup; surfaced via
// getCapturePerfSummary → RenderPerfSummary → the render_complete event) ──
// `armed` derives from the verified staticFrames set. Predicted count is stored
@@ -2742,6 +2750,9 @@ export async function computeStaticFrameSet(
): Promise<{
totalFrames: number;
staticFrameSet: Set<number>;
/** Frames whose only active motion is non-spatial (opacity, colour); safe to skip
* the adaptive probe. Empty (not meaningful) when `!eligible`. */
nonSpatialOnlyFrameSet: Set<number>;
hasVideo: boolean;
hasCanvas: boolean;
hasNonGsapAnim: boolean;
@@ -2749,7 +2760,7 @@ export async function computeStaticFrameSet(
eligible: boolean;
reason: string;
}> {
const result = await page.evaluate(() => {
const result = await page.evaluate((spatialProps: readonly string[]) => {
type AnyTween = {
startTime(): number;
duration(): number;
@@ -2757,7 +2768,16 @@ export async function computeStaticFrameSet(
getChildren?(nested: boolean, tweens: boolean, timelines: boolean): AnyTween[];
vars?: Record<string, unknown>;
};
const intervals: Array<{ start: number; end: number }> = [];
// Passed in from SPATIAL_TWEEN_PROPERTIES rather than declared here, so the list
// stays the one this module's own tests exercise (the matching code itself must
// stay inline — this closure is serialized and runs in the page realm).
const SPATIAL_PROPS = new Set(spatialProps);
function isSpatial(vars: Record<string, unknown> | undefined): boolean {
if (!vars) return false;
for (const key of Object.keys(vars)) if (SPATIAL_PROPS.has(key)) return true;
return false;
}
const intervals: Array<{ start: number; end: number; spatial: boolean }> = [];
let tweenCount = 0;
// totalDuration() (NOT duration()): a repeat/yoyo tween animates past one iteration;
// a repeating timeline is marked opaque over its whole span (conservative).
@@ -2781,7 +2801,8 @@ export async function computeStaticFrameSet(
// Mark its entire span as animated so those frames are never deduped.
if (typeof tl.vars?.onUpdate === "function") {
const total = typeof tl.totalDuration === "function" ? tl.totalDuration() : 0;
if (total > 0) intervals.push({ start: offset, end: offset + total });
// onUpdate can move anything (e.g. x/y from Math.random) — conservatively spatial.
if (total > 0) intervals.push({ start: offset, end: offset + total, spatial: true });
}
for (const child of tl.getChildren(false, true, true)) {
const start = offset + (typeof child.startTime === "function" ? child.startTime() : 0);
@@ -2789,7 +2810,10 @@ export async function computeStaticFrameSet(
const total = typeof child.totalDuration === "function" ? child.totalDuration() : single;
if (typeof child.getChildren === "function") {
if (total > single + 1e-6) {
intervals.push({ start, end: start + total });
// A repeating nested timeline's own children are walked for their real
// vars below; this span is a conservative (spatial) placeholder for the
// repeat/yoyo overrun, which a per-child walk doesn't otherwise cover.
intervals.push({ start, end: start + total, spatial: true });
// Still descend for hasTimelineCall even though the repeating
// span is already opaque (its frames are excluded from dedup
// regardless): a call() inside it is a review-flagged detection
@@ -2802,7 +2826,7 @@ export async function computeStaticFrameSet(
}
} else {
tweenCount++;
intervals.push({ start, end: start + total });
intervals.push({ start, end: start + total, spatial: isSpatial(child.vars) });
if (
total <= 1e-6 &&
(typeof child.vars?.onComplete === "function" ||
@@ -2855,7 +2879,7 @@ export async function computeStaticFrameSet(
hasUnresolvableClipStart,
hasTimelineCall,
};
});
}, SPATIAL_TWEEN_PROPERTIES);
const {
intervals,
@@ -2867,7 +2891,7 @@ export async function computeStaticFrameSet(
hasUnresolvableClipStart,
hasTimelineCall,
} = result as {
intervals: Array<{ start: number; end: number }>;
intervals: Array<{ start: number; end: number; spatial: boolean }>;
tweenCount: number;
duration: number;
hasVideo: boolean;
@@ -2881,6 +2905,7 @@ export async function computeStaticFrameSet(
return {
totalFrames,
staticFrameSet: new Set<number>(),
nonSpatialOnlyFrameSet: new Set<number>(),
hasVideo,
hasCanvas,
hasNonGsapAnim,
@@ -2890,10 +2915,14 @@ export async function computeStaticFrameSet(
};
}
const animated = new Set<number>();
for (const { start, end } of intervals) {
const spatialFrameSet = new Set<number>();
for (const { start, end, spatial } of intervals) {
const lo = Math.max(0, Math.floor(start * fps));
const hi = Math.min(totalFrames - 1, Math.ceil(end * fps));
for (let f = lo; f <= hi; f++) animated.add(f);
for (let f = lo; f <= hi; f++) {
animated.add(f);
if (spatial) spatialFrameSet.add(f);
}
}
for (const f of await computeClipBoundaryFrames(page, fps)) animated.add(f);
const reasons: string[] = [];
@@ -2916,9 +2945,14 @@ export async function computeStaticFrameSet(
if (!animated.has(f) && !animated.has(f - 1)) staticFrameSet.add(f);
}
}
const nonSpatialOnlyFrameSet = new Set<number>();
if (eligible) {
for (const f of animated) if (!spatialFrameSet.has(f)) nonSpatialOnlyFrameSet.add(f);
}
return {
totalFrames,
staticFrameSet,
nonSpatialOnlyFrameSet,
hasVideo,
hasCanvas,
hasNonGsapAnim,
@@ -3281,6 +3315,23 @@ async function armStaticDedup(
page: Page,
logInitPhase: (phase: string) => void,
): Promise<void> {
// Adaptive motion-blur sample-count classification shares the GSAP-timeline walk
// below but is gated independently of dedup (capture mode / before-capture hooks are
// about buffer-reuse safety, irrelevant to which properties a tween touches), so it
// is computed here, once, ahead of dedup's own idempotency check. Cached in
// `sharedStaticFrameStats` so dedup's own call further down does not repeat the walk.
// Reads `session.options.motionBlur` (the raw caller options), not `session.motionBlur`
// (the resolved plan) — every armStaticDedup call site runs before finalizeSessionInit
// resolves the plan, so the resolved field is never set yet at this point.
let sharedStaticFrameStats: Awaited<ReturnType<typeof computeStaticFrameSet>> | undefined;
if (
session.options.motionBlur &&
session.options.motionBlur.samplesPerFrame === undefined &&
!session.motionBlurNonSpatialFrames
) {
sharedStaticFrameStats = await computeStaticFrameSet(page, fpsToNumber(session.options.fps));
session.motionBlurNonSpatialFrames = sharedStaticFrameStats.nonSpatialOnlyFrameSet;
}
// Idempotent: the drawElement init path arms dedup BEFORE canvas injection
// (verification screenshots need the un-injected DOM), and initializeSession
// calls this again unconditionally afterwards. Once staticFrames is
@@ -3330,7 +3381,7 @@ async function armStaticDedup(
return;
}
const fps = fpsToNumber(session.options.fps);
const stats = await computeStaticFrameSet(page, fps);
const stats = sharedStaticFrameStats ?? (await computeStaticFrameSet(page, fps));
if (!stats.eligible || stats.staticFrameSet.size === 0) {
session.staticDedupSkipReason = "ineligible";
logInitPhase(`static-frame dedup: disabled (${stats.reason})`);
@@ -3761,13 +3812,52 @@ function finalizeSessionInit(session: CaptureSession): void {
session.isInitialized = true;
}
/** Choose how many samples this frame gets when the plan is adaptive: the floor with
* no probe for a frame confirmed non-spatial, otherwise two edge probes (see
* motionBlurProbeTimes) mapped through adaptiveSampleCount. Probe cost is returned
* alongside the count so the caller folds it into the frame's timing totals. */
async function resolveAdaptiveSampleCount(
session: CaptureSession,
frameIndex: number,
absFrameIndex: number,
plan: MotionBlurPlan,
fps: number,
sampleSeek: HfSeekOptions,
): Promise<{
samplesPerFrame: number;
seekMs: number;
beforeCaptureMs: number;
screenshotMs: number;
}> {
if (session.motionBlurNonSpatialFrames?.has(absFrameIndex)) {
return {
samplesPerFrame: DEFAULT_SAMPLES_PER_FRAME,
seekMs: 0,
beforeCaptureMs: 0,
screenshotMs: 0,
};
}
const { windowStart, windowEnd } = motionBlurProbeTimes(plan, absFrameIndex, fps);
const probeA = await captureFrameSurface(session, frameIndex, windowStart, sampleSeek);
const probeB = await captureFrameSurface(session, frameIndex, windowEnd, sampleSeek);
return {
samplesPerFrame: adaptiveSampleCount(probeDiffMagnitude(probeA.buffer, probeB.buffer)),
seekMs: probeA.seekMs + probeB.seekMs,
beforeCaptureMs: probeA.beforeCaptureMs + probeB.beforeCaptureMs,
screenshotMs: probeA.screenshotMs + probeB.screenshotMs,
};
}
/**
* Capture one output frame as the average of `plan.samplesPerFrame` sub-frame captures.
* Capture one output frame as the average of some number of sub-frame captures — fixed
* by the caller, or chosen per frame from measured motion (see
* `resolveAdaptiveSampleCount`).
*
* Callback invariant: exactly one eventful seek per output frame, at the frame time,
* arriving from the previous frame's time. Every sample seek suppresses events and the
* playhead is restored to the frame time afterwards, so a composition's own
* onUpdate/onComplete fire on the same interval boundaries as a render with blur off.
* arriving from the previous frame's time. Every sample seek (including a probe)
* suppresses events and the playhead is restored to the frame time afterwards, so a
* composition's own onUpdate/onComplete fire on the same interval boundaries as a
* render with blur off.
*/
async function captureAccumulatedFrame(
session: CaptureSession,
@@ -3786,8 +3876,25 @@ async function captureAccumulatedFrame(
suppressEvents: true,
subFrameDivisions: plan.subFrameDivisions,
};
let samplesPerFrame = plan.fixedSamplesPerFrame;
if (samplesPerFrame === null) {
const chosen = await resolveAdaptiveSampleCount(
session,
frameIndex,
absFrameIndex,
plan,
fps,
sampleSeek,
);
samplesPerFrame = chosen.samplesPerFrame;
totals.seekMs += chosen.seekMs;
totals.beforeCaptureMs += chosen.beforeCaptureMs;
totals.screenshotMs += chosen.screenshotMs;
}
const accumulator = new MotionBlurAccumulator(plan.blend);
for (const sampleTime of motionBlurSampleTimes(plan, absFrameIndex, fps)) {
for (const sampleTime of motionBlurSampleTimes(plan, absFrameIndex, fps, samplesPerFrame)) {
const sample = await captureFrameSurface(session, frameIndex, sampleTime, sampleSeek);
totals.seekMs += sample.seekMs;
totals.beforeCaptureMs += sample.beforeCaptureMs;
+99 -23
View File
@@ -1,13 +1,27 @@
import { describe, it, expect } from "vitest";
import {
ADAPTIVE_SAMPLE_STEPS,
MAX_SAMPLES_PER_FRAME,
MotionBlurAccumulator,
adaptiveSampleCount,
motionBlurProbeTimes,
motionBlurSampleTimes,
probeDiffMagnitude,
resolveMotionBlurPlan,
type MotionBlurBlendSpace,
} from "./motionBlur.js";
import { encodePng } from "../utils/alphaBlit.js";
/** Tick offsets from the frame instant for `samplesPerFrame` samples of `plan`, read
* back from `motionBlurSampleTimes` at a frame far enough from 0 that no sample clamps. */
const offsetsAtFrame10Fps30 = (
plan: NonNullable<ReturnType<typeof resolveMotionBlurPlan>>,
samplesPerFrame: number,
): number[] =>
motionBlurSampleTimes(plan, 10, 30, samplesPerFrame).map(
(t) => Math.round(t * 30 * plan.subFrameDivisions) - 10 * plan.subFrameDivisions,
);
const solid = (count: number, r: number, g: number, b: number, a = 255): Uint8Array => {
const out = new Uint8Array(count * 4);
for (let i = 0; i < count * 4; i += 4) {
@@ -37,55 +51,56 @@ describe("resolveMotionBlurPlan", () => {
expect(resolveMotionBlurPlan(undefined)).toBeNull();
});
it("defaults to After Effects: 16 samples, 180 degree shutter, phase -90, sRGB", () => {
it("defaults to After Effects: 180 degree shutter, phase -90, sRGB, adaptive count", () => {
const plan = resolveMotionBlurPlan({});
expect(plan).not.toBeNull();
expect(plan?.sampleTickOffsets).toHaveLength(16);
expect(plan?.blend).toBe("srgb");
expect(plan?.subFrameDivisions).toBe(4096);
// A caller that never sets samplesPerFrame gets adaptive (null), not a fixed 16.
expect(plan?.fixedSamplesPerFrame).toBeNull();
// 180 degrees at phase -90 is a window of a quarter frame either side of the instant.
expect(plan?.sampleTickOffsets[0]).toBe(-960);
expect(plan?.sampleTickOffsets[15]).toBe(960);
expect(plan?.windowStartFrames).toBeCloseTo(-0.25, 10);
expect(plan?.shutterFrames).toBeCloseTo(0.5, 10);
});
it("places the default samples symmetrically across a quarter frame either side", () => {
it("places 16 samples symmetrically across a quarter frame either side", () => {
// From the shutter formula, by hand: offset_k = (-90/360 + ((k+0.5)/16) * 180/360)
// frames, so k=0 is -0.234375 and k=15 is +0.234375. At 4096 ticks per frame that
// is exactly -960 and +960, with 128 ticks between neighbours.
const plan = resolveMotionBlurPlan({});
expect(plan?.subFrameDivisions).toBe(4096);
expect(plan?.sampleTickOffsets[0]).toBe(-960);
expect(plan?.sampleTickOffsets[15]).toBe(960);
expect(plan?.sampleTickOffsets).toHaveLength(16);
const gaps = plan?.sampleTickOffsets
.slice(1)
.map((t, i) => t - (plan.sampleTickOffsets[i] as number));
if (!plan) throw new Error("plan");
const offsets = offsetsAtFrame10Fps30(plan, 16);
expect(offsets[0]).toBe(-960);
expect(offsets[15]).toBe(960);
expect(offsets).toHaveLength(16);
const gaps = offsets.slice(1).map((t, i) => t - (offsets[i] as number));
expect(new Set(gaps)).toEqual(new Set([128]));
});
it("clamps the sample count to 1..64 and ignores non-finite input", () => {
expect(resolveMotionBlurPlan({ samplesPerFrame: 200 })?.sampleTickOffsets).toHaveLength(
it("clamps an explicit sample count to 1..64 and ignores non-finite input", () => {
expect(resolveMotionBlurPlan({ samplesPerFrame: 200 })?.fixedSamplesPerFrame).toBe(
MAX_SAMPLES_PER_FRAME,
);
expect(resolveMotionBlurPlan({ samplesPerFrame: 0 })?.sampleTickOffsets).toHaveLength(1);
expect(resolveMotionBlurPlan({ samplesPerFrame: Number.NaN })?.sampleTickOffsets).toHaveLength(
16,
);
expect(resolveMotionBlurPlan({ samplesPerFrame: 0 })?.fixedSamplesPerFrame).toBe(1);
expect(resolveMotionBlurPlan({ samplesPerFrame: Number.NaN })?.fixedSamplesPerFrame).toBe(16);
});
it("follows the shutter window when the angle and phase are changed", () => {
// The AD5 measurement of the real After Effects export: a window one frame either
// side of t, which is shutterAngle 720 at phase -360.
const plan = resolveMotionBlurPlan({ shutterAngle: 720, shutterPhase: -360 });
expect(plan?.sampleTickOffsets[0]).toBe(Math.round((-1 + 1 / 16) * 4096));
expect(plan?.sampleTickOffsets[15]).toBe(Math.round((1 - 1 / 16) * 4096));
if (!plan) throw new Error("plan");
const offsets = offsetsAtFrame10Fps30(plan, 16);
expect(offsets[0]).toBe(Math.round((-1 + 1 / 16) * 4096));
expect(offsets[15]).toBe(Math.round((1 - 1 / 16) * 4096));
});
});
describe("motionBlurSampleTimes", () => {
it("returns ascending distinct sub-frame times around the frame instant", () => {
it("returns ascending distinct sub-frame times around the frame instant, for the requested count", () => {
const plan = resolveMotionBlurPlan({});
if (!plan) throw new Error("plan");
const times = motionBlurSampleTimes(plan, 10, 30);
const times = motionBlurSampleTimes(plan, 10, 30, 16);
// Frame 10 at 30fps is tick 40960 on the 122880-tick-per-second grid; the first
// sample sits 960 ticks earlier and the last 960 later.
@@ -99,11 +114,72 @@ describe("motionBlurSampleTimes", () => {
it("clamps samples before the composition start to zero", () => {
const plan = resolveMotionBlurPlan({});
if (!plan) throw new Error("plan");
const times = motionBlurSampleTimes(plan, 0, 30);
const times = motionBlurSampleTimes(plan, 0, 30, 16);
expect(times[0]).toBe(0);
expect(times[15]).toBe(960 / 122880);
});
it("changing the count only changes how densely the SAME window is filled", () => {
const plan = resolveMotionBlurPlan({});
if (!plan) throw new Error("plan");
const { windowStart, windowEnd } = motionBlurProbeTimes(plan, 10, 30);
const times16 = motionBlurSampleTimes(plan, 10, 30, 16);
const times64 = motionBlurSampleTimes(plan, 10, 30, 64);
expect(times64).toHaveLength(64);
// Every sample, at either count, stays strictly inside the true window edges, and
// more samples pack closer to those edges (finer, not a wider or shifted window).
for (const t of [...times16, ...times64]) {
expect(t).toBeGreaterThan(windowStart);
expect(t).toBeLessThan(windowEnd);
}
expect(times64[0] as number).toBeLessThan(times16[0] as number);
expect(times64[63] as number).toBeGreaterThan(times16[15] as number);
});
});
describe("motionBlurProbeTimes", () => {
it("returns the TRUE window edges, outside every sample regardless of count", () => {
const plan = resolveMotionBlurPlan({});
if (!plan) throw new Error("plan");
const { windowStart, windowEnd } = motionBlurProbeTimes(plan, 10, 30);
const samples = motionBlurSampleTimes(plan, 10, 30, 16);
expect(windowStart).toBeLessThan(samples[0] as number);
expect(windowEnd).toBeGreaterThan(samples[15] as number);
});
});
describe("adaptiveSampleCount", () => {
it("is monotone non-decreasing and bottoms out at the first step's sample count", () => {
expect(adaptiveSampleCount(0)).toBe(ADAPTIVE_SAMPLE_STEPS[0]?.samples);
let previous = 0;
for (const diff of [0, 1, 2, 3, 4, 5, 6, 7, 50]) {
const count = adaptiveSampleCount(diff);
expect(count).toBeGreaterThanOrEqual(previous);
previous = count;
}
expect(adaptiveSampleCount(1000)).toBe(MAX_SAMPLES_PER_FRAME);
});
});
describe("probeDiffMagnitude", () => {
const png = (r: number, g: number, b: number, a = 255) => encodePng(1, 1, solid(1, r, g, b, a));
it("is zero for identical probes", () => {
expect(probeDiffMagnitude(png(10, 20, 30), png(10, 20, 30))).toBe(0);
});
it("scales with the byte difference between probes", () => {
// Per channel |0-255| for r,g,b and 0 for alpha (both opaque): mean = 3*255/4.
expect(probeDiffMagnitude(png(0, 0, 0), png(255, 255, 255))).toBeCloseTo((3 * 255) / 4, 6);
});
it("rejects a geometry mismatch rather than diffing garbage", () => {
expect(() => probeDiffMagnitude(encodePng(2, 1, solid(2, 0, 0, 0)), png(0, 0, 0))).toThrow(
/geometry mismatch/,
);
});
});
describe("MotionBlurAccumulator", () => {
+147 -26
View File
@@ -35,10 +35,8 @@ export interface MotionBlurOptions {
export type MotionBlurBlendSpace = "srgb" | "linear";
/**
* A resolved plan. Only `resolveMotionBlurPlan` produces one, so every field is already
* clamped and the sample offsets are integers on a fixed sub-frame tick grid.
*/
/** A resolved plan. The window is fixed by shutter angle/phase alone, independent of
* how many samples fill it, which is what lets the count vary per frame. */
export interface MotionBlurPlan {
readonly blend: MotionBlurBlendSpace;
/**
@@ -46,13 +44,19 @@ export interface MotionBlurPlan {
* contains the output frame grid, so the frame-time instant never moves.
*/
readonly subFrameDivisions: number;
/** Tick offsets from the frame instant, ascending. Integers, so every host agrees. */
readonly sampleTickOffsets: readonly number[];
/** Window start, in frames relative to the frame instant (`shutterPhase / 360`). */
readonly windowStartFrames: number;
/** Window width, in frames (`shutterAngle / 360`). */
readonly shutterFrames: number;
/** Explicit count from the caller (clamped to 1..64), or null for adaptive per-frame. */
readonly fixedSamplesPerFrame: number | null;
}
export const MAX_SAMPLES_PER_FRAME = 64;
const DEFAULT_SAMPLES_PER_FRAME = 16;
/** Sample count for a frame confirmed to need no measurement (the adaptive floor, and
* the count an explicit `samplesPerFrame` defaults to). */
export const DEFAULT_SAMPLES_PER_FRAME = 16;
const DEFAULT_SHUTTER_ANGLE = 180;
const DEFAULT_SHUTTER_PHASE = -90;
@@ -72,27 +76,42 @@ export function resolveMotionBlurPlan(
): MotionBlurPlan | null {
if (!options) return null;
const samplesPerFrame = Math.min(
MAX_SAMPLES_PER_FRAME,
Math.max(1, Math.round(finiteOr(options.samplesPerFrame, DEFAULT_SAMPLES_PER_FRAME))),
);
const fixedSamplesPerFrame =
options.samplesPerFrame === undefined
? null
: Math.min(
MAX_SAMPLES_PER_FRAME,
Math.max(1, Math.round(finiteOr(options.samplesPerFrame, DEFAULT_SAMPLES_PER_FRAME))),
);
const shutterAngle = finiteOr(options.shutterAngle, DEFAULT_SHUTTER_ANGLE);
const shutterPhase = finiteOr(options.shutterPhase, DEFAULT_SHUTTER_PHASE);
const blend: MotionBlurBlendSpace = options.blend === "linear" ? "linear" : "srgb";
const windowStartFrames = shutterPhase / 360;
const shutterFrames = shutterAngle / 360;
const sampleTickOffsets: number[] = [];
for (let k = 0; k < samplesPerFrame; k++) {
const offsetFrames = windowStartFrames + ((k + 0.5) / samplesPerFrame) * shutterFrames;
sampleTickOffsets.push(Math.round(offsetFrames * SUB_FRAME_DIVISIONS));
}
return {
blend,
subFrameDivisions: SUB_FRAME_DIVISIONS,
windowStartFrames: shutterPhase / 360,
shutterFrames: shutterAngle / 360,
fixedSamplesPerFrame,
};
}
return { blend, subFrameDivisions: SUB_FRAME_DIVISIONS, sampleTickOffsets };
/** Tick offsets from the frame instant for `samplesPerFrame` samples spread evenly
* across the plan's window, ascending. Integers, so every host agrees. */
function sampleTickOffsets(plan: MotionBlurPlan, samplesPerFrame: number): number[] {
const offsets: number[] = [];
for (let k = 0; k < samplesPerFrame; k++) {
const offsetFrames =
plan.windowStartFrames + ((k + 0.5) / samplesPerFrame) * plan.shutterFrames;
offsets.push(Math.round(offsetFrames * plan.subFrameDivisions));
}
return offsets;
}
/**
* Absolute seek times for one output frame, in ascending order.
* Absolute seek times for one output frame, in ascending order, for `samplesPerFrame`
* samples spread across the plan's window (independent of `plan.fixedSamplesPerFrame` —
* the caller decides how many samples this particular frame gets).
*
* Built from integer ticks rather than by adding floats to `frameIndex / fps`, so the
* page-side quantizer recovers the intended tick exactly on every host. Times before the
@@ -102,10 +121,35 @@ export function motionBlurSampleTimes(
plan: MotionBlurPlan,
frameIndex: number,
fps: number,
samplesPerFrame: number,
): number[] {
const grid = fps * plan.subFrameDivisions;
const frameTicks = frameIndex * plan.subFrameDivisions;
return plan.sampleTickOffsets.map((offset) => Math.max(0, frameTicks + offset) / grid);
return sampleTickOffsets(plan, samplesPerFrame).map(
(offset) => Math.max(0, frameTicks + offset) / grid,
);
}
/** The two true shutter-window edges (not samples — see `motionBlurProbeTimes`'s
* caller), for measuring motion magnitude before a sample count is chosen. */
export function motionBlurProbeTimes(
plan: MotionBlurPlan,
frameIndex: number,
fps: number,
): {
windowStart: number;
windowEnd: number;
} {
const grid = fps * plan.subFrameDivisions;
const frameTicks = frameIndex * plan.subFrameDivisions;
const startTick = Math.round(plan.windowStartFrames * plan.subFrameDivisions);
const endTick = Math.round(
(plan.windowStartFrames + plan.shutterFrames) * plan.subFrameDivisions,
);
return {
windowStart: Math.max(0, frameTicks + startTick) / grid,
windowEnd: Math.max(0, frameTicks + endTick) / grid,
};
}
/**
@@ -127,19 +171,96 @@ export function motionBlurWindowIsStatic(
frameIndex: number,
staticFrames: ReadonlySet<number>,
): boolean {
const first = Math.floor(
frameIndex - 1 + Math.min(...plan.sampleTickOffsets) / plan.subFrameDivisions,
);
// Bound by the outermost sample this frame could take (its half-step inset from the
// true edge), not the edge itself. An adaptive plan's K is unknown here, so use the
// widest possible inset (K=64) rather than the true edge, which any chosen K stays inside.
const worstCaseSamples = plan.fixedSamplesPerFrame ?? MAX_SAMPLES_PER_FRAME;
const inset = plan.shutterFrames / (2 * worstCaseSamples);
const minOffsetFrames = plan.windowStartFrames + inset;
const maxOffsetFrames = plan.windowStartFrames + plan.shutterFrames - inset;
const first = Math.floor(frameIndex - 1 + Math.min(0, minOffsetFrames));
// A sample landing inside [F, F+1) reads content the frame set only pins down at both
// ends, so the frame after the last one touched has to be static as well.
const last =
Math.floor(frameIndex + Math.max(...plan.sampleTickOffsets) / plan.subFrameDivisions) + 1;
const last = Math.floor(frameIndex + Math.max(0, maxOffsetFrames)) + 1;
for (let frame = first; frame <= last; frame++) {
if (!staticFrames.has(frame)) return false;
}
return true;
}
/** Diff magnitude → sample count, ascending and exhaustive. Calibration provenance and
* measured numbers are in the PR description for #4029; the 32-sample step is
* interpolated, not independently measured. */
export const ADAPTIVE_SAMPLE_STEPS: ReadonlyArray<{
readonly maxDiff: number;
readonly samples: number;
}> = [
{ maxDiff: 3, samples: DEFAULT_SAMPLES_PER_FRAME },
{ maxDiff: 6, samples: 32 },
{ maxDiff: Infinity, samples: MAX_SAMPLES_PER_FRAME },
];
/** Sample count for a frame whose probe diff magnitude is `diff` (see `ADAPTIVE_SAMPLE_STEPS`). */
export function adaptiveSampleCount(diff: number): number {
for (const step of ADAPTIVE_SAMPLE_STEPS) {
if (diff <= step.maxDiff) return step.samples;
}
return MAX_SAMPLES_PER_FRAME;
}
/** GSAP property names that move an element in screen space. Matches the Transform
* categories of SUPPORTED_PROPS in packages/parsers/src/gsapConstants.ts, minus
* transformOrigin (that file's own classifyTweenPropertyGroup excludes it: a pivot-point
* modifier, not independent motion). Passed into computeStaticFrameSet's page.evaluate as data. */
export const SPATIAL_TWEEN_PROPERTIES: readonly string[] = [
"x",
"y",
"z",
"xPercent",
"yPercent",
"rotation",
"rotate",
"rotationX",
"rotationY",
"rotationZ",
"scale",
"scaleX",
"scaleY",
"scaleZ",
"skewX",
"skewY",
"perspective",
"transformPerspective",
"top",
"left",
"right",
"bottom",
"width",
"height",
"translate",
"translateX",
"translateY",
"translateZ",
"transform",
];
/** Mean absolute byte difference between two probe captures (never a sample that
* enters the average) — the motion signal `adaptiveSampleCount` maps to a count. */
export function probeDiffMagnitude(a: Buffer, b: Buffer): number {
const decodedA = decodePng(a);
const decodedB = decodePng(b);
if (decodedA.width !== decodedB.width || decodedA.height !== decodedB.height) {
throw new Error(
`probeDiffMagnitude: sample geometry mismatch (${decodedA.width}x${decodedA.height} vs ${decodedB.width}x${decodedB.height})`,
);
}
let sum = 0;
for (let i = 0; i < decodedA.data.length; i++) {
sum += Math.abs((decodedA.data[i] as number) - (decodedB.data[i] as number));
}
return sum / decodedA.data.length;
}
const SRGB_TO_LINEAR = (() => {
const lut = new Float64Array(256);
for (let i = 0; i < 256; i++) lut[i] = srgbByteToLinear(i);