Files
hyperframes/packages/player/tests/perf/runner.ts
Vance Ingalls 10d2725b54 perf(player): p0-1a perf test infra + composition-load smoke test (#399)
## Summary

First slice of `P0-1` from the player perf proposal: lays the foundation for a player perf gate so later PRs can plug in fps / scrub / drift / parity scenarios without rebuilding infrastructure. Ships one smoke scenario (`03-load`, cold + warm composition load) to prove the gate end-to-end on real numbers.

## Why

There was no automated way to catch player perf regressions. Every perf concern in the existing proposal — composition load time, sustained FPS, scrub p95, mirror-clock drift, live-vs-seek parity — needs the same plumbing: a same-origin harness, a Puppeteer runner, a baseline file, a gate that emits structured results, and a CI workflow that runs the right scenarios on the right changes. Building that up-front in one reviewable PR lets every subsequent perf PR (`P0-1b`, `P0-1c`, and beyond) be a 100-line scenario file plus a baseline entry instead of re-litigating the framework.

## What changed

### Harness — `packages/player/tests/perf/server.ts`

- `Bun.serve` on a free port, single same-origin host for the player IIFE bundle, hyperframe runtime, GSAP from `node_modules`, and fixture HTML.
- Same-origin matters: cross-origin would force every probe through `postMessage`, hiding bugs and inflating numbers in ways production never sees. Tests should measure the path the studio editor actually takes.
- Routes:
  - `/player.js` → built IIFE bundle (rebuilt on demand).
  - `/vendor/runtime.js`, `/vendor/gsap.min.js` → resolved from `node_modules` so fixtures don't need to ship copies.
  - `/fixtures/*` → fixture HTML.

### Runner — `packages/player/tests/perf/runner.ts`

- `puppeteer-core` thin wrappers (`launchBrowser`, `loadHostPage`).
- Uses the system Chrome detected by `setup-chrome` in CI rather than the bundled puppeteer revision — keeps the action smaller, lets us pin Chrome version policy at the workflow level, and matches what users actually run.

### Gate — `packages/player/tests/perf/perf-gate.ts` + `baseline.json`

- Loads `baseline.json` (initial budgets: cold/warm comp load, fps, scrub p95 isolated/inline, drift max/p95) with a 10% `allowedRegressionRatio`.
- Per-metric direction (`lower-is-better` / `higher-is-better`) so the same evaluator handles latency and throughput.
- Returns a structured `GateReport` consumed by both the CLI (table output) and `metrics.json` (CI artifact).
- Two modes: `measure` (log only — used during the rollout) and `enforce` (fail the build) — flip per-metric once we trust the signal, without touching the harness.

### CLI orchestrator — `packages/player/tests/perf/index.ts`

- Parses `--mode` / `--scenarios` / `--runs` / `--fixture` in both space- and equals-separated form (so `--scenarios fps,scrub` and `--scenarios=fps,scrub` both work — matches what humans type and what GitHub Actions emits).
- Runs scenarios, runs the gate, and **always** writes `results/metrics.json` with schema version, git SHA, metrics, and gate rows — so failed runs are still investigable from the artifact alone.

### Fixture + smoke scenario

- `fixtures/gsap-heavy/index.html`: 200 stagger-animated tiles, no media. Heavy enough to make load time meaningful, light enough to be deterministic.
- `scenarios/03-load.ts`: cold + warm composition load. Measures from navigation start to player `ready` event, reports p95 across runs.

### CI — `.github/workflows/player-perf.yml`

- `paths-filter` on `player` / `core` / `runtime` — perf only runs when something that could move the needle actually changed.
- Sets up bun + node + chrome, runs perf in `measure` mode on a shard matrix (so future scenarios shard naturally), uploads `metrics.json` artifacts, and a summary job aggregates shard results into a single PR comment.

### Wiring

- `packages/player`: `puppeteer-core`, `gsap`, `@types/bun` devDeps; typecheck extended to cover the perf `tsconfig`; new `perf` script.
- Root `package.json`: `player:perf` workspace script so `bun run player:perf` runs the whole suite locally with the same flags CI uses.
- `.gitignore`: `packages/player/tests/perf/results/`.
- Separate `tests/perf/tsconfig.json` so test code doesn't pollute the package `rootDir` while still being typechecked.

## Test plan

- [x] Local: `bun run player:perf` passes — cold p95 ≈ 386 ms, warm p95 ≈ 375 ms, both well under the seeded baselines.
- [x] Typecheck, lint, format pass on the perf workspace.
- [x] Existing player unit tests (71/71) still green.
- [ ] First CI run after merge will be the real signal: confirms `setup-chrome` works on hosted runners, the shard matrix wires up, and `metrics.json` artifacts upload.

## Stack

Step `P0-1a` of the player perf proposal. The next two slices are content-only — they don't touch the harness:

- `P0-1b` (#400): adds `02-fps`, `04-scrub`, `05-drift` scenarios on a 10-video-grid fixture.
- `P0-1c` (#401): adds `06-parity` (live playback vs. synchronously-seeked reference, compared via SSIM).

Wiring this gate up first means each follow-up is a self-contained scenario file + baseline row + workflow shard.
2026-04-22 18:04:05 -07:00

138 lines
4.5 KiB
TypeScript

import { existsSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import puppeteer, { type Browser, type LaunchOptions, type Page } from "puppeteer-core";
/**
* Puppeteer browser + page helpers shared across all perf scenarios.
*
* Browser launch args mirror packages/producer/src/parity-harness.ts so we get
* the same SwiftShader-backed WebGL output and font hinting between perf runs
* and visual parity runs. That parity matters for P0-1c (live-playback parity)
* and is harmless for the load/scrub/drift scenarios.
*/
const HERE = dirname(fileURLToPath(import.meta.url));
const PLAYER_PKG = resolve(HERE, "../..");
export type LaunchOpts = {
width?: number;
height?: number;
headless?: boolean;
};
export type LoadOpts = {
/** Fixture name (must match a directory under tests/perf/fixtures/). */
fixture: string;
width?: number;
height?: number;
/** Override timeout in ms for the player `ready` event. Default 30s. */
readyTimeoutMs?: number;
};
export type LoadResult = {
/** Wall-clock ms from page navigation start to player `ready` event. */
loadMs: number;
/** Composition duration as reported by the player (seconds). */
duration: number;
};
declare global {
interface Window {
__playerReady?: boolean;
__playerReadyAt?: number;
__playerNavStart?: number;
__playerDuration?: number;
__playerError?: string;
}
}
function findChromeExecutable(): string | undefined {
if (process.env.CHROME_PATH) return process.env.CHROME_PATH;
if (process.env.PUPPETEER_EXECUTABLE_PATH) return process.env.PUPPETEER_EXECUTABLE_PATH;
const candidates = [
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
"/usr/bin/google-chrome",
"/usr/bin/chromium-browser",
"/usr/bin/chromium",
];
for (const path of candidates) {
if (existsSync(path)) return path;
}
return undefined;
}
export async function launchBrowser(options: LaunchOpts = {}): Promise<Browser> {
const width = options.width ?? 1920;
const height = options.height ?? 1080;
const executablePath = findChromeExecutable();
if (!executablePath) {
throw new Error(
`[player-perf] no chrome executable found. Set CHROME_PATH or install Google Chrome. (looked in: $CHROME_PATH, $PUPPETEER_EXECUTABLE_PATH, /Applications/Google Chrome.app, /usr/bin/google-chrome)`,
);
}
const launchOptions: LaunchOptions = {
executablePath,
headless: options.headless ?? true,
defaultViewport: {
width,
height,
deviceScaleFactor: 1,
},
args: [
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage",
"--disable-accelerated-2d-canvas",
"--enable-webgl",
"--ignore-gpu-blocklist",
"--use-gl=angle",
"--use-angle=swiftshader",
"--font-render-hinting=none",
"--force-color-profile=srgb",
"--autoplay-policy=no-user-gesture-required",
`--window-size=${width},${height}`,
],
};
return puppeteer.launch(launchOptions);
}
/**
* Navigate a page to the host shell and wait for the player's `ready` event.
* Returns the wall-clock ms between `Page.goto` start and the `ready` event,
* along with the composition duration the player reported.
*/
export async function loadHostPage(
page: Page,
origin: string,
options: LoadOpts,
): Promise<LoadResult> {
const width = options.width ?? 1920;
const height = options.height ?? 1080;
const readyTimeoutMs = options.readyTimeoutMs ?? 30_000;
const url = `${origin}/host.html?fixture=${encodeURIComponent(options.fixture)}&width=${width}&height=${height}`;
const t0 = performance.now();
await page.goto(url, { waitUntil: "domcontentloaded", timeout: readyTimeoutMs });
await page.waitForFunction(() => window.__playerReady === true || !!window.__playerError, {
timeout: readyTimeoutMs,
});
const error = await page.evaluate(() => window.__playerError ?? null);
if (error) throw new Error(`[player-perf] player reported error during load: ${error}`);
const loadMs = performance.now() - t0;
const duration = (await page.evaluate(() => window.__playerDuration ?? 0)) ?? 0;
return { loadMs, duration };
}
export function percentile(samples: number[], pct: number): number {
if (samples.length === 0) return 0;
const sorted = [...samples].sort((a, b) => a - b);
const idx = Math.min(sorted.length - 1, Math.max(0, Math.ceil((pct / 100) * sorted.length) - 1));
return sorted[idx] ?? 0;
}
export function repoPlayerDir(): string {
return PLAYER_PKG;
}