feat(cli): check finds text over a face, measure cuts names resumed takes, measure hands (#8074)

* feat(cli): check reports text drawn over a face in a video

The layout audit reports text painted in front of an opaque video with its
word boxes and where the video's frame lands; Node finds the faces in the
video file with the detector `measure face` uses and adds a `text_over_face`
finding (info from 5% of the face covered, warning from 30%) with
`coveredFraction` and `faceRect`. `layout.faceCheck` in the report says
whether the pass looked. `data-layout-allow-on-face` waives it.

Also reads a two-token `object-position` whose first token is "50%" as the
horizontal one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* feat(cli): measure cuts names the shot a take resumes

A shot that returns to an earlier camera setup after at least one other
shot (a wide take, a close-up cut into it, the wide take again) carries
`continues`: the index in `shots` of the shot it resumes. It is read from
the signatures `cuts` already decodes, by the scene's edges, so a person
who has moved or an exposure that drifted does not hide the match.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* feat(cli): measure hands follows a hand through a clip

`measure hands <video>` reports where a hand's palm (with the hand's
size), index fingertip or pinch is in each sampled frame of a stretch of
up to 15 s, from MediaPipe's hand landmarker. A frame with no such hand
is null, never filled in. The landmarker runs in the CLI's headless
Chrome on a page answered from disk; its code and model are not shipped
and are downloaded on first use, pinned by size and sha256, into the
trusted cache.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* fix(cli): text_over_face is info only, and a stalled OpenCV download no longer stalls check

The face detector can take texture for a face, so text_over_face is
reported as info however much of the face is covered and never fails
--strict.

check passes its deadline and a 30 s limit into the detector's first-use
OpenCV download. A download that does not answer costs the face pass
only: faceCheck is { ran: false, reason }, the other findings are kept
and the process exits.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
WaterrrForever
2026-11-17 17:16:20 +00:00
committed by GitHub
co-authored by Claude Fable 5.2
parent f5ccfcc960
commit 820d63d7f9
26 changed files with 2530 additions and 35 deletions
+24
View File
@@ -1386,6 +1386,25 @@ transparent. A fully transparent frame is treated as unread on purpose, since a
failed draw looks the same; mark text that sits under a cutout before its subject
enters with `data-layout-allow-occlusion`.
Text painted in front of an opaque `<video>` is judged against the faces in the
frame that video shows at the sample. `text_over_face` is an info finding when
the words cover 5% or more of a face; `coveredFraction` is the share of the face
covered and `faceRect` is the face in canvas pixels. It is never more than info,
so it never fails `--strict`: the detector can take texture for a face. The
faces are found in the video file itself, with the detector
[`measure face`](#measure) uses, so only a video served from the project is
looked at. That detector's first use downloads OpenCV; when the download has not
answered within 30 seconds the check goes on without this pass and `faceCheck`
says why. A rotated video and a frame not yet
decoded are skipped, a transparent cutout between the text and the take is looked
through, and text behind a cutout stays `text_occluded`'s to judge. Mark an
intended overlap with `data-layout-allow-on-face` on the text or an ancestor.
`layout.faceCheck` in the JSON says whether the pass looked, so no finding reads
apart from no look: `{ "ran": true, "candidates": 3, "faces": 2 }` counts the
text-over-video samples examined and the distinct video frames a face was found
in (`unreadVideos` is added when some files could not be read), and
`{ "ran": false, "reason": "..." }` says why it could not.
`canvas_covered` is an error when an opaque element spanning the canvas sits above
every other visible clip (shape, image, video or text) for most of the film, so the
frame shows only it: typically a full-frame background placed last in the DOM. It
@@ -1726,8 +1745,13 @@ npx hyperframes measure face ref.mp4 --start 1 --end 2.5
| `energy` | How much two clips move, and whether at the same moments |
| `look` | Brightness, contrast, saturation and warmth, alone or against another video |
| `face` | The face's size and position over a window |
| `hands` | Where a hand's palm, index fingertip or pinch is in each frame of a stretch |
| `sync` | The lag between mouth motion and the voice |
In `cuts`, a shot that returns to an earlier camera setup after at least one other shot (a wide take, a close-up
cut into it, the wide take again) carries `continues`: the index in `shots` of the shot it resumes, so one take can
feed both. A shot that resumes nothing has no `continues`.
### `diff`
Scores a render, or a project folder without rendering it, against the original
@@ -10,6 +10,8 @@ const SRC = fileURLToPath(new URL("..", import.meta.url));
const OPENS_NO_COMPOSITION: Record<string, string> = {
"browser/launch.ts": "defines the launch, and its GPU probe opens a blank page",
"capture/launchCaptureBrowser.ts": "`capture` opens a website, not a composition",
"measure/hands.ts":
"`measure hands` opens its own page of pinned MediaPipe code; every request is answered from disk or aborted",
};
const WATCHED = new Set(["launchManagedBrowser", "acquireBrowser", "launchWithPageNetwork"]);
+28
View File
@@ -1726,6 +1726,7 @@ function reportWithFindings(overrides: Partial<CheckReport> = {}): CheckReport {
tolerance: 2,
totalIssueCount: 0,
truncated: false,
faceCheck: { ran: true, candidates: 0, faces: 0 },
},
motion: { ...emptySection(), enabled: false, samples: 0 },
contrast: { ...emptySection(), enabled: true, samples: [], checked: 0, passed: 0 },
@@ -4374,6 +4375,33 @@ it("applies transition sample limits within the requested range", async () => {
expect(report.layout.transitionSamplesDropped).toBe(3);
});
describe("layout.faceCheck", () => {
it("says the face pass ran when no text was in front of a video", async () => {
const { report } = await runScenario(fakeDriver({ collectTextOverVideo: async () => [] }));
expect(report.layout.faceCheck).toEqual({ ran: true, candidates: 0, faces: 0 });
});
it("says why when the pass could not look", async () => {
const { report } = await runScenario(fakeDriver());
expect(report.layout.faceCheck).toEqual({
ran: false,
reason: "this check driver does not report text over video",
});
});
it("is present with a reason when the browser never ran", async () => {
const { deps } = dependencies(fakeDriver());
deps.runBrowserCheck = async () => {
throw new Error("Chrome did not start");
};
const report = await runCheckPipeline(PROJECT, DEFAULT_CHECK_OPTIONS, deps);
expect(report.layout.faceCheck).toEqual({
ran: false,
reason: "the check stopped before looking for faces under text",
});
});
});
describe("check cache", () => {
const dirs: string[] = [];
afterEach(() => {
@@ -1294,11 +1294,14 @@
if (tokens[0] === "top" || tokens[0] === "bottom") y = tokens[0];
else x = tokens[0];
} else {
// The first token that names no side is the horizontal one, even when it is the default "50%".
let xGiven = false;
for (const token of tokens) {
if (token === "top" || token === "bottom") y = token;
else if (token === "left" || token === "right") x = token;
else if (x === "50%") x = token;
else y = token;
else if (token === "left" || token === "right" || !xGiven) {
x = token;
xGiven = true;
} else y = token;
}
}
return { x: objectPositionOffset(x, freeX), y: objectPositionOffset(y, freeY) };
@@ -1965,6 +1968,139 @@
};
}
// text_over_face: headless Chrome has no face detector, so the audit only reports which text is
// painted in front of which opaque video and where that video's frame lands; Node finds the
// faces in the file and judges the cover (utils/textOverFace.ts).
const FACE_PROBE_MAX_COLUMNS = 16;
let faceVideoKinds = new Map();
let textOverVideo = { time: Number.NaN, candidates: [] };
// Kept apart from videoAlphaFrames: a frame that cannot be read here is skipped, never reported
// as judged opaque.
function faceVideoKind(video) {
let kind = faceVideoKinds.get(video);
if (!kind) {
const frame = videoAlphaFrames.get(video) || readVideoAlphaFrame(video);
const cutout = frame.kind === "matte" && !hasOpaqueBackground(getComputedStyle(video));
kind = cutout ? "cutout" : frame.kind === "unread" ? "unread" : "opaque";
faceVideoKinds.set(video, kind);
}
return kind;
}
// The opaque video the text is painted directly in front of at (x, y). Null where something
// opaque above hides the text, where the text sits on an opaque panel, and where the video's
// frame cannot be read or mapped. A transparent cutout beneath the text is looked through to the
// video under it; text inside the box of one above it is text_occluded's to judge, because the
// cutout repaints its subject, face included, over the text.
function videoBeneathTextAt(element, x, y) {
let beneath = false;
for (const hit of document.elementsFromPoint(x, y)) {
if (!isForeignElement(element, hit)) {
beneath = true;
continue;
}
if (hasIgnoreFlag(hit)) continue;
const media = authoredElement(hit);
if (media.tagName === "VIDEO" && opacityChain(hit) >= OCCLUDING_ALPHA) {
const kind = faceVideoKind(media);
if (kind === "cutout" && beneath) continue;
if (kind === "cutout" || !beneath) return null;
return kind === "opaque" ? media : null;
}
if (isOpaqueOccluder(hit, x, y, element)) return null;
}
return null;
}
// The stretches of one word box in front of a video, probed in columns half a line high, so a
// word that runs off the video or under an opaque element counts only for the part in front.
function wordPiecesOverVideo(element, rect, add) {
const columns = Math.max(
1,
Math.min(FACE_PROBE_MAX_COLUMNS, Math.round((2 * rect.width) / rect.height)),
);
const step = rect.width / columns;
const y = rect.top + rect.height / 2;
let run = null;
for (let column = 0; column <= columns; column++) {
const video =
column < columns ? videoBeneathTextAt(element, rect.left + (column + 0.5) * step, y) : null;
if (run && run.video !== video) {
add(run.video, {
x: round(rect.left + run.from * step),
y: round(rect.top),
width: round((column - run.from) * step),
height: round(rect.height),
});
run = null;
}
if (video && !run) run = { video, from: column };
}
}
// Where the video's whole frame lands after object-fit and object-position (it can spill past
// the element's box, which crops it), and the part of the box the canvas shows.
function videoPlacement(video, rootRect) {
const mapping = paintedSourceMapping(video, video.videoWidth, video.videoHeight);
if (!mapping) return null;
const { rect, offsets, paintedWidth, paintedHeight } = mapping;
const left = Math.max(rect.left, rootRect.left);
const top = Math.max(rect.top, rootRect.top);
const right = Math.min(rect.right, rootRect.right);
const bottom = Math.min(rect.bottom, rootRect.bottom);
if (right <= left || bottom <= top) return null;
return {
selector: uniqueSelectorFor(video),
src: video.currentSrc || video.src || "",
mediaTime: video.currentTime,
box: {
x: round(left),
y: round(top),
width: round(right - left),
height: round(bottom - top),
},
frame: {
x: round(mapping.flipX ? rect.right - offsets.x - paintedWidth : rect.left + offsets.x),
y: round(mapping.flipY ? rect.bottom - offsets.y - paintedHeight : rect.top + offsets.y),
width: round(paintedWidth),
height: round(paintedHeight),
},
flipX: mapping.flipX,
flipY: mapping.flipY,
};
}
function textOverVideoCandidates(element, videoRects, rootRect) {
if (videoRects.length === 0 || typeof document.elementsFromPoint !== "function") return [];
if (authoredElement(element).closest("[data-layout-allow-on-face]")) return [];
if (opacityChain(element) < OCCLUDING_ALPHA) return [];
const textRect = textRectFor(element, true);
if (!textRect || !videoRects.some((rect) => intersectionArea(textRect, rect) > 0)) return [];
const piecesByVideo = new Map();
const add = (video, piece) => {
const pieces = piecesByVideo.get(video);
if (pieces) pieces.push(piece);
else piecesByVideo.set(video, [piece]);
};
for (const { rects } of wordClientRects(element)) {
for (const rect of rects) wordPiecesOverVideo(element, rect, add);
}
const candidates = [];
for (const [video, wordRects] of piecesByVideo) {
const placement = videoPlacement(video, rootRect);
if (!placement || !placement.src) continue;
candidates.push({
...candidateAnchor(element),
text: textContentFor(element, true),
rect: textRect,
wordRects,
video: placement,
});
}
return candidates;
}
// canvas_covered: an opaque element over most of the canvas with every other painted clip beneath it.
const COVER_CANVAS_SHARE = 0.9;
const COVERED_CONTENT_SHARE = 0.9;
@@ -2897,6 +3033,11 @@
const issues = [];
const matteResults = new Map();
videoAlphaFrames = new Map();
faceVideoKinds = new Map();
const overVideo = [];
const videoRects = Array.from(root.querySelectorAll("video"), (video) =>
video.getBoundingClientRect(),
);
const restoreHits = restoreHitTesting(root);
try {
@@ -2913,6 +3054,7 @@
}
const invisible = invisibleTextIssue(element, time);
if (invisible) issues.push(invisible);
overVideo.push(...textOverVideoCandidates(element, videoRects, rootRect));
}
const covered = canvasCoveredIssue(root, rootRect, elements, time);
if (covered) issues.push(covered);
@@ -2930,9 +3072,16 @@
issues.push(...canvasEdgeIssues(root, rootRect, time, tolerance));
issues.push(...connectorDetachmentIssues(root, rootRect, time));
issues.push(...connectorOrphanIssues(root, rootRect, time));
textOverVideo = { time, candidates: overVideo };
return issues;
};
// What the audit call for this time found, collected in its pass: hit testing is restored there
// and each video's frame is read once.
window.__hyperframesTextOverVideo = function collectTextOverVideo(options) {
return options && options.time === textOverVideo.time ? textOverVideo.candidates : [];
};
// Reruns only the overlap detector (same threshold, no new surface) on a fine grid for the dense motion re-sampling pass.
window.__hyperframesOverlapAudit = function auditOverlap(options) {
const time = options && typeof options.time === "number" ? options.time : 0;
@@ -0,0 +1,389 @@
// @vitest-environment happy-dom
import { afterEach, describe, expect, it, vi } from "vitest";
import { readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
const script = readFileSync(
join(dirname(fileURLToPath(import.meta.url)), "layout-audit.browser.js"),
"utf-8",
);
interface Box {
x: number;
y: number;
width: number;
height: number;
}
interface Candidate {
selector: string;
dataAttributes: Record<string, string>;
sourceFile: string;
text: string;
wordRects: Box[];
video: {
selector: string;
src: string;
mediaTime: number;
box: Box;
frame: Box;
flipX: boolean;
flipY: boolean;
};
}
interface VideoSpec {
box?: Box;
/** Alpha (0-255) of its decoded frame; a frame with any clear pixel is a cutout. */
alpha?: number;
style?: Record<string, string>;
readyState?: number;
}
interface Word {
word: string;
x: number;
width: number;
}
interface Scene {
/** Words of the one text element (#title), all on one line. */
words?: Word[];
titleAttrs?: string;
wrapperAttrs?: string;
titleStyle?: Record<string, string>;
videos?: Record<string, VideoSpec>;
/** Ids of opaque panels. */
panels?: string[];
/** Ids painted at a point, topmost first; "title" is the text. */
stack?: (x: number) => string[];
}
// A 9:16 canvas; every video decodes a 192x108 (16:9) frame.
const CANVAS: Box = { x: 0, y: 0, width: 1080, height: 1920 };
const SOURCE = { width: 192, height: 108 };
const WORD_Y = 800;
const WORD_HEIGHT = 100;
const HELLO = [{ word: "HELLO", x: 400, width: 200 }];
const PANEL_STYLE = { backgroundColor: "rgb(10, 10, 10)" };
const BASE_STYLE: Record<string, string> = {
display: "block",
visibility: "visible",
opacity: "1",
overflow: "visible",
overflowX: "visible",
overflowY: "visible",
backgroundColor: "rgba(0, 0, 0, 0)",
backgroundImage: "none",
objectFit: "fill",
objectPosition: "50% 50%",
fontSize: "36px",
};
function domRect({ x, y, width, height }: Box): DOMRect {
return {
left: x,
top: y,
width,
height,
right: x + width,
bottom: y + height,
x,
y,
toJSON() {
return this;
},
} as DOMRect;
}
const byId = (id: string) => document.getElementById(id) as HTMLElement;
function installVideos(videos: Record<string, VideoSpec>): void {
for (const [id, spec] of Object.entries(videos)) {
const video = byId(id);
Object.defineProperties(video, {
videoWidth: { configurable: true, value: SOURCE.width },
videoHeight: { configurable: true, value: SOURCE.height },
readyState: { configurable: true, value: spec.readyState ?? 4 },
seeking: { configurable: true, value: false },
currentTime: { configurable: true, value: 2.5 },
currentSrc: { configurable: true, value: `http://127.0.0.1:4000/takes/${id}.mp4` },
});
vi.spyOn(video, "getBoundingClientRect").mockReturnValue(domRect(spec.box ?? CANVAS));
}
}
// Range boxes for #title: every word for a whole-node range, one word for a word's own range.
function installTitleRanges(words: Word[]): void {
const boxes = (picked: Word[]) =>
picked.map(({ x, width }) =>
domRect({ x, y: WORD_Y, width, height: WORD_HEIGHT }),
) as unknown as DOMRectList;
vi.spyOn(document, "createRange").mockImplementation(() => {
let node: Node | null = null;
let range: [number, number] | null = null;
return {
selectNodeContents(selected: Node) {
node = selected;
},
setStart(selected: Node, offset: number) {
node = selected;
range = [offset, offset];
},
setEnd(_selected: Node, offset: number) {
range = [range ? range[0] : 0, offset];
},
getClientRects() {
if (node?.parentElement?.id !== "title") return boxes([]);
if (!range) return boxes(words);
const text = (node.textContent ?? "").slice(range[0], range[1]);
return boxes(words.filter(({ word }) => word === text));
},
detach() {},
} as unknown as Range;
});
}
// Each video's decoded frame is opaque except, for a cutout, one clear corner pixel.
function installFrames(videos: Record<string, VideoSpec>): ReturnType<typeof vi.fn> {
let drawn = "";
const getImageData = vi.fn((_x: number, _y: number, width: number, height: number) => {
const data = new Uint8ClampedArray(width * height * 4).fill(255);
data[3] = videos[drawn]?.alpha ?? 255;
return { data };
});
// getContext's overloads resolve to the WebGPU one; the audit only asks for "2d".
const getContextSpy = vi.spyOn(HTMLCanvasElement.prototype, "getContext") as unknown as {
mockReturnValue(value: CanvasRenderingContext2D): void;
};
getContextSpy.mockReturnValue({
clearRect() {},
drawImage(source: Element) {
drawn = source.id;
},
getImageData,
} as unknown as CanvasRenderingContext2D);
return getImageData;
}
function installScene(scene: Scene): { getImageData: ReturnType<typeof vi.fn> } {
const words = scene.words ?? HELLO;
const videos = scene.videos ?? { take: {} };
const panels = scene.panels ?? [];
const tags = [
...Object.keys(videos).map((id) => `<video id="${id}"></video>`),
...panels.map((id) => `<div id="${id}"></div>`),
];
document.body.innerHTML = `
<div id="root" data-composition-id="main" data-width="1080" data-height="1920">
${tags.join("")}
<div id="wrapper" ${scene.wrapperAttrs ?? ""}><span id="title" ${scene.titleAttrs ?? ""}>${words
.map(({ word }) => word)
.join(" ")}</span></div>
</div>`;
for (const element of Array.from(document.querySelectorAll("*"))) {
vi.spyOn(element, "getBoundingClientRect").mockReturnValue(domRect(CANVAS));
}
installVideos(videos);
const styles = new Map<Element, Record<string, string>>([
[byId("title"), scene.titleStyle ?? {}],
...panels.map((id): [Element, Record<string, string>] => [byId(id), PANEL_STYLE]),
...Object.entries(videos).map(([id, spec]): [Element, Record<string, string>] => [
byId(id),
spec.style ?? {},
]),
]);
vi.spyOn(window, "getComputedStyle").mockImplementation(
(element) => ({ ...BASE_STYLE, ...styles.get(element) }) as unknown as CSSStyleDeclaration,
);
installTitleRanges(words);
const stack = scene.stack ?? (() => ["title", ...Object.keys(videos)]);
Object.defineProperty(document, "elementsFromPoint", {
configurable: true,
value: (x: number) => [...stack(x).map(byId), byId("wrapper"), byId("root")],
});
const getImageData = installFrames(videos);
window.eval(script);
return { getImageData };
}
interface AuditWindow {
__hyperframesLayoutAudit(options: { time: number; tolerance: number }): unknown[];
__hyperframesTextOverVideo(options: { time: number }): Candidate[];
}
function textOverVideo(time = 1, askedTime = time): Candidate[] {
const page = window as unknown as AuditWindow;
page.__hyperframesLayoutAudit({ time, tolerance: 2 });
return page.__hyperframesTextOverVideo({ time: askedTime });
}
const CUTOUT: VideoSpec = { alpha: 0 };
describe("layout-audit.browser text in front of a <video>", () => {
afterEach(() => {
vi.restoreAllMocks();
document.body.innerHTML = "";
Reflect.deleteProperty(document, "elementsFromPoint");
Reflect.deleteProperty(window, "__hyperframesLayoutAudit");
Reflect.deleteProperty(window, "__hyperframesTextOverVideo");
});
it("reports text painted above an opaque video with its word boxes and the video's state", () => {
installScene({ titleAttrs: 'data-layer-id="promo"' });
expect(textOverVideo()).toEqual([
{
selector: "#title",
dataAttributes: { "data-layer-id": "promo" },
sourceFile: "index.html",
text: "HELLO",
rect: expect.objectContaining({ left: 400, top: 800, width: 200, height: 100 }),
wordRects: [{ x: 400, y: 800, width: 200, height: 100 }],
video: {
selector: "#take",
src: "http://127.0.0.1:4000/takes/take.mp4",
mediaTime: 2.5,
box: CANVAS,
frame: CANVAS,
flipX: false,
flipY: false,
},
},
]);
});
it("gives one box per word", () => {
installScene({
words: [
{ word: "15%", x: 300, width: 150 },
{ word: "OFF", x: 500, width: 180 },
],
});
expect(textOverVideo()[0]?.wordRects).toEqual([
{ x: 300, y: 800, width: 150, height: 100 },
{ x: 500, y: 800, width: 180, height: 100 },
]);
});
it.each([
// 16:9 into 9:16: cover scales to the height and spills sideways, contain letterboxes.
["cover", "50% 50%", { x: -1166.67, y: 0, width: 3413.33, height: 1920 }],
["cover", "0% 50%", { x: 0, y: 0, width: 3413.33, height: 1920 }],
["contain", "50% 50%", { x: 0, y: 656.25, width: 1080, height: 607.5 }],
["contain", "50% 0%", { x: 0, y: 0, width: 1080, height: 607.5 }],
["none", "50% 50%", { x: 444, y: 906, width: 192, height: 108 }],
])("places the frame for object-fit %s at %s", (objectFit, objectPosition, frame) => {
installScene({ videos: { take: { style: { objectFit, objectPosition } } } });
expect(textOverVideo()[0]?.video).toMatchObject({ box: CANVAS, frame });
});
it("cuts the video's box to the canvas and keeps the frame whole", () => {
const box = { x: -100, y: 400, width: 1280, height: 720 };
installScene({ videos: { take: { box } } });
expect(textOverVideo()[0]?.video).toMatchObject({
box: { x: 0, y: 400, width: 1080, height: 720 },
frame: box,
});
});
it("marks a mirrored video", () => {
installScene({ videos: { take: { style: { transform: "matrix(-1, 0, 0, 1, 0, 0)" } } } });
expect(textOverVideo()[0]?.video).toMatchObject({ flipX: true, flipY: false, frame: CANVAS });
});
it("skips a rotated video instead of guessing where its frame lands", () => {
installScene({
videos: { take: { style: { transform: "matrix(0.866, 0.5, -0.5, 0.866, 0, 0)" } } },
});
expect(textOverVideo()).toEqual([]);
});
it("skips a video whose frame is not decoded", () => {
installScene({ videos: { take: { readyState: 1 } } });
expect(textOverVideo()).toEqual([]);
});
it.each([
["the element", { titleAttrs: "data-layout-allow-on-face" }],
["an ancestor", { wrapperAttrs: "data-layout-allow-on-face" }],
])("honours data-layout-allow-on-face on %s", (_where, attrs) => {
installScene(attrs);
expect(textOverVideo()).toEqual([]);
});
it("looks through a transparent cutout beneath the text to the take under it", () => {
installScene({
videos: { cutout: CUTOUT, take: {} },
stack: () => ["title", "cutout", "take"],
});
expect(textOverVideo().map((candidate) => candidate.video.selector)).toEqual(["#take"]);
});
it("does not take a lone cutout for a video to look for faces in", () => {
installScene({ videos: { cutout: CUTOUT } });
expect(textOverVideo()).toEqual([]);
});
it("leaves text behind a cutout to text_occluded", () => {
installScene({
videos: { cutout: CUTOUT, take: {} },
stack: () => ["cutout", "title", "take"],
});
expect(textOverVideo()).toEqual([]);
});
it("does not report text that is beneath the video", () => {
installScene({ stack: () => ["take", "title"] });
expect(textOverVideo()).toEqual([]);
});
it("does not report text that sits on an opaque panel over the video", () => {
installScene({ panels: ["card"], stack: () => ["title", "card", "take"] });
expect(textOverVideo()).toEqual([]);
});
it("counts only the stretch of a word that is in front of the video", () => {
installScene({
panels: ["card"],
stack: (x) => (x < 500 ? ["title", "card", "take"] : ["title", "take"]),
});
expect(textOverVideo()[0]?.wordRects).toEqual([{ x: 500, y: 800, width: 100, height: 100 }]);
});
it("splits a word across two videos side by side", () => {
installScene({
videos: {
left: { box: { x: 0, y: 0, width: 500, height: 1920 } },
right: { box: { x: 500, y: 0, width: 580, height: 1920 } },
},
stack: (x) => ["title", x < 500 ? "left" : "right"],
});
expect(
textOverVideo().map((candidate) => [candidate.video.selector, candidate.wordRects]),
).toEqual([
["#left", [{ x: 400, y: 800, width: 100, height: 100 }]],
["#right", [{ x: 500, y: 800, width: 100, height: 100 }]],
]);
});
it("does not report text that is still fading in", () => {
installScene({ titleStyle: { opacity: "0.3" } });
expect(textOverVideo()).toEqual([]);
});
it("costs nothing without a video: no candidates and no frame read", () => {
const { getImageData } = installScene({ videos: {}, stack: () => ["title"] });
expect(textOverVideo()).toEqual([]);
expect(getImageData).not.toHaveBeenCalled();
});
it("answers only for the time its audit ran at", () => {
installScene({});
expect(textOverVideo(1, 2)).toEqual([]);
});
});
@@ -430,17 +430,32 @@ describe("layout-audit.browser text behind a <video>", () => {
expect(occlusionOf(runAudit(), "ship")).toBeUndefined();
});
it("reads each probed video's frame once per sample and leaves videos under no text alone", () => {
it("reads each probed video's frame once per sample, above the text or beneath it", () => {
const covering = installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });
runAudit();
expect(covering.getImageData).toHaveBeenCalledTimes(1);
runAudit();
expect(covering.getImageData).toHaveBeenCalledTimes(2);
// A video beneath text is read too: text_over_face looks through a cutout to the take under it.
vi.restoreAllMocks();
const beneath = installVideoScene({ text: SHIP, video: { overlapsText: false } });
runAudit();
expect(beneath.getImageData).not.toHaveBeenCalled();
expect(beneath.getImageData).toHaveBeenCalledTimes(1);
});
it("leaves a video no text touches alone", () => {
const apart = installVideoScene({
text: SHIP,
sizedText: true,
video: { overlapsText: false },
});
const video = document.getElementById("clip") as HTMLVideoElement;
vi.spyOn(video, "getBoundingClientRect").mockReturnValue(
domRect({ left: 0, top: 0, width: 400, height: 300 }),
);
runAudit();
expect(apart.getImageData).not.toHaveBeenCalled();
});
it.each([
@@ -15,9 +15,23 @@ interface AuditIssue {
coveredFraction?: number;
}
interface Box {
x: number;
y: number;
width: number;
height: number;
}
interface TextOverVideo {
selector: string;
wordRects: Box[];
video: { selector: string; src: string; mediaTime: number; box: Box; frame: Box };
}
declare global {
interface Window {
__hyperframesLayoutAudit(options: { time: number; tolerance: number }): AuditIssue[];
__hyperframesTextOverVideo(options: { time: number }): TextOverVideo[];
}
}
@@ -162,7 +176,7 @@ describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in C
if (clipDir) rmSync(clipDir, { recursive: true, force: true });
});
async function audit(html: string, seekTo: number | null = 0.25): Promise<AuditIssue[]> {
async function audited<T>(html: string, seekTo: number | null, read: () => T): Promise<T> {
const path = `/page-${pages.size}.html`;
pages.set(path, html);
const tab = await browser.newPage();
@@ -171,22 +185,31 @@ describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in C
await tab.goto(`${origin}${path}`);
if (seekTo !== null) {
await tab.evaluate(async (time) => {
const video = document.querySelector("video");
if (!video) return;
if (video.readyState < 2) {
await new Promise((resolve) => video.addEventListener("loadeddata", resolve));
for (const video of Array.from(document.querySelectorAll("video"))) {
if (video.readyState < 2) {
await new Promise((resolve) => video.addEventListener("loadeddata", resolve));
}
video.currentTime = time;
await new Promise((resolve) => video.addEventListener("seeked", resolve));
}
video.currentTime = time;
await new Promise((resolve) => video.addEventListener("seeked", resolve));
}, seekTo);
}
await tab.addScriptTag({ content: script });
return await tab.evaluate(() => window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 }));
return (await tab.evaluate(read)) as T;
} finally {
await tab.close();
}
}
const audit = (html: string, seekTo: number | null = 0.25): Promise<AuditIssue[]> =>
audited(html, seekTo, () => window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 }));
const textOverVideo = (html: string): Promise<TextOverVideo[]> =>
audited(html, 0.25, () => {
window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 });
return window.__hyperframesTextOverVideo({ time: 1 });
});
const occlusionOf = (issues: AuditIssue[], id: string) =>
issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);
@@ -278,6 +301,65 @@ describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in C
expect(occlusionOf(issues, "waived")).toBeUndefined();
});
// text_over_face's page half: a title at z-index 3 over a take, with a cutout above or beneath it.
const title = (extra = "") =>
`<span id="title" ${extra} style="position:absolute;left:100px;top:150px;z-index:3;margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">SHIP IT</span>`;
const take = (style = "") =>
`<video id="take" src="/opaque.webm" muted playsinline style="position:absolute;left:0;top:0;width:640px;height:360px;z-index:1;${style}"></video>`;
const cutoutAt = (zIndex: number) =>
`<video id="cutout" src="/cutout.webm" muted playsinline style="position:absolute;left:0;top:0;width:640px;height:360px;z-index:${zIndex}"></video>`;
const stage = (body: string) =>
`<!doctype html><body style="margin:0"><div data-composition-id="main" data-width="640" data-height="360" style="position:relative;width:640px;height:360px;overflow:hidden;background:#111">${body}</div></body>`;
it("reports a title in front of a take with its word boxes and the take's clock", async () => {
const found = await textOverVideo(stage(take() + title()));
expect(found).toHaveLength(1);
const { selector, video, wordRects } = found[0]!;
expect(selector).toBe("#title");
expect(video).toMatchObject({
selector: "#take",
src: `${origin}/opaque.webm`,
mediaTime: 0.25,
box: { x: 0, y: 0, width: 640, height: 360 },
frame: { x: 0, y: 0, width: 640, height: 360 },
});
expect(wordRects).toHaveLength(2);
const [ship, it] = [wordRects[0]!, wordRects[1]!];
expect(ship.x).toBe(100);
expect(it.x).toBeGreaterThan(ship.x + ship.width);
expect(it).toMatchObject({ y: ship.y, height: ship.height });
});
it("places the frame of a take that object-fit crops", async () => {
// 320x180 covering a 220x360 box anchored top right: 640x360 painted, 420 px cut off the left.
const cropped = take("width:220px;object-fit:cover;object-position:right top");
const found = await textOverVideo(stage(cropped + title()));
expect(found).toHaveLength(1);
const { video, wordRects } = found[0]!;
expect(video).toMatchObject({
box: { x: 0, y: 0, width: 220, height: 360 },
frame: { x: -420, y: 0, width: 640, height: 360 },
});
// The title runs past the box onto the stage: only what is in front of the take is reported.
expect(wordRects.length).toBeGreaterThan(0);
for (const piece of wordRects) expect(piece.x + piece.width / 2).toBeLessThan(220);
});
it("looks through a cutout between the title and the take", async () => {
const found = await textOverVideo(stage(take() + cutoutAt(2) + title()));
expect(found.map((candidate) => candidate.video.selector)).toEqual(["#take"]);
});
it("leaves a title behind a cutout to text_occluded", async () => {
expect(await textOverVideo(stage(take() + cutoutAt(4) + title()))).toEqual([]);
});
it("skips a rotated take and honours data-layout-allow-on-face", async () => {
expect(await textOverVideo(stage(take("transform:rotate(8deg)") + title()))).toEqual([]);
const waived = stage(take() + title("data-layout-allow-on-face"));
expect(await textOverVideo(waived)).toEqual([]);
});
it("falls back to opaque with an info note when no frame has been decoded", async () => {
const issues = await audit(
page(videoTag("cutout.webm", 'preload="none"'), word("unknown", 480, "SHIP")),
+2 -1
View File
@@ -24,6 +24,7 @@ const subCommands = {
ink: () => import("./measure/ink.js").then((m) => m.default),
look: () => import("./measure/look.js").then((m) => m.default),
face: () => import("./measure/face.js").then((m) => m.default),
hands: () => import("./measure/hands.js").then((m) => m.default),
sync: () => import("./measure/sync.js").then((m) => m.default),
};
@@ -33,7 +34,7 @@ export default defineCommand({
meta: {
name: "measure",
description:
"Measure a video file as JSON: clock, cuts, events, frames, crops, eases, motion energy, ink, look, faces, lip sync",
"Measure a video file as JSON: clock, cuts, events, frames, crops, eases, motion energy, ink, look, faces, hands, lip sync",
},
subCommands,
});
+6 -2
View File
@@ -1,6 +1,7 @@
import { defineCommand } from "../../utils/defineCommand.js";
import type { Example } from "../_examples.js";
import { jsonArg, runMeasure, videoArg, window } from "../../measure/command.js";
import { continuedShots } from "../../measure/continues.js";
import { classifiedCuts, signatures } from "../../measure/cuts.js";
import { probeVideo, round, videoClip } from "../../measure/media.js";
@@ -13,7 +14,7 @@ export default defineCommand({
meta: {
name: "cuts",
description:
"Hard cuts from the frame-to-frame difference (scene scores miss some), and the shots between them; light flashes, whips and fast moves are listed apart as not_cuts",
"Hard cuts from the frame-to-frame difference (scene scores miss some), and the shots between them, a shot that resumes an earlier camera setup naming that shot as continues; light flashes, whips and fast moves are listed apart as not_cuts",
},
args: {
video: videoArg,
@@ -26,8 +27,10 @@ export default defineCommand({
const info = probeVideo(String(args.video));
const [start, end] = window(info, args.start, args.end);
const clip = videoClip(info.path, start, end - start, info);
const all = classifiedCuts(signatures(clip.input, clip.filters).frames);
const frames = signatures(clip.input, clip.filters).frames;
const all = classifiedCuts(frames);
const found = all.filter((c) => !c.kind);
const continues = continuedShots(frames, found);
const f0 = Math.ceil(start * info.fps - 1e-6);
const not_cuts = all.flatMap((c) =>
c.kind ? [{ t: round((f0 + c.frame) / info.fps, 3), kind: c.kind }] : [],
@@ -50,6 +53,7 @@ export default defineCommand({
start: round(edges[i]!, 3),
end: round(e, 3),
seconds: round(e - edges[i]!, 2),
...(continues[i] !== undefined ? { continues: continues[i] } : {}),
})),
};
});
@@ -0,0 +1,80 @@
import { defineCommand } from "../../utils/defineCommand.js";
import type { Example } from "../_examples.js";
import { jsonArg, num, runMeasure, videoArg, window } from "../../measure/command.js";
import { HAND_CHOICES, HAND_POINTS, followHand } from "../../measure/hands.js";
import { Fix, probeVideo } from "../../measure/media.js";
export const examples: Example[] = [
[
"Where the palm is through 3-5 s, for a title that rides it",
"hyperframes measure hands ref.mp4 --start 3 --end 5",
],
[
"The right hand's pinch at the video's own frame rate",
"hyperframes measure hands ref.mp4 --start 1.2 --end 2.4 --point pinch --which right --fps 30",
],
];
function choice<T extends string>(
value: unknown,
flag: string,
choices: readonly T[],
fallback: T,
): T {
if (value === undefined) return fallback;
const hit = choices.find((c) => c === value);
if (hit === undefined)
throw new Fix(
`${flag} must be one of ${choices.join(", ")}, got "${String(value)}"`,
`${flag} ${fallback}`,
);
return hit;
}
export default defineCommand({
meta: {
name: "hands",
description:
"Where a hand is through a stretch of up to 15 s: its palm, index fingertip or pinch in each sampled frame, from MediaPipe's hand landmarker",
},
args: {
video: videoArg,
start: { type: "string", alias: ["from"], description: "First second (default 0)" },
end: { type: "string", alias: ["to"], description: "Last second (default: the end)" },
point: {
type: "string",
description:
"palm (the wrist and knuckles' middle, with the hand's size), index_tip, or pinch (midway between the thumb and index tips). Default palm",
},
which: {
type: "string",
description:
"The hand followed: largest, or the largest one labelled left or right. Default largest",
},
fps: { type: "string", description: "Frames sampled per second, 1 to 30 (default 15)" },
json: jsonArg,
},
async run({ args }) {
await runMeasure(async () => {
const info = probeVideo(String(args.video));
const [start, end] = window(info, args.start, args.end);
const followed = await followHand(
info,
start,
end,
{
point: choice(args.point, "--point", HAND_POINTS, "palm"),
which: choice(args.which, "--which", HAND_CHOICES, "largest"),
fps: num(args.fps, "--fps", 15),
},
(message) => console.error(message),
);
return {
video: info.path,
...followed,
reading:
"points[i] is the hand's point at window[0] + i * step seconds: x, y as shares of the frame from its top left (palm adds the hand's width and height); null is a frame with no such hand, never filled in",
};
});
},
});
+171
View File
@@ -0,0 +1,171 @@
import { describe, expect, it } from "vitest";
import { continuedShots, sameSetup, setupLikeness } from "./continues.js";
import { findCuts, SIG_H, SIG_W, type Cut } from "./cuts.js";
/** A room: 40 boxes of seeded sizes and grey levels on a mid-grey wall, so every part of the frame has edges. */
function room(seed: number): Uint8Array {
let state = seed;
const next = () => {
state = (state * 1664525 + 1013904223) >>> 0;
return state / 2 ** 32;
};
const f = new Uint8Array(SIG_W * SIG_H).fill(120);
for (let k = 0; k < 40; k++) {
const [x0, y0] = [Math.floor(next() * SIG_W), Math.floor(next() * SIG_H)];
const [w, h, level] = [4 + Math.floor(next() * 14), 3 + Math.floor(next() * 10), next() * 255];
for (let y = y0; y < Math.min(SIG_H, y0 + h); y++)
for (let x = x0; x < Math.min(SIG_W, x0 + w); x++) f[y * SIG_W + x] = level;
}
return f;
}
/** `f` with a box of `level` drawn over it: a person standing at x, or a title. */
function over(f: Uint8Array, x0: number, y0: number, w: number, h: number, level = 20) {
const out = Uint8Array.from(f);
for (let y = y0; y < y0 + h; y++) for (let x = x0; x < x0 + w; x++) out[y * SIG_W + x] = level;
return out;
}
const person = (f: Uint8Array, x: number) => over(f, x, 14, 6, 16);
/** `f` scaled by `s` about the centre and shifted right by `dx` pixels, edges held. */
function moved(f: Uint8Array, s: number, dx = 0): Uint8Array {
const out = new Uint8Array(f.length);
const [cx, cy] = [(SIG_W - 1) / 2, (SIG_H - 1) / 2];
for (let y = 0; y < SIG_H; y++)
for (let x = 0; x < SIG_W; x++) {
const sx = Math.min(SIG_W - 1, Math.max(0, Math.round((x - cx - dx) / s + cx)));
const sy = Math.min(SIG_H - 1, Math.max(0, Math.round((y - cy) / s + cy)));
out[y * SIG_W + x] = f[sy * SIG_W + sx]!;
}
return out;
}
const hold = (f: Uint8Array, n: number) => Array.from({ length: n }, () => f);
const cut = (frame: number, frames = 1): Cut => ({ frame, frames, change: 40 });
describe("sameSetup", () => {
const wide = room(1);
const other = room(2);
it("knows a setup again when the person in it has moved and the exposure has drifted", () => {
const later = person(wide, 41).map((v) => Math.min(255, v * 1.1 + 6));
expect(sameSetup([person(wide, 9)], [later])).toBe(true);
expect(setupLikeness([wide], [wide])).toEqual({ textured: 48, agreeing: 48 });
});
it("tells two scenes apart, with or without the same person in the same place", () => {
expect(sameSetup([wide], [other])).toBe(false);
expect(sameSetup([person(wide, 25)], [person(other, 25)])).toBe(false);
});
it("allows a title over part of the frame", () => {
expect(sameSetup([over(wide, 16, 0, 32, 6, 250)], [person(wide, 33)])).toBe(true);
});
it("allows a pixel of drift, and refuses a punch-in on the same scene", () => {
expect(sameSetup([wide], [moved(wide, 1, 1)])).toBe(true);
expect(sameSetup([wide], [moved(wide, 1.3)])).toBe(false);
});
it("refuses a flat card against anything, itself included: there is nothing to compare", () => {
const card = new Uint8Array(SIG_W * SIG_H).fill(160);
expect(sameSetup([card], [wide])).toBe(false);
expect(sameSetup([wide], [card])).toBe(false);
expect(sameSetup([card], [card])).toBe(false);
});
it("refuses a split screen that keeps its top half and changes its bottom half", () => {
const split = (bottom: Uint8Array) =>
wide.map((v, i) => (i < (SIG_W * SIG_H) / 2 ? v : bottom[i]!));
// The kept half agrees, less a cell at the seam: about half the frame, well under the bar.
expect(setupLikeness([split(other)], [split(room(3))])).toEqual({ textured: 48, agreeing: 23 });
expect(sameSetup([split(other)], [split(room(3))])).toBe(false);
expect(sameSetup([split(other)], [split(other)])).toBe(true);
});
it("reads the median of three frames, so one flashed frame at a cut does not count", () => {
const flash = new Uint8Array(SIG_W * SIG_H).fill(250);
expect(sameSetup([wide, wide, wide], [flash, wide, wide])).toBe(true);
expect(sameSetup([wide], [flash])).toBe(false);
});
});
describe("continuedShots", () => {
const wide = room(1);
const insert = room(2);
const third = room(3);
it("names the shot a take resumes after an insert, the person further along", () => {
// A wide shot, a close-up cut into it, and the wide shot again.
const frames = [
...hold(person(wide, 9), 20),
...hold(insert, 10),
...hold(person(wide, 41), 20),
];
const cuts = findCuts(frames);
expect(cuts.map((c) => c.frame)).toEqual([20, 30]);
expect(continuedShots(frames, cuts)).toEqual([undefined, undefined, 0]);
});
it("names nothing among three different shots, or for a single shot", () => {
const frames = [...hold(wide, 10), ...hold(insert, 10), ...hold(third, 10)];
expect(continuedShots(frames, [cut(10), cut(20)])).toEqual([undefined, undefined, undefined]);
expect(continuedShots(hold(wide, 10), [])).toEqual([undefined]);
});
it("names the most recent shot of the setup, where the take was left", () => {
// Wide, a jump cut in the wide take, an insert, the wide take again.
const frames = [
...hold(person(wide, 9), 10),
...hold(person(wide, 25), 10),
...hold(insert, 10),
...hold(person(wide, 41), 10),
];
expect(continuedShots(frames, [cut(10), cut(20), cut(30)])).toEqual([
undefined,
undefined,
undefined,
1,
]);
});
it("leaves a jump cut alone: a shot that matches the one before it resumes nothing", () => {
// Wide, an insert, wide again, then a jump cut in that wide take.
const frames = [
...hold(person(wide, 9), 10),
...hold(insert, 10),
...hold(person(wide, 25), 10),
...hold(person(wide, 41), 10),
];
expect(continuedShots(frames, [cut(10), cut(20), cut(30)])).toEqual([
undefined,
undefined,
0,
undefined,
]);
});
it("follows two setups cut back and forth", () => {
const frames = [wide, insert, wide, insert, wide].flatMap((f) => hold(f, 10));
expect(continuedShots(frames, [cut(10), cut(20), cut(30), cut(40)])).toEqual([
undefined,
undefined,
0,
1,
2,
]);
});
it("compares where a shot was left with where the next one starts, not its first frames", () => {
// The wide shot pans away to another view before the insert: the take was left there, so nothing resumes.
const frames = [...hold(wide, 10), ...hold(third, 10), ...hold(insert, 10), ...hold(wide, 10)];
expect(continuedShots(frames, [cut(20), cut(30)])).toEqual([undefined, undefined, undefined]);
});
it("starts a shot after its transition: a three-frame dissolve back into the take still resumes it", () => {
const mix = (k: number) => wide.map((v, i) => (insert[i]! * (3 - k) + v * k) / 3);
const frames = [...hold(wide, 10), ...hold(insert, 10), mix(1), mix(2), ...hold(wide, 10)];
expect(continuedShots(frames, [cut(10), cut(20, 3)])).toEqual([undefined, undefined, 0]);
});
});
+138
View File
@@ -0,0 +1,138 @@
// Which shots resume an earlier camera setup, by the scene's edges: a person moves and exposure drifts, walls do not.
import { SIG_H, SIG_W, type Cut } from "./cuts.js";
/** Cells across and down, 8 x 6 pixels each: small enough that a person or a title spoils only a few. */
const COLS = 8;
const ROWS = 6;
const CELL_W = SIG_W / COLS;
const CELL_H = SIG_H / ROWS;
/** Frames taken from each end of a shot, a median a pixel: one blended or flashed frame at a cut drops out. */
const PLATE_FRAMES = 3;
/** Mean squared gradient a pixel for a cell to say anything: under it a wall or a sky is flat, and noise decides. */
const TEXTURED = 15;
/** The weaker picture's share of the stronger one's edges in a cell: a flat card matches nothing. */
const BOTH_TEXTURED = 0.25;
/** Gradient correlation for a cell to be the same piece of scene: a still camera gives 0.9 and up. */
const CELL_AGREES = 0.75;
/** Textured cells out of all: with fewer, the picture has too little structure to tell two setups apart. */
const MIN_TEXTURED = 0.35;
/** Agreeing cells out of the textured ones. Other scenes read under 0.6 and the same room framed tighter or
* wider 0.6 to 0.7: a wrong match joins two takes, a miss costs none. */
const MIN_AGREEING = 0.75;
/** Pixels (of 64 x 36) the picture may have slid between the two ends: a hand-held take never returns exactly. */
const DRIFT = 1;
const WORTH_SLIDING = MIN_AGREEING / 2;
export interface Likeness {
textured: number;
agreeing: number;
}
interface Plate {
gx: Float32Array;
gy: Float32Array;
}
const median3 = (a: number, b: number, c: number) =>
Math.max(Math.min(a, b), Math.min(Math.max(a, b), c));
function meanOf3x3Around(px: Float32Array, x: number, y: number): number {
let [sum, n] = [0, 0];
for (let v = Math.max(0, y - 1); v <= Math.min(SIG_H - 1, y + 1); v++)
for (let u = Math.max(0, x - 1); u <= Math.min(SIG_W - 1, x + 1); u++) {
sum += px[v * SIG_W + u]!;
n++;
}
return sum / n;
}
function plate(frames: Uint8Array[]): Plate {
const gx = new Float32Array(SIG_W * SIG_H);
const gy = new Float32Array(SIG_W * SIG_H);
const a = frames[0];
if (!a) return { gx, gy };
const b = frames[1] ?? a;
const c = frames[2] ?? b;
const still = gx.map((_, i) => median3(a[i]!, b[i]!, c[i]!));
const soft = still.map((_, i) => meanOf3x3Around(still, i % SIG_W, Math.floor(i / SIG_W)));
for (let y = 1; y < SIG_H - 1; y++)
for (let x = 1; x < SIG_W - 1; x++) {
const i = y * SIG_W + x;
gx[i] = soft[i + 1]! - soft[i - 1]!;
gy[i] = soft[i + SIG_W]! - soft[i - SIG_W]!;
}
return { gx, gy };
}
function cellAgrees(
a: Plate,
b: Plate,
[col, row]: [number, number],
[dx, dy]: [number, number],
): boolean | undefined {
let [aa, bb, ab, n] = [0, 0, 0, 0];
const yEnd = Math.min((row + 1) * CELL_H, SIG_H - dy);
const xEnd = Math.min((col + 1) * CELL_W, SIG_W - dx);
for (let y = Math.max(row * CELL_H, -dy); y < yEnd; y++)
for (let x = Math.max(col * CELL_W, -dx); x < xEnd; x++) {
const i = y * SIG_W + x;
const j = i + dy * SIG_W + dx;
aa += a.gx[i]! ** 2 + a.gy[i]! ** 2;
bb += b.gx[j]! ** 2 + b.gy[j]! ** 2;
ab += a.gx[i]! * b.gx[j]! + a.gy[i]! * b.gy[j]!;
n++;
}
if (Math.max(aa, bb) <= TEXTURED * n) return undefined;
return (
Math.min(aa, bb) >= BOTH_TEXTURED * Math.max(aa, bb) && ab >= CELL_AGREES * Math.sqrt(aa * bb)
);
}
function likenessAt(a: Plate, b: Plate, dx: number, dy: number): Likeness {
let [textured, agreeing] = [0, 0];
for (let cell = 0; cell < COLS * ROWS; cell++) {
const agrees = cellAgrees(a, b, [cell % COLS, Math.floor(cell / COLS)], [dx, dy]);
if (agrees === undefined) continue;
textured++;
if (agrees) agreeing++;
}
return { textured, agreeing };
}
function likeness(a: Plate, b: Plate): Likeness {
let best = likenessAt(a, b, 0, 0);
if (best.agreeing < WORTH_SLIDING * best.textured) return best;
for (let dy = -DRIFT; dy <= DRIFT; dy++)
for (let dx = -DRIFT; dx <= DRIFT; dx++) {
const at = dx || dy ? likenessAt(a, b, dx, dy) : best;
if (at.agreeing > best.agreeing) best = at;
}
return best;
}
const isSame = ({ textured, agreeing }: Likeness) =>
textured >= MIN_TEXTURED * COLS * ROWS && agreeing >= MIN_AGREEING * textured;
export function setupLikeness(a: Uint8Array[], b: Uint8Array[]): Likeness {
return likeness(plate(a), plate(b));
}
export function sameSetup(a: Uint8Array[], b: Uint8Array[]): boolean {
return isSame(setupLikeness(a, b));
}
/**
* For each shot between `cuts`, the index of the earlier shot whose camera setup it resumes after at least one other
* shot. A match with the shot right before is a jump cut, which the cut already tells; the most recent match wins.
*/
export function continuedShots(frames: Uint8Array[], cuts: Cut[]): (number | undefined)[] {
const starts = [0, ...cuts.map((c) => c.frame + c.frames - 1)];
const ends = [...cuts.map((c) => c.frame), frames.length];
const heads = starts.map((s, i) => plate(frames.slice(s, Math.min(s + PLATE_FRAMES, ends[i]!))));
const tails = ends.map((e, i) => plate(frames.slice(Math.max(starts[i]!, e - PLATE_FRAMES), e)));
return heads.map((head, i) => {
if (i < 2 || isSame(likeness(tails[i - 1]!, head))) return undefined;
for (let j = i - 2; j >= 0; j--) if (isSame(likeness(tails[j]!, head))) return j;
return undefined;
});
}
+13 -6
View File
@@ -86,10 +86,6 @@ export function readSamples(info: VideoInfo, ks: number[], w: number, h: number)
});
}
/**
* Each sampled frame's faces as [x, y, w, h] shares of the frame. `minFace` is the smallest face looked for as a
* share of the frame's height (default FACE_MIN; 0 looks down to the cascades' 20 px, for crowds).
*/
export async function faceBoxes(
info: VideoInfo,
start: number,
@@ -97,8 +93,19 @@ export async function faceBoxes(
samples: number,
opts: { minFace?: number } = {},
) {
const cv = await loadOpenCv((m) => console.error(m));
const ks = sampleFrames(info.fps, start, end, samples);
return faceBoxesAt(info, sampleFrames(info.fps, start, end, samples), opts);
}
/**
* The faces of frames `ks` (ascending indices) as [x, y, w, h] shares of the frame. `minFace` is the smallest face
* looked for as a share of the frame's height (default FACE_MIN; 0 looks down to the cascades' 20 px, for crowds).
*/
export async function faceBoxesAt(
info: VideoInfo,
ks: number[],
opts: { minFace?: number; signal?: AbortSignal } = {},
) {
const cv = await loadOpenCv((m) => console.error(m), opts.signal);
const w = Math.min(WORK_W, info.width);
const h =
w === info.width
+170
View File
@@ -0,0 +1,170 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterAll, describe, expect, it } from "vitest";
import { findFFmpeg, findFFprobe } from "../browser/ffmpeg.js";
import {
MEDIAPIPE_DIR,
followHand,
handFrames,
handPath,
pageFile,
pickHand,
placeHand,
readHands,
type Hand,
} from "./hands.js";
import { Fix, probeVideo, runFfmpeg } from "./media.js";
const media = !!findFFmpeg() && !!findFFprobe();
const dir = mkdtempSync(join(tmpdir(), "hf-hands-test-"));
afterAll(() => rmSync(dir, { recursive: true, force: true }));
/** A hand whose 21 landmarks all sit at (x, y), then the ones in `moved` put elsewhere. */
function hand(side: string, x: number, y: number, moved: Record<number, [number, number]> = {}) {
const points = Array.from({ length: 21 }, (_, i): [number, number] => moved[i] ?? [x, y]);
return { side, points } satisfies Hand;
}
/** A hand that spans `w` x `h` from (x, y): its wrist at one corner, its little finger's tip at the other. */
const sized = (side: string, x: number, y: number, w: number, h: number) =>
hand(side, x, y, { 20: [x + w, y + h] });
describe("placeHand", () => {
it("puts the palm at the mean of the wrist and the four knuckles, with the hand's size", () => {
const h = hand("Left", 0.5, 0.5, {
0: [0.4, 0.6],
5: [0.42, 0.5],
9: [0.44, 0.48],
13: [0.46, 0.5],
17: [0.48, 0.52],
// A fingertip far from the palm widens the hand and leaves the palm where it is.
12: [0.6, 0.2],
});
expect(placeHand(h, "palm")).toEqual([0.44, 0.52, 0.2, 0.4]);
});
it("gives the index fingertip, and the pinch midway between it and the thumb's tip", () => {
const h = hand("Right", 0.5, 0.5, { 4: [0.3, 0.4], 8: [0.35, 0.3] });
expect(placeHand(h, "index_tip")).toEqual([0.35, 0.3]);
expect(placeHand(h, "pinch")).toEqual([0.325, 0.35]);
});
it("rounds to four places", () => {
const h = hand("Right", 0.5, 0.5, { 8: [0.123449, 0.98766] });
expect(placeHand(h, "index_tip")).toEqual([0.1234, 0.9877]);
});
});
describe("pickHand", () => {
const small = sized("Left", 0.1, 0.1, 0.05, 0.05);
const big = sized("Right", 0.5, 0.5, 0.2, 0.2);
const bigger = sized("Right", 0.3, 0.3, 0.3, 0.2);
it("follows the largest hand by default", () => {
expect(pickHand([small, big], "largest")).toBe(big);
expect(pickHand([big, small], "largest")).toBe(big);
});
it("follows the largest hand with the side asked for, however large the other is", () => {
expect(pickHand([small, big], "left")).toBe(small);
expect(pickHand([big, bigger, small], "right")).toBe(bigger);
});
it("keeps the first of two hands the same size", () => {
// Quarters, so both areas are the same number exactly.
const [one, twin] = [
sized("Right", 0.5, 0.25, 0.25, 0.25),
sized("Left", 0.25, 0.5, 0.25, 0.25),
];
expect(pickHand([one, twin], "largest")).toBe(one);
expect(pickHand([twin, one], "largest")).toBe(twin);
});
it("finds none when no hand, or none on that side, is in the frame", () => {
expect(pickHand([], "largest")).toBeNull();
expect(pickHand([big], "left")).toBeNull();
});
});
describe("handPath", () => {
it("leaves a frame without the hand null between the frames that have it", () => {
const left = (x: number) => hand("Left", x, 0.5, { 8: [x, 0.25] });
const frames = [[left(0.2)], [], [hand("Right", 0.9, 0.9)], [left(0.4)]];
expect(handPath(frames, "index_tip", "left")).toEqual([[0.2, 0.25], null, null, [0.4, 0.25]]);
expect(handPath(frames, "index_tip", "largest")[2]).toEqual([0.9, 0.9]);
});
});
describe("readHands", () => {
it("takes what the page answers", () => {
const answer: unknown = JSON.parse(JSON.stringify([hand("Left", 0.5, 0.5)]));
expect(readHands(answer)).toEqual([hand("Left", 0.5, 0.5)]);
expect(readHands([])).toEqual([]);
});
it("refuses an answer that is not hands of 21 landmarks", () => {
const short = { side: "Left", points: [[0.5, 0.5]] };
const unlabelled = { points: hand("Left", 0.5, 0.5).points };
const holed = { side: "Left", points: hand("Left", 0.5, 0.5).points.map(() => [0.5, null]) };
for (const bad of [null, "hands", [short], [unlabelled], [holed]])
expect(() => readHands(bad)).toThrow(/shape/);
});
});
describe("pageFile", () => {
it("answers the page, the pinned files and the stretch's frames, and nothing else", () => {
expect(pageFile("/", dir)?.text).toContain("HandLandmarker");
expect(pageFile("/vision_wasm_internal.wasm", dir)).toEqual({
type: "application/wasm",
file: join(MEDIAPIPE_DIR, "vision_wasm_internal.wasm"),
});
expect(pageFile("/frames/t00012.jpg", dir)).toEqual({
type: "image/jpeg",
file: join(dir, "t00012.jpg"),
});
for (const path of ["/frames/../t00012.jpg", "/frames/notes.txt", "/etc/passwd", "/frames/"])
expect(pageFile(path, dir)).toBeNull();
});
});
describe.skipIf(!media)("handFrames", () => {
it("writes the stretch at the asked rate, 480 wide, one frame a step from its start", () => {
const video = join(dir, "clip.mp4");
runFfmpeg([
"-y",
"-f",
"lavfi",
"-i",
"testsrc=size=320x240:rate=30:duration=2",
"-pix_fmt",
"yuv420p",
video,
]);
const out = mkdtempSync(join(dir, "frames-"));
const names = handFrames(probeVideo(video), 0.5, 1.5, 15, out);
expect(names).toHaveLength(15);
expect(names[0]).toBe("t00001.jpg");
expect(names.at(-1)).toBe("t00015.jpg");
const first = probeVideo(join(out, "t00001.jpg"));
expect([first.width, first.height]).toEqual([480, 360]);
});
it("refuses a stretch longer than it follows before reading a frame", async () => {
const video = join(dir, "long.mp4");
runFfmpeg([
"-y",
"-f",
"lavfi",
"-i",
"color=c=gray:size=64x64:rate=5:duration=16",
"-pix_fmt",
"yuv420p",
video,
]);
const opts = { point: "palm", which: "largest", fps: 15 } as const;
const refused = followHand(probeVideo(video), 0, 16, opts);
await expect(refused).rejects.toBeInstanceOf(Fix);
await expect(refused).rejects.toMatchObject({ fix: "--start 0 --end 15" });
});
});
+310
View File
@@ -0,0 +1,310 @@
// Hand keypoints through a stretch of video: where the palm, the index
// fingertip or the pinch (thumb and index tips together) is, frame by frame, from MediaPipe's hand landmarker. A hand
// closes, turns and spreads its fingers, which loses a colour or patch tracker; landmarks follow the hand itself.
//
// The landmarker is MediaPipe's own WebAssembly build with the model the Python package uses, so both give the same
// numbers. It draws each frame through a WebGL canvas, which Node has not, so it runs in the CLI's headless Chrome on a
// page whose every request is answered from disk. Neither is shipped in the CLI: the first use downloads them, pinned
// by size and sha256, into ~/.cache/hyperframes/mediapipe/<version>/.
import { mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { HTTPRequest, Page } from "puppeteer-core";
import { trustedCacheRoot } from "@hyperframes/core/cache-dir";
import { Fix, round, runFfmpeg, type VideoInfo } from "./media.js";
import { ensurePinnedFiles, type PinnedFile } from "./pinnedFiles.js";
const VERSION = "1.0.0";
const CODE_ROOT = trustedCacheRoot();
export const MEDIAPIPE_DIR = join(CODE_ROOT, "mediapipe", VERSION);
interface Served extends PinnedFile {
type: string;
}
// The code is @mediapipe/tasks-vision 1.0.0 and the model is MediaPipe's float16 hand landmarker (both Apache-2.0).
const TASKS_VISION = `https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@${VERSION}`;
const FILES: readonly Served[] = [
{
name: "vision_bundle.mjs",
bytes: 154_839,
sha256: "54ad628b478853dadb23261c1c7d14cfe3760b363ccf62b8c02ef73e5c7fb87a",
url: `${TASKS_VISION}/vision_bundle.mjs`,
type: "text/javascript",
},
{
name: "vision_wasm_internal.js",
bytes: 323_343,
sha256: "e2da04fe2ab9f58eb2aeb8f8724c80ad8919d31d7f5fb99057565879f05f51ca",
url: `${TASKS_VISION}/wasm/vision_wasm_internal.js`,
type: "text/javascript",
},
{
name: "vision_wasm_internal.wasm",
bytes: 11_532_084,
sha256: "c266c65f3789bea74ab12dd6609edc96c68655da4540d6097d0732d8be80b742",
url: `${TASKS_VISION}/wasm/vision_wasm_internal.wasm`,
type: "application/wasm",
},
{
name: "hand_landmarker.task",
bytes: 7_819_105,
sha256: "fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1",
url: "https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task",
type: "application/octet-stream",
},
];
export const HAND_POINTS = ["palm", "index_tip", "pinch"] as const;
export type HandPoint = (typeof HAND_POINTS)[number];
export const HAND_CHOICES = ["largest", "left", "right"] as const;
export type WhichHand = (typeof HAND_CHOICES)[number];
// The wrist and the four knuckles: the palm's middle, which stays put while the fingers move.
const PALM = [0, 5, 9, 13, 17];
const THUMB_TIP = 4;
const INDEX_TIP = 8;
const LANDMARKS = 21;
const MAX_HANDS = 2;
const WIDTH = 480;
const MAX_FPS = 30;
const MAX_SECONDS = 15;
type Xy = [x: number, y: number];
/** One hand in one frame: MediaPipe's side label and its 21 landmarks as shares of the frame. */
export interface Hand {
side: string;
points: Xy[];
}
const span = (v: number[]) => Math.max(...v) - Math.min(...v);
const xs = (hand: Hand) => hand.points.map((p) => p[0]);
const ys = (hand: Hand) => hand.points.map((p) => p[1]);
function handArea(hand: Hand): number {
return span(xs(hand)) * span(ys(hand));
}
/** The point as a track point: the palm with the hand's size [x, y, w, h], a fingertip or a pinch as [x, y]. */
export function placeHand(hand: Hand, point: HandPoint): number[] {
if (point === "palm") {
const mean = (axis: 0 | 1) =>
PALM.reduce((sum, i) => sum + hand.points[i]![axis], 0) / PALM.length;
return [mean(0), mean(1), span(xs(hand)), span(ys(hand))].map((v) => round(v, 4));
}
const tip = hand.points[INDEX_TIP]!;
if (point === "index_tip") return [round(tip[0], 4), round(tip[1], 4)];
const thumb = hand.points[THUMB_TIP]!;
return [round((thumb[0] + tip[0]) / 2, 4), round((thumb[1] + tip[1]) / 2, 4)];
}
/** The hand to follow: the largest (nearest the lens), or the largest one MediaPipe labels left or right. */
export function pickHand(hands: Hand[], which: WhichHand): Hand | null {
const side = which === "left" ? "Left" : "Right";
return hands
.filter((hand) => which === "largest" || hand.side === side)
.reduce<Hand | null>(
(best, hand) => (best && handArea(best) >= handArea(hand) ? best : hand),
null,
);
}
/** The followed hand's point in each frame; a frame without that hand stays null. */
export function handPath(frames: Hand[][], point: HandPoint, which: WhichHand) {
return frames.map((hands) => {
const hand = pickHand(hands, which);
return hand ? placeHand(hand, point) : null;
});
}
const isXy = (v: unknown): v is Xy =>
Array.isArray(v) && v.length === 2 && v.every((n) => typeof n === "number" && Number.isFinite(n));
const isHand = (v: unknown): v is Hand =>
typeof v === "object" &&
v !== null &&
"side" in v &&
typeof v.side === "string" &&
"points" in v &&
Array.isArray(v.points) &&
v.points.length === LANDMARKS &&
v.points.every(isXy);
/** What the page answered for one frame, checked: it crosses from Chrome as plain JSON. */
export function readHands(answer: unknown): Hand[] {
if (!Array.isArray(answer) || !answer.every(isHand))
throw new Error("the hand landmarker answered in a shape this CLI does not read");
return answer;
}
const FRAME_NAME = /^t\d{5}\.jpg$/;
/**
* The stretch as numbered JPEGs in `dir`, `fps` a second and WIDTH wide. Returns their names in order.
*/
export function handFrames(
info: VideoInfo,
start: number,
end: number,
fps: number,
dir: string,
): string[] {
runFfmpeg([
"-y",
"-ss",
start.toFixed(3),
"-i",
info.path,
"-t",
(end - start).toFixed(3),
"-vf",
`fps=${fps}:round=near,scale=${WIDTH}:-2`,
"-q:v",
"3",
join(dir, "t%05d.jpg"),
]);
return readdirSync(dir)
.filter((name) => FRAME_NAME.test(name))
.sort();
}
// No such host exists: the page is answered from disk before Chrome looks one up.
const ORIGIN = "https://hands.hyperframes.invalid";
// The CPU delegate, as in the Python, so a machine's GPU cannot move the numbers.
const PAGE = `<!doctype html><meta charset="utf-8"><title>hands</title><script type="module">
import { HandLandmarker } from "/vision_bundle.mjs";
let landmarker;
window.loadHands = async () => {
landmarker = await HandLandmarker.createFromOptions(
{ wasmLoaderPath: "/vision_wasm_internal.js", wasmBinaryPath: "/vision_wasm_internal.wasm" },
{
baseOptions: { modelAssetPath: "/hand_landmarker.task", delegate: "CPU" },
runningMode: "VIDEO",
numHands: ${MAX_HANDS},
},
);
};
window.handsIn = async (name, ms) => {
const file = await (await fetch("/frames/" + name)).blob();
const picture = await createImageBitmap(file, { colorSpaceConversion: "none", premultiplyAlpha: "none" });
try {
const found = landmarker.detectForVideo(picture, ms);
return found.landmarks.map((marks, i) => ({
side: found.handedness[i]?.[0]?.categoryName ?? "",
points: marks.map((mark) => [mark.x, mark.y]),
}));
} finally {
picture.close();
}
};
</script>`;
/** The body and type for a path on the page's origin, or null for one it has no business asking for. */
export function pageFile(
path: string,
frames: string,
): { type: string; file?: string; text?: string } | null {
if (path === "/") return { type: "text/html", text: PAGE };
const served = FILES.find((f) => path === `/${f.name}`);
if (served) return { type: served.type, file: join(MEDIAPIPE_DIR, served.name) };
const frame = path.startsWith("/frames/") ? path.slice("/frames/".length) : "";
return FRAME_NAME.test(frame) ? { type: "image/jpeg", file: join(frames, frame) } : null;
}
function answer(request: HTTPRequest, frames: string): void {
const url = new URL(request.url());
const hit = url.origin === ORIGIN ? pageFile(url.pathname, frames) : null;
const sent = hit
? request.respond({
status: 200,
contentType: hit.type,
body: hit.file ? readFileSync(hit.file) : (hit.text ?? ""),
})
: request.abort();
// A request still open when the page closes has nobody left to answer.
sent.catch(() => {});
}
async function openHandsPage(page: Page, frames: string): Promise<void> {
await page.setRequestInterception(true);
page.on("request", (request) => answer(request, frames));
try {
await page.goto(`${ORIGIN}/`);
await page.evaluate("loadHands()");
} catch (err) {
throw new Fix(
`the hand landmarker could not start in Chrome (${err instanceof Error ? err.message.split("\n")[0] : String(err)})`,
"run `hyperframes doctor` to check Chrome, then run the command again",
);
}
}
/** The landmarker run over the frames in order as a video, so it carries a hand from one frame to the next. */
async function detectHands(
dir: string,
frames: string[],
fps: number,
onStatus?: (message: string) => void,
): Promise<Hand[][]> {
await ensurePinnedFiles(MEDIAPIPE_DIR, CODE_ROOT, FILES, onStatus);
const { ensureBrowser } = await import("../browser/manager.js");
const { closeManagedBrowser, launchManagedBrowser } = await import("../browser/launch.js");
const { buildChromeArgs } = await import("@hyperframes/engine");
const puppeteer = await import("puppeteer-core");
const chrome = await ensureBrowser();
const browser = await launchManagedBrowser(puppeteer.default, {
headless: true,
executablePath: chrome.executablePath,
// Software WebGL: the frames only pass through it, and it is there on a machine with no GPU.
args: buildChromeArgs(
{ width: WIDTH, height: WIDTH, captureMode: "screenshot" },
{ browserGpuMode: "software" },
),
});
try {
const page = await browser.newPage();
await openHandsPage(page, dir);
const found: Hand[][] = [];
for (const [k, name] of frames.entries()) {
const ms = Math.trunc((k * 1000) / fps);
found.push(readHands(await page.evaluate(`handsIn(${JSON.stringify(name)}, ${ms})`)));
}
return found;
} finally {
await closeManagedBrowser(browser);
}
}
/** The hand's point from `start` to `end`, one entry every `step` seconds, and how many frames lost it. */
export async function followHand(
info: VideoInfo,
start: number,
end: number,
opts: { point: HandPoint; which: WhichHand; fps: number },
onStatus?: (message: string) => void,
) {
if (end - start > MAX_SECONDS)
throw new Fix(
`hands follows up to ${MAX_SECONDS} seconds at a time, not ${round(end - start, 3)}`,
`--start ${start} --end ${round(start + MAX_SECONDS, 3)}`,
);
const fps = Math.max(1, Math.min(MAX_FPS, opts.fps));
const dir = mkdtempSync(join(tmpdir(), "hf-hands-"));
try {
const frames = handFrames(info, start, end, fps, dir);
const found = await detectHands(dir, frames, fps, onStatus);
const points = handPath(found, opts.point, opts.which);
return {
window: [round(start, 3), round(end, 3)],
step: round(1 / fps, 4),
point: opts.point,
which: opts.which,
points,
lost: points.filter((p) => p === null).length,
};
} finally {
rmSync(dir, { recursive: true, force: true });
}
}
+13 -1
View File
@@ -5,6 +5,7 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest";
const ownedByAnotherAccount = vi.hoisted(() => new Set<string>());
const downloads = vi.hoisted(() => [] as string[]);
const signals = vi.hoisted(() => [] as (AbortSignal | undefined)[]);
vi.mock("node:fs", async (importOriginal) => {
const fs = await importOriginal<typeof import("node:fs")>();
const lstatSync = ((path: string, options?: object) => {
@@ -16,8 +17,9 @@ vi.mock("node:fs", async (importOriginal) => {
return { ...fs, lstatSync };
});
vi.mock("../cloud/download.js", () => ({
downloadToFile: async (url: string) => {
downloadToFile: async (url: string, _dest: string, options?: { signal?: AbortSignal }) => {
downloads.push(url);
signals.push(options?.signal);
},
}));
@@ -30,6 +32,7 @@ beforeEach(() => {
afterEach(() => {
ownedByAnotherAccount.clear();
downloads.length = 0;
signals.length = 0;
vi.unstubAllEnvs();
rmSync(cache, { recursive: true, force: true });
});
@@ -46,3 +49,12 @@ it.skipIf(!process.getuid)(
expect(downloads).toEqual([]);
},
);
it("hands the caller's signal to the download, so a check that stops ends it", async () => {
const { loadOpenCv } = await import("./opencv.js");
const stop = new AbortController();
await expect(loadOpenCv(undefined, stop.signal)).rejects.toThrow(
"did not match its pinned size and sha256",
);
expect(signals).toEqual([stop.signal]);
});
+10 -4
View File
@@ -125,7 +125,10 @@ const isCv = (v: unknown): v is Cv =>
/** Every pinned file present and verified; the missing ones fetched. */
// fallow-ignore-next-line complexity
async function ensureOpenCv(onStatus?: (message: string) => void): Promise<void> {
async function ensureOpenCv(
onStatus?: (message: string) => void,
signal?: AbortSignal,
): Promise<void> {
const missing: Pinned[] = [];
for (const f of FILES) {
const file = join(OPENCV_DIR, f.name);
@@ -137,7 +140,7 @@ async function ensureOpenCv(onStatus?: (message: string) => void): Promise<void>
onStatus?.(`Downloading ${f.name} (${Math.round(f.bytes / 1e5) / 10} MB, once)...`);
const temp = join(OPENCV_DIR, `${f.name}.part-${process.pid}`);
try {
await downloadToFile(f.url, temp);
await downloadToFile(f.url, temp, { signal });
if (!(await verifies(temp, f)))
throw new Fix(
`${f.name} did not match its pinned size and sha256`,
@@ -161,9 +164,12 @@ async function ensureOpenCv(onStatus?: (message: string) => void): Promise<void>
let loaded: Promise<Cv> | null = null;
/** opencv.js, ready to call, with the face models in its file system. Loaded once per process. */
export function loadOpenCv(onStatus?: (message: string) => void): Promise<Cv> {
export function loadOpenCv(
onStatus?: (message: string) => void,
signal?: AbortSignal,
): Promise<Cv> {
loaded ??= (async () => {
await ensureOpenCv(onStatus);
await ensureOpenCv(onStatus, signal);
const mod: unknown = createRequire(import.meta.url)(join(OPENCV_DIR, "opencv.js"));
if (typeof mod !== "object" || mod === null) throw new Error("opencv.js did not load");
// Wait on its init hook, never on the module: it is a thenable that does not settle under await.
+53
View File
@@ -0,0 +1,53 @@
// Files a measuring tool needs but the CLI does not ship: fetched on first use into a folder under the trusted cache
// root, each one checked against its pinned size and sha256 before it is used, and the folder sealed.
import { mkdirSync, renameSync, rmSync } from "node:fs";
import { join } from "node:path";
import { foreignWritablePath, sealCacheInstall } from "@hyperframes/core/cache-dir";
import { downloadToFile } from "../cloud/download.js";
import { verifies, type ModelFile } from "../whisper/modelFile.js";
import { Fix } from "./media.js";
export interface PinnedFile extends ModelFile {
url: string;
}
async function fetchPinned(dir: string, file: PinnedFile): Promise<void> {
const temp = join(dir, `${file.name}.part-${process.pid}`);
try {
await downloadToFile(file.url, temp).catch((err: unknown) => {
throw new Fix(
`could not download ${file.name} (${err instanceof Error ? err.message : String(err)})`,
"check the network and run the command again",
);
});
if (!(await verifies(temp, file)))
throw new Fix(
`${file.name} did not match its pinned size and sha256`,
"run the command again to retry the download",
);
renameSync(temp, join(dir, file.name));
} finally {
rmSync(temp, { force: true });
}
}
/** Every file present in `dir` (under the cache `root`) and verified; the missing ones fetched. */
export async function ensurePinnedFiles(
dir: string,
root: string,
files: readonly PinnedFile[],
onStatus?: (message: string) => void,
): Promise<void> {
const wanted: PinnedFile[] = [];
for (const file of files) {
const path = join(dir, file.name);
if (foreignWritablePath(path, root) || !(await verifies(path, file))) wanted.push(file);
}
if (!wanted.length) return;
mkdirSync(dir, { recursive: true, mode: 0o700 });
for (const file of wanted) {
onStatus?.(`Downloading ${file.name} (${Math.round(file.bytes / 1e5) / 10} MB, once)...`);
await fetchPinned(dir, file);
}
sealCacheInstall(dir, root);
}
+33 -1
View File
@@ -2,7 +2,7 @@
// fallow-ignore-file code-duplication
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { DEFAULT_CONFIG } from "@hyperframes/engine";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import os, { tmpdir } from "node:os";
import { join } from "node:path";
import {
@@ -21,6 +21,7 @@ import {
preResolveHostileMediaProxies,
dropFailedProxyEchoes,
runBrowserCheck,
servedProjectFile,
collectSeekClock,
mergeHelperDrafts,
parseLayoutIssue,
@@ -1378,6 +1379,37 @@ describe("mergeHelperDrafts", () => {
});
});
describe("servedProjectFile", () => {
const page = "http://127.0.0.1:4000/";
let root: string;
let projectDir: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "hf-served-file-"));
projectDir = join(root, "project");
mkdirSync(join(projectDir, "takes"), { recursive: true });
writeFileSync(join(projectDir, "takes", "my take.mp4"), "");
writeFileSync(join(root, "outside.mp4"), "");
});
afterEach(() => rmSync(root, { recursive: true, force: true }));
it("resolves a page-origin source to its project file, proxied or not", () => {
const file = join(projectDir, "takes", "my take.mp4");
expect(servedProjectFile(`${page}takes/my%20take.mp4`, page, projectDir)).toBe(file);
expect(servedProjectFile(`${page}takes/my%20take.mp4?hf-proxy=h264`, page, projectDir)).toBe(
file,
);
});
it("gives no file for a remote source, a missing file or a path outside the project", () => {
const remote = "https://cdn.example.com/takes/my%20take.mp4";
expect(servedProjectFile(remote, page, projectDir)).toBeNull();
expect(servedProjectFile(`${page}takes/gone.mp4`, page, projectDir)).toBeNull();
expect(servedProjectFile(`${page}..%2Foutside.mp4`, page, projectDir)).toBeNull();
expect(servedProjectFile("", page, projectDir)).toBeNull();
});
});
describe("dropFailedProxyEchoes", () => {
it("keeps one finding for a failed proxy and drops the preview's repeats of it", () => {
const at = (code: string, message: string, url?: string) =>
+82 -5
View File
@@ -7,7 +7,7 @@ import { collectMotionBlurDrafts, dropMotionBlurConsoleEcho } from "./checkMotio
import { checkSnapshotDirectory } from "./checkSnapshotDirectory.js";
import { existsSync, writeFileSync } from "node:fs";
import os from "node:os";
import { join, resolve } from "node:path";
import { isAbsolute, join, relative, resolve } from "node:path";
import type { Browser, Page } from "puppeteer-core";
import {
AUDIT_SEEK_OPTIONS,
@@ -92,6 +92,7 @@ import type {
RotationSample,
RunAuditGrid,
SeekClock,
TextOverVideoCandidate,
} from "./checkTypes.js";
import type { ProjectDir } from "./project.js";
@@ -474,7 +475,7 @@ export async function runBrowserCheck(
let chromeBrowser: import("puppeteer-core").Browser | undefined;
const helperSession =
samplingBrowserCount(options) > 1
? startHelperSession(html, server.url, options, deadline)
? startHelperSession(html, server.url, options, deadline, project.dir)
: undefined;
const helperOpen = helperSession?.opened;
const helpers = helperOpen?.then(
@@ -527,7 +528,7 @@ export async function runBrowserCheck(
const blur = await collectMotionBlurDrafts(page);
drafts.push(...blur);
const sampleLog = sampleWindow(drafts);
const driver = createPageDriver(page, clock, sampleLog);
const driver = createPageDriver(page, clock, sampleLog, project.dir);
const result = await runGrid(
driver,
options,
@@ -536,6 +537,7 @@ export async function runBrowserCheck(
deadline?.enter,
logCheckProgress,
deadline?.keepPartial,
deadline?.signal,
);
helperSession?.stop();
const helper = await helperOpen?.catch(() => undefined);
@@ -611,10 +613,11 @@ function startHelperSession(
url: string,
options: CheckOptions,
deadline: CheckDeadline | undefined,
projectDir: string,
) {
const abort = new AbortController();
let browser: Browser | undefined;
const opened = openHelperSession(html, url, options, deadline, {
const opened = openHelperSession(html, url, options, deadline, projectDir, {
signal: deadline ? AbortSignal.any([deadline.signal, abort.signal]) : abort.signal,
onBrowser: (launched) => {
browser = launched;
@@ -637,6 +640,7 @@ async function openHelperSession(
url: string,
options: CheckOptions,
deadline: CheckDeadline | undefined,
projectDir: string,
owner: { signal: AbortSignal; onBrowser: (browser: Browser) => void },
): Promise<HelperSession> {
const sampleLog = sampleWindow([]);
@@ -655,7 +659,7 @@ async function openHelperSession(
});
await waitForPreferredSeekTarget(session.page);
return {
driver: createPageDriver(session.page, clock, sampleLog),
driver: createPageDriver(session.page, clock, sampleLog, projectDir),
sampleLog,
};
}
@@ -1089,6 +1093,7 @@ export function createPageDriver(
page: Page,
clock: RuntimeClock = createRuntimeClock(),
sampleLog?: SampleWindow,
projectDir?: string,
): CheckAuditDriver {
return failWhenPageDies(page, {
initialize: (contrast) => injectAuditScripts(page, contrast),
@@ -1142,6 +1147,9 @@ export function createPageDriver(
collectSeekClock: () => collectSeekClock(page),
collectRotationSample: (time) => collectRotationSample(page, time),
collectOffPivotRotationSample: (time) => collectOffPivotRotationSample(page, time),
...(projectDir === undefined
? {}
: { collectTextOverVideo: (time) => collectTextOverVideo(page, time, projectDir) }),
collectGeometryCandidates: (time, request) => collectGeometryCandidates(page, time, request),
collectMotionFrame: (time, selectors, scopes) =>
collectMotionFrame(page, time, selectors, scopes),
@@ -1572,6 +1580,75 @@ function parseRotationSample(value: unknown, time: number): RotationSample[] {
return [{ time, selector, cx, cy, w, h, angle }];
}
async function collectTextOverVideo(
page: Page,
time: number,
projectDir: string,
): Promise<TextOverVideoCandidate[]> {
const raw = await page.evaluate(
(options: { time: number }) => {
const collect = Reflect.get(window, "__hyperframesTextOverVideo");
if (typeof collect !== "function") return [];
const result = Reflect.apply(collect, window, [options]);
return Array.isArray(result) ? result : [];
},
{ time },
);
return raw.flatMap((value) => parseTextOverVideo(value, time, page.url(), projectDir));
}
/** The project file a page-origin URL is served from; null for a remote source or a path outside the project. */
export function servedProjectFile(src: string, pageUrl: string, projectDir: string): string | null {
const url = parseUrl(src);
const path = decodedPath(url);
if (!url || !path || url.origin !== parseUrl(pageUrl)?.origin) return null;
const file = resolve(projectDir, `.${path}`);
const inside = relative(projectDir, file);
if (!inside || inside.startsWith("..") || isAbsolute(inside)) return null;
return existsSync(file) ? file : null;
}
function parsePlacedVideo(
value: Record<string, unknown> | null,
pageUrl: string,
projectDir: string,
): TextOverVideoCandidate["video"] | null {
if (!value) return null;
const selector = stringValue(value, "selector");
const file = servedProjectFile(stringValue(value, "src") ?? "", pageUrl, projectDir);
const mediaTime = numberValue(value, "mediaTime");
const box = parseBbox(Reflect.get(value, "box"));
const frame = parseBbox(Reflect.get(value, "frame"));
if (!selector || !file || mediaTime === null || !box || !frame) return null;
return {
selector,
file,
mediaTime,
box,
frame,
flipX: booleanValue(value, "flipX") === true,
flipY: booleanValue(value, "flipY") === true,
};
}
function parseTextOverVideo(
value: unknown,
time: number,
pageUrl: string,
projectDir: string,
): TextOverVideoCandidate[] {
if (!isRecord(value)) return [];
const rect = parseRect(Reflect.get(value, "rect"));
const anchor = rect ? parseGeometryAnchor(value, rect, time) : null;
const video = parsePlacedVideo(recordField(value, "video"), pageUrl, projectDir);
const rawWords: unknown = Reflect.get(value, "wordRects");
const wordRects = Array.isArray(rawWords)
? rawWords.flatMap((word) => parseBbox(word) ?? [])
: [];
if (!rect || !anchor || !video || wordRects.length === 0) return [];
return [{ ...anchor, text: stringValue(value, "text") ?? "", rect, wordRects, video }];
}
async function collectOffPivotRotationSample(page: Page, time: number): Promise<OffPivotFrame> {
const raw = await evaluateSampler(page, "__hyperframesOffPivotRotationSample");
return { time, samples: raw.flatMap(parseOffPivotRotationSample) };
@@ -0,0 +1,173 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { DEFAULT_CHECK_OPTIONS, runAuditGrid } from "./checkPipeline.js";
import type { CheckAuditDriver, TextOverVideoCandidate } from "./checkTypes.js";
import type { VideoInfo } from "../measure/media.js";
const mocks = vi.hoisted(() => ({
faceBoxesAt: vi.fn(),
probeVideo: vi.fn(),
}));
vi.mock("../measure/face.js", () => ({ faceBoxesAt: mocks.faceBoxesAt }));
vi.mock("../measure/media.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../measure/media.js")>()),
probeVideo: mocks.probeVideo,
}));
const SAMPLES = [1, 4, 7];
const OPTIONS = { ...DEFAULT_CHECK_OPTIONS, at: SAMPLES, contrast: false, craft: false };
function driver(overrides: Partial<CheckAuditDriver> = {}): CheckAuditDriver {
let state = 0;
return {
initialize: async () => undefined,
getDuration: async () => 9,
hasNoTimelineDeclaration: async () => false,
getTransitionBoundaries: async () => [],
getClipEdges: async () => [],
getCanvas: async () => ({ width: 1080, height: 1920 }),
findAmbiguousSelectors: async () => [],
seek: async () => undefined,
seekGeometry: async () => undefined,
collectLayout: async () => [],
collectOverlap: async () => [],
collectOverlapGrid: async (times) => times.map(() => []),
collectLayoutGeometry: async () => `state-${state++}`,
captureFrame: async () => "frame",
framesDiffer: async () => true,
collectSeekClock: async () => [{ id: 1, time: 0, done: false }],
collectRotationSample: async () => [],
collectOffPivotRotationSample: async (time) => ({ time, samples: [] }),
collectGeometryCandidates: async () => [],
collectMotionFrame: async (time) => ({ time, data: {}, liveness: {} }),
anchorMotionIssues: async () => [],
collectContrast: async () => ({ entries: [] }),
...overrides,
};
}
// A title over the middle of a full-canvas video; the video's clock runs half a second behind.
function title(time: number): TextOverVideoCandidate {
return {
selector: "#title",
dataAttributes: {},
sourceFile: "index.html",
bbox: { x: 340, y: 860, width: 400, height: 200 },
time,
text: "HELLO",
rect: { left: 340, top: 860, right: 740, bottom: 1060, width: 400, height: 200 },
wordRects: [{ x: 340, y: 860, width: 400, height: 200 }],
video: {
selector: "#take",
file: "/project/takes/take.mp4",
mediaTime: time - 0.5,
box: { x: 0, y: 0, width: 1080, height: 1920 },
frame: { x: 0, y: 0, width: 1080, height: 1920 },
flipX: false,
flipY: false,
},
};
}
const INFO = { path: "/project/takes/take.mp4", fps: 24, frames: 216 } as VideoInfo;
// Centred, 216 px square: the title covers its whole width and 196 of its 216 px.
const CENTRE_FACE = [0.4, 0.45, 0.2, 0.1125];
describe("text_over_face in the check grid", () => {
afterEach(() => {
vi.restoreAllMocks();
mocks.faceBoxesAt.mockReset();
mocks.probeVideo.mockReset();
});
it("looks for faces once per video at the frames its samples showed, and reports the cover", async () => {
mocks.probeVideo.mockReturnValue(INFO);
// She faces the camera at the media times 0.5 s and 6.5 s (frames 12 and 156), not at 3.5 s.
mocks.faceBoxesAt.mockImplementation(async (_info: VideoInfo, frames: number[]) =>
frames.map((frame) => ({ frame, t: frame / 24, boxes: frame === 84 ? [] : [CENTRE_FACE] })),
);
const result = await runAuditGrid(
driver({ collectTextOverVideo: async (time) => [title(time)] }),
OPTIONS,
{ kind: "none" },
);
expect(mocks.probeVideo).toHaveBeenCalledTimes(1);
expect(mocks.faceBoxesAt).toHaveBeenCalledTimes(1);
expect(mocks.faceBoxesAt.mock.calls[0]?.[1]).toEqual([12, 84, 156]);
const found = result.layoutIssues.filter((issue) => issue.code === "text_over_face");
expect(found.map((issue) => issue.time).sort()).toEqual([1, 7]);
expect(found[0]).toMatchObject({
severity: "info",
selector: "#title",
containerSelector: "#take",
coveredFraction: 0.91,
faceRect: { x: 432, y: 864, width: 216, height: 216 },
message: '"HELLO" covers 91% of a face in the video beneath it.',
});
expect(result.faceCheck).toEqual({ ran: true, candidates: 3, faces: 2 });
});
it("hands the detector the signal that ends with the check", async () => {
mocks.probeVideo.mockReturnValue(INFO);
mocks.faceBoxesAt.mockResolvedValue([]);
vi.spyOn(console, "error").mockImplementation(() => undefined);
const stop = new AbortController();
const noop = undefined;
await runAuditGrid(
driver({ collectTextOverVideo: async () => [title(1)] }),
{ ...OPTIONS, at: [1] },
{ kind: "none" },
noop,
noop,
noop,
noop,
stop.signal,
);
const given: AbortSignal | undefined = mocks.faceBoxesAt.mock.calls[0]?.[2]?.signal;
expect(given?.aborted).toBe(false);
stop.abort();
expect(given?.aborted).toBe(true);
});
it("does not load the detector when no text is in front of a video", async () => {
const result = await runAuditGrid(driver({ collectTextOverVideo: async () => [] }), OPTIONS, {
kind: "none",
});
expect(mocks.probeVideo).not.toHaveBeenCalled();
expect(mocks.faceBoxesAt).not.toHaveBeenCalled();
expect(result.layoutIssues).toEqual([]);
expect(result.faceCheck).toEqual({ ran: true, candidates: 0, faces: 0 });
});
it("finishes without the finding when OpenCV cannot load, and says so once", async () => {
const stderr = vi.spyOn(console, "error").mockImplementation(() => undefined);
mocks.probeVideo.mockReturnValue(INFO);
mocks.faceBoxesAt.mockRejectedValue(new Error("could not download opencv.js (fetch failed)"));
const result = await runAuditGrid(
driver({ collectTextOverVideo: async (time) => [title(time)] }),
OPTIONS,
{ kind: "none" },
);
expect(result.layoutIssues).toEqual([]);
expect(result.faceCheck).toEqual({
ran: false,
reason: "could not download opencv.js (fetch failed)",
});
const said = stderr.mock.calls.filter(([line]) => String(line).includes("face"));
expect(said).toEqual([
[
"[hyperframes] check: text over a face was not looked for in 1 of 1 video(s): could not download opencv.js (fetch failed)",
],
]);
});
it("says the pass did not run on a driver that cannot report text over video", async () => {
const result = await runAuditGrid(driver(), OPTIONS, { kind: "none" });
expect(result.faceCheck).toEqual({
ran: false,
reason: "this check driver does not report text over video",
});
});
});
+39
View File
@@ -90,12 +90,14 @@ import type {
CraftFinding,
CraftFrame,
CraftResult,
FaceCheck,
GeometryCandidateRequest,
MotionSpecResolution,
OffPivotFrame,
OffPivotRotationSample,
RotationSample,
SeekClock,
TextOverVideoCandidate,
} from "./checkTypes.js";
export type {
@@ -313,6 +315,10 @@ interface GridSamples {
* material-point geometry + dial hub per layout sample; flattened by selector
* to detect off_pivot_rotation. */
indicatorFrames: OffPivotFrame[];
/** Text painted in front of a video at each layout sample; faces are looked for after the run. */
textOverVideo: TextOverVideoCandidate[];
/** No page that sampled could report text over video. */
textOverVideoUnavailable: boolean;
craftFrames: CraftFrame[];
}
@@ -498,6 +504,8 @@ function emptyGridSamples(): GridSamples {
layoutStateSignatures: [],
rotationSamples: [],
indicatorFrames: [],
textOverVideo: [],
textOverVideoUnavailable: false,
craftFrames: [],
};
}
@@ -512,6 +520,8 @@ function appendGridSamples(into: GridSamples, from: GridSamples): void {
into.layoutStateSignatures.push(...from.layoutStateSignatures);
into.rotationSamples.push(...from.rotationSamples);
into.indicatorFrames.push(...from.indicatorFrames);
into.textOverVideo.push(...from.textOverVideo);
into.textOverVideoUnavailable ||= from.textOverVideoUnavailable;
into.craftFrames.push(...from.craftFrames);
}
@@ -549,6 +559,9 @@ async function collectSampleAt(
});
collected.rotationSamples.push(...(await driver.collectRotationSample(time)));
collected.indicatorFrames.push(await driver.collectOffPivotRotationSample(time));
if (driver.collectTextOverVideo) {
collected.textOverVideo.push(...(await driver.collectTextOverVideo(time)));
} else collected.textOverVideoUnavailable = true;
}
if (sets.canvas) {
const geometryIssues = await collectGeometryAt(driver, options, grid, sets, sets.canvas, time);
@@ -1588,6 +1601,24 @@ export function reportingProgress<T extends object>(
});
}
async function textOverFace(
collected: GridSamples,
onFile: () => void,
stop?: AbortSignal,
): Promise<{ issues: AnchoredLayoutIssue[]; faceCheck: FaceCheck }> {
if (collected.textOverVideoUnavailable) {
return {
issues: [],
faceCheck: { ran: false, reason: "this check driver does not report text over video" },
};
}
if (collected.textOverVideo.length === 0) {
return { issues: [], faceCheck: { ran: true, candidates: 0, faces: 0 } };
}
const { findTextOverFace } = await import("./textOverFace.js");
return findTextOverFace(collected.textOverVideo, undefined, onFile, stop);
}
export async function runAuditGrid(
driver: CheckAuditDriver,
options: CheckOptions,
@@ -1596,6 +1627,7 @@ export async function runAuditGrid(
enter: (phase: CheckPhase, detail?: string) => void = () => {},
report: CheckProgress = () => {},
keep: (read: () => Partial<CheckBrowserResult>) => void = () => {},
stop?: AbortSignal,
): Promise<CheckBrowserResult> {
enter("samples");
await driver.initialize(options.contrast);
@@ -1653,6 +1685,7 @@ export async function runAuditGrid(
await analysing.getCanvas(),
);
const offPivotFindings = detectOffPivotRotation(collected.indicatorFrames);
const overFace = await textOverFace(collected, () => enter("analysis", "faces under text"), stop);
const contrast = buildContrastResults(collected.contrastEntries);
report("analyze", 2, 3);
const craft = craftEnabled(analysing, options)
@@ -1682,6 +1715,7 @@ export async function runAuditGrid(
...sweepFindings,
...rotationFindings,
...offPivotFindings,
...overFace.issues,
...motionIssues,
...contrast.findings,
...craft.findings,
@@ -1702,7 +1736,9 @@ export async function runAuditGrid(
...sweepFindings,
...rotationFindings,
...offPivotFindings,
...overFace.issues,
],
faceCheck: overFace.faceCheck,
motionIssues,
motionSampleCount: collected.motionFrames.length,
contrastSamples: grid.contrastSamples,
@@ -2224,6 +2260,8 @@ function filterComponentReport(report: CheckReport, options: CheckOptions): Chec
return report;
}
const FACE_CHECK_NOT_REACHED = "the check stopped before looking for faces under text";
function shapeLayoutSection(
issues: AnchoredLayoutIssue[],
browser: CheckBrowserResult,
@@ -2239,6 +2277,7 @@ function shapeLayoutSection(
tolerance: options.tolerance,
totalIssueCount: shaped.totalIssueCount,
truncated: shaped.truncated,
faceCheck: browser.faceCheck ?? { ran: false, reason: FACE_CHECK_NOT_REACHED },
};
}
+24
View File
@@ -183,6 +183,26 @@ export interface CheckGeometryCandidate extends CheckAnchor {
overflow?: LayoutOverflow;
}
export interface TextOverVideoCandidate extends CheckAnchor {
text: string;
rect: LayoutRect;
wordRects: CheckBbox[];
video: {
selector: string;
file: string;
mediaTime: number;
box: CheckBbox;
frame: CheckBbox;
flipX: boolean;
flipY: boolean;
};
}
/** `candidates`: text-over-video samples examined. `faces`: distinct video frames a face was found in. */
export type FaceCheck =
| { ran: true; candidates: number; faces: number; unreadVideos?: number }
| { ran: false; reason: string };
/** One rotatable element's geometry at a single seeked sample, produced by
* `__hyperframesRotationSample` (layout-audit.browser.js) and accumulated
* across the grid to detect `rotation_pivot_drift`. */
@@ -294,6 +314,7 @@ export interface CheckAuditDriver {
* geometry + resolved dial hub at the current seeked state, as a time-hoisted
* frame envelope. */
collectOffPivotRotationSample(time: number): Promise<OffPivotFrame>;
collectTextOverVideo?(time: number): Promise<TextOverVideoCandidate[]>;
collectGeometryCandidates(
time: number,
request: GeometryCandidateRequest,
@@ -342,6 +363,7 @@ export interface CheckBrowserResult {
transitionSamplesDropped: number;
runtimeFindings: CheckFinding[];
layoutIssues: AnchoredLayoutIssue[];
faceCheck?: FaceCheck;
motionIssues: AnchoredLayoutIssue[];
motionSampleCount: number;
contrastSamples: number[];
@@ -383,6 +405,7 @@ export type RunAuditGrid = (
enter?: (phase: CheckPhase) => void,
report?: CheckProgress,
keep?: (read: () => Partial<CheckBrowserResult>) => void,
stop?: AbortSignal,
) => Promise<CheckBrowserResult>;
export interface CheckSection<T extends CheckFinding = CheckFinding> {
@@ -414,6 +437,7 @@ export interface CheckReport {
tolerance: number;
totalIssueCount: number;
truncated: boolean;
faceCheck: FaceCheck;
};
motion: CheckSection & { enabled: boolean; specPath?: string; samples: number };
contrast: CheckSection<CheckContrastFinding> & {
+4 -2
View File
@@ -19,6 +19,7 @@ export type LayoutIssueCode =
| "content_overlap"
| "visibility_escapes_segment"
| "text_occluded"
| "text_over_face"
| "canvas_covered"
| "video_alpha_unread"
| "text_not_painted"
@@ -66,9 +67,10 @@ export interface LayoutIssue {
rect: LayoutRect;
containerRect?: LayoutRect;
overflow?: LayoutOverflow;
/** `text_occluded` only (0-1): probe-grid share hit by an opaque occluder, or behind a
* transparent video the most covered word's share under its alpha (layout-audit.browser.js). */
/** 0-1. `text_occluded`: probe-grid share hit by an opaque occluder, or behind a transparent video
* the most covered word's share under its alpha. `text_over_face`: the share of `faceRect` covered. */
coveredFraction?: number;
faceRect?: { x: number; y: number; width: number; height: number };
/** `text_occluded` only: the global window the element's own clip is on screen. */
lifetime?: { start: number; end: number };
fixHint?: string;
+295
View File
@@ -0,0 +1,295 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { CheckBbox, TextOverVideoCandidate } from "./checkTypes.js";
import {
FACE_COVER_REPORT,
coveredShare,
faceRectOnCanvas,
findTextOverFace,
textOverFaceIssue,
type FaceFinder,
} from "./textOverFace.js";
const box = (x: number, y: number, width: number, height: number): CheckBbox => ({
x,
y,
width,
height,
});
// A 768x1344 take cover-fitted into a 1080x1920 canvas: the frame is 1097.14 wide and spills
// 8.57 px past each side of the box.
const COVER = {
box: box(0, 0, 1080, 1920),
frame: box(-8.57, 0, 1097.14, 1920),
flipX: false,
flipY: false,
};
function candidate(overrides: Partial<TextOverVideoCandidate> = {}): TextOverVideoCandidate {
return {
selector: "#title",
dataAttributes: { "data-layer-id": "title" },
sourceFile: "index.html",
bbox: box(400, 760, 200, 100),
time: 7,
text: "15% OFF",
rect: { left: 400, top: 760, right: 600, bottom: 860, width: 200, height: 100 },
wordRects: [box(400, 760, 200, 100)],
video: { selector: "#take", file: "/project/take.mp4", mediaTime: 7, ...COVER },
...overrides,
};
}
// On COVER this face lands at x 430, y 768, 175.5 x 192.
const FACE = [0.4, 0.4, 0.16, 0.1];
describe("faceRectOnCanvas", () => {
it("places a face through the frame object-fit painted, not through the element's box", () => {
const rect = faceRectOnCanvas(FACE, COVER);
expect(rect?.x).toBeCloseTo(-8.57 + 0.4 * 1097.14, 2);
expect(rect?.y).toBeCloseTo(768, 2);
expect(rect?.width).toBeCloseTo(0.16 * 1097.14, 2);
expect(rect?.height).toBeCloseTo(192, 2);
});
it("follows a letterboxed frame inside its box", () => {
const contain = { ...COVER, frame: box(0, 656.25, 1080, 607.5) };
expect(faceRectOnCanvas([0.5, 0.5, 0.1, 0.2], contain)).toEqual(box(540, 960, 108, 121.5));
});
it("mirrors the face with the video", () => {
const fill = { ...COVER, frame: box(0, 0, 1000, 1000), box: box(0, 0, 1000, 1000) };
expect(faceRectOnCanvas([0.125, 0.25, 0.25, 0.25], { ...fill, flipX: true })).toEqual(
box(625, 250, 250, 250),
);
expect(faceRectOnCanvas([0.125, 0.25, 0.25, 0.25], { ...fill, flipY: true })).toEqual(
box(125, 500, 250, 250),
);
});
it("cuts a face to what the video's box shows", () => {
const cropped = { ...COVER, frame: box(-100, 0, 1280, 1920) };
// The face spans x -100..156; the box shows 0..156, more than half of it.
expect(faceRectOnCanvas([0, 0.4, 0.2, 0.1], cropped)).toEqual(box(0, 768, 156, 192));
});
it("drops a face the box crops to less than half", () => {
const cropped = { ...COVER, frame: box(-200, 0, 1480, 1920) };
// The face spans x -200..96: only 96 of 296 px are inside the box.
expect(faceRectOnCanvas([0, 0.4, 0.2, 0.1], cropped)).toBeNull();
});
});
describe("coveredShare", () => {
const face = box(100, 100, 200, 200);
it("is the share of the face under the text, not the share of the text on the face", () => {
// A small caption wholly inside a large face hides little of it.
expect(coveredShare([box(150, 150, 100, 40)], face)).toBeCloseTo(0.1, 6);
// A large title that only clips the face hides a lot of it.
expect(coveredShare([box(0, 0, 2000, 200)], face)).toBeCloseTo(0.5, 6);
});
it("counts overlapping word boxes once", () => {
const words = [box(100, 100, 150, 100), box(200, 100, 100, 100), box(200, 150, 100, 100)];
// Their union inside the face is 200x100 plus the 100x50 below its right half.
expect(coveredShare(words, face)).toBeCloseTo((200 * 100 + 100 * 50) / (200 * 200), 6);
});
it("is 0 for text clear of the face and 1 for text over all of it", () => {
expect(coveredShare([box(400, 100, 200, 200)], face)).toBe(0);
expect(coveredShare([box(0, 0, 150, 500), box(150, 0, 500, 500)], face)).toBe(1);
});
});
describe("textOverFaceIssue", () => {
it("builds the finding the contract names", () => {
// The face is 430..605.5 x 768..960; the title covers its top 92 px across 170 px of it.
const [issue] = textOverFaceIssue(candidate(), [FACE]);
expect(issue).toEqual({
code: "text_over_face",
severity: "info",
time: 7,
selector: "#title",
containerSelector: "#take",
text: "15% OFF",
message: '"15% OFF" covers 46% of a face in the video beneath it.',
rect: { left: 400, top: 760, right: 600, bottom: 860, width: 200, height: 100 },
coveredFraction: 0.46,
faceRect: { x: 430.29, y: 768, width: 175.54, height: 192 },
fixHint:
"Move the text off the face, or mark an intended overlap with data-layout-allow-on-face.",
dataAttributes: { "data-layer-id": "title" },
sourceFile: "index.html",
bbox: box(400, 760, 200, 100),
});
});
it("is info however much of the face is covered, so a face found in texture never fails --strict", () => {
const grazing = candidate({ wordRects: [box(400, 760, 200, 40)] });
expect(textOverFaceIssue(grazing, [FACE])[0]).toMatchObject({
severity: "info",
coveredFraction: 0.16,
});
const whole = candidate({ wordRects: [box(0, 0, 1080, 1920)] });
expect(textOverFaceIssue(whole, [FACE])[0]).toMatchObject({
severity: "info",
coveredFraction: 1,
});
});
it("reports nothing under 5% of the face", () => {
expect(FACE_COVER_REPORT).toBe(0.05);
const touching = candidate({ wordRects: [box(400, 760, 200, 15)] });
expect(textOverFaceIssue(touching, [FACE])).toEqual([]);
});
it("reports the face the text covers most", () => {
const clear = [0.8, 0.1, 0.1, 0.05];
const [issue] = textOverFaceIssue(candidate(), [clear, FACE]);
expect(issue?.faceRect?.x).toBeCloseTo(430.29, 2);
});
it("reports nothing without a face", () => {
expect(textOverFaceIssue(candidate(), [])).toEqual([]);
});
it("shortens a long line in the message only", () => {
const text = "A very long caption line that runs well past forty characters";
const [issue] = textOverFaceIssue(candidate({ text }), [FACE]);
expect(issue?.message).toBe(
'"A very long caption line that runs we..." covers 46% of a face in the video beneath it.',
);
expect(issue?.text).toBe(text);
});
});
describe("findTextOverFace", () => {
afterEach(() => vi.restoreAllMocks());
const at = (time: number, file = "/project/take.mp4") =>
candidate({ time, video: { ...candidate().video, file, mediaTime: time } });
const finder = (faceTimes: number[]) =>
vi.fn<FaceFinder>(
async (_file, times) =>
new Map(
times.map((time) => [
time,
{ frame: Math.round(time * 24), boxes: faceTimes.includes(time) ? [FACE] : [] },
]),
),
);
it("reads each video once for its distinct times and reports only where a face is", async () => {
const facesIn = finder([1, 7]);
const { issues, faceCheck } = await findTextOverFace(
[at(1), at(4), at(7), at(7), at(7, "/project/other.mp4")],
facesIn,
);
expect(facesIn.mock.calls.map(([file, times]) => [file, times])).toEqual([
["/project/take.mp4", [1, 4, 7]],
["/project/other.mp4", [7]],
]);
expect(issues.map((issue) => issue.time)).toEqual([1, 7, 7, 7]);
expect(faceCheck).toEqual({ ran: true, candidates: 5, faces: 3 });
});
it("puts the most covered sample first, so a collapsed finding carries the worst numbers", async () => {
const grazing = { ...at(1), wordRects: [box(400, 760, 200, 40)] };
const { issues } = await findTextOverFace([grazing, at(7)], finder([1, 7]));
expect(issues.map((issue) => [issue.time, issue.coveredFraction])).toEqual([
[7, 0.46],
[1, 0.16],
]);
});
it("says the pass ran when it looked and found no face", async () => {
const { issues, faceCheck } = await findTextOverFace([at(4)], finder([]));
expect(issues).toEqual([]);
expect(faceCheck).toEqual({ ran: true, candidates: 1, faces: 0 });
});
it("survives a detector that cannot load: no finding, one stderr line, the reason kept", async () => {
const stderr = vi.spyOn(console, "error").mockImplementation(() => undefined);
const facesIn = vi.fn<FaceFinder>(async () => {
throw new Error("could not download opencv.js (fetch failed)");
});
const result = await findTextOverFace([at(1), at(7, "/project/other.mp4")], facesIn);
expect(result).toEqual({
issues: [],
faceCheck: { ran: false, reason: "could not download opencv.js (fetch failed)" },
});
expect(stderr).toHaveBeenCalledTimes(1);
expect(stderr.mock.calls[0]?.[0]).toBe(
"[hyperframes] check: text over a face was not looked for in 2 of 2 video(s): could not download opencv.js (fetch failed)",
);
});
// A detector whose first-use download never answers: it settles only when its signal ends.
const neverReady = () =>
vi.fn<FaceFinder>(
(_file, _times, signal) =>
new Promise((_resolve, reject) => {
signal?.addEventListener("abort", () => reject(signal.reason));
}),
);
it("gives up on a detector that is not ready within 30 s and says how to get it", async () => {
vi.useFakeTimers();
try {
vi.spyOn(console, "error").mockImplementation(() => undefined);
const facesIn = neverReady();
const pending = findTextOverFace([at(1)], facesIn);
await vi.advanceTimersByTimeAsync(29_999);
expect(facesIn.mock.calls[0]?.[2]?.aborted).toBe(false);
await vi.advanceTimersByTimeAsync(1);
expect(await pending).toEqual({
issues: [],
faceCheck: {
ran: false,
reason:
"no answer within 30 s; `hyperframes measure face <video>` downloads it with no time limit",
},
});
} finally {
vi.useRealTimers();
}
});
it("stops the detector when the check stops", async () => {
vi.spyOn(console, "error").mockImplementation(() => undefined);
const stop = new AbortController();
const pending = findTextOverFace([at(1)], neverReady(), undefined, stop.signal);
stop.abort(new Error("Check stopped before finishing (SIGINT)."));
expect((await pending).faceCheck).toEqual({
ran: false,
reason: "Check stopped before finishing (SIGINT).",
});
});
it("leaves no timer running once the faces are read", async () => {
vi.useFakeTimers();
try {
await findTextOverFace([at(7)], finder([7]));
expect(vi.getTimerCount()).toBe(0);
} finally {
vi.useRealTimers();
}
});
it("keeps the findings of the videos it could read and counts the ones it could not", async () => {
vi.spyOn(console, "error").mockImplementation(() => undefined);
const readable = finder([7]);
const facesIn = vi.fn<FaceFinder>(async (file, times) => {
if (file === "/project/other.mp4") throw new Error("ffprobe cannot read it");
return readable(file, times);
});
const { issues, faceCheck } = await findTextOverFace(
[at(7), at(7, "/project/other.mp4")],
facesIn,
);
expect(issues).toHaveLength(1);
expect(faceCheck).toEqual({ ran: true, candidates: 2, faces: 1, unreadVideos: 1 });
});
});
+212
View File
@@ -0,0 +1,212 @@
// text_over_face: text painted in front of a video, over a face in it. The page reports the text's
// word boxes and where the video's frame lands (layout-audit.browser.js); the faces come from the
// file itself, with the detector `measure face` uses, because headless Chrome has none.
import type {
AnchoredLayoutIssue,
CheckBbox,
FaceCheck,
TextOverVideoCandidate,
} from "./checkTypes.js";
import { normalizeErrorMessage } from "./errorMessage.js";
/** Below this share of the face a descender or a letter's corner touches it, which hides nothing. */
export const FACE_COVER_REPORT = 0.05;
// A face the video's own box crops to less than half is not one the viewer is shown.
const FACE_SHOWN_MIN = 0.5;
const MESSAGE_TEXT_MAX = 40;
// The detector's first use downloads OpenCV: a network that never answers costs the finding, not the check.
const DETECTOR_READY_MS = 30_000;
const DETECTOR_LATE = `no answer within ${DETECTOR_READY_MS / 1000} s; \`hyperframes measure face <video>\` downloads it with no time limit`;
type VideoPlacement = Pick<TextOverVideoCandidate["video"], "box" | "frame" | "flipX" | "flipY">;
/** Each asked media time's frame, and that frame's faces as [x, y, w, h] shares of it. */
export type FaceFinder = (
file: string,
times: number[],
signal?: AbortSignal,
) => Promise<Map<number, { frame: number; boxes: number[][] }>>;
const area = (box: CheckBbox) => box.width * box.height;
function intersect(a: CheckBbox, b: CheckBbox): CheckBbox | null {
const x = Math.max(a.x, b.x);
const y = Math.max(a.y, b.y);
const width = Math.min(a.x + a.width, b.x + b.width) - x;
const height = Math.min(a.y + a.height, b.y + b.height) - y;
return width > 0 && height > 0 ? { x, y, width, height } : null;
}
/** A face given in shares of the video frame, in canvas pixels, cut to what the video's box shows. */
export function faceRectOnCanvas(face: readonly number[], video: VideoPlacement): CheckBbox | null {
const [fx = 0, fy = 0, fw = 0, fh = 0] = face;
const { frame } = video;
const whole = {
x: frame.x + (video.flipX ? 1 - fx - fw : fx) * frame.width,
y: frame.y + (video.flipY ? 1 - fy - fh : fy) * frame.height,
width: fw * frame.width,
height: fh * frame.height,
};
const shown = intersect(whole, video.box);
return shown && area(shown) >= FACE_SHOWN_MIN * area(whole) ? shown : null;
}
/** The share of `face` under the union of `rects`: overlapping word boxes count once. */
export function coveredShare(rects: readonly CheckBbox[], face: CheckBbox): number {
const inside = rects.flatMap((rect) => intersect(rect, face) ?? []);
const edges = [...new Set(inside.flatMap((rect) => [rect.x, rect.x + rect.width]))].sort(
(a, b) => a - b,
);
let covered = 0;
edges.slice(1).forEach((right, index) => {
const left = edges[index] ?? right;
const spans = inside
.filter((rect) => rect.x <= left && rect.x + rect.width >= right)
.map((rect) => ({ top: rect.y, bottom: rect.y + rect.height }))
.sort((a, b) => a.top - b.top);
let reached = Number.NEGATIVE_INFINITY;
for (const { top, bottom } of spans) {
if (bottom <= reached) continue;
covered += (bottom - Math.max(top, reached)) * (right - left);
reached = bottom;
}
});
return area(face) > 0 ? covered / area(face) : 0;
}
const round2 = (value: number) => Math.round(value * 100) / 100;
function quoted(text: string): string {
return `"${text.length > MESSAGE_TEXT_MAX ? `${text.slice(0, MESSAGE_TEXT_MAX - 3)}...` : text}"`;
}
/** The finding for the face this text covers most, or none when it covers no face enough. */
export function textOverFaceIssue(
candidate: TextOverVideoCandidate,
faces: readonly (readonly number[])[],
): AnchoredLayoutIssue[] {
let most: { faceRect: CheckBbox; coveredFraction: number } | null = null;
for (const face of faces) {
const faceRect = faceRectOnCanvas(face, candidate.video);
if (!faceRect) continue;
const coveredFraction = round2(coveredShare(candidate.wordRects, faceRect));
if (!most || coveredFraction > most.coveredFraction) most = { faceRect, coveredFraction };
}
if (!most || most.coveredFraction < FACE_COVER_REPORT) return [];
const { faceRect, coveredFraction } = most;
return [
{
code: "text_over_face",
// The detector can take texture for a face, so a cover is reported and never fails a strict check.
severity: "info",
time: candidate.time,
selector: candidate.selector,
containerSelector: candidate.video.selector,
text: candidate.text,
message: `${quoted(candidate.text)} covers ${Math.round(coveredFraction * 100)}% of a face in the video beneath it.`,
rect: candidate.rect,
coveredFraction,
faceRect: {
x: round2(faceRect.x),
y: round2(faceRect.y),
width: round2(faceRect.width),
height: round2(faceRect.height),
},
fixHint:
"Move the text off the face, or mark an intended overlap with data-layout-allow-on-face.",
dataAttributes: candidate.dataAttributes,
sourceFile: candidate.sourceFile,
bbox: candidate.bbox,
},
];
}
// Loaded on first use: a check with no text in front of a video never pays for OpenCV.
const facesInFile: FaceFinder = async (file, times, signal) => {
const [{ faceBoxesAt }, { frameAt, probeVideo }] = await Promise.all([
import("../measure/face.js"),
import("../measure/media.js"),
]);
const info = probeVideo(file);
const frameOf = (time: number) => Math.min(Math.max(0, info.frames - 1), frameAt(time, info.fps));
const frames = [...new Set(times.map(frameOf))].sort((a, b) => a - b);
const found = await faceBoxesAt(info, frames, { signal });
// A frame that did not decode shifts every later one, so the boxes no longer belong to their times.
if (found.length !== frames.length) throw new Error(`could not read every frame of ${file}`);
const byFrame = new Map(found.map((sample) => [sample.frame, sample.boxes]));
return new Map(
times.map((time) => [time, { frame: frameOf(time), boxes: byFrame.get(frameOf(time)) ?? [] }]),
);
};
type FacesByTime = Awaited<ReturnType<FaceFinder>>;
async function readFaces(
timesByFile: Map<string, Set<number>>,
facesIn: FaceFinder,
onFile: () => void,
stop?: AbortSignal,
): Promise<{ faces: Map<string, FacesByTime>; unread: string[] }> {
const faces = new Map<string, FacesByTime>();
const unread: string[] = [];
const late = new AbortController();
const timer = setTimeout(() => late.abort(new Error(DETECTOR_LATE)), DETECTOR_READY_MS);
const signal = stop ? AbortSignal.any([stop, late.signal]) : late.signal;
for (const [file, times] of timesByFile) {
onFile();
await facesIn(file, [...times], signal).then(
(found) => faces.set(file, found),
(error: unknown) => unread.push(normalizeErrorMessage(error)),
);
}
clearTimeout(timer);
return { faces, unread };
}
/**
* The text_over_face findings for what the page saw in front of videos, and whether the pass
* could look. Each video file is read once, for the distinct frames its samples landed on. A file
* or a detector that cannot be read costs the finding, never the check: stderr says so in one line.
*/
export async function findTextOverFace(
candidates: readonly TextOverVideoCandidate[],
facesIn: FaceFinder = facesInFile,
onFile: () => void = () => {},
stop?: AbortSignal,
): Promise<{ issues: AnchoredLayoutIssue[]; faceCheck: FaceCheck }> {
const timesByFile = new Map<string, Set<number>>();
for (const { video } of candidates) {
const times = timesByFile.get(video.file) ?? new Set<number>();
timesByFile.set(video.file, times.add(video.mediaTime));
}
const { faces, unread } = await readFaces(timesByFile, facesIn, onFile, stop);
const reason = unread[0];
if (reason !== undefined) {
console.error(
`[hyperframes] check: text over a face was not looked for in ${unread.length} of ${timesByFile.size} video(s): ${reason}`,
);
if (faces.size === 0) return { issues: [], faceCheck: { ran: false, reason } };
}
const framesWithFace = new Set<string>();
for (const [file, byTime] of faces) {
for (const { frame, boxes } of byTime.values()) {
if (boxes.length > 0) framesWithFace.add(`${file}\0${frame}`);
}
}
const issues = candidates.flatMap((candidate) =>
textOverFaceIssue(
candidate,
faces.get(candidate.video.file)?.get(candidate.video.mediaTime)?.boxes ?? [],
),
);
return {
// Worst first: a finding collapsed over time keeps the numbers of its first sample.
issues: issues.sort((a, b) => (b.coveredFraction ?? 0) - (a.coveredFraction ?? 0)),
faceCheck: {
ran: true,
candidates: candidates.length,
faces: framesWithFace.size,
...(unread.length > 0 ? { unreadVideos: unread.length } : {}),
},
};
}