mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-18 07:50:13 +00:00
feat(cli): check finds text over a face, measure cuts names resumed takes, measure hands (#8074)
* feat(cli): check reports text drawn over a face in a video The layout audit reports text painted in front of an opaque video with its word boxes and where the video's frame lands; Node finds the faces in the video file with the detector `measure face` uses and adds a `text_over_face` finding (info from 5% of the face covered, warning from 30%) with `coveredFraction` and `faceRect`. `layout.faceCheck` in the report says whether the pass looked. `data-layout-allow-on-face` waives it. Also reads a two-token `object-position` whose first token is "50%" as the horizontal one. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * feat(cli): measure cuts names the shot a take resumes A shot that returns to an earlier camera setup after at least one other shot (a wide take, a close-up cut into it, the wide take again) carries `continues`: the index in `shots` of the shot it resumes. It is read from the signatures `cuts` already decodes, by the scene's edges, so a person who has moved or an exposure that drifted does not hide the match. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * feat(cli): measure hands follows a hand through a clip `measure hands <video>` reports where a hand's palm (with the hand's size), index fingertip or pinch is in each sampled frame of a stretch of up to 15 s, from MediaPipe's hand landmarker. A frame with no such hand is null, never filled in. The landmarker runs in the CLI's headless Chrome on a page answered from disk; its code and model are not shipped and are downloaded on first use, pinned by size and sha256, into the trusted cache. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(cli): text_over_face is info only, and a stalled OpenCV download no longer stalls check The face detector can take texture for a face, so text_over_face is reported as info however much of the face is covered and never fails --strict. check passes its deadline and a 30 s limit into the detector's first-use OpenCV download. A download that does not answer costs the face pass only: faceCheck is { ran: false, reason }, the other findings are kept and the process exits. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.2
parent
f5ccfcc960
commit
820d63d7f9
@@ -1386,6 +1386,25 @@ transparent. A fully transparent frame is treated as unread on purpose, since a
|
||||
failed draw looks the same; mark text that sits under a cutout before its subject
|
||||
enters with `data-layout-allow-occlusion`.
|
||||
|
||||
Text painted in front of an opaque `<video>` is judged against the faces in the
|
||||
frame that video shows at the sample. `text_over_face` is an info finding when
|
||||
the words cover 5% or more of a face; `coveredFraction` is the share of the face
|
||||
covered and `faceRect` is the face in canvas pixels. It is never more than info,
|
||||
so it never fails `--strict`: the detector can take texture for a face. The
|
||||
faces are found in the video file itself, with the detector
|
||||
[`measure face`](#measure) uses, so only a video served from the project is
|
||||
looked at. That detector's first use downloads OpenCV; when the download has not
|
||||
answered within 30 seconds the check goes on without this pass and `faceCheck`
|
||||
says why. A rotated video and a frame not yet
|
||||
decoded are skipped, a transparent cutout between the text and the take is looked
|
||||
through, and text behind a cutout stays `text_occluded`'s to judge. Mark an
|
||||
intended overlap with `data-layout-allow-on-face` on the text or an ancestor.
|
||||
`layout.faceCheck` in the JSON says whether the pass looked, so no finding reads
|
||||
apart from no look: `{ "ran": true, "candidates": 3, "faces": 2 }` counts the
|
||||
text-over-video samples examined and the distinct video frames a face was found
|
||||
in (`unreadVideos` is added when some files could not be read), and
|
||||
`{ "ran": false, "reason": "..." }` says why it could not.
|
||||
|
||||
`canvas_covered` is an error when an opaque element spanning the canvas sits above
|
||||
every other visible clip (shape, image, video or text) for most of the film, so the
|
||||
frame shows only it: typically a full-frame background placed last in the DOM. It
|
||||
@@ -1726,8 +1745,13 @@ npx hyperframes measure face ref.mp4 --start 1 --end 2.5
|
||||
| `energy` | How much two clips move, and whether at the same moments |
|
||||
| `look` | Brightness, contrast, saturation and warmth, alone or against another video |
|
||||
| `face` | The face's size and position over a window |
|
||||
| `hands` | Where a hand's palm, index fingertip or pinch is in each frame of a stretch |
|
||||
| `sync` | The lag between mouth motion and the voice |
|
||||
|
||||
In `cuts`, a shot that returns to an earlier camera setup after at least one other shot (a wide take, a close-up
|
||||
cut into it, the wide take again) carries `continues`: the index in `shots` of the shot it resumes, so one take can
|
||||
feed both. A shot that resumes nothing has no `continues`.
|
||||
|
||||
### `diff`
|
||||
|
||||
Scores a render, or a project folder without rendering it, against the original
|
||||
|
||||
@@ -10,6 +10,8 @@ const SRC = fileURLToPath(new URL("..", import.meta.url));
|
||||
const OPENS_NO_COMPOSITION: Record<string, string> = {
|
||||
"browser/launch.ts": "defines the launch, and its GPU probe opens a blank page",
|
||||
"capture/launchCaptureBrowser.ts": "`capture` opens a website, not a composition",
|
||||
"measure/hands.ts":
|
||||
"`measure hands` opens its own page of pinned MediaPipe code; every request is answered from disk or aborted",
|
||||
};
|
||||
|
||||
const WATCHED = new Set(["launchManagedBrowser", "acquireBrowser", "launchWithPageNetwork"]);
|
||||
|
||||
@@ -1726,6 +1726,7 @@ function reportWithFindings(overrides: Partial<CheckReport> = {}): CheckReport {
|
||||
tolerance: 2,
|
||||
totalIssueCount: 0,
|
||||
truncated: false,
|
||||
faceCheck: { ran: true, candidates: 0, faces: 0 },
|
||||
},
|
||||
motion: { ...emptySection(), enabled: false, samples: 0 },
|
||||
contrast: { ...emptySection(), enabled: true, samples: [], checked: 0, passed: 0 },
|
||||
@@ -4374,6 +4375,33 @@ it("applies transition sample limits within the requested range", async () => {
|
||||
expect(report.layout.transitionSamplesDropped).toBe(3);
|
||||
});
|
||||
|
||||
describe("layout.faceCheck", () => {
|
||||
it("says the face pass ran when no text was in front of a video", async () => {
|
||||
const { report } = await runScenario(fakeDriver({ collectTextOverVideo: async () => [] }));
|
||||
expect(report.layout.faceCheck).toEqual({ ran: true, candidates: 0, faces: 0 });
|
||||
});
|
||||
|
||||
it("says why when the pass could not look", async () => {
|
||||
const { report } = await runScenario(fakeDriver());
|
||||
expect(report.layout.faceCheck).toEqual({
|
||||
ran: false,
|
||||
reason: "this check driver does not report text over video",
|
||||
});
|
||||
});
|
||||
|
||||
it("is present with a reason when the browser never ran", async () => {
|
||||
const { deps } = dependencies(fakeDriver());
|
||||
deps.runBrowserCheck = async () => {
|
||||
throw new Error("Chrome did not start");
|
||||
};
|
||||
const report = await runCheckPipeline(PROJECT, DEFAULT_CHECK_OPTIONS, deps);
|
||||
expect(report.layout.faceCheck).toEqual({
|
||||
ran: false,
|
||||
reason: "the check stopped before looking for faces under text",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("check cache", () => {
|
||||
const dirs: string[] = [];
|
||||
afterEach(() => {
|
||||
|
||||
@@ -1294,11 +1294,14 @@
|
||||
if (tokens[0] === "top" || tokens[0] === "bottom") y = tokens[0];
|
||||
else x = tokens[0];
|
||||
} else {
|
||||
// The first token that names no side is the horizontal one, even when it is the default "50%".
|
||||
let xGiven = false;
|
||||
for (const token of tokens) {
|
||||
if (token === "top" || token === "bottom") y = token;
|
||||
else if (token === "left" || token === "right") x = token;
|
||||
else if (x === "50%") x = token;
|
||||
else y = token;
|
||||
else if (token === "left" || token === "right" || !xGiven) {
|
||||
x = token;
|
||||
xGiven = true;
|
||||
} else y = token;
|
||||
}
|
||||
}
|
||||
return { x: objectPositionOffset(x, freeX), y: objectPositionOffset(y, freeY) };
|
||||
@@ -1965,6 +1968,139 @@
|
||||
};
|
||||
}
|
||||
|
||||
// text_over_face: headless Chrome has no face detector, so the audit only reports which text is
|
||||
// painted in front of which opaque video and where that video's frame lands; Node finds the
|
||||
// faces in the file and judges the cover (utils/textOverFace.ts).
|
||||
const FACE_PROBE_MAX_COLUMNS = 16;
|
||||
let faceVideoKinds = new Map();
|
||||
let textOverVideo = { time: Number.NaN, candidates: [] };
|
||||
|
||||
// Kept apart from videoAlphaFrames: a frame that cannot be read here is skipped, never reported
|
||||
// as judged opaque.
|
||||
function faceVideoKind(video) {
|
||||
let kind = faceVideoKinds.get(video);
|
||||
if (!kind) {
|
||||
const frame = videoAlphaFrames.get(video) || readVideoAlphaFrame(video);
|
||||
const cutout = frame.kind === "matte" && !hasOpaqueBackground(getComputedStyle(video));
|
||||
kind = cutout ? "cutout" : frame.kind === "unread" ? "unread" : "opaque";
|
||||
faceVideoKinds.set(video, kind);
|
||||
}
|
||||
return kind;
|
||||
}
|
||||
|
||||
// The opaque video the text is painted directly in front of at (x, y). Null where something
|
||||
// opaque above hides the text, where the text sits on an opaque panel, and where the video's
|
||||
// frame cannot be read or mapped. A transparent cutout beneath the text is looked through to the
|
||||
// video under it; text inside the box of one above it is text_occluded's to judge, because the
|
||||
// cutout repaints its subject, face included, over the text.
|
||||
function videoBeneathTextAt(element, x, y) {
|
||||
let beneath = false;
|
||||
for (const hit of document.elementsFromPoint(x, y)) {
|
||||
if (!isForeignElement(element, hit)) {
|
||||
beneath = true;
|
||||
continue;
|
||||
}
|
||||
if (hasIgnoreFlag(hit)) continue;
|
||||
const media = authoredElement(hit);
|
||||
if (media.tagName === "VIDEO" && opacityChain(hit) >= OCCLUDING_ALPHA) {
|
||||
const kind = faceVideoKind(media);
|
||||
if (kind === "cutout" && beneath) continue;
|
||||
if (kind === "cutout" || !beneath) return null;
|
||||
return kind === "opaque" ? media : null;
|
||||
}
|
||||
if (isOpaqueOccluder(hit, x, y, element)) return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// The stretches of one word box in front of a video, probed in columns half a line high, so a
|
||||
// word that runs off the video or under an opaque element counts only for the part in front.
|
||||
function wordPiecesOverVideo(element, rect, add) {
|
||||
const columns = Math.max(
|
||||
1,
|
||||
Math.min(FACE_PROBE_MAX_COLUMNS, Math.round((2 * rect.width) / rect.height)),
|
||||
);
|
||||
const step = rect.width / columns;
|
||||
const y = rect.top + rect.height / 2;
|
||||
let run = null;
|
||||
for (let column = 0; column <= columns; column++) {
|
||||
const video =
|
||||
column < columns ? videoBeneathTextAt(element, rect.left + (column + 0.5) * step, y) : null;
|
||||
if (run && run.video !== video) {
|
||||
add(run.video, {
|
||||
x: round(rect.left + run.from * step),
|
||||
y: round(rect.top),
|
||||
width: round((column - run.from) * step),
|
||||
height: round(rect.height),
|
||||
});
|
||||
run = null;
|
||||
}
|
||||
if (video && !run) run = { video, from: column };
|
||||
}
|
||||
}
|
||||
|
||||
// Where the video's whole frame lands after object-fit and object-position (it can spill past
|
||||
// the element's box, which crops it), and the part of the box the canvas shows.
|
||||
function videoPlacement(video, rootRect) {
|
||||
const mapping = paintedSourceMapping(video, video.videoWidth, video.videoHeight);
|
||||
if (!mapping) return null;
|
||||
const { rect, offsets, paintedWidth, paintedHeight } = mapping;
|
||||
const left = Math.max(rect.left, rootRect.left);
|
||||
const top = Math.max(rect.top, rootRect.top);
|
||||
const right = Math.min(rect.right, rootRect.right);
|
||||
const bottom = Math.min(rect.bottom, rootRect.bottom);
|
||||
if (right <= left || bottom <= top) return null;
|
||||
return {
|
||||
selector: uniqueSelectorFor(video),
|
||||
src: video.currentSrc || video.src || "",
|
||||
mediaTime: video.currentTime,
|
||||
box: {
|
||||
x: round(left),
|
||||
y: round(top),
|
||||
width: round(right - left),
|
||||
height: round(bottom - top),
|
||||
},
|
||||
frame: {
|
||||
x: round(mapping.flipX ? rect.right - offsets.x - paintedWidth : rect.left + offsets.x),
|
||||
y: round(mapping.flipY ? rect.bottom - offsets.y - paintedHeight : rect.top + offsets.y),
|
||||
width: round(paintedWidth),
|
||||
height: round(paintedHeight),
|
||||
},
|
||||
flipX: mapping.flipX,
|
||||
flipY: mapping.flipY,
|
||||
};
|
||||
}
|
||||
|
||||
function textOverVideoCandidates(element, videoRects, rootRect) {
|
||||
if (videoRects.length === 0 || typeof document.elementsFromPoint !== "function") return [];
|
||||
if (authoredElement(element).closest("[data-layout-allow-on-face]")) return [];
|
||||
if (opacityChain(element) < OCCLUDING_ALPHA) return [];
|
||||
const textRect = textRectFor(element, true);
|
||||
if (!textRect || !videoRects.some((rect) => intersectionArea(textRect, rect) > 0)) return [];
|
||||
const piecesByVideo = new Map();
|
||||
const add = (video, piece) => {
|
||||
const pieces = piecesByVideo.get(video);
|
||||
if (pieces) pieces.push(piece);
|
||||
else piecesByVideo.set(video, [piece]);
|
||||
};
|
||||
for (const { rects } of wordClientRects(element)) {
|
||||
for (const rect of rects) wordPiecesOverVideo(element, rect, add);
|
||||
}
|
||||
const candidates = [];
|
||||
for (const [video, wordRects] of piecesByVideo) {
|
||||
const placement = videoPlacement(video, rootRect);
|
||||
if (!placement || !placement.src) continue;
|
||||
candidates.push({
|
||||
...candidateAnchor(element),
|
||||
text: textContentFor(element, true),
|
||||
rect: textRect,
|
||||
wordRects,
|
||||
video: placement,
|
||||
});
|
||||
}
|
||||
return candidates;
|
||||
}
|
||||
|
||||
// canvas_covered: an opaque element over most of the canvas with every other painted clip beneath it.
|
||||
const COVER_CANVAS_SHARE = 0.9;
|
||||
const COVERED_CONTENT_SHARE = 0.9;
|
||||
@@ -2897,6 +3033,11 @@
|
||||
const issues = [];
|
||||
const matteResults = new Map();
|
||||
videoAlphaFrames = new Map();
|
||||
faceVideoKinds = new Map();
|
||||
const overVideo = [];
|
||||
const videoRects = Array.from(root.querySelectorAll("video"), (video) =>
|
||||
video.getBoundingClientRect(),
|
||||
);
|
||||
|
||||
const restoreHits = restoreHitTesting(root);
|
||||
try {
|
||||
@@ -2913,6 +3054,7 @@
|
||||
}
|
||||
const invisible = invisibleTextIssue(element, time);
|
||||
if (invisible) issues.push(invisible);
|
||||
overVideo.push(...textOverVideoCandidates(element, videoRects, rootRect));
|
||||
}
|
||||
const covered = canvasCoveredIssue(root, rootRect, elements, time);
|
||||
if (covered) issues.push(covered);
|
||||
@@ -2930,9 +3072,16 @@
|
||||
issues.push(...canvasEdgeIssues(root, rootRect, time, tolerance));
|
||||
issues.push(...connectorDetachmentIssues(root, rootRect, time));
|
||||
issues.push(...connectorOrphanIssues(root, rootRect, time));
|
||||
textOverVideo = { time, candidates: overVideo };
|
||||
return issues;
|
||||
};
|
||||
|
||||
// What the audit call for this time found, collected in its pass: hit testing is restored there
|
||||
// and each video's frame is read once.
|
||||
window.__hyperframesTextOverVideo = function collectTextOverVideo(options) {
|
||||
return options && options.time === textOverVideo.time ? textOverVideo.candidates : [];
|
||||
};
|
||||
|
||||
// Reruns only the overlap detector (same threshold, no new surface) on a fine grid for the dense motion re-sampling pass.
|
||||
window.__hyperframesOverlapAudit = function auditOverlap(options) {
|
||||
const time = options && typeof options.time === "number" ? options.time : 0;
|
||||
|
||||
@@ -0,0 +1,389 @@
|
||||
// @vitest-environment happy-dom
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const script = readFileSync(
|
||||
join(dirname(fileURLToPath(import.meta.url)), "layout-audit.browser.js"),
|
||||
"utf-8",
|
||||
);
|
||||
|
||||
interface Box {
|
||||
x: number;
|
||||
y: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
|
||||
interface Candidate {
|
||||
selector: string;
|
||||
dataAttributes: Record<string, string>;
|
||||
sourceFile: string;
|
||||
text: string;
|
||||
wordRects: Box[];
|
||||
video: {
|
||||
selector: string;
|
||||
src: string;
|
||||
mediaTime: number;
|
||||
box: Box;
|
||||
frame: Box;
|
||||
flipX: boolean;
|
||||
flipY: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
interface VideoSpec {
|
||||
box?: Box;
|
||||
/** Alpha (0-255) of its decoded frame; a frame with any clear pixel is a cutout. */
|
||||
alpha?: number;
|
||||
style?: Record<string, string>;
|
||||
readyState?: number;
|
||||
}
|
||||
|
||||
interface Word {
|
||||
word: string;
|
||||
x: number;
|
||||
width: number;
|
||||
}
|
||||
|
||||
interface Scene {
|
||||
/** Words of the one text element (#title), all on one line. */
|
||||
words?: Word[];
|
||||
titleAttrs?: string;
|
||||
wrapperAttrs?: string;
|
||||
titleStyle?: Record<string, string>;
|
||||
videos?: Record<string, VideoSpec>;
|
||||
/** Ids of opaque panels. */
|
||||
panels?: string[];
|
||||
/** Ids painted at a point, topmost first; "title" is the text. */
|
||||
stack?: (x: number) => string[];
|
||||
}
|
||||
|
||||
// A 9:16 canvas; every video decodes a 192x108 (16:9) frame.
|
||||
const CANVAS: Box = { x: 0, y: 0, width: 1080, height: 1920 };
|
||||
const SOURCE = { width: 192, height: 108 };
|
||||
const WORD_Y = 800;
|
||||
const WORD_HEIGHT = 100;
|
||||
const HELLO = [{ word: "HELLO", x: 400, width: 200 }];
|
||||
const PANEL_STYLE = { backgroundColor: "rgb(10, 10, 10)" };
|
||||
|
||||
const BASE_STYLE: Record<string, string> = {
|
||||
display: "block",
|
||||
visibility: "visible",
|
||||
opacity: "1",
|
||||
overflow: "visible",
|
||||
overflowX: "visible",
|
||||
overflowY: "visible",
|
||||
backgroundColor: "rgba(0, 0, 0, 0)",
|
||||
backgroundImage: "none",
|
||||
objectFit: "fill",
|
||||
objectPosition: "50% 50%",
|
||||
fontSize: "36px",
|
||||
};
|
||||
|
||||
function domRect({ x, y, width, height }: Box): DOMRect {
|
||||
return {
|
||||
left: x,
|
||||
top: y,
|
||||
width,
|
||||
height,
|
||||
right: x + width,
|
||||
bottom: y + height,
|
||||
x,
|
||||
y,
|
||||
toJSON() {
|
||||
return this;
|
||||
},
|
||||
} as DOMRect;
|
||||
}
|
||||
|
||||
const byId = (id: string) => document.getElementById(id) as HTMLElement;
|
||||
|
||||
function installVideos(videos: Record<string, VideoSpec>): void {
|
||||
for (const [id, spec] of Object.entries(videos)) {
|
||||
const video = byId(id);
|
||||
Object.defineProperties(video, {
|
||||
videoWidth: { configurable: true, value: SOURCE.width },
|
||||
videoHeight: { configurable: true, value: SOURCE.height },
|
||||
readyState: { configurable: true, value: spec.readyState ?? 4 },
|
||||
seeking: { configurable: true, value: false },
|
||||
currentTime: { configurable: true, value: 2.5 },
|
||||
currentSrc: { configurable: true, value: `http://127.0.0.1:4000/takes/${id}.mp4` },
|
||||
});
|
||||
vi.spyOn(video, "getBoundingClientRect").mockReturnValue(domRect(spec.box ?? CANVAS));
|
||||
}
|
||||
}
|
||||
|
||||
// Range boxes for #title: every word for a whole-node range, one word for a word's own range.
|
||||
function installTitleRanges(words: Word[]): void {
|
||||
const boxes = (picked: Word[]) =>
|
||||
picked.map(({ x, width }) =>
|
||||
domRect({ x, y: WORD_Y, width, height: WORD_HEIGHT }),
|
||||
) as unknown as DOMRectList;
|
||||
vi.spyOn(document, "createRange").mockImplementation(() => {
|
||||
let node: Node | null = null;
|
||||
let range: [number, number] | null = null;
|
||||
return {
|
||||
selectNodeContents(selected: Node) {
|
||||
node = selected;
|
||||
},
|
||||
setStart(selected: Node, offset: number) {
|
||||
node = selected;
|
||||
range = [offset, offset];
|
||||
},
|
||||
setEnd(_selected: Node, offset: number) {
|
||||
range = [range ? range[0] : 0, offset];
|
||||
},
|
||||
getClientRects() {
|
||||
if (node?.parentElement?.id !== "title") return boxes([]);
|
||||
if (!range) return boxes(words);
|
||||
const text = (node.textContent ?? "").slice(range[0], range[1]);
|
||||
return boxes(words.filter(({ word }) => word === text));
|
||||
},
|
||||
detach() {},
|
||||
} as unknown as Range;
|
||||
});
|
||||
}
|
||||
|
||||
// Each video's decoded frame is opaque except, for a cutout, one clear corner pixel.
|
||||
function installFrames(videos: Record<string, VideoSpec>): ReturnType<typeof vi.fn> {
|
||||
let drawn = "";
|
||||
const getImageData = vi.fn((_x: number, _y: number, width: number, height: number) => {
|
||||
const data = new Uint8ClampedArray(width * height * 4).fill(255);
|
||||
data[3] = videos[drawn]?.alpha ?? 255;
|
||||
return { data };
|
||||
});
|
||||
// getContext's overloads resolve to the WebGPU one; the audit only asks for "2d".
|
||||
const getContextSpy = vi.spyOn(HTMLCanvasElement.prototype, "getContext") as unknown as {
|
||||
mockReturnValue(value: CanvasRenderingContext2D): void;
|
||||
};
|
||||
getContextSpy.mockReturnValue({
|
||||
clearRect() {},
|
||||
drawImage(source: Element) {
|
||||
drawn = source.id;
|
||||
},
|
||||
getImageData,
|
||||
} as unknown as CanvasRenderingContext2D);
|
||||
return getImageData;
|
||||
}
|
||||
|
||||
function installScene(scene: Scene): { getImageData: ReturnType<typeof vi.fn> } {
|
||||
const words = scene.words ?? HELLO;
|
||||
const videos = scene.videos ?? { take: {} };
|
||||
const panels = scene.panels ?? [];
|
||||
const tags = [
|
||||
...Object.keys(videos).map((id) => `<video id="${id}"></video>`),
|
||||
...panels.map((id) => `<div id="${id}"></div>`),
|
||||
];
|
||||
document.body.innerHTML = `
|
||||
<div id="root" data-composition-id="main" data-width="1080" data-height="1920">
|
||||
${tags.join("")}
|
||||
<div id="wrapper" ${scene.wrapperAttrs ?? ""}><span id="title" ${scene.titleAttrs ?? ""}>${words
|
||||
.map(({ word }) => word)
|
||||
.join(" ")}</span></div>
|
||||
</div>`;
|
||||
|
||||
for (const element of Array.from(document.querySelectorAll("*"))) {
|
||||
vi.spyOn(element, "getBoundingClientRect").mockReturnValue(domRect(CANVAS));
|
||||
}
|
||||
installVideos(videos);
|
||||
const styles = new Map<Element, Record<string, string>>([
|
||||
[byId("title"), scene.titleStyle ?? {}],
|
||||
...panels.map((id): [Element, Record<string, string>] => [byId(id), PANEL_STYLE]),
|
||||
...Object.entries(videos).map(([id, spec]): [Element, Record<string, string>] => [
|
||||
byId(id),
|
||||
spec.style ?? {},
|
||||
]),
|
||||
]);
|
||||
vi.spyOn(window, "getComputedStyle").mockImplementation(
|
||||
(element) => ({ ...BASE_STYLE, ...styles.get(element) }) as unknown as CSSStyleDeclaration,
|
||||
);
|
||||
installTitleRanges(words);
|
||||
|
||||
const stack = scene.stack ?? (() => ["title", ...Object.keys(videos)]);
|
||||
Object.defineProperty(document, "elementsFromPoint", {
|
||||
configurable: true,
|
||||
value: (x: number) => [...stack(x).map(byId), byId("wrapper"), byId("root")],
|
||||
});
|
||||
|
||||
const getImageData = installFrames(videos);
|
||||
window.eval(script);
|
||||
return { getImageData };
|
||||
}
|
||||
|
||||
interface AuditWindow {
|
||||
__hyperframesLayoutAudit(options: { time: number; tolerance: number }): unknown[];
|
||||
__hyperframesTextOverVideo(options: { time: number }): Candidate[];
|
||||
}
|
||||
|
||||
function textOverVideo(time = 1, askedTime = time): Candidate[] {
|
||||
const page = window as unknown as AuditWindow;
|
||||
page.__hyperframesLayoutAudit({ time, tolerance: 2 });
|
||||
return page.__hyperframesTextOverVideo({ time: askedTime });
|
||||
}
|
||||
|
||||
const CUTOUT: VideoSpec = { alpha: 0 };
|
||||
|
||||
describe("layout-audit.browser text in front of a <video>", () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
document.body.innerHTML = "";
|
||||
Reflect.deleteProperty(document, "elementsFromPoint");
|
||||
Reflect.deleteProperty(window, "__hyperframesLayoutAudit");
|
||||
Reflect.deleteProperty(window, "__hyperframesTextOverVideo");
|
||||
});
|
||||
|
||||
it("reports text painted above an opaque video with its word boxes and the video's state", () => {
|
||||
installScene({ titleAttrs: 'data-layer-id="promo"' });
|
||||
expect(textOverVideo()).toEqual([
|
||||
{
|
||||
selector: "#title",
|
||||
dataAttributes: { "data-layer-id": "promo" },
|
||||
sourceFile: "index.html",
|
||||
text: "HELLO",
|
||||
rect: expect.objectContaining({ left: 400, top: 800, width: 200, height: 100 }),
|
||||
wordRects: [{ x: 400, y: 800, width: 200, height: 100 }],
|
||||
video: {
|
||||
selector: "#take",
|
||||
src: "http://127.0.0.1:4000/takes/take.mp4",
|
||||
mediaTime: 2.5,
|
||||
box: CANVAS,
|
||||
frame: CANVAS,
|
||||
flipX: false,
|
||||
flipY: false,
|
||||
},
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("gives one box per word", () => {
|
||||
installScene({
|
||||
words: [
|
||||
{ word: "15%", x: 300, width: 150 },
|
||||
{ word: "OFF", x: 500, width: 180 },
|
||||
],
|
||||
});
|
||||
expect(textOverVideo()[0]?.wordRects).toEqual([
|
||||
{ x: 300, y: 800, width: 150, height: 100 },
|
||||
{ x: 500, y: 800, width: 180, height: 100 },
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
// 16:9 into 9:16: cover scales to the height and spills sideways, contain letterboxes.
|
||||
["cover", "50% 50%", { x: -1166.67, y: 0, width: 3413.33, height: 1920 }],
|
||||
["cover", "0% 50%", { x: 0, y: 0, width: 3413.33, height: 1920 }],
|
||||
["contain", "50% 50%", { x: 0, y: 656.25, width: 1080, height: 607.5 }],
|
||||
["contain", "50% 0%", { x: 0, y: 0, width: 1080, height: 607.5 }],
|
||||
["none", "50% 50%", { x: 444, y: 906, width: 192, height: 108 }],
|
||||
])("places the frame for object-fit %s at %s", (objectFit, objectPosition, frame) => {
|
||||
installScene({ videos: { take: { style: { objectFit, objectPosition } } } });
|
||||
expect(textOverVideo()[0]?.video).toMatchObject({ box: CANVAS, frame });
|
||||
});
|
||||
|
||||
it("cuts the video's box to the canvas and keeps the frame whole", () => {
|
||||
const box = { x: -100, y: 400, width: 1280, height: 720 };
|
||||
installScene({ videos: { take: { box } } });
|
||||
expect(textOverVideo()[0]?.video).toMatchObject({
|
||||
box: { x: 0, y: 400, width: 1080, height: 720 },
|
||||
frame: box,
|
||||
});
|
||||
});
|
||||
|
||||
it("marks a mirrored video", () => {
|
||||
installScene({ videos: { take: { style: { transform: "matrix(-1, 0, 0, 1, 0, 0)" } } } });
|
||||
expect(textOverVideo()[0]?.video).toMatchObject({ flipX: true, flipY: false, frame: CANVAS });
|
||||
});
|
||||
|
||||
it("skips a rotated video instead of guessing where its frame lands", () => {
|
||||
installScene({
|
||||
videos: { take: { style: { transform: "matrix(0.866, 0.5, -0.5, 0.866, 0, 0)" } } },
|
||||
});
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it("skips a video whose frame is not decoded", () => {
|
||||
installScene({ videos: { take: { readyState: 1 } } });
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["the element", { titleAttrs: "data-layout-allow-on-face" }],
|
||||
["an ancestor", { wrapperAttrs: "data-layout-allow-on-face" }],
|
||||
])("honours data-layout-allow-on-face on %s", (_where, attrs) => {
|
||||
installScene(attrs);
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it("looks through a transparent cutout beneath the text to the take under it", () => {
|
||||
installScene({
|
||||
videos: { cutout: CUTOUT, take: {} },
|
||||
stack: () => ["title", "cutout", "take"],
|
||||
});
|
||||
expect(textOverVideo().map((candidate) => candidate.video.selector)).toEqual(["#take"]);
|
||||
});
|
||||
|
||||
it("does not take a lone cutout for a video to look for faces in", () => {
|
||||
installScene({ videos: { cutout: CUTOUT } });
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it("leaves text behind a cutout to text_occluded", () => {
|
||||
installScene({
|
||||
videos: { cutout: CUTOUT, take: {} },
|
||||
stack: () => ["cutout", "title", "take"],
|
||||
});
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it("does not report text that is beneath the video", () => {
|
||||
installScene({ stack: () => ["take", "title"] });
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it("does not report text that sits on an opaque panel over the video", () => {
|
||||
installScene({ panels: ["card"], stack: () => ["title", "card", "take"] });
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it("counts only the stretch of a word that is in front of the video", () => {
|
||||
installScene({
|
||||
panels: ["card"],
|
||||
stack: (x) => (x < 500 ? ["title", "card", "take"] : ["title", "take"]),
|
||||
});
|
||||
expect(textOverVideo()[0]?.wordRects).toEqual([{ x: 500, y: 800, width: 100, height: 100 }]);
|
||||
});
|
||||
|
||||
it("splits a word across two videos side by side", () => {
|
||||
installScene({
|
||||
videos: {
|
||||
left: { box: { x: 0, y: 0, width: 500, height: 1920 } },
|
||||
right: { box: { x: 500, y: 0, width: 580, height: 1920 } },
|
||||
},
|
||||
stack: (x) => ["title", x < 500 ? "left" : "right"],
|
||||
});
|
||||
expect(
|
||||
textOverVideo().map((candidate) => [candidate.video.selector, candidate.wordRects]),
|
||||
).toEqual([
|
||||
["#left", [{ x: 400, y: 800, width: 100, height: 100 }]],
|
||||
["#right", [{ x: 500, y: 800, width: 100, height: 100 }]],
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not report text that is still fading in", () => {
|
||||
installScene({ titleStyle: { opacity: "0.3" } });
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
});
|
||||
|
||||
it("costs nothing without a video: no candidates and no frame read", () => {
|
||||
const { getImageData } = installScene({ videos: {}, stack: () => ["title"] });
|
||||
expect(textOverVideo()).toEqual([]);
|
||||
expect(getImageData).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("answers only for the time its audit ran at", () => {
|
||||
installScene({});
|
||||
expect(textOverVideo(1, 2)).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -430,17 +430,32 @@ describe("layout-audit.browser text behind a <video>", () => {
|
||||
expect(occlusionOf(runAudit(), "ship")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("reads each probed video's frame once per sample and leaves videos under no text alone", () => {
|
||||
it("reads each probed video's frame once per sample, above the text or beneath it", () => {
|
||||
const covering = installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });
|
||||
runAudit();
|
||||
expect(covering.getImageData).toHaveBeenCalledTimes(1);
|
||||
runAudit();
|
||||
expect(covering.getImageData).toHaveBeenCalledTimes(2);
|
||||
|
||||
// A video beneath text is read too: text_over_face looks through a cutout to the take under it.
|
||||
vi.restoreAllMocks();
|
||||
const beneath = installVideoScene({ text: SHIP, video: { overlapsText: false } });
|
||||
runAudit();
|
||||
expect(beneath.getImageData).not.toHaveBeenCalled();
|
||||
expect(beneath.getImageData).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("leaves a video no text touches alone", () => {
|
||||
const apart = installVideoScene({
|
||||
text: SHIP,
|
||||
sizedText: true,
|
||||
video: { overlapsText: false },
|
||||
});
|
||||
const video = document.getElementById("clip") as HTMLVideoElement;
|
||||
vi.spyOn(video, "getBoundingClientRect").mockReturnValue(
|
||||
domRect({ left: 0, top: 0, width: 400, height: 300 }),
|
||||
);
|
||||
runAudit();
|
||||
expect(apart.getImageData).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
|
||||
@@ -15,9 +15,23 @@ interface AuditIssue {
|
||||
coveredFraction?: number;
|
||||
}
|
||||
|
||||
interface Box {
|
||||
x: number;
|
||||
y: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
|
||||
interface TextOverVideo {
|
||||
selector: string;
|
||||
wordRects: Box[];
|
||||
video: { selector: string; src: string; mediaTime: number; box: Box; frame: Box };
|
||||
}
|
||||
|
||||
declare global {
|
||||
interface Window {
|
||||
__hyperframesLayoutAudit(options: { time: number; tolerance: number }): AuditIssue[];
|
||||
__hyperframesTextOverVideo(options: { time: number }): TextOverVideo[];
|
||||
}
|
||||
}
|
||||
|
||||
@@ -162,7 +176,7 @@ describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in C
|
||||
if (clipDir) rmSync(clipDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function audit(html: string, seekTo: number | null = 0.25): Promise<AuditIssue[]> {
|
||||
async function audited<T>(html: string, seekTo: number | null, read: () => T): Promise<T> {
|
||||
const path = `/page-${pages.size}.html`;
|
||||
pages.set(path, html);
|
||||
const tab = await browser.newPage();
|
||||
@@ -171,22 +185,31 @@ describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in C
|
||||
await tab.goto(`${origin}${path}`);
|
||||
if (seekTo !== null) {
|
||||
await tab.evaluate(async (time) => {
|
||||
const video = document.querySelector("video");
|
||||
if (!video) return;
|
||||
if (video.readyState < 2) {
|
||||
await new Promise((resolve) => video.addEventListener("loadeddata", resolve));
|
||||
for (const video of Array.from(document.querySelectorAll("video"))) {
|
||||
if (video.readyState < 2) {
|
||||
await new Promise((resolve) => video.addEventListener("loadeddata", resolve));
|
||||
}
|
||||
video.currentTime = time;
|
||||
await new Promise((resolve) => video.addEventListener("seeked", resolve));
|
||||
}
|
||||
video.currentTime = time;
|
||||
await new Promise((resolve) => video.addEventListener("seeked", resolve));
|
||||
}, seekTo);
|
||||
}
|
||||
await tab.addScriptTag({ content: script });
|
||||
return await tab.evaluate(() => window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 }));
|
||||
return (await tab.evaluate(read)) as T;
|
||||
} finally {
|
||||
await tab.close();
|
||||
}
|
||||
}
|
||||
|
||||
const audit = (html: string, seekTo: number | null = 0.25): Promise<AuditIssue[]> =>
|
||||
audited(html, seekTo, () => window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 }));
|
||||
|
||||
const textOverVideo = (html: string): Promise<TextOverVideo[]> =>
|
||||
audited(html, 0.25, () => {
|
||||
window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 });
|
||||
return window.__hyperframesTextOverVideo({ time: 1 });
|
||||
});
|
||||
|
||||
const occlusionOf = (issues: AuditIssue[], id: string) =>
|
||||
issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);
|
||||
|
||||
@@ -278,6 +301,65 @@ describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in C
|
||||
expect(occlusionOf(issues, "waived")).toBeUndefined();
|
||||
});
|
||||
|
||||
// text_over_face's page half: a title at z-index 3 over a take, with a cutout above or beneath it.
|
||||
const title = (extra = "") =>
|
||||
`<span id="title" ${extra} style="position:absolute;left:100px;top:150px;z-index:3;margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">SHIP IT</span>`;
|
||||
const take = (style = "") =>
|
||||
`<video id="take" src="/opaque.webm" muted playsinline style="position:absolute;left:0;top:0;width:640px;height:360px;z-index:1;${style}"></video>`;
|
||||
const cutoutAt = (zIndex: number) =>
|
||||
`<video id="cutout" src="/cutout.webm" muted playsinline style="position:absolute;left:0;top:0;width:640px;height:360px;z-index:${zIndex}"></video>`;
|
||||
const stage = (body: string) =>
|
||||
`<!doctype html><body style="margin:0"><div data-composition-id="main" data-width="640" data-height="360" style="position:relative;width:640px;height:360px;overflow:hidden;background:#111">${body}</div></body>`;
|
||||
|
||||
it("reports a title in front of a take with its word boxes and the take's clock", async () => {
|
||||
const found = await textOverVideo(stage(take() + title()));
|
||||
expect(found).toHaveLength(1);
|
||||
const { selector, video, wordRects } = found[0]!;
|
||||
expect(selector).toBe("#title");
|
||||
expect(video).toMatchObject({
|
||||
selector: "#take",
|
||||
src: `${origin}/opaque.webm`,
|
||||
mediaTime: 0.25,
|
||||
box: { x: 0, y: 0, width: 640, height: 360 },
|
||||
frame: { x: 0, y: 0, width: 640, height: 360 },
|
||||
});
|
||||
expect(wordRects).toHaveLength(2);
|
||||
const [ship, it] = [wordRects[0]!, wordRects[1]!];
|
||||
expect(ship.x).toBe(100);
|
||||
expect(it.x).toBeGreaterThan(ship.x + ship.width);
|
||||
expect(it).toMatchObject({ y: ship.y, height: ship.height });
|
||||
});
|
||||
|
||||
it("places the frame of a take that object-fit crops", async () => {
|
||||
// 320x180 covering a 220x360 box anchored top right: 640x360 painted, 420 px cut off the left.
|
||||
const cropped = take("width:220px;object-fit:cover;object-position:right top");
|
||||
const found = await textOverVideo(stage(cropped + title()));
|
||||
expect(found).toHaveLength(1);
|
||||
const { video, wordRects } = found[0]!;
|
||||
expect(video).toMatchObject({
|
||||
box: { x: 0, y: 0, width: 220, height: 360 },
|
||||
frame: { x: -420, y: 0, width: 640, height: 360 },
|
||||
});
|
||||
// The title runs past the box onto the stage: only what is in front of the take is reported.
|
||||
expect(wordRects.length).toBeGreaterThan(0);
|
||||
for (const piece of wordRects) expect(piece.x + piece.width / 2).toBeLessThan(220);
|
||||
});
|
||||
|
||||
it("looks through a cutout between the title and the take", async () => {
|
||||
const found = await textOverVideo(stage(take() + cutoutAt(2) + title()));
|
||||
expect(found.map((candidate) => candidate.video.selector)).toEqual(["#take"]);
|
||||
});
|
||||
|
||||
it("leaves a title behind a cutout to text_occluded", async () => {
|
||||
expect(await textOverVideo(stage(take() + cutoutAt(4) + title()))).toEqual([]);
|
||||
});
|
||||
|
||||
it("skips a rotated take and honours data-layout-allow-on-face", async () => {
|
||||
expect(await textOverVideo(stage(take("transform:rotate(8deg)") + title()))).toEqual([]);
|
||||
const waived = stage(take() + title("data-layout-allow-on-face"));
|
||||
expect(await textOverVideo(waived)).toEqual([]);
|
||||
});
|
||||
|
||||
it("falls back to opaque with an info note when no frame has been decoded", async () => {
|
||||
const issues = await audit(
|
||||
page(videoTag("cutout.webm", 'preload="none"'), word("unknown", 480, "SHIP")),
|
||||
|
||||
@@ -24,6 +24,7 @@ const subCommands = {
|
||||
ink: () => import("./measure/ink.js").then((m) => m.default),
|
||||
look: () => import("./measure/look.js").then((m) => m.default),
|
||||
face: () => import("./measure/face.js").then((m) => m.default),
|
||||
hands: () => import("./measure/hands.js").then((m) => m.default),
|
||||
sync: () => import("./measure/sync.js").then((m) => m.default),
|
||||
};
|
||||
|
||||
@@ -33,7 +34,7 @@ export default defineCommand({
|
||||
meta: {
|
||||
name: "measure",
|
||||
description:
|
||||
"Measure a video file as JSON: clock, cuts, events, frames, crops, eases, motion energy, ink, look, faces, lip sync",
|
||||
"Measure a video file as JSON: clock, cuts, events, frames, crops, eases, motion energy, ink, look, faces, hands, lip sync",
|
||||
},
|
||||
subCommands,
|
||||
});
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, runMeasure, videoArg, window } from "../../measure/command.js";
|
||||
import { continuedShots } from "../../measure/continues.js";
|
||||
import { classifiedCuts, signatures } from "../../measure/cuts.js";
|
||||
import { probeVideo, round, videoClip } from "../../measure/media.js";
|
||||
|
||||
@@ -13,7 +14,7 @@ export default defineCommand({
|
||||
meta: {
|
||||
name: "cuts",
|
||||
description:
|
||||
"Hard cuts from the frame-to-frame difference (scene scores miss some), and the shots between them; light flashes, whips and fast moves are listed apart as not_cuts",
|
||||
"Hard cuts from the frame-to-frame difference (scene scores miss some), and the shots between them, a shot that resumes an earlier camera setup naming that shot as continues; light flashes, whips and fast moves are listed apart as not_cuts",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
@@ -26,8 +27,10 @@ export default defineCommand({
|
||||
const info = probeVideo(String(args.video));
|
||||
const [start, end] = window(info, args.start, args.end);
|
||||
const clip = videoClip(info.path, start, end - start, info);
|
||||
const all = classifiedCuts(signatures(clip.input, clip.filters).frames);
|
||||
const frames = signatures(clip.input, clip.filters).frames;
|
||||
const all = classifiedCuts(frames);
|
||||
const found = all.filter((c) => !c.kind);
|
||||
const continues = continuedShots(frames, found);
|
||||
const f0 = Math.ceil(start * info.fps - 1e-6);
|
||||
const not_cuts = all.flatMap((c) =>
|
||||
c.kind ? [{ t: round((f0 + c.frame) / info.fps, 3), kind: c.kind }] : [],
|
||||
@@ -50,6 +53,7 @@ export default defineCommand({
|
||||
start: round(edges[i]!, 3),
|
||||
end: round(e, 3),
|
||||
seconds: round(e - edges[i]!, 2),
|
||||
...(continues[i] !== undefined ? { continues: continues[i] } : {}),
|
||||
})),
|
||||
};
|
||||
});
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
import { defineCommand } from "../../utils/defineCommand.js";
|
||||
import type { Example } from "../_examples.js";
|
||||
import { jsonArg, num, runMeasure, videoArg, window } from "../../measure/command.js";
|
||||
import { HAND_CHOICES, HAND_POINTS, followHand } from "../../measure/hands.js";
|
||||
import { Fix, probeVideo } from "../../measure/media.js";
|
||||
|
||||
export const examples: Example[] = [
|
||||
[
|
||||
"Where the palm is through 3-5 s, for a title that rides it",
|
||||
"hyperframes measure hands ref.mp4 --start 3 --end 5",
|
||||
],
|
||||
[
|
||||
"The right hand's pinch at the video's own frame rate",
|
||||
"hyperframes measure hands ref.mp4 --start 1.2 --end 2.4 --point pinch --which right --fps 30",
|
||||
],
|
||||
];
|
||||
|
||||
function choice<T extends string>(
|
||||
value: unknown,
|
||||
flag: string,
|
||||
choices: readonly T[],
|
||||
fallback: T,
|
||||
): T {
|
||||
if (value === undefined) return fallback;
|
||||
const hit = choices.find((c) => c === value);
|
||||
if (hit === undefined)
|
||||
throw new Fix(
|
||||
`${flag} must be one of ${choices.join(", ")}, got "${String(value)}"`,
|
||||
`${flag} ${fallback}`,
|
||||
);
|
||||
return hit;
|
||||
}
|
||||
|
||||
export default defineCommand({
|
||||
meta: {
|
||||
name: "hands",
|
||||
description:
|
||||
"Where a hand is through a stretch of up to 15 s: its palm, index fingertip or pinch in each sampled frame, from MediaPipe's hand landmarker",
|
||||
},
|
||||
args: {
|
||||
video: videoArg,
|
||||
start: { type: "string", alias: ["from"], description: "First second (default 0)" },
|
||||
end: { type: "string", alias: ["to"], description: "Last second (default: the end)" },
|
||||
point: {
|
||||
type: "string",
|
||||
description:
|
||||
"palm (the wrist and knuckles' middle, with the hand's size), index_tip, or pinch (midway between the thumb and index tips). Default palm",
|
||||
},
|
||||
which: {
|
||||
type: "string",
|
||||
description:
|
||||
"The hand followed: largest, or the largest one labelled left or right. Default largest",
|
||||
},
|
||||
fps: { type: "string", description: "Frames sampled per second, 1 to 30 (default 15)" },
|
||||
json: jsonArg,
|
||||
},
|
||||
async run({ args }) {
|
||||
await runMeasure(async () => {
|
||||
const info = probeVideo(String(args.video));
|
||||
const [start, end] = window(info, args.start, args.end);
|
||||
const followed = await followHand(
|
||||
info,
|
||||
start,
|
||||
end,
|
||||
{
|
||||
point: choice(args.point, "--point", HAND_POINTS, "palm"),
|
||||
which: choice(args.which, "--which", HAND_CHOICES, "largest"),
|
||||
fps: num(args.fps, "--fps", 15),
|
||||
},
|
||||
(message) => console.error(message),
|
||||
);
|
||||
return {
|
||||
video: info.path,
|
||||
...followed,
|
||||
reading:
|
||||
"points[i] is the hand's point at window[0] + i * step seconds: x, y as shares of the frame from its top left (palm adds the hand's width and height); null is a frame with no such hand, never filled in",
|
||||
};
|
||||
});
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,171 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { continuedShots, sameSetup, setupLikeness } from "./continues.js";
|
||||
import { findCuts, SIG_H, SIG_W, type Cut } from "./cuts.js";
|
||||
|
||||
/** A room: 40 boxes of seeded sizes and grey levels on a mid-grey wall, so every part of the frame has edges. */
|
||||
function room(seed: number): Uint8Array {
|
||||
let state = seed;
|
||||
const next = () => {
|
||||
state = (state * 1664525 + 1013904223) >>> 0;
|
||||
return state / 2 ** 32;
|
||||
};
|
||||
const f = new Uint8Array(SIG_W * SIG_H).fill(120);
|
||||
for (let k = 0; k < 40; k++) {
|
||||
const [x0, y0] = [Math.floor(next() * SIG_W), Math.floor(next() * SIG_H)];
|
||||
const [w, h, level] = [4 + Math.floor(next() * 14), 3 + Math.floor(next() * 10), next() * 255];
|
||||
for (let y = y0; y < Math.min(SIG_H, y0 + h); y++)
|
||||
for (let x = x0; x < Math.min(SIG_W, x0 + w); x++) f[y * SIG_W + x] = level;
|
||||
}
|
||||
return f;
|
||||
}
|
||||
|
||||
/** `f` with a box of `level` drawn over it: a person standing at x, or a title. */
|
||||
function over(f: Uint8Array, x0: number, y0: number, w: number, h: number, level = 20) {
|
||||
const out = Uint8Array.from(f);
|
||||
for (let y = y0; y < y0 + h; y++) for (let x = x0; x < x0 + w; x++) out[y * SIG_W + x] = level;
|
||||
return out;
|
||||
}
|
||||
|
||||
const person = (f: Uint8Array, x: number) => over(f, x, 14, 6, 16);
|
||||
|
||||
/** `f` scaled by `s` about the centre and shifted right by `dx` pixels, edges held. */
|
||||
function moved(f: Uint8Array, s: number, dx = 0): Uint8Array {
|
||||
const out = new Uint8Array(f.length);
|
||||
const [cx, cy] = [(SIG_W - 1) / 2, (SIG_H - 1) / 2];
|
||||
for (let y = 0; y < SIG_H; y++)
|
||||
for (let x = 0; x < SIG_W; x++) {
|
||||
const sx = Math.min(SIG_W - 1, Math.max(0, Math.round((x - cx - dx) / s + cx)));
|
||||
const sy = Math.min(SIG_H - 1, Math.max(0, Math.round((y - cy) / s + cy)));
|
||||
out[y * SIG_W + x] = f[sy * SIG_W + sx]!;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const hold = (f: Uint8Array, n: number) => Array.from({ length: n }, () => f);
|
||||
const cut = (frame: number, frames = 1): Cut => ({ frame, frames, change: 40 });
|
||||
|
||||
describe("sameSetup", () => {
|
||||
const wide = room(1);
|
||||
const other = room(2);
|
||||
|
||||
it("knows a setup again when the person in it has moved and the exposure has drifted", () => {
|
||||
const later = person(wide, 41).map((v) => Math.min(255, v * 1.1 + 6));
|
||||
expect(sameSetup([person(wide, 9)], [later])).toBe(true);
|
||||
expect(setupLikeness([wide], [wide])).toEqual({ textured: 48, agreeing: 48 });
|
||||
});
|
||||
|
||||
it("tells two scenes apart, with or without the same person in the same place", () => {
|
||||
expect(sameSetup([wide], [other])).toBe(false);
|
||||
expect(sameSetup([person(wide, 25)], [person(other, 25)])).toBe(false);
|
||||
});
|
||||
|
||||
it("allows a title over part of the frame", () => {
|
||||
expect(sameSetup([over(wide, 16, 0, 32, 6, 250)], [person(wide, 33)])).toBe(true);
|
||||
});
|
||||
|
||||
it("allows a pixel of drift, and refuses a punch-in on the same scene", () => {
|
||||
expect(sameSetup([wide], [moved(wide, 1, 1)])).toBe(true);
|
||||
expect(sameSetup([wide], [moved(wide, 1.3)])).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses a flat card against anything, itself included: there is nothing to compare", () => {
|
||||
const card = new Uint8Array(SIG_W * SIG_H).fill(160);
|
||||
expect(sameSetup([card], [wide])).toBe(false);
|
||||
expect(sameSetup([wide], [card])).toBe(false);
|
||||
expect(sameSetup([card], [card])).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses a split screen that keeps its top half and changes its bottom half", () => {
|
||||
const split = (bottom: Uint8Array) =>
|
||||
wide.map((v, i) => (i < (SIG_W * SIG_H) / 2 ? v : bottom[i]!));
|
||||
// The kept half agrees, less a cell at the seam: about half the frame, well under the bar.
|
||||
expect(setupLikeness([split(other)], [split(room(3))])).toEqual({ textured: 48, agreeing: 23 });
|
||||
expect(sameSetup([split(other)], [split(room(3))])).toBe(false);
|
||||
expect(sameSetup([split(other)], [split(other)])).toBe(true);
|
||||
});
|
||||
|
||||
it("reads the median of three frames, so one flashed frame at a cut does not count", () => {
|
||||
const flash = new Uint8Array(SIG_W * SIG_H).fill(250);
|
||||
expect(sameSetup([wide, wide, wide], [flash, wide, wide])).toBe(true);
|
||||
expect(sameSetup([wide], [flash])).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("continuedShots", () => {
|
||||
const wide = room(1);
|
||||
const insert = room(2);
|
||||
const third = room(3);
|
||||
|
||||
it("names the shot a take resumes after an insert, the person further along", () => {
|
||||
// A wide shot, a close-up cut into it, and the wide shot again.
|
||||
const frames = [
|
||||
...hold(person(wide, 9), 20),
|
||||
...hold(insert, 10),
|
||||
...hold(person(wide, 41), 20),
|
||||
];
|
||||
const cuts = findCuts(frames);
|
||||
expect(cuts.map((c) => c.frame)).toEqual([20, 30]);
|
||||
expect(continuedShots(frames, cuts)).toEqual([undefined, undefined, 0]);
|
||||
});
|
||||
|
||||
it("names nothing among three different shots, or for a single shot", () => {
|
||||
const frames = [...hold(wide, 10), ...hold(insert, 10), ...hold(third, 10)];
|
||||
expect(continuedShots(frames, [cut(10), cut(20)])).toEqual([undefined, undefined, undefined]);
|
||||
expect(continuedShots(hold(wide, 10), [])).toEqual([undefined]);
|
||||
});
|
||||
|
||||
it("names the most recent shot of the setup, where the take was left", () => {
|
||||
// Wide, a jump cut in the wide take, an insert, the wide take again.
|
||||
const frames = [
|
||||
...hold(person(wide, 9), 10),
|
||||
...hold(person(wide, 25), 10),
|
||||
...hold(insert, 10),
|
||||
...hold(person(wide, 41), 10),
|
||||
];
|
||||
expect(continuedShots(frames, [cut(10), cut(20), cut(30)])).toEqual([
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
1,
|
||||
]);
|
||||
});
|
||||
|
||||
it("leaves a jump cut alone: a shot that matches the one before it resumes nothing", () => {
|
||||
// Wide, an insert, wide again, then a jump cut in that wide take.
|
||||
const frames = [
|
||||
...hold(person(wide, 9), 10),
|
||||
...hold(insert, 10),
|
||||
...hold(person(wide, 25), 10),
|
||||
...hold(person(wide, 41), 10),
|
||||
];
|
||||
expect(continuedShots(frames, [cut(10), cut(20), cut(30)])).toEqual([
|
||||
undefined,
|
||||
undefined,
|
||||
0,
|
||||
undefined,
|
||||
]);
|
||||
});
|
||||
|
||||
it("follows two setups cut back and forth", () => {
|
||||
const frames = [wide, insert, wide, insert, wide].flatMap((f) => hold(f, 10));
|
||||
expect(continuedShots(frames, [cut(10), cut(20), cut(30), cut(40)])).toEqual([
|
||||
undefined,
|
||||
undefined,
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
]);
|
||||
});
|
||||
|
||||
it("compares where a shot was left with where the next one starts, not its first frames", () => {
|
||||
// The wide shot pans away to another view before the insert: the take was left there, so nothing resumes.
|
||||
const frames = [...hold(wide, 10), ...hold(third, 10), ...hold(insert, 10), ...hold(wide, 10)];
|
||||
expect(continuedShots(frames, [cut(20), cut(30)])).toEqual([undefined, undefined, undefined]);
|
||||
});
|
||||
|
||||
it("starts a shot after its transition: a three-frame dissolve back into the take still resumes it", () => {
|
||||
const mix = (k: number) => wide.map((v, i) => (insert[i]! * (3 - k) + v * k) / 3);
|
||||
const frames = [...hold(wide, 10), ...hold(insert, 10), mix(1), mix(2), ...hold(wide, 10)];
|
||||
expect(continuedShots(frames, [cut(10), cut(20, 3)])).toEqual([undefined, undefined, 0]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,138 @@
|
||||
// Which shots resume an earlier camera setup, by the scene's edges: a person moves and exposure drifts, walls do not.
|
||||
import { SIG_H, SIG_W, type Cut } from "./cuts.js";
|
||||
|
||||
/** Cells across and down, 8 x 6 pixels each: small enough that a person or a title spoils only a few. */
|
||||
const COLS = 8;
|
||||
const ROWS = 6;
|
||||
const CELL_W = SIG_W / COLS;
|
||||
const CELL_H = SIG_H / ROWS;
|
||||
/** Frames taken from each end of a shot, a median a pixel: one blended or flashed frame at a cut drops out. */
|
||||
const PLATE_FRAMES = 3;
|
||||
/** Mean squared gradient a pixel for a cell to say anything: under it a wall or a sky is flat, and noise decides. */
|
||||
const TEXTURED = 15;
|
||||
/** The weaker picture's share of the stronger one's edges in a cell: a flat card matches nothing. */
|
||||
const BOTH_TEXTURED = 0.25;
|
||||
/** Gradient correlation for a cell to be the same piece of scene: a still camera gives 0.9 and up. */
|
||||
const CELL_AGREES = 0.75;
|
||||
/** Textured cells out of all: with fewer, the picture has too little structure to tell two setups apart. */
|
||||
const MIN_TEXTURED = 0.35;
|
||||
/** Agreeing cells out of the textured ones. Other scenes read under 0.6 and the same room framed tighter or
|
||||
* wider 0.6 to 0.7: a wrong match joins two takes, a miss costs none. */
|
||||
const MIN_AGREEING = 0.75;
|
||||
/** Pixels (of 64 x 36) the picture may have slid between the two ends: a hand-held take never returns exactly. */
|
||||
const DRIFT = 1;
|
||||
const WORTH_SLIDING = MIN_AGREEING / 2;
|
||||
|
||||
export interface Likeness {
|
||||
textured: number;
|
||||
agreeing: number;
|
||||
}
|
||||
|
||||
interface Plate {
|
||||
gx: Float32Array;
|
||||
gy: Float32Array;
|
||||
}
|
||||
|
||||
const median3 = (a: number, b: number, c: number) =>
|
||||
Math.max(Math.min(a, b), Math.min(Math.max(a, b), c));
|
||||
|
||||
function meanOf3x3Around(px: Float32Array, x: number, y: number): number {
|
||||
let [sum, n] = [0, 0];
|
||||
for (let v = Math.max(0, y - 1); v <= Math.min(SIG_H - 1, y + 1); v++)
|
||||
for (let u = Math.max(0, x - 1); u <= Math.min(SIG_W - 1, x + 1); u++) {
|
||||
sum += px[v * SIG_W + u]!;
|
||||
n++;
|
||||
}
|
||||
return sum / n;
|
||||
}
|
||||
|
||||
function plate(frames: Uint8Array[]): Plate {
|
||||
const gx = new Float32Array(SIG_W * SIG_H);
|
||||
const gy = new Float32Array(SIG_W * SIG_H);
|
||||
const a = frames[0];
|
||||
if (!a) return { gx, gy };
|
||||
const b = frames[1] ?? a;
|
||||
const c = frames[2] ?? b;
|
||||
const still = gx.map((_, i) => median3(a[i]!, b[i]!, c[i]!));
|
||||
const soft = still.map((_, i) => meanOf3x3Around(still, i % SIG_W, Math.floor(i / SIG_W)));
|
||||
for (let y = 1; y < SIG_H - 1; y++)
|
||||
for (let x = 1; x < SIG_W - 1; x++) {
|
||||
const i = y * SIG_W + x;
|
||||
gx[i] = soft[i + 1]! - soft[i - 1]!;
|
||||
gy[i] = soft[i + SIG_W]! - soft[i - SIG_W]!;
|
||||
}
|
||||
return { gx, gy };
|
||||
}
|
||||
|
||||
function cellAgrees(
|
||||
a: Plate,
|
||||
b: Plate,
|
||||
[col, row]: [number, number],
|
||||
[dx, dy]: [number, number],
|
||||
): boolean | undefined {
|
||||
let [aa, bb, ab, n] = [0, 0, 0, 0];
|
||||
const yEnd = Math.min((row + 1) * CELL_H, SIG_H - dy);
|
||||
const xEnd = Math.min((col + 1) * CELL_W, SIG_W - dx);
|
||||
for (let y = Math.max(row * CELL_H, -dy); y < yEnd; y++)
|
||||
for (let x = Math.max(col * CELL_W, -dx); x < xEnd; x++) {
|
||||
const i = y * SIG_W + x;
|
||||
const j = i + dy * SIG_W + dx;
|
||||
aa += a.gx[i]! ** 2 + a.gy[i]! ** 2;
|
||||
bb += b.gx[j]! ** 2 + b.gy[j]! ** 2;
|
||||
ab += a.gx[i]! * b.gx[j]! + a.gy[i]! * b.gy[j]!;
|
||||
n++;
|
||||
}
|
||||
if (Math.max(aa, bb) <= TEXTURED * n) return undefined;
|
||||
return (
|
||||
Math.min(aa, bb) >= BOTH_TEXTURED * Math.max(aa, bb) && ab >= CELL_AGREES * Math.sqrt(aa * bb)
|
||||
);
|
||||
}
|
||||
|
||||
function likenessAt(a: Plate, b: Plate, dx: number, dy: number): Likeness {
|
||||
let [textured, agreeing] = [0, 0];
|
||||
for (let cell = 0; cell < COLS * ROWS; cell++) {
|
||||
const agrees = cellAgrees(a, b, [cell % COLS, Math.floor(cell / COLS)], [dx, dy]);
|
||||
if (agrees === undefined) continue;
|
||||
textured++;
|
||||
if (agrees) agreeing++;
|
||||
}
|
||||
return { textured, agreeing };
|
||||
}
|
||||
|
||||
function likeness(a: Plate, b: Plate): Likeness {
|
||||
let best = likenessAt(a, b, 0, 0);
|
||||
if (best.agreeing < WORTH_SLIDING * best.textured) return best;
|
||||
for (let dy = -DRIFT; dy <= DRIFT; dy++)
|
||||
for (let dx = -DRIFT; dx <= DRIFT; dx++) {
|
||||
const at = dx || dy ? likenessAt(a, b, dx, dy) : best;
|
||||
if (at.agreeing > best.agreeing) best = at;
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
const isSame = ({ textured, agreeing }: Likeness) =>
|
||||
textured >= MIN_TEXTURED * COLS * ROWS && agreeing >= MIN_AGREEING * textured;
|
||||
|
||||
export function setupLikeness(a: Uint8Array[], b: Uint8Array[]): Likeness {
|
||||
return likeness(plate(a), plate(b));
|
||||
}
|
||||
|
||||
export function sameSetup(a: Uint8Array[], b: Uint8Array[]): boolean {
|
||||
return isSame(setupLikeness(a, b));
|
||||
}
|
||||
|
||||
/**
|
||||
* For each shot between `cuts`, the index of the earlier shot whose camera setup it resumes after at least one other
|
||||
* shot. A match with the shot right before is a jump cut, which the cut already tells; the most recent match wins.
|
||||
*/
|
||||
export function continuedShots(frames: Uint8Array[], cuts: Cut[]): (number | undefined)[] {
|
||||
const starts = [0, ...cuts.map((c) => c.frame + c.frames - 1)];
|
||||
const ends = [...cuts.map((c) => c.frame), frames.length];
|
||||
const heads = starts.map((s, i) => plate(frames.slice(s, Math.min(s + PLATE_FRAMES, ends[i]!))));
|
||||
const tails = ends.map((e, i) => plate(frames.slice(Math.max(starts[i]!, e - PLATE_FRAMES), e)));
|
||||
return heads.map((head, i) => {
|
||||
if (i < 2 || isSame(likeness(tails[i - 1]!, head))) return undefined;
|
||||
for (let j = i - 2; j >= 0; j--) if (isSame(likeness(tails[j]!, head))) return j;
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
@@ -86,10 +86,6 @@ export function readSamples(info: VideoInfo, ks: number[], w: number, h: number)
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Each sampled frame's faces as [x, y, w, h] shares of the frame. `minFace` is the smallest face looked for as a
|
||||
* share of the frame's height (default FACE_MIN; 0 looks down to the cascades' 20 px, for crowds).
|
||||
*/
|
||||
export async function faceBoxes(
|
||||
info: VideoInfo,
|
||||
start: number,
|
||||
@@ -97,8 +93,19 @@ export async function faceBoxes(
|
||||
samples: number,
|
||||
opts: { minFace?: number } = {},
|
||||
) {
|
||||
const cv = await loadOpenCv((m) => console.error(m));
|
||||
const ks = sampleFrames(info.fps, start, end, samples);
|
||||
return faceBoxesAt(info, sampleFrames(info.fps, start, end, samples), opts);
|
||||
}
|
||||
|
||||
/**
|
||||
* The faces of frames `ks` (ascending indices) as [x, y, w, h] shares of the frame. `minFace` is the smallest face
|
||||
* looked for as a share of the frame's height (default FACE_MIN; 0 looks down to the cascades' 20 px, for crowds).
|
||||
*/
|
||||
export async function faceBoxesAt(
|
||||
info: VideoInfo,
|
||||
ks: number[],
|
||||
opts: { minFace?: number; signal?: AbortSignal } = {},
|
||||
) {
|
||||
const cv = await loadOpenCv((m) => console.error(m), opts.signal);
|
||||
const w = Math.min(WORK_W, info.width);
|
||||
const h =
|
||||
w === info.width
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, describe, expect, it } from "vitest";
|
||||
import { findFFmpeg, findFFprobe } from "../browser/ffmpeg.js";
|
||||
import {
|
||||
MEDIAPIPE_DIR,
|
||||
followHand,
|
||||
handFrames,
|
||||
handPath,
|
||||
pageFile,
|
||||
pickHand,
|
||||
placeHand,
|
||||
readHands,
|
||||
type Hand,
|
||||
} from "./hands.js";
|
||||
import { Fix, probeVideo, runFfmpeg } from "./media.js";
|
||||
|
||||
const media = !!findFFmpeg() && !!findFFprobe();
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-hands-test-"));
|
||||
afterAll(() => rmSync(dir, { recursive: true, force: true }));
|
||||
|
||||
/** A hand whose 21 landmarks all sit at (x, y), then the ones in `moved` put elsewhere. */
|
||||
function hand(side: string, x: number, y: number, moved: Record<number, [number, number]> = {}) {
|
||||
const points = Array.from({ length: 21 }, (_, i): [number, number] => moved[i] ?? [x, y]);
|
||||
return { side, points } satisfies Hand;
|
||||
}
|
||||
|
||||
/** A hand that spans `w` x `h` from (x, y): its wrist at one corner, its little finger's tip at the other. */
|
||||
const sized = (side: string, x: number, y: number, w: number, h: number) =>
|
||||
hand(side, x, y, { 20: [x + w, y + h] });
|
||||
|
||||
describe("placeHand", () => {
|
||||
it("puts the palm at the mean of the wrist and the four knuckles, with the hand's size", () => {
|
||||
const h = hand("Left", 0.5, 0.5, {
|
||||
0: [0.4, 0.6],
|
||||
5: [0.42, 0.5],
|
||||
9: [0.44, 0.48],
|
||||
13: [0.46, 0.5],
|
||||
17: [0.48, 0.52],
|
||||
// A fingertip far from the palm widens the hand and leaves the palm where it is.
|
||||
12: [0.6, 0.2],
|
||||
});
|
||||
expect(placeHand(h, "palm")).toEqual([0.44, 0.52, 0.2, 0.4]);
|
||||
});
|
||||
|
||||
it("gives the index fingertip, and the pinch midway between it and the thumb's tip", () => {
|
||||
const h = hand("Right", 0.5, 0.5, { 4: [0.3, 0.4], 8: [0.35, 0.3] });
|
||||
expect(placeHand(h, "index_tip")).toEqual([0.35, 0.3]);
|
||||
expect(placeHand(h, "pinch")).toEqual([0.325, 0.35]);
|
||||
});
|
||||
|
||||
it("rounds to four places", () => {
|
||||
const h = hand("Right", 0.5, 0.5, { 8: [0.123449, 0.98766] });
|
||||
expect(placeHand(h, "index_tip")).toEqual([0.1234, 0.9877]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("pickHand", () => {
|
||||
const small = sized("Left", 0.1, 0.1, 0.05, 0.05);
|
||||
const big = sized("Right", 0.5, 0.5, 0.2, 0.2);
|
||||
const bigger = sized("Right", 0.3, 0.3, 0.3, 0.2);
|
||||
|
||||
it("follows the largest hand by default", () => {
|
||||
expect(pickHand([small, big], "largest")).toBe(big);
|
||||
expect(pickHand([big, small], "largest")).toBe(big);
|
||||
});
|
||||
|
||||
it("follows the largest hand with the side asked for, however large the other is", () => {
|
||||
expect(pickHand([small, big], "left")).toBe(small);
|
||||
expect(pickHand([big, bigger, small], "right")).toBe(bigger);
|
||||
});
|
||||
|
||||
it("keeps the first of two hands the same size", () => {
|
||||
// Quarters, so both areas are the same number exactly.
|
||||
const [one, twin] = [
|
||||
sized("Right", 0.5, 0.25, 0.25, 0.25),
|
||||
sized("Left", 0.25, 0.5, 0.25, 0.25),
|
||||
];
|
||||
expect(pickHand([one, twin], "largest")).toBe(one);
|
||||
expect(pickHand([twin, one], "largest")).toBe(twin);
|
||||
});
|
||||
|
||||
it("finds none when no hand, or none on that side, is in the frame", () => {
|
||||
expect(pickHand([], "largest")).toBeNull();
|
||||
expect(pickHand([big], "left")).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("handPath", () => {
|
||||
it("leaves a frame without the hand null between the frames that have it", () => {
|
||||
const left = (x: number) => hand("Left", x, 0.5, { 8: [x, 0.25] });
|
||||
const frames = [[left(0.2)], [], [hand("Right", 0.9, 0.9)], [left(0.4)]];
|
||||
expect(handPath(frames, "index_tip", "left")).toEqual([[0.2, 0.25], null, null, [0.4, 0.25]]);
|
||||
expect(handPath(frames, "index_tip", "largest")[2]).toEqual([0.9, 0.9]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("readHands", () => {
|
||||
it("takes what the page answers", () => {
|
||||
const answer: unknown = JSON.parse(JSON.stringify([hand("Left", 0.5, 0.5)]));
|
||||
expect(readHands(answer)).toEqual([hand("Left", 0.5, 0.5)]);
|
||||
expect(readHands([])).toEqual([]);
|
||||
});
|
||||
|
||||
it("refuses an answer that is not hands of 21 landmarks", () => {
|
||||
const short = { side: "Left", points: [[0.5, 0.5]] };
|
||||
const unlabelled = { points: hand("Left", 0.5, 0.5).points };
|
||||
const holed = { side: "Left", points: hand("Left", 0.5, 0.5).points.map(() => [0.5, null]) };
|
||||
for (const bad of [null, "hands", [short], [unlabelled], [holed]])
|
||||
expect(() => readHands(bad)).toThrow(/shape/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("pageFile", () => {
|
||||
it("answers the page, the pinned files and the stretch's frames, and nothing else", () => {
|
||||
expect(pageFile("/", dir)?.text).toContain("HandLandmarker");
|
||||
expect(pageFile("/vision_wasm_internal.wasm", dir)).toEqual({
|
||||
type: "application/wasm",
|
||||
file: join(MEDIAPIPE_DIR, "vision_wasm_internal.wasm"),
|
||||
});
|
||||
expect(pageFile("/frames/t00012.jpg", dir)).toEqual({
|
||||
type: "image/jpeg",
|
||||
file: join(dir, "t00012.jpg"),
|
||||
});
|
||||
for (const path of ["/frames/../t00012.jpg", "/frames/notes.txt", "/etc/passwd", "/frames/"])
|
||||
expect(pageFile(path, dir)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!media)("handFrames", () => {
|
||||
it("writes the stretch at the asked rate, 480 wide, one frame a step from its start", () => {
|
||||
const video = join(dir, "clip.mp4");
|
||||
runFfmpeg([
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"testsrc=size=320x240:rate=30:duration=2",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
video,
|
||||
]);
|
||||
const out = mkdtempSync(join(dir, "frames-"));
|
||||
const names = handFrames(probeVideo(video), 0.5, 1.5, 15, out);
|
||||
expect(names).toHaveLength(15);
|
||||
expect(names[0]).toBe("t00001.jpg");
|
||||
expect(names.at(-1)).toBe("t00015.jpg");
|
||||
const first = probeVideo(join(out, "t00001.jpg"));
|
||||
expect([first.width, first.height]).toEqual([480, 360]);
|
||||
});
|
||||
|
||||
it("refuses a stretch longer than it follows before reading a frame", async () => {
|
||||
const video = join(dir, "long.mp4");
|
||||
runFfmpeg([
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=gray:size=64x64:rate=5:duration=16",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
video,
|
||||
]);
|
||||
const opts = { point: "palm", which: "largest", fps: 15 } as const;
|
||||
const refused = followHand(probeVideo(video), 0, 16, opts);
|
||||
await expect(refused).rejects.toBeInstanceOf(Fix);
|
||||
await expect(refused).rejects.toMatchObject({ fix: "--start 0 --end 15" });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,310 @@
|
||||
// Hand keypoints through a stretch of video: where the palm, the index
|
||||
// fingertip or the pinch (thumb and index tips together) is, frame by frame, from MediaPipe's hand landmarker. A hand
|
||||
// closes, turns and spreads its fingers, which loses a colour or patch tracker; landmarks follow the hand itself.
|
||||
//
|
||||
// The landmarker is MediaPipe's own WebAssembly build with the model the Python package uses, so both give the same
|
||||
// numbers. It draws each frame through a WebGL canvas, which Node has not, so it runs in the CLI's headless Chrome on a
|
||||
// page whose every request is answered from disk. Neither is shipped in the CLI: the first use downloads them, pinned
|
||||
// by size and sha256, into ~/.cache/hyperframes/mediapipe/<version>/.
|
||||
import { mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import type { HTTPRequest, Page } from "puppeteer-core";
|
||||
import { trustedCacheRoot } from "@hyperframes/core/cache-dir";
|
||||
import { Fix, round, runFfmpeg, type VideoInfo } from "./media.js";
|
||||
import { ensurePinnedFiles, type PinnedFile } from "./pinnedFiles.js";
|
||||
|
||||
const VERSION = "1.0.0";
|
||||
const CODE_ROOT = trustedCacheRoot();
|
||||
export const MEDIAPIPE_DIR = join(CODE_ROOT, "mediapipe", VERSION);
|
||||
|
||||
interface Served extends PinnedFile {
|
||||
type: string;
|
||||
}
|
||||
|
||||
// The code is @mediapipe/tasks-vision 1.0.0 and the model is MediaPipe's float16 hand landmarker (both Apache-2.0).
|
||||
const TASKS_VISION = `https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@${VERSION}`;
|
||||
const FILES: readonly Served[] = [
|
||||
{
|
||||
name: "vision_bundle.mjs",
|
||||
bytes: 154_839,
|
||||
sha256: "54ad628b478853dadb23261c1c7d14cfe3760b363ccf62b8c02ef73e5c7fb87a",
|
||||
url: `${TASKS_VISION}/vision_bundle.mjs`,
|
||||
type: "text/javascript",
|
||||
},
|
||||
{
|
||||
name: "vision_wasm_internal.js",
|
||||
bytes: 323_343,
|
||||
sha256: "e2da04fe2ab9f58eb2aeb8f8724c80ad8919d31d7f5fb99057565879f05f51ca",
|
||||
url: `${TASKS_VISION}/wasm/vision_wasm_internal.js`,
|
||||
type: "text/javascript",
|
||||
},
|
||||
{
|
||||
name: "vision_wasm_internal.wasm",
|
||||
bytes: 11_532_084,
|
||||
sha256: "c266c65f3789bea74ab12dd6609edc96c68655da4540d6097d0732d8be80b742",
|
||||
url: `${TASKS_VISION}/wasm/vision_wasm_internal.wasm`,
|
||||
type: "application/wasm",
|
||||
},
|
||||
{
|
||||
name: "hand_landmarker.task",
|
||||
bytes: 7_819_105,
|
||||
sha256: "fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1",
|
||||
url: "https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task",
|
||||
type: "application/octet-stream",
|
||||
},
|
||||
];
|
||||
|
||||
export const HAND_POINTS = ["palm", "index_tip", "pinch"] as const;
|
||||
export type HandPoint = (typeof HAND_POINTS)[number];
|
||||
export const HAND_CHOICES = ["largest", "left", "right"] as const;
|
||||
export type WhichHand = (typeof HAND_CHOICES)[number];
|
||||
|
||||
// The wrist and the four knuckles: the palm's middle, which stays put while the fingers move.
|
||||
const PALM = [0, 5, 9, 13, 17];
|
||||
const THUMB_TIP = 4;
|
||||
const INDEX_TIP = 8;
|
||||
const LANDMARKS = 21;
|
||||
const MAX_HANDS = 2;
|
||||
const WIDTH = 480;
|
||||
const MAX_FPS = 30;
|
||||
const MAX_SECONDS = 15;
|
||||
|
||||
type Xy = [x: number, y: number];
|
||||
|
||||
/** One hand in one frame: MediaPipe's side label and its 21 landmarks as shares of the frame. */
|
||||
export interface Hand {
|
||||
side: string;
|
||||
points: Xy[];
|
||||
}
|
||||
|
||||
const span = (v: number[]) => Math.max(...v) - Math.min(...v);
|
||||
const xs = (hand: Hand) => hand.points.map((p) => p[0]);
|
||||
const ys = (hand: Hand) => hand.points.map((p) => p[1]);
|
||||
|
||||
function handArea(hand: Hand): number {
|
||||
return span(xs(hand)) * span(ys(hand));
|
||||
}
|
||||
|
||||
/** The point as a track point: the palm with the hand's size [x, y, w, h], a fingertip or a pinch as [x, y]. */
|
||||
export function placeHand(hand: Hand, point: HandPoint): number[] {
|
||||
if (point === "palm") {
|
||||
const mean = (axis: 0 | 1) =>
|
||||
PALM.reduce((sum, i) => sum + hand.points[i]![axis], 0) / PALM.length;
|
||||
return [mean(0), mean(1), span(xs(hand)), span(ys(hand))].map((v) => round(v, 4));
|
||||
}
|
||||
const tip = hand.points[INDEX_TIP]!;
|
||||
if (point === "index_tip") return [round(tip[0], 4), round(tip[1], 4)];
|
||||
const thumb = hand.points[THUMB_TIP]!;
|
||||
return [round((thumb[0] + tip[0]) / 2, 4), round((thumb[1] + tip[1]) / 2, 4)];
|
||||
}
|
||||
|
||||
/** The hand to follow: the largest (nearest the lens), or the largest one MediaPipe labels left or right. */
|
||||
export function pickHand(hands: Hand[], which: WhichHand): Hand | null {
|
||||
const side = which === "left" ? "Left" : "Right";
|
||||
return hands
|
||||
.filter((hand) => which === "largest" || hand.side === side)
|
||||
.reduce<Hand | null>(
|
||||
(best, hand) => (best && handArea(best) >= handArea(hand) ? best : hand),
|
||||
null,
|
||||
);
|
||||
}
|
||||
|
||||
/** The followed hand's point in each frame; a frame without that hand stays null. */
|
||||
export function handPath(frames: Hand[][], point: HandPoint, which: WhichHand) {
|
||||
return frames.map((hands) => {
|
||||
const hand = pickHand(hands, which);
|
||||
return hand ? placeHand(hand, point) : null;
|
||||
});
|
||||
}
|
||||
|
||||
const isXy = (v: unknown): v is Xy =>
|
||||
Array.isArray(v) && v.length === 2 && v.every((n) => typeof n === "number" && Number.isFinite(n));
|
||||
|
||||
const isHand = (v: unknown): v is Hand =>
|
||||
typeof v === "object" &&
|
||||
v !== null &&
|
||||
"side" in v &&
|
||||
typeof v.side === "string" &&
|
||||
"points" in v &&
|
||||
Array.isArray(v.points) &&
|
||||
v.points.length === LANDMARKS &&
|
||||
v.points.every(isXy);
|
||||
|
||||
/** What the page answered for one frame, checked: it crosses from Chrome as plain JSON. */
|
||||
export function readHands(answer: unknown): Hand[] {
|
||||
if (!Array.isArray(answer) || !answer.every(isHand))
|
||||
throw new Error("the hand landmarker answered in a shape this CLI does not read");
|
||||
return answer;
|
||||
}
|
||||
|
||||
const FRAME_NAME = /^t\d{5}\.jpg$/;
|
||||
|
||||
/**
|
||||
* The stretch as numbered JPEGs in `dir`, `fps` a second and WIDTH wide. Returns their names in order.
|
||||
*/
|
||||
export function handFrames(
|
||||
info: VideoInfo,
|
||||
start: number,
|
||||
end: number,
|
||||
fps: number,
|
||||
dir: string,
|
||||
): string[] {
|
||||
runFfmpeg([
|
||||
"-y",
|
||||
"-ss",
|
||||
start.toFixed(3),
|
||||
"-i",
|
||||
info.path,
|
||||
"-t",
|
||||
(end - start).toFixed(3),
|
||||
"-vf",
|
||||
`fps=${fps}:round=near,scale=${WIDTH}:-2`,
|
||||
"-q:v",
|
||||
"3",
|
||||
join(dir, "t%05d.jpg"),
|
||||
]);
|
||||
return readdirSync(dir)
|
||||
.filter((name) => FRAME_NAME.test(name))
|
||||
.sort();
|
||||
}
|
||||
|
||||
// No such host exists: the page is answered from disk before Chrome looks one up.
|
||||
const ORIGIN = "https://hands.hyperframes.invalid";
|
||||
|
||||
// The CPU delegate, as in the Python, so a machine's GPU cannot move the numbers.
|
||||
const PAGE = `<!doctype html><meta charset="utf-8"><title>hands</title><script type="module">
|
||||
import { HandLandmarker } from "/vision_bundle.mjs";
|
||||
let landmarker;
|
||||
window.loadHands = async () => {
|
||||
landmarker = await HandLandmarker.createFromOptions(
|
||||
{ wasmLoaderPath: "/vision_wasm_internal.js", wasmBinaryPath: "/vision_wasm_internal.wasm" },
|
||||
{
|
||||
baseOptions: { modelAssetPath: "/hand_landmarker.task", delegate: "CPU" },
|
||||
runningMode: "VIDEO",
|
||||
numHands: ${MAX_HANDS},
|
||||
},
|
||||
);
|
||||
};
|
||||
window.handsIn = async (name, ms) => {
|
||||
const file = await (await fetch("/frames/" + name)).blob();
|
||||
const picture = await createImageBitmap(file, { colorSpaceConversion: "none", premultiplyAlpha: "none" });
|
||||
try {
|
||||
const found = landmarker.detectForVideo(picture, ms);
|
||||
return found.landmarks.map((marks, i) => ({
|
||||
side: found.handedness[i]?.[0]?.categoryName ?? "",
|
||||
points: marks.map((mark) => [mark.x, mark.y]),
|
||||
}));
|
||||
} finally {
|
||||
picture.close();
|
||||
}
|
||||
};
|
||||
</script>`;
|
||||
|
||||
/** The body and type for a path on the page's origin, or null for one it has no business asking for. */
|
||||
export function pageFile(
|
||||
path: string,
|
||||
frames: string,
|
||||
): { type: string; file?: string; text?: string } | null {
|
||||
if (path === "/") return { type: "text/html", text: PAGE };
|
||||
const served = FILES.find((f) => path === `/${f.name}`);
|
||||
if (served) return { type: served.type, file: join(MEDIAPIPE_DIR, served.name) };
|
||||
const frame = path.startsWith("/frames/") ? path.slice("/frames/".length) : "";
|
||||
return FRAME_NAME.test(frame) ? { type: "image/jpeg", file: join(frames, frame) } : null;
|
||||
}
|
||||
|
||||
function answer(request: HTTPRequest, frames: string): void {
|
||||
const url = new URL(request.url());
|
||||
const hit = url.origin === ORIGIN ? pageFile(url.pathname, frames) : null;
|
||||
const sent = hit
|
||||
? request.respond({
|
||||
status: 200,
|
||||
contentType: hit.type,
|
||||
body: hit.file ? readFileSync(hit.file) : (hit.text ?? ""),
|
||||
})
|
||||
: request.abort();
|
||||
// A request still open when the page closes has nobody left to answer.
|
||||
sent.catch(() => {});
|
||||
}
|
||||
|
||||
async function openHandsPage(page: Page, frames: string): Promise<void> {
|
||||
await page.setRequestInterception(true);
|
||||
page.on("request", (request) => answer(request, frames));
|
||||
try {
|
||||
await page.goto(`${ORIGIN}/`);
|
||||
await page.evaluate("loadHands()");
|
||||
} catch (err) {
|
||||
throw new Fix(
|
||||
`the hand landmarker could not start in Chrome (${err instanceof Error ? err.message.split("\n")[0] : String(err)})`,
|
||||
"run `hyperframes doctor` to check Chrome, then run the command again",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** The landmarker run over the frames in order as a video, so it carries a hand from one frame to the next. */
|
||||
async function detectHands(
|
||||
dir: string,
|
||||
frames: string[],
|
||||
fps: number,
|
||||
onStatus?: (message: string) => void,
|
||||
): Promise<Hand[][]> {
|
||||
await ensurePinnedFiles(MEDIAPIPE_DIR, CODE_ROOT, FILES, onStatus);
|
||||
const { ensureBrowser } = await import("../browser/manager.js");
|
||||
const { closeManagedBrowser, launchManagedBrowser } = await import("../browser/launch.js");
|
||||
const { buildChromeArgs } = await import("@hyperframes/engine");
|
||||
const puppeteer = await import("puppeteer-core");
|
||||
const chrome = await ensureBrowser();
|
||||
const browser = await launchManagedBrowser(puppeteer.default, {
|
||||
headless: true,
|
||||
executablePath: chrome.executablePath,
|
||||
// Software WebGL: the frames only pass through it, and it is there on a machine with no GPU.
|
||||
args: buildChromeArgs(
|
||||
{ width: WIDTH, height: WIDTH, captureMode: "screenshot" },
|
||||
{ browserGpuMode: "software" },
|
||||
),
|
||||
});
|
||||
try {
|
||||
const page = await browser.newPage();
|
||||
await openHandsPage(page, dir);
|
||||
const found: Hand[][] = [];
|
||||
for (const [k, name] of frames.entries()) {
|
||||
const ms = Math.trunc((k * 1000) / fps);
|
||||
found.push(readHands(await page.evaluate(`handsIn(${JSON.stringify(name)}, ${ms})`)));
|
||||
}
|
||||
return found;
|
||||
} finally {
|
||||
await closeManagedBrowser(browser);
|
||||
}
|
||||
}
|
||||
|
||||
/** The hand's point from `start` to `end`, one entry every `step` seconds, and how many frames lost it. */
|
||||
export async function followHand(
|
||||
info: VideoInfo,
|
||||
start: number,
|
||||
end: number,
|
||||
opts: { point: HandPoint; which: WhichHand; fps: number },
|
||||
onStatus?: (message: string) => void,
|
||||
) {
|
||||
if (end - start > MAX_SECONDS)
|
||||
throw new Fix(
|
||||
`hands follows up to ${MAX_SECONDS} seconds at a time, not ${round(end - start, 3)}`,
|
||||
`--start ${start} --end ${round(start + MAX_SECONDS, 3)}`,
|
||||
);
|
||||
const fps = Math.max(1, Math.min(MAX_FPS, opts.fps));
|
||||
const dir = mkdtempSync(join(tmpdir(), "hf-hands-"));
|
||||
try {
|
||||
const frames = handFrames(info, start, end, fps, dir);
|
||||
const found = await detectHands(dir, frames, fps, onStatus);
|
||||
const points = handPath(found, opts.point, opts.which);
|
||||
return {
|
||||
window: [round(start, 3), round(end, 3)],
|
||||
step: round(1 / fps, 4),
|
||||
point: opts.point,
|
||||
which: opts.which,
|
||||
points,
|
||||
lost: points.filter((p) => p === null).length,
|
||||
};
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest";
|
||||
|
||||
const ownedByAnotherAccount = vi.hoisted(() => new Set<string>());
|
||||
const downloads = vi.hoisted(() => [] as string[]);
|
||||
const signals = vi.hoisted(() => [] as (AbortSignal | undefined)[]);
|
||||
vi.mock("node:fs", async (importOriginal) => {
|
||||
const fs = await importOriginal<typeof import("node:fs")>();
|
||||
const lstatSync = ((path: string, options?: object) => {
|
||||
@@ -16,8 +17,9 @@ vi.mock("node:fs", async (importOriginal) => {
|
||||
return { ...fs, lstatSync };
|
||||
});
|
||||
vi.mock("../cloud/download.js", () => ({
|
||||
downloadToFile: async (url: string) => {
|
||||
downloadToFile: async (url: string, _dest: string, options?: { signal?: AbortSignal }) => {
|
||||
downloads.push(url);
|
||||
signals.push(options?.signal);
|
||||
},
|
||||
}));
|
||||
|
||||
@@ -30,6 +32,7 @@ beforeEach(() => {
|
||||
afterEach(() => {
|
||||
ownedByAnotherAccount.clear();
|
||||
downloads.length = 0;
|
||||
signals.length = 0;
|
||||
vi.unstubAllEnvs();
|
||||
rmSync(cache, { recursive: true, force: true });
|
||||
});
|
||||
@@ -46,3 +49,12 @@ it.skipIf(!process.getuid)(
|
||||
expect(downloads).toEqual([]);
|
||||
},
|
||||
);
|
||||
|
||||
it("hands the caller's signal to the download, so a check that stops ends it", async () => {
|
||||
const { loadOpenCv } = await import("./opencv.js");
|
||||
const stop = new AbortController();
|
||||
await expect(loadOpenCv(undefined, stop.signal)).rejects.toThrow(
|
||||
"did not match its pinned size and sha256",
|
||||
);
|
||||
expect(signals).toEqual([stop.signal]);
|
||||
});
|
||||
|
||||
@@ -125,7 +125,10 @@ const isCv = (v: unknown): v is Cv =>
|
||||
|
||||
/** Every pinned file present and verified; the missing ones fetched. */
|
||||
// fallow-ignore-next-line complexity
|
||||
async function ensureOpenCv(onStatus?: (message: string) => void): Promise<void> {
|
||||
async function ensureOpenCv(
|
||||
onStatus?: (message: string) => void,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> {
|
||||
const missing: Pinned[] = [];
|
||||
for (const f of FILES) {
|
||||
const file = join(OPENCV_DIR, f.name);
|
||||
@@ -137,7 +140,7 @@ async function ensureOpenCv(onStatus?: (message: string) => void): Promise<void>
|
||||
onStatus?.(`Downloading ${f.name} (${Math.round(f.bytes / 1e5) / 10} MB, once)...`);
|
||||
const temp = join(OPENCV_DIR, `${f.name}.part-${process.pid}`);
|
||||
try {
|
||||
await downloadToFile(f.url, temp);
|
||||
await downloadToFile(f.url, temp, { signal });
|
||||
if (!(await verifies(temp, f)))
|
||||
throw new Fix(
|
||||
`${f.name} did not match its pinned size and sha256`,
|
||||
@@ -161,9 +164,12 @@ async function ensureOpenCv(onStatus?: (message: string) => void): Promise<void>
|
||||
let loaded: Promise<Cv> | null = null;
|
||||
|
||||
/** opencv.js, ready to call, with the face models in its file system. Loaded once per process. */
|
||||
export function loadOpenCv(onStatus?: (message: string) => void): Promise<Cv> {
|
||||
export function loadOpenCv(
|
||||
onStatus?: (message: string) => void,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Cv> {
|
||||
loaded ??= (async () => {
|
||||
await ensureOpenCv(onStatus);
|
||||
await ensureOpenCv(onStatus, signal);
|
||||
const mod: unknown = createRequire(import.meta.url)(join(OPENCV_DIR, "opencv.js"));
|
||||
if (typeof mod !== "object" || mod === null) throw new Error("opencv.js did not load");
|
||||
// Wait on its init hook, never on the module: it is a thenable that does not settle under await.
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
// Files a measuring tool needs but the CLI does not ship: fetched on first use into a folder under the trusted cache
|
||||
// root, each one checked against its pinned size and sha256 before it is used, and the folder sealed.
|
||||
import { mkdirSync, renameSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { foreignWritablePath, sealCacheInstall } from "@hyperframes/core/cache-dir";
|
||||
import { downloadToFile } from "../cloud/download.js";
|
||||
import { verifies, type ModelFile } from "../whisper/modelFile.js";
|
||||
import { Fix } from "./media.js";
|
||||
|
||||
export interface PinnedFile extends ModelFile {
|
||||
url: string;
|
||||
}
|
||||
|
||||
async function fetchPinned(dir: string, file: PinnedFile): Promise<void> {
|
||||
const temp = join(dir, `${file.name}.part-${process.pid}`);
|
||||
try {
|
||||
await downloadToFile(file.url, temp).catch((err: unknown) => {
|
||||
throw new Fix(
|
||||
`could not download ${file.name} (${err instanceof Error ? err.message : String(err)})`,
|
||||
"check the network and run the command again",
|
||||
);
|
||||
});
|
||||
if (!(await verifies(temp, file)))
|
||||
throw new Fix(
|
||||
`${file.name} did not match its pinned size and sha256`,
|
||||
"run the command again to retry the download",
|
||||
);
|
||||
renameSync(temp, join(dir, file.name));
|
||||
} finally {
|
||||
rmSync(temp, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
/** Every file present in `dir` (under the cache `root`) and verified; the missing ones fetched. */
|
||||
export async function ensurePinnedFiles(
|
||||
dir: string,
|
||||
root: string,
|
||||
files: readonly PinnedFile[],
|
||||
onStatus?: (message: string) => void,
|
||||
): Promise<void> {
|
||||
const wanted: PinnedFile[] = [];
|
||||
for (const file of files) {
|
||||
const path = join(dir, file.name);
|
||||
if (foreignWritablePath(path, root) || !(await verifies(path, file))) wanted.push(file);
|
||||
}
|
||||
if (!wanted.length) return;
|
||||
mkdirSync(dir, { recursive: true, mode: 0o700 });
|
||||
for (const file of wanted) {
|
||||
onStatus?.(`Downloading ${file.name} (${Math.round(file.bytes / 1e5) / 10} MB, once)...`);
|
||||
await fetchPinned(dir, file);
|
||||
}
|
||||
sealCacheInstall(dir, root);
|
||||
}
|
||||
@@ -2,7 +2,7 @@
|
||||
// fallow-ignore-file code-duplication
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { DEFAULT_CONFIG } from "@hyperframes/engine";
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import os, { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
@@ -21,6 +21,7 @@ import {
|
||||
preResolveHostileMediaProxies,
|
||||
dropFailedProxyEchoes,
|
||||
runBrowserCheck,
|
||||
servedProjectFile,
|
||||
collectSeekClock,
|
||||
mergeHelperDrafts,
|
||||
parseLayoutIssue,
|
||||
@@ -1378,6 +1379,37 @@ describe("mergeHelperDrafts", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("servedProjectFile", () => {
|
||||
const page = "http://127.0.0.1:4000/";
|
||||
let root: string;
|
||||
let projectDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "hf-served-file-"));
|
||||
projectDir = join(root, "project");
|
||||
mkdirSync(join(projectDir, "takes"), { recursive: true });
|
||||
writeFileSync(join(projectDir, "takes", "my take.mp4"), "");
|
||||
writeFileSync(join(root, "outside.mp4"), "");
|
||||
});
|
||||
afterEach(() => rmSync(root, { recursive: true, force: true }));
|
||||
|
||||
it("resolves a page-origin source to its project file, proxied or not", () => {
|
||||
const file = join(projectDir, "takes", "my take.mp4");
|
||||
expect(servedProjectFile(`${page}takes/my%20take.mp4`, page, projectDir)).toBe(file);
|
||||
expect(servedProjectFile(`${page}takes/my%20take.mp4?hf-proxy=h264`, page, projectDir)).toBe(
|
||||
file,
|
||||
);
|
||||
});
|
||||
|
||||
it("gives no file for a remote source, a missing file or a path outside the project", () => {
|
||||
const remote = "https://cdn.example.com/takes/my%20take.mp4";
|
||||
expect(servedProjectFile(remote, page, projectDir)).toBeNull();
|
||||
expect(servedProjectFile(`${page}takes/gone.mp4`, page, projectDir)).toBeNull();
|
||||
expect(servedProjectFile(`${page}..%2Foutside.mp4`, page, projectDir)).toBeNull();
|
||||
expect(servedProjectFile("", page, projectDir)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("dropFailedProxyEchoes", () => {
|
||||
it("keeps one finding for a failed proxy and drops the preview's repeats of it", () => {
|
||||
const at = (code: string, message: string, url?: string) =>
|
||||
|
||||
@@ -7,7 +7,7 @@ import { collectMotionBlurDrafts, dropMotionBlurConsoleEcho } from "./checkMotio
|
||||
import { checkSnapshotDirectory } from "./checkSnapshotDirectory.js";
|
||||
import { existsSync, writeFileSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { isAbsolute, join, relative, resolve } from "node:path";
|
||||
import type { Browser, Page } from "puppeteer-core";
|
||||
import {
|
||||
AUDIT_SEEK_OPTIONS,
|
||||
@@ -92,6 +92,7 @@ import type {
|
||||
RotationSample,
|
||||
RunAuditGrid,
|
||||
SeekClock,
|
||||
TextOverVideoCandidate,
|
||||
} from "./checkTypes.js";
|
||||
import type { ProjectDir } from "./project.js";
|
||||
|
||||
@@ -474,7 +475,7 @@ export async function runBrowserCheck(
|
||||
let chromeBrowser: import("puppeteer-core").Browser | undefined;
|
||||
const helperSession =
|
||||
samplingBrowserCount(options) > 1
|
||||
? startHelperSession(html, server.url, options, deadline)
|
||||
? startHelperSession(html, server.url, options, deadline, project.dir)
|
||||
: undefined;
|
||||
const helperOpen = helperSession?.opened;
|
||||
const helpers = helperOpen?.then(
|
||||
@@ -527,7 +528,7 @@ export async function runBrowserCheck(
|
||||
const blur = await collectMotionBlurDrafts(page);
|
||||
drafts.push(...blur);
|
||||
const sampleLog = sampleWindow(drafts);
|
||||
const driver = createPageDriver(page, clock, sampleLog);
|
||||
const driver = createPageDriver(page, clock, sampleLog, project.dir);
|
||||
const result = await runGrid(
|
||||
driver,
|
||||
options,
|
||||
@@ -536,6 +537,7 @@ export async function runBrowserCheck(
|
||||
deadline?.enter,
|
||||
logCheckProgress,
|
||||
deadline?.keepPartial,
|
||||
deadline?.signal,
|
||||
);
|
||||
helperSession?.stop();
|
||||
const helper = await helperOpen?.catch(() => undefined);
|
||||
@@ -611,10 +613,11 @@ function startHelperSession(
|
||||
url: string,
|
||||
options: CheckOptions,
|
||||
deadline: CheckDeadline | undefined,
|
||||
projectDir: string,
|
||||
) {
|
||||
const abort = new AbortController();
|
||||
let browser: Browser | undefined;
|
||||
const opened = openHelperSession(html, url, options, deadline, {
|
||||
const opened = openHelperSession(html, url, options, deadline, projectDir, {
|
||||
signal: deadline ? AbortSignal.any([deadline.signal, abort.signal]) : abort.signal,
|
||||
onBrowser: (launched) => {
|
||||
browser = launched;
|
||||
@@ -637,6 +640,7 @@ async function openHelperSession(
|
||||
url: string,
|
||||
options: CheckOptions,
|
||||
deadline: CheckDeadline | undefined,
|
||||
projectDir: string,
|
||||
owner: { signal: AbortSignal; onBrowser: (browser: Browser) => void },
|
||||
): Promise<HelperSession> {
|
||||
const sampleLog = sampleWindow([]);
|
||||
@@ -655,7 +659,7 @@ async function openHelperSession(
|
||||
});
|
||||
await waitForPreferredSeekTarget(session.page);
|
||||
return {
|
||||
driver: createPageDriver(session.page, clock, sampleLog),
|
||||
driver: createPageDriver(session.page, clock, sampleLog, projectDir),
|
||||
sampleLog,
|
||||
};
|
||||
}
|
||||
@@ -1089,6 +1093,7 @@ export function createPageDriver(
|
||||
page: Page,
|
||||
clock: RuntimeClock = createRuntimeClock(),
|
||||
sampleLog?: SampleWindow,
|
||||
projectDir?: string,
|
||||
): CheckAuditDriver {
|
||||
return failWhenPageDies(page, {
|
||||
initialize: (contrast) => injectAuditScripts(page, contrast),
|
||||
@@ -1142,6 +1147,9 @@ export function createPageDriver(
|
||||
collectSeekClock: () => collectSeekClock(page),
|
||||
collectRotationSample: (time) => collectRotationSample(page, time),
|
||||
collectOffPivotRotationSample: (time) => collectOffPivotRotationSample(page, time),
|
||||
...(projectDir === undefined
|
||||
? {}
|
||||
: { collectTextOverVideo: (time) => collectTextOverVideo(page, time, projectDir) }),
|
||||
collectGeometryCandidates: (time, request) => collectGeometryCandidates(page, time, request),
|
||||
collectMotionFrame: (time, selectors, scopes) =>
|
||||
collectMotionFrame(page, time, selectors, scopes),
|
||||
@@ -1572,6 +1580,75 @@ function parseRotationSample(value: unknown, time: number): RotationSample[] {
|
||||
return [{ time, selector, cx, cy, w, h, angle }];
|
||||
}
|
||||
|
||||
async function collectTextOverVideo(
|
||||
page: Page,
|
||||
time: number,
|
||||
projectDir: string,
|
||||
): Promise<TextOverVideoCandidate[]> {
|
||||
const raw = await page.evaluate(
|
||||
(options: { time: number }) => {
|
||||
const collect = Reflect.get(window, "__hyperframesTextOverVideo");
|
||||
if (typeof collect !== "function") return [];
|
||||
const result = Reflect.apply(collect, window, [options]);
|
||||
return Array.isArray(result) ? result : [];
|
||||
},
|
||||
{ time },
|
||||
);
|
||||
return raw.flatMap((value) => parseTextOverVideo(value, time, page.url(), projectDir));
|
||||
}
|
||||
|
||||
/** The project file a page-origin URL is served from; null for a remote source or a path outside the project. */
|
||||
export function servedProjectFile(src: string, pageUrl: string, projectDir: string): string | null {
|
||||
const url = parseUrl(src);
|
||||
const path = decodedPath(url);
|
||||
if (!url || !path || url.origin !== parseUrl(pageUrl)?.origin) return null;
|
||||
const file = resolve(projectDir, `.${path}`);
|
||||
const inside = relative(projectDir, file);
|
||||
if (!inside || inside.startsWith("..") || isAbsolute(inside)) return null;
|
||||
return existsSync(file) ? file : null;
|
||||
}
|
||||
|
||||
function parsePlacedVideo(
|
||||
value: Record<string, unknown> | null,
|
||||
pageUrl: string,
|
||||
projectDir: string,
|
||||
): TextOverVideoCandidate["video"] | null {
|
||||
if (!value) return null;
|
||||
const selector = stringValue(value, "selector");
|
||||
const file = servedProjectFile(stringValue(value, "src") ?? "", pageUrl, projectDir);
|
||||
const mediaTime = numberValue(value, "mediaTime");
|
||||
const box = parseBbox(Reflect.get(value, "box"));
|
||||
const frame = parseBbox(Reflect.get(value, "frame"));
|
||||
if (!selector || !file || mediaTime === null || !box || !frame) return null;
|
||||
return {
|
||||
selector,
|
||||
file,
|
||||
mediaTime,
|
||||
box,
|
||||
frame,
|
||||
flipX: booleanValue(value, "flipX") === true,
|
||||
flipY: booleanValue(value, "flipY") === true,
|
||||
};
|
||||
}
|
||||
|
||||
function parseTextOverVideo(
|
||||
value: unknown,
|
||||
time: number,
|
||||
pageUrl: string,
|
||||
projectDir: string,
|
||||
): TextOverVideoCandidate[] {
|
||||
if (!isRecord(value)) return [];
|
||||
const rect = parseRect(Reflect.get(value, "rect"));
|
||||
const anchor = rect ? parseGeometryAnchor(value, rect, time) : null;
|
||||
const video = parsePlacedVideo(recordField(value, "video"), pageUrl, projectDir);
|
||||
const rawWords: unknown = Reflect.get(value, "wordRects");
|
||||
const wordRects = Array.isArray(rawWords)
|
||||
? rawWords.flatMap((word) => parseBbox(word) ?? [])
|
||||
: [];
|
||||
if (!rect || !anchor || !video || wordRects.length === 0) return [];
|
||||
return [{ ...anchor, text: stringValue(value, "text") ?? "", rect, wordRects, video }];
|
||||
}
|
||||
|
||||
async function collectOffPivotRotationSample(page: Page, time: number): Promise<OffPivotFrame> {
|
||||
const raw = await evaluateSampler(page, "__hyperframesOffPivotRotationSample");
|
||||
return { time, samples: raw.flatMap(parseOffPivotRotationSample) };
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { DEFAULT_CHECK_OPTIONS, runAuditGrid } from "./checkPipeline.js";
|
||||
import type { CheckAuditDriver, TextOverVideoCandidate } from "./checkTypes.js";
|
||||
import type { VideoInfo } from "../measure/media.js";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
faceBoxesAt: vi.fn(),
|
||||
probeVideo: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("../measure/face.js", () => ({ faceBoxesAt: mocks.faceBoxesAt }));
|
||||
vi.mock("../measure/media.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../measure/media.js")>()),
|
||||
probeVideo: mocks.probeVideo,
|
||||
}));
|
||||
|
||||
const SAMPLES = [1, 4, 7];
|
||||
const OPTIONS = { ...DEFAULT_CHECK_OPTIONS, at: SAMPLES, contrast: false, craft: false };
|
||||
|
||||
function driver(overrides: Partial<CheckAuditDriver> = {}): CheckAuditDriver {
|
||||
let state = 0;
|
||||
return {
|
||||
initialize: async () => undefined,
|
||||
getDuration: async () => 9,
|
||||
hasNoTimelineDeclaration: async () => false,
|
||||
getTransitionBoundaries: async () => [],
|
||||
getClipEdges: async () => [],
|
||||
getCanvas: async () => ({ width: 1080, height: 1920 }),
|
||||
findAmbiguousSelectors: async () => [],
|
||||
seek: async () => undefined,
|
||||
seekGeometry: async () => undefined,
|
||||
collectLayout: async () => [],
|
||||
collectOverlap: async () => [],
|
||||
collectOverlapGrid: async (times) => times.map(() => []),
|
||||
collectLayoutGeometry: async () => `state-${state++}`,
|
||||
captureFrame: async () => "frame",
|
||||
framesDiffer: async () => true,
|
||||
collectSeekClock: async () => [{ id: 1, time: 0, done: false }],
|
||||
collectRotationSample: async () => [],
|
||||
collectOffPivotRotationSample: async (time) => ({ time, samples: [] }),
|
||||
collectGeometryCandidates: async () => [],
|
||||
collectMotionFrame: async (time) => ({ time, data: {}, liveness: {} }),
|
||||
anchorMotionIssues: async () => [],
|
||||
collectContrast: async () => ({ entries: [] }),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// A title over the middle of a full-canvas video; the video's clock runs half a second behind.
|
||||
function title(time: number): TextOverVideoCandidate {
|
||||
return {
|
||||
selector: "#title",
|
||||
dataAttributes: {},
|
||||
sourceFile: "index.html",
|
||||
bbox: { x: 340, y: 860, width: 400, height: 200 },
|
||||
time,
|
||||
text: "HELLO",
|
||||
rect: { left: 340, top: 860, right: 740, bottom: 1060, width: 400, height: 200 },
|
||||
wordRects: [{ x: 340, y: 860, width: 400, height: 200 }],
|
||||
video: {
|
||||
selector: "#take",
|
||||
file: "/project/takes/take.mp4",
|
||||
mediaTime: time - 0.5,
|
||||
box: { x: 0, y: 0, width: 1080, height: 1920 },
|
||||
frame: { x: 0, y: 0, width: 1080, height: 1920 },
|
||||
flipX: false,
|
||||
flipY: false,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const INFO = { path: "/project/takes/take.mp4", fps: 24, frames: 216 } as VideoInfo;
|
||||
// Centred, 216 px square: the title covers its whole width and 196 of its 216 px.
|
||||
const CENTRE_FACE = [0.4, 0.45, 0.2, 0.1125];
|
||||
|
||||
describe("text_over_face in the check grid", () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
mocks.faceBoxesAt.mockReset();
|
||||
mocks.probeVideo.mockReset();
|
||||
});
|
||||
|
||||
it("looks for faces once per video at the frames its samples showed, and reports the cover", async () => {
|
||||
mocks.probeVideo.mockReturnValue(INFO);
|
||||
// She faces the camera at the media times 0.5 s and 6.5 s (frames 12 and 156), not at 3.5 s.
|
||||
mocks.faceBoxesAt.mockImplementation(async (_info: VideoInfo, frames: number[]) =>
|
||||
frames.map((frame) => ({ frame, t: frame / 24, boxes: frame === 84 ? [] : [CENTRE_FACE] })),
|
||||
);
|
||||
const result = await runAuditGrid(
|
||||
driver({ collectTextOverVideo: async (time) => [title(time)] }),
|
||||
OPTIONS,
|
||||
{ kind: "none" },
|
||||
);
|
||||
|
||||
expect(mocks.probeVideo).toHaveBeenCalledTimes(1);
|
||||
expect(mocks.faceBoxesAt).toHaveBeenCalledTimes(1);
|
||||
expect(mocks.faceBoxesAt.mock.calls[0]?.[1]).toEqual([12, 84, 156]);
|
||||
const found = result.layoutIssues.filter((issue) => issue.code === "text_over_face");
|
||||
expect(found.map((issue) => issue.time).sort()).toEqual([1, 7]);
|
||||
expect(found[0]).toMatchObject({
|
||||
severity: "info",
|
||||
selector: "#title",
|
||||
containerSelector: "#take",
|
||||
coveredFraction: 0.91,
|
||||
faceRect: { x: 432, y: 864, width: 216, height: 216 },
|
||||
message: '"HELLO" covers 91% of a face in the video beneath it.',
|
||||
});
|
||||
expect(result.faceCheck).toEqual({ ran: true, candidates: 3, faces: 2 });
|
||||
});
|
||||
|
||||
it("hands the detector the signal that ends with the check", async () => {
|
||||
mocks.probeVideo.mockReturnValue(INFO);
|
||||
mocks.faceBoxesAt.mockResolvedValue([]);
|
||||
vi.spyOn(console, "error").mockImplementation(() => undefined);
|
||||
const stop = new AbortController();
|
||||
const noop = undefined;
|
||||
await runAuditGrid(
|
||||
driver({ collectTextOverVideo: async () => [title(1)] }),
|
||||
{ ...OPTIONS, at: [1] },
|
||||
{ kind: "none" },
|
||||
noop,
|
||||
noop,
|
||||
noop,
|
||||
noop,
|
||||
stop.signal,
|
||||
);
|
||||
const given: AbortSignal | undefined = mocks.faceBoxesAt.mock.calls[0]?.[2]?.signal;
|
||||
expect(given?.aborted).toBe(false);
|
||||
stop.abort();
|
||||
expect(given?.aborted).toBe(true);
|
||||
});
|
||||
|
||||
it("does not load the detector when no text is in front of a video", async () => {
|
||||
const result = await runAuditGrid(driver({ collectTextOverVideo: async () => [] }), OPTIONS, {
|
||||
kind: "none",
|
||||
});
|
||||
expect(mocks.probeVideo).not.toHaveBeenCalled();
|
||||
expect(mocks.faceBoxesAt).not.toHaveBeenCalled();
|
||||
expect(result.layoutIssues).toEqual([]);
|
||||
expect(result.faceCheck).toEqual({ ran: true, candidates: 0, faces: 0 });
|
||||
});
|
||||
|
||||
it("finishes without the finding when OpenCV cannot load, and says so once", async () => {
|
||||
const stderr = vi.spyOn(console, "error").mockImplementation(() => undefined);
|
||||
mocks.probeVideo.mockReturnValue(INFO);
|
||||
mocks.faceBoxesAt.mockRejectedValue(new Error("could not download opencv.js (fetch failed)"));
|
||||
const result = await runAuditGrid(
|
||||
driver({ collectTextOverVideo: async (time) => [title(time)] }),
|
||||
OPTIONS,
|
||||
{ kind: "none" },
|
||||
);
|
||||
expect(result.layoutIssues).toEqual([]);
|
||||
expect(result.faceCheck).toEqual({
|
||||
ran: false,
|
||||
reason: "could not download opencv.js (fetch failed)",
|
||||
});
|
||||
const said = stderr.mock.calls.filter(([line]) => String(line).includes("face"));
|
||||
expect(said).toEqual([
|
||||
[
|
||||
"[hyperframes] check: text over a face was not looked for in 1 of 1 video(s): could not download opencv.js (fetch failed)",
|
||||
],
|
||||
]);
|
||||
});
|
||||
|
||||
it("says the pass did not run on a driver that cannot report text over video", async () => {
|
||||
const result = await runAuditGrid(driver(), OPTIONS, { kind: "none" });
|
||||
expect(result.faceCheck).toEqual({
|
||||
ran: false,
|
||||
reason: "this check driver does not report text over video",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -90,12 +90,14 @@ import type {
|
||||
CraftFinding,
|
||||
CraftFrame,
|
||||
CraftResult,
|
||||
FaceCheck,
|
||||
GeometryCandidateRequest,
|
||||
MotionSpecResolution,
|
||||
OffPivotFrame,
|
||||
OffPivotRotationSample,
|
||||
RotationSample,
|
||||
SeekClock,
|
||||
TextOverVideoCandidate,
|
||||
} from "./checkTypes.js";
|
||||
|
||||
export type {
|
||||
@@ -313,6 +315,10 @@ interface GridSamples {
|
||||
* material-point geometry + dial hub per layout sample; flattened by selector
|
||||
* to detect off_pivot_rotation. */
|
||||
indicatorFrames: OffPivotFrame[];
|
||||
/** Text painted in front of a video at each layout sample; faces are looked for after the run. */
|
||||
textOverVideo: TextOverVideoCandidate[];
|
||||
/** No page that sampled could report text over video. */
|
||||
textOverVideoUnavailable: boolean;
|
||||
craftFrames: CraftFrame[];
|
||||
}
|
||||
|
||||
@@ -498,6 +504,8 @@ function emptyGridSamples(): GridSamples {
|
||||
layoutStateSignatures: [],
|
||||
rotationSamples: [],
|
||||
indicatorFrames: [],
|
||||
textOverVideo: [],
|
||||
textOverVideoUnavailable: false,
|
||||
craftFrames: [],
|
||||
};
|
||||
}
|
||||
@@ -512,6 +520,8 @@ function appendGridSamples(into: GridSamples, from: GridSamples): void {
|
||||
into.layoutStateSignatures.push(...from.layoutStateSignatures);
|
||||
into.rotationSamples.push(...from.rotationSamples);
|
||||
into.indicatorFrames.push(...from.indicatorFrames);
|
||||
into.textOverVideo.push(...from.textOverVideo);
|
||||
into.textOverVideoUnavailable ||= from.textOverVideoUnavailable;
|
||||
into.craftFrames.push(...from.craftFrames);
|
||||
}
|
||||
|
||||
@@ -549,6 +559,9 @@ async function collectSampleAt(
|
||||
});
|
||||
collected.rotationSamples.push(...(await driver.collectRotationSample(time)));
|
||||
collected.indicatorFrames.push(await driver.collectOffPivotRotationSample(time));
|
||||
if (driver.collectTextOverVideo) {
|
||||
collected.textOverVideo.push(...(await driver.collectTextOverVideo(time)));
|
||||
} else collected.textOverVideoUnavailable = true;
|
||||
}
|
||||
if (sets.canvas) {
|
||||
const geometryIssues = await collectGeometryAt(driver, options, grid, sets, sets.canvas, time);
|
||||
@@ -1588,6 +1601,24 @@ export function reportingProgress<T extends object>(
|
||||
});
|
||||
}
|
||||
|
||||
async function textOverFace(
|
||||
collected: GridSamples,
|
||||
onFile: () => void,
|
||||
stop?: AbortSignal,
|
||||
): Promise<{ issues: AnchoredLayoutIssue[]; faceCheck: FaceCheck }> {
|
||||
if (collected.textOverVideoUnavailable) {
|
||||
return {
|
||||
issues: [],
|
||||
faceCheck: { ran: false, reason: "this check driver does not report text over video" },
|
||||
};
|
||||
}
|
||||
if (collected.textOverVideo.length === 0) {
|
||||
return { issues: [], faceCheck: { ran: true, candidates: 0, faces: 0 } };
|
||||
}
|
||||
const { findTextOverFace } = await import("./textOverFace.js");
|
||||
return findTextOverFace(collected.textOverVideo, undefined, onFile, stop);
|
||||
}
|
||||
|
||||
export async function runAuditGrid(
|
||||
driver: CheckAuditDriver,
|
||||
options: CheckOptions,
|
||||
@@ -1596,6 +1627,7 @@ export async function runAuditGrid(
|
||||
enter: (phase: CheckPhase, detail?: string) => void = () => {},
|
||||
report: CheckProgress = () => {},
|
||||
keep: (read: () => Partial<CheckBrowserResult>) => void = () => {},
|
||||
stop?: AbortSignal,
|
||||
): Promise<CheckBrowserResult> {
|
||||
enter("samples");
|
||||
await driver.initialize(options.contrast);
|
||||
@@ -1653,6 +1685,7 @@ export async function runAuditGrid(
|
||||
await analysing.getCanvas(),
|
||||
);
|
||||
const offPivotFindings = detectOffPivotRotation(collected.indicatorFrames);
|
||||
const overFace = await textOverFace(collected, () => enter("analysis", "faces under text"), stop);
|
||||
const contrast = buildContrastResults(collected.contrastEntries);
|
||||
report("analyze", 2, 3);
|
||||
const craft = craftEnabled(analysing, options)
|
||||
@@ -1682,6 +1715,7 @@ export async function runAuditGrid(
|
||||
...sweepFindings,
|
||||
...rotationFindings,
|
||||
...offPivotFindings,
|
||||
...overFace.issues,
|
||||
...motionIssues,
|
||||
...contrast.findings,
|
||||
...craft.findings,
|
||||
@@ -1702,7 +1736,9 @@ export async function runAuditGrid(
|
||||
...sweepFindings,
|
||||
...rotationFindings,
|
||||
...offPivotFindings,
|
||||
...overFace.issues,
|
||||
],
|
||||
faceCheck: overFace.faceCheck,
|
||||
motionIssues,
|
||||
motionSampleCount: collected.motionFrames.length,
|
||||
contrastSamples: grid.contrastSamples,
|
||||
@@ -2224,6 +2260,8 @@ function filterComponentReport(report: CheckReport, options: CheckOptions): Chec
|
||||
return report;
|
||||
}
|
||||
|
||||
const FACE_CHECK_NOT_REACHED = "the check stopped before looking for faces under text";
|
||||
|
||||
function shapeLayoutSection(
|
||||
issues: AnchoredLayoutIssue[],
|
||||
browser: CheckBrowserResult,
|
||||
@@ -2239,6 +2277,7 @@ function shapeLayoutSection(
|
||||
tolerance: options.tolerance,
|
||||
totalIssueCount: shaped.totalIssueCount,
|
||||
truncated: shaped.truncated,
|
||||
faceCheck: browser.faceCheck ?? { ran: false, reason: FACE_CHECK_NOT_REACHED },
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -183,6 +183,26 @@ export interface CheckGeometryCandidate extends CheckAnchor {
|
||||
overflow?: LayoutOverflow;
|
||||
}
|
||||
|
||||
export interface TextOverVideoCandidate extends CheckAnchor {
|
||||
text: string;
|
||||
rect: LayoutRect;
|
||||
wordRects: CheckBbox[];
|
||||
video: {
|
||||
selector: string;
|
||||
file: string;
|
||||
mediaTime: number;
|
||||
box: CheckBbox;
|
||||
frame: CheckBbox;
|
||||
flipX: boolean;
|
||||
flipY: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
/** `candidates`: text-over-video samples examined. `faces`: distinct video frames a face was found in. */
|
||||
export type FaceCheck =
|
||||
| { ran: true; candidates: number; faces: number; unreadVideos?: number }
|
||||
| { ran: false; reason: string };
|
||||
|
||||
/** One rotatable element's geometry at a single seeked sample, produced by
|
||||
* `__hyperframesRotationSample` (layout-audit.browser.js) and accumulated
|
||||
* across the grid to detect `rotation_pivot_drift`. */
|
||||
@@ -294,6 +314,7 @@ export interface CheckAuditDriver {
|
||||
* geometry + resolved dial hub at the current seeked state, as a time-hoisted
|
||||
* frame envelope. */
|
||||
collectOffPivotRotationSample(time: number): Promise<OffPivotFrame>;
|
||||
collectTextOverVideo?(time: number): Promise<TextOverVideoCandidate[]>;
|
||||
collectGeometryCandidates(
|
||||
time: number,
|
||||
request: GeometryCandidateRequest,
|
||||
@@ -342,6 +363,7 @@ export interface CheckBrowserResult {
|
||||
transitionSamplesDropped: number;
|
||||
runtimeFindings: CheckFinding[];
|
||||
layoutIssues: AnchoredLayoutIssue[];
|
||||
faceCheck?: FaceCheck;
|
||||
motionIssues: AnchoredLayoutIssue[];
|
||||
motionSampleCount: number;
|
||||
contrastSamples: number[];
|
||||
@@ -383,6 +405,7 @@ export type RunAuditGrid = (
|
||||
enter?: (phase: CheckPhase) => void,
|
||||
report?: CheckProgress,
|
||||
keep?: (read: () => Partial<CheckBrowserResult>) => void,
|
||||
stop?: AbortSignal,
|
||||
) => Promise<CheckBrowserResult>;
|
||||
|
||||
export interface CheckSection<T extends CheckFinding = CheckFinding> {
|
||||
@@ -414,6 +437,7 @@ export interface CheckReport {
|
||||
tolerance: number;
|
||||
totalIssueCount: number;
|
||||
truncated: boolean;
|
||||
faceCheck: FaceCheck;
|
||||
};
|
||||
motion: CheckSection & { enabled: boolean; specPath?: string; samples: number };
|
||||
contrast: CheckSection<CheckContrastFinding> & {
|
||||
|
||||
@@ -19,6 +19,7 @@ export type LayoutIssueCode =
|
||||
| "content_overlap"
|
||||
| "visibility_escapes_segment"
|
||||
| "text_occluded"
|
||||
| "text_over_face"
|
||||
| "canvas_covered"
|
||||
| "video_alpha_unread"
|
||||
| "text_not_painted"
|
||||
@@ -66,9 +67,10 @@ export interface LayoutIssue {
|
||||
rect: LayoutRect;
|
||||
containerRect?: LayoutRect;
|
||||
overflow?: LayoutOverflow;
|
||||
/** `text_occluded` only (0-1): probe-grid share hit by an opaque occluder, or behind a
|
||||
* transparent video the most covered word's share under its alpha (layout-audit.browser.js). */
|
||||
/** 0-1. `text_occluded`: probe-grid share hit by an opaque occluder, or behind a transparent video
|
||||
* the most covered word's share under its alpha. `text_over_face`: the share of `faceRect` covered. */
|
||||
coveredFraction?: number;
|
||||
faceRect?: { x: number; y: number; width: number; height: number };
|
||||
/** `text_occluded` only: the global window the element's own clip is on screen. */
|
||||
lifetime?: { start: number; end: number };
|
||||
fixHint?: string;
|
||||
|
||||
@@ -0,0 +1,295 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import type { CheckBbox, TextOverVideoCandidate } from "./checkTypes.js";
|
||||
import {
|
||||
FACE_COVER_REPORT,
|
||||
coveredShare,
|
||||
faceRectOnCanvas,
|
||||
findTextOverFace,
|
||||
textOverFaceIssue,
|
||||
type FaceFinder,
|
||||
} from "./textOverFace.js";
|
||||
|
||||
const box = (x: number, y: number, width: number, height: number): CheckBbox => ({
|
||||
x,
|
||||
y,
|
||||
width,
|
||||
height,
|
||||
});
|
||||
|
||||
// A 768x1344 take cover-fitted into a 1080x1920 canvas: the frame is 1097.14 wide and spills
|
||||
// 8.57 px past each side of the box.
|
||||
const COVER = {
|
||||
box: box(0, 0, 1080, 1920),
|
||||
frame: box(-8.57, 0, 1097.14, 1920),
|
||||
flipX: false,
|
||||
flipY: false,
|
||||
};
|
||||
|
||||
function candidate(overrides: Partial<TextOverVideoCandidate> = {}): TextOverVideoCandidate {
|
||||
return {
|
||||
selector: "#title",
|
||||
dataAttributes: { "data-layer-id": "title" },
|
||||
sourceFile: "index.html",
|
||||
bbox: box(400, 760, 200, 100),
|
||||
time: 7,
|
||||
text: "15% OFF",
|
||||
rect: { left: 400, top: 760, right: 600, bottom: 860, width: 200, height: 100 },
|
||||
wordRects: [box(400, 760, 200, 100)],
|
||||
video: { selector: "#take", file: "/project/take.mp4", mediaTime: 7, ...COVER },
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// On COVER this face lands at x 430, y 768, 175.5 x 192.
|
||||
const FACE = [0.4, 0.4, 0.16, 0.1];
|
||||
|
||||
describe("faceRectOnCanvas", () => {
|
||||
it("places a face through the frame object-fit painted, not through the element's box", () => {
|
||||
const rect = faceRectOnCanvas(FACE, COVER);
|
||||
expect(rect?.x).toBeCloseTo(-8.57 + 0.4 * 1097.14, 2);
|
||||
expect(rect?.y).toBeCloseTo(768, 2);
|
||||
expect(rect?.width).toBeCloseTo(0.16 * 1097.14, 2);
|
||||
expect(rect?.height).toBeCloseTo(192, 2);
|
||||
});
|
||||
|
||||
it("follows a letterboxed frame inside its box", () => {
|
||||
const contain = { ...COVER, frame: box(0, 656.25, 1080, 607.5) };
|
||||
expect(faceRectOnCanvas([0.5, 0.5, 0.1, 0.2], contain)).toEqual(box(540, 960, 108, 121.5));
|
||||
});
|
||||
|
||||
it("mirrors the face with the video", () => {
|
||||
const fill = { ...COVER, frame: box(0, 0, 1000, 1000), box: box(0, 0, 1000, 1000) };
|
||||
expect(faceRectOnCanvas([0.125, 0.25, 0.25, 0.25], { ...fill, flipX: true })).toEqual(
|
||||
box(625, 250, 250, 250),
|
||||
);
|
||||
expect(faceRectOnCanvas([0.125, 0.25, 0.25, 0.25], { ...fill, flipY: true })).toEqual(
|
||||
box(125, 500, 250, 250),
|
||||
);
|
||||
});
|
||||
|
||||
it("cuts a face to what the video's box shows", () => {
|
||||
const cropped = { ...COVER, frame: box(-100, 0, 1280, 1920) };
|
||||
// The face spans x -100..156; the box shows 0..156, more than half of it.
|
||||
expect(faceRectOnCanvas([0, 0.4, 0.2, 0.1], cropped)).toEqual(box(0, 768, 156, 192));
|
||||
});
|
||||
|
||||
it("drops a face the box crops to less than half", () => {
|
||||
const cropped = { ...COVER, frame: box(-200, 0, 1480, 1920) };
|
||||
// The face spans x -200..96: only 96 of 296 px are inside the box.
|
||||
expect(faceRectOnCanvas([0, 0.4, 0.2, 0.1], cropped)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("coveredShare", () => {
|
||||
const face = box(100, 100, 200, 200);
|
||||
|
||||
it("is the share of the face under the text, not the share of the text on the face", () => {
|
||||
// A small caption wholly inside a large face hides little of it.
|
||||
expect(coveredShare([box(150, 150, 100, 40)], face)).toBeCloseTo(0.1, 6);
|
||||
// A large title that only clips the face hides a lot of it.
|
||||
expect(coveredShare([box(0, 0, 2000, 200)], face)).toBeCloseTo(0.5, 6);
|
||||
});
|
||||
|
||||
it("counts overlapping word boxes once", () => {
|
||||
const words = [box(100, 100, 150, 100), box(200, 100, 100, 100), box(200, 150, 100, 100)];
|
||||
// Their union inside the face is 200x100 plus the 100x50 below its right half.
|
||||
expect(coveredShare(words, face)).toBeCloseTo((200 * 100 + 100 * 50) / (200 * 200), 6);
|
||||
});
|
||||
|
||||
it("is 0 for text clear of the face and 1 for text over all of it", () => {
|
||||
expect(coveredShare([box(400, 100, 200, 200)], face)).toBe(0);
|
||||
expect(coveredShare([box(0, 0, 150, 500), box(150, 0, 500, 500)], face)).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("textOverFaceIssue", () => {
|
||||
it("builds the finding the contract names", () => {
|
||||
// The face is 430..605.5 x 768..960; the title covers its top 92 px across 170 px of it.
|
||||
const [issue] = textOverFaceIssue(candidate(), [FACE]);
|
||||
expect(issue).toEqual({
|
||||
code: "text_over_face",
|
||||
severity: "info",
|
||||
time: 7,
|
||||
selector: "#title",
|
||||
containerSelector: "#take",
|
||||
text: "15% OFF",
|
||||
message: '"15% OFF" covers 46% of a face in the video beneath it.',
|
||||
rect: { left: 400, top: 760, right: 600, bottom: 860, width: 200, height: 100 },
|
||||
coveredFraction: 0.46,
|
||||
faceRect: { x: 430.29, y: 768, width: 175.54, height: 192 },
|
||||
fixHint:
|
||||
"Move the text off the face, or mark an intended overlap with data-layout-allow-on-face.",
|
||||
dataAttributes: { "data-layer-id": "title" },
|
||||
sourceFile: "index.html",
|
||||
bbox: box(400, 760, 200, 100),
|
||||
});
|
||||
});
|
||||
|
||||
it("is info however much of the face is covered, so a face found in texture never fails --strict", () => {
|
||||
const grazing = candidate({ wordRects: [box(400, 760, 200, 40)] });
|
||||
expect(textOverFaceIssue(grazing, [FACE])[0]).toMatchObject({
|
||||
severity: "info",
|
||||
coveredFraction: 0.16,
|
||||
});
|
||||
const whole = candidate({ wordRects: [box(0, 0, 1080, 1920)] });
|
||||
expect(textOverFaceIssue(whole, [FACE])[0]).toMatchObject({
|
||||
severity: "info",
|
||||
coveredFraction: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it("reports nothing under 5% of the face", () => {
|
||||
expect(FACE_COVER_REPORT).toBe(0.05);
|
||||
const touching = candidate({ wordRects: [box(400, 760, 200, 15)] });
|
||||
expect(textOverFaceIssue(touching, [FACE])).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports the face the text covers most", () => {
|
||||
const clear = [0.8, 0.1, 0.1, 0.05];
|
||||
const [issue] = textOverFaceIssue(candidate(), [clear, FACE]);
|
||||
expect(issue?.faceRect?.x).toBeCloseTo(430.29, 2);
|
||||
});
|
||||
|
||||
it("reports nothing without a face", () => {
|
||||
expect(textOverFaceIssue(candidate(), [])).toEqual([]);
|
||||
});
|
||||
|
||||
it("shortens a long line in the message only", () => {
|
||||
const text = "A very long caption line that runs well past forty characters";
|
||||
const [issue] = textOverFaceIssue(candidate({ text }), [FACE]);
|
||||
expect(issue?.message).toBe(
|
||||
'"A very long caption line that runs we..." covers 46% of a face in the video beneath it.',
|
||||
);
|
||||
expect(issue?.text).toBe(text);
|
||||
});
|
||||
});
|
||||
|
||||
describe("findTextOverFace", () => {
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
const at = (time: number, file = "/project/take.mp4") =>
|
||||
candidate({ time, video: { ...candidate().video, file, mediaTime: time } });
|
||||
const finder = (faceTimes: number[]) =>
|
||||
vi.fn<FaceFinder>(
|
||||
async (_file, times) =>
|
||||
new Map(
|
||||
times.map((time) => [
|
||||
time,
|
||||
{ frame: Math.round(time * 24), boxes: faceTimes.includes(time) ? [FACE] : [] },
|
||||
]),
|
||||
),
|
||||
);
|
||||
|
||||
it("reads each video once for its distinct times and reports only where a face is", async () => {
|
||||
const facesIn = finder([1, 7]);
|
||||
const { issues, faceCheck } = await findTextOverFace(
|
||||
[at(1), at(4), at(7), at(7), at(7, "/project/other.mp4")],
|
||||
facesIn,
|
||||
);
|
||||
expect(facesIn.mock.calls.map(([file, times]) => [file, times])).toEqual([
|
||||
["/project/take.mp4", [1, 4, 7]],
|
||||
["/project/other.mp4", [7]],
|
||||
]);
|
||||
expect(issues.map((issue) => issue.time)).toEqual([1, 7, 7, 7]);
|
||||
expect(faceCheck).toEqual({ ran: true, candidates: 5, faces: 3 });
|
||||
});
|
||||
|
||||
it("puts the most covered sample first, so a collapsed finding carries the worst numbers", async () => {
|
||||
const grazing = { ...at(1), wordRects: [box(400, 760, 200, 40)] };
|
||||
const { issues } = await findTextOverFace([grazing, at(7)], finder([1, 7]));
|
||||
expect(issues.map((issue) => [issue.time, issue.coveredFraction])).toEqual([
|
||||
[7, 0.46],
|
||||
[1, 0.16],
|
||||
]);
|
||||
});
|
||||
|
||||
it("says the pass ran when it looked and found no face", async () => {
|
||||
const { issues, faceCheck } = await findTextOverFace([at(4)], finder([]));
|
||||
expect(issues).toEqual([]);
|
||||
expect(faceCheck).toEqual({ ran: true, candidates: 1, faces: 0 });
|
||||
});
|
||||
|
||||
it("survives a detector that cannot load: no finding, one stderr line, the reason kept", async () => {
|
||||
const stderr = vi.spyOn(console, "error").mockImplementation(() => undefined);
|
||||
const facesIn = vi.fn<FaceFinder>(async () => {
|
||||
throw new Error("could not download opencv.js (fetch failed)");
|
||||
});
|
||||
const result = await findTextOverFace([at(1), at(7, "/project/other.mp4")], facesIn);
|
||||
expect(result).toEqual({
|
||||
issues: [],
|
||||
faceCheck: { ran: false, reason: "could not download opencv.js (fetch failed)" },
|
||||
});
|
||||
expect(stderr).toHaveBeenCalledTimes(1);
|
||||
expect(stderr.mock.calls[0]?.[0]).toBe(
|
||||
"[hyperframes] check: text over a face was not looked for in 2 of 2 video(s): could not download opencv.js (fetch failed)",
|
||||
);
|
||||
});
|
||||
|
||||
// A detector whose first-use download never answers: it settles only when its signal ends.
|
||||
const neverReady = () =>
|
||||
vi.fn<FaceFinder>(
|
||||
(_file, _times, signal) =>
|
||||
new Promise((_resolve, reject) => {
|
||||
signal?.addEventListener("abort", () => reject(signal.reason));
|
||||
}),
|
||||
);
|
||||
|
||||
it("gives up on a detector that is not ready within 30 s and says how to get it", async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
vi.spyOn(console, "error").mockImplementation(() => undefined);
|
||||
const facesIn = neverReady();
|
||||
const pending = findTextOverFace([at(1)], facesIn);
|
||||
await vi.advanceTimersByTimeAsync(29_999);
|
||||
expect(facesIn.mock.calls[0]?.[2]?.aborted).toBe(false);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(await pending).toEqual({
|
||||
issues: [],
|
||||
faceCheck: {
|
||||
ran: false,
|
||||
reason:
|
||||
"no answer within 30 s; `hyperframes measure face <video>` downloads it with no time limit",
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("stops the detector when the check stops", async () => {
|
||||
vi.spyOn(console, "error").mockImplementation(() => undefined);
|
||||
const stop = new AbortController();
|
||||
const pending = findTextOverFace([at(1)], neverReady(), undefined, stop.signal);
|
||||
stop.abort(new Error("Check stopped before finishing (SIGINT)."));
|
||||
expect((await pending).faceCheck).toEqual({
|
||||
ran: false,
|
||||
reason: "Check stopped before finishing (SIGINT).",
|
||||
});
|
||||
});
|
||||
|
||||
it("leaves no timer running once the faces are read", async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
await findTextOverFace([at(7)], finder([7]));
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps the findings of the videos it could read and counts the ones it could not", async () => {
|
||||
vi.spyOn(console, "error").mockImplementation(() => undefined);
|
||||
const readable = finder([7]);
|
||||
const facesIn = vi.fn<FaceFinder>(async (file, times) => {
|
||||
if (file === "/project/other.mp4") throw new Error("ffprobe cannot read it");
|
||||
return readable(file, times);
|
||||
});
|
||||
const { issues, faceCheck } = await findTextOverFace(
|
||||
[at(7), at(7, "/project/other.mp4")],
|
||||
facesIn,
|
||||
);
|
||||
expect(issues).toHaveLength(1);
|
||||
expect(faceCheck).toEqual({ ran: true, candidates: 2, faces: 1, unreadVideos: 1 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,212 @@
|
||||
// text_over_face: text painted in front of a video, over a face in it. The page reports the text's
|
||||
// word boxes and where the video's frame lands (layout-audit.browser.js); the faces come from the
|
||||
// file itself, with the detector `measure face` uses, because headless Chrome has none.
|
||||
import type {
|
||||
AnchoredLayoutIssue,
|
||||
CheckBbox,
|
||||
FaceCheck,
|
||||
TextOverVideoCandidate,
|
||||
} from "./checkTypes.js";
|
||||
import { normalizeErrorMessage } from "./errorMessage.js";
|
||||
|
||||
/** Below this share of the face a descender or a letter's corner touches it, which hides nothing. */
|
||||
export const FACE_COVER_REPORT = 0.05;
|
||||
// A face the video's own box crops to less than half is not one the viewer is shown.
|
||||
const FACE_SHOWN_MIN = 0.5;
|
||||
const MESSAGE_TEXT_MAX = 40;
|
||||
// The detector's first use downloads OpenCV: a network that never answers costs the finding, not the check.
|
||||
const DETECTOR_READY_MS = 30_000;
|
||||
const DETECTOR_LATE = `no answer within ${DETECTOR_READY_MS / 1000} s; \`hyperframes measure face <video>\` downloads it with no time limit`;
|
||||
|
||||
type VideoPlacement = Pick<TextOverVideoCandidate["video"], "box" | "frame" | "flipX" | "flipY">;
|
||||
|
||||
/** Each asked media time's frame, and that frame's faces as [x, y, w, h] shares of it. */
|
||||
export type FaceFinder = (
|
||||
file: string,
|
||||
times: number[],
|
||||
signal?: AbortSignal,
|
||||
) => Promise<Map<number, { frame: number; boxes: number[][] }>>;
|
||||
|
||||
const area = (box: CheckBbox) => box.width * box.height;
|
||||
|
||||
function intersect(a: CheckBbox, b: CheckBbox): CheckBbox | null {
|
||||
const x = Math.max(a.x, b.x);
|
||||
const y = Math.max(a.y, b.y);
|
||||
const width = Math.min(a.x + a.width, b.x + b.width) - x;
|
||||
const height = Math.min(a.y + a.height, b.y + b.height) - y;
|
||||
return width > 0 && height > 0 ? { x, y, width, height } : null;
|
||||
}
|
||||
|
||||
/** A face given in shares of the video frame, in canvas pixels, cut to what the video's box shows. */
|
||||
export function faceRectOnCanvas(face: readonly number[], video: VideoPlacement): CheckBbox | null {
|
||||
const [fx = 0, fy = 0, fw = 0, fh = 0] = face;
|
||||
const { frame } = video;
|
||||
const whole = {
|
||||
x: frame.x + (video.flipX ? 1 - fx - fw : fx) * frame.width,
|
||||
y: frame.y + (video.flipY ? 1 - fy - fh : fy) * frame.height,
|
||||
width: fw * frame.width,
|
||||
height: fh * frame.height,
|
||||
};
|
||||
const shown = intersect(whole, video.box);
|
||||
return shown && area(shown) >= FACE_SHOWN_MIN * area(whole) ? shown : null;
|
||||
}
|
||||
|
||||
/** The share of `face` under the union of `rects`: overlapping word boxes count once. */
|
||||
export function coveredShare(rects: readonly CheckBbox[], face: CheckBbox): number {
|
||||
const inside = rects.flatMap((rect) => intersect(rect, face) ?? []);
|
||||
const edges = [...new Set(inside.flatMap((rect) => [rect.x, rect.x + rect.width]))].sort(
|
||||
(a, b) => a - b,
|
||||
);
|
||||
let covered = 0;
|
||||
edges.slice(1).forEach((right, index) => {
|
||||
const left = edges[index] ?? right;
|
||||
const spans = inside
|
||||
.filter((rect) => rect.x <= left && rect.x + rect.width >= right)
|
||||
.map((rect) => ({ top: rect.y, bottom: rect.y + rect.height }))
|
||||
.sort((a, b) => a.top - b.top);
|
||||
let reached = Number.NEGATIVE_INFINITY;
|
||||
for (const { top, bottom } of spans) {
|
||||
if (bottom <= reached) continue;
|
||||
covered += (bottom - Math.max(top, reached)) * (right - left);
|
||||
reached = bottom;
|
||||
}
|
||||
});
|
||||
return area(face) > 0 ? covered / area(face) : 0;
|
||||
}
|
||||
|
||||
const round2 = (value: number) => Math.round(value * 100) / 100;
|
||||
|
||||
function quoted(text: string): string {
|
||||
return `"${text.length > MESSAGE_TEXT_MAX ? `${text.slice(0, MESSAGE_TEXT_MAX - 3)}...` : text}"`;
|
||||
}
|
||||
|
||||
/** The finding for the face this text covers most, or none when it covers no face enough. */
|
||||
export function textOverFaceIssue(
|
||||
candidate: TextOverVideoCandidate,
|
||||
faces: readonly (readonly number[])[],
|
||||
): AnchoredLayoutIssue[] {
|
||||
let most: { faceRect: CheckBbox; coveredFraction: number } | null = null;
|
||||
for (const face of faces) {
|
||||
const faceRect = faceRectOnCanvas(face, candidate.video);
|
||||
if (!faceRect) continue;
|
||||
const coveredFraction = round2(coveredShare(candidate.wordRects, faceRect));
|
||||
if (!most || coveredFraction > most.coveredFraction) most = { faceRect, coveredFraction };
|
||||
}
|
||||
if (!most || most.coveredFraction < FACE_COVER_REPORT) return [];
|
||||
const { faceRect, coveredFraction } = most;
|
||||
return [
|
||||
{
|
||||
code: "text_over_face",
|
||||
// The detector can take texture for a face, so a cover is reported and never fails a strict check.
|
||||
severity: "info",
|
||||
time: candidate.time,
|
||||
selector: candidate.selector,
|
||||
containerSelector: candidate.video.selector,
|
||||
text: candidate.text,
|
||||
message: `${quoted(candidate.text)} covers ${Math.round(coveredFraction * 100)}% of a face in the video beneath it.`,
|
||||
rect: candidate.rect,
|
||||
coveredFraction,
|
||||
faceRect: {
|
||||
x: round2(faceRect.x),
|
||||
y: round2(faceRect.y),
|
||||
width: round2(faceRect.width),
|
||||
height: round2(faceRect.height),
|
||||
},
|
||||
fixHint:
|
||||
"Move the text off the face, or mark an intended overlap with data-layout-allow-on-face.",
|
||||
dataAttributes: candidate.dataAttributes,
|
||||
sourceFile: candidate.sourceFile,
|
||||
bbox: candidate.bbox,
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
// Loaded on first use: a check with no text in front of a video never pays for OpenCV.
|
||||
const facesInFile: FaceFinder = async (file, times, signal) => {
|
||||
const [{ faceBoxesAt }, { frameAt, probeVideo }] = await Promise.all([
|
||||
import("../measure/face.js"),
|
||||
import("../measure/media.js"),
|
||||
]);
|
||||
const info = probeVideo(file);
|
||||
const frameOf = (time: number) => Math.min(Math.max(0, info.frames - 1), frameAt(time, info.fps));
|
||||
const frames = [...new Set(times.map(frameOf))].sort((a, b) => a - b);
|
||||
const found = await faceBoxesAt(info, frames, { signal });
|
||||
// A frame that did not decode shifts every later one, so the boxes no longer belong to their times.
|
||||
if (found.length !== frames.length) throw new Error(`could not read every frame of ${file}`);
|
||||
const byFrame = new Map(found.map((sample) => [sample.frame, sample.boxes]));
|
||||
return new Map(
|
||||
times.map((time) => [time, { frame: frameOf(time), boxes: byFrame.get(frameOf(time)) ?? [] }]),
|
||||
);
|
||||
};
|
||||
|
||||
type FacesByTime = Awaited<ReturnType<FaceFinder>>;
|
||||
|
||||
async function readFaces(
|
||||
timesByFile: Map<string, Set<number>>,
|
||||
facesIn: FaceFinder,
|
||||
onFile: () => void,
|
||||
stop?: AbortSignal,
|
||||
): Promise<{ faces: Map<string, FacesByTime>; unread: string[] }> {
|
||||
const faces = new Map<string, FacesByTime>();
|
||||
const unread: string[] = [];
|
||||
const late = new AbortController();
|
||||
const timer = setTimeout(() => late.abort(new Error(DETECTOR_LATE)), DETECTOR_READY_MS);
|
||||
const signal = stop ? AbortSignal.any([stop, late.signal]) : late.signal;
|
||||
for (const [file, times] of timesByFile) {
|
||||
onFile();
|
||||
await facesIn(file, [...times], signal).then(
|
||||
(found) => faces.set(file, found),
|
||||
(error: unknown) => unread.push(normalizeErrorMessage(error)),
|
||||
);
|
||||
}
|
||||
clearTimeout(timer);
|
||||
return { faces, unread };
|
||||
}
|
||||
|
||||
/**
|
||||
* The text_over_face findings for what the page saw in front of videos, and whether the pass
|
||||
* could look. Each video file is read once, for the distinct frames its samples landed on. A file
|
||||
* or a detector that cannot be read costs the finding, never the check: stderr says so in one line.
|
||||
*/
|
||||
export async function findTextOverFace(
|
||||
candidates: readonly TextOverVideoCandidate[],
|
||||
facesIn: FaceFinder = facesInFile,
|
||||
onFile: () => void = () => {},
|
||||
stop?: AbortSignal,
|
||||
): Promise<{ issues: AnchoredLayoutIssue[]; faceCheck: FaceCheck }> {
|
||||
const timesByFile = new Map<string, Set<number>>();
|
||||
for (const { video } of candidates) {
|
||||
const times = timesByFile.get(video.file) ?? new Set<number>();
|
||||
timesByFile.set(video.file, times.add(video.mediaTime));
|
||||
}
|
||||
const { faces, unread } = await readFaces(timesByFile, facesIn, onFile, stop);
|
||||
const reason = unread[0];
|
||||
if (reason !== undefined) {
|
||||
console.error(
|
||||
`[hyperframes] check: text over a face was not looked for in ${unread.length} of ${timesByFile.size} video(s): ${reason}`,
|
||||
);
|
||||
if (faces.size === 0) return { issues: [], faceCheck: { ran: false, reason } };
|
||||
}
|
||||
const framesWithFace = new Set<string>();
|
||||
for (const [file, byTime] of faces) {
|
||||
for (const { frame, boxes } of byTime.values()) {
|
||||
if (boxes.length > 0) framesWithFace.add(`${file}\0${frame}`);
|
||||
}
|
||||
}
|
||||
const issues = candidates.flatMap((candidate) =>
|
||||
textOverFaceIssue(
|
||||
candidate,
|
||||
faces.get(candidate.video.file)?.get(candidate.video.mediaTime)?.boxes ?? [],
|
||||
),
|
||||
);
|
||||
return {
|
||||
// Worst first: a finding collapsed over time keeps the numbers of its first sample.
|
||||
issues: issues.sort((a, b) => (b.coveredFraction ?? 0) - (a.coveredFraction ?? 0)),
|
||||
faceCheck: {
|
||||
ran: true,
|
||||
candidates: candidates.length,
|
||||
faces: framesWithFace.size,
|
||||
...(unread.length > 0 ? { unreadVideos: unread.length } : {}),
|
||||
},
|
||||
};
|
||||
}
|
||||
Reference in New Issue
Block a user