mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-10-30 09:16:55 +00:00
feat(cli): judge text behind a transparent video by its decoded alpha in check (#5657)
* feat(cli): judge text behind a transparent video by its decoded alpha in check The layout audit treated every <video> as opaque over its whole box, so a caption between footage and a remove-background cutout always failed text_occluded and needed data-layout-allow-occlusion, which also hid a word the subject's head fully covered. check now draws the video's current (seeked) frame into one reused, downscaled scratch canvas, once per sample and only for a video a text probe lands under. A fully opaque frame keeps today's behaviour. A frame with transparency is judged per rendered word against its alpha: partial cover is an info finding carrying the most covered word's share, and a word 80% hidden or two neighbouring words each half hidden is an error. A frame that cannot be read falls back to opaque with a video_alpha_unread info note. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(cli): serve the video-alpha fixtures from memory, not request paths Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(cli): check finds words under narrow, mirrored or backed cutouts Review follow-ups on #5657: - A matte the probe walk passes over, even at clear-alpha points, now triggers the per-word check, so a feature between two probe columns (an arm, a mic stand) still fails the word it hides. - Mirrored videos and images map viewport points through the flip; a rotated, skewed or 3D-transformed video falls back to opaque with video_alpha_unread. - A video with an opaque CSS background keeps its whole box occluding. - cli.mdx lists every unread cause and why a fully transparent frame is one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
WaterrrForever
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
08ee1bd5ad
commit
cea8aa8a7d
@@ -84,6 +84,11 @@ The plate keeps the original surroundings and leaves a transparent
|
||||
person-shaped hole. It is not an inpainted empty background. Place an opaque
|
||||
graphic or scene beneath the hole, then place `subject.webm` above it.
|
||||
|
||||
`hyperframes check` judges text behind the cutout by the cutout's own alpha at
|
||||
each sample time. A word partly behind the subject is reported as info with the
|
||||
share it hides; a word 80% hidden, or two neighbouring words each half hidden,
|
||||
is an error. You do not need `data-layout-allow-occlusion` for this layering.
|
||||
|
||||
## Know when it will work well
|
||||
|
||||
The included model is designed for people. It works best with a clear human
|
||||
|
||||
@@ -857,6 +857,21 @@ cover every finding from the full check.
|
||||
configured seek samples falls inside `--range`; a requested gate never silently
|
||||
passes without running.
|
||||
|
||||
Text under a `<video>` is judged by the frame the video shows at that sample.
|
||||
An opaque video covers its whole box, as an opaque image does. A video with
|
||||
transparency, such as a `remove-background` cutout, covers only where its
|
||||
alpha is: a word partly behind the subject is an info finding whose
|
||||
`coveredFraction` is the most covered word's share, and a word 80% hidden or two
|
||||
neighbouring words each half hidden are an error. A CSS background on the video
|
||||
still covers its whole box, and a mirrored video (`scaleX(-1)`) is read on the
|
||||
side it is painted. When a video's frame cannot be read, the audit treats the
|
||||
video as opaque and adds a `video_alpha_unread` info finding saying why: no frame
|
||||
decoded yet, the seek still landing, a cross-origin source, no frame size, a
|
||||
rotated, skewed or 3D-transformed video, or a frame that reads back fully
|
||||
transparent. A fully transparent frame is treated as unread on purpose, since a
|
||||
failed draw looks the same; mark text that sits under a cutout before its subject
|
||||
enters with `data-layout-allow-occlusion`.
|
||||
|
||||
Contrast failures are errors, and each one reports the sampled foreground and
|
||||
background colours, the measured ratio against the required one, and a
|
||||
compliant colour you could use instead.
|
||||
|
||||
@@ -64,7 +64,10 @@ Two more ways an SVG "line draws itself" effect (animated `strokeDasharray` / `s
|
||||
`hyperframes check` flags a text block covered by another element as `text_occluded`, and
|
||||
two attributes waive it: `data-layout-allow-overlap` for intentional layering, and
|
||||
`data-layout-allow-occlusion` for something deliberately painted over type. Both are
|
||||
legitimate — a caption designed to sit behind a matted subject needs one.
|
||||
legitimate. A caption behind a transparent `<video>` cutout needs neither: `check` reads the
|
||||
cutout's decoded alpha at each sample, reports how much of each word the subject covers as
|
||||
info, and fails only a word that is 80% hidden or two neighbouring words that are each half
|
||||
hidden.
|
||||
|
||||
Two things about them are worth knowing before you reach for either:
|
||||
|
||||
|
||||
@@ -1026,14 +1026,46 @@
|
||||
return { x: objectPositionOffset(x, freeX), y: objectPositionOffset(y, freeY) };
|
||||
}
|
||||
|
||||
// Return the alpha painted by an <img> at a viewport point. `null` means the
|
||||
// browser would not let us inspect the image (not loaded or cross-origin), in
|
||||
// which case callers preserve the conservative opaque fallback.
|
||||
function imageAlphaAt(element, x, y) {
|
||||
const sourceWidth = element.naturalWidth;
|
||||
const sourceHeight = element.naturalHeight;
|
||||
const IDENTITY_FLIP = { flipX: false, flipY: false };
|
||||
|
||||
// 2D CSS matrix components [a, b, c, d], or null when the transform is 3D or unparsed.
|
||||
function transformMatrix(value) {
|
||||
if (!value || value === "none") return [1, 0, 0, 1];
|
||||
const match = /^matrix\(([^)]*)\)$/.exec(value.trim());
|
||||
if (!match) return null;
|
||||
const parts = match[1].split(",").map(Number);
|
||||
return parts.length === 6 && parts.every(Number.isFinite) ? parts.slice(0, 4) : null;
|
||||
}
|
||||
|
||||
// Which axes the element's paint is mirrored on, or null when a rotation, skew or 3D transform
|
||||
// (on it or an ancestor) means its bounding box no longer maps axis-aligned onto its pixels.
|
||||
function paintFlips(element) {
|
||||
let flipX = false;
|
||||
let flipY = false;
|
||||
for (let current = element; current; current = current.parentElement) {
|
||||
const style = getComputedStyle(current);
|
||||
const matrix = transformMatrix(style.transform);
|
||||
if (!matrix) return null;
|
||||
const [a, b, c, d] = matrix;
|
||||
if (Math.abs(b) > 1e-6 || Math.abs(c) > 1e-6) return null;
|
||||
if (style.rotate && style.rotate !== "none" && parseFloat(style.rotate) !== 0) return null;
|
||||
const scale = String(style.scale || "none")
|
||||
.trim()
|
||||
.split(/\s+/)
|
||||
.map(Number);
|
||||
const scaleX = Number.isFinite(scale[0]) ? scale[0] : 1;
|
||||
const scaleY = Number.isFinite(scale[1]) ? scale[1] : scaleX;
|
||||
if (a * scaleX < 0) flipX = !flipX;
|
||||
if (d * scaleY < 0) flipY = !flipY;
|
||||
}
|
||||
return flipX || flipY ? { flipX, flipY } : IDENTITY_FLIP;
|
||||
}
|
||||
|
||||
function paintedSourceMapping(element, sourceWidth, sourceHeight) {
|
||||
const rect = element.getBoundingClientRect();
|
||||
if (!sourceWidth || !sourceHeight || !rect.width || !rect.height) return null;
|
||||
const flips = paintFlips(element);
|
||||
if (!flips) return null;
|
||||
|
||||
const style = getComputedStyle(element);
|
||||
const fit = style.objectFit || "fill";
|
||||
@@ -1061,9 +1093,48 @@
|
||||
rect.width - paintedWidth,
|
||||
rect.height - paintedHeight,
|
||||
);
|
||||
const localX = x - rect.left - offsets.x;
|
||||
const localY = y - rect.top - offsets.y;
|
||||
if (localX < 0 || localY < 0 || localX >= paintedWidth || localY >= paintedHeight) return 0;
|
||||
return {
|
||||
rect: { left: rect.left, top: rect.top, right: rect.right, bottom: rect.bottom },
|
||||
offsets,
|
||||
...flips,
|
||||
paintedWidth,
|
||||
paintedHeight,
|
||||
scaleX,
|
||||
scaleY,
|
||||
sourceWidth,
|
||||
sourceHeight,
|
||||
};
|
||||
}
|
||||
|
||||
// The source pixel painted at a viewport point, or null outside the painted content box.
|
||||
function sourcePixelAt(mapping, x, y) {
|
||||
const { rect, offsets } = mapping;
|
||||
const localX = (mapping.flipX ? rect.right - x : x - rect.left) - offsets.x;
|
||||
const localY = (mapping.flipY ? rect.bottom - y : y - rect.top) - offsets.y;
|
||||
if (
|
||||
localX < 0 ||
|
||||
localY < 0 ||
|
||||
localX >= mapping.paintedWidth ||
|
||||
localY >= mapping.paintedHeight
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
x: Math.min(mapping.sourceWidth - 1, Math.max(0, Math.floor(localX / mapping.scaleX))),
|
||||
y: Math.min(mapping.sourceHeight - 1, Math.max(0, Math.floor(localY / mapping.scaleY))),
|
||||
};
|
||||
}
|
||||
|
||||
// Return the alpha painted by an <img> at a viewport point. `null` means the
|
||||
// browser would not let us inspect the image (not loaded or cross-origin), in
|
||||
// which case callers preserve the conservative opaque fallback.
|
||||
function imageAlphaAt(element, x, y) {
|
||||
const sourceWidth = element.naturalWidth;
|
||||
const sourceHeight = element.naturalHeight;
|
||||
const mapping = paintedSourceMapping(element, sourceWidth, sourceHeight);
|
||||
if (!mapping) return null;
|
||||
const pixel = sourcePixelAt(mapping, x, y);
|
||||
if (!pixel) return 0;
|
||||
|
||||
try {
|
||||
let cached = imageAlphaCanvases.get(element);
|
||||
@@ -1083,25 +1154,129 @@
|
||||
cached = { context, width: sourceWidth, height: sourceHeight, source };
|
||||
imageAlphaCanvases.set(element, cached);
|
||||
}
|
||||
const sourceX = Math.min(sourceWidth - 1, Math.max(0, Math.floor(localX / scaleX)));
|
||||
const sourceY = Math.min(sourceHeight - 1, Math.max(0, Math.floor(localY / scaleY)));
|
||||
return cached.context.getImageData(sourceX, sourceY, 1, 1).data[3] / 255;
|
||||
return cached.context.getImageData(pixel.x, pixel.y, 1, 1).data[3] / 255;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
const OCCLUDING_ALPHA = 0.6;
|
||||
const VIDEO_ALPHA_MAX_SIDE = 480;
|
||||
let videoAlphaFrames = new Map();
|
||||
let videoAlphaScratch = null;
|
||||
|
||||
function videoAlphaContext(width, height) {
|
||||
if (!videoAlphaScratch) {
|
||||
const canvas = document.createElement("canvas");
|
||||
const context = canvas.getContext("2d", { willReadFrequently: true });
|
||||
if (!context) return null;
|
||||
videoAlphaScratch = { canvas, context };
|
||||
}
|
||||
videoAlphaScratch.canvas.width = width;
|
||||
videoAlphaScratch.canvas.height = height;
|
||||
return videoAlphaScratch.context;
|
||||
}
|
||||
|
||||
// Read once per audit call, so per sample time, and only for a video some text probe lands under.
|
||||
function videoAlphaFrame(video) {
|
||||
let frame = videoAlphaFrames.get(video);
|
||||
if (!frame) {
|
||||
frame = readVideoAlphaFrame(video);
|
||||
videoAlphaFrames.set(video, frame);
|
||||
}
|
||||
return frame;
|
||||
}
|
||||
|
||||
function unreadVideoFrame(reason) {
|
||||
return { kind: "unread", reason };
|
||||
}
|
||||
|
||||
// "opaque" keeps a video's whole box occluding; "matte" judges each point by the decoded
|
||||
// frame's alpha; "unread" falls back to opaque and is reported, never guessed.
|
||||
function readVideoAlphaFrame(video) {
|
||||
if (video.seeking) return unreadVideoFrame("the seek had not landed");
|
||||
if (video.readyState < 2) return unreadVideoFrame("no frame was decoded at this time");
|
||||
if (!video.videoWidth || !video.videoHeight)
|
||||
return unreadVideoFrame("the video has no frame size");
|
||||
const mapping = paintedSourceMapping(video, video.videoWidth, video.videoHeight);
|
||||
if (!mapping) return unreadVideoFrame("it is rotated, skewed or transformed in 3D");
|
||||
const downscale = Math.min(
|
||||
1,
|
||||
VIDEO_ALPHA_MAX_SIDE / Math.max(mapping.sourceWidth, mapping.sourceHeight),
|
||||
);
|
||||
const width = Math.max(1, Math.round(mapping.sourceWidth * downscale));
|
||||
const height = Math.max(1, Math.round(mapping.sourceHeight * downscale));
|
||||
let rgba;
|
||||
try {
|
||||
const context = videoAlphaContext(width, height);
|
||||
if (!context) return unreadVideoFrame("no 2D canvas was available");
|
||||
context.clearRect(0, 0, width, height);
|
||||
context.drawImage(video, 0, 0, width, height);
|
||||
rgba = context.getImageData(0, 0, width, height).data;
|
||||
} catch (error) {
|
||||
return unreadVideoFrame(
|
||||
error && error.name === "SecurityError"
|
||||
? "its source is cross-origin"
|
||||
: "its frame could not be drawn",
|
||||
);
|
||||
}
|
||||
return classifyVideoAlpha(rgba, width, height, mapping, opacityChain(video));
|
||||
}
|
||||
|
||||
function classifyVideoAlpha(rgba, width, height, mapping, opacity) {
|
||||
const alpha = new Uint8Array(width * height);
|
||||
let clear = 0;
|
||||
let painted = 0;
|
||||
for (let index = 0; index < alpha.length; index++) {
|
||||
alpha[index] = rgba[index * 4 + 3];
|
||||
if (alpha[index] < OCCLUDING_ALPHA * 255) clear += 1;
|
||||
if (alpha[index] > 0) painted += 1;
|
||||
}
|
||||
if (clear === 0) return { kind: "opaque" };
|
||||
if (painted === 0) return unreadVideoFrame("its frame read back fully transparent");
|
||||
return {
|
||||
kind: "matte",
|
||||
alpha,
|
||||
width,
|
||||
height,
|
||||
opacity,
|
||||
mapping: {
|
||||
...mapping,
|
||||
sourceWidth: width,
|
||||
sourceHeight: height,
|
||||
scaleX: mapping.paintedWidth / width,
|
||||
scaleY: mapping.paintedHeight / height,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function matteOccludesAt(frame, x, y) {
|
||||
const pixel = sourcePixelAt(frame.mapping, x, y);
|
||||
if (!pixel) return false;
|
||||
return (frame.alpha[pixel.y * frame.width + pixel.x] / 255) * frame.opacity >= OCCLUDING_ALPHA;
|
||||
}
|
||||
|
||||
// A video with an opaque CSS background paints that background behind its clear pixels.
|
||||
function matteFrameFor(element) {
|
||||
if (element.tagName !== "VIDEO") return null;
|
||||
if (hasOpaqueBackground(getComputedStyle(element))) return null;
|
||||
const frame = videoAlphaFrame(element);
|
||||
return frame.kind === "matte" ? frame : null;
|
||||
}
|
||||
|
||||
// An element hides text beneath it when it paints opaque pixels at near-full
|
||||
// opacity: raster content (img/video/canvas), a background image, or a solid
|
||||
// background colour. Low-opacity overlays (grain, scrims) do not occlude.
|
||||
function isOpaqueOccluder(element, x, y) {
|
||||
const opacity = opacityChain(element);
|
||||
if (opacity < 0.6) return false;
|
||||
if (opacity < OCCLUDING_ALPHA) return false;
|
||||
if (IGNORE_TAGS.has(element.tagName)) return false;
|
||||
if (element.tagName === "IMG") {
|
||||
const alpha = imageAlphaAt(element, x, y);
|
||||
if (alpha !== null) return alpha * opacity >= 0.6;
|
||||
if (alpha !== null) return alpha * opacity >= OCCLUDING_ALPHA;
|
||||
}
|
||||
const matte = matteFrameFor(element);
|
||||
if (matte) return matteOccludesAt(matte, x, y);
|
||||
if (RASTER_TAGS.has(element.tagName)) return true;
|
||||
return hasOpaqueBackground(getComputedStyle(element));
|
||||
}
|
||||
@@ -1150,7 +1325,7 @@
|
||||
// there is related to the text, non-opaque, sharing a 3D context with it, or
|
||||
// part of a transient crossfade overlap.
|
||||
// fallow-ignore-next-line complexity
|
||||
function occluderAt(element, x, y) {
|
||||
function occluderAt(element, x, y, seenMattes) {
|
||||
// Walk the paint-ordered stack: a transparent layer on top must not mask an opaque one below it.
|
||||
const stack =
|
||||
typeof document.elementsFromPoint === "function"
|
||||
@@ -1163,6 +1338,8 @@
|
||||
// Pair-specific exemptions excuse this hit only; keep walking for deeper occluders.
|
||||
if (sharedPreserve3d(element, hit)) continue;
|
||||
if (isCrossSceneTransitionOverlap(element, hit)) continue;
|
||||
const matte = seenMattes && matteFrameFor(hit);
|
||||
if (matte && !seenMattes.has(matte)) seenMattes.set(matte, hit);
|
||||
if (isOpaqueOccluder(hit, x, y)) return hit;
|
||||
}
|
||||
return null;
|
||||
@@ -1196,23 +1373,164 @@
|
||||
function occlusionCoverage(element, textRects) {
|
||||
let occluder = null;
|
||||
let hits = 0;
|
||||
let opaqueHits = 0;
|
||||
// Every matte above the text, including where its alpha is clear: a feature narrower than the
|
||||
// probe grid (an arm, a mic stand) can hide a word between two probe columns.
|
||||
const seenMattes = new Map();
|
||||
for (const textRect of textRects) {
|
||||
for (const yFraction of OCCLUSION_PROBE_Y_FRACTIONS) {
|
||||
const y = textRect.top + textRect.height * yFraction;
|
||||
for (const xFraction of OCCLUSION_PROBE_X_FRACTIONS) {
|
||||
const hit = occluderAt(element, textRect.left + textRect.width * xFraction, y);
|
||||
const x = textRect.left + textRect.width * xFraction;
|
||||
const hit = occluderAt(element, x, y, seenMattes);
|
||||
if (!hit) continue;
|
||||
hits += 1;
|
||||
if (!occluder) occluder = hit;
|
||||
if (!matteFrameFor(hit)) opaqueHits += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
const judgeByMatte = opaqueHits === 0 && seenMattes.size > 0;
|
||||
return {
|
||||
occluder,
|
||||
occluder: occluder || (judgeByMatte ? seenMattes.values().next().value : null),
|
||||
coveredFraction: round(hits / (OCCLUSION_GRID_POINTS * textRects.length)),
|
||||
mattes: judgeByMatte ? [...seenMattes.keys()] : [],
|
||||
};
|
||||
}
|
||||
|
||||
// Thresholds from measuring rendered words against real subject mattes: a word reads as
|
||||
// deliberately tucked behind the subject at 30-55% covered, and is lost at 80%, or when it
|
||||
// and its neighbour are both half covered.
|
||||
const MATTE_WORD_LOST_COVERAGE = 0.8;
|
||||
const MATTE_PAIR_LOST_COVERAGE = 0.5;
|
||||
const MATTE_MAX_SAMPLES_PER_AXIS = 48;
|
||||
|
||||
function wordClientRects(element) {
|
||||
const words = [];
|
||||
for (const node of directTextNodes(element)) {
|
||||
for (const match of (node.textContent || "").matchAll(/\S+/g)) {
|
||||
const range = document.createRange();
|
||||
range.setStart(node, match.index);
|
||||
range.setEnd(node, match.index + match[0].length);
|
||||
const rects = Array.from(range.getClientRects()).filter(
|
||||
(rect) => rect.width > 0.5 && rect.height > 0.5,
|
||||
);
|
||||
range.detach();
|
||||
if (rects.length > 0) words.push({ word: match[0], rects });
|
||||
}
|
||||
}
|
||||
return words;
|
||||
}
|
||||
|
||||
function matteSampleCount(length, mattes) {
|
||||
const pitch = Math.max(1, Math.min(...mattes.map((matte) => matte.mapping.scaleX)));
|
||||
return Math.max(1, Math.min(MATTE_MAX_SAMPLES_PER_AXIS, Math.ceil(length / pitch)));
|
||||
}
|
||||
|
||||
// Area-weighted share of the rects under any matte's occluding alpha, sampled at the matte's resolution.
|
||||
function matteCoverage(rects, mattes) {
|
||||
let coveredArea = 0;
|
||||
let totalArea = 0;
|
||||
for (const rect of rects) {
|
||||
const cols = matteSampleCount(rect.width, mattes);
|
||||
const rows = matteSampleCount(rect.height, mattes);
|
||||
let covered = 0;
|
||||
for (let row = 0; row < rows; row++) {
|
||||
const y = rect.top + ((row + 0.5) * rect.height) / rows;
|
||||
for (let col = 0; col < cols; col++) {
|
||||
const x = rect.left + ((col + 0.5) * rect.width) / cols;
|
||||
if (mattes.some((matte) => matteOccludesAt(matte, x, y))) covered += 1;
|
||||
}
|
||||
}
|
||||
const area = rect.width * rect.height;
|
||||
coveredArea += (area * covered) / (rows * cols);
|
||||
totalArea += area;
|
||||
}
|
||||
return totalArea > 0 ? round(coveredArea / totalArea) : 0;
|
||||
}
|
||||
|
||||
function lostNeighbourPair(words) {
|
||||
for (let index = 1; index < words.length; index++) {
|
||||
const pair = [words[index - 1], words[index]];
|
||||
if (pair.every((word) => word.coverage >= MATTE_PAIR_LOST_COVERAGE)) return pair;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function percent(fraction) {
|
||||
return `${Math.round(fraction * 100)}%`;
|
||||
}
|
||||
|
||||
function lostWordsMessage(lostWord, pair) {
|
||||
if (lostWord) {
|
||||
return `"${lostWord.word}" is ${percent(lostWord.coverage)} behind a transparent video's matte.`;
|
||||
}
|
||||
return `Neighbouring words "${pair[0].word}" and "${pair[1].word}" are ${percent(pair[0].coverage)} and ${percent(pair[1].coverage)} behind a transparent video's matte.`;
|
||||
}
|
||||
|
||||
function markMatteWordsLost(issue, lostWord, pair) {
|
||||
issue.severity = "error";
|
||||
issue.message = lostWordsMessage(lostWord, pair);
|
||||
issue.fixHint =
|
||||
"Move or resize the text so no word is 80% behind the matte and no two neighbouring words are each half behind it, or mark intentional layering with data-layout-allow-occlusion.";
|
||||
}
|
||||
|
||||
// Text behind a cutout is judged per rendered word against the decoded alpha: partial cover is
|
||||
// reported as info with its share, a lost word is an error.
|
||||
function matteOcclusionIssue(element, time, textRect, text, occluder, mattes) {
|
||||
const words = wordClientRects(element).map(({ word, rects }) => ({
|
||||
word,
|
||||
coverage: matteCoverage(rects, mattes),
|
||||
}));
|
||||
if (words.length === 0) return null;
|
||||
const peak = words.reduce((top, word) => (word.coverage > top.coverage ? word : top));
|
||||
if (peak.coverage <= 0) return null;
|
||||
const issue = {
|
||||
code: "text_occluded",
|
||||
severity: "info",
|
||||
time,
|
||||
selector: uniqueSelectorFor(element),
|
||||
containerSelector: uniqueSelectorFor(occluder),
|
||||
text,
|
||||
message: `Text sits partly behind a transparent video's matte; its most covered word "${peak.word}" is ${percent(peak.coverage)} hidden.`,
|
||||
rect: textRect,
|
||||
coveredFraction: peak.coverage,
|
||||
};
|
||||
const lostWord = peak.coverage >= MATTE_WORD_LOST_COVERAGE ? peak : null;
|
||||
const pair = lostNeighbourPair(words);
|
||||
if (lostWord || pair) markMatteWordsLost(issue, lostWord, pair);
|
||||
return { issue, first: words[0], last: words[words.length - 1] };
|
||||
}
|
||||
|
||||
// Captions often render one word per element, so the neighbour rule also spans sibling elements.
|
||||
function markLostSiblingWords(matteResults) {
|
||||
for (const [element, result] of matteResults) {
|
||||
const next = element.nextElementSibling && matteResults.get(element.nextElementSibling);
|
||||
if (!next) continue;
|
||||
const pair = [result.last, next.first];
|
||||
if (!pair.every((word) => word.coverage >= MATTE_PAIR_LOST_COVERAGE)) continue;
|
||||
for (const { issue } of [result, next]) {
|
||||
if (issue.severity !== "error") markMatteWordsLost(issue, null, pair);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function unreadVideoIssues(time) {
|
||||
const issues = [];
|
||||
for (const [video, frame] of videoAlphaFrames) {
|
||||
if (frame.kind !== "unread") continue;
|
||||
issues.push({
|
||||
code: "video_alpha_unread",
|
||||
severity: "info",
|
||||
time,
|
||||
selector: uniqueSelectorFor(video),
|
||||
message: `Could not read this video's frame alpha (${frame.reason}), so text under it was judged as if the video were opaque.`,
|
||||
rect: toRect(video.getBoundingClientRect()),
|
||||
});
|
||||
}
|
||||
return issues;
|
||||
}
|
||||
|
||||
// pointer-events:none hides elements from elementFromPoint — both probed text AND occluders.
|
||||
function restoreHitTesting(root) {
|
||||
const restores = [];
|
||||
@@ -1240,18 +1558,23 @@
|
||||
}
|
||||
|
||||
// text_occluded: atomic labels flag at any hit; prose needs coveredFraction >= proseCoverageFloor (default 0.15).
|
||||
function occludedTextIssue(element, time, proseCoverageFloor) {
|
||||
function occludedTextIssue(element, time, proseCoverageFloor, matteResults) {
|
||||
if (hasAllowOcclusionFlag(element)) return null;
|
||||
if (!hasVisibleTextInk(element)) return null;
|
||||
const textRect = textRectFor(element, true);
|
||||
if (!textRect) return null;
|
||||
const textRects = textClientRects(element, true);
|
||||
const text = textContentFor(element, true);
|
||||
const { occluder, coveredFraction } = occlusionCoverage(
|
||||
const { occluder, coveredFraction, mattes } = occlusionCoverage(
|
||||
element,
|
||||
textRects.length > 0 ? textRects : [textRect],
|
||||
);
|
||||
if (!occluder) return null;
|
||||
if (mattes.length > 0) {
|
||||
const result = matteOcclusionIssue(element, time, textRect, text, occluder, mattes);
|
||||
if (result) matteResults.set(element, result);
|
||||
return result ? result.issue : null;
|
||||
}
|
||||
if (!isAtomicLabel(text) && coveredFraction < proseCoverageFloor) return null;
|
||||
return {
|
||||
code: "text_occluded",
|
||||
@@ -2015,6 +2338,8 @@
|
||||
isVisibleElement(element, 0.05),
|
||||
);
|
||||
const issues = [];
|
||||
const matteResults = new Map();
|
||||
videoAlphaFrames = new Map();
|
||||
|
||||
const restoreHits = restoreHitTesting(root);
|
||||
try {
|
||||
@@ -2023,7 +2348,7 @@
|
||||
const clipped = clippedTextIssue(element, time, tolerance);
|
||||
if (clipped) issues.push(clipped);
|
||||
issues.push(...textOverflowIssues(element, root, rootRect, time, tolerance, clipped));
|
||||
const occluded = occludedTextIssue(element, time, proseCoverageFloor);
|
||||
const occluded = occludedTextIssue(element, time, proseCoverageFloor, matteResults);
|
||||
if (occluded) issues.push(occluded);
|
||||
const invisible = invisibleTextIssue(element, time);
|
||||
if (invisible) issues.push(invisible);
|
||||
@@ -2031,6 +2356,8 @@
|
||||
} finally {
|
||||
restoreHits();
|
||||
}
|
||||
markLostSiblingWords(matteResults);
|
||||
issues.push(...unreadVideoIssues(time));
|
||||
|
||||
issues.push(...containerOverflowIssues(root, time, tolerance));
|
||||
issues.push(...contentOverlapIssues(root, time));
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
// @vitest-environment happy-dom
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const script = readFileSync(
|
||||
join(dirname(fileURLToPath(import.meta.url)), "layout-audit.browser.js"),
|
||||
"utf-8",
|
||||
);
|
||||
|
||||
interface AuditIssue {
|
||||
code: string;
|
||||
severity: string;
|
||||
selector: string;
|
||||
containerSelector?: string;
|
||||
message: string;
|
||||
coveredFraction?: number;
|
||||
}
|
||||
|
||||
interface Box {
|
||||
left: number;
|
||||
top: number;
|
||||
width: number;
|
||||
height: number;
|
||||
}
|
||||
|
||||
// The <video> fills a 1920x1080 frame from a 192x108 source, so one source pixel is 10x10 px.
|
||||
const SOURCE_WIDTH = 192;
|
||||
const SOURCE_HEIGHT = 108;
|
||||
const FRAME: Box = { left: 0, top: 0, width: 1920, height: 1080 };
|
||||
const WORD_TOP = 500;
|
||||
const WORD_HEIGHT = 80;
|
||||
|
||||
interface VideoScene {
|
||||
/** Text elements: id -> words with their left edge and width, all on one line. */
|
||||
text: Record<string, Array<{ word: string; left: number; width: number }>>;
|
||||
/** Alpha (0-255) the decoded frame paints at a composition point. */
|
||||
alphaAt?: (x: number, y: number) => number;
|
||||
video?: { readyState?: number; seeking?: boolean; readError?: Error; overlapsText?: boolean };
|
||||
/** Computed style the <video> adds to the base, e.g. a transform or a background. */
|
||||
videoStyle?: Record<string, string>;
|
||||
/** Report each element's text as one line box, as Chromium does, rather than one per word. */
|
||||
wholeLine?: boolean;
|
||||
textAttrs?: string;
|
||||
}
|
||||
|
||||
function domRect({ left, top, width, height }: Box): DOMRect {
|
||||
return {
|
||||
left,
|
||||
top,
|
||||
width,
|
||||
height,
|
||||
right: left + width,
|
||||
bottom: top + height,
|
||||
x: left,
|
||||
y: top,
|
||||
toJSON() {
|
||||
return this;
|
||||
},
|
||||
} as DOMRect;
|
||||
}
|
||||
|
||||
const BASE_STYLE: Record<string, string> = {
|
||||
display: "block",
|
||||
visibility: "visible",
|
||||
opacity: "1",
|
||||
overflow: "visible",
|
||||
overflowX: "visible",
|
||||
overflowY: "visible",
|
||||
backgroundColor: "rgba(0, 0, 0, 0)",
|
||||
backgroundImage: "none",
|
||||
objectFit: "fill",
|
||||
objectPosition: "50% 50%",
|
||||
fontSize: "36px",
|
||||
};
|
||||
|
||||
function wordBox(left: number, width: number): Box {
|
||||
return { left, top: WORD_TOP, width, height: WORD_HEIGHT };
|
||||
}
|
||||
|
||||
function installVideoScene(scene: VideoScene): { getImageData: ReturnType<typeof vi.fn> } {
|
||||
const spans = Object.entries(scene.text)
|
||||
.map(
|
||||
([id, words]) =>
|
||||
`<span id="${id}" ${scene.textAttrs ?? ""}>${words.map(({ word }) => word).join(" ")}</span>`,
|
||||
)
|
||||
.join("");
|
||||
document.body.innerHTML = `
|
||||
<div id="root" data-composition-id="main" data-width="1920" data-height="1080">
|
||||
<div id="line">${spans}</div>
|
||||
<video id="clip"></video>
|
||||
</div>`;
|
||||
|
||||
const video = document.getElementById("clip") as HTMLVideoElement;
|
||||
Object.defineProperties(video, {
|
||||
videoWidth: { configurable: true, value: SOURCE_WIDTH },
|
||||
videoHeight: { configurable: true, value: SOURCE_HEIGHT },
|
||||
readyState: { configurable: true, value: scene.video?.readyState ?? 4 },
|
||||
seeking: { configurable: true, value: scene.video?.seeking ?? false },
|
||||
});
|
||||
|
||||
vi.spyOn(window, "getComputedStyle").mockImplementation(
|
||||
(element) =>
|
||||
({
|
||||
...BASE_STYLE,
|
||||
...(element === video ? scene.videoStyle : {}),
|
||||
}) as unknown as CSSStyleDeclaration,
|
||||
);
|
||||
for (const element of Array.from(document.querySelectorAll("*"))) {
|
||||
vi.spyOn(element, "getBoundingClientRect").mockReturnValue(domRect(FRAME));
|
||||
}
|
||||
|
||||
const wordRect = (node: Node | null, start: number, end: number): DOMRect[] => {
|
||||
const owner = node?.parentElement?.id ?? "";
|
||||
const word = (node?.textContent ?? "").slice(start, end);
|
||||
const entry = scene.text[owner]?.find((candidate) => candidate.word === word);
|
||||
return entry ? [domRect(wordBox(entry.left, entry.width))] : [];
|
||||
};
|
||||
vi.spyOn(document, "createRange").mockImplementation(() => {
|
||||
let node: Node | null = null;
|
||||
let start = 0;
|
||||
let end: number | null = null;
|
||||
return {
|
||||
selectNodeContents(selected: Node) {
|
||||
node = selected;
|
||||
},
|
||||
setStart(selected: Node, offset: number) {
|
||||
node = selected;
|
||||
start = offset;
|
||||
},
|
||||
setEnd(_selected: Node, offset: number) {
|
||||
end = offset;
|
||||
},
|
||||
getClientRects() {
|
||||
const owner = node?.parentElement?.id ?? "";
|
||||
if (end === null) {
|
||||
const boxes = (scene.text[owner] ?? []).map(({ left, width }) => wordBox(left, width));
|
||||
if (scene.wholeLine && boxes.length > 0) {
|
||||
const left = Math.min(...boxes.map((box) => box.left));
|
||||
const right = Math.max(...boxes.map((box) => box.left + box.width));
|
||||
return [domRect(wordBox(left, right - left))] as unknown as DOMRectList;
|
||||
}
|
||||
return boxes.map(domRect) as unknown as DOMRectList;
|
||||
}
|
||||
return wordRect(node, start, end) as unknown as DOMRectList;
|
||||
},
|
||||
detach() {},
|
||||
} as unknown as Range;
|
||||
});
|
||||
|
||||
const overlapsText = scene.video?.overlapsText ?? true;
|
||||
Object.defineProperty(document, "elementsFromPoint", {
|
||||
configurable: true,
|
||||
value: (x: number) => {
|
||||
const textElement = Object.keys(scene.text)
|
||||
.map((id) => document.getElementById(id))
|
||||
.find((element) => {
|
||||
const words = scene.text[element?.id ?? ""] ?? [];
|
||||
return words.some(({ left, width }) => x >= left && x < left + width);
|
||||
});
|
||||
return [...(overlapsText ? [video] : []), textElement, video].filter(Boolean);
|
||||
},
|
||||
});
|
||||
|
||||
const alphaAt = scene.alphaAt ?? (() => 255);
|
||||
const getImageData = vi.fn((_x: number, _y: number, width: number, height: number) => {
|
||||
if (scene.video?.readError) throw scene.video.readError;
|
||||
const data = new Uint8ClampedArray(width * height * 4);
|
||||
for (let row = 0; row < height; row++) {
|
||||
for (let col = 0; col < width; col++) {
|
||||
const x = ((col + 0.5) * FRAME.width) / width;
|
||||
const y = ((row + 0.5) * FRAME.height) / height;
|
||||
data[(row * width + col) * 4 + 3] = alphaAt(x, y);
|
||||
}
|
||||
}
|
||||
return { data };
|
||||
});
|
||||
// getContext's overloads resolve to the WebGPU one; the audit only asks for "2d".
|
||||
const getContextSpy = vi.spyOn(HTMLCanvasElement.prototype, "getContext") as unknown as {
|
||||
mockReturnValue(value: CanvasRenderingContext2D): void;
|
||||
};
|
||||
getContextSpy.mockReturnValue({
|
||||
clearRect() {},
|
||||
drawImage() {},
|
||||
getImageData,
|
||||
} as unknown as CanvasRenderingContext2D);
|
||||
|
||||
window.eval(script);
|
||||
return { getImageData };
|
||||
}
|
||||
|
||||
function runAudit(): AuditIssue[] {
|
||||
const audit = (
|
||||
window as unknown as {
|
||||
__hyperframesLayoutAudit: (options: { time: number; tolerance: number }) => AuditIssue[];
|
||||
}
|
||||
).__hyperframesLayoutAudit;
|
||||
return audit({ time: 1, tolerance: 2 });
|
||||
}
|
||||
|
||||
const occlusionOf = (issues: AuditIssue[], id: string) =>
|
||||
issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);
|
||||
|
||||
const SHIP = { ship: [{ word: "SHIP", left: 800, width: 200 }] };
|
||||
const subjectLeftOf = (edge: number) => (x: number) => (x < edge ? 255 : 0);
|
||||
|
||||
describe("layout-audit.browser text behind a <video>", () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
document.body.innerHTML = "";
|
||||
Reflect.deleteProperty(document, "elementsFromPoint");
|
||||
Reflect.deleteProperty(window, "__hyperframesLayoutAudit");
|
||||
});
|
||||
|
||||
it("keeps a fully opaque frame occluding like any raster", () => {
|
||||
installVideoScene({ text: SHIP });
|
||||
expect(occlusionOf(runAudit(), "ship")).toMatchObject({
|
||||
severity: "error",
|
||||
containerSelector: "#clip",
|
||||
coveredFraction: 1,
|
||||
message: "Text is hidden beneath an opaque element.",
|
||||
});
|
||||
});
|
||||
|
||||
it("does not flag text under the transparent part of a cutout", () => {
|
||||
installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(400) });
|
||||
expect(occlusionOf(runAudit(), "ship")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("reports a partly covered word as info with its share under the matte", () => {
|
||||
installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });
|
||||
expect(occlusionOf(runAudit(), "ship")).toMatchObject({
|
||||
severity: "info",
|
||||
coveredFraction: 0.3,
|
||||
message: `Text sits partly behind a transparent video's matte; its most covered word "SHIP" is 30% hidden.`,
|
||||
});
|
||||
});
|
||||
|
||||
it("fails a word that is 80% or more behind the matte", () => {
|
||||
installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(970) });
|
||||
expect(occlusionOf(runAudit(), "ship")).toMatchObject({
|
||||
severity: "error",
|
||||
coveredFraction: 0.85,
|
||||
message: `"SHIP" is 85% behind a transparent video's matte.`,
|
||||
});
|
||||
});
|
||||
|
||||
it("judges each rendered word, so one lost word fails a mostly clear line", () => {
|
||||
installVideoScene({
|
||||
text: {
|
||||
line1: [
|
||||
{ word: "a", left: 200, width: 40 },
|
||||
{ word: "big", left: 260, width: 100 },
|
||||
{ word: "launch", left: 380, width: 240 },
|
||||
],
|
||||
},
|
||||
alphaAt: (x) => (x >= 370 ? 255 : 0),
|
||||
});
|
||||
expect(occlusionOf(runAudit(), "line1")).toMatchObject({
|
||||
severity: "error",
|
||||
coveredFraction: 1,
|
||||
message: `"launch" is 100% behind a transparent video's matte.`,
|
||||
});
|
||||
});
|
||||
|
||||
it("fails two neighbouring words that are each half covered", () => {
|
||||
installVideoScene({
|
||||
text: {
|
||||
pair: [
|
||||
{ word: "IN", left: 200, width: 100 },
|
||||
{ word: "SHIP", left: 320, width: 200 },
|
||||
],
|
||||
},
|
||||
alphaAt: (_x, y) => (y >= 530 ? 255 : 0),
|
||||
});
|
||||
expect(occlusionOf(runAudit(), "pair")).toMatchObject({
|
||||
severity: "error",
|
||||
coveredFraction: 0.63,
|
||||
message: `Neighbouring words "IN" and "SHIP" are 63% and 63% behind a transparent video's matte.`,
|
||||
});
|
||||
});
|
||||
|
||||
it("applies the neighbour rule across one-word sibling elements", () => {
|
||||
installVideoScene({
|
||||
text: {
|
||||
first: [{ word: "IN", left: 200, width: 100 }],
|
||||
second: [{ word: "SHIP", left: 320, width: 200 }],
|
||||
},
|
||||
alphaAt: (_x, y) => (y >= 530 ? 255 : 0),
|
||||
});
|
||||
const issues = runAudit();
|
||||
expect(occlusionOf(issues, "first")?.severity).toBe("error");
|
||||
expect(occlusionOf(issues, "second")?.severity).toBe("error");
|
||||
});
|
||||
|
||||
it("keeps a single half-covered word at info", () => {
|
||||
installVideoScene({
|
||||
text: {
|
||||
first: [{ word: "IN", left: 200, width: 100 }],
|
||||
second: [{ word: "SHIP", left: 320, width: 200 }],
|
||||
},
|
||||
alphaAt: (x, y) => (x < 310 && y >= 530 ? 255 : 0),
|
||||
});
|
||||
const issues = runAudit();
|
||||
expect(occlusionOf(issues, "first")?.severity).toBe("info");
|
||||
expect(occlusionOf(issues, "second")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("fails a word hidden by a matte feature that falls between two probe columns", () => {
|
||||
// One 1800px line box: its 20% and 35% probe columns sit at x 360 and 630, the strip at 440-540.
|
||||
installVideoScene({
|
||||
text: {
|
||||
caption: [
|
||||
{ word: "a", left: 0, width: 40 },
|
||||
{ word: "gh", left: 460, width: 60 },
|
||||
{ word: "end", left: 1700, width: 100 },
|
||||
],
|
||||
},
|
||||
wholeLine: true,
|
||||
alphaAt: (x) => (x >= 440 && x < 540 ? 255 : 0),
|
||||
});
|
||||
expect(occlusionOf(runAudit(), "caption")).toMatchObject({
|
||||
severity: "error",
|
||||
containerSelector: "#clip",
|
||||
message: `"gh" is 100% behind a transparent video's matte.`,
|
||||
});
|
||||
});
|
||||
|
||||
it("reads a mirrored video's alpha on the side it is painted", () => {
|
||||
installVideoScene({
|
||||
text: {
|
||||
shown: [{ word: "SHIP", left: 200, width: 200 }],
|
||||
hidden: [{ word: "SHIP", left: 1500, width: 200 }],
|
||||
},
|
||||
videoStyle: { transform: "matrix(-1, 0, 0, 1, 0, 0)" },
|
||||
// Source alpha: the subject fills the source's left half, so it paints on the right.
|
||||
alphaAt: subjectLeftOf(960),
|
||||
});
|
||||
const issues = runAudit();
|
||||
expect(occlusionOf(issues, "shown")).toBeUndefined();
|
||||
expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });
|
||||
});
|
||||
|
||||
it("keeps a video's opaque CSS background occluding behind its clear pixels", () => {
|
||||
installVideoScene({
|
||||
text: SHIP,
|
||||
videoStyle: { backgroundColor: "rgb(18, 52, 86)" },
|
||||
alphaAt: subjectLeftOf(400),
|
||||
});
|
||||
expect(occlusionOf(runAudit(), "ship")).toMatchObject({
|
||||
severity: "error",
|
||||
message: "Text is hidden beneath an opaque element.",
|
||||
});
|
||||
});
|
||||
|
||||
it("respects data-layout-allow-occlusion", () => {
|
||||
installVideoScene({
|
||||
text: SHIP,
|
||||
alphaAt: subjectLeftOf(970),
|
||||
textAttrs: "data-layout-allow-occlusion",
|
||||
});
|
||||
expect(occlusionOf(runAudit(), "ship")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("reads each probed video's frame once per sample and leaves videos under no text alone", () => {
|
||||
const covering = installVideoScene({ text: SHIP, alphaAt: subjectLeftOf(860) });
|
||||
runAudit();
|
||||
expect(covering.getImageData).toHaveBeenCalledTimes(1);
|
||||
runAudit();
|
||||
expect(covering.getImageData).toHaveBeenCalledTimes(2);
|
||||
|
||||
vi.restoreAllMocks();
|
||||
const beneath = installVideoScene({ text: SHIP, video: { overlapsText: false } });
|
||||
runAudit();
|
||||
expect(beneath.getImageData).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{
|
||||
name: "a video with no decoded frame",
|
||||
video: { readyState: 1 },
|
||||
alphaAt: undefined,
|
||||
reason: "no frame was decoded at this time",
|
||||
},
|
||||
{
|
||||
name: "a video still seeking",
|
||||
video: { seeking: true },
|
||||
alphaAt: undefined,
|
||||
reason: "the seek had not landed",
|
||||
},
|
||||
{
|
||||
name: "a cross-origin video",
|
||||
video: { readError: Object.assign(new Error("tainted"), { name: "SecurityError" }) },
|
||||
alphaAt: undefined,
|
||||
reason: "its source is cross-origin",
|
||||
},
|
||||
{
|
||||
name: "a rotated video",
|
||||
video: {},
|
||||
videoStyle: { transform: "matrix(0, 1, -1, 0, 0, 0)" },
|
||||
alphaAt: subjectLeftOf(400),
|
||||
reason: "it is rotated, skewed or transformed in 3D",
|
||||
},
|
||||
{
|
||||
name: "a frame that reads back fully transparent",
|
||||
video: {},
|
||||
alphaAt: () => 0,
|
||||
reason: "its frame read back fully transparent",
|
||||
},
|
||||
])(
|
||||
"falls back to opaque with an info note for $name",
|
||||
({ video, videoStyle, alphaAt, reason }) => {
|
||||
installVideoScene({ text: SHIP, video, videoStyle, alphaAt });
|
||||
const issues = runAudit();
|
||||
expect(occlusionOf(issues, "ship")).toMatchObject({ severity: "error", coveredFraction: 1 });
|
||||
expect(issues).toContainEqual(
|
||||
expect.objectContaining({
|
||||
code: "video_alpha_unread",
|
||||
severity: "info",
|
||||
selector: "#clip",
|
||||
message: `Could not read this video's frame alpha (${reason}), so text under it was judged as if the video were opaque.`,
|
||||
}),
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,291 @@
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
|
||||
import { createServer, type Server, type ServerResponse } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import puppeteer, { type Browser } from "puppeteer-core";
|
||||
|
||||
interface AuditIssue {
|
||||
code: string;
|
||||
severity: string;
|
||||
selector: string;
|
||||
message: string;
|
||||
coveredFraction?: number;
|
||||
}
|
||||
|
||||
declare global {
|
||||
interface Window {
|
||||
__hyperframesLayoutAudit(options: { time: number; tolerance: number }): AuditIssue[];
|
||||
}
|
||||
}
|
||||
|
||||
const executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
|
||||
const hasFfmpeg = spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0;
|
||||
const script = readFileSync(new URL("./layout-audit.browser.js", import.meta.url), "utf8");
|
||||
|
||||
// 320x180 clips shown at 640x360. The cutout's subject fills the left half until 0.5s and the
|
||||
// right half after it, the horizon's fills the bottom half, the strip's is opaque only at source
|
||||
// x 76-104 (page x 152-208), and the opaque clip has no alpha plane.
|
||||
const CLIPS: Record<string, string[]> = {
|
||||
"cutout.webm": [
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(lt(T,0.5),if(lt(X,W/2),255,0),if(gte(X,W/2),255,0))'",
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-pix_fmt",
|
||||
"yuva420p",
|
||||
"-auto-alt-ref",
|
||||
"0",
|
||||
],
|
||||
"horizon.webm": [
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(gte(Y,H/2),255,0)'",
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-pix_fmt",
|
||||
"yuva420p",
|
||||
"-auto-alt-ref",
|
||||
"0",
|
||||
],
|
||||
"strip.webm": [
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=red:s=320x180:d=1:r=10,format=rgba,geq=r='255':g='0':b='0':a='if(between(X,76,104),255,0)'",
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-pix_fmt",
|
||||
"yuva420p",
|
||||
"-auto-alt-ref",
|
||||
"0",
|
||||
],
|
||||
"opaque.webm": [
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=blue:s=320x180:d=1:r=10",
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
};
|
||||
|
||||
const word = (id: string, left: number, text: string, extra = "") =>
|
||||
`<span id="${id}" ${extra} style="position:absolute;left:${left}px;top:150px;transform:translateX(-50%);margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">${text}</span>`;
|
||||
|
||||
function page(video: string, words: string): string {
|
||||
return `<!doctype html><body style="margin:0">
|
||||
<div data-composition-id="main" data-width="640" data-height="360" style="position:relative;width:640px;height:360px;background:#111">
|
||||
<div style="position:absolute;inset:0;z-index:1">${words}</div>
|
||||
${video}
|
||||
</div></body>`;
|
||||
}
|
||||
|
||||
const videoTag = (src: string, extra = "", style = "") =>
|
||||
`<video id="clip" src="/${src}" muted playsinline ${extra} style="position:absolute;left:0;top:0;width:640px;height:360px;z-index:2;${style}"></video>`;
|
||||
|
||||
// A ~590px caption whose word "gh" is moved onto the strip's centre (page x 180). The coarse probe
|
||||
// columns at 20% and 35% of the line then fall either side of the strip, never on it.
|
||||
const STRIP_CAPTION = `<span id="line" style="position:absolute;left:0;top:150px;margin:0;font:24px/1 monospace;color:#fff;white-space:nowrap">abcdefghi gh ij kl mn op qr st uv wx yz ab cd</span>
|
||||
<script>
|
||||
const line = document.getElementById("line");
|
||||
const text = line.firstChild;
|
||||
const range = document.createRange();
|
||||
range.setStart(text, 10);
|
||||
range.setEnd(text, 12);
|
||||
const word = range.getBoundingClientRect();
|
||||
line.style.left = 180 - (word.left + word.width / 2) + "px";
|
||||
</script>`;
|
||||
|
||||
// Chromium only seeks a media resource its server serves by byte range.
|
||||
function serveByteRange(body: Buffer, rangeHeader: string | undefined, response: ServerResponse) {
|
||||
const range = /bytes=(\d+)-(\d*)/.exec(rangeHeader ?? "");
|
||||
const start = range ? Number(range[1]) : 0;
|
||||
const end = range?.[2] ? Number(range[2]) : body.length - 1;
|
||||
response.writeHead(range ? 206 : 200, {
|
||||
"content-type": "video/webm",
|
||||
"accept-ranges": "bytes",
|
||||
"content-length": end - start + 1,
|
||||
...(range ? { "content-range": `bytes ${start}-${end}/${body.length}` } : {}),
|
||||
});
|
||||
response.end(body.subarray(start, end + 1));
|
||||
}
|
||||
|
||||
describe.runIf(executablePath && hasFfmpeg)("text_occluded behind a <video> in Chromium", () => {
|
||||
let browser: Browser;
|
||||
let server: Server;
|
||||
let origin: string;
|
||||
let clipDir: string;
|
||||
const pages = new Map<string, string>();
|
||||
const clips = new Map<string, Buffer>();
|
||||
|
||||
beforeAll(async () => {
|
||||
clipDir = mkdtempSync(join(tmpdir(), "hf-video-alpha-"));
|
||||
for (const [name, args] of Object.entries(CLIPS)) {
|
||||
const result = spawnSync("ffmpeg", [
|
||||
"-y",
|
||||
"-loglevel",
|
||||
"error",
|
||||
...args,
|
||||
join(clipDir, name),
|
||||
]);
|
||||
if (result.status !== 0) throw new Error(`ffmpeg ${name}: ${result.stderr}`);
|
||||
clips.set(`/${name}`, readFileSync(join(clipDir, name)));
|
||||
}
|
||||
server = createServer((request, response) => {
|
||||
const path = (request.url ?? "/").split("?")[0] ?? "/";
|
||||
const html = pages.get(path);
|
||||
if (html) {
|
||||
response.writeHead(200, { "content-type": "text/html" });
|
||||
response.end(html);
|
||||
return;
|
||||
}
|
||||
const body = clips.get(path);
|
||||
if (body) serveByteRange(body, request.headers.range, response);
|
||||
else response.writeHead(404).end();
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||
origin = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
|
||||
browser = await puppeteer.launch({ executablePath, args: ["--no-sandbox"] });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await browser?.close();
|
||||
await new Promise((resolve) => server?.close(resolve));
|
||||
if (clipDir) rmSync(clipDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function audit(html: string, seekTo: number | null = 0.25): Promise<AuditIssue[]> {
|
||||
const path = `/page-${pages.size}.html`;
|
||||
pages.set(path, html);
|
||||
const tab = await browser.newPage();
|
||||
try {
|
||||
await tab.setViewport({ width: 640, height: 360, deviceScaleFactor: 1 });
|
||||
await tab.goto(`${origin}${path}`);
|
||||
if (seekTo !== null) {
|
||||
await tab.evaluate(async (time) => {
|
||||
const video = document.querySelector("video");
|
||||
if (!video) return;
|
||||
if (video.readyState < 2) {
|
||||
await new Promise((resolve) => video.addEventListener("loadeddata", resolve));
|
||||
}
|
||||
video.currentTime = time;
|
||||
await new Promise((resolve) => video.addEventListener("seeked", resolve));
|
||||
}, seekTo);
|
||||
}
|
||||
await tab.addScriptTag({ content: script });
|
||||
return await tab.evaluate(() => window.__hyperframesLayoutAudit({ time: 1, tolerance: 2 }));
|
||||
} finally {
|
||||
await tab.close();
|
||||
}
|
||||
}
|
||||
|
||||
const occlusionOf = (issues: AuditIssue[], id: string) =>
|
||||
issues.find((issue) => issue.code === "text_occluded" && issue.selector === `#${id}`);
|
||||
|
||||
it("keeps an opaque video occluding its whole box", async () => {
|
||||
const issues = await audit(page(videoTag("opaque.webm"), word("hidden", 480, "SHIP")));
|
||||
expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });
|
||||
expect(issues.some((issue) => issue.code === "video_alpha_unread")).toBe(false);
|
||||
});
|
||||
|
||||
it("does not flag text behind the transparent part of a cutout", async () => {
|
||||
const issues = await audit(page(videoTag("cutout.webm"), word("clear", 480, "SHIP")));
|
||||
expect(occlusionOf(issues, "clear")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("reports a word half under the subject's matte as info with its coverage", async () => {
|
||||
const issues = await audit(page(videoTag("cutout.webm"), word("tucked", 320, "SHIP")));
|
||||
const finding = occlusionOf(issues, "tucked");
|
||||
expect(finding).toMatchObject({ severity: "info", containerSelector: "#clip" });
|
||||
expect(finding?.coveredFraction).toBeGreaterThan(0.4);
|
||||
expect(finding?.coveredFraction).toBeLessThan(0.6);
|
||||
expect(finding?.message).toMatch(/"SHIP" is \d+% hidden/);
|
||||
});
|
||||
|
||||
it("fails a word that the matte covers entirely", async () => {
|
||||
const issues = await audit(page(videoTag("cutout.webm"), word("eaten", 160, "SHIP")));
|
||||
expect(occlusionOf(issues, "eaten")).toMatchObject({ severity: "error", coveredFraction: 1 });
|
||||
});
|
||||
|
||||
it("fails two neighbouring words that are each at least half covered", async () => {
|
||||
const line = `<span id="pair" style="position:absolute;left:200px;top:160px;margin:0;font:700 48px/1 Arial;color:#fff;white-space:nowrap">IN SHIP</span>`;
|
||||
const issues = await audit(page(videoTag("horizon.webm"), line));
|
||||
const finding = occlusionOf(issues, "pair");
|
||||
expect(finding?.severity).toBe("error");
|
||||
expect(finding?.message).toMatch(/Neighbouring words "IN" and "SHIP"/);
|
||||
});
|
||||
|
||||
it("reads the frame the video was seeked to", async () => {
|
||||
const html = page(
|
||||
videoTag("cutout.webm"),
|
||||
word("left", 160, "SHIP") + word("right", 480, "SHIP"),
|
||||
);
|
||||
const early = await audit(html, 0.25);
|
||||
const late = await audit(html, 0.75);
|
||||
expect(occlusionOf(early, "left")?.severity).toBe("error");
|
||||
expect(occlusionOf(early, "right")).toBeUndefined();
|
||||
expect(occlusionOf(late, "left")).toBeUndefined();
|
||||
expect(occlusionOf(late, "right")?.severity).toBe("error");
|
||||
});
|
||||
|
||||
it("fails a word hidden by a matte feature narrower than the probe grid", async () => {
|
||||
const issues = await audit(page(videoTag("strip.webm"), STRIP_CAPTION));
|
||||
const finding = occlusionOf(issues, "line");
|
||||
expect(finding).toMatchObject({ severity: "error", containerSelector: "#clip" });
|
||||
expect(finding?.message).toMatch(/"gh" is 100% behind/);
|
||||
});
|
||||
|
||||
it("reads a mirrored cutout's alpha on the side it is painted", async () => {
|
||||
const issues = await audit(
|
||||
page(
|
||||
videoTag("cutout.webm", "", "transform:scaleX(-1)"),
|
||||
word("shown", 160, "SHIP") + word("hidden", 480, "SHIP"),
|
||||
),
|
||||
);
|
||||
expect(occlusionOf(issues, "shown")).toBeUndefined();
|
||||
expect(occlusionOf(issues, "hidden")).toMatchObject({ severity: "error", coveredFraction: 1 });
|
||||
});
|
||||
|
||||
it("keeps a cutout's opaque CSS background occluding behind its clear pixels", async () => {
|
||||
const issues = await audit(
|
||||
page(videoTag("cutout.webm", "", "background:#123456"), word("boxed", 480, "SHIP")),
|
||||
);
|
||||
expect(occlusionOf(issues, "boxed")).toMatchObject({ severity: "error" });
|
||||
});
|
||||
|
||||
it("judges a rotated cutout as opaque and says why", async () => {
|
||||
const issues = await audit(
|
||||
page(videoTag("cutout.webm", "", "transform:rotate(90deg)"), word("turned", 480, "SHIP")),
|
||||
);
|
||||
expect(occlusionOf(issues, "turned")).toMatchObject({ severity: "error" });
|
||||
expect(issues).toContainEqual(
|
||||
expect.objectContaining({ code: "video_alpha_unread", selector: "#clip" }),
|
||||
);
|
||||
});
|
||||
|
||||
it("respects data-layout-allow-occlusion on text behind a cutout", async () => {
|
||||
const issues = await audit(
|
||||
page(videoTag("cutout.webm"), word("waived", 160, "SHIP", "data-layout-allow-occlusion")),
|
||||
);
|
||||
expect(occlusionOf(issues, "waived")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("falls back to opaque with an info note when no frame has been decoded", async () => {
|
||||
const issues = await audit(
|
||||
page(videoTag("cutout.webm", 'preload="none"'), word("unknown", 480, "SHIP")),
|
||||
null,
|
||||
);
|
||||
expect(occlusionOf(issues, "unknown")?.severity).toBe("error");
|
||||
expect(issues).toContainEqual(
|
||||
expect.objectContaining({ code: "video_alpha_unread", severity: "info", selector: "#clip" }),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -1441,6 +1441,7 @@ const LAYOUT_ISSUE_CODES: readonly LayoutIssueCode[] = [
|
||||
"content_overlap",
|
||||
"visibility_escapes_segment",
|
||||
"text_occluded",
|
||||
"video_alpha_unread",
|
||||
"text_not_painted",
|
||||
"caption_zone_collision",
|
||||
"frame_out_of_frame",
|
||||
|
||||
@@ -18,6 +18,7 @@ export type LayoutIssueCode =
|
||||
| "content_overlap"
|
||||
| "visibility_escapes_segment"
|
||||
| "text_occluded"
|
||||
| "video_alpha_unread"
|
||||
| "text_not_painted"
|
||||
| "caption_zone_collision"
|
||||
| "frame_out_of_frame"
|
||||
@@ -62,8 +63,8 @@ export interface LayoutIssue {
|
||||
rect: LayoutRect;
|
||||
containerRect?: LayoutRect;
|
||||
overflow?: LayoutOverflow;
|
||||
/** `text_occluded` only: approximate fraction (0-1) of the occlusion probe
|
||||
* grid that hit an opaque occluder — see layout-audit.browser.js. */
|
||||
/** `text_occluded` only (0-1): probe-grid share hit by an opaque occluder, or behind a
|
||||
* transparent video the most covered word's share under its alpha (layout-audit.browser.js). */
|
||||
coveredFraction?: number;
|
||||
fixHint?: string;
|
||||
}
|
||||
|
||||
@@ -30,7 +30,7 @@
|
||||
"files": 8
|
||||
},
|
||||
"hyperframes-cli": {
|
||||
"hash": "f08c429fb0f8b82c",
|
||||
"hash": "3c8d1fac9e0a02fe",
|
||||
"files": 11
|
||||
},
|
||||
"hyperframes-core": {
|
||||
|
||||
@@ -44,7 +44,7 @@ npx hyperframes check --strict # exit non-zero on warnings too (defaul
|
||||
One command, one Chrome boot. `check` runs the linter first and skips the browser entirely when lint reports errors. When the browser never ran (lint errors, a linter crash, or a browser launch failure), `browserSkipped` is `true` and the `layout`, `motion` and `contrast` sections are empty, not clean. Otherwise it loads the bundled composition once, wires runtime listeners before navigation, and sweeps one seek grid running every audit per sample:
|
||||
|
||||
- **Runtime**: JavaScript console errors, unhandled exceptions, failed network requests (media-file `ERR_ABORTED` filtered out), HTTP 4xx/5xx.
|
||||
- **Layout**: text extending outside its container or the canvas, text clipped by its own box, held text overlaps and occlusion (with an approximate covered fraction), children escaping clipping containers.
|
||||
- **Layout**: text extending outside its container or the canvas, text clipped by its own box, held text overlaps and occlusion (with an approximate covered fraction), children escaping clipping containers. Text under a transparent `<video>` (a `remove-background` cutout) is measured per word against the frame's alpha: partial cover is info with the share hidden, a word 80% hidden or two neighbouring words each half hidden is an error, and a frame that cannot be read falls back to opaque with a `video_alpha_unread` note.
|
||||
- **Motion**: `*.motion.json` sidecar assertions against the same seeked timeline (see below).
|
||||
- **Contrast**: WCAG AA on visible text, sampled at 5 grid points. Failures are **errors** and each finding carries the sampled fg/bg colors, measured vs required ratio, and a suggested compliant color in the same palette direction, so most contrast fixes need no screenshot at all.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user