ci: fail when the search index stops covering the registry

The catalog vector artifact is regenerated by hand. Nothing in CI, in
package.json or in a hook rebuilds it, because embedding needs the 32 MB
model. So adding a registry item silently makes it invisible to meaning
search until someone remembers to regenerate.

The failure is asymmetric, which is what makes it easy to miss. Removing an
item is self-healing: the ranker still scores the dead vector, then filters
the name before display, so a user is never offered something they cannot
install. Adding one is not: the item is absent from the candidate set
entirely, not ranked low.

Comparing the two name lists needs neither the model nor a network call, so
the gate runs in seconds. CI checks rather than fixes, for the same reason
it cannot regenerate.

Scoped to blocks and components. Examples are starter projects a user
scaffolds, never something catalog ranks, and the artifact carries no vector
for them, so demanding one would keep this gate permanently red and it would
be ignored within a week.

Verified in both directions rather than assumed: adding an unindexed item
exits 1 and names it, restoring the registry exits 0.
This commit is contained in:
Miguel Ángel
2026-08-09 17:45:59 +00:00
parent e203b199e8
commit f85301cbec
2 changed files with 97 additions and 0 deletions
@@ -0,0 +1,77 @@
/**
* Fails when the committed catalog vector artifact no longer covers the
* registry it is meant to search.
*
* The artifact is regenerated by hand: nothing in CI or in a hook rebuilds it,
* because embedding needs the 32 MB model. So adding a registry item silently
* makes it invisible to meaning search until someone remembers. Removing one is
* self-healing (the ranker filters names it cannot install), which is why the
* asymmetry is easy to miss.
*
* Comparing the two name lists needs no model and no network, so it is cheap
* enough to run on every change that touches either side.
*/
import { readFileSync } from "node:fs";
type RegistryItem = { name: string; type?: string };
type Registry = { items: RegistryItem[] };
type Artifact = { model?: string; dimensions?: number; names?: string[] };
const REGISTRY = "registry/registry.json";
const ARTIFACT = "registry/catalog-artifact/local-vectors.json";
function read<T>(path: string): T {
try {
return JSON.parse(readFileSync(path, "utf8")) as T;
} catch (error) {
console.error(`Could not read ${path}: ${(error as Error).message}`);
process.exit(1);
}
}
const registry = read<Registry>(REGISTRY);
const artifact = read<Artifact>(ARTIFACT);
// Only blocks and components are searchable moves. Examples are starter
// projects a user scaffolds, never something `catalog` ranks, so the artifact
// deliberately carries no vector for them and demanding one would keep this
// gate permanently red.
const SEARCHABLE = new Set(["hyperframes:block", "hyperframes:component"]);
const registryNames = new Set(
registry.items.filter((i) => SEARCHABLE.has(i.type ?? "")).map((i) => i.name),
);
const artifactNames = new Set(artifact.names ?? []);
const unindexed = [...registryNames].filter((n) => !artifactNames.has(n)).sort();
const dropped = [...artifactNames].filter((n) => !registryNames.has(n)).sort();
const show = (names: string[]) =>
names
.slice(0, 10)
.map((n) => ` ${n}`)
.join("\n") + (names.length > 10 ? `\n ... and ${names.length - 10} more` : "");
console.log(`registry: ${registryNames.size} searchable items (blocks + components)`);
console.log(`artifact: ${artifactNames.size} vectors (${artifact.model ?? "unknown model"})`);
if (dropped.length > 0) {
// Not fatal: the CLI filters these before a user ever sees them.
console.log(`\nnote: ${dropped.length} vector(s) name items the registry no longer has.`);
console.log(show(dropped));
console.log(" These are filtered at search time, so they cost space, not correctness.");
}
if (unindexed.length === 0) {
console.log("\nEvery registry item has a vector. Meaning search can see the whole catalog.");
process.exit(0);
}
console.error(`\n${unindexed.length} registry item(s) have no vector:`);
console.error(show(unindexed));
console.error(
"\nMeaning search cannot return these at all: they are absent from the candidate\n" +
"set, not ranked low. Word search still finds them.\n\n" +
"Regenerate and commit the artifact:\n" +
" bun scripts/catalog/build-local-vectors.ts\n",
);
process.exit(1);