ci: fail when the search index stops covering the registry

The catalog vector artifact is regenerated by hand. Nothing in CI, in
package.json or in a hook rebuilds it, because embedding needs the 32 MB
model. So adding a registry item silently makes it invisible to meaning
search until someone remembers to regenerate.

The failure is asymmetric, which is what makes it easy to miss. Removing an
item is self-healing: the ranker still scores the dead vector, then filters
the name before display, so a user is never offered something they cannot
install. Adding one is not: the item is absent from the candidate set
entirely, not ranked low.

Comparing the two name lists needs neither the model nor a network call, so
the gate runs in seconds. CI checks rather than fixes, for the same reason
it cannot regenerate.

Scoped to blocks and components. Examples are starter projects a user
scaffolds, never something catalog ranks, and the artifact carries no vector
for them, so demanding one would keep this gate permanently red and it would
be ignored within a week.

Verified in both directions rather than assumed: adding an unindexed item
exits 1 and names it, restoring the registry exits 0.
This commit is contained in:
Miguel Ángel
2026-08-09 17:45:59 +00:00
parent e203b199e8
commit f85301cbec
2 changed files with 97 additions and 0 deletions
+20
View File
@@ -34,6 +34,7 @@ jobs:
code: ${{ steps.filter.outputs.code }}
cli: ${{ steps.filter.outputs.cli }}
skills: ${{ steps.filter.outputs.skills }}
catalog_index: ${{ steps.filter.outputs.catalog_index }}
codex_plugin: ${{ steps.filter.outputs.codex_plugin }}
gcp_beginframe: ${{ steps.filter.outputs.gcp_beginframe }}
studio: ${{ steps.filter.outputs.studio }}
@@ -49,6 +50,10 @@ jobs:
with:
token: ""
filters: |
catalog_index:
- "registry/registry.json"
- "registry/catalog-artifact/**"
- "scripts/catalog/check-artifact-coverage.ts"
code:
- "packages/**"
- "scripts/**"
@@ -360,6 +365,21 @@ jobs:
# `gen:skills-manifest --check`, which compares per-skill content hashes; the
# manifest carries no version/timestamp, so it only fails on real content
# drift. bun runs the TS script directly, no install needed.
catalog-index-coverage:
name: "Catalog: search index covers the registry"
needs: changes
if: needs.changes.outputs.catalog_index == 'true'
runs-on: ubuntu-latest
timeout-minutes: 3
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
# Comparing two name lists needs neither the embedding model nor a
# network call, so this stays a seconds-long gate. Regenerating the
# vectors does need the model, which is why CI checks rather than fixes.
- name: Verify every searchable registry item has a vector
run: bun scripts/catalog/check-artifact-coverage.ts
skills-manifest:
name: "Skills: manifest in sync"
needs: changes