Compare commits

...
Author SHA1 Message Date
ethernet c24a1a7e25 perf(tools): text prefilter before AST parse in tool discovery
`_module_registers_tools()` reads each `tools/*.py` file and fully
AST-parses it to check for a top-level `registry.register()` call.
90 files are scanned on every process start — but only 32 actually
register tools.

Add a cheap text prefilter: after reading the file (which we need to
do anyway for AST), check that both `"registry"` and `"register"`
appear in the source before calling `ast.parse`. A file with a
top-level `registry.register()` call must contain both strings, so
this is a perfect superset — zero false negatives. 50 of 90 files
skip the AST parse entirely.

The `source=` parameter is not threaded through `discover_builtin_tools`;
the prefilter lives entirely inside `_module_registers_tools`, keeping
the public API unchanged.

Benchmark (median of 10 runs, scanning 90 files):

  before (read + ast.parse all):  305.9ms
  after  (text prefilter + ast):   187.8ms
  speedup: 1.6x  (118ms saved)

Identical module set: 32 modules, same names, same order.
2026-07-13 15:35:53 -04:00
ethernet 92d083b77d feat(ci): python test speedups 2026-07-13 15:07:50 -04:00
3 changed files with 20 additions and 4 deletions
+9 -1
View File
@@ -71,6 +71,15 @@ echo " (TZ=UTC LANG=C.UTF-8 PYTHONHASHSEED=0; clean env)"
cd "$REPO_ROOT"
# ── Pre-compile .pyc bytecode cache ─────────────────────────────────────────
# Each test file runs in its own subprocess via run_tests_parallel.py.
# Pre-building the bytecode cache once here (instead of each subprocess
# compiling on first import) avoids redundant work across ~2000 processes.
# Uses git to list tracked .py files (skips venv, node_modules, etc).
echo "▶ pre-compiling bytecode cache"
"$PYTHON" -m compileall -q -j 0 -- $(git ls-files '*.py') >/dev/null 2>&1 || true
echo "▶ launching test runner"
exec env -i \
PATH="$PATH" \
HOME="$HOME" \
@@ -78,7 +87,6 @@ exec env -i \
LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
PYTHONHASHSEED=0 \
PYTHONDONTWRITEBYTECODE=1 \
${HERMES_RUN_SLOW_PET_TESTS:+HERMES_RUN_SLOW_PET_TESTS="$HERMES_RUN_SLOW_PET_TESTS"} \
${EXTRA_PYTHONPATH:+PYTHONPATH="$EXTRA_PYTHONPATH"} \
${EXTRA_PYTEST_PLUGINS:+PYTEST_PLUGINS="$EXTRA_PYTEST_PLUGINS"} \
+1 -2
View File
@@ -260,8 +260,7 @@ def _run_one_file(
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
# skipping writing bytecode because we're running a bunch of parallel python processes on the same code
env={**os.environ, 'PYTHONDONTWRITEBYTECODE': '1'},
env=os.environ,
# POSIX: place the child at the head of its own process group so
# _kill_tree can SIGKILL the group atomically.
# Windows: this maps to CREATE_NEW_PROCESS_GROUP in CPython 3.12+;
+10 -1
View File
@@ -45,11 +45,20 @@ def _module_registers_tools(module_path: Path) -> bool:
Only inspects module-body statements so that helper modules which happen
to call ``registry.register()`` inside a function are not picked up.
A cheap text prefilter avoids the ``ast.parse`` cost for files that do not
mention both ``registry`` and ``register`` — a necessary condition for a
top-level ``registry.register()`` call to exist.
"""
try:
source = module_path.read_text(encoding="utf-8")
except OSError:
return False
if "registry" not in source or "register" not in source:
return False
try:
tree = ast.parse(source, filename=str(module_path))
except (OSError, SyntaxError):
except SyntaxError:
return False
return any(_is_registry_register_call(stmt) for stmt in tree.body)