mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-08-31 02:41:44 +00:00
Miguel Ángel
a562946e11
`docs/changelog.mdx` reached 512 KB and started failing the 500 KB pre-commit check, so `chore: release v0.8.14` could only be committed by passing `HF_MAX_NONLFS_KB`. Every release from here would need the same override, and the file grows a few KB each time. The check is for binaries. Its own error message says "large binaries are being committed to git instead of LFS", and its header explains why: an ONNX model, HDR-regression MP4s, demo clips, each of which lives in history forever and is paid for by every clone. That cost is specific to binaries. Git delta-compresses text, so release notes that grow a few KB per commit add a few KB to the pack, while a binary of the same size re-enters the pack whole on every edit. Text is now exempt regardless of size, detected with `grep -I` (a file with NUL bytes is binary), the same heuristic git uses for "Binary files differ". The binary rule, the `registry/` exemption, the LFS check and the size threshold are all unchanged. The alternative was an allowlist entry for the one file, which would leave the next legitimately growing text file to hit the same wall and get the same one-off exemption. `scripts/check-large-files.sh` had no test. It has one now, wired into `test:scripts` so it actually runs: an over-limit binary fails, an over-limit text file passes, an under-limit binary passes, and multiple offenders are all named. Verified the text case fails with the exemption removed, so the test pins the behaviour rather than describing it.
98 lines
4.2 KiB
Bash
Executable File
98 lines
4.2 KiB
Bash
Executable File
#!/bin/sh
|
|
# Reject large binaries committed straight into the git pack instead of LFS.
|
|
#
|
|
# Why this exists: the repo's history carries hundreds of MB of binaries that
|
|
# should have been LFS — a 31 MB ONNX model, nested HDR-regression MP4s that
|
|
# dodged non-recursive .gitattributes globs, demo clips, scratch renders. Each
|
|
# was "noticed later and deleted," but a raw commit lives in history forever and
|
|
# every clone pays for it. This hook stops the next one at commit time.
|
|
#
|
|
# Rule: any staged file larger than $MAX_KB that is NOT routed through Git LFS
|
|
# fails the commit. Fix by either adding an LFS pattern in .gitattributes for
|
|
# that path/extension, or not committing the file (assets/, gitignore, etc.).
|
|
#
|
|
# Usage:
|
|
# check-large-files.sh # default: check the staged file set
|
|
# check-large-files.sh <file> [<file>] # explicit files (handy for testing)
|
|
#
|
|
# We read the staged set ourselves rather than taking lefthook's {staged_files}
|
|
# expansion: that expands to a bare space-separated string, which splits paths
|
|
# containing spaces into separate args. `git diff --cached` + a line-based read
|
|
# keeps whole paths intact (only a literal newline in a filename would break it,
|
|
# which git quotes/escapes anyway).
|
|
|
|
set -u
|
|
|
|
MAX_KB="${HF_MAX_NONLFS_KB:-500}"
|
|
|
|
# Emit the list of paths to check, one per line.
|
|
list_files() {
|
|
if [ "$#" -gt 0 ]; then
|
|
printf '%s\n' "$@"
|
|
else
|
|
# Added/Copied/Modified/Renamed staged paths (skip Deleted — nothing to size).
|
|
git diff --cached --name-only --diff-filter=ACMR
|
|
fi
|
|
}
|
|
|
|
violations="$(mktemp)"
|
|
trap 'rm -f "$violations"' EXIT INT TERM
|
|
|
|
list_files "$@" | while IFS= read -r f; do
|
|
[ -n "$f" ] || continue
|
|
|
|
# Skip symlinks: `wc -c` would measure the link *target's* bytes, so a symlink
|
|
# to a large LFS-tracked asset could be flagged even though the real blob is a
|
|
# tiny pointer. Symlinks themselves are never the bloat we're hunting.
|
|
[ -L "$f" ] && continue
|
|
[ -f "$f" ] || continue
|
|
|
|
# registry/ intentionally ships raw binary assets (block backgrounds, avatar
|
|
# PNGs, .glb models, audio) so installed blocks stay portable without an LFS
|
|
# round-trip. Those are the product, not accidental bloat — skip them here.
|
|
case "$f" in registry/*) continue ;; esac
|
|
|
|
# Text is exempt, whatever its size, because the cost this hook exists to stop
|
|
# is a binary one. Git delta-compresses text, so a file that grows by a few KB
|
|
# per commit adds a few KB to the pack. A binary of the same size re-enters the
|
|
# pack whole on every edit, which is exactly how the history got its hundreds
|
|
# of megabytes. `docs/changelog.mdx` is the case that forced this: half a
|
|
# megabyte of release notes, a little larger every release, tripping a check
|
|
# whose own error message says "large binaries".
|
|
#
|
|
# `grep -I` treats a file containing NUL bytes as binary, the same heuristic
|
|
# git uses to print "Binary files differ". A generated blob of text is still
|
|
# caught by review, not here.
|
|
grep -qI . "$f" 2>/dev/null && continue
|
|
|
|
bytes="$(wc -c < "$f" | tr -d ' ')"
|
|
# Ceiling division: a sub-1024-byte file must report >=1 KB, never 0, so it
|
|
# can't slip past a strict threshold (e.g. HF_MAX_NONLFS_KB=0). Plain
|
|
# `bytes / 1024` would round a 512-byte binary down to 0 and pass it.
|
|
kb=$(( (bytes + 1023) / 1024 ))
|
|
[ "$kb" -le "$MAX_KB" ] && continue
|
|
|
|
# Is this path routed through LFS? `git check-attr` reads .gitattributes.
|
|
filter="$(git check-attr filter -- "$f" | sed 's/.*: //')"
|
|
[ "$filter" = "lfs" ] && continue
|
|
|
|
printf '%s\t%s\n' "$kb" "$f" >> "$violations"
|
|
done
|
|
|
|
# `while` ran in a pipeline subshell, so it couldn't set a parent-shell flag —
|
|
# the violations file is the durable signal.
|
|
if [ -s "$violations" ]; then
|
|
echo "ERROR: large binaries are being committed to git instead of LFS." >&2
|
|
echo " (limit: ${MAX_KB} KB — override per-commit with HF_MAX_NONLFS_KB)" >&2
|
|
echo >&2
|
|
while IFS=' ' read -r kb f; do
|
|
echo " • ${f} (${kb} KB)" >&2
|
|
done < "$violations"
|
|
echo >&2
|
|
echo "Fix: add an LFS pattern for it in .gitattributes, e.g." >&2
|
|
echo " path/to/**/*.ext filter=lfs diff=lfs merge=lfs -text" >&2
|
|
echo " then re-stage the file. Or, if it should not be committed at all," >&2
|
|
echo " add it to .gitignore." >&2
|
|
exit 1
|
|
fi
|