Skip to content

Run git cleanup's branch comparisons in parallel - #90

Open
mattmenefee wants to merge 1 commit into
mainfrom
speed-up-git-cleanup
Open

Run git cleanup's branch comparisons in parallel#90
mattmenefee wants to merge 1 commit into
mainfrom
speed-up-git-cleanup

Conversation

@mattmenefee

@mattmenefee mattmenefee commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • git pc took ~30 seconds in a repository with a few hundred local branches. Profiling showed nearly all of it inside cleanup rather than the pull — the pull itself is ~1s
  • The loop spends one git cherry per branch, and each one rebuilds a patch-id table covering every commit that branch sits behind the base. Cost scales with how stale a branch is: a branch forked ~7,000 commits ago measured 271ms against 22ms for one forked last week, and that table is rebuilt from scratch for every branch
  • Each iteration also piped git cherry output into grep just to test for a leading +, paying a second process per branch for something the shell can do itself
  • The comparisons are independent, so they now fan out across xargs -P sized to the machine's processors. Workers only classify and print a verdict; deletion stays serial and runs after every verdict is collected, so no ref disappears while another process is still walking the ref table
  • Verdicts are sorted by ref name before anything is deleted. Workers finish out of order, and the list of deleted branches is this command's audit trail — scanning a scrambled one for a branch you did not expect to lose is materially harder
  • The grep is replaced by scanning the output words for a bare +. That is exact only while git cherry stays unadorned — adding -v would put commit subjects in the stream and a subject containing a lone + would read as unmerged. The comments record that constraint, along with why branches travel NUL-delimited (ref names may contain quotes, which xargs would otherwise treat as syntax) and what the cleanup-worker argument guards against (without it xargs binds the first branch of every batch to $0, where the loop over $@ never reaches it — roughly one branch in twenty would go silently uncompared)
  • Measured on a 397-branch repository: 22.6s → 3.2s

Splitting classification from deletion introduces failure modes the serial loop could not have, so four guards come with it:

  • The reader re-checks the base, current, and repository base branches before deleting. Serially, "a protected branch can never reach the delete" was visible in one loop; split apart it becomes a claim about the whole pipeline
  • A fan-out exiting non-zero deletes nothing, rather than acting on a partial verdict list
  • The reader ignores any verdict whose ref is not under refs/heads/, so a torn or truncated line cannot reach git branch -D as a bare or half-formed name
  • The worker count rejects 0 (which means unbounded to GNU xargs) and caps at 16, since getconf reports host processors rather than any cgroup quota

One deliberate behavior change

A symbolic ref under refs/heads/ used to be kept or deleted purely on name order. Its comparison only failed if its target happened to sort ahead of it and had already been deleted — so aaa-aliaszzz-target was deleted, while zzz-aliasaaa-target survived, in otherwise identical repositories. Deciding every branch against one pre-deletion snapshot makes the case reachable consistently, and symbolic refs are now skipped outright: a symref is not a branch, and which of the two names git branch -D would take is the kind of question this command should decline.

Everything else is unchanged. Given this alias once force-deleted every branch in a master-based repository, that was verified by differential testing rather than inspection.

Test plan

Both implementations were extracted from the actual parsed alias (git config --get alias.cleanup) with git branch -D stubbed to an echo, then run side by side. Comparison is cmp on raw stdout and stderr plus exit code — not sorted, so ordering regressions cannot hide.

  • 397-branch repository — byte-identical
  • 370-branch repository where 250 branches are deleted in a single run, spanning ~19 worker batches — byte-identical, including deletion order
  • Branch merged by rebase (patch-equivalent, different SHA) — still deleted
  • Branch carrying a merge commit — still skipped with unique merge commits
  • Branch whose tip is an ancestor of the base — still deleted
  • Genuinely unmerged branch — still kept
  • Branch names containing ' and " — handled identically (confirmed refs cannot contain spaces, so the space-delimited verdict format is safe)
  • A tag shadowing the base branch's name — the documented ambiguity hazard, identical behavior
  • Empty candidate set — no spurious output, exit 0
  • Detached HEAD — identical
  • Base resolving only on origin — identical
  • Base that does not resolve — same abort message, exit 1
  • Explicit base argument — identical
  • Worker count resolves correctly for empty, non-numeric, 0, normal, and oversized getconf output
  • Fan-out returns 0 across repeated runs on the 397-branch repository, confirming the new non-zero check will not fire spuriously
  • Symbolic ref under refs/heads/, both name orderings, with real deletions inspecting .git/refs/heads/ rather than for-each-ref (which does not list a dangling symref)
  • Branch held by a linked worktree — git branch -D refuses, and the exit code matches the serial version
  • Partial fan-out: xargs aborting mid-run, and xargs absent entirely — both delete nothing, report comparison failed, and exit 1
  • Verified tr '\n' '\0' and xargs -0 -P behave as required on BSD/macOS, and that git's config parser passes the escapes through without introducing literal newlines or tabs

Review

Reviewed by four expert agents in isolated worktrees. Findings folded in:

  • Documentation — the cleanup-worker comment stated the wrong rationale and the wrong failure mode. Corrected: it guards against silent per-batch loss of the first branch, not an empty-input edge case that never triggers
  • Security — approved, no critical or high findings. Adversarial ref names (x;touch$IFS/tmp/pwn;y, backtick and $(...) payloads, --force, -D, 6000-byte names, a hostile clone controlling origin/HEAD) could not achieve injection or forge a verdict. Their protected-branch recheck and worker-count hardening are included above
  • Best practices — caught that the original differential testing compared sorted output, which hid the ordering regression now fixed and covered by the 250-deletion case above
  • Test-suite architect — found that stubbing git branch -D structurally cannot observe refs disappearing mid-run, which is how the symref divergence above went unnoticed; also that a partial fan-out failed silently while returning 0. Both are fixed and covered by real-deletion tests. Their remaining checks came back clean: 36 hostile-but-legal ref names, 3000 branches against ARG_MAX, batch boundaries at 19/20/21/40/400, octopus and orphan-root merges, hostile IFS, and every base-resolution path

@mattmenefee mattmenefee self-assigned this Aug 4, 2026
@mattmenefee
mattmenefee force-pushed the speed-up-git-cleanup branch 3 times, most recently from 692f4f4 to 901ad59 Compare August 4, 2026 21:14
`git pc` took around thirty seconds in a repository holding a few
hundred local branches, nearly all of it inside `cleanup` rather than
the pull. The loop spends one `git cherry` per branch, and each of
those rebuilds a patch-id table covering every commit that branch sits
behind the base — so a branch forked seven thousand commits ago costs
more than ten times one forked last week, and that table is rebuilt
from scratch for every branch. On top of that, each iteration piped
`git cherry` output into `grep` just to look for a leading `+`, paying
a second process per branch for a test the shell can do itself.

The comparisons never depended on one another, so they now fan out
across `xargs -P` sized to the machine's processors. Workers only
classify and print a verdict; deletion stays serial and happens after
every verdict is in, so no ref vanishes while another process is still
walking the ref table. Verdicts are sorted by ref name before anything
is deleted, since workers finish out of order and the list of deleted
branches is this command's audit trail. The `grep` is gone in favor of
scanning the output words for a bare `+`, which is exact as long as
`git cherry` stays unadorned — the comments record that constraint
along with why branches travel NUL-delimited and what the
`cleanup-worker` argument is guarding against.

Splitting classification from deletion adds failure modes the serial
loop could not have, so four guards come with it. The reader re-checks
the base, current and repository base branches before deleting,
because "a protected branch never reaches the delete" used to be
visible in a single loop and is now a claim about a whole pipeline. It
also ignores any verdict whose ref is not under refs/heads/, so a torn
line cannot arrive as a bare or half-formed name. A fan-out that exits
non-zero deletes nothing rather than acting on a partial verdict list.
And the worker count rejects zero, which means unbounded to GNU xargs,
and caps at sixteen, since `getconf` reports host processors rather
than any cgroup quota.

One behavior does change. A symbolic ref under refs/heads/ used to be
kept or deleted purely on name order: its comparison only failed if
its target happened to sort ahead of it and had already been deleted,
so the same repository answered differently depending on what the two
were called. Deciding every branch against one pre-deletion snapshot
makes that reachable consistently, and the answer chosen is to skip
symbolic refs outright — a symref is not a branch, and which of the
two names `git branch -D` would take is the kind of question this
command should decline.

Nothing else moves. Old and new produce byte-identical stdout, stderr
and exit codes on the repository that prompted this, on a 370-branch
repository where 250 branches are deleted in one run, and for branches
merged by rebase, branches carrying merge commits, branch names
containing quotes, a tag shadowing the base branch's name, an empty
candidate set, a detached HEAD, a branch held by a linked worktree, a
base living only on origin, and a base that does not resolve. The run
drops from roughly twenty-three seconds to under four.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant