Improve parallel scaling of e-matching

[ezrosent]

(branched off of perf-shared-trie-nodes)

This change improves parallel e-matching scaling with a few optimizations:

• Avoids highly-contended Arc::clone calls with index construction: handles on shared indexes are grabbed before the plan starts running to avoid big clone calls. This alone was a large win.
• Add a "fast path" where the top relation is coarse-grained parallelized rather than the fine-grained behavior we had before. This is another big speedup, when the top relation is large enough. (Removing Arc clones was still higher though!).
• Reintroduce a limited form of work-stealing by adding a separate spawn_local method onto the thread pool that allows for work to be appended to a local deque first, only migrating to the global queue if sufficient threads are stalled. This appears to help cases where there isn't a lot of parallelism to expose. The thread pool essentially sprayed data randomly across threads, which makes cache locality much harder to achieve. (The original morsel-driven parallelism paper talks about this too). Local spawns maintain locality while still improving parallel utilization if there's a lot of skew.

To evaluate this I had codex write up a benchmark using the dataset and queries from the Honeycomb paper. This is just a single query over a large dataset, so it's a helpful test case for this part of the code: other egglog benchmarks have complex schedules and many rules that we parallelize across sometimes.

Here are the overall results on my m4 max laptop:

(Note that M4 max only has 12 P-cores, so some amount of flattening after 12 is expected. Still, I suspect there's more to do here)

Workload	1 thread	8 threads	12 threads	16 threads
Honeycomb q6/count	9.271s → 9.126s (1.02×)	3.693s → 1.778s (2.08×)	3.476s → 1.480s (2.35×)	3.691s → 1.411s (2.62×)
gemma.egg	3.436s → 3.615s (0.95×)	1.462s → 1.214s (1.20×)	1.524s → 1.200s (1.27×)	1.651s → 1.273s (1.30×)
gemma4_moe.egg	12.366s → 12.864s (0.96×)	4.695s → 3.865s (1.21×)	4.748s → 3.713s (1.28×)	4.870s → 3.894s (1.25×)
hardboiled_conv1d_128.egg	0.220s → 0.219s (1.00×)	0.256s → 0.210s (1.22×)	0.339s → 0.286s (1.19×)	0.415s → 0.330s (1.26×)
hardboiled_conv1d_32.egg	0.089s → 0.090s (0.99×)	0.106s → 0.107s (1.00×)	0.109s → 0.109s (1.00×)	0.115s → 0.116s (0.99×)
llama.egg	0.306s → 0.308s (1.00×)	0.340s → 0.345s (0.99×)	0.344s → 0.346s (1.00×)	0.350s → 0.350s (1.00×)
luminal-llama.egg	0.065s → 0.081s (0.80×)	0.078s → 0.095s (0.82×)	0.085s → 0.102s (0.83×)	0.085s → 0.102s (0.83×)
paged_llama.egg	1.204s → 1.204s (1.00×)	1.262s → 1.262s (1.00×)	1.289s → 1.295s (1.00×)	1.281s → 1.287s (0.99×)
qwen.egg	0.349s → 0.354s (0.99×)	0.394s → 0.364s (1.08×)	0.423s → 0.391s (1.08×)	0.446s → 0.407s (1.09×)
qwen3_moe.egg	0.458s → 0.466s (0.98×)	0.502s → 0.440s (1.14×)	0.535s → 0.468s (1.14×)	0.587s → 0.510s (1.15×)
whisper.egg	0.994s → 1.018s (0.98×)	0.742s → 0.605s (1.23×)	0.784s → 0.629s (1.25×)	0.868s → 0.679s (1.28×)
Geomean, .egg workloads	0.96×	1.08×	1.09×	1.10×

[codecov-commenter]

Codecov Report

❌ Patch coverage is 89.86961% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.82%. Comparing base (95f539c) to head (e4ebef2).

Files with missing lines	Patch %	Lines
core-relations/src/free_join/execute.rs	83.91%	92 Missing ⚠️
concurrency/src/threadpool/mod.rs	97.89%	8 Missing ⚠️
core-relations/src/table/mod.rs	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #954      +/-   ##
==========================================
+ Coverage   86.65%   86.82%   +0.17%     
==========================================
  Files          95       95              
  Lines       29695    30591     +896     
==========================================
+ Hits        25732    26561     +829     
- Misses       3963     4030      +67     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[codspeed-hq]

Merging this PR will not alter performance

✅ 37 untouched benchmarks
⏩ 227 skipped benchmarks¹

---

_{Comparing ezr-better-parallel (e4ebef2) with main (95f539c)}

Footnotes

1. 227 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[yihozhang]

Had some nits and questions. I didn't carefully review the new threadpool implementation besides the high-level description. I also didn't carefully read the new tests.

[yihozhang]

From running uv run scripts/bench.py run perf-shared-trie-nodes ezr-better-parallel (serial mode).

# Benchmark diff

  Generated  : 2026-07-25T14:17:12
  Baseline   : 225ad44d (225ad44d2379)  —  add a comment
  Comparison : a384d253 (a384d25325da)  —  Remove retired generic-join tuning paths

  Benchmark                                  Before (s)   After (s)     Δ (s)      Δ %
  ────────────────────────────────────────────────────────────────────────────────────────
  hardboiled_conv1d_32.egg                        0.118       0.115   -0.003    -2.5%  ▼ faster
  hardboiled_conv1d_128.egg                       0.301       0.291   -0.010    -3.3%  ▼ faster
  luminal-llama.egg                               0.083       0.111  +  0.028  +  33.5%  ▲ slower
  python_array_optimize.egg                       0.233       0.235  +  0.002  +   0.8%  ▲ slower
  cykjson.egg                                     0.032       0.034  +  0.002  +   5.5%  ▲ slower
  eggcc-extraction.egg                            0.203       0.216  +  0.012  +   6.0%  ▲ slower
  llama.egg                                       0.373       0.379  +  0.006  +   1.7%  ▲ slower
  paged_llama.egg                                 1.616       1.634  +  0.018  +   1.1%  ▲ slower
  qwen.egg                                        0.424       0.447  +  0.023  +   5.4%  ▲ slower
  qwen3_moe.egg                                   0.586       0.588  +  0.002  +   0.3%  ·
  whisper.egg                                     1.249       1.278  +  0.029  +   2.3%  ▲ slower

Update: numbers from nightly are mostly consistent with the above

• ezr-better-parallel: https://nightly.cs.washington.edu/reports/egglog/1784552964:ezr-better-parallel:a384d253/
• perf-shared-trie-nodes: https://nightly.cs.washington.edu/reports/egglog/1784364193:perf-shared-trie-nodes:225ad44d/

But the packed trie PR #959 shows some significant speedups.

• ezr-better-parallel-packed-trie: https://nightly.cs.washington.edu/reports/egglog/1784967636:ezr-better-parallel-packed-trie:c7fc1d77/

[ezrosent]

I think the llama slowdown is due to the buggy variable ordering heuristic that codex snuck in; it's reverted in the packed-trie branch.

[yihozhang]

new perf numbers using this script

(I had another run using the existing bench.py that's worse but I think it's the artifact of that script. The updated regression is much slower.)

# Benchmark diff

  Generated  : 2026-08-11T17:18:41
  Baseline   : 95f539cc (95f539cca027)  —  Merge pull request #973 from egraphs-good/fix/971-intersect-transient-unsorted
  Comparison : e4ebef25 (e4ebef25b5d3)  —  Address parallel join review feedback

  Benchmark                                  Before (s)   After (s)     Δ (s)      Δ %
  ────────────────────────────────────────────────────────────────────────────────────────
  hardboiled_conv1d_32.egg                        0.115       0.116  +  0.001  +   1.0%  ▲ slower
  hardboiled_conv1d_128.egg                       0.291       0.293  +  0.002  +   0.6%  ▲ slower
  luminal-llama.egg                               0.071       0.072  +  0.001  +   0.7%  ▲ slower
  python_array_optimize.egg                       0.230       0.234  +  0.004  +   1.9%  ▲ slower
  cykjson.egg                                     0.032       0.033  +  0.001  +   1.9%  ▲ slower
  eggcc-extraction.egg                            0.213       0.215  +  0.002  +   1.0%  ▲ slower
  llama.egg                                       0.371       0.372  +  0.000  +   0.1%  ·
  paged_llama.egg                                 1.655       1.680  +  0.024  +   1.5%  ▲ slower
  qwen.egg                                        0.414       0.415  +  0.000  +   0.1%  ·
  qwen3_moe.egg                                   0.549       0.550  +  0.001  +   0.2%  ·
  whisper.egg                                     1.148       1.154  +  0.007  +   0.6%  ▲ slower

  Summary: 0 faster  ·  8 slower  ·  3 unchanged  ·  0 missing
  Overall average Δ: +0.87%

gemma4_moe.egg is up to ~10% slower

• https://nightly.cs.washington.edu/reports/egglog/1786383546:ezr-better-parallel:e4ebef25/
• https://nightly.cs.washington.edu/reports/egglog/1786263356:main:95f539cc/