perf(hash-join): eliminate intermediate array allocations in probe-side collision filter

[LiaCastaneda]

Which issue does this PR close?

• Closes High HashJoin probe latency under high fanout and key skew (Partitioned mode) #23237

Rationale for this change

When HashJoinExec builds the probe the hash table, it collects all candidate (build_idx, probe_idx) pairs and then rechecks every pair with equal_rows_arr (shows in the profile, taking 20% of the query CPU) to filter out hash collisions. This recheck materializes intermediate Arrow arrays sized to the number of matched pairs (take + eq_dyn_null + FilterBuilder), making it O(matched_pairs) in both allocations and comparisons.

For high-fanout joins (many build rows per distinct key), matched_pairs = probe_rows × fanout. At fanout 78× over 2.3M probe rows this produces ~176M pairs — and the recheck runs on all of them, even though nearly all pass.

This cost is most visible when:

• The build side has duplicate keys
• The join output is large (high fanout × large probe side)
• Hash partition skew concentrates most matched pairs onto a single partition, making the per-partition allocation cost directly observable as query latency

What changes are included in this PR?

Step 1: Replace equal_rows_arr with an in-place JoinKeyComparator loop , eliminating the O(matched_pairs) Arrow allocations in the fallback path.

Step 2: At build time, scan the next chain once to detect whether the map is collision-free (all adjacent linked pairs share the same key). If so, the probe phase checks once per run of consecutive same-probe-idx pairs and accepts the entire run.

Are these changes tested?

Existing tests pass, I also added some tests for the new function that detects hash collisions detect_key_collisions

I did some profiling on the added Q23:

• profile of Q23 main
• profile

We can see the CPU for the HashJoin was cut in half.
Also the benchmarks results show Q23 runs ~x2.5 times faster:

• main: 1.01s
• this PR: 0.42s

Are there any user-facing changes?

No

[LiaCastaneda]

I checked in the other engine (Trino) and there when a build row is inserted, Trino only inserts rows that share the same key, so by construction, every chain is already "pure" and hash collisions are not possible.
This happens because Trino uses a hash table that resolves key equality at insert time. DataFusion's update_from_iter only receives hashes and row indices (it has no access to key values) so it chains all rows in the same hash bucket together regardless of whether they share a key or not.

[LiaCastaneda]

Note doing the same in DF would be non trivial because it would require modifying update_from_iter trait signature to accept key columns alongside the hashes, and updating all call sites accordingly.