Support LEFT/RIGHT JOIN in the DuckDB IEJoin dialect by decomposing the outer join into INNER pairs plus unmatched rows — Closes #95

[conradbzura]

Summary

Accelerate LEFT / RIGHT JOIN on a column-to-column INTERSECTS by decomposing the outer join rather than emitting one. Both halves reach DuckDB's IE_JOIN, where the shape previously declined to the naive overlap predicate — a hash join on a 24-value chromosome key with the position inequalities as a residual filter, quadratic and unable to finish at a million intervals per side. At 2^20 the decomposition returns 1,334,564 rows in 0.78s; at 2^22 it returns 7,758,528 rows in 2.0s, scaling linearly.

A LEFT JOIN is exactly the INNER pairs unioned with the row-preserving side's unmatched rows, NULL-filled on the other side. Both halves already had fast paths, so this emits them and unions the result instead of hoping the planner chooses well for an outer join. RIGHT is the mirror, reached by swapping the FROM and joined tables.

The issue originally prescribed emitting a per-chromosome LEFT JOIN. That design was benchmarked and rejected: it is faster below ~1e5 rows and then collapses, failing to finish within 300s at 2^22. EXPLAIN reports IE_JOIN for it throughout, so plan inspection cannot distinguish the two designs and only execution at scale does — which is why the test suite includes a scale test rather than relying on plan assertions.

The trade-off is two passes over the data, making the decomposition roughly 2x slower than a per-chromosome outer join at toy sizes. Predictable linear scaling is worth that.

Two rounds of independent review shaped what landed. The first found five defects a green suite passed straight through, each a shape that previously declined to the naive plan and answered correctly. The second found one behavioral defect — the rewrite renamed duplicate output columns — plus a regression lock that never reached the gate it was written to protect: deleting that gate left all 2131 tests passing while the dialect answered a semi join as an outer join. Roughly 1,900 randomized differential cases across seven independent fuzz runs found no row or schema divergence, so the correctness work here is in the gates and the tests rather than in the plan.

bedtools intersect -wao remains on the naive plan: its CASE projection is blocked by the projection gate independently of the outer join, and unblocks with #109.

Closes #95
Closes #226

Proposed changes

Return the IEJoin setup and SELECT separately

The dialect emitted a multi-statement script, and any rewrite composing on top recovered the pieces by splitting the rendered string on the statement separator. An identifier containing that sequence splits the script inside a quoted alias and the query no longer parses. _build_sql now returns (setup, select), transform_to_sql is a thin joiner, and composing builders consume the parts directly. This also repairs the same latent defect in the shipped count_overlaps path.

Decompose the outer join

_match_outer_join_decomposition claims LEFT/RIGHT shapes whose projections are side-attributable columns; _build_outer_join_parts emits the matched half, the unmatched half, and the union. The halves partition chromosomes differently by design — the matched half intersects both sides, the unmatched half enumerates the preserved side alone — which is what lets rows on a chromosome the other table lacks still surface.

Dispatch sits after the count_overlaps matcher, which keeps its faster zero-fill path, and before the existing outer-join decline, which still catches FULL OUTER and the WHERE-INTERSECTS shape.

Preserve rows whose chromosome is NULL

Such rows can never match, and neither half surfaces them on its own: both partitions come from SELECT DISTINCT over the chromosome, where a NULL renders as a NULL literal that string_agg skips, so no branch is emitted for it. They are unioned in directly. The root cause is shared with the standalone ANTI path, which has dropped these rows since #208 and is filed separately; what this fixes is LEFT/RIGHT inheriting it instead of declining safely.

Preserve duplicate output column names

The matched half is a UNION ALL branch directly rather than a derived table. SELECT * over a subquery makes DuckDB de-duplicate repeated output names, and that renaming becomes the union's schema, so the canonical bedtools projection — a.chrom, a.start, a.end, b.chrom, b.start, b.end — came back as chrom, start, end, chrom_1, start_1, end_1 under dialect="duckdb" and unchanged under every other dialect. A flag documented as a performance opt-in must not alter the result schema.

Gate on a whitelist rather than a blocklist

The rewrite re-emits the query as a union of two independently transpiled halves, so any top-level clause it does not itself read would be applied per half instead of over the union — LIMIT 2 returning four rows, QUALIFY silently dropped. Rejecting anything outside the set the builder consumes forecloses the class rather than the instance. TABLESAMPLE is rejected separately since it rides on the table node, not the top-level SELECT.

Decline case-insensitively colliding output names

DuckDB resolves identifiers case-insensitively even when quoted, so AS x alongside AS X bound both positions to the first column, returning the wrong value and widening that column to VARCHAR for the matched rows too. The uniqueness gate now case-folds through _normalize_alias.

Share one gate prelude between both matchers

_match_count_overlaps and _match_outer_join_decomposition opened with near-identical preludes differing only in the accepted join side. _resolve_intersects_join now performs those checks once and returns the resolved join. The duplication had a concrete cost: the untested-gate defect below existed in both copies, so it had to be found twice.

Mark the bedtools integration modules

pytest does not honour pytestmark declared in a conftest, so the entire bedtools oracle lane was invisible to marker-based selection — pytest -m integration collected 250 of 333 tests and -m "not integration" ran the binary-dependent lane it exists to skip. Each module now declares the marker, matching the datafusion lane. Collection is 335 of 335.

Correct stale documentation

The public transpile docstring listed LEFT/RIGHT among the shapes the dialect declines. The README, spatial-operators page, and performance guide carried the same INNER/SEMI/ANTI-only claim, as did the canonical grammar line on the page describing the feature, the registry entry point's decline list, and the naive-predicate module documenting when it is the fallback.

The performance guide now documents the third UNION ALL branch, which is load-bearing correctness the previous text omitted, and completes the decline list with self-joins, TABLESAMPLE, and case-only name collisions. Both documented -loj recipes project a star and therefore decline, so the migration guide says so rather than leaving users to conclude the flag does nothing.

Two execution details also changed: the emitted script is no longer always two statements, and a decomposed query declares one session variable per half.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A LEFT-join INTERSECTS with matched, unmatched, and other-chromosome rows	The query is transpiled with dialect="duckdb" and executed	Rows equal the Python LEFT-join reference	Core semantics
2	TestTranspileDuckDBIEJoinOuterJoinDecomposition	Duplicate rows on both sides	The decomposed query runs	Row multiplicity matches the reference exactly	Union multiplicity
3	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A RIGHT-join INTERSECTS with unmatched right rows	The query is transpiled and executed	Rows equal the mirrored reference	FROM/join swap
4	TestTranspileDuckDBIEJoinSQLStructure	A RIGHT-join INTERSECTS	The query is transpiled	The unmatched half partitions on the preserved side alone	Swap emission
5	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A projection whose six columns carry three output names	The query is transpiled and executed	Column labels match the naive plan rather than being renumbered	Duplicate output names
6	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A NULL-fill alias shadowing a preserved output name ahead of it	The query is transpiled and executed	The bare reference binds the relation column, not the lateral NULL alias	Name-resolution near-miss
7	TestTranspileDuckDBIEJoinOuterJoinDecomposition	Preserved output names differing only by letter case	The query is transpiled and executed	It declines and matches the naive plan in rows and result schema	Case-insensitive collision
8	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A preserved side containing NULL chromosomes, including all-NULL	The query is transpiled and executed	Every NULL-chromosome row surfaces NULL-filled	NULL chromosome preservation
9	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A top-level QUALIFY, LIMIT, OFFSET, GROUP BY, or ORDER BY	The query is transpiled and executed	It declines and matches the naive plan	Clause whitelist
10	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A TABLESAMPLE on the FROM table or on the joined table	The query is transpiled and executed	It declines and remains executable	Sampled operand, gated per side
11	TestTranspileDuckDBIEJoinOuterJoinDecomposition	An output alias containing the statement separator	The decomposition and count_overlaps queries run	Both execute and match the naive plan	Structural parts seam
12	TestTranspileDuckDBIEJoinOuterJoinDecomposition	Colliding preserved names, a WHERE, a star, a self-join, an ON residual, a repeated INTERSECTS, or a subquery operand	The query is transpiled and executed	It emits no session variable and matches the naive plan	Decline as one unit
13	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A LEFT SEMI or LEFT ANTI join, which parses with side='LEFT' and reaches the kind gate	The query is transpiled	At most one session variable is declared and no unmatched half is emitted	Kind gate
14	TestTranspileDuckDBIEJoinOuterJoinDecomposition	A bare SEMI or ANTI join, which parses with no side and is rejected a gate earlier	The query is transpiled	Exactly one session variable is declared and no unmatched half is emitted	Side gate
15	TestTranspileDuckDBIEJoinOuterJoinDecomposition	Tables configured with a custom chromosome column and an other-side-only projection	The query is transpiled and executed	The configured column drives the unmatched half	Table config resolution
16	TestTranspileDuckDBIEJoinOuterJoinDecomposition	20,000 rows per side with a share unmatched	Each session variable is planned separately	Both halves plan through IE_JOIN and neither through BLOCKWISE_NL_JOIN	Both halves reach the fast operator
17	TestTranspileDuckDBIEJoinOuterJoinDecomposition	Two decomposed queries sharing one connection	Their setup statements are interleaved before either SELECT	Each returns its own rows	Session-variable isolation
18	TestTranspileDuckDBIEJoinOuterJoinDecomposition	Empty, half-empty, and chromosome-disjoint tables	The query is transpiled and executed	Rows and result schema match the naive plan	NULL-fill typing
19	TestTranspileDuckDBIEJoinOuterJoinDecomposition	262,144 rows per side with a sixth unmatched	The decomposed query runs	Every left row survives as a distinct key, the unmatched half contributes, and it finishes inside the bound	Scale, which plan assertions cannot cover
20	TestTranspileDuckDBIEJoinOuterJoinDecomposition	Hypothesis-generated intervals including NULL chromosomes and zero-length spans	The query is transpiled and executed	Rows equal the Python reference as a multiset	Randomized correctness
21	tests/integration/bedtools/test_intersect.py	Interval sets with an unmatched row and a chromosome absent from B	The decomposed query is compared to bedtools intersect -loj	Output matches exactly	Real-oracle agreement
22	tests/integration/bedtools/test_intersect.py	A RIGHT join against the same inputs exchanged	The decomposed query is compared to bedtools intersect -loj	Output matches exactly	RIGHT against a real oracle
23	tests/integration/bedtools/test_intersect.py	Byte-identical duplicate rows on both sides	The decomposed query is compared to bedtools intersect -loj	Output matches exactly	Multiplicity against a real oracle
24	tests/integration/bedtools/test_intersect_property.py	Hypothesis-generated interval sets	A non-DISTINCT LEFT join is transpiled with dialect="duckdb"	The fast path fires and output matches bedtools -loj	Randomized oracle agreement
25	tests/integration/datafusion/test_cross_target_oracle.py	The same query across generic, datafusion, and duckdb targets	Each target runs it	All agree and the duckdb target is asserted to have decomposed	Cross-target equivalence