feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2]

[schenksj]

Part 2 of the Delta Lake contrib PR breakup. Part 1 (#4700, the core SPI for contrib leaf scans) is merged. This part establishes the contrib-delta build gate and the inert wiring that lets a gated build compile and link end to end — while the default build stays byte-for-byte unchanged (zero Delta surface). It ships no real Delta read logic: a Delta read that reaches native returns a clean "not implemented" error and falls back to vanilla Spark. The full sequence and dependency graph live in the tracking umbrella, #4366.

The whole point of this part is that everything it adds is gated or inert, so it is safe to land on main well ahead of the read path, and reviewers can review the wire format, the build machinery, and the gate enforcement once, in isolation, before any Delta code shows up.

Changes

Build gate

• Maven contrib-delta profile (spark/pom.xml) with a per-Spark delta.version (3.5 → 3.3.2, 4.0 → 4.0.0, 4.1 → 4.1.0) and an add-source of contrib/delta/src. Default delta.version floor in pom.xml. The default spark.version stays 4.1.2 — the delta-spark 4.1.1 pin is a separate, deferred decision (raised for later parts).
• Cargo contrib-delta feature on core (optional path dep on comet-contrib-delta); native/Cargo.toml excludes ../contrib from the workspace so non-Delta committers never build it.
• dev/verify-contrib-delta-gate.sh proves the default cargo tree, Maven dependency set, compiled classes, and libcomet all carry zero Delta surface, and that the gated build pulls the right deps (delta-spark per Spark profile, comet-contrib-delta in the cargo tree). Wired into a minimal delta_build_gate.yml CI job. The full test-suite and regression workflows land in later parts.

Inert wiring

• Proto: Delta* messages and delta_scan = 118 (117 is BroadcastNestedLoopJoin). One-time wire-format review here, early.
• Native dispatch: an OpStruct::DeltaScan arm that returns a not-compiled-in error on default builds, and a feature-gated delta_scan shim that calls the contrib; exhaustive-match arms in operator_registry / jni_api; convert_spark_types_to_arrow_schema promoted to pub(crate).
• Stub contrib crate (contrib/delta/native): plan_delta_scan returns DataFusionError::NotImplemented — just enough to satisfy the core shim's contract so --features contrib-delta links. This makes the exact core↔contrib contract visible in a small PR.
• JVM bridge DeltaIntegration (reflective; every lookup returns None until the contrib classes exist), the CometExecRule Delta-marker hook (the CDF hook is deferred to a later part), the CometScanRule Delta delegation + metadata-column reorder, and the leaf DeltaConf.

What this part deliberately does NOT do yet

• No Rust read path — plan_delta_scan is a stub. Log replay, predicate pushdown, deletion vectors, and the kernel read land in parts 3a/3b.
• No Scala claim/decline or execution — DeltaIntegration's lookups resolve to None, so the marker always falls back. The claim/decline layer and native exec land in parts 4a/4b.
• No CDF — the CometExecRule CDF hook and DeltaIntegration's CDF members are held back to part 5.

Why it is safe on default builds

Without -Pcontrib-delta: the cargo feature is off (no comet-contrib-delta, no delta_kernel in the tree), the Maven profile is inactive (no io.delta:*, no contrib/delta sources compiled), and the native DeltaScan arm is #[cfg]-stubbed to an error that is never reached because nothing emits the proto message. DeltaIntegration is the only always-present class and every one of its reflective lookups returns None. The gate script asserts all of this mechanically: default libcomet has 0 Delta symbols and is the same size as before.

The CometScanRule change reorders the metadata-column guard so V1 scans reach transformV1Scan (which delegates to any V1 contrib) before the generic metadata-column rejection — but for a non-contrib V1 scan the guard is re-applied inside transformV1Scan, so vanilla behavior is unchanged.

Verification

Run against this branch rebased on current main:

• Default native build + gated native build (--features contrib-delta): green.
• cargo clippy both feature states: clean.
• dev/verify-contrib-delta-gate.sh: all checks pass (0 Delta symbols in the default dylib).
• Gated JVM compile (spark-3.5 / Scala 2.13, -Pcontrib-delta) and default JVM compile (spark-3.4 / Scala 2.12): both compile.
• spotless + scalastyle: clean.

Roadmap

Parts still to come: Rust driver-side planning (3a), Rust executor-side read path (3b), Scala claim/decline (4a), Scala execution — end-to-end native reads (4b), Change Data Feed (5), test battery + regression harness (6), and docs (7). Each is gated behind -Pcontrib-delta, so every intermediate state on main is safe for default builds. Tracking umbrella: #4366; part 1: #4700.

---

🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.

[schenksj]

@parthchandra @andygrove this is part 2 of the Delta contrib split — the build gate + inert wiring — following on from part 1 (#4700, now merged). Thank you both for the reviews on part 1! 🙏

Sorry it took a while to get this next phase out there — I was heads-down getting https://github.com/capitalone/vulnhunter to market. Back on the Delta series now.

This part is deliberately inert/gated (zero Delta surface on default builds), so it should be a fairly self-contained review of the wire format, the build machinery, and the gate-enforcement script. Would appreciate your eyes when you have a chance.