[WIP] feat(raster): view machinery for non-identity band views

[james-willis]

Summary

Adds the view-machinery layer on top of the N-D raster type. Bands can now be constructed and read with non-identity view entries — slices, broadcasts, axis permutations, and reverse-iteration — and the byte-access surface (data / contiguous_data / nd_buffer) handles the strided materialization transparently. The slice/manipulation functions in #750 depend on this layer.

What's in this PR

Single commit, five files. Three layers:

Schema / trait surface (rust/sedona-raster/src/traits.rs)

• validate_view() + 22 unit tests (length checks, axis range, negative step, broadcast / step=0, i64 overflow guards, etc.).
• visible_shape_from_view() helper.
• Both pub(crate) — validation runs implicitly inside start_band_with_view (writer) and RasterRef::band (reader), so external callers don't need to invoke them.

Builder (rust/sedona-raster/src/builder.rs)

• start_band_with_view() API.
• ~12 view-construction tests: slice, broadcast, transpose, negative-step, multi-band mixed views, Arrow IPC round-trip.

Reader (rust/sedona-raster/src/array.rs)

• RasterRefImpl::band composes view → byte strides + byte offset using checked arithmetic; malformed views and arithmetic overflows route through sedona_internal_datafusion_err! so they surface with the standard "SedonaDB internal error" framing.
• nd_buffer() returns the source buffer + shape/strides/offset (zero-allocation strided access).
• contiguous_data() delegates to nd_buffer() for the OutDb check + identity-view fast path, then walks the strided copy via a shared materialize_strided helper.
• BandRef::data() on BandRefImpl: identity views borrow from the Arrow column directly (zero-allocation); non-identity views lazy-materialize a row-major copy into an internal OnceCell on first access and reuse it on subsequent calls.
• Reader tests for explicit views (negative steps, OOB axis, length mismatch, malformed view rejection).

GDAL bridge (rust/sedona-raster-gdal/src/{gdal_common,gdal_dataset_provider}.rs)

• raster_ref_to_gdal_mem returns (Dataset, Vec<Vec<u8>>) so the Vec backing GDAL's pointer for view-materialized (Cow::Owned) bands outlives the dataset. RasterDataset carries the matching _owned_band_bytes field.

Behavior change vs main

• A band with a non-null view row now decodes via the composition path (previously rejected as a read error).
• BandRef::data() on a non-identity view returns the materialized row-major visible bytes via an internal OnceCell cache, rather than the raw source-column bytes. Repeat calls reuse the cache.

Test plan

• cargo build -p sedona-schema -p sedona-raster -p sedona-raster-gdal -p sedona-raster-functions -p sedona-testing
• cargo test -p sedona-raster — 113 tests pass.
• cargo test -p sedona-raster-gdal — 53 tests pass.
• cargo clippy --all-targets -- -D warnings
• cargo fmt --all --check

[paleolimbot]

This is very cool!

I think I got there by the end in the inline comments, but I think a struct ViewEntries with associated functions in src/view_entries.rs where the non serialize-to-arrow and deserialize-from-arrow logic behind this lives will make this clearer.

The main issue I have with the implementation here is that it relies on hidden Vec<u8>s that aren't reusable. To manage memory properly we are going to have to register some scratch spaces with the session's memory pool and reuse them between iterations of a loop. Hiding them within various structures for convenience is going to make it tricky to implement a correct pattern going forward.

[paleolimbot]

Suggested change

	/// with no allocation or copy. Non-identity views must materialize a
	/// row-major copy on first byte access.
	/// with no allocation or copy. Non-identity views must materialize a
	/// row-major copy on first byte access when using these functions
	/// (use `nd_buffer()` to guarantee that a copy is never produced).

[paleolimbot]

Should source_axis be a usize? I'm wary of using as here given that this is sensitive territory (will panic on out of bounds) although it is probably fine.

[paleolimbot]

These should probably be all i64s unless there is a compelling reason to use the unsigned version to minimize the number of times we have to as between signed and unsigned types.

[james-willis]

I cleaned out some of the u64s. These are the ones I left in the raster ecosystem:

1. BandRef::width() -> u64 / height() -> u64 (and the RasterMetadata shim equivalents) — 2D-compat accessors. Changing these is API-breaking across sedona-functions, sedona-raster-gdal, callers in tests, etc.
2. BandRef::raw_source_shape() -> &[u64] and dim_size() -> Option — same reason; they sit on the public trait.
3. source_shape, spatial_shape Arrow columns type → could be done as a follow up.

[paleolimbot]

Clippy has a good point here. Define struct StartBandWithViewArgs<'a> { ... } with all pub fields, which will force callers to write start_band_with_view(StartBandWithViewArgs { <named params> }).

You could also expose more than one function or have a BandBuilder of some kind to separate these things into more than one call.

[paleolimbot]

Here as well (you can probably use a common Args struct)

[paleolimbot]

A helper struct to manage Vec<ViewEntry> would make it more likely that other people will write reasonable tests. Or a macro (view_entries![0:4, 1:5]; use Python slice syntax since that's what you'd type in numpy to do this).

[paleolimbot]

A BandBuilder may help you quite a bit here since there are a lot of None that a default constructor could fill in

[paleolimbot]

These kinds of tests you probably don't want in this file. This file is about serializing stuff into Arrow and it's tricky to see what's going on...I'm not sure what the perfect abstraction is here but it should probably live in src/views.rs with tests and some pub functions that can be benchmarked.

Proabably struct ViewEntries { inner: Vec<ViewEntry> } with associated methods would make this a bit cleaner than the &[ViewEntry] + free functions.

[james-willis]

Somewhat offtopic but are you going to have the same concern with lazy outdb loading as well? (Outdb lazy loading + zarr loader is going to be higher on my priority list next week than this PR)

The main issue I have with the implementation here is that it relies on hidden Vecs that aren't reusable. To manage memory properly we are going to have to register some scratch spaces with the session's memory pool and reuse them between iterations of a loop. Hiding them within various structures for convenience is going to make it tricky to implement a correct pattern going forward.

[paleolimbot]

Possibly? I know it seems like a lot of comments but what I'm getting at is:

let mut scratch = Vec::new();
for raster in rasters {
  let data_slice = raster.band(0)?.contiguous_data(&mut scratch)?;
  do_stuff(data_slice);
}

Or on the day we need to track memory:

let mut max_capacity = 0;
for raster in rasters {
  max_capacity = max_capacity.max(raster.band(0)?.contiguous_data_alloc_size());
}

if max_capacity > config_options.raster.max_scratch_alloc {
  // error
}

// If this ends up being a lot, frequently, we may also want to use a MemoryReservation to track it
// Probably this helps contiguous_data be faster since in theory there is only one heap allocation.
// You could probably use some of the unsafe modifiers to also ensure that there are no bounds
// checks in your materializer slowing things down
let mut scratch = Vec::with_capacity(max_capacity);
for raster in rasters {
  let data_slice = raster.band(0)?.contiguous_data(&mut scratch)?;
  do_stuff(data_slice);
}

I'm not sure exactly how loading works yet but you might want something similar (with the hiccup that you need some number of async workers doing IO so you also need the same number of scratch buffers).

(I have the same comment for pretty much everything that heap allocates in a loop, which is why, for example, our geometry writers write WKB directly into the Arrow output instead of returning Vec<u8>).