Is your feature request related to a problem or challenge?
arrow_cast::base64::b64_encode sizes the output from every physical offset pair and calls encode_slice for every row, including rows already masked out by the null bitmap:
https://github.com/apache/arrow-rs/blob/master/arrow-cast/src/base64.rs
GenericByteArray::value documents that the physical bytes of a null slot are arbitrary but well-defined. Variable-width arrays may legally retain non-empty payloads in null slots. This is not only a hand-built case: filter on byte arrays explicitly copies null-slot bytes so the validity bitmap can be handled separately (filter_bytes in arrow-select).
The result is that encode:
- computes an encoded length for unobservable null payloads
- allocates those bytes in the output values buffer
- actually Base64-encodes them
b64_decode already skips decoding null rows, but it still sizes the output buffer from array.values().len() (the full physical buffer) and does not truncate the unused tail. At high null density with retained payloads, that is a large zeroed allocation that is never used.
Logical values are unchanged if null slots get a zero output length: physical bytes in null slots are not part of the array's logical content, and array equality already ignores them.
Describe the solution you'd like
- When the input has nulls, give null rows output length 0 and skip
encode_slice for them.
- Keep a no-null fast path so the common dense case does not pay a per-row validity check.
- For decode, size the buffer from valid input bytes (or truncate to the actual decoded length) so unused tail is not retained.
- Preserve the current generic
i32/i64 offset API.
- Add unit coverage for arrays that retain non-empty payloads in null slots, plus a Criterion bench.
I independently remeasured on arm64 macOS against 381eea177, 16,384 rows × 1,024-byte values, BASE64_STANDARD. Null slots in the "retained" case keep the full 1,024-byte payload. Logical encode output of a skip-null prototype matched the current kernel.
| Case |
Current |
Skip-null prototype |
Speedup |
| encode, 50% null retained |
4.081 ms |
2.012 ms |
2.03× |
| encode, 90% null retained |
3.930 ms |
434 µs |
9.07× |
| encode, 99% null retained |
3.888 ms |
76 µs |
51.2× |
| encode, no nulls |
4.001 ms |
3.935 ms |
~1.02× (no regression) |
| decode, 99% null retained |
188 µs |
82 µs |
2.31× |
| decode, no nulls |
4.292 ms |
4.277 ms |
unchanged |
On current encode, a 99% null array with compact (empty) null slots is already ~36× faster than the same logical array with retained payloads, which is the wasted work this change removes.
I can open a PR covering encode and decode together: same file, same retained-payload semantics, shared tests.
Describe alternatives you've considered
- Compacting null payloads in
filter/concat instead. That would help more kernels, but it is a much larger behavior change, and encode would still be wrong for FFI / manually constructed arrays.
- Only truncating the decode buffer and leaving encode alone. That misses the large CPU cost on encode.
- Always checking validity even with
null_count() == 0. A dedicated no-null path is cheaper and matches other kernels.
Additional context
Related but not the same: #10284 / #10324 made b64_encode reject invalid UTF-8 from a misbehaving Engine. No open issue or PR currently covers skipping null payloads.
There is currently no base64_encode bench in arrow-cast; a PR should add one, including a retained-null-payload case.
Is your feature request related to a problem or challenge?
arrow_cast::base64::b64_encodesizes the output from every physical offset pair and callsencode_slicefor every row, including rows already masked out by the null bitmap:https://github.com/apache/arrow-rs/blob/master/arrow-cast/src/base64.rs
GenericByteArray::valuedocuments that the physical bytes of a null slot are arbitrary but well-defined. Variable-width arrays may legally retain non-empty payloads in null slots. This is not only a hand-built case:filteron byte arrays explicitly copies null-slot bytes so the validity bitmap can be handled separately (filter_bytesinarrow-select).The result is that encode:
b64_decodealready skips decoding null rows, but it still sizes the output buffer fromarray.values().len()(the full physical buffer) and does not truncate the unused tail. At high null density with retained payloads, that is a large zeroed allocation that is never used.Logical values are unchanged if null slots get a zero output length: physical bytes in null slots are not part of the array's logical content, and array equality already ignores them.
Describe the solution you'd like
encode_slicefor them.i32/i64offset API.I independently remeasured on arm64 macOS against
381eea177, 16,384 rows × 1,024-byte values,BASE64_STANDARD. Null slots in the "retained" case keep the full 1,024-byte payload. Logical encode output of a skip-null prototype matched the current kernel.On current encode, a 99% null array with compact (empty) null slots is already ~36× faster than the same logical array with retained payloads, which is the wasted work this change removes.
I can open a PR covering encode and decode together: same file, same retained-payload semantics, shared tests.
Describe alternatives you've considered
filter/concatinstead. That would help more kernels, but it is a much larger behavior change, and encode would still be wrong for FFI / manually constructed arrays.null_count() == 0. A dedicated no-null path is cheaper and matches other kernels.Additional context
Related but not the same: #10284 / #10324 made
b64_encodereject invalid UTF-8 from a misbehavingEngine. No open issue or PR currently covers skipping null payloads.There is currently no
base64_encodebench inarrow-cast; a PR should add one, including a retained-null-payload case.