Minify Kernel

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Often people want to store arrow data in memory, and in some cases are potentially willing to pay a premium for reduced memory usage.

Currently various array types can have "bloated" memory footprints, for example:

• Arrays can be sliced, with potentially unreferenced data buffers and child arrays
• Dictionaries can contain duplicate entries
• View arrays can contain unreferenced data

Describe the solution you'd like

I would like to propose a minify kernel in arrow-select that allows performing this minification. It should take a non-exhaustive/build-pattern MinifyOptions struct to allow controlling how this logic is performed. Ideally it would be possible to move the logic that currently resides in the IPCWriter over to use this kernel.

Describe alternatives you've considered

Additional context

[alamb]

This came up as part of

• feat: add cleanup_non_empty_nulls kernel #9970

It also comes up regularly in DataFusion and we have various workarounds for it

[rluvaton]

Although I like the idea of minify kernel with builder pattern non-exhaustive options and have use cases that I would use that in my code right now, some options are not relevant for all array types.

for example, the cleanup non empty nulls, this is not relevant to Primitive/Struct/Boolean and possibly more

also in minify some people might want to cleanup all non empty nulls and reduce the unused space but this specific option as oppose to only clean top level is different that only match for some data types.

also, if minify is for reduce memory, in case the buffer is shared, calling minify will just increase memory as it will require data copy (which is still valuable)

[alamb]

also, if minify is for reduce memory, in case the buffer is shared, calling minify will just increase memory as it will require data copy (which is still valuable)

I think this is the case for almost all kernels (they make a new array from an existing one, temporarily increasing memory usage)

also in minify some people might want to cleanup all non empty nulls and reduce the unused space but this specific option as oppose to only clean top level is different that only match for some data types.

I was thinking minify would always create a new Array (with new allocations, etc) that consumes the minimum amount of new space (so for lists and other structured types it would avoid copying unused data for child arrays, as proposed in #9970).

[Jefffrey]

also in minify some people might want to cleanup all non empty nulls and reduce the unused space but this specific option as oppose to only clean top level is different that only match for some data types.

The way I read the original issue, ideally this behaviour would be composable via the options. e.g. could specify only want to zero size nulls but don't minimise sliced data, so only lists/strings would be affected.

(Though in implementation it could get a bit tricky)