Describe the bug
The comparison kernels in arrow_ord::cmp (eq, lt, distinct, ...) panic with internal error: entered unreachable code when given a Dictionary array whose values are themselves a Dictionary (or a RunEndEncoded). They return Err(InvalidArgumentError) for other unsupported inputs, such as nested types or mismatched types, so a panic here is inconsistent with the rest of the kernel and takes down the caller's thread.
compare_op unwraps one run-end-encoded layer and one dictionary layer, then checks is_nested() and type equality on what is left. DataType::is_nested returns false for Dictionary(_, Utf8), and both sides have the same type, so the checks pass and downcast_primitive_array! falls into its _ => unreachable!() arm (arrow-ord/src/cmp.rs:290 in 59.3.0, line 314 on main).
main already has the right predicate, supports_distinct (added in #9642), but only partition.rs consults it; compare_op itself still panics.
To Reproduce
use std::sync::Arc;
use arrow_array::{Array, ArrayRef, DictionaryArray, Int32Array, StringArray, UInt32Array};
use arrow_ord::cmp::eq;
fn main() {
// Dictionary<Int32, Dictionary<UInt32, Utf8>>
let inner: ArrayRef = Arc::new(DictionaryArray::new(
UInt32Array::from(vec![0, 1]),
Arc::new(StringArray::from(vec!["a", "b"])),
));
let outer = DictionaryArray::new(Int32Array::from(vec![0, 1]), inner);
println!("type: {}", outer.data_type());
match eq(&outer, &outer) {
Ok(a) => println!("ok: {a:?}"),
Err(e) => println!("err: {e}"),
}
}
Output with arrow-array/arrow-ord/arrow-schema =59.3.0:
type: Dictionary(Int32, Dictionary(UInt32, Utf8))
thread 'main' panicked at .../arrow-ord-59.3.0/src/cmp.rs:290:18:
internal error: entered unreachable code
Found through DataFusion, where ScalarValue::partial_cmp on a List<Dictionary<Dictionary<Utf8>>> reaches this kernel (apache/datafusion#24916).
Expected behavior
Err(ArrowError::InvalidArgumentError(..)), in line with the existing Nested comparison: ... (hint: use make_comparator instead) error, or support for the nested encoding.
Additional context
I am happy to send a PR that returns an error from compare_op when the unwrapped type is still a Dictionary or RunEndEncoded, reusing supports_distinct.
Describe the bug
The comparison kernels in
arrow_ord::cmp(eq,lt,distinct, ...) panic withinternal error: entered unreachable codewhen given aDictionaryarray whose values are themselves aDictionary(or aRunEndEncoded). They returnErr(InvalidArgumentError)for other unsupported inputs, such as nested types or mismatched types, so a panic here is inconsistent with the rest of the kernel and takes down the caller's thread.compare_opunwraps one run-end-encoded layer and one dictionary layer, then checksis_nested()and type equality on what is left.DataType::is_nestedreturnsfalseforDictionary(_, Utf8), and both sides have the same type, so the checks pass anddowncast_primitive_array!falls into its_ => unreachable!()arm (arrow-ord/src/cmp.rs:290in 59.3.0, line 314 onmain).mainalready has the right predicate,supports_distinct(added in #9642), but onlypartition.rsconsults it;compare_opitself still panics.To Reproduce
Output with
arrow-array/arrow-ord/arrow-schema=59.3.0:Found through DataFusion, where
ScalarValue::partial_cmpon aList<Dictionary<Dictionary<Utf8>>>reaches this kernel (apache/datafusion#24916).Expected behavior
Err(ArrowError::InvalidArgumentError(..)), in line with the existingNested comparison: ... (hint: use make_comparator instead)error, or support for the nested encoding.Additional context
I am happy to send a PR that returns an error from
compare_opwhen the unwrapped type is still aDictionaryorRunEndEncoded, reusingsupports_distinct.