Skip to content

cmp kernels panic with unreachable!() on nested dictionary arrays instead of returning an error #11094

Description

@edubraqd

Describe the bug

The comparison kernels in arrow_ord::cmp (eq, lt, distinct, ...) panic with internal error: entered unreachable code when given a Dictionary array whose values are themselves a Dictionary (or a RunEndEncoded). They return Err(InvalidArgumentError) for other unsupported inputs, such as nested types or mismatched types, so a panic here is inconsistent with the rest of the kernel and takes down the caller's thread.

compare_op unwraps one run-end-encoded layer and one dictionary layer, then checks is_nested() and type equality on what is left. DataType::is_nested returns false for Dictionary(_, Utf8), and both sides have the same type, so the checks pass and downcast_primitive_array! falls into its _ => unreachable!() arm (arrow-ord/src/cmp.rs:290 in 59.3.0, line 314 on main).

main already has the right predicate, supports_distinct (added in #9642), but only partition.rs consults it; compare_op itself still panics.

To Reproduce

use std::sync::Arc;

use arrow_array::{Array, ArrayRef, DictionaryArray, Int32Array, StringArray, UInt32Array};
use arrow_ord::cmp::eq;

fn main() {
    // Dictionary<Int32, Dictionary<UInt32, Utf8>>
    let inner: ArrayRef = Arc::new(DictionaryArray::new(
        UInt32Array::from(vec![0, 1]),
        Arc::new(StringArray::from(vec!["a", "b"])),
    ));
    let outer = DictionaryArray::new(Int32Array::from(vec![0, 1]), inner);
    println!("type: {}", outer.data_type());

    match eq(&outer, &outer) {
        Ok(a) => println!("ok: {a:?}"),
        Err(e) => println!("err: {e}"),
    }
}

Output with arrow-array/arrow-ord/arrow-schema =59.3.0:

type: Dictionary(Int32, Dictionary(UInt32, Utf8))

thread 'main' panicked at .../arrow-ord-59.3.0/src/cmp.rs:290:18:
internal error: entered unreachable code

Found through DataFusion, where ScalarValue::partial_cmp on a List<Dictionary<Dictionary<Utf8>>> reaches this kernel (apache/datafusion#24916).

Expected behavior

Err(ArrowError::InvalidArgumentError(..)), in line with the existing Nested comparison: ... (hint: use make_comparator instead) error, or support for the nested encoding.

Additional context

I am happy to send a PR that returns an error from compare_op when the unwrapped type is still a Dictionary or RunEndEncoded, reusing supports_distinct.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions