JSON index with inner ZONEMAP / FmIndex fails: "Expected option to have value"

[ztorchan]

Description

Creating a scalar index with IndexConfig(index_type="json", ...) fails when
target_index_type is "zonemap" (or "fmindex"). It works fine when the
inner type is "btree" or "label_list".

Error

RuntimeError: Encountered internal error. Expected option to have value

Root cause

JsonIndexPlugin::convert_stream_by_type always outputs batches with ROW_ID,
but ZONEMAP (and FmIndex) trainers expect ROW_ADDR.

What convert_stream_by_type produces

[lance-index/src/scalar/json.rs]

let new_schema = Arc::new(Schema::new(vec![
    ArrowField::new(VALUE_COLUMN_NAME, target_type.clone(), true),
    ArrowField::new(ROW_ID, DataType::UInt64, false),   // ← hard-coded ROW_ID
]));

What ZONEMAP expects

[lance-index/src/scalar/zoned.rs]

let row_addr_col = batch
    .column_by_name(ROW_ADDR)     // ← looks for ROW_ADDR, not ROW_ID
    .unwrap()
    ...
let fragment_id = row_addr >> 32; // needs fragment_id from 64-bit row_addr

ZONEMAP needs ROW_ADDR (upper 32 bits = fragment_id, lower 32 bits =
row_offset) so it can enforce that zones never span fragment boundaries.
ROW_ID doesn't carry fragment information.

Why BTREE / LabelList work

Inner index	Column used	TrainingCriteria pattern	Works?
BTREE	ROW_ID	TrainingOrdering::Values + .with_row_id()	✅
LabelList	ROW_ID	Values + .with_row_id()	✅
ZONEMAP	ROW_ADDR	Addresses + .with_row_addr()	❌
FmIndex	ROW_ADDR	Addresses	❌

The JSON plugin hard-codes ROW_ID in both:

• extract_json_with_type_info (line ~408, 425-426)
• convert_stream_by_type (line ~648-651)

which only matches the BTREE/LabelList contract.

Suggested fix

convert_stream_by_type should check the inner plugin's TrainingCriteria
and output the appropriate row-address column:

1. Minimal: query the inner target plugin's training criteria; output
   ROW_ADDR when with_row_addr() is set, ROW_ID otherwise.
2. Better: have the JSON plugin pass through whatever row-address column
   already exists in the input stream (which load_training_data tailors to
   each plugin's criteria), rather than hard-coding ROW_ID.

Workaround

Use a different inner index type (e.g. "btree") inside the JSON index, or
avoid wrapping ZONEMAP inside a JSON index.

Steps to reproduce

import lance
from lance.indices import IndexConfig

ds = lance.dataset("/path/to/dataset")
ds.create_scalar_index(
    "my_json_column",
    index_type=IndexConfig(
        index_type="json",
        parameters={
            "path": "some_key",
            "target_index_type": "zonemap",
            "target_index_parameters": "{}",
        },
    ),
    replace=True,
)

Expected behavior

No response

Lance version

v8.0.0

Language binding

Rust

Environment

No response

Logs / traceback

[westonpace]

This may be inadvertently solved by #8085