v9.1.0-beta.2

What's Changed

New Features 🎉

• feat(compaction): compact fragments over an overlay-count limit by @wjones127 in #7772
• feat(fts): read inverted index params without opening the segment by @LuQQiu in #7816

Full Changelog: v9.1.0-beta.1...v9.1.0-beta.2

v9.1.0-beta.1

What's Changed

New Features 🎉

• feat(fts): record scorer_build_ms for the exec-local BM25 scorer fallback by @LuQQiu in #7807

Bug Fixes 🐛

• fix: bracket match query display by @jackye1995 in #7796

Full Changelog: release-root/9.1.0-beta.N...v9.1.0-beta.1

v9.0.0-rc.1

What's Changed

Breaking Changes 🛠

• refactor!: rename FMIndexIndexDetails to FMIndexDetails by @westonpace in #7397
• feat(fts)!: add configurable posting block size by @BubbleCal in #7466
• feat(fts)!: make v2 the default index format by @BubbleCal in #7512

New Features 🎉

• feat(namespace-dir): add alter table column operations (add/alter/drop) by @XuQianJin-Stars in #6273
• feat: compile-time feature-gated Rust backtrace for JNI error diagnostics by @summaryzb in #6365
• feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds by @tobocop2 in #6630
• feat(namespace-dir): implement update_table and delete_from_table by @XuQianJin-Stars in #6923
• feat(namespace-dir): implement alter_transaction for DirectoryNamespace by @XuQianJin-Stars in #6974
• feat(mem_wal): support prefiltered LSM vector and FTS search by @touch-of-grey in #7138
• feat(index): support distributed LabelList scalar index builds by @jackye1995 in #7223
• feat(compaction): introduce RowAddrRemap structure to avoid remap OOM caused by HashMap by @zhangyue19921010 in #7237
• feat: support cleanup explain for python and java by @yanghua in #7248
• feat(schema_evolution): allow Dict <-> value-type casts in alter_columns by @valkum in #7289
• feat: add SpillStore trait with local-disk implementation by @wjones127 in #7311
• feat(blob-v2): configure packed blob file max size with field metadata by @jo-migo in #7322
• feat(index): support Utf8View prefixes in SargableQueryParser by @wombatu-kun in #7351
• feat: support COUNT(*) pushdown on stable row id datasets by @wkalt in #7360
• feat(mem-wal): add ShardWriter::abort + Sealed manifest fence for drop-table by @hamersaw in #7361
• feat(mem-wal): add ShardWriter::put_no_wait by @hamersaw in #7362
• feat: add RLE v2 run length widths by @Xuanwo in #7376
• feat: support hamming clustering by @brendanclement in #7379
• feat(file): v2 writer/reader support columns of unequal length by @wjones127 in #7406
• feat(mem_wal): row-level delete via tombstone rows by @hamersaw in #7417
• feat(python): add list_with_delimiter, delete_file, and read_range to LanceFileSession by @jmhsieh in #7426
• feat(python): expose fragment-reuse remap and delete-by-offset by @wkalt in #7438
• feat(index): read a column's min/max from ZoneMap without a scan by @Ali2Arslan in #7463
• feat: add ICU split tokenizer variant by @Xuanwo in #7474
• feat(mem_wal): tombstone-preserving point lookup by @hamersaw in #7482
• feat(mem_wal): non-blocking ShardWriter::delete_no_wait by @hamersaw in #7483
• feat(bitpacking): add owned bitpacking codecs by @BubbleCal in #7496
• feat(io): publish object store metrics via the metrics crate by @wjones127 in #7533
• feat: data overlay files — model, feature flag, and write/commit path by @wjones127 in #7535
• feat: resolve data overlay files on the take (and scan) read path by @wjones127 in #7536
• feat(python): expose Lance metrics via OpenTelemetry by @wjones127 in #7537
• feat: add blob write preparation API by @Xuanwo in #7558
• feat: support update with blob encoding column by @nyl3532016 in #7579
• feat(fts): impact skip data for posting lists by @BubbleCal in #7602
• feat: support per-base storage options with base_. prefix by @jackye1995 in #7608
• feat: support multi-base tables in merge insert with target base routing by @jackye1995 in #7610
• feat(python): expose MemWAL shard delete by @u70b3 in #7649
• feat: compress RLE child buffers by @Xuanwo in #7663
• feat: support list blob scans by @Xuanwo in #7664
• feat: support segment selection in pylance prewarm by @jackye1995 in #7677
• feat: support nested field FTS by @Xuanwo in #7686
• feat(java): expose MemWAL shard delete by @hamersaw in #7688
• feat(java): forward table properties to declareTable on namespace create by @puchengy in #7711
• feat(mem_wal): thread store_params + session through WAL write/read paths by @hamersaw in #7735
• feat(python): add bulk packed blob writer API by @justinrmiller in #7743
• feat(credentials): vend AWS credentials via AssumeRoleWithWebIdentity to avoid role chaining by @rpgreen in #7757
• feat: support multi-segment indices in hamming clustering by @jackye1995 in #7758
• feat: support wider hamming hashes by @jackye1995 in #7767

Bug Fixes 🐛

• fix(python): fix tuple bug in _coerce_query_vector condition check by @hushengquan in #6555
• fix: map object_store::Error::NotFound to Error::NotFound instead of Error::IO by @LuciferYang in #6569
• fix: shift offsets after trimming values in merge_with_schema by @kaan-simbe in #6581
• fix(transaction): reduce redundant code for new_version calculations by @yangshangqing95 in #6589
• fix(index): keep HNSW IVF scratch dir alive during partition writing by @geserdugarov in #6980
• fix: support manifests >5 GB via size-aware copy by @lixmgl in #7047
• fix: route JSON index queries to the correct sub-parser by path by @ztorchan in #7072
• fix(index): cap IVF train prefetch memory for nullable columns by @geserdugarov in #7143
• fix: evaluate all list-element docs in FTS prefilter walk-the-allowlist branch by @Ar-maan05 in #7246
• fix: apply per-segment filters and frag-reuse remap in BTree segment merge by @zhangyue19921010 in #7320
• fix(index): drop stale scalar index entries after stable-row-id update by @wkalt in #7359
• fix: reject DataReplacement racing concurrent Update/Delete/Merge by @wkalt in #7373
• fix(ci): replace deprecated array.shape assignment for NumPy 2.5 by @zhangyue19921010 in #7384
• fix(fts): enforce required terms for and queries by @BubbleCal in #7385
• fix(index): use range block max for fts conjunction by @BubbleCal in #7387
• fix(merge_insert): keep nested-field index correct on stable row id update by @jackye1995 in #7410
• fix(update): keep nested-field index correct when updating a struct column by @jackye1995 in #7412
• fix(io): clamp ObjectStore::io_parallelism() to at least 1 by @LuciferYang in #7414
• fix(namespace): allow create_branch bootstrap on managed tables by @geruh in #7415
• fix(fts): use async send in FTS index builder to prevent thread-pool … by @a-agmon in #7423
• fix(rowids): union overlapping segments in RowAddrTreeMap::from by @wkalt in #7429
• fix: relax DataReplacement conflict with concurrent Delete/Update by @brendanclement in #7433
• fix(python): validate delete_rows offsets and add missing type stubs by @wkalt in #7448
• fix(index): give each FTS partition a distinct scheduler base priority by @LuQQiu in #7449
• fix(encoding): honor zstd compression scheme and level on large per-value buffers by @claydugo in #7460
• fix(index): record IVF_HNSW index file sizes after writing footer by @Ali2Arslan in https://github.com/lance-form...

v9.0.0-beta.24

What's Changed

New Features 🎉

• feat(mem_wal): thread store_params + session through WAL write/read paths by @hamersaw in #7735
• feat(python): add bulk packed blob writer API by @justinrmiller in #7743

Full Changelog: v9.0.0-beta.23...v9.0.0-beta.24

v9.0.0-beta.23

What's Changed

Bug Fixes 🐛

• fix: reconstruct protobuf schema fields in linear time by @Xuanwo in #7766

Full Changelog: v9.0.0-beta.22...v9.0.0-beta.23

v9.0.0-beta.22

What's Changed

Breaking Changes 🛠

• feat(fts)!: add configurable posting block size by @BubbleCal in #7466

New Features 🎉

• feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds by @tobocop2 in #6630
• feat: data overlay files — model, feature flag, and write/commit path by @wjones127 in #7535
• feat(fts): impact skip data for posting lists by @BubbleCal in #7602
• feat: support list blob scans by @Xuanwo in #7664
• feat(credentials): vend AWS credentials via AssumeRoleWithWebIdentity to avoid role chaining by @rpgreen in #7757
• feat: support multi-segment indices in hamming clustering by @jackye1995 in #7758
• feat: support wider hamming hashes by @jackye1995 in #7767

Bug Fixes 🐛

• fix(index): keep HNSW IVF scratch dir alive during partition writing by @geserdugarov in #6980
• fix(encoding): honor zstd compression scheme and level on large per-value buffers by @claydugo in #7460
• fix(python): preserve PQ num_bits in model training by @ddupg in #7583
• fix(scanner): honor batch_readahead to bound v2 scan decode concurrency by @zhangyue19921010 in #7632
• fix: preserve case in generated field path expressions by @skyshineb in #7698
• fix(encoding): decode empty inline-bitpacked blocks by @Xuanwo in #7752
• fix: train IVF indexes on fragment subsets by @jackye1995 in #7768
• perf: skip fragment-reuse index for compactions with no indexed data by @xuanyu-z in #7774

Documentation 📚

• docs(index): document public API of LogicalScalarIndex by @westonpace in #6804
• docs: specify data overlay files for the table format by @wjones127 in #7381
• docs: clarify Lance format stability by @Xuanwo in #7475

Performance Improvements 🚀

• perf(fts): bulk MAXSCORE search path for top-k disjunctions by @BubbleCal in #7603

Other Changes

• refactor(encoding): clarify mini-block repdef budget by @Xuanwo in #7751

Full Changelog: v9.0.0-beta.21...v9.0.0-beta.22

v9.0.0-beta.21

What's Changed

Bug Fixes 🐛

• fix(lance-core): avoid aliasing &mut Runtime in global_cpu_runtime by @LuciferYang in #7682
• fix: round-trip empty blob values in the 2.1+ structural encoding by @wkalt in #7717

Performance Improvements 🚀

• perf(index): pack prewarmed FTS posting groups by @BubbleCal in #7720

Full Changelog: v9.0.0-beta.20...v9.0.0-beta.21

v9.0.0-beta.20

What's Changed

New Features 🎉

• feat: compress RLE child buffers by @Xuanwo in #7663
• feat(java): expose MemWAL shard delete by @hamersaw in #7688
• feat(java): forward table properties to declareTable on namespace create by @puchengy in #7711

Bug Fixes 🐛

• fix(transaction): reduce redundant code for new_version calculations by @yangshangqing95 in #6589
• fix: late-materialize blob columns read as binary by @wkalt in #7593
• fix(lance-io): drop reservation when future is cancelled by @valkum in #7638
• fix: return error instead of panicking on values too wide for miniblock by @zhangyue19921010 in #7650
• fix(index): batch IVF streaming partition search off the CPU pool by @a-agmon in #7680
• fix(encoding): tolerate legacy RLE chunks that overflow the declared value count by @jackye1995 in #7708

Performance Improvements 🚀

• perf(dataset): reuse session-cached manifest on checkout by @zhangyue19921010 in #7661

Other Changes

• refactor(java): rename WriteDatasetBuilder.tableProperties to properties by @puchengy in #7715

Full Changelog: v9.0.0-beta.19...v9.0.0-beta.20

v9.0.0-beta.19

What's Changed

New Features 🎉

• feat: compile-time feature-gated Rust backtrace for JNI error diagnostics by @summaryzb in #6365
• feat(python): expose Lance metrics via OpenTelemetry by @wjones127 in #7537
• feat(python): expose MemWAL shard delete by @u70b3 in #7649
• feat: support nested field FTS by @Xuanwo in #7686

Bug Fixes 🐛

• fix(index): cap IVF train prefetch memory for nullable columns by @geserdugarov in #7143
• fix(index): translate address-domain scalar index results to row ids under stable row ids by @charleshuang119 in #7565
• fix(fts): use complete English stop-word list for ICU tokenizer by @LuQQiu in #7621
• fix: relax test_query_delta_indices assertion for approximate index by @XuQianJin-Stars in #7622
• fix(index): describe bitmap-less system indices in describe_indices by @hamersaw in #7667
• fix: avoid creating directory namespace manifest on read by @Xuanwo in #7687
• fix(python): ignore PyTorch 3.14 script_method warning by @Xuanwo in #7689

Performance Improvements 🚀

• perf(index): add exact null-row bitmap to zone map and bloom filter by @westonpace in #7372
• perf: increase default memory pool to 150MB with 40MB sort spill reservation by @westonpace in #7675

Other Changes

• refactor(python): move tensorflow integration out of tree by @Xuanwo in #7685

Full Changelog: v9.0.0-beta.18...v9.0.0-beta.19

v9.0.0-beta.18

What's Changed

New Features 🎉

• feat(io): publish object store metrics via the metrics crate by @wjones127 in #7533
• feat: support segment selection in pylance prewarm by @jackye1995 in #7677

Bug Fixes 🐛

• fix(python): fix tuple bug in _coerce_query_vector condition check by @hushengquan in #6555
• fix: map object_store::Error::NotFound to Error::NotFound instead of Error::IO by @LuciferYang in #6569
• fix: shift offsets after trimming values in merge_with_schema by @kaan-simbe in #6581
• fix: convert Arrow JSON to Lance JSON in single-fragment create path by @xloya in #7469
• fix: convert Arrow JSON when updating columns via merge/update path by @xloya in #7472
• fix(fsst): correct decoder output-buffer size contract to 8x by @LuciferYang in #7589
• fix(fts): enforce fuzzy max_expansions globally across index partitions by @BubbleCal in #7634
• fix: build list FTS indexes as row documents by @Xuanwo in #7656
• fix: solve hang in train_streaming_coreset_ivf_model by @cmccabe in #7676

Documentation 📚

• docs: add cleanup and auto cleanup documentation by @westonpace in #6546
• docs: add fragment sizing guidance to performance guide by @hamersaw in #6606
• docs(core): document spawn_cpu limitations and audit call sites by @wjones127 in #7643
• docs: update MemWAL format spec by @jackye1995 in #7655

Performance Improvements 🚀

• perf(arrow): zero-copy BFloat16Array::from(Vec) via Buffer::from_vec by @LuciferYang in #7614

Full Changelog: v9.0.0-beta.17...v9.0.0-beta.18