Skip to content

Update arrow_datafusion (major)#1330

Open
dashql-renovate[bot] wants to merge 1 commit intomainfrom
renovate/major-arrow_datafusion
Open

Update arrow_datafusion (major)#1330
dashql-renovate[bot] wants to merge 1 commit intomainfrom
renovate/major-arrow_datafusion

Conversation

@dashql-renovate
Copy link
Contributor

@dashql-renovate dashql-renovate bot commented Jul 27, 2025 •
edited
Loading

This PR contains the following updates:

Package Type Update Change
arrow dependencies major 55.2.0 -> 57.2.0
datafusion (source) dependencies major 48.0.1 -> 52.1.0
datafusion-common (source) dependencies major 48.0.1 -> 52.1.0
datafusion-datasource (source) dependencies major 48.0.1 -> 52.1.0
datafusion-execution (source) dependencies major 48.0.1 -> 52.1.0
datafusion-expr (source) dependencies major 48.0.1 -> 52.1.0
datafusion-functions (source) dependencies major 48.0.1 -> 52.1.0
datafusion-functions-aggregate (source) dependencies major 48.0.1 -> 52.1.0
datafusion-functions-nested (source) dependencies major 48.0.1 -> 52.1.0
datafusion-functions-window (source) dependencies major 48.0.1 -> 52.1.0
datafusion-optimizer (source) dependencies major 48.0.1 -> 52.1.0
datafusion-physical-expr (source) dependencies major 48.0.1 -> 52.1.0
datafusion-physical-plan (source) dependencies major 48.0.1 -> 52.1.0

Release Notes

apache/arrow-rs (arrow)

v57.2.0

Compare Source

Full Changelog

Breaking changes:

Fixed bugs:

v57.1.0: arrow 57.1.0

Compare Source

Changelog

57.1.0 (2025-11-20)

Full Changelog

Implemented enhancements:

  • Eliminate bound checks in filter kernels #​8865 [arrow]
  • Respect page index policy option for ParquetObjectReader when it's not skip #​8856 [parquet]
  • Speed up collect_bool and remove unsafe #​8848 [arrow]
  • Error reading parquet FileMetaData with empty lists encoded as element-type=0 #​8826 [parquet]
  • ValueStatistics methods can't be used from generic context in external crate #​8823 [parquet]
  • Custom Pretty-Printing Implementation for Column when Formatting Record Batches #​8821 [arrow]
  • Parquet-concat: supports bloom filter and page index #​8804 [parquet]
  • [Parquet] virtual row group number support #​8800
  • [Variant] Enforce shredded-type validation in shred_variant #​8795 [arrow]
  • Simplify decision logic to call FilterBuilder::optimize or not #​8781 [arrow]
  • [Variant] Add variant to arrow for DataType::{Binary, LargeBinary, BinaryView} #​8767 [arrow]
  • Provide algorithm that allows zipping arrays whose values are not prealigned #​8752 [arrow]
  • [Parquet] ParquetMetadataReader decodes too much metadata under point-get scenerio #​8751 [parquet]
  • arrow-json supports encoding binary arrays, but not decoding #​8736 [arrow]
  • Allow FilterPredicate instances to be reused for RecordBatches #​8692 [arrow]
  • ArrowJsonBatch::from_batch is incomplete #​8684 [arrow]
  • parquet-layout: More info about layout including footer size, page index, bloom filter? #​8682 [parquet]
  • Rewrite ParquetRecordBatchStream (async API) in terms of the PushDecoder #​8677 [parquet]
  • [JSON] Add encoding for binary view #​8674 [arrow]
  • Refactor arrow-cast decimal casting to unify the rescale logic used in Parquet variant casts #​8670 [arrow]
  • [Variant] Support Uuid/FixedSizeBinary(16) shredding #​8665
  • [Parquet]There should be an encoding counter to know how many encodings the repo supports in total #​8662 [parquet]
  • Improve parse_data_type for List, ListView, LargeList, LargeListView, FixedSizeList, Union, Map, RunEndCoded. #​8648 [arrow]
  • [Variant] Support variant to arrow primitive support null/time/decimal_* #​8637
  • Return error from RleDecoder::reset rather than panic #​8632 [parquet]
  • Add bitwise ops on BooleanBufferBuilder and MutableBuffer that mutate directly the buffer #​8618 [arrow]
  • [Variant] Add variant_to_arrow Utf-8, LargeUtf8, Utf8View types support #​8567 [arrow]

Fixed bugs:

  • Regression: Parsing List(Int64) results in nullable list in 57.0.0 and a non-nullable list in 57.1.0 #​8883
  • Regression: FixedSlizeList data type parsing fails on 57.1.0 #​8880
  • (dyn ArrayFormatterFactory + 'static) can't be safely shared between threads #​8875
  • RowNumber reader has wrong row group ordering #​8864 [parquet]
  • ThriftMetadataWriter::write_column_indexes cannot handle a ColumnIndexMetaData::NONE #​8815 [parquet]
  • "Archery test With other arrows" Integration test failing on main: #​8813 [arrow]
  • [Parquet] Writing in 57.0.0 seems 10% slower than 56.0.0 #​8783 [parquet]
  • Parquet reader cannot handle files with unknown logical types #​8776 [parquet]
  • zip now treats nulls as false in provided mask regardless of the underlying bit value #​8721 [arrow]
  • [avro] Incorrect version in crate.io landing page #​8691 [arrow]
  • Array: ViewType gc() has bug when array sum length exceed i32::MAX #​8681 [arrow]
  • Parquet 56: encounter error: item_reader def levels are None when reading nested field with row filter #​8657 [parquet]
  • Degnerate and non-nullable FixedSizeListArrays are not handled #​8623 [arrow]
  • [Parquet]Performance Degradation with RowFilter on Unsorted Columns due to Fragmented ReadPlan #​8565 [parquet]

Documentation updates:

Performance improvements:

Closed issues:

  • Variant to NullType conversion ignores strict casting #​8810
  • Unify display representation for Field #​8784
  • Misleading configuration name: skip_arrow_metadata #​8780
  • Inconsistent display for types with Metadata #​8761 [arrow]
  • Internal arrow-integration-test crate is linked from arrow docs #​8739 [arrow]
  • Add benchmark for RunEndEncoded casting #​8709 [arrow]
  • [Varaint] Support VariantArray::value to return a Result<Variant> #​8672

Merged pull requests:

* This Changelog was automatically generated by github_changelog_generator

v57.0.0: arrow 57.0.0

Compare Source

Changelog

57.0.0 (2025-10-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Added Avro support (new arrow-avro crate) #​4886
  • parquet-rewrite: supports compression level and write batch size #​8639
  • Error not panic when int96 stastistics aren't size 12 #​8614 [parquet]
  • [Variant] Make VariantArray iterable #​8612
  • [Variant] impl PartialEq for VariantArray #​8610
  • [Variant] Remove potential panics when probing VariantArray #​8609
  • [Variant] Remove ceremony of going from list of Variant to VariantArray #​8606
  • Eliminate redundant validation in RecordBatch::project #​8591 [arrow]
  • [PARQUET][BENCH] Arrow writer bench with compression and/or page v2 #​8559 [parquet]
  • [Variant] casting functions are confusingly named #​8531 [parquet]
  • Support writing GeospatialStatistics in Parquet writer #​8523 [parquet]
  • [thrift-remodel] Optimize convert_row_groups #​8517 [parquet]
  • [Variant] Add variant to arrow primitive support for boolean/timestamp/time #​8515
  • Test thrift-remodel branch with DataFusion #​8513 [parquet]
  • Make UnionArray::is_dense Method Public #​8503 [arrow]
  • Add append_n method to FixedSizeBinaryDictionaryBuilder #​8497 [arrow]
  • [Parquet] Reduce size of ParquetMetadata when encryption feature is enabled #​8469 [parquet]
  • [Parquet] Remove useless mut requirements in geting bloom filter function #​8461 [parquet]
  • Change serde dependency to serde_core where applicable #​8451 [arrow]
  • [Parquet] Split ParquetMetadataReader into IO/decoder state machine and thrift parsing #​8439 [parquet]
  • Remove compiler warning for redundant config enablement #​8412 [arrow]
  • Add geospatial statistics creation support for GEOMETRY/GEOGRAPHY Parquet logical types #​8411 [arrow]
  • arrow_json lacks with_timestamp_format functions like arrow_csv had offered #​8398 [arrow]
  • Unify API for writing column chunks / row groups in parallel #​8389 [parquet]
  • Reuse zstd context in arrow IPC writer #​8386 [arrow] [arrow-flight]
  • [Variant] Support reading/writing Parquet Variant LogicalType #​8370 [parquet]
  • [Variant] Implement a shred_variant function #​8361
  • [Parquet] Expose ReadPlan and ReadPlanBuilder #​8347 [parquet]
  • [Variant] [Shredding] Support typed_access for List #​8337 [parquet]
  • [Variant] [Shredding] Support typed_access for Struct #​8336 [parquet]
  • [Variant] [Shredding] Support typed_access for Time64(Microsecond) #​8334 [parquet]
  • [Variant] [Shredding] Support typed_access for Decimal128 #​8332 [parquet]
  • [Variant] [Shredding] Support typed_access for Timestamp(Microsecond, _) and Timestamp(Nanosecond, _) #​8331 [parquet]
  • [Variant] [Shredding] Support typed_access for Date32 #​8330 [parquet]
  • [Variant] Support strict casting for all data types #​8303
  • [Variant] Support typed access for string types in variant_get #​8285
  • [Variant]: Implement DataType::FixedSizeList support for cast_to_variant kernel #​8281

Fixed bugs:

  • Fix arrow-avro Writer Documentation related to AvroBinaryFormat #​8631 [arrow]
  • Decimal -> Decimal cast wrongly fails for large scale reduction #​8579 [arrow]
  • [Parquet] Avoid fetching multiple pages when max_predicate_cache_sizeis 0 #​8542 [parquet]
  • DataType parsing no longer works correctly for old formatted timestamps #​8539 [parquet] [arrow]
  • [Parquet] ArrowWriter flush does not work #​8534 [parquet]
  • arrow::compute::interleave fails with struct arrays with no fields #​8533 [arrow]
  • [Parquet] Over memory consumation for writer page v1 compressed #​8526 [parquet]
  • Incorrect Behavior of Collecting a filtered iterator to a BooleanArray #​8505 [arrow]
  • [Parquet] ProjectionMask::columns name handling is bug prone #​8443 [parquet]
  • [Variant] Shredded typed_value columns must have valid variant types #​8435 [parquet]
  • cargo test -p parquet fails with default ulimit #​8406 [parquet]
  • Column with List(Struct) causes failed to decode level data for struct array #​8404 [parquet]
  • Binaryview Utf8 Cast Issue #​8403 [arrow]
  • Decimal precision validation displays value without accounting for scale #​8382 [arrow]
  • [Variant] VariantArray::data_type returns StructType, causing Array::as_struct to panic #​8319 [parquet]
  • [Variant] writing a VariantArray to parquet panics #​8296 [parquet]

Documentation updates:

Performance improvements:

Closed issues:

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

  • If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

@dashql-renovate dashql-renovate bot added dependencies Pull requests that update a dependency file rust labels Jul 27, 2025
@dashql-renovate dashql-renovate bot enabled auto-merge (squash) July 27, 2025 05:08
@dashql-renovate dashql-renovate bot added the rust label Jul 27, 2025
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 6 times, most recently from 69e2744 to 6227e91 Compare August 2, 2025 05:06
@dashql-renovate dashql-renovate bot changed the title Update arrow_datafusion to v49 (major) Update arrow_datafusion (major) Aug 2, 2025
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 6 times, most recently from dc2d071 to bad2c41 Compare August 10, 2025 05:06
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 10 times, most recently from 5c93824 to b9aaf45 Compare August 20, 2025 05:04
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 2 times, most recently from ba2a299 to 9c27c8d Compare August 22, 2025 05:04
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 7 times, most recently from 657f5d5 to 76823aa Compare November 11, 2025 05:04
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 3 times, most recently from faebdda to 5a5ad25 Compare November 20, 2025 05:03
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch from 5a5ad25 to 7a8fa33 Compare November 25, 2025 05:03
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 11 times, most recently from 11901fd to 41ebf7c Compare January 17, 2026 05:05
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 3 times, most recently from a699180 to 57afb36 Compare January 27, 2026 05:09
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch 3 times, most recently from de75465 to c9ca812 Compare February 4, 2026 05:20
@dashql-renovate dashql-renovate bot force-pushed the renovate/major-arrow_datafusion branch from c9ca812 to 3ef7f8a Compare February 5, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

dependencies Pull requests that update a dependency file rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

0 participants