Vortex Integrations

[gatesn]

A thread to track the current status and roadmap of Vortex integrations

• 🟢 DataFusion
  • Considered a first-class integration.
  • We will continue to integrate with additional DataFusion features as they become available.
• 🟢 DuckDB
  • Considered a first-class integration.
  • Vortex extension is available to install directly via DuckDB
• 🟠 Apache Arrow
  • Arrow import and export is a core part of Vortex. However we do not have full coverage of the Arrow specification, including map types, run-length encoding, and possibly others.
• 🟠 Apache Spark
  • Limited support for expression push-down.
• 🟠 Polars
  • Limited support for expression push-down.
• 🔴 cuDF
  • In prototype + design phase.
• 🔴 SiriusDB
  • Please open a discussion.
• 🔴 Velox
  • Please open a discussion.
• 🔴 Trino
  • Please open a discussion.

[arlyon]

I have been playing with Vortex and Polars to work with larger-than-memory data and decided to write an AnonymousScan impl for polars but was dismayed to see that next_batch and allows_slice_pushdown literally do nothing meaning that providing an offset to the scanner causes it to scan the entire dataset rather than use vortex' filter mechanisms.

In lieu of this I am looking at adding a new io backend to polars directly. When you mention limited support I suppose you mean at the python layer scanning the entire set into memory and converting it to an apache arrow df?

(I am working at the rust level)

Having lots of fun with this so for, so TY!

[vkingnew]

it's strongly desired support duckdb windows platform and apache spark, which these are widely used in Data engineering and data analyse.