(feat) Additional Arrow support

[Fred-Wu]

Additional Arrow support

With the current on demand data loading, I think Data Viewer could extend existing Arrow table support to lazy Arrow-backed data sources:

• Dataset and its subclasses, including FileSystemDataset
• arrow_dplyr_query

This allows file-backed Arrow datasets and dplyr queries on Arrow data to be opened directly with View() without first collecting the full dataset into memory. Rows are fetched on demand as they are requested by the Data Viewer, allowing large Arrow datasets to be browsed efficiently.

This should only be a small change in the data viewer model.

[grantmcdermott]

Could we not implement via a more lightweight solution like nanoarrow? https://arrow.apache.org/nanoarrow/latest/index.html

[Fred-Wu]

Could we not implement via a more lightweight solution like nanoarrow? https://arrow.apache.org/nanoarrow/latest/index.html

It looks like nanoarrow is mainly designed for data streaming. It still relies on arrow for Dataset and arrow_dplyr_query. For our data viewer, we also need efficient forward and backward scrolling, filtering, sorting, and caching, so arrow seems better fitted here. Type conversion might also be an issue with nanoarrow.

For those files sit on disc rather than in memory, I think some trade-offs are expected. For example, loading data might be slower compared to in-memory data, especially after sorting.