In bigdata scenario，need to write vortex file on HDFS\JFS etc. filesystem

[blackflash997997]

As mentioned above, will you develop a feature that can write to these file systems in the future?

[danking]

@blackflash997997 Do you have an example of code that doesn't work as you expect? In Rust, Vortex's read (respectively writing) API accepts anything that implements the VortexReadAt (respectively VortexWrite) trait.

The Python API is currently a bit limited. We can read from GCS, S3, ABS, and local files, but we can only write to local files. We could easily support GCS, S3, and ABS. Supporting HDFS requires the Python package to ship with the hdfs-native-object-store Rust crate. That might be a bigger lift due to the complexity of optionally shipping extra functionality in the Python package.

In the Java API (including Spark), I think we'd need a feature-flag that enables HDFS object store support. Then you could build a JAR with that flag enabled. There is some work involved in implementing this.