add support for hf://buckets in httpfs#293
Open
cfahlgren1 wants to merge 2 commits into
Open
Conversation
cfahlgren1
commented
Mar 23, 2026
|
|
||
| # FIXME: currently this will not fail the Linux HTTPFS ci job if it fails, because it might do so due to networking issues | ||
| # however having a CI job dedicated to remote tests that may spuriously fail would solve this | ||
| # TODO: add hf://buckets integration coverage here once there is a maintainer-owned public bucket fixture |
Author
There was a problem hiding this comment.
A core contributor would need to create a small bucket fixture so we can add hf://buckets/... integration coverage in here, similar to the existing remote Hugging Face fixtures there under samansmink/duckdb_ci_tests for public datasets, private datasets, and spaces?
|
Yay ! let us know @carlopi if it looks good to you :) |
Author
|
@carlopi friendly bump if this makes sense to add 🤗 |
|
Very interested in this!!! |
|
Gentle ping on this one :) |
|
This sounds great! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hugging Face recently introduced Storage Buckets as mutable, non-versioned object storage on the Hub, addressable as
hf://buckets/<namespace>/<bucket>.This PR adds
hf://buckets/...support tohttpfsalongside the existinghf://datasets/...andhf://spaces/...handling.Example usage
Verified locally against the public bucket
cfahlgren1/duckdb-httpfs-fineweb-edu-5gb:I tested locally against the following buckets:
Public bucket:
hf://buckets/cfahlgren1/duckdb-httpfs-fineweb-edu-5gb/data/**/*.parquetPrivate bucket:
hf://buckets/cfahlgren1/duckdb-httpfs-fineweb-edu-5gb-private/README.mdCREATE SECRET ... TYPE HUGGINGFACE, PROVIDER credential_chainChanges
bucketsas a valid Hugging Face repo type