Skip to content

add support for hf://buckets in httpfs#293

Open
cfahlgren1 wants to merge 2 commits into
duckdb:mainfrom
cfahlgren1:feat/hf-buckets-support
Open

add support for hf://buckets in httpfs#293
cfahlgren1 wants to merge 2 commits into
duckdb:mainfrom
cfahlgren1:feat/hf-buckets-support

Conversation

@cfahlgren1

Copy link
Copy Markdown

Hugging Face recently introduced Storage Buckets as mutable, non-versioned object storage on the Hub, addressable as hf://buckets/<namespace>/<bucket>.

This PR adds hf://buckets/... support to httpfs alongside the existing hf://datasets/... and hf://spaces/... handling.

Example usage

Verified locally against the public bucket cfahlgren1/duckdb-httpfs-fineweb-edu-5gb:

SELECT filename, token_count
FROM parquet_scan(
    'hf://buckets/cfahlgren1/duckdb-httpfs-fineweb-edu-5gb/data/**/*.parquet',
    filename = true
)
LIMIT 1;

I tested locally against the following buckets:

Public bucket:

  • hf://buckets/cfahlgren1/duckdb-httpfs-fineweb-edu-5gb/data/**/*.parquet

Private bucket:

  • hf://buckets/cfahlgren1/duckdb-httpfs-fineweb-edu-5gb-private/README.md
  • verified with CREATE SECRET ... TYPE HUGGINGFACE, PROVIDER credential_chain

Changes

  • accept buckets as a valid Hugging Face repo type
  • treat buckets as non-revisioned resources
  • build bucket file URLs without inserting a revision segment


# FIXME: currently this will not fail the Linux HTTPFS ci job if it fails, because it might do so due to networking issues
# however having a CI job dedicated to remote tests that may spuriously fail would solve this
# TODO: add hf://buckets integration coverage here once there is a maintainer-owned public bucket fixture

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A core contributor would need to create a small bucket fixture so we can add hf://buckets/... integration coverage in here, similar to the existing remote Hugging Face fixtures there under samansmink/duckdb_ci_tests for public datasets, private datasets, and spaces?

@lhoestq

lhoestq commented Apr 1, 2026

Copy link
Copy Markdown

Yay ! let us know @carlopi if it looks good to you :)

@cfahlgren1

Copy link
Copy Markdown
Author

@carlopi friendly bump if this makes sense to add 🤗

@nleroy917

Copy link
Copy Markdown

Very interested in this!!!

@lhoestq

lhoestq commented May 28, 2026

Copy link
Copy Markdown

Gentle ping on this one :)

@davanstrien

Copy link
Copy Markdown

This sounds great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants