docs: blob storage documentation #19194

mrzeszutko · 2025-12-22T16:18:51Z

Summary

Add comprehensive blob storage documentation for node operators
Rename BLOB_SINK_ARCHIVE_API_URL → BLOB_ARCHIVE_API_URL (cleanup after BlobSink removal)
Remove dead environment variables BLOB_SINK_PORT and BLOB_SINK_URL

Description

Following the removal of the BlobSink HTTP server (#19143), this PR:

Adds new documentation (blob_storage.md) explaining how Aztec nodes store and retrieve blob data, including:
- Overview of blob sources (FileStore, L1 Consensus, Archive API)
- PeerDAS and supernode requirements for L1 consensus
- Configuration examples for GCS, S3, and Cloudflare R2
- Authentication setup
- Troubleshooting guide
Adds blob upload documentation (blob_upload.md) for node operators who want to contribute to the network by hosting a blob file store, including:
- Upload configuration with BLOB_FILE_STORE_UPLOAD_URL
- How to expose public HTTP endpoints for GCS, S3, and R2
- Authentication with write permissions
Cleans up legacy naming by renaming BLOB_SINK_ARCHIVE_API_URL to BLOB_ARCHIVE_API_URL - the "sink" terminology is no longer accurate since the HTTP server was removed
Removes dead code - BLOB_SINK_PORT and BLOB_SINK_URL env vars that were left behind after BlobSink removal

Fixes A-389

spalladino

Let's split the instructions related to retrieval and storage, since they are meant for different users. Also let's please delete the generic or redundant instructions inserted by Claude, like "search logs for troubleshooting".

docs/docs-network/setup/blob_storage.md

spalladino · 2026-01-05T12:53:47Z

docs/docs-network/setup/blob_storage.md

+
+### Environment variables
+
+Configure blob sources using environment variables in your `.env` file:


Suggested change

Configure blob sources using environment variables in your `.env` file:

Configure blob sources using environment variables:

Unless we're elsewhere making a point to use .env files to run an Aztec node.

spalladino · 2026-01-05T12:56:11Z

docs/docs-network/setup/blob_storage.md

+### Cloudflare R2 configuration
+
+```bash
+BLOB_FILE_STORE_URLS=s3://my-bucket/?endpoint=https://[ACCOUNT_ID].r2.cloudflarestorage.com


Is this the proper R2 config for a consumer from cloudflare? Seems weird it goes through S3.

I based it on syncing best practices description from docs and actually tested that it works (just tested parts of the code on my own R2), but I cannot find any information about this format in cloudflare docs

Cloudflare's API is S3-compatible

Ngl, looks odd. @spypsy can you confirm? I'm worried the other docs may also be wrong.

spalladino · 2026-01-05T12:57:20Z

docs/docs-network/setup/blob_storage.md

+### Docker Compose integration
+
+Add the environment variables to your `docker-compose.yml`:
+
+```yaml
+environment:
+  # ... other environment variables
+  BLOB_FILE_STORE_URLS: ${BLOB_FILE_STORE_URLS}
+  L1_CONSENSUS_HOST_URLS: ${L1_CONSENSUS_HOST_URLS}
+  BLOB_ARCHIVE_API_URL: ${BLOB_ARCHIVE_API_URL}
+```


Let's not go into different ways to configure env vars here. We should just explain the env vars to set, and let the operator decide how to handle them.

spalladino · 2026-01-05T12:58:06Z

docs/docs-network/setup/blob_storage.md

+:::tip Automatic Configuration
+When using `--network [NETWORK_NAME]`, blob sources are automatically configured for you. Most users don't need to manually configure blob storage.
+:::


Let's also clarify what happens if the user manually sets one of the env vars: they end up replacing the blob sources, not appending to the ones automatically configured.

I added a warning with clarification, additionally updated the initial tip to mention that only filestore sources are configured on the network level

spalladino · 2026-01-05T12:58:30Z

docs/docs-network/setup/blob_storage.md

+  BLOB_ARCHIVE_API_URL: ${BLOB_ARCHIVE_API_URL}
+```
+
+## Authentication


Isn't this whole section only relevant for upload?

I don't think so - depends on the storage configuration but you might need read access keys/permissions to read the data, only HTTP access should always work without any additional permissions

only HTTP access should always work without any additional permissions

My guess is most endpoints (S3, R2, whatever it is) will just be accessed via public http, that's why I was pushing for removing this section. But I guess we can leave it in in case someone wants to set up a permissioned repository.

spalladino · 2026-01-05T12:59:52Z

docs/docs-network/setup/blob_upload.md

+
+- **Google Cloud Storage** - `gs://bucket-name/path/`
+- **Amazon S3** - `s3://bucket-name/path/`
+- **Cloudflare R2** - `s3://bucket-name/path/?endpoint=https://[ACCOUNT_ID].r2.cloudflarestorage.com`


Same question on R2 URL

Same answer - these are urls already used in the docs and they do work

spalladino · 2026-01-05T13:00:50Z

docs/docs-network/setup/blob_upload.md

+BLOB_FILE_STORE_UPLOAD_URL=file:///data/blobs
+```
+
+### Docker Compose integration


Same as in the other doc

spalladino · 2026-01-05T13:01:19Z

docs/docs-network/setup/blob_upload.md

Let's add a section on the .healthcheck file, and request to exclude it from any pruning policies.

spalladino · 2026-01-07T12:17:47Z

docs/docs-network/setup/blob_storage.md

+### Cloudflare R2 configuration
+
+```bash
+BLOB_FILE_STORE_URLS=s3://my-bucket/?endpoint=https://[ACCOUNT_ID].r2.cloudflarestorage.com


Ngl, looks odd. @spypsy can you confirm? I'm worried the other docs may also be wrong.

spalladino · 2026-01-07T12:20:26Z

docs/docs-network/setup/blob_storage.md

+  BLOB_ARCHIVE_API_URL: ${BLOB_ARCHIVE_API_URL}
+```
+
+## Authentication


only HTTP access should always work without any additional permissions

My guess is most endpoints (S3, R2, whatever it is) will just be accessed via public http, that's why I was pushing for removing this section. But I guess we can leave it in in case someone wants to set up a permissioned repository.

spalladino · 2026-01-07T12:51:55Z

@mrzeszutko heads up this failed due to spellchecks

mrzeszutko · 2026-01-08T08:34:41Z

@mrzeszutko heads up this failed due to spellchecks

@spalladino I added missing new words to docs-words.txt

spypsy · 2026-01-08T09:25:00Z

bit late but just to confirm @spalladino that format does look correct and it's what we use currently for snapshot uploading to our bucket. we also have a &publicBaseUrl=https://aztec-labs-snapshots.com at the end but don't think it's needed for uploading

## Summary - Add `docs-network`, `docs-developers`, `developer_versioned_docs`, and `network_versioned_docs` to `.rebuild_patterns` so that changes to markdown files in these directories invalidate the test cache for spellcheck ## Problem PR #19194 passed spellcheck on the PR but failed on merge queue. This happened because: 1. The spellcheck test cache hash is computed from `.rebuild_patterns` 2. `.rebuild_patterns` didn't include the docs content directories (`docs-network/`, etc.) 3. On PR: hash didn't change → cache hit → spellcheck "passed" using stale cached result 4. On merge queue: `USE_TEST_CACHE=0` → spellcheck ran fresh → found spelling errors ## Solution Add the missing docs content directories to `.rebuild_patterns` so any markdown changes invalidate the cache and spellcheck runs fresh. ## Test plan - [x] Verify `.rebuild_patterns` syntax is correct - [ ] CI should run spellcheck with the new hash 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Josh Crites <[email protected]>

## Summary - Add `docs-network`, `docs-developers`, `developer_versioned_docs`, and `network_versioned_docs` to `.rebuild_patterns` so that changes to markdown files in these directories invalidate the test cache for spellcheck ## Problem PR #19194 passed spellcheck on the PR but failed on merge queue. This happened because: 1. The spellcheck test cache hash is computed from `.rebuild_patterns` 2. `.rebuild_patterns` didn't include the docs content directories (`docs-network/`, etc.) 3. On PR: hash didn't change → cache hit → spellcheck "passed" using stale cached result 4. On merge queue: `USE_TEST_CACHE=0` → spellcheck ran fresh → found spelling errors ## Solution Add the missing docs content directories to `.rebuild_patterns` so any markdown changes invalidate the cache and spellcheck runs fresh. ## Test plan - [x] Verify `.rebuild_patterns` syntax is correct - [ ] CI should run spellcheck with the new hash 🤖 Generated with [Claude Code](https://claude.com/claude-code)

mrzeszutko changed the title ~~Blob storage documentation~~ docs: blob storage documentation Dec 22, 2025

mrzeszutko marked this pull request as ready for review December 22, 2025 21:07

spalladino requested changes Dec 23, 2025

View reviewed changes

mrzeszutko force-pushed the feature/blob-storage-docs branch from 2078cd3 to 3c4be61 Compare January 5, 2026 12:12

spalladino reviewed Jan 5, 2026

View reviewed changes

mrzeszutko force-pushed the feature/blob-storage-docs branch from 3c4be61 to 66e0fdc Compare January 7, 2026 11:09

mrzeszutko requested a review from spalladino January 7, 2026 11:10

spalladino approved these changes Jan 7, 2026

View reviewed changes

spalladino added this pull request to the merge queue Jan 7, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 7, 2026

critesjosh mentioned this pull request Jan 7, 2026

fix(docs): include docs content folders in cache hash #19385

Merged

2 tasks

Blob storage documentation

556a9a9

mrzeszutko force-pushed the feature/blob-storage-docs branch from 66e0fdc to 556a9a9 Compare January 8, 2026 08:28

mrzeszutko added this pull request to the merge queue Jan 8, 2026

Merged via the queue into next with commit 651732d Jan 8, 2026
16 checks passed

mrzeszutko deleted the feature/blob-storage-docs branch January 8, 2026 09:08


		### Environment variables

		Configure blob sources using environment variables in your `.env` file:

docs: blob storage documentation #19194

docs: blob storage documentation #19194

Uh oh!

Conversation

mrzeszutko commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

Uh oh!

spalladino left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrzeszutko Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spalladino commented Jan 7, 2026

Uh oh!

mrzeszutko commented Jan 8, 2026

Uh oh!

Uh oh!

spypsy commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mrzeszutko commented Dec 22, 2025 •

edited

Loading

mrzeszutko Jan 7, 2026 •

edited

Loading