Skip to content

feat(sync): add gRPC-based block downloader#503

Open
kariy wants to merge 7 commits intomainfrom
feat/gateway-sync-grpc
Open

feat(sync): add gRPC-based block downloader#503
kariy wants to merge 7 commits intomainfrom
feat/gateway-sync-grpc

Conversation

@kariy
Copy link
Copy Markdown
Member

@kariy kariy commented Mar 22, 2026

Adds gRPC as a third sync source for full node block synchronization, alongside the existing feeder gateway and JSON-RPC downloaders. Usage: --sync.grpc http://host:5051. This is a Katana-specific endpoint — it only works when syncing from another Katana node with --grpc enabled.

Implementation

Block downloader (crates/sync/stage/src/blocks/downloader.rs):

  • GrpcBatchBlockDownloader implements BlockDownloader using BatchDownloader<GrpcDownloader> for concurrent downloads with retry logic
  • Downloads GetBlockWithReceipts and GetStateUpdate concurrently per block via tokio::try_join!
  • Converts proto responses directly to BlockData (proto → primitives), avoiding intermediate rpc_types conversions and keeping the code self-contained in the sync crate
  • Lazy connection via OnceCell<GrpcClient> — connects on first download rather than at startup, so Node::build() stays synchronous and transient connection failures are handled by the existing retry logic in BatchDownloader
  • Clones GrpcClient per download call since tonic's Channel is Arc-backed (cheap to clone), avoiding Mutex contention on concurrent downloads where &mut self is required by tonic

Class downloader (crates/sync/stage/src/classes/downloader.rs):

  • GrpcClassDownloader implements ClassDownloader using the same lazy connection and clone-per-download patterns
  • Deserializes the class from the JSON payload in the gRPC response's abi field

gRPC server fix (crates/grpc/src/handlers/starknet.rs):

  • Fixed get_class and get_class_at handlers which previously returned stubbed/empty responses
  • JSON-serializes the full Class (Sierra or Legacy) and stores it in the abi field of the appropriate proto variant (ContractClass or DeprecatedContractClass)
  • Uses JSON rather than a binary format (e.g. postcard) because the class types are not compatible with non-human-readable serialization

CLI (crates/cli/src/options.rs):

  • --sync.grpc <URL> flag, mutually exclusive with --sync.gateway and --sync.rpc

Error handling:

  • Retryable gRPC status codes (Unavailable, ResourceExhausted, DeadlineExceeded) and transport errors trigger BatchDownloader's exponential backoff retry
  • All other status codes are treated as permanent failures

🤖 Generated with Claude Code

kariy and others added 2 commits March 21, 2026 23:09
Add a third sync source alongside the existing feeder gateway and
JSON-RPC downloaders, enabling block synchronization via Katana's
gRPC endpoint.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implement GrpcClassDownloader for the Classes stage, completing
gRPC sync support for both blocks and classes. The class data is
deserialized from the JSON payload in the gRPC response's abi field.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark

Details
Benchmark suite Current: aaff16f Previous: b4121df Ratio
CompiledClass(fixture)/compress 2625505 ns/iter (± 48439) 2625621 ns/iter (± 88165) 1.00
CompiledClass(fixture)/decompress 2912041 ns/iter (± 183236) 2905010 ns/iter (± 12454) 1.00
ExecutionCheckpoint/compress 33 ns/iter (± 0) 32 ns/iter (± 3) 1.03
ExecutionCheckpoint/decompress 27 ns/iter (± 11) 27 ns/iter (± 3) 1
PruningCheckpoint/compress 32 ns/iter (± 5) 33 ns/iter (± 6) 0.97
PruningCheckpoint/decompress 27 ns/iter (± 4) 27 ns/iter (± 5) 1
VersionedHeader/compress 653 ns/iter (± 14) 655 ns/iter (± 25) 1.00
VersionedHeader/decompress 812 ns/iter (± 42) 806 ns/iter (± 7) 1.01
StoredBlockBodyIndices/compress 76 ns/iter (± 5) 76 ns/iter (± 6) 1
StoredBlockBodyIndices/decompress 36 ns/iter (± 8) 35 ns/iter (± 5) 1.03
StorageEntry/compress 140 ns/iter (± 1) 140 ns/iter (± 10) 1
StorageEntry/decompress 141 ns/iter (± 2) 139 ns/iter (± 3) 1.01
ContractNonceChange/compress 140 ns/iter (± 2) 140 ns/iter (± 11) 1
ContractNonceChange/decompress 235 ns/iter (± 3) 248 ns/iter (± 4) 0.95
ContractClassChange/compress 207 ns/iter (± 1) 209 ns/iter (± 3) 0.99
ContractClassChange/decompress 253 ns/iter (± 3) 252 ns/iter (± 3) 1.00
ContractStorageEntry/compress 147 ns/iter (± 2) 155 ns/iter (± 3) 0.95
ContractStorageEntry/decompress 310 ns/iter (± 12) 309 ns/iter (± 15) 1.00
GenericContractInfo/compress 137 ns/iter (± 2) 138 ns/iter (± 2) 0.99
GenericContractInfo/decompress 107 ns/iter (± 3) 108 ns/iter (± 4) 0.99
Felt/compress 72 ns/iter (± 3) 71 ns/iter (± 6) 1.01
Felt/decompress 56 ns/iter (± 4) 54 ns/iter (± 4) 1.04
BlockHash/compress 72 ns/iter (± 1) 71 ns/iter (± 5) 1.01
BlockHash/decompress 56 ns/iter (± 4) 55 ns/iter (± 5) 1.02
TxHash/compress 72 ns/iter (± 1) 71 ns/iter (± 4) 1.01
TxHash/decompress 56 ns/iter (± 6) 54 ns/iter (± 4) 1.04
ClassHash/compress 72 ns/iter (± 2) 71 ns/iter (± 6) 1.01
ClassHash/decompress 56 ns/iter (± 9) 54 ns/iter (± 5) 1.04
CompiledClassHash/compress 71 ns/iter (± 1) 71 ns/iter (± 5) 1
CompiledClassHash/decompress 56 ns/iter (± 5) 54 ns/iter (± 5) 1.04
BlockNumber/compress 35 ns/iter (± 2) 34 ns/iter (± 2) 1.03
BlockNumber/decompress 26 ns/iter (± 0) 26 ns/iter (± 1) 1
TxNumber/compress 35 ns/iter (± 2) 34 ns/iter (± 2) 1.03
TxNumber/decompress 25 ns/iter (± 1) 25 ns/iter (± 1) 1
FinalityStatus/compress 1 ns/iter (± 0) 1 ns/iter (± 0) 1
FinalityStatus/decompress 12 ns/iter (± 1) 12 ns/iter (± 1) 1
TypedTransactionExecutionInfo/compress 15949 ns/iter (± 152) 18191 ns/iter (± 108) 0.88
TypedTransactionExecutionInfo/decompress 3609 ns/iter (± 63) 3570 ns/iter (± 71) 1.01
VersionedContractClass/compress 379 ns/iter (± 4) 370 ns/iter (± 5) 1.02
VersionedContractClass/decompress 760 ns/iter (± 4) 780 ns/iter (± 29) 0.97
MigratedCompiledClassHash/compress 148 ns/iter (± 4) 142 ns/iter (± 5) 1.04
MigratedCompiledClassHash/decompress 143 ns/iter (± 6) 151 ns/iter (± 4) 0.95
ContractInfoChangeList/compress 1705 ns/iter (± 40) 1692 ns/iter (± 34) 1.01
ContractInfoChangeList/decompress 2092 ns/iter (± 377) 2100 ns/iter (± 375) 1.00
BlockChangeList/compress 711 ns/iter (± 57) 714 ns/iter (± 19) 1.00
BlockChangeList/decompress 847 ns/iter (± 154) 860 ns/iter (± 152) 0.98
ReceiptEnvelope/compress 31151 ns/iter (± 419) 31191 ns/iter (± 302) 1.00
ReceiptEnvelope/decompress 6096 ns/iter (± 534) 6149 ns/iter (± 299) 0.99
TrieDatabaseValue/compress 182 ns/iter (± 3) 188 ns/iter (± 3) 0.97
TrieDatabaseValue/decompress 244 ns/iter (± 1) 243 ns/iter (± 2) 1.00
TrieHistoryEntry/compress 321 ns/iter (± 1) 319 ns/iter (± 21) 1.01
TrieHistoryEntry/decompress 263 ns/iter (± 8) 267 ns/iter (± 9) 0.99

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 22, 2026

Runner: AMD EPYC 7763 64-Core Processor (4 cores) · 15Gi RAM

kariy and others added 2 commits March 21, 2026 23:51
Remove ClassProtoResponse trait abstraction — both call sites
construct their response directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the structured ContractClass/DeprecatedContractClass proto
messages with raw bytes variants in the GetClass and GetClassAt
responses. The oneof still distinguishes Sierra vs Legacy, but each
variant is just the JSON-serialized class bytes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 22, 2026

Codecov Report

❌ Patch coverage is 0.79745% with 622 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.37%. Comparing base (9bde0ae) to head (aaff16f).
⚠️ Report is 330 commits behind head on main.

Files with missing lines Patch % Lines
crates/sync/stage/src/blocks/downloader.rs 0.00% 506 Missing ⚠️
crates/sync/stage/src/classes/downloader.rs 0.00% 63 Missing ⚠️
crates/node/full/src/lib.rs 0.00% 41 Missing ⚠️
crates/grpc/src/handlers/starknet.rs 35.71% 9 Missing ⚠️
crates/cli/src/full.rs 0.00% 2 Missing ⚠️
crates/cli/src/options.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #503      +/-   ##
==========================================
- Coverage   73.32%   67.37%   -5.95%     
==========================================
  Files         209      292      +83     
  Lines       23132    37388   +14256     
==========================================
+ Hits        16961    25191    +8230     
- Misses       6171    12197    +6026     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant