With the improved storage provider architectures, like Curio, SPs are now capable of fetching advertisement entries fast (within 10s of milliseconds). In that world, the bottleneck shifts from entry chuck generation to roundtrips between indexer and SP.
Consider supporting streaming response when fetching ad entries:
- SP continuously writes multihashes in expected order
- indexers cumulatively calculate the CID of the hashes received, until the CID matches the expected CID in advertisement (with some upper bound to the max number of multihashes accumulated, currently at ~16K)
- SP uses keep-alive to signal WIP in returning further entities to indexers (with some upper bound for maximum wait time on indexer side).
This approach should significantly improve ingestion time for SPs that are able to serve entries fast enough. It shifts the ingest architecture to a reactive model that would avoid roundtrip bottlenecks.
With the improved storage provider architectures, like Curio, SPs are now capable of fetching advertisement entries fast (within 10s of milliseconds). In that world, the bottleneck shifts from entry chuck generation to roundtrips between indexer and SP.
Consider supporting streaming response when fetching ad entries:
This approach should significantly improve ingestion time for SPs that are able to serve entries fast enough. It shifts the ingest architecture to a reactive model that would avoid roundtrip bottlenecks.