Skip to content

Conversation

@ShreyasSinha
Copy link
Collaborator

@ShreyasSinha ShreyasSinha commented Dec 22, 2025

Implementing a WriteChannel for uploading objects to Google Cloud Storage using the multipart upload API.
We breaks the data into 5MB chunks and uploads them in parallel using a fixed-size thread pool.
When the channel is closed, it sends a request to GCS to complete the multipart upload, assembling the parts into the final object.

Sample Benchmark: (Writing 5GB via cloudtop)
Using JSON writer with XML MPU API (Pool_SIZE = 16)
--- Aggregate Metrics ---
Total time: 21.179 s
File size: 5000.000 MB
Throughput: 236.082 MB/s

Using gRPC (existing)
--- Aggregate Metrics ---
Total time: 52.980 s
File size: 5000.000 MB
Throughput: 94.376 MB/s

Code used for benchmarking: https://github.com/ShreyasSinha/gcs-xml-mpu-benchmark/blob/main/src/main/java/org/example/ResumableUploadBenchmark.java

Test failures expected as its a prototype code.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: storage Issues related to the googleapis/java-storage API. labels Dec 22, 2025
@ShreyasSinha ShreyasSinha changed the title Sample code to prototype using MPU to implement write methods that currently use resumable uploads chore: Sample code to prototype using MPU to implement write methods that currently use resumable uploads Dec 22, 2025
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Dec 22, 2025
@ShreyasSinha ShreyasSinha added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jan 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the googleapis/java-storage API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. size: l Pull request size is large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant