Redundant multipart uploads for files with the same digest

[hannes-ucsc]

(Posted by Claude Code)

When the mirror queue contains multiple MirrorFileAction messages for files with the same digest but different UUIDs, each one starts a separate multipart upload for the same file object. Because the FIFO group serializes message handling but not upload lifecycles, all uploads progress roughly in lockstep — each handler invocation uploads one part and yields the next MirrorPartAction. As a result, all uploads finish at roughly the same time, wasting bandwidth and Lambda execution time on redundant downloads and uploads.

The per-part _info_exists check added in #8142 helps when there is a significant time gap between the duplicate actions, but does not help in the common case where the actions are clustered together in the queue.

[hannes-ucsc]

(Posted by Claude Code)

Proposed solution

Before uploading each part, check for concurrent multipart uploads to the same file object. If another upload with a higher upload ID exists, the current upload is the loser. The loser stops uploading parts but keeps its upload alive, and queues a WaitForFileAction carrying the suspended FileUpload state.

The WaitForFileAction handler has three outcomes:

1. Info object exists: the winner completed successfully. Abort the loser's upload, update the info object (to merge content types), done.
2. Winner's upload is still active: the winner is still working. Re-queue another WaitForFileAction to check again later.
3. Winner's upload disappeared but no info object: the winner failed. Resume uploading parts from where the loser left off by queuing a MirrorPartAction with the loser's preserved FileUpload state.

This avoids redundant downloads and uploads in the common case where duplicate actions are clustered together in the queue. The tradeoff is that only the winner verifies the file digest, so corrupt duplicates in the source repository would go undetected by the losers. The optimization could be made optional via a CLI flag.

State machine

Based on the diagram in #8142 (comment), with the proposed changes from both #8142 and this issue:

stateDiagram-v2

    [*] --> MirrorFileAction

    state MirrorFileAction {
        direction TB
        fa1: Check info object
        fa2: Check file object
        fa3: Check file size
        fa1 --> fa2 : not found
        fa1 --> UpdateInfo : found
        fa2 --> fa3 : not found
        state "AssertionError" as ae1
        fa2 --> ae1 : found
        fa3 --> s1 : size ≤ part size
        fa3 --> c1 : size > part size
        UpdateInfo --> [*] : exists
        UpdateInfo --> StorageObjectNotFound : doesn't exist
    }

    state SingleUpload {
        direction TB
        s1: Download
        s2: Hash
        s3: Verify digest
        s4: Upload (PUT)
        s5: Create info
        s1 --> s2
        s2 --> s3
        s3 --> s4 : match
        state "AssertionError" as ae2
        s3 --> ae2 : mismatch
        s4 --> s5
        s5 --> [*] : doesn't exist
        state "StorageObjectExists" as soe1
        s5 --> soe1 : already exists
    }

    state CreateMultipartUpload {
        direction TB
        c1: Create upload
        c2: Download part 0
        c3: Update hash
        c4: Upload part 0
        c5: Assert more parts remain
        c1 --> c2
        c2 --> c3
        c3 --> c4
        c4 --> c5
        c5 --> p1
    }

    state MirrorPartAction {
        direction TB
        p1: Check info object
        p2: List concurrent uploads
        p3: Download part n
        p4: Update hash
        p5: Upload part n
        p6: Check for more parts
        state "Abort upload and update info" as pa1
        p1 --> pa1 : found
        pa1 --> [*]
        p1 --> p2 : not found
        p2 --> w1 : higher upload ID exists
        p2 --> p3 : no higher upload ID
        p3 --> p4
        p4 --> p5
        p5 --> p6
        p6 --> p1 : more parts remain
        p6 --> z1 : last part uploaded
    }

    state WaitForFileAction {
        direction TB
        w1: Check info object
        w2: Check winner upload still active
        state "Abort upload and update info" as wa1
        w1 --> wa1 : found
        wa1 --> [*]
        w1 --> w2 : not found
        w2 --> w1 : winner still active
        w2 --> p1 : winner failed
    }

    state FinalizeFileAction {
        direction TB
        z1: Verify digest
        z2: Complete multipart upload
        z3: Abort upload
        z4: Create info
        z5: Update info
        z1 --> z2 : match
        state "AssertionError" as ae3
        z1 --> ae3 : mismatch
        z2 --> z4 : object doesn't exist
        z2 --> z3 : object already exists
        z3 --> z4
        z4 --> [*] : doesn't exist
        z4 --> z5 : already exists
        z5 --> [*]
    }

Loading

[hannes-ucsc]

Message consolidation would be another approach, similar to how we handle tallies in the indexer. This would require receiving multiple messages at once, grouping them by digest (the message group ID), thereby reducing, but not eliminating, the possibility of redundant downloads.