fetch-pack: aggregate progress for parallel URI packs - #106
Closed
ttaylorr-oai wants to merge 14 commits into
Closed
fetch-pack: aggregate progress for parallel URI packs#106ttaylorr-oai wants to merge 14 commits into
ttaylorr-oai wants to merge 14 commits into
Conversation
The --packfile mode accepts one --index-pack-arg=<arg> option per argument passed to index-pack, but its documentation and option dependency errors still refer to the plural --index-pack-args form. Correct the spelling and describe the repeatable per-argument form. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
finish_http_pack_request() passes its staging-file descriptor to index-pack through child_process.in. start_command() takes ownership of a supplied descriptor and closes it, even when starting the child fails. Do not close the descriptor again after run_command() returns. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
A resumed pack request may already have all bytes of the remote pack. A server can respond to the resulting Range request with HTTP 416 instead of returning an empty response. Accept that response in each pack-download caller and let index-pack validate the completed staging file. This can happen without concurrent downloads when a previous attempt completed the transfer but failed before indexing it. Add a regression test that seeds a complete partial pack and checks that http-fetch indexes it after the server returns HTTP 416. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Pack requests stage downloads in a predictable partial-pack file so an interrupted transfer can be resumed. Both packfile URI and ordinary dumb HTTP requests use this staging path. Opening it in append mode forces each write to the current end of the file, so concurrent responses can append duplicate data and corrupt the pack. Open the partial pack read-write without O_APPEND and seek once to its current end. Each downloader then retains the offset matching the Range it requested. Because the staging key must uniquely identify immutable pack contents, overlapping responses write the same bytes at the same offsets instead of extending the file with duplicate data. Duplicate the staging descriptor for index-pack instead of reopening the path after closing the stream. Another downloader may unlink the staging path before indexing begins, but index-pack can still read the retained descriptor. Exercise resumed transfers and overlapping 200 and 206 responses, and clarify the staging-key documentation. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
On Windows, an open file must permit FILE_SHARE_DELETE before another process can unlink it. MinGW's non-append O_RDWR open enables that sharing mode only for an existing file; adding O_CREAT falls back to _wopen(), which cannot set it. First try opening the partial pack without O_CREAT. If it does not exist, create it exclusively, close that descriptor, and retry through the existing-file path. A racing creator retries after EEXIST. This ensures that every retained descriptor permits another downloader to unlink the staging path. Add an unlink-while-indexing test that does not require FIFOs and can therefore run on MinGW. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
When index-pack finds an existing keep file it reports pack rather than keep. Accept either result from http-fetch, and only register a keep lockfile when this fetch created it. Read the pack/keep prefix and hash without consuming any following fsck output, validate the reported pack hash against the advertised hash, and exercise a packfile URI fetch with a pre-existing keep file. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the following commit(s), some tests will need to distinguish between `REF_DELTA`s and `OFS_DELTA`s to exercise a new '--no-ref-delta' option for 'pack-objects'. Existing tools report delta relationships, but not how their bases are represented in the pack. Teach 'test-tool pack-deltas' a '--list-deltas' mode. For each delta entry, print the object ID, its REF_DELTA or OFS_DELTA type, and the base object ID or pack offset, respectively. This lets tests inspect pack headers without open-coding a parser. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Some consumers of 'pack-objects' may wish to avoid packs which contain `REF_DELTA` entries. For instance, a 'receive-pack' implementation which retains the resulting pack without building an index of object IDs may prefer every delta base to be discoverable from an earlier entry in the same pack. Teach 'pack-objects' a new `--no-ref-delta` option to avoid writing `REF_DELTA` entries, without changing whether `OFS_DELTA` is allowed. When used without `--delta-base-offset`, no delta representation remains, so avoid delta search entirely. Otherwise, allow new deltas whose bases appear earlier in the same pack. For now, disable delta- and bitmap-reuse under `--no-ref-delta`, since either may copy an existing `REF_DELTA` entry. This is overly pessimistic, but simplifies the changes in this commit. The next commit re-enables reuse in the cases which do not require `REF_DELTA`. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The previous commit disables delta- and bitmap-reuse entirely whenever pack-objects is given '--no-ref-delta' for the sake of simplicity. This is overly pessimistic. When '--delta-base-offset' is also given, delta reuse can remain enabled. A reused delta whose base is written earlier in the output can be encoded as an `OFS_DELTA`, even when its source copy was encoded as a `REF_DELTA`. Preferred bases and external thin-pack bases are different: neither appears in the output, so deltas against either still require encoding the object as a `REF_DELTA`, and thus cannot be reused. Without '--delta-base-offset', delta reuse remains disabled, since no delta representation remains. Bitmap reuse follows a different path, since selected entries may be copied without passing through the code which chooses a delta representation. When given '--no-ref-delta', we must inspect candidate objects individually, and leave `REF_DELTA` entries to the normal object path outside of pack-reuse. We must likewise avoid the special-case for reusing either the single or preferred pack corresponding to the bitmap by whole `eword_t`'s at a time. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Add a 'no-ref-delta' receive-pack capability and teach send-pack to pass '--no-ref-delta' to 'pack-objects' when the server advertises it. Keep this separate from 'ofs-delta' so that a server may request that `send-pack` omit `REF_DELTA` without also accepting `OFS_DELTA`. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A client can rely on a server no-ref-delta promise only if it checks every received pack. Add --no-ref-delta to reject a REF_DELTA while parsing the pack. OFS_DELTA remains accepted. By definition, an OFS_DELTA refers to an earlier entry in the same pack, independently of this option. Test file and stdin input containing REF_DELTA. Retain coverage that accepts OFS_DELTA. Signed-off-by: Friel <friel@openai.com>
The existing no-ref-delta capability lets a receive-pack server ask send-pack not to emit REF_DELTA entries. Protocol v2 fetch needs the same guarantee in the other direction before a client can treat packfile URI responses as independent packs. Add uploadpack.allowNoRefDelta and advertise no-ref-delta as a fetch feature when it is enabled. When a client requests it, pass --no-ref-delta to the inline pack-objects process. The same promise covers every configured packfile URI; upload-pack cannot inspect those packs, so the administrator must create or verify them accordingly. The promise remains independent of ofs-delta. Test advertisement, rejection of an unadvertised request, and the pack-objects argument. Signed-off-by: Friel <friel@openai.com>
Packfile URI responses are indexed in advertised order because a later URI pack can contain a REF_DELTA whose base is installed by an earlier response pack. That ordering prevents independent URI packs from using more than one indexer. When upload-pack advertises no-ref-delta and more than one URI job is configured, request it with packfile-uris. Pass --no-ref-delta to the inline indexer and every URI indexer, then run URI packs concurrently. Keep each URI indexer at one thread so the configured job count also bounds indexer threads. Precreate each expected keep path before starting its http-fetch child, record newly-created paths in the existing pack-lock list, and omit the child's --keep argument. Existing fetch cleanup then owns keeps created by this process while pre-existing keeps remain untouched. Mark both helper process layers for exit cleanup. Use a small poll loop to refill a finished slot until all URI packs are indexed. One job, one URI, or callers without pack-lock ownership retain the serial path. A promised URI pack that contains a REF_DELTA fails closed in index-pack. Test the negotiated request, inline verifier, and three URI indexers. Signed-off-by: Friel <friel@openai.com>
Each URI indexer writes its own progress to stderr. With concurrent workers, their updates overwrite one another and leave a completion line for every pack. The indexers also read from downloaded files, so their throughput does not measure the HTTP transfer. Report cumulative download bytes through http-fetch's existing stdout pipe and render one progress display in fetch-pack. Count packs after indexing and hash verification, and use Git's progress machinery for bytes and throughput. Count newly received bytes, excluding any existing prefix when a download resumes. Suppress the URI indexers' verbose progress and drain both worker pipes while they run. Parse complete stdout records incrementally so partial reads cannot block the other workers. Clear the progress line before forwarding diagnostics, and defer repainting until the diagnostic's line ends. Quiet and no-progress fetches do not request byte reports. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
friel-openai
approved these changes
Sep 10, 2026
This was referenced Sep 10, 2026
dreynaud-oai
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallel URI-pack indexers each write their own progress to stderr. Their lines overwrite one another and leave a completion line for every pack; the reported throughput measures indexing of a downloaded file rather than the HTTP transfer.
Use Git's progress API to show one
Fetching packsline with completed packs, downloaded bytes, and throughput. Count a pack after indexing and hash verification, exclude any existing prefix from resumed-download byte counts, and preserve worker diagnostics. Quiet and no-progress fetches do not request byte reports.The new work is one commit. The earlier topic history was reviewed in #65; this comparison isolates the change from the currently enrolled source pin.
Validation
t0500-progress-display.sh: 18/18, including clearing a split line and repainting a display with an unknown total.t5550-http-fetch-dumb.sh: 63/63, including exact byte counts for resumed and already-complete downloads.t5702-protocol-v2.sh: 90/90, including aggregate progress, quiet/no-progress, and worker errors.codexat0d779fa987a6d214f24d68d36b2d22c32fe85fbcalso passes its developer build and all 94 protocol-v2 tests, including URI authentication coverage.git fsck --full. Three workers emitting 128 KiB of diagnostics each complete without a pipe deadlock.The source topic needs integration with the newer HTTP authentication changes in
codex. A local replay resolves the two overlapping files while retaining the newer authentication and error-response handling. This PR requests approval of the source topic; release-plan admission and the controller rebuild follow separately.GitHub CI currently stops during setup because this older source topic's existing workflow references
actions/github-script@v9, while repository policy requires full-SHA action pins (failed run). No GitHub build or test job ran. The release plan carries action pinning as a separate topic.