Skip to content

archive: cache parent directory fd during tar extraction - #49

Draft
thaJeztah wants to merge 5 commits into
moby:mainfrom
thaJeztah:add_cache
Draft

archive: cache parent directory fd during tar extraction#49
thaJeztah wants to merge 5 commits into
moby:mainfrom
thaJeztah:add_cache

Conversation

@thaJeztah

Copy link
Copy Markdown
Member

Each root.*(path) call re-walks every path component via doInRoot, costing 2D syscalls per call at path depth D. A typical file entry triggers ~5 root.* calls, so at depth 4 that is ~40 syscalls in path traversal alone.

This PR adds a dirCache that keeps one parent-directory fd open between entries. Consecutive entries in the same directory reuse the cached fd for all *at(2) operations, reducing path-traversal cost to ~5 syscalls per entry regardless of depth.

@codecov-commenter

codecov-commenter commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 62.50000% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.91%. Comparing base (216738e) to head (8b247ec).
⚠️ Report is 32 commits behind head on main.

Files with missing lines Patch % Lines
dircache_unix.go 69.23% 10 Missing and 10 partials ⚠️
archive.go 42.10% 3 Missing and 8 partials ⚠️
diff.go 50.00% 3 Missing ⚠️
archive_unix.go 75.00% 1 Missing ⚠️
time_nonwindows.go 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #49      +/-   ##
==========================================
- Coverage   65.81%   63.91%   -1.91%     
==========================================
  Files          42       35       -7     
  Lines        2039     2203     +164     
==========================================
+ Hits         1342     1408      +66     
- Misses        519      597      +78     
- Partials      178      198      +20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a dirCache abstraction to reduce repeated path-walk overhead during tar extraction by caching the most recently used parent-directory handle and performing *at(2) operations relative to it (Unix), while keeping Windows behavior as a thin delegation to os.Root.

Changes:

  • Add dirCache implementations for Unix (openat/*at) and Windows (delegation).
  • Thread dirCache through Unpack, UnpackLayer, and createTarFile to reuse parent-directory context across consecutive entries.
  • Adjust chmod/chtime/lchown/lchtime call sites to go through the cache, and update tests to use the new createTarFile signature.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
dircache_unix.go Adds Unix dirCache using cached parent dir fd + *at(2) syscalls for extraction operations.
dircache_windows.go Adds Windows dirCache delegating to os.Root (no fd caching).
archive.go Updates createTarFile and Unpack to use dirCache for extraction operations.
diff.go Updates UnpackLayer to use dirCache, including AUFS temp-root handling.
archive_unix.go Updates handleLChmod to route chmod through dirCache.
archive_windows.go Updates handleLChmod signature for the new dirCache plumbing.
archive_test.go Adjusts tests to construct/close dirCache when calling createTarFile.
time_nonwindows.go Marks lchtimes helper as unused after moving call sites to dirCache.
time_windows.go Removes the Windows stub lchtimes helper.
Comments suppressed due to low confidence (2)

dircache_unix.go:147

  • UtimesNanoAt is called with flags=0 (follows symlinks). If a path is swapped to a symlink concurrently, this can apply timestamps to an unintended target. Using AT_SYMLINK_NOFOLLOW avoids following symlinks for non-symlink entries while keeping the syscall count low.
	if err := unix.UtimesNanoAt(int(d.Fd()), filepath.Base(path), utimes[0:], 0); err != nil {

dircache_unix.go:132

  • chmod currently uses fchmodat with flags=0, which follows symlinks; if the path is swapped to a symlink concurrently, this can chmod an unintended target (potentially outside the extraction root). Use AT_SYMLINK_NOFOLLOW with a fallback (like the previous chmodNoSymlink implementation) to avoid following symlinks.
	if err := unix.Fchmodat(int(d.Fd()), filepath.Base(path), fileModeToPerm(mode), 0); err != nil {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dircache_unix.go Outdated
Comment thread archive.go
Comment on lines 483 to 487
// Source is a regular file. Use os.Root.OpenFile so that all
// path resolution is bounded within root using openat(2) semantics.
// os.Root.OpenFile only accepts the nine least-significant permission
// bits; special bits are applied afterward by handleLChmod.
// We use sequential file access to avoid depleting the standby list
Comment thread time_nonwindows.go Outdated
ctalledo and others added 4 commits July 27, 2026 18:34
Each root.*(path) call re-walks every path component via doInRoot,
costing 2D syscalls per call at path depth D. A typical file entry
triggers ~5 root.* calls, so at depth 4 that is ~40 syscalls in path
traversal alone.

Add a dirCache that keeps one parent-directory fd open between entries.
Consecutive entries in the same directory reuse the cached fd for all
*at(2) operations, reducing path-traversal cost to ~5 syscalls per
entry regardless of depth.

Signed-off-by: Cesar Talledo <cesar.talledo@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah

Copy link
Copy Markdown
Member Author

Last commit probably won't work on Windows (yet); working on that

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants