Skip to content

Fix stale dependent framework caches by propagating dependency cache keys#271

Merged
Ckitakishi merged 19 commits into
mainfrom
fix/cache-key-dependency-propagation
Jul 15, 2026
Merged

Fix stale dependent framework caches by propagating dependency cache keys#271
Ckitakishi merged 19 commits into
mainfrom
fix/cache-key-dependency-propagation

Conversation

@ikesyo

@ikesyo ikesyo commented May 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes stale cache reuse for dependent frameworks by including dependency build state in cache keys.

Previously, a downstream target could keep reusing a cached artifact even after a dependency was rebuilt with incompatible changes.

Related issue: #265

What Changed

  • Added dependencyCacheKeyChecksums to SwiftPMCacheKey.
  • Added graph-based cache key calculation (calculateCacheKeys(for:)) that computes keys in dependency order and embeds direct dependency key checksums.
  • Updated framework cache flows to reuse a single cache-key map across existing framework validation, restore checks, cache writes, version file generation, and restored cache sharing.
  • Preserved backward compatibility for old version files by decoding missing dependencyCacheKeyChecksums as an empty list.
  • Stabilized dependency checksum ordering with a deterministic tie-breaker (targetName, then checksum).
  • Kept historical behavior when cache policies are empty (cache feature is fully off).
  • Fall back to building without cache participation when cache key precomputation fails, so missing local repository metadata does not abort the build before the intended build path.

Tests

  • Extended CacheSystemTests to cover encoding with dependency checksums, backward-compatible decode, direct dependency checksum propagation, and dependent key invalidation when dependency build state changes.
  • Updated cache key tests to copy the PartialCacheTestPackage fixture into a temporary tagged git repository, keeping them stable in CI merge checkouts.
  • FrameworkProducerTests passes locally.
  • CI is passing.

Impact

  • Dependency changes now correctly invalidate dependent cache keys.
  • Reduces risk of link/runtime failures caused by stale downstream cache artifacts.
  • Maintains existing non-cache behavior for packages whose cache keys cannot be resolved.

@ikesyo ikesyo changed the title Propagate dependency build state in framework cache keys Fix stale dependent framework caches by propagating dependency cache keys May 13, 2026
@ikesyo
ikesyo force-pushed the fix/cache-key-dependency-propagation branch from 0dc268b to 3cab5ce Compare May 14, 2026 04:42
@ikesyo
ikesyo marked this pull request as ready for review May 14, 2026 05:09
@ikesyo
ikesyo requested review from freddi-kit and giginet May 14, 2026 05:09
@giginet

giginet commented May 20, 2026

Copy link
Copy Markdown
Owner

Thanks for the fix — the propagation approach via dependencyCacheKeyChecksums looks right and the backward-compatible decode is well handled. Two concerns I'd like to raise before merging.

1. precomputeCacheKeysIfPossible swallows all errors silently

} catch {
    logger.warning(
        "⚠️ Cache key precomputation failed. Continue build without cache restore/save for this run.",
        metadata: .color(.yellow)
    )
    return [:]
}

Previously, errors from calculateCacheKey(of:) such as revisionNotDetected, compilerVersionNotDetected, and xcodeVersionNotDetected would propagate and abort the build. With this change, any error falls back silently to a "cache fully disabled" mode for the whole run. Two problems:

  • The warning does not include the underlying error, so a user running prepare mode on a large dependency tree has no way to discover why caching was silently disabled. They will just observe slow rebuilds.
  • The catch is too broad. If the intent is to tolerate missing git metadata in fixtures (as the PR description says), only Error.revisionNotDetected should warrant the fallback. compilerVersionNotDetected / xcodeVersionNotDetected indicate genuinely broken environments and should still abort.

Suggested changes:

  • At minimum, include error.localizedDescription (or \(error)) in the warning so users can diagnose.
  • Narrow the catch to the specific case(s) that justify a fallback and rethrow the rest.
  • Optionally, gate the fallback behind a CLI flag so strict cache validation can remain the default.

2. Cache key precomputation is now fully serialized

calculateCacheKeys(for:) iterates graph.allNodes and awaits each calculateCacheKeyRecursively call sequentially. Each call internally spawns subprocesses for clang --version, xcodebuild -version, git describe, and git rev-parse. Previously these were invoked inside withTaskGroup blocks in validateExistingFrameworks / restoreCaches / shareCachesToStorage and ran in parallel across targets. For projects with many dependencies this could regress wall-clock time noticeably.

Suggested changes (in order of effort/payoff):

  • clangVersion and xcodeVersion are constants for the lifetime of a Scipio run. Fetch them once (e.g. at CacheSystem initialization or at the start of calculateCacheKeys(for:)) and reuse, instead of re-running the subprocesses for every target. This is cheap and recovers most of the regression.
  • Parallelize same-level dependencies inside calculateCacheKeys(for:) while preserving topological ordering — for example, processing the graph layer by layer with withTaskGroup.

The first one alone is low-risk and a clear win.

@ikesyo
ikesyo force-pushed the fix/cache-key-dependency-propagation branch from 3cab5ce to df83ef2 Compare May 22, 2026 15:22
@ikesyo

ikesyo commented May 22, 2026

Copy link
Copy Markdown
Collaborator Author

@giginet Thanks for the feedback. I addressed the points. Please re-review the changes.

520a4eb...df83ef2

@ikesyo
ikesyo requested review from Ckitakishi, S-Shimotori and giginet and removed request for freddi-kit and giginet May 22, 2026 16:10
Comment thread Sources/ScipioKit/Producer/FrameworkProducer.swift Outdated
Comment thread Sources/ScipioKit/Producer/Cache/CacheSystem.swift Outdated
Comment thread Sources/ScipioKit/Producer/FrameworkProducer.swift Outdated
Comment thread Sources/ScipioKit/Producer/Cache/CacheSystem.swift Outdated
Comment thread Sources/ScipioKit/Producer/Cache/CacheSystem.swift Outdated
Comment thread Sources/ScipioKit/Producer/FrameworkProducer.swift Outdated
ikesyo and others added 9 commits July 15, 2026 08:38
This change addresses stale cache reuse in dependent frameworks by including direct dependency cache key checksums in SwiftPMCacheKey and computing keys in dependency order.

It also preserves backward compatibility for existing version files, stabilizes dependency checksum ordering, and keeps historical behavior when cache policies are disabled (no restore/save/version file generation).

Tests were extended to cover backward-compatible decoding, dependency checksum propagation, and cache key invalidation cascading when dependency build state changes.

Co-authored-by: Codex <noreply@openai.com>
Move helper methods into an extension without behavior changes to keep FrameworkProducer within SwiftLint type_body_length limits.

Co-authored-by: Codex <noreply@openai.com>
When cache key graph precomputation fails, fallback to cache-disabled behavior for the run and continue building targets. This restores expected behavior for fixtures without repository version metadata and avoids failing tests before the intended compile-failure path.

Co-authored-by: Codex <noreply@openai.com>
Keep FrameworkProducer helper definitions aligned with the order they are used from processAllTargets while preserving behavior.

Co-authored-by: Codex <noreply@openai.com>
Copy the PartialCacheTestPackage fixture into a temporary git repository and tag it before calculating cache keys. This keeps the tests stable in CI merge checkouts where the repository itself does not expose the fixture as a tagged package revision.

Co-authored-by: Codex <noreply@openai.com>
Keep the no-cache fallback for local package revisions that cannot be resolved, but propagate other cache key errors such as compiler or Xcode version detection failures.

Co-authored-by: Codex <noreply@openai.com>
Fetch clang and Xcode versions once for graph-wide cache key calculation and pass the shared environment into each target key calculation.

Co-authored-by: Codex <noreply@openai.com>
Process cache key calculation from leaf nodes upward and calculate targets in the same dependency layer concurrently while keeping parent targets blocked until all child keys are available.

Co-authored-by: Codex <noreply@openai.com>
Copy the TestingPackage fixture into a per-test temporary directory and derive project and local disk cache paths from that isolated location so Swift Testing can keep running cases in parallel without cache cleanup races.

Co-authored-by: Codex <noreply@openai.com>
@ikesyo
ikesyo force-pushed the fix/cache-key-dependency-propagation branch from df83ef2 to a39eca0 Compare July 15, 2026 01:48
ikesyo and others added 5 commits July 15, 2026 10:59
Keep cache keys for targets whose revisions are available when another target cannot resolve its revision. Propagate unavailable status to dependents so only the affected cache participation is skipped, and cover the behavior with a regression test.

Co-authored-by: Codex <noreply@openai.com>
Check that a cache key exists before logging a framework cache save, so targets intentionally excluded from cache participation are not reported as cached.

Co-authored-by: Codex <noreply@openai.com>
Report the same VersionFile warning when a target has no precomputed cache key, instead of silently skipping version file generation.

Co-authored-by: Codex <noreply@openai.com>
Remove the keyless generateVersionFile API, which cannot represent dependency cache key checksums for non-leaf targets. Migrate RunnerTests to calculate one cache key map from the resolved dependency graph and pass explicit keys when generating version files.

Co-authored-by: Codex <noreply@openai.com>
Cap AsyncOperations asyncMap to CacheSystem.defaultParallelNumber while preserving parallel processing for same-level graph nodes.

Co-authored-by: Codex <noreply@openai.com>
ikesyo and others added 5 commits July 15, 2026 11:00
Calculate graph cache keys directly from FrameworkProducer now that revision failures are handled per target by CacheSystem. Remove the redundant helper and its do-catch wrapper so non-revision errors propagate unchanged.

Co-authored-by: Codex <noreply@openai.com>
Identify the framework when VersionFile generation is skipped because its cache key is unavailable or the version file cannot be written.

Co-authored-by: Codex <noreply@openai.com>
Omit empty dependency checksum lists from serialized cache keys so legacy dependency-free artifacts remain compatible. Add coverage for decoding and re-encoding the legacy format.
Keep graph traversal and cache state management in calculateCacheKeys while isolating concurrent per-layer calculation.
Use the graph-based cache key API throughout the production tests and update revision-missing expectations to match partial graph results.
@ikesyo
ikesyo force-pushed the fix/cache-key-dependency-propagation branch from a39eca0 to 3fd4dc3 Compare July 15, 2026 02:02
@ikesyo
ikesyo requested a review from Ckitakishi July 15, 2026 03:22

@Ckitakishi Ckitakishi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@Ckitakishi
Ckitakishi merged commit 201f21f into main Jul 15, 2026
5 checks passed
@Ckitakishi
Ckitakishi deleted the fix/cache-key-dependency-propagation branch July 15, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache key should account for dependency build state to prevent stale cache usage

3 participants