Repository navigation
Use sbt 2 native task caching in the sbt plugin - #2024
Draft
denisrosca wants to merge 6 commits into
Draft
denisrosca wants to merge 6 commits into
denisrosca wants to merge 6 commits into
Conversation
On sbt 2, `smithy4sCodegen` and `smithy4sGeneratedSmithyFiles` now run as `Def.cachedTask` actions with declared outputs instead of the hand-rolled `Tracked` input/output stores. `smithyBuild` defaults to `Def.uncached`. `@transient` is removed from these three keys. The public tasks keep their `Seq[File]` / `Option[File]` types and return exactly the configured destinations. They are uncached facades over private native actions that return `HashedVirtualFileRef`s. - Outputs outside `rootOutputDirectory` (e.g. a custom `smithy4sOutputDir`) are staged under `smithy-cache/<project>/<config>/<scalaVersion>/<action>/` and materialized at the configured paths. Missing or content-different files are restored on a cache hit without regenerating; identical files are not rewritten. - All writes go through `IO.writeFileAtomically` so a destination that sbt restored as a symlink into the content-addressed store is replaced rather than written through (sbt/sbt#9826). - sbt 1 behavior is unchanged: `CachedTask` and the legacy task bodies moved to `src-2` verbatim. - sbt 2 minimum is now 2.0.4 (sbt/sbt#9492); documented in the installation guide, including the need for `Def.uncached` when overriding these keys on sbt 2. - Scripted coverage: new `native-cache` and `metadata-cache` tests, plus extended `extra-configs`, `generate-openapi-specs`, `smithy-build` and `wildcard-config` tests.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
disclosure: This PR has been prepared with the help of AI coding agents
Summary
On sbt 2,
smithy4sCodegenandsmithy4sGeneratedSmithyFilesnow use sbt'snative task cache (
Def.cachedTask+Def.declareOutput) instead of thehand-rolled
Tracked/CachedTaskstores.smithyBuildloses@transientanddefaults to
Def.uncached. sbt 1 behavior is unchanged.Design
(
Seq[File],Option[File]) and return exactly the configured destinations.They are
Def.uncachedfacades over private actions that returnHashedVirtualFileRefs.Codegen.generateToDiskand thecodegen,codegen-cliand mill plugin modules are untouched.CodegenArgsvalue, withoutput/resourceOutputhashed as path strings. Dependency jars are part ofthe key by content, so changing a transformer jar regenerates.
rootOutputDirectory, butsmithy4sOutputDir/smithy4sResourceDirarecommonly set elsewhere. Those files are generated into an isolated staging tree
(
<root>/smithy4s-cache/<project digest>/<config>/<scalaVersion>/<action>/...).The staged path reversibly encodes the destination, so the returned refs
identify the final files with no manifest or directory scan. The facade then
installs missing or content-different files only (size check, then
Files.mismatch). On a cache hit, damaged or deleted outputs are restoredwithout regenerating, and identical files are not rewritten (mtimes stay
stable).
IO.writeFileAtomically. sbt 2.0.4 restoresdeclared outputs as symlinks into the shared content-addressed store, and
writing through such a link in place can corrupt older cache entries
(A -> B -> A then restores B), see [2.x] Restored outputs are symlinks into the content-addressed store. A task that reopens its output on a rerun corrupts the cached blob, and a later hit restores the wrong content sbt/sbt#9826. The rationale is documented
in
CacheSupport.writeOutput.CachedTask.scalaand the legacy task bodies moved tosrc-2verbatim. The sbt 2 code lives insrc-3.Behavior changes and compatibility
Def.declareOutputinsideloops over dynamic outputs). The plugin cross-build targets 2.0.4. The
sbt2-compat(0.1.0) and sbt 1.12.4 versions are unchanged.@transientis gone, builds thatreplace
smithyBuild,smithy4sCodegenorsmithy4sGeneratedSmithyFilesmustwrap the definition in
Def.uncached. Documented in the installation guide,including a no-op shim for cross-built sbt 1/2 builds.
files are reverted: inside the native root by sbt's output sync, outside it by
our materialization step. sbt 1 generally keeps such edits. This is intentional
and documented.
Both user-facing changes are in the changelog and the installation docs.
Testing
New sbt-test scripted tests, run on both plugin axes:
native-cache(new). Asserts the exact output inventory and the number ofgenerator invocations, with sbt 1 and sbt 2 expectations where they differ:
reload;same-size-different-content files are repaired with no extra generation
(the last case reaches the
Files.mismatchcomparison);metadata-cache(new). Covers wildcard-argument changes, deleted/damaged/relocated metadata files and unchanged mtimes. On sbt 2 only, it also asserts
native cache hits through the cache event log, with no generation on a hit.
smithy-buildandsmithy-build-kind-projector: thesmithyBuildoverridesnow use
Def.uncached(with a build-local sbt 1 no-op shim), as requiredafter removing
@transient.Checklist