Skip to content

Use sbt 2 native task caching in the sbt plugin - #2024

Draft
denisrosca wants to merge 6 commits into
disneystreaming:series/0.19from
denisrosca:sbt2_caching
Draft

denisrosca wants to merge 6 commits into
disneystreaming:series/0.19from
denisrosca:sbt2_caching

Conversation

@denisrosca

@denisrosca denisrosca commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

disclosure: This PR has been prepared with the help of AI coding agents

Summary

On sbt 2, smithy4sCodegen and smithy4sGeneratedSmithyFiles now use sbt's
native task cache (Def.cachedTask + Def.declareOutput) instead of the
hand-rolled Tracked/CachedTask stores. smithyBuild loses @transient and
defaults to Def.uncached. sbt 1 behavior is unchanged.

Design

  • Facade over native actions. The public keys keep their types
    (Seq[File], Option[File]) and return exactly the configured destinations.
    They are Def.uncached facades over private actions that return
    HashedVirtualFileRefs. Codegen.generateToDisk and the codegen,
    codegen-cli and mill plugin modules are untouched.
  • Explicit input policy. The cache key is the CodegenArgs value, with
    output/resourceOutput hashed as path strings. Dependency jars are part of
    the key by content, so changing a transformer jar regenerates.
  • External destinations. sbt only stores declared outputs under
    rootOutputDirectory, but smithy4sOutputDir/smithy4sResourceDir are
    commonly set elsewhere. Those files are generated into an isolated staging tree
    (<root>/smithy4s-cache/<project digest>/<config>/<scalaVersion>/<action>/...).
    The staged path reversibly encodes the destination, so the returned refs
    identify the final files with no manifest or directory scan. The facade then
    installs missing or content-different files only (size check, then
    Files.mismatch). On a cache hit, damaged or deleted outputs are restored
    without regenerating, and identical files are not rewritten (mtimes stay
    stable).
  • Atomic writes. All writes use IO.writeFileAtomically. sbt 2.0.4 restores
    declared outputs as symlinks into the shared content-addressed store, and
    writing through such a link in place can corrupt older cache entries
    (A -> B -> A then restores B), see [2.x] Restored outputs are symlinks into the content-addressed store. A task that reopens its output on a rerun corrupts the cached blob, and a later hit restores the wrong content sbt/sbt#9826. The rationale is documented
    in CacheSupport.writeOutput.
  • sbt 1 untouched. CachedTask.scala and the legacy task bodies moved to
    src-2 verbatim. The sbt 2 code lives in src-3.

Behavior changes and compatibility

  • Minimum sbt 2 version is now 2.0.4 ([2.x] fix: Fixes Def.declareOutput invoked in a loop sbt/sbt#9492, Def.declareOutput inside
    loops over dynamic outputs). The plugin cross-build targets 2.0.4. The
    sbt2-compat (0.1.0) and sbt 1.12.4 versions are unchanged.
  • Overriding the keys on sbt 2. Because @transient is gone, builds that
    replace smithyBuild, smithy4sCodegen or smithy4sGeneratedSmithyFiles must
    wrap the definition in Def.uncached. Documented in the installation guide,
    including a no-op shim for cross-built sbt 1/2 builds.
  • Modified generated files are restored on sbt 2. Hand-edits to generated
    files are reverted: inside the native root by sbt's output sync, outside it by
    our materialization step. sbt 1 generally keeps such edits. This is intentional
    and documented.
  • Obsolete generated files are not removed automatically (same as before).

Both user-facing changes are in the changelog and the installation docs.

Testing

New sbt-test scripted tests, run on both plugin axes:

  • native-cache (new). Asserts the exact output inventory and the number of
    generator invocations, with sbt 1 and sbt 2 expectations where they differ:
    • cache hit on repeated calls, and after reload;
    • model A -> B -> A (the return to A is a cache hit on sbt 2);
    • generated outputs deleted;
    • destinations moved outside the build; on sbt 2, damaged, deleted and
      same-size-different-content files are repaired with no extra generation
      (the last case reaches the Files.mismatch comparison);
    • unchanged mtimes of external outputs on no-op calls.
  • metadata-cache (new). Covers wildcard-argument changes, deleted/damaged/
    relocated metadata files and unchanged mtimes. On sbt 2 only, it also asserts
    native cache hits through the cache event log, with no generation on a hit.
  • smithy-build and smithy-build-kind-projector: the smithyBuild overrides
    now use Def.uncached (with a build-local sbt 1 no-op shim), as required
    after removing @transient.

Checklist

  • Added unit-tests (for runtime code)
  • Added bootstrapped code + smoke tests (when the rendering logic is modified)
  • Added build-plugins integration tests (when reflection loading is required at codegen-time)
  • Added alloy compliance tests (when simpleRestJson protocol behaviour is expanded/updated)
  • Updated dynamic module to match generated-code behaviour
  • Added documentation
  • Updated changelog

On sbt 2, `smithy4sCodegen` and `smithy4sGeneratedSmithyFiles` now run
as `Def.cachedTask` actions with declared outputs instead of the
hand-rolled `Tracked` input/output stores. `smithyBuild` defaults to
`Def.uncached`. `@transient` is removed from these three keys.

The public tasks keep their `Seq[File]` / `Option[File]` types and
return exactly the configured destinations. They are uncached facades
over private native actions that return `HashedVirtualFileRef`s.

- Outputs outside `rootOutputDirectory` (e.g. a custom
  `smithy4sOutputDir`) are staged under
  `smithy-cache/<project>/<config>/<scalaVersion>/<action>/` and
  materialized at the configured paths. Missing or content-different
  files are restored on a cache hit without regenerating; identical
  files are not rewritten.
- All writes go through `IO.writeFileAtomically` so a destination that
  sbt restored as a symlink into the content-addressed store is
  replaced rather than written through (sbt/sbt#9826).
- sbt 1 behavior is unchanged: `CachedTask` and the legacy task bodies
  moved to `src-2` verbatim.
- sbt 2 minimum is now 2.0.4 (sbt/sbt#9492); documented in the
  installation guide, including the need for `Def.uncached` when
  overriding these keys on sbt 2.
- Scripted coverage: new `native-cache` and `metadata-cache` tests, plus
  extended `extra-configs`, `generate-openapi-specs`, `smithy-build` and
  `wildcard-config` tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant