perf: memoize subprocess results across documents - #80
Draft
schloerke wants to merge 1 commit into
Draft
Conversation
The filter re-runs identical subprocesses for every document in a project render (extension info, base-htmldeps, language-resources, plus two per code block). Memoize their results to .quarto/shinylive-cache/ so a full-site render pays each unique call once. Cache keys embed the live version of the tool that produced the result (py/r shinylive package version, and the quarto version for the codeblock-to-json call), so stale reuse is impossible; the version discovery calls themselves are never cached. Cached results whose referenced files no longer exist are re-run, so the asset download that base-htmldeps performs as a side effect can never be skipped when needed. Any cache failure falls back to running the real subprocess. Fixes quarto-ext#79
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #79 — see the issue for full measurements on a 520-page site (CI build 47.5 → 38.1 min; local warm re-renders skip every shinylive subprocess call).
What this does
Adds a
cachedPipedrop-in forpandoc.pipethat memoizes subprocess results to.quarto/shinylive-cache/(one content-addressed file per result, atomic tmp-file + rename writes), so a full-project render pays each unique subprocess call once instead of once per document.Design
--versionruns (per document, never cached), the R path afterextension infoestablishesversions.r, and thecodeblock-to-jsoncall is keyed on both the language package version andquarto.version. Every cache key embeds those versions, so upgrading shinylive or quarto invalidates naturally; a stale hit is impossible.extension base-htmldepsdownloads the shinylive web assets as a side effect; a naive cache elides that and emits HTML referencing files that don't exist (we hit this on a fresh CI runner). Cached results are only served if every"source"/"path"file they reference still exists — otherwise the subprocess re-runs and re-creates them.pandoc.pipe. Failed subprocess calls are never cached (errors propagate exactly as before). Projects without a project directory get plain passthrough.Testing
On a minimal one-page test site (quarto 1.8.25, py-shinylive 0.8.9):
Deployed at scale (a slightly different site-specific variant) on shiny.posit.co/py: posit-dev/py-shiny-site#382 — 520 pages, byte-identical output verified, CI + local measurements in the issue.
Opening as a draft for design feedback — happy to adjust key layout, cache location, or gate it behind an option.