refactor: move PyPI resolution and installation into the command dispatcher by wolfv · Pull Request #6324 · prefix-dev/pixi

wolfv · 2026-06-10T06:55:37Z

Description

This PR moves the PyPI resolution and installation pipeline out of pixi_core so that it can be reused outside the workspace flow (e.g. by pixi global).

The PyPI logic now lives in crates below the command dispatcher and is driven through the compute engine, following the same architecture as the other pixi_compute_* crates:

pixi_compute_pypi (new): exposes solve_pypi_environment / install_pypi_environment as extension traits on ComputeCtx. Shared resources come from the engine's data store: the UvResolutionContext is registered once per engine, the workspace root comes from the existing RootDir data object, and link options are shared with the conda install path.
pixi_install_pypi: gains the resolve pipeline (moved from pixi_core::lock_file::resolve). The workspace coupling was replaced by a CondaPrefixProvider trait: when uv has to build a source distribution, the provider supplies a conda prefix with a python interpreter and activation env vars on demand. pixi_core implements it on top of the memoized CondaPrefixUpdater.
pixi_uv_reporter (new): the uv progress reporter, extracted from pixi_reporters so lower crates can use it.

Progress reporting is unified with the other solve/install paths: SolvePypiReporter / InstallPypiReporter live in the engine's data store and are implemented by TopLevelProgress. PyPI solves render as rows in the same section as conda solves.

The command dispatcher has no PyPI code or dependencies anymore. A generic CommandDispatcherBuilder::with_engine_data hook lets engine-based crates register their resources and reporters without the dispatcher knowing about them, and ComputeEngine gained a public global_data() accessor.

How Has This Been Tested?

The resolve/install logic itself is unchanged — it was moved and re-wired; the existing unit suites (including the resolve conversion tests, which moved along with the code) and the workspace lock-file update tests cover it.
Verified end-to-end against real conda-forge/pypi.org with a debug build: lock-file update with PyPI dependencies, install, and re-install with --reinstall.

Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas

https://claude.ai/code/session_01S4nY8g4frki9JvtuUnZm85

Add an install-pypi operation to the CommandDispatcher so that PyPI packages can be installed into a conda prefix through the same handle that already drives conda solves and installs. This makes the PyPI install pipeline reusable outside the workspace install path, e.g. for pixi global. - Extract UvReporter/UvReporterOptions into a new pixi_uv_reporter crate. pixi_reporters depends on pixi_command_dispatcher, so the reporter had to move below the dispatcher to let it depend on pixi_install_pypi without a cycle. pixi_reporters re-exports the types so existing users are unaffected. - Add InstallPypiEnvironmentSpec and CommandDispatcher::install_pypi_environment, which wraps PyPIEnvironmentUpdater and derives the wheel link mode from the dispatcher's configured link options. - Switch the workspace install path in pixi_core to build the spec and go through the dispatcher instead of wiring up the updater configs inline.

Complete the PyPI refactor started with install: resolution now also runs through the CommandDispatcher, so both halves of the PyPI story (solve + install) are reachable outside the workspace code, e.g. for pixi global. - Move the resolve pipeline (resolve_pypi, LazyBuildDispatch, CondaResolverProvider) from pixi_core::lock_file::resolve into pixi_install_pypi::resolve. - Decouple it from the workspace with a new CondaPrefixProvider trait: when uv must build an sdist to get metadata, the provider supplies a conda prefix (python interpreter + activation env vars) on demand. pixi_core implements it as WorkspaceCondaPrefixProvider wrapping the memoized CondaPrefixUpdater and environment activation; both the lock-file update path and the satisfiability metadata checks use it. - Move PypiPackageIdentifier into pixi_install_pypi; the satisfiability-specific satisfies() check stays in pixi_core as an extension trait. - Add SolvePypiEnvironmentSpec and CommandDispatcher::solve_pypi_environment, deriving the link mode from the dispatcher's configured link options. - Drop the dead conda_task plumbing: LazyBuildDispatch::conda_task was never assigned, so resolve_pypi now returns just the locked records and the PypiGroupSolved task keeps forwarding None. - Remove the uv-* dependencies pixi_core no longer needs.

rattler_digest and uv-preview were only used by the PyPI resolve code that moved to pixi_install_pypi.

baszalmstra

Can you properly fill out the PR template. I dont really care too much about the technical details, a bit higher level is more useful I think.

On an architectural level, I want to get rid of the command dispatcher crate and replace it with calls on a compute engine. I have been making PRs to split it up into distinct crates. I would prefer to move the entire pypi logic into a separate crate as well, that mostly uses the compute engine instead of the command dispatcher.

I also commented about the reporter. The pypi reporter is currently still using an "old" system, its not handled by the same unified reporter infrastructure. I think this would be the perfect opportunity to unify that as well.

baszalmstra · 2026-06-12T07:56:33Z

+    /// The directory against which relative paths in the records (e.g. local
+    /// wheels or editable installs) are resolved. For workspaces this is the
+    /// directory that holds the lock file.
+    pub lock_file_dir: PathBuf,


I believe there is a compute engine key for the workspace root that you can use for this.

baszalmstra · 2026-06-12T07:58:14Z

+    pub ignored_extraneous: HashSet<uv_normalize::PackageName>,
+
+    /// The shared uv context (cache, concurrency, http settings) to use.
+    pub uv_context: UvResolutionContext,


Maybe we can insert this as a data object into the compute engine? Or is this specific to this request?

baszalmstra · 2026-06-12T07:59:17Z

+    /// The directory against which relative paths (e.g. local wheels or
+    /// editable installs) are resolved. For workspaces this is the directory
+    /// that holds the lock file.
+    pub project_root: PathBuf,


This is a key in the compute engine I believe.

baszalmstra · 2026-06-12T07:59:43Z

+    pub build_dispatch_cache: Arc<LazyBuildDispatchDependencies>,
+
+    /// The shared uv context (cache, concurrency, http settings) to use.
+    pub uv_context: UvResolutionContext,


Is this shared? Then maybe create it as a data object on the compute engine.

baszalmstra · 2026-06-12T08:00:20Z

+
+    /// Progress bar to report resolution progress on. A hidden bar is used
+    /// when not provided.
+    pub progress_bar: Option<ProgressBar>,


Ideally we use the same mechanism for progress as the other solve/install things! We can then unify that part as well!

The progress bar before was indeed ugly and broken, so this is probably a good change :)

baszalmstra · 2026-06-12T08:04:08Z

 use uv_configuration::initialize_rayon_once;
-// Re-export the uv_reporter types for external use
-pub use uv_reporter::{UvReporter, UvReporterOptions};
+// Re-export the uv reporter types for external use; they live in their own


This comment is not really needed.

tdejager

just a small question

tdejager · 2026-06-12T08:25:09Z

        .map_err(|e| miette::miette!("{}", e));
-    let lazy_build_dispatch = LazyBuildDispatch::new(
-        build_params,
+    let prefix_provider = WorkspaceCondaPrefixProvider::new(


Is the LazyBuilDispatch and the WorkspaceCondaPrefixProvider now the same thing?

…e engine Address review feedback: the PyPI solve and install operations no longer live in (or depend on) the command dispatcher. A new pixi_compute_pypi crate exposes them as extension traits on ComputeCtx, following the pattern of the other pixi_compute_* crates: - SolvePypiEnvironmentSpec / InstallPypiEnvironmentSpec move to the new crate. The specs no longer carry the workspace root (read from the engine's RootDir data object), the UvResolutionContext (registered once as an engine data object by the workspace), or a progress bar. - Progress reporting is unified with the other solve/install paths: SolvePypiReporter and InstallPypiReporter are registered in the engine's data store and implemented by TopLevelProgress. PyPI solves now render as rows in the same solve section as conda solves, and the detailed uv progress is created through the reporter instead of directly inside pixi_install_pypi. - The command dispatcher loses all PyPI dependencies; a generic CommandDispatcherBuilder::with_engine_data hook lets crates built on the compute engine register their shared resources without the dispatcher knowing about them. ComputeEngine gains a public global_data() accessor. - AllowLinkOptions moves to pixi_utils so both the conda and PyPI install paths can read the same data-store value. - The satisfiability metadata checks reuse the engine-wide uv context instead of constructing their own, and the per-solve build-dispatch cache is created per resolve (it only ever cached state for repeated solves of the same environment; the satisfiability path keeps its own cache).

Several jobs of the previous run were cancelled when the cargo-shear fix was pushed mid-run.

Registering the UvResolutionContext eagerly forced the reqwest client (and with it the system certificate store) on every command dispatcher construction, slowing every pixi invocation and pushing several CI jobs past their timeouts. The data-store object is now a UvResolutionContextSource that builds the context on first use and memoizes it for the engine's lifetime.

Several jobs of the previous runs were cancelled by hung artifact downloads on the CI infrastructure.

wolfv · 2026-06-16T14:39:40Z

@copilot resolve the merge conflicts in this pull request

Copilot · 2026-06-16T14:52:29Z

@copilot resolve the merge conflicts in this pull request

Done — I resolved the merge conflicts by merging origin/main into this branch in commit 34fc7ba.

claude added 3 commits June 10, 2026 06:00

chore: remove unused dependencies flagged by cargo-shear

247b0d6

rattler_digest and uv-preview were only used by the PyPI resolve code that moved to pixi_install_pypi.

wolfv changed the title ~~Refactor PyPI resolution and installation into command dispatcher~~ refactor: move PyPI resolution and installation into the command dispatcher Jun 10, 2026

wolfv force-pushed the claude/amazing-davinci-2k2j16 branch from b09716a to 247b0d6 Compare June 10, 2026 14:25

wolfv mentioned this pull request Jun 10, 2026

feat: add PyPI dependencies to pixi global environments #6334

Open

wolfv requested a review from tdejager June 12, 2026 07:47

tdejager added the test:extra_slow Run the extra slow tests label Jun 12, 2026

baszalmstra requested changes Jun 12, 2026

View reviewed changes

tdejager reviewed Jun 12, 2026

View reviewed changes

claude added 5 commits June 12, 2026 10:52

chore: remove unused dependencies flagged by cargo-shear

2d109e4

chore: retrigger CI

b892483

Several jobs of the previous run were cancelled when the cargo-shear fix was pushed mid-run.

chore: retrigger CI

6252ec4

Several jobs of the previous runs were cancelled by hung artifact downloads on the CI infrastructure.

Copilot started work on behalf of wolfv June 16, 2026 14:40 View session

Merge origin/main into claude/amazing-davinci-2k2j16

34fc7ba

Copilot finished work on behalf of wolfv June 16, 2026 14:53

Merge branch 'main' into claude/amazing-davinci-2k2j16

8c7eb1f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: move PyPI resolution and installation into the command dispatcher#6324

refactor: move PyPI resolution and installation into the command dispatcher#6324
wolfv wants to merge 10 commits into
mainfrom
claude/amazing-davinci-2k2j16

wolfv commented Jun 10, 2026 •

edited

Loading

Uh oh!

baszalmstra left a comment

Uh oh!

baszalmstra Jun 12, 2026

Uh oh!

baszalmstra Jun 12, 2026

Uh oh!

baszalmstra Jun 12, 2026

Uh oh!

baszalmstra Jun 12, 2026

Uh oh!

baszalmstra Jun 12, 2026

Uh oh!

wolfv Jun 12, 2026

Uh oh!

baszalmstra Jun 12, 2026

Uh oh!

tdejager left a comment

Uh oh!

tdejager Jun 12, 2026

Uh oh!

wolfv commented Jun 16, 2026

Uh oh!

Copilot AI commented Jun 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

wolfv commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Checklist:

Uh oh!

baszalmstra left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdejager left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wolfv commented Jun 16, 2026

Uh oh!

Copilot AI commented Jun 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wolfv commented Jun 10, 2026 •

edited

Loading