feat: make conda-pypi mappings configurable#6333
Conversation
…eplace modes, inline mappings and cache-ttl
- conda-pypi-map now accepts false (global disable), and per-channel
values can be a bare string, false, or a table with location,
inline mapping entries, mode = "extend"|"replace" and cache-ttl.
- BREAKING: bare location strings now use the additive extend mode
(overlay over the prefix.dev chain) instead of the exclusive
replace mode. The old behavior is available via mode = "replace".
- conda-pypi-map = {} is soft-deprecated in favor of false and emits
a deprecation warning.
- pixi_core wires the manifest entries into the per-channel mapping
configuration; inline keys are lowercased to match normalized conda
names, and cache-ttl is validated to require an http(s) location.
…{} disable to false
…tion and breaking-change notes
- Move all mapping/purl tests from solve_group_tests.rs into a new
conda_pypi_map_tests.rs integration module.
- Split CondaPypiMapEntry::Map into CondaPypiMapSpec with a dedicated
MappingLocationSpec { location, cache_ttl } so the TTL is structurally
tied to the location source it applies to.
- Clarify in the Disabled doc comments that the offline conda-forge
verbatim fallback still applies when lookups are disabled.
- Deduplicate the offline help text into a shared MAPPING_OFFLINE_HELP
const used by both the prefix.dev and project-defined fetch errors, and
mention pointing at a custom mapping location (with cache-ttl) as an
escape hatch.
- Document why the TTL cache cannot reuse the http-cache middleware
(header-driven freshness, client-global max_ttl, no stale-on-error).
- Docs: add a parselmouth raw-URL pinning recipe (and a note that blob
URLs serve HTML).
- pixi_toml: add a custom_error(message, span) constructor and use it for the conda-pypi-map validation errors. - pixi_core: extract the conda-pypi-map manifest conversion out of workspace/mod.rs into a workspace::conda_pypi_map module with named, unit-testable functions (incl. the channel-membership validation). - pixi_core: classify mapping locations with rattler_lock::UrlOrPath instead of hand-rolled starts_with checks; file:// urls normalize to paths and non-http(s) remote schemes are rejected with a clear error. - pypi_mapping: make the per-record fallback policy explicit with a Fallback enum (PrefixThenVerbatim | Verbatim | None) instead of a mutable suppression flag. - pixi-build-python: dedupe the requirement version conversion into convert_requirement_version, shared by the user-map and service paths. - test: pin that a mapping for one channel no longer suppresses the verbatim fallback for records from other, unmapped channels (online).
- TTL cache: treat a future mtime (clock skew) as age zero instead of
making the cached copy invisible to the freshness check and the stale
fallback; write cache files atomically via tempfile + persist; unit
tests for the age computation.
- pypi-conda-map: an invalid conda name in an override now falls through
to the mapping service instead of silently dropping the dependency.
- Split the offline help text: failures fetching a user-configured
location now suggest checking the URL / adding cache-ttl instead of
the firewall-framed prefix.dev advice; clearer HTTP status error.
- Warn when a mapping location uses plain http://, since a tampered
mapping influences dependency resolution.
- Encode the manifest-mode to MappingMode conversion in a documented
convert_mode function (a From impl is impossible: neither crate
depends on the other, so the orphan rule forces it into pixi_core).
- Error wording: cache-ttl duration errors show example values;
cache-ttl-without-location message no longer implies location must be
a URL; {} deprecation help reworded; stale Disabled doc hedge fixed;
duplicated doc comment removed.
- Docs: warning box now also covers the verbatim-fallback scope change
for unmapped channels; cache-ttl docs state the no-cache hard-failure;
inline-mapping example no longer reuses 'pytorch' as both channel and
package name.
- New tests: mixed-case inline keys, cache-ttl on a local path rejected,
file:// table-form location works (pins UrlOrPath normalization),
Skip entries with markers, vacuous purls assertion fixed, unit tests
for parse_mapping_location/convert_entry; re-documented what the
fresh-cache TTL test actually pins (cache layout + no network).
- typos: reword 'mis-mapped' in the conda/PyPI concepts page. - basedpyright: no implicit string concatenation in the new schema/model.py field descriptions (schema output unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…jager/pixi into feat-additive-conda-pypi-map
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-pypi-map # Conflicts: # Cargo.lock # crates/pixi_manifest/Cargo.toml
|
What is the reason for the addition of the |
|
I've done some user testing and it all seems to work as designed. I've got one question about the I believe there is currently no way accept making an empty map and assigning that as replace. Was this behavior already existing? |
We want to cache the requested mapping and don't fetch it multiple times during the solve - for this reason, we thought that it would be great to allow users to choose how "fresh" the mapping should be for them, so we added this conf option |
@ruben-arts for reference the compressed mapping for conda-forge on GitHub which you might use is about 1MB :) |
Rename the additive channel mode to `mapping-mode = "overlay"` and add a per-channel `same-name-heuristic` switch. The heuristic now defaults to enabled for conda-forge and disabled elsewhere, but can be explicitly enabled for any channel. Treat `conda-pypi-map = false` and `<channel> = false` as hard disables, while preserving the legacy empty-map behavior as a deprecated no-default-lookup conda-forge same-name configuration. Allow mapping-mode-only entries to express empty replacement mappings explicitly. Update schema, docs, and integration coverage for the revised hierarchy, and clean up pypi_mapping resolver naming.
Remote mapping fetches already pass through the http-cache middleware (CacheMode::Default), and the real mapping hosts (prefix.dev, parselmouth on raw.githubusercontent.com) send Cache-Control + ETag. Rely on that entirely instead of a bespoke per-entry cache-ttl plus a hand-rolled mtime/stale-on-error file cache: - Remove the cache-ttl manifest option, its parsing/validation and snapshots. - Collapse the now single-field MappingLocationSpec to a plain String. - Delete the custom project-defined file cache; fetch straight through the cache-aware client. The middleware handles freshness, ETag revalidation, and use-stale-on-error (conditional_fetch serves the cached copy when a refresh fails, unless the response is no-store/must-revalidate). - Drop now-unused deps (humantime, filetime, rattler_digest dev-dep). Verified that a second, independent client reuses the on-disk HTTP cache without any network access (test_remote_mapping_reused_from_cache_offline).
0db381f to
665ed56
Compare
- test_mapping_location asserted same-name=true for a non-conda-forge channel (https://prefix.dev/test-channel) via extend(); the heuristic correctly defaults to false there, so build the expected value explicitly. - Remove the now-unused tempfile dependency from pypi_mapping (flagged by cargo-shear after the custom stale-cache file writer was deleted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
665ed56 to
d539cae
Compare
Description
This PR makes the conda↔PyPI name mapping configurable in both directions:
conda-pypi-mapnow supports per-channel overlay/replacement mappings, inline entries, URL/file locations, hard disables, and an explicit same-name heuristicpixi-build-pythonnow supportspypi-conda-mapoverrides before the PyPI→conda mapping serviceThe forward and reverse mappings intentionally have different shapes:
conda-pypi-mapis per channel and maps conda package names → one or more PyPI names because it decides which installed conda packages satisfy PyPI requirements/PURLs.pypi-conda-mapis a flat map of PyPI package name → one conda package name or false because the build backend converts each Python requirement into at most one conda recipe dependency.Example:
Common configurations:
Concretely this PR:
Adds table entries for
conda-pypi-mapchannels with:locationmappingmapping-mode = "overlay" | "replace"same-name-heuristic = true | falseKeeps bare string entries as shorthand for an additive overlay location.
Supports multiple PyPI names for one conda package (
airflow = ["airflow", "apache-airflow"]).Makes
falseconsistent at its scope:conda-pypi-map = falsedisables all PyPI name derivation globally<channel> = falsedisables all PyPI name derivation for that channelmapping.package = falsemeans that package is explicitly not on PyPIMakes the same-name heuristic explicit:
Preserves the legacy behavior of
conda-pypi-map = {}as a soft-deprecated spelling for “avoid default mapping lookups but keep conda-forge same-name guessing”. The explicit replacement is:Caches URL mapping locations via standard HTTP cache semantics: remote fetches go through the existing http-cache middleware (
CacheMode::Default), which honors the server'sCache-Control/ETag. A freshly fetched mapping is reused on later solves without network access, a stale one is revalidated cheaply, and a previously fetched copy is reused when a refresh fails (unless the server marked the responseno-store/must-revalidate) so offline solves keep working. No bespoke per-entry TTL knob is needed.Adds
pypi-conda-maptopixi-build-python, including target-specific per-key merge behavior.Improves diagnostics for offline/firewalled mapping failures and HTML/GitHub-blob mapping URLs.
Cleans up
pypi_mapping::resolversnaming (PrefixHash,PrefixCompressed,ProjectDefined,SameName).Behavior changes
conda-forge = "mapping.json"conda-forge = { location = "mapping.json", mapping-mode = "replace", same-name-heuristic = false }B = falseto disable it.conda-pypi-map = {}conda-pypi-map = { conda-forge = { mapping-mode = "replace" } }conda-pypi-map = false{ conda-forge = { mapping-mode = "replace" } }if you only wanted no network/default lookups.How Has This Been Tested?
Automated/unit/integration coverage includes:
mapping-mode,same-name-heuristic, false/null/list mapping values, and empty-map deprecationfalsepreventing fallbackconda-pypi-map = {}behaviorpypi-conda-mapoverride/skip/marker/merge behaviorCommands run locally:
Schema files were regenerated from
schema/model.pyand validated withpixi run -e schema test-schema.AI Disclosure
Tools: Claude Code / pi coding agent
Checklist
schema/model.py