Honor the CLI key specification in LMDBDataset - #1516
Merged
aacostadiaz merged 2 commits intoJul 8, 2026
Conversation
`LMDBDataset.__getitem__` hardcoded `KeySpecification.from_defaults()`, so CLI key overrides such as `--total_charge_key` / `--total_spin_key` were silently ignored for `.aselmdb` / `.lmdb` datasets. Because the defaults map `total_charge <- "total_charge"` and `total_spin <- "total_spin"` (keys absent from OMol data, which stores `charge` / `spin`), every system was loaded as a neutral closed-shell singlet regardless of its true charge/spin. Thread `head_config.key_specification` through `load_dataset_for_path` into the `LMDBDataset` constructors, and honor it in `__getitem__` (falling back to the defaults when none is supplied, so existing behavior is unchanged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regression guard for the previous commit: a key spec built from --total_charge_key / --total_spin_key must reach the loaded data (and the defaults still apply when none is passed). Fails on the pre-fix code with total_charge defaulting to 0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aacostadiaz
approved these changes
Jul 8, 2026
aacostadiaz
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the fix and the very thorough write-up!
LGTM! I'll go ahead and merge this now.
ilyes319
pushed a commit
to ACEsuit/mace-foundations
that referenced
this pull request
Jul 12, 2026
The polar-1 medium/large scripts did not pass --total_charge_key / --total_spin_key, so on OMol data every system trained as a neutral closed-shell singlet. They also set --forces_key='forces'; once mace honors CLI key overrides (ACEsuit/mace#1516) that selects an empty array, so forces must read 'REF_forces'. Aligns the key block with mace_omol/mace-omol.sh. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
aacostadiaz
added a commit
that referenced
this pull request
Aug 7, 2026
* permit first block to be non-linear in 'MACE'-type models
* add head name to test parity plots
* remove redundant scatter labels and label test head
* log num of property not sum of weights
* use first-party testing
* doc
* add option to return fukui function in ase calculator
* simplify
* add cube file write
* add quality report to the cube
* add scheduler
* update cuda test
* Adding a MACE-MDP foundation model calculator and supporting cueq=True for AtomicDielectricMACE (#1439)
* Integrate AtomicDielectric CuEq matrix support
* Apply formatting updates
* Fix CI pre-commit issues
* Add MACE-MDP foundation calculator tests
* fix: GatedEquivariantBlock layout wrong in cueq readout path
transpose_mul_ir converts CuEq linear output ir_mul→mul_ir before the
block, so hardcode layout="mul_ir" instead of get_layout(). Fixes
cueq=True/False inconsistency for models with non-scalar MLP_irreps.
Update cueq matrix test to use MLP_irreps with 1o+2e components and
add rotation equivariance checks for dipole and polarizability.
* ci: run unit tests on push to mdp branch
* refactor: remove redundant transpose wrappers from dipole/polar readout blocks
GatedEquivariantBlock natively supports both "mul_ir" and "ir_mul" layouts,
so the transpose sandwich (ir_mul→mul_ir before gate, mul_ir→ir_mul after)
is unnecessary. Use get_layout(cueq_config) directly instead.
* ci: remove mdp branch from push triggers in unittest workflow
* feat: add fine-tuning support for MACE-MDP (AtomicDielectricMACE)
New --finetune_dipoles_polarizabilities flag enables fine-tuning a
pretrained MACE-MDP checkpoint on dipoles and polarizabilities:
- Validates that --model=AtomicDielectricMACE and --foundation_model
are set; forces --loss=dipole_polar
- Loads the MDP checkpoint via the existing foundation_model path and
copies all weights (incl. readouts) with load_foundations()
- After configure_model(), freezes all parameters except model.readouts
- Reuses existing DipolePolarLoss, data pipeline, and training loop
without modification
Also adds tests/test_mdp_finetune.py verifying that non-readout params
are frozen and readout params update after a short fine-tuning run.
* feat: add --finetune_dipoles_polarizabilities flag for MACE-MDP fine-tuning
* fix: accept default --loss=weighted in MDP fine-tuning validation
* fix: robust MDP foundation loading with species remapping and higher-order irreps
- Guard --E0s=foundation/estimated in MDP fine-tuning path (fall back to average)
- Replace load_foundations() call with new load_foundations_mdp() that handles
higher-order irreps (0e+1o+2e) in skip_tp and transfers all angular momentum
channels in symmetric_contractions for all product layers
- Guard scale_shift access in load_foundations_elements against MDP models
* Restore Dynamo import
* Restore Dynamo import
* Fix CPU DDP, broken console entry points, and pre-commit checking (#1511)
- distributed_tools: initialize a gloo process group when --distributed runs
on CPU (previously no process group was ever created outside cuda/xpu, so
CPU DDP could never work)
- run_train: pass DDP device_ids only on cuda (device_ids with CPU modules is
invalid and raised at wrap time)
- polar_density_cube: PEP-604/PEP-585 annotations crashed the entry point on
python 3.9 at import time; add 'from __future__ import annotations'. Also
silence wrong-import-position like the other CLIs (env vars must be set
before imports) and apply current black formatting
- plot_train: matplotlib/pandas are not package dependencies, so the entry
point crashed on a clean install; guard the import and fail main() with an
actionable message instead
- preprocess_data wrapper: fix copy-pasted comment
- pre-commit: update hook revs (pre-commit-hooks v2.5.0->v6.0.0, black
24.4.0->26.5.1, isort 5.13.2->8.0.1)
- gitignore: stop ignoring /scripts (the repo's own scripts/ is tracked),
allow requirements/*.txt, ignore .venv
* Honor the CLI key specification in LMDBDataset (#1516)
* Honor the CLI key specification in LMDBDataset
`LMDBDataset.__getitem__` hardcoded `KeySpecification.from_defaults()`, so
CLI key overrides such as `--total_charge_key` / `--total_spin_key` were
silently ignored for `.aselmdb` / `.lmdb` datasets. Because the defaults map
`total_charge <- "total_charge"` and `total_spin <- "total_spin"` (keys absent
from OMol data, which stores `charge` / `spin`), every system was loaded as a
neutral closed-shell singlet regardless of its true charge/spin.
Thread `head_config.key_specification` through `load_dataset_for_path` into the
`LMDBDataset` constructors, and honor it in `__getitem__` (falling back to the
defaults when none is supplied, so existing behavior is unchanged).
* Add test that LMDBDataset honors the key specification
Regression guard for the previous commit: a key spec built from
--total_charge_key / --total_spin_key must reach the loaded data (and the
defaults still apply when none is passed). Fails on the pre-fix code with
total_charge defaulting to 0.
---------
* Restructure the test suite around capabilities; replace unittest/pre-commit workflows with ci-core + ci-extensions (#1515)
Test suite (601 tests, was 510 in a flat directory):
- tests/ organised by execution requirements: unit/ (fast CPU),
workflows/ (e2e subprocess trainings), backends/ (e3nn<->cueq/oeq parity,
shared harness in backend_parity.py), extensions/{polar,les,torchsim,
schedulefree}/, foundations/ (network), integrations/lammps/, benchmarks/
- capability model in conftest.py: markers (gpu/cueq/oeq/polar/les/torchsim/
schedulefree/network/bin_lammps) probe availability; missing capability
skips locally but FAILS under MACE_REQUIRE_CAPS (CI jobs declare what they
guarantee) and zero-collection of a guaranteed capability is an error --
no more silently-skipped-forever tests (oeq had zero coverage this way)
- directory-derived markers via pytest_collection_modifyitems: a new file
inherits the capability/cost of its folder; --ignore lists are gone
- fixtures consolidated (fitting_configs/pretraining_configs in conftest,
run_mace_train/base_mace_params/availability flags in tests/helpers.py):
~1080 duplicated lines removed; session-scoped trained_tiny_model_path
- test_calculator/test_cueq_oeq split by concern; TestOeq no longer
parametrizes to an empty (invisible) list on CPU hosts
- NEW: LAMMPS integration contract tier (export CLI both formats, TorchScript
reload, real-vs-ghost parity of the lammps branches in blocks.py, virials
path) + DDP CPU smoke (2-process gloo training) + les xfails documenting
two pre-existing breakages found on first-ever run
- pytest-split durations committed (tests/.test_durations), refreshed by CI
CI:
- ci-core.yaml: lint (pre-commit in CHECK mode -- black could never fail
before) -> unit (py3.9-3.13) -> workflows e2e (frontier on PRs, full matrix
on push; split 2 via committed durations) -> informative coverage artifact
- ci-extensions.yaml: one job per optional dep with MACE_REQUIRE_CAPS
(les/schedulefree run in CI for the first time), paths-filtered on PRs,
external git deps pinned in requirements/*.txt
- composite action .github/actions/setup-mace; every job body is a script
under scripts/ci/ (bash scripts/ci/<job>.sh reproduces the job locally);
actions pinned by SHA; permissions/timeouts/concurrency everywhere
* Add GPU fleet, integration, nightly and gated release workflows (#1513)
* Restructure the test suite around capabilities; replace unittest/pre-commit workflows with ci-core + ci-extensions
Test suite (601 tests, was 510 in a flat directory):
- tests/ organised by execution requirements: unit/ (fast CPU),
workflows/ (e2e subprocess trainings), backends/ (e3nn<->cueq/oeq parity,
shared harness in backend_parity.py), extensions/{polar,les,torchsim,
schedulefree}/, foundations/ (network), integrations/lammps/, benchmarks/
- capability model in conftest.py: markers (gpu/cueq/oeq/polar/les/torchsim/
schedulefree/network/bin_lammps) probe availability; missing capability
skips locally but FAILS under MACE_REQUIRE_CAPS (CI jobs declare what they
guarantee) and zero-collection of a guaranteed capability is an error --
no more silently-skipped-forever tests (oeq had zero coverage this way)
- directory-derived markers via pytest_collection_modifyitems: a new file
inherits the capability/cost of its folder; --ignore lists are gone
- fixtures consolidated (fitting_configs/pretraining_configs in conftest,
run_mace_train/base_mace_params/availability flags in tests/helpers.py):
~1080 duplicated lines removed; session-scoped trained_tiny_model_path
- test_calculator/test_cueq_oeq split by concern; TestOeq no longer
parametrizes to an empty (invisible) list on CPU hosts
- NEW: LAMMPS integration contract tier (export CLI both formats, TorchScript
reload, real-vs-ghost parity of the lammps branches in blocks.py, virials
path) + DDP CPU smoke (2-process gloo training) + les xfails documenting
two pre-existing breakages found on first-ever run
- pytest-split durations committed (tests/.test_durations), refreshed by CI
CI:
- ci-core.yaml: lint (pre-commit in CHECK mode -- black could never fail
before) -> unit (py3.9-3.13) -> workflows e2e (frontier on PRs, full matrix
on push; split 2 via committed durations) -> informative coverage artifact
- ci-extensions.yaml: one job per optional dep with MACE_REQUIRE_CAPS
(les/schedulefree run in CI for the first time), paths-filtered on PRs,
external git deps pinned in requirements/*.txt
- composite action .github/actions/setup-mace; every job body is a script
under scripts/ci/ (bash scripts/ci/<job>.sh reproduces the job locally);
actions pinned by SHA; permissions/timeouts/concurrency everywhere
* Add GPU fleet, integration, nightly and gated release workflows
- ci-gpu.yaml: self-hosted GPU testing driven by a declarative fleet file
(.github/gpu-fleet.json). A plan job emits one matrix entry per ENABLED
vendor, so a disabled/absent vendor never queues a job forever. Tests are
selected by marker expression per vendor (nvidia: 'gpu'; amd, pre-wired
and disabled: 'gpu and not cueq' since cuEquivariance is NVIDIA-only;
tests always use device=cuda, valid on ROCm). Replaces the 2 hand-picked
A100 tests with the full gpu selection (~50 tests: cueq/oeq parity incl.
first-ever oeq coverage, torch.compile on GPU, GPU training, calculator
GPU paths), with MACE_REQUIRE_CAPS turning broken backend installs into
failures. Runs on PRs, push to main/develop, nightly cron and dispatch;
superseded runs are cancelled per ref
- ci-integrations.yaml: LAMMPS contract tier on PRs (paths-filtered), no
binary required; template documented in tests/integrations/README.md
- nightly.yaml: foundation-model tests (network opt-in + download cache),
full-matrix e2e workflows, extension refresh (external git deps break on
their own schedule), CPU benchmarks artifact, real-tier LAMMPS via
conda-forge (ML-IAP path, continue-on-error until proven), durations
refresh artifact
- release.yaml: build (twine check) -> wheel smoke on py3.9+py3.13
(scripts/ci/smoke_release.sh: clean-venv install, all 12 console entry
points, mini train/eval/LAMMPS-export) -> publish; previously a tag
published with no checks at all
* ci-gpu: retry pip installs to ride out transient DNS/network blips on the self-hosted host
pip's --retries does not cover DNS failures inside git clone (pinned git deps
in requirements/*.txt); seen on bluesky: 'Could not resolve host: github.com'
90s after a successful checkout.
* Add unit tests for previously uncovered core modules; document the test suite (#1514)
* Restructure the test suite around capabilities; replace unittest/pre-commit workflows with ci-core + ci-extensions
Test suite (601 tests, was 510 in a flat directory):
- tests/ organised by execution requirements: unit/ (fast CPU),
workflows/ (e2e subprocess trainings), backends/ (e3nn<->cueq/oeq parity,
shared harness in backend_parity.py), extensions/{polar,les,torchsim,
schedulefree}/, foundations/ (network), integrations/lammps/, benchmarks/
- capability model in conftest.py: markers (gpu/cueq/oeq/polar/les/torchsim/
schedulefree/network/bin_lammps) probe availability; missing capability
skips locally but FAILS under MACE_REQUIRE_CAPS (CI jobs declare what they
guarantee) and zero-collection of a guaranteed capability is an error --
no more silently-skipped-forever tests (oeq had zero coverage this way)
- directory-derived markers via pytest_collection_modifyitems: a new file
inherits the capability/cost of its folder; --ignore lists are gone
- fixtures consolidated (fitting_configs/pretraining_configs in conftest,
run_mace_train/base_mace_params/availability flags in tests/helpers.py):
~1080 duplicated lines removed; session-scoped trained_tiny_model_path
- test_calculator/test_cueq_oeq split by concern; TestOeq no longer
parametrizes to an empty (invisible) list on CPU hosts
- NEW: LAMMPS integration contract tier (export CLI both formats, TorchScript
reload, real-vs-ghost parity of the lammps branches in blocks.py, virials
path) + DDP CPU smoke (2-process gloo training) + les xfails documenting
two pre-existing breakages found on first-ever run
- pytest-split durations committed (tests/.test_durations), refreshed by CI
CI:
- ci-core.yaml: lint (pre-commit in CHECK mode -- black could never fail
before) -> unit (py3.9-3.13) -> workflows e2e (frontier on PRs, full matrix
on push; split 2 via committed durations) -> informative coverage artifact
- ci-extensions.yaml: one job per optional dep with MACE_REQUIRE_CAPS
(les/schedulefree run in CI for the first time), paths-filtered on PRs,
external git deps pinned in requirements/*.txt
- composite action .github/actions/setup-mace; every job body is a script
under scripts/ci/ (bash scripts/ci/<job>.sh reproduces the job locally);
actions pinned by SHA; permissions/timeouts/concurrency everywhere
* Add GPU fleet, integration, nightly and gated release workflows
- ci-gpu.yaml: self-hosted GPU testing driven by a declarative fleet file
(.github/gpu-fleet.json). A plan job emits one matrix entry per ENABLED
vendor, so a disabled/absent vendor never queues a job forever. Tests are
selected by marker expression per vendor (nvidia: 'gpu'; amd, pre-wired
and disabled: 'gpu and not cueq' since cuEquivariance is NVIDIA-only;
tests always use device=cuda, valid on ROCm). Replaces the 2 hand-picked
A100 tests with the full gpu selection (~50 tests: cueq/oeq parity incl.
first-ever oeq coverage, torch.compile on GPU, GPU training, calculator
GPU paths), with MACE_REQUIRE_CAPS turning broken backend installs into
failures. Runs on PRs, push to main/develop, nightly cron and dispatch;
superseded runs are cancelled per ref
- ci-integrations.yaml: LAMMPS contract tier on PRs (paths-filtered), no
binary required; template documented in tests/integrations/README.md
- nightly.yaml: foundation-model tests (network opt-in + download cache),
full-matrix e2e workflows, extension refresh (external git deps break on
their own schedule), CPU benchmarks artifact, real-tier LAMMPS via
conda-forge (ML-IAP path, continue-on-error until proven), durations
refresh artifact
- release.yaml: build (twine check) -> wheel smoke on py3.9+py3.13
(scripts/ci/smoke_release.sh: clean-venv install, all 12 console entry
points, mini train/eval/LAMMPS-export) -> publish; previously a tag
published with no checks at all
* Add unit tests for previously uncovered core modules; document the test suite
81 new unit tests (all CPU, no network, <10s combined) for the highest-risk
modules that had no direct coverage:
- modules/loss.py: every loss variant against hand-computed closed-form
values on tiny batches (weighted energy/forces/stress/virials/dipole,
Huber regimes, conditional-MSE force-norm regimes, per-config weights)
- tools/arg_parser.py: the ~22 defaults the rest of the code assumes, YAML
--config round-trip with CLI precedence, swa/stage_two alias, invalid
choices, preprocess parser
- tools/multihead_tools.py: HeadConfig, prepare_default_head against the
real parser, dict_head_to_dataclass errors, both prepare_pt_head branches
without network
- tools/finetuning_utils.py: element-subset weight transfer verified tensor
by tensor between two in-situ models, plus a working forward
- data/utils.py: KeySpecification, config_from_atoms key mapping,
load_from_xyz round-trip, compute_average_E0s on an exactly solvable
system
README: document the suite layout, the capability contract
(MACE_REQUIRE_CAPS / MACE_CI_ALLOW_NETWORK) and how to reproduce any CI job
locally via scripts/ci/<job>.sh
* ci-gpu: retry pip installs to ride out transient DNS/network blips on the self-hosted host
pip's --retries does not cover DNS failures inside git clone (pinned git deps
in requirements/*.txt); seen on bluesky: 'Could not resolve host: github.com'
90s after a successful checkout.
* fix(imports): break the mace.data <-> mace.data.utils import cycle (#1521)
mace/data/__init__ imports .atomic_data and .utils, while
mace/data/utils.py closed the loop with a function-level
`from mace.data import AtomicData`. pylint reports this as R0401.
Import AtomicData from its leaf module instead, and keep the
`Configuration` annotation in atomic_data.py behind TYPE_CHECKING
(it is only ever used as an annotation) with `from __future__
import annotations` so the runtime behaviour is unchanged.
* Drop Python 3.9, make 3.12 the CI default (#1520)
* build: drop Python 3.9, make 3.12 the CI default
Python 3.9 reached EOL in October 2025. The concrete cost in this repo was
not the EOL date: torch stopped publishing cp39 wheels after 2.8.0, so the
`unit (py3.9)` job silently resolved to torch <=2.8 while every other job ran
2.9+. That leg was validating a different stack than the one we ship.
- python_requires >=3.10; drop the 3.9 classifier and the README floor.
- unit / workflows / workflows-full matrices: 3.10-3.13.
- PR and release frontiers move to 3.10 + 3.13.
- setup-mace default, the lint job and the release build move to 3.12. The
per-job pins that spread across 3.10/3.11/3.12 are left alone on purpose:
collapsing them onto 3.12 would shrink the set of versions CI exercises
outside the matrices.
- pylint: pin py-version to the floor (3.10) so lint tracks what we support
rather than whichever interpreter the job runs on. Verified message-set
identical between py-version 3.10 and 3.12.
- mypy: python_version 3.8 -> 3.10, which was stale under the old floor too.
- ruff: py38 -> py39, deliberately still below the floor. FA102 is the only
enabled rule and it degrades to a no-op at py310; py39 keeps `X | Y`
flagged because TorchScript cannot parse it.
* refactor: drop the python-3.9 annotation workaround in polar_density_cube
f358b99 added `from __future__ import annotations` to polar_density_cube
because its PEP-604/PEP-585 annotations crashed the entry point at import time
on python 3.9. With the floor at 3.10 that workaround is dead: `str | Path`,
`tuple[...]` and `list[Path]` all evaluate natively. Verified by importing all
12 console-script modules on 3.10 and 3.12.
Left alone on purpose:
- mace/cli/plot_train.py keeps its future import. It is NOT a 3.9 workaround:
matplotlib/pandas are import-guarded and `pd` is None when they are missing,
so `data: pd.DataFrame` would be evaluated at def time and raise
AttributeError — precisely the clean-install crash the guard exists to
prevent.
- tests/helpers.py keeps its `list | None` future import; tests are excluded
from lint/format anyway.
Removing the import made ruff's FA102 demand it back, because ruff.toml was
pinned to py39 globally. That was too broad: the constraint is torch, not
python. TorchScript rejects `X | Y` on torch < 2.2 (and the future import does
not rescue it), so the guard only belongs on code carrying
@compile_mode("script"). Global target-version now states the real floor
(py310) and per-file-target-version keeps py39 on the scripted surface.
Verified with a positive control (planting `int | None` in modules/blocks.py
still trips FA102) and a negative one (the same line in mace/cli does not).
* build: pin pylint
The pre-commit pylint hook uses `language: system`, so its `rev: pylint-2.5.2`
is decorative and CI lints with whatever pylint the `dev` extra happens to
install. A new pylint release can therefore turn any PR red with no code change
— pylint 4 is what surfaced the long-standing `mace.data` import cycle that
currently blocks #1244.
This belongs in this PR rather than a standalone one: pylint 4.0.6 requires
Python >= 3.10.0, so a single-line pin is only satisfiable once the floor moves
to 3.10. On the old 3.9 floor it would need a python_version marker, and the
`unit (py3.9)` / `workflows (py3.9)` jobs — which install `extras: dev` — would
fail dependency resolution outright.
Verified `.[dev]` resolves to pylint==4.0.6 on 3.10, 3.11, 3.12 and 3.13.
* Adding`MagneticMACE` (#1244)
* magnetic model evaluation done
* add calculator and training
* clean ups
* move random rotation to to augmentation.py
* add fitting to magnetic forces and logging, more logistics
* fix data augmentation
* fix pylint
* feat(magnetic): calculator outputs + forward parity + smoke tests
BREAKING: rename the magnetic calculator result key from mace_magmom
to MACE_magmoms. Old .model files are not expected to reload through
the new MagneticMACECalculator surface.
Wire compute_edge_forces and compute_atomic_stresses through
MagneticScaleShiftMACE.forward, mirroring the ScaleShiftMACE output
contract (returns edge_forces, atomic_virials, atomic_stresses).
Remove the stale MagneticSCFMACE data["mace_magmom"] dead write —
nothing downstream consumes that key.
Add tests/test_magmace.py smoke coverage for the MagneticSCFMACE
forward path. test_run_eval_magnetic_mace is currently @pytest.mark.skip
pending an eval_configs.py update to plumb magmom into the data dict
post-merge; tracked as follow-up. test_run_train_magnetic_mace is left
in the file but not exercised by this commit (training is heavy and
out of scope for prep smoke).
Pre-commit-config bump --max-module-lines=1500 -> 2000 already
covered by the merge commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(magnetic): eval_configs.py magmom plumbing + canonicalize REF/MACE keys
Restore magmom support in mace/cli/eval_configs.py so magnetic MACE models
can be evaluated end-to-end:
- Add --magmom_key (default REF_magmom) and --return_magforces.
- Thread KeySpecification with the magmom_key through config_from_atoms
so AtomicData.from_config populates data["magmom"].
- Plumb compute_magforces through get_model_output and write MACE_magforces
to atoms.arrays when --return_magforces is set.
Canonicalize the magmom keys across the package so input and output do not
share a buffer:
- INPUT reference data: REF_magmom (alongside REF_energy/REF_forces)
- OUTPUT model prediction: MACE_magmoms (alongside MACE_energy/MACE_forces)
MagneticMACECalculator default magmom_key changes MACE_magmoms -> REF_magmom.
The test_run_train_magnetic_mace fixture no longer needs to copy
REF_magmom into MACE_magmoms before calling get_potential_energy().
Unskip test_run_eval_magnetic_mace and add --return_magforces=True to its
Namespace. Both test_run_eval_magnetic_mace and test_run_magnetic_scf now
pass in clean-pr-test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(magnetic): honor --mean for MagneticScaleShiftMACE atomic_inter_shift
In _build_model the MagneticScaleShiftMACE branch hardcoded
atomic_inter_shift=[0.0]*len(heads), so --mean was silently ignored for
magnetic models. Replace with _determine_atomic_inter_shift(args.mean,
heads), matching the regular MACE/ScaleShiftMACE branches. The helper
already returns [0.0]*len(heads) when args.mean is None, so the previous
behaviour is preserved for callers that do not pass --mean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(magnetic): round-trip magnetic hparams through extract_config_mace_model
Add MagneticScaleShiftMACE to the whitelist in
mace/tools/scripts_utils.extract_config_mace_model. Round-trip the
magnetic hyperparameters needed to rebuild an identical instance:
- m_max (per-element saturation, registered buffer)
- max_m_ell (max ell of the magmom spherical-harmonics basis)
- num_mag_radial_basis (Chebyshev count in mag_radial_embedding)
- use_magmom_one_body
- num_mag_radial_basis_one_body (only when use_magmom_one_body=True,
read from onebody_magmombasis_coeffs.shape[1])
Adds tests/test_magmace.py::test_extract_config_magnetic_round_trip
covering the keys above. Required upstream of any foundation-FT
work that wants to reconstruct a magnetic foundation from its config.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(magnetic): inherit magnetic hparams from foundation
Add multihead_tools.inherit_magnetic_hyperparameters_from_foundation:
copies m_max, max_m_ell, num_mag_radial_basis, and
num_mag_radial_basis_one_body from the foundation checkpoint onto
args, so users do not have to pass them by hand when fine-tuning a
saved magnetic foundation. Ported from CheukHinHoJerry/mace@482c250.
run_train calls it right after args.r_max = model_foundation.r_max.item().
No-op for non-magnetic foundations (extract_config_mace_model does not
emit m_max etc. for them).
tests/test_magmace.py covers the helper directly with a synthetic
foundation_config fixture.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(magnetic): species-dependent --m_max CLI input
Previously --m_max required a positional list of N floats matching the
order of atomic_numbers — fragile across datasets with different element
coverage. Mirror the --E0s pattern by also accepting a dict literal mapping
atomic number to m_max value:
--m_max '{26: 1.8, 28: 1.2}'
Only listed elements are required; unspecified ones default to 1.0.
scripts_utils.resolve_m_max handles the conversion to a per-element list
ordered by atomic_numbers. It accepts:
* None -> None (caller decides default)
* list[float] -> fast path, validated length
* single dict-literal string token -> ordered by atomic_numbers
* single float-literal string token -> broadcast across all elements
* legacy list of float-literal strings -> parsed and length-validated
argparse now uses type=str, nargs="+" so both the dict form and the
legacy space-separated float form keep working. The resolver is called
in model_script_utils._build_model's MagneticScaleShiftMACE branch right
before the constructor call, using model_config["atomic_numbers"] as the
ordering.
tests/test_magmace.py adds 7 focused unit tests covering all input
shapes and the error cases (wrong length, unknown element).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(magnetic): tolerate --m_max dict keys not in z_table
The previous resolve_m_max raised ValueError on any dict key that wasn't
in atomic_numbers. That breaks the natural use case of passing a generic
over-spec dict (e.g. all Z=1..94) on a dataset whose z_table only covers
a subset — which is what we want for FT-ready foundation training.
Change: silently ignore extra dict keys (log INFO listing them), only
use the dict entries that match the current z_table; missing entries
fall back to `default`.
Also normalize atomic_numbers to python int before set-comparison so
np.int64 z_tables (the common case) work the same as python int.
Tests:
* test_resolve_m_max_unknown_element_raises -> replaced with
test_resolve_m_max_extra_dict_keys_ignored (asserts the new behavior)
* new test_resolve_m_max_numpy_atomic_numbers covers np.int64 zs
Reported from a real MATPES_V2_BADER training run where the dict covered
Z=1..94 but the train data lacked Po (84), At (85), Rn (86), Fr (87), Ra (88).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(magnetic): accept use_edge_irreps_first kwarg in MagneticMACE
Upstream main added `use_edge_irreps_first` to ScaleShiftMACE/MACE
(commit 6dd9134) and the foundation_filtered _build_model branches
pass it through to whatever __init__ they call. MagneticMACE.__init__
on this branch didn't accept it, so MATPES_V2_BADER training failed at
model construction with:
TypeError: MagneticMACE.__init__() got an unexpected keyword argument
'use_edge_irreps_first'
Mirror upstream commit a13910d ("add unused use_edge_irreps_first arg to
models that don't inherit from MACE baseclass"): add the kwarg with a
False default and the pylint:unused-argument silence. The magnetic stack
doesn't have the edge-irreps-first behavior so this is intentionally a
silent ignore.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(magnetic): allow MagneticScaleShiftMACE through cueq/oeq conversion gate
The post-merge run_train.py asserts model.__class__.__name__ is in a
whitelist before calling run_e3nn_to_cueq / run_e3nn_to_oeq. Add
MagneticScaleShiftMACE to both whitelists.
The e3nn->cueq conversion is a generic walk-and-replace over tensor
products and works on the magnetic class as well — the source branch
(magnetic-fix-nonSOC) has been training MAGNETIC models with
--enable_cueq=True for months; it just had the assertion commented out
there. Whitelisting is cleaner than commenting out.
Same change for the oeq path for consistency.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(magnetic): clamp magmom/m_max ratio before Chebyshev transform
The radial magmom transform `1 - 2*(|m|/m_max)**2` assumed
`|m|/m_max ∈ [0, 1]` so the result stays in `[-1, 1]` (where the
Chebyshev basis is numerically stable). It was unclamped, so any sample
with |bader_magmom| > m_max for its element pushed the transform below
-1, where the Chebyshev recurrence is unstable and produced single-step
gradient blowups (loss → 2.85e+176 at step 348 of a MATPES bader run on
2026-05-24, then NaN forever after).
Clamping the ratio to [0, 1] is the targeted fix; the MATPES bader
training started 2026-05-24 10:00 has been running cleanly since.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(magnetic): fall back to False when output_args lacks 'magforces'
The magforces branches in take_step() and evaluate() indexed output_args
with a hard ['magforces'] key, which raised KeyError for any caller that
didn't populate the key (e.g. tests/unit/test_lora.py::
test_lora_evaluate_preserves_frozen_state, which passes only forces/
virials/stress). Use .get('magforces', False) so magforces stays opt-in
and legacy callers keep working.
* fix(magnetic): default args.return_magforces=False in eval_configs.run
The magnetic-pr magforces path required args.return_magforces on the
Namespace, but callers that construct partial Namespaces (e.g. the LES
extension tests: test_run_eval_with_bec, test_run_eval_no_bec) hit
AttributeError. Normalize once at the top of run() so magforces stays
opt-in and legacy callers keep working.
* fix(magnetic): read magmom_mace attrs via getattr fallback in MagneticMACECalculator
MagneticMACECalculator reads model.magmom_mace.<attr> in six places, but
that attribute only exists when the loaded model is a MagneticSCFMACE
wrapper. Training via mace_run saves a raw MagneticScaleShiftMACE, so
loading a trained checkpoint blew up with:
AttributeError: 'MagneticScaleShiftMACE' object has no attribute 'magmom_mace'
Replace the six accesses with getattr(model, 'magmom_mace', model).<attr>
so raw and SCF-wrapped models both work. Rejected the alternative of
wrapping raw models in MagneticSCFMACE(use_scf=False) at load time — that
would push every model into the equilibrated_magmom branch of
calculate() and break committee inference (assert len(self.models) == 1).
* ci(magnetic): add magnetic extension job mirroring polar/les
Bring the magnetic tests into the CI extension pattern introduced in the
develop-side test-suite restructure:
- tests/test_magmace.py -> tests/extensions/magnetic/test_magmace.py.
- requirements/magnetic.txt pins sphericart-torch==2.0.2.
- tests/conftest.py: 'magnetic' capability probe (sphericart) + directory
marker so tests/extensions/magnetic/* auto-carry the marker.
- pyproject.toml registers the 'magnetic' marker.
- .github/workflows/ci-extensions.yaml adds the paths filter entry and
a 'magnetic:' job (python 3.10, extras dev, pip-packages
requirements/magnetic.txt), same shape as les:.
- mace/calculators/__init__.py exposes MagneticMACECalculator (needed by
the test to load a trained magnetic model with magmom_key plumbing).
- The moved test uses MagneticMACECalculator instead of MACECalculator
so magmom is read from atoms.arrays['REF_magmom'].
* fix(magnetic): guard MACE_magmoms writeback in MagneticMACECalculator.calculate
The MACE_magmoms result is only populated when the model output includes
it (guarded set at mace/calculators/mace.py:1236). But the writeback to
atoms.arrays['MACE_magmoms'] at the end of calculate() was unconditional,
so MagneticMACECalculator crashed with KeyError: 'MACE_magmoms' on any
model/inference path that doesn't emit magmoms (surfaced by
tests/extensions/magnetic/test_magmace.py::test_run_train_magnetic_mace
running through the ASE calculator path after training).
Guard the writeback with the same 'MACE_magmoms' in self.results check.
* feat(magnetic): rotate magforces alongside magmom in Random3DRotation
For non-SOC training, augmenting with a random SO(3) rotation of just the
magnetic moments teaches the model the magmom-only rotation invariance
that the equivariant layers don't encode by construction. But magforces
are magmom-space vectors too (magforce_i = -dE/dm_i), so they must
transform under the same rotation to keep training targets consistent
with augmented inputs. Ports the missing line from the
magnetic-fix-nonSOC branch.
* fix(magnetic): unblock legacy checkpoint loading
Two changes so legacy magnetic checkpoints (produced before the one-body
magmom feature landed) load cleanly through the current class layout:
- mace/modules/extensions.py: MagneticScaleShiftMACE.__init__ now uses
kwargs.pop('num_mag_radial_basis_one_body', 0). The value is only
consumed inside the use_magmom_one_body branch, so legacy configs
that don't set it get a harmless default instead of KeyError.
- requirements/magnetic.txt: pin sphericart-torch to 1.0.9. 1.x
registers torch.classes.sphericart_torch.SolidHarmonics via
torch::class_, which pickled magnetic checkpoints reference by that
exact TorchScript class path. 2.x moved to a different API and can't
deserialize the C++ custom-class handle. The magnetic tests pass on
either; the pin only matters when loading legacy weights.
* feat(magnetic): route MagneticScaleShiftMACE through paper-branch load_foundations
The vanilla load_foundations_elements reshapes skip_tp weights and copies
scale_shift assuming a non-magnetic InteractionBlock layout. Magnetic
interaction blocks
(MagneticRealAgnosticSpinOrbitCoupledDensityInteractionBlock and its
residue variant) use magmom_skip_tp and a different skip layout, so the
vanilla path AttributeErrors on 'skip_tp'.
Introduce a small dispatcher:
load_foundations_elements(model, ...) # public entry point
├── if 'Magnetic' in class name: load_foundations_elements_magnetic(...)
└── else: load_foundations_elements_default(...)
- load_foundations_elements_default is the previously-existing function,
renamed but byte-identical in body (default FT path is unchanged).
- load_foundations_elements_magnetic is ported verbatim from the paper
branch (magmace-examples-paper/mace/mace/tools/finetuning_utils.py):
handles magmom_skip_tp weight transfer, per-head readout replication,
and scale_shift copy for magnetic models.
- default_dtype is only meaningful on the default branch (the paper cut
we ported predates that kwarg); the dispatcher forwards it only there.
Unblocks the tests/extensions/magnetic/test_run_train_magnetic_mace path
when starting from a MagneticScaleShiftMACE foundation checkpoint.
* fix(magnetic): route MagneticScaleShiftMACE through per-head atomic_inter_shift
The foundation-loading branch in configure_model computes atomic_inter_shift
from args.mean/heads via _determine_atomic_inter_shift for ScaleShiftMACE
and PolarMACE, and falls back to [0.0] * len(heads) for everything else.
MagneticScaleShiftMACE was in the fallback branch, so magnetic FT inherited
a zero shift regardless of the training data — surfacing as a large,
constant per-atom energy offset at the initial validation step.
Add MagneticScaleShiftMACE to both arms of the condition (args.model and
model_foundation.__class__.__name__) so magnetic FT gets the same
per-head shift computation as ScaleShiftMACE/PolarMACE. Additive change:
other model classes retain their existing behavior.
* build(magnetic): declare torch-geometric as a required magnetic-extension dep
mace/data/augmentation.py's Random3DRotation transform inherits
torch_geometric.transforms.BaseTransform, and create_random_rotation_loader
imports torch_geometric-provided loader utilities. Magnetic training uses
this via --data_aug_magmom=True (rotates magmom + magforces jointly to
teach the non-SOC magmom-only rotation invariance). Without torch-geometric
installed, the CLI hits:
ImportError: torch_geometric is required for DataLoader functionality.
Pin it in requirements/magnetic.txt so the ci-extensions magnetic job (and
anyone running magnetic FT locally) has it available.
* feat(magnetic): add use_collinear flag to MagneticSCFMACE
Ports the collinear-SCF pattern from magnetic-fix/mace/modules/models.py:
when use_collinear=True, the closure zeros the x and y components of the
LBFGS gradient (magmom.grad[:, 0:2] = 0.0), so LBFGS moves magmom only
along z during the SCF. Combined with an initial magmom projected onto the
z-axis (caller's responsibility), this enforces a collinear self-consistent
relaxation without touching any other code path — default remains False.
Callers can now do:
MagneticSCFMACE(model, n_scf_step=20, use_collinear=True)
with everything else — cache_magmom behaviour, energy_history population,
final_output shape — unchanged.
* fix(magnetic): use LinearReadoutBlock for MagneticMACE intermediate readouts
The MagneticMACE constructor's intermediate-iteration branch called
readout_cls(hidden_irreps, output_irreps, cueq_config, oeq_config) — a
4-arg signature that matches LinearReadoutBlock. But readout_cls defaults
to NonLinearReadoutBlock, which takes a different (7-arg) ctor signature.
Any MagneticScaleShiftMACE built with num_interactions > 2 would blow up
during construction. Base MACE handles this exactly the same way we now
do: hard-code LinearReadoutBlock for intermediate layers, only apply
readout_cls on the final layer.
* fix(magnetic): pass --magforces_weight into UniversalLoss
get_loss_fn constructed UniversalLoss for --loss=universal without ever
forwarding args.magforces_weight (default 100.0), silently falling back
to UniversalLoss's own default of 1.0. Same for the stage-two branch and
args.swa_magforces_weight. Wire both through so magnetic training honors
the CLI settings, and mention the weight in the stage-two log line for
symmetry with the others.
* fix(magnetic): preserve DistributedSampler in create_random_rotation_loader
Under --distributed training, train() calls train_sampler.set_epoch(epoch)
on the DistributedSampler that shards the dataset per rank. The wrapped
loader for --data_aug_magmom was building a fresh DataLoader with
shuffle=True and discarding original_loader.sampler entirely, so every
rank iterated the full dataset — duplicating samples and warping effective
batch size / epoch length.
Detect a DistributedSampler on the source loader and pass it through
(omitting shuffle, which DataLoader forbids when a sampler is set).
Non-distributed loaders keep the previous shuffle-based behavior.
* fix(magnetic): safely compare Atoms.info in MagneticMACECalculator.check_state
Raw dict != on Atoms.info raises when a value is a numpy array (arr != arr
returns an array, not a bool). The regular MACECalculator above already
uses an ndarray-skipping _infos_equal helper for exactly this reason.
Port the same nested helper into MagneticMACECalculator so repeated
atoms.get_potential_energy() calls on frames carrying ndarray info values
(e.g. stress, or metadata parsed from extxyz) no longer break during
ASE's cache check — matters e.g. for the SCF-notebook loop.
* chore(lint): disable pylint cyclic-import (package/submodule pattern is intentional)
CI has been failing the lint step since bdb01c2 on:
mace/calculators/lammps_mliap_mace.py:1:0: R0401: Cyclic import
(mace.data -> mace.data.utils) (cyclic-import)
The 'cycle' is between the mace.data package and its own utils submodule
because mace/data/__init__.py re-exports symbols from .utils — a normal
Python packaging pattern, not a real circular dependency. Local pylint
4.0.6 doesn't flag it; a newer version on the CI image does. Silencing
matches how the rest of the noise-class checks (line-too-long,
too-many-locals, duplicate-code, etc.) are already handled.
Add the disable to both the pre-commit hook args (used by CI) and the
pyproject.toml disable list (used by any standalone pylint invocation).
* fix(imports): break mace.data <-> mace.data.utils cycle via leaf-module imports
pylint on the CI image detects a cyclic-import through
mace.data -> mace.data.utils -> mace.tools -> ... -> mace.data.
The middle hop only exists because mace/data/utils.py imports from the
mace.tools package (which pulls in every re-export in its __init__).
Import AtomicNumberTable and DefaultKeys directly from their leaf
modules (mace.tools.utils, mace.tools.default_keys) so mace.data.utils
no longer transitively depends on the mace.tools package init. Zero
behavior change, and the cyclic-import lint disable added in b067ec5
becomes redundant (kept in place — the fix is orthogonal).
* fix(loss): compute magforces loss in DDP branch; declare magforces_weight on FakeBatch
UniversalLoss.forward's DDP branch previously set loss_magforces = 0
unconditionally, silently dropping the magforces term under distributed
training. Mirror the energy/stress DDP pattern (huber_loss with
reduction='none' then reduce_loss) inside a matching 'if pred.magforces
is not None' guard, so the term is contributed in both DDP and non-DDP
paths and each branch stays symmetric with the other loss terms.
tests/unit/test_loss.py::FakeBatch was missing 'magforces_weight' from
its default fields — real Batch objects always declare it alongside
'forces_weight' (see mace/data/atomic_data.py). Add it so
test_universal_loss no longer AttributeErrors on eager attribute access.
* Revert "chore(lint): disable pylint cyclic-import (package/submodule pattern is intentional)"
This reverts commit b067ec505030f37c819af9457a752ba870d7be4b.
* fix(imports): use leaf-module import in mace.data.augmentation
Same pattern as 39fa7ec applied to the other cycle-suspect edge:
replace 'from mace.tools import torch_geometric as tg_mace' with a direct
'from mace.tools.torch_geometric.dataloader import DataLoader' so
mace.data.augmentation no longer transitively pulls in the whole
mace.tools package init.
Behavior unchanged: create_random_rotation_loader used exactly one
symbol from tg_mace (tg_mace.dataloader.DataLoader); the direct import
gives access to the same class, verified by re-running
tests/extensions/magnetic (13/13 pass).
* fix(imports): leaf-module imports in atomic_data & padding_tools
Continue the leaf-module import refactor started in 39fa7ec and f0d5658
so mace/data/atomic_data.py and mace/data/padding_tools.py no longer
transitively pull in mace/tools/__init__.py (which eagerly imports
train, finetuning_utils, etc.).
- atomic_data.py: split the multi-symbol 'from mace.tools import (...)'
into three leaf imports (torch_geometric, torch_tools, utils).
- padding_tools.py: replace 'from mace.tools import torch_geometric'
with 'from mace.tools.torch_geometric.data import Data'.
Local pylint on the whole mace tree no longer detects the
mace.data -> mace.tools -> mace.tools.train cycles (5 of the earlier
warnings drop to 5 within-mace.data package/submodule warnings, which
develop's tree exhibits too and CI accepts there). Behavior unchanged;
tests/extensions/magnetic + tests/unit/test_loss.py + test_data_utils
all pass locally (58 tests).
* mannual import fix
* disable pylint cuclic import
* minor fix
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix one body terms
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* narrow too-many-lines disable to extensions.py
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add rotation + inversion tests for magnetic MACE
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* address theochemtheo review comments
- Factor radial-weight transfer into `_copy_radial_weights` helper and call
it from all three loaders (default / magnetic / mdp). The helper uses
`.data.copy_(...)` in-place so a target registered as `register_buffer`
(BesselBasis/GaussianBasis default `trainable=False`) is no longer
silently promoted to a Parameter. Adds GaussianBasis parallel to
BesselBasis handling.
- Drop dead `.reshape/.flatten/.clone/*num_species_foundations` comment
blocks around magmom_skip_tp, skip_tp, and conv_tp_weights inside
`load_foundations_elements_magnetic`.
- test_magmace.py test_resolve_m_max_numpy_atomic_numbers: drop redundant
`import numpy as np` (already imported at module top).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* move magnetic deps to setup.cfg extras_require
Address @AntObi's review comment on PR #1244: sphericart-torch and
torch-geometric are on PyPI so magnetic can live in
`[options.extras_require]` rather than a separate requirements file
(LES/Polar still need requirements.txt because they point at github URLs).
- setup.cfg: new `magnetic` extra listing sphericart-torch==1.0.9 and
torch-geometric.
- ci-extensions.yaml: magnetic job now uses `extras: dev, magnetic`
instead of `pip-packages: -r requirements/magnetic.txt`; drop the
now-stale `'requirements/magnetic.txt'` path-filter entry (setup.cfg
is already in the shared filter, so extras edits still trigger CI).
- requirements/magnetic.txt: deleted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* codex-pass cleanup: typos, dead code, logging convention, LBFGS closure bug fix
Cleanup pass driven by 3 iterations of codex-style review of the PR
(P1/P2 findings only, style-only nits skipped).
- utils.py: assert magmoms -> ValueError (assert stripped under python -O);
fix "Magnetic momenet must be inputed" -> "moment must be provided".
- extensions.py: rename magmom_lenghts -> magmom_lengths (5 sites); drop
misleading pretraining-hook comment and stale "specicies dependent
transform" comment (species-dep scaling is implemented in forward).
- calculators/mace.py: 3x print -> logging.info to match sibling
MACECalculator; delete commented dev leftovers; use
DefaultKeys.MAGMOM.value for magmom_key default.
- eval_configs.py: use DefaultKeys.MAGMOM.value instead of hard-coded
"REF_magmom" (single source of truth).
- default_keys.py: add MAGMOM/MAGFORCES entries so keydict() picks them
up alongside energy/forces/etc.
- arg_parser.py: fix "agumentation"/"manetic" typos in --data_aug_magmom
help; switch --magmom_key / --magforces_key to DefaultKeys.
- augmentation.py: sample quaternion in fp32 when magmom.dtype is
bf16/fp16 then cast R back, so mixed-precision runs get proper SO(3)
sampling instead of ~256-discrete-angle bf16 rotations; drop the
misleading "hemisphere rotation" docstring (implementation is uniform
SO(3)).
- train.py: collapse take_step.closure duplicated model() branch into a
single call with conditional compute_magforces kwarg; apply the same
pattern to take_step_lbfgs.closure -- the LBFGS closure previously
hardcoded the no-magforces branch, so magforces were silently dropped
during LBFGS steps even when --loss requested them.
- tests/extensions/magnetic/test_magmace.py: drop 3 debug print(...) in
test bodies.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* address aacostadiaz PR review: node_energy units + zeros defaults for magmom/magforces
- scale MagneticMACECalculator.node_energy by energy_units_to_eV
- default magmom/magforces to zeros in atomic_data (match forces/charges)
- gate MagFs metric with filter_nonzero_weight so unlabeled configs
(magforces_weight=0) do not contaminate rel_mae/rel_rmse
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Bump version to 0.3.17
* Fix non-periodic cell in get_neighborhood: extent-based padding + physical cell on partial PBC (#1533)
* Update neighborhood.py
* Fix stress normalization for partially periodic systems (#1509)
get_neighborhood extends the cell along non-periodic directions so that
matscipy can bin atoms, but the extended cell leaked out of the function
and was stored in AtomicData. Stress is normalized by det(cell), so for
slabs the analytic stress was silently scaled down by the ratio of the
artificial extent to the true vacuum extent (~18x in the reported case),
while energies and forces were unaffected.
Keep the extended cell internal to the neighbour-list call and return
the physical cell whenever at least one direction is periodic. Shifts
are unchanged because unit_shifts are zero along non-periodic
directions. Fully non-periodic systems retain the previous behaviour.
Add a finite-difference regression test comparing analytic stress with
the strain derivative of the energy on a slab.
* Add partial-PBC PolarMACE regression tests for the neighborhood cell fix
The combined #1473 + #1509 fix makes get_neighborhood return the physical
cell for partially periodic systems. That cell also becomes AtomicData's
volume/rcell, which drive PolarMACE's k-space electrostatics, so the change
is not stress-only for electrostatic models.
- test_polar_slab_partial_pbc_cell_contract: deterministic guard that a slab
gets the physical cell (not the matscipy blow-up), that volume/rcell derive
from it, that the short-range graph is invariant to the vacuum, and that the
forward is finite on a partial-PBC electrostatic model (a path neither PR
exercised).
- test_polar_slab_electrostatics_converge_with_vacuum: physics check for the
reviewer's concern that the vacuum affects the electrostatic part. On a
dipole-free slab the 3D k-space sum must converge as the vacuum grows;
non-convergence would signal the need for a 2D / slab-corrected sum.
* Loosen cueq/e3nn stress parity tolerance for the smaller non-periodic cell
The extent-based non-periodic cell (from the neighborhood fix) has ~2 orders
smaller volume than the old max(|pos|)*5*cutoff cell, so stress = virial/det(cell)
scales up proportionally. The e3nn<->cueq parity is unchanged, but the absolute
stress bounds (1e-8 float32, 1e-11 float64) were calibrated against the old
inflated cell and now trip at ~2.3e-8. Loosen to 1e-6 / 1e-9; energy and force
bounds are untouched (they don't divide by det(cell)).
* Strengthen neighborhood test coverage (mutation-verified)
Mutation testing showed two gaps in the added tests:
- Nothing guarded the extent-based non-periodic cell (the #1473 OOM fix):
reverting to max(|positions|)*5*cutoff kept every test green. Add
test_nonperiodic_cell_is_extent_based, which asserts the fictitious cell is
origin-independent (same molecule shifted far away gets the same cell) and
tracks the atom extent + cutoff padding. Verified to fail on the old formula.
- test_polar_slab_electrostatics_converge_with_vacuum was documented as if it
exercised the Yeh-Berkowitz slab correction, but its slab is dipole-free and
converges with the correction on or off (verified by toggling
include_pbc_corrections). Reword the docstring to scope it honestly as a
finite/non-diverging smoke check, not a guard of the correction.
* Fix black formatting: drop trailing blank line in neighborhood.py
* Add non-orthogonal partial-PBC neighbour-list regression test
get_neighborhood replaces the non-periodic (vacuum) row of the cell with an
axis-aligned vector for the matscipy binning. This verifies the resulting
neighbour list is still exact on non-orthogonal slabs, where the periodic
vectors keep their true skewed directions. Parametrized over hexagonal,
monoclinic and triclinic in-plane lattices x two cutoffs x three random
configurations (18 cases), compared edge-by-edge (indices + shift vectors)
against a brute-force reference whose image range grows until the edge set
converges (complete regardless of skew).
---------
Co-authored-by: Samuel M. Blau <smblau@lbl.gov>
Co-authored-by: darthjaja6 <darthjaja6@gmail.com>
* Register all parameters in optimizer groups explicitly (#1529)
* Register all trainable parameters in optimizer groups explicitly
get_params_options previously built named parameter groups for the MACE
backbone only; submodules it did not know about (the PolarMACE
electrostatics blocks, ~36% of the weights of the released POLAR-1
models) were silently excluded from the optimizer and stayed at their
initialization during training.
- Register optional submodules by name (radial_embedding,
pair_repulsion_fn, joint_embedding, embedding_readout, les_readouts,
les, lr_source_maps, fukui_source_map, field_dependent_charges_maps,
local_electron_energy, layer_feature_mixer), skipping absent or
parameter-free ones.
- Raise ValueError naming any trainable parameter no group claims, so
future submodules fail loudly at optimizer construction instead of
silently not training.
- Regenerate the hardcoded reference energies of the single-head
foundation finetuning tests (workflows/test_freeze.py,
workflows/test_run_train.py::test_run_train_foundation): the previous
values encoded a removed artifact where the foundation model's
bessel_weights, promoted to an unregistered trainable Parameter by
the old finetuning loader, inflated the global clip_grad_norm_ norm
(22% on the first step) and rescaled every real update. All other
training tests pass unchanged.
- Add tests/unit/test_optimizer_param_groups.py covering every model
class, the trainable-Bessel case, and the unregistered-submodule
raise.
* Address PR review: checkpoint restart warning, LES guard note, extension test coverage.
* lint mace/tools/scripts_utils.py
Co-authored-by: Alejandro Acosta <alejandroacosdi@gmail.com>
---------
Co-authored-by: Alejandro Acosta <alejandroacosdi@gmail.com>
* Run the GPU suite on MPCDF's GitLab runners (#1546)
* Add MPCDF GitLab GPU pipeline driven from GitHub Actions
The MPCDF instance runs GitLab Community Edition, where the built-in
integration that mirrors a GitHub repository and reports statuses back is a
paid feature, and shell executors are forbidden so an Actions runner cannot
live on those machines. Invert the direction instead: a workflow job pushes
the commit under test onto a throwaway branch of a mirror project, starts the
pipeline through the API, waits for it, and mirrors the GitLab job traces into
its own log, since MACE maintainers have no accounts on that instance.
Manual trigger only for now, so the new path cannot add noise to pull requests
before it has proven itself. Verified against the shared runners: the suite
passes on an A30 MIG slice and, for the first time, on an MI210.
* Split the MPCDF checks per vendor and enable OpenEquivariance
The two vendors now drive one single-job pipeline each, selected with
ONLY_VENDOR, so they surface as independent checks: the shared Nvidia runners
are contended enough that a combined check would hold a finished AMD result
behind a Nvidia queue wait of several minutes. That also moves the decision of
whether a red AMD blocks a merge to branch protection, where it belongs, so
the pipeline no longer hides it behind allow_failure.
The Nvidia job moves to a CUDA toolkit image, which is what containers buy us
over the self-hosted host: OpenEquivariance JIT-compiles its kernels and
cannot run without nvcc, so those 24 tests were skipped everywhere until now.
Both jobs also install the polar/les/schedulefree stack and require those
capabilities, so a broken install fails instead of quietly skipping.
* Treat undefined ONLY_VENDOR as null and survive polling blips
GitLab evaluates an undefined variable in a rule as null, and null == "" is
false, so a pipeline started without ONLY_VENDOR matched no job at all and the
API rejected it as empty. Compare against null explicitly so a hand-started
pipeline still runs both vendors.
The wait loop ran a bare curl under 'set -e', so one timeout was enough to
abort a job that had already sat through a queue wait and a full test run.
Retry, and treat an unreadable status as unknown rather than fatal.
* Install rocblas-dev for the AMD OpenEquivariance build
OpenEquivariance includes rocblas/rocblas.h, which the ROCm dev image does not
carry, so every oeq test died at JIT time on a missing header.
Also trims the comments down to what a reader needs to maintain the files.
* Cache only what is small enough to be stored
Caching the venv put the archive in the multi-GB range, which the cache store
rejected outright. A rejected upload does not fail the job, so the cache had
never worked: every run reinstalled and recompiled everything while reporting
green.
Cache the wheels pip has to build instead, plus the JIT-compiled kernels via
TORCH_EXTENSIONS_DIR, which otherwise land in a home directory the cache
cannot reach. Verified over two consecutive runs: the archive uploads, the
second run reuses the OpenEquivariance wheel rather than rebuilding it, and
the job is ~6 minutes shorter.
* Run the MPCDF GPU path on pushes, pull requests and nightly
It runs alongside the self-hosted fleet rather than in front of it: until the
two have been seen agreeing, a disagreement between them is something to
investigate, so this one stays out of branch protection's required checks.
* Retire the self-hosted GPU fleet
The MPCDF path covers everything this did and more: the self-hosted host had
no CUDA toolkit, so OpenEquivariance could never run there, and there was no
AMD hardware at all. Both are covered now.
Carries the gpu-internal/gpu-external setup notes over into the surviving
workflow, since that was the only place they were written down.
* Give fork pull requests GPU coverage
Neither secrets nor repository variables are passed to a workflow triggered by
a fork's pull request, so the gate on MPCDF_GITLAB_PROJECT_ID silently skipped
every job on exactly the contributions whose GPU behaviour is least known.
Switch to pull_request_target, which runs the base branch's copy of the
workflow with the base repository's credentials.
Nothing here executes the code under test — the steps are git and curl, and
the tests run on the GitLab side behind the environment gate — but three
things have to hold for that to stay true, and each of them is silent when
wrong:
- the pipeline definition is read from the base ref rather than the checked-out
tree, so a fork cannot choose what runs on MPCDF's hardware
- the checkout is explicitly refs/pull/N/head, since pull_request_target
otherwise hands over the base branch and the suite passes on the wrong code
- the environment expression compares against pull_request_target; left as
pull_request it sends every fork to gpu-internal and drops the reviewer gate
Also fixes two latent bugs that only the first non-skipped run would have
found: the pipeline definition was committed with a pathspec git had never
seen, which fails outright, and the concurrency group keyed on github.ref,
which under pull_request_target is the base branch and would have had every
open pull request cancel the last.
* Stop the bridge reporting green when it is not
Four ways this could pass while covering less than it claims, all of them
silent:
Network tests were skipped, not run. The retired workflow passed
`allow-network: true`; the GitLab pipeline set no equivalent, and the `network`
capability is never autodetected, so a foundation-model test carrying `gpu` was
collected and then skipped. Export MACE_CI_ALLOW_NETWORK=1 and list `network`
in REQUIRE_CAPS, so dropping it again fails the job rather than shrinking the
suite. One test changes state, on Nvidia only: the other gpu+network test is a
benchmark, which the marker expression already excludes.
Giving up on the wait left the pipeline running. The cancel step fired on
`cancelled()`, which only covers GitHub cancelling the job, so our own deadline
abandoned a pipeline on a contended GPU runner. Cancel on both, and raise the
deadline to cover queue time on top of the pipeline's own 2h limit while
staying under the job timeout — if GitHub kills the job first, the cancel never
runs, which is the case that matters.
Mirroring the logs could fail the job. The job-list request ran under `set -e`
with --fail-with-body, so an API blip turned a green pipeline red. That step
reports a result rather than producing one: retry it, and let it be
continue-on-error. The verdict stays with the report step.
Cleanup was gated on the deployment environment. It deletes a branch and runs
nothing from the pull request, so the gate bought nothing and an approval
nobody gave would leak the throwaway branch. The token is a repository secret,
not an environment one, so removing the gate keeps it reachable.
* Route same-repo pull requests through pull_request
`pull_request_target` reads the workflow from the base branch, so a pull
request that ADDS this workflow gets no GPU check at all — including the one
adding it. The bootstrap case is not a curiosity: it is the only way to see
this pipeline work before it is merged.
Take both triggers and give each the half it can serve. Same-repository pull
requests come in as `pull_request`, which carries secrets and variables
normally and runs the PR's own copy of the workflow. Fork pull requests come in
as `pull_request_target`, which is the only way they see credentials at all.
Both events fire for every pull request, so each job declines the half it is
not responsible for; without that the suite would run twice on runners that are
already contended.
This also shrinks the privileged path to the case that actually needs it: fork
pull requests, which are the ones the environment gate exists for.
Two consequences that would otherwise be silent:
- the pipeline definition is read from the base ref only under
pull_request_target now. A same-repo PR is trusted, and must test the
definition it ships: reading it from the base would mean a change to the GPU
pipeline never exercises itself, and would fail outright against a base that
has no definition yet.
- the concurrency group includes the event name. Both PR triggers fire for the
same pull request, and without it the run that no-ops shares a group with the
run doing the work and cancels it.
* Match the cueq ops to the toolkit, and claim only what runs
Two things the first green run made visible.
The Nvidia job installs cu12 cueq ops next to a cu13 torch. Default PyPI torch
is `2.13.0+cu130` — the toolkit image was right, the extra was not — so pip
resolved a whole second CUDA runtime into the venv, cu12 libraries alongside
torch's cu13 ones. It works today, which is why the job passed and why this is
worth doing before it stops working. Add a `cueq-cuda-13` extra and use it; the
cu13 ops start at 0.7.0, hence the different floor from its cu11/cu12 siblings.
REQUIRE_CAPS promised more than either job can keep. `polar` has no `gpu`-marked
test anywhere, so no marker expression can select one; on AMD, `not cueq` also
removes the only `gpu` `les` test and the only `gpu` `network` test, both of
which carry `cueq` as well. Those entries read as coverage that does not exist.
The guard in tests/conftest.py cannot catch this on our behalf, because it
counts collected tests rather than selected ones — so a required capability
whose tests are all deselected passes it silently. List only what the marker
expression can reach; the polar and les installs stay, so a future `gpu`-marked
test in either needs no change here.
* Retry the installs, as the self-hosted job did
`requirements/polar.txt` and `requirements/les.txt` are git+https pins, and
pip's --retries covers pip's own HTTP requests, not the `git clone` it shells
out to for a VCS requirement: one DNS blip fails the install and the job. The
retired self-hosted workflow wrapped every install in a shell retry for exactly
this reason, with a comment saying so, and dropping it made the replacement more
brittle than the thing it replaced — on runners that are shared rather than
ours, so no less exposed.
Restore the same helper, same…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
LMDBDataset.__getitem__hardcodesKeySpecification.from_defaults()when building each config:So CLI key overrides such as
--total_charge_key/--total_spin_keyare silently ignored for.aselmdb/.lmdbdatasets —load_dataset_for_pathnever passes the parsed key specification toLMDBDataset, and__getitem__would ignore it even if it did.This bites OMol-style data. The defaults map
total_charge <- "total_charge"andtotal_spin <- "total_spin", but OMol stores these underdata["charge"]/data["spin"](surfaced byget_atomsasatoms.info["charge"]/["spin"]). The mapped keys are absent, sototal_charge/total_spinfall back to0.0/1.0— every system is loaded as a neutral closed-shell singlet, regardless of the flags on the command line.Fix
Thread
head_config.key_specificationthroughload_dataset_for_pathinto theLMDBDatasetconstructors, and honor it in__getitem__— falling back toKeySpecification.from_defaults()when none is supplied, so existing behavior is unchanged.LMDBDatasetis the only on-the-fly reader that appliedconfig_from_atomswith a hardcoded spec; HDF5 datasets bake the key mapping in at write time.Affected training recipes (mace-foundations)
mace_omol/mace-omol.shalready passes--total_charge_key="charge" --total_spin_key="spin"and--energy_key=REF_energy --forces_key=REF_forces. Those charge/spin flags are currently dead and start working with this change; its energy/forces keys are unaffected.mace_polar_1/mace-polar-1-{medium,large}.shdo not pass charge/spin keys, and set--forces_key='forces'. Companion PR: mace_polar_1: pass charge/spin keys and use REF_energy/REF_forces mace-foundations#79.develop,--forces_key='forces'is silently ignored and forces fall back to theREF_forcesdefault, so they load correctly by accident. Once overrides are honored (this PR),--forces_key='forces'actively selects the emptyatoms.arrays["forces"]and forces load as zero. The companion PR updates the polar scripts toREF_forces(and adds the charge/spin keys), matchingmace-omol.sh. Verified empirically both ways.Tests
Two regression tests in
tests/test_lmdb_database.py, each writing a one-molecule.aselmdbwithdata={"charge": -1, "spin": 3}(the OMol layout) and asserting a key spec built from--total_charge_key=charge --total_spin_key=spinyieldstotal_charge == -1,total_spin == 3:test_lmdb_dataset_honors_key_specification— constructsLMDBDataset(..., key_specification=...)directly (also checks the no-spec default still gives0 / 1). Guards thelmdb_dataset.pyhalf.test_load_dataset_for_path_forwards_key_specification— callsload_dataset_for_pathwith a minimalSimpleNamespace(head_name=..., key_specification=...). Guards therun_train_utils.pydispatch half.Both fail on the corresponding pre-fix code.
Reproduction
Self-contained (synthesizes a one-molecule
.aselmdb, no external data):Before:
total_charge = 0.0,total_spin = 1.0— After:total_charge = -1.0,total_spin = 3.0.