Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,26 @@ defaults:
shell: bash -euo pipefail {0}

jobs:
quality:
name: Lint, types, and docs
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
filter: blob:none
fetch-depth: 0
- uses: astral-sh/setup-uv@v7
with:
cache-dependency-glob: |
pyproject.toml
hatch.toml
- name: Run pre-commit
env:
SKIP: no-commit-to-branch
run: uvx hatch run lint:check
- name: Build documentation
run: uvx hatch run docs:build

build:
name: Build
runs-on: ubuntu-latest
Expand Down Expand Up @@ -113,6 +133,7 @@ jobs:
name: All CI Green
if: always()
needs:
- quality
- build
- get-test-environments
- test
Expand Down
34 changes: 9 additions & 25 deletions .github/workflows/python-publish.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,3 @@
# This workflow will upload a Python Package to PyPI when a release is created
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries

# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

name: Upload Python Package

on:
Expand All @@ -20,42 +12,34 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
- uses: actions/checkout@v6
with:
python-version: "3.x"
fetch-depth: 0

- uses: astral-sh/setup-uv@v7

- name: Build release distributions
- name: Build and validate release distributions
run: |
# NOTE: put your own distribution build steps here.
python -m pip install build
python -m build
uv build
uvx twine check --strict dist/*

- name: Upload distributions
uses: actions/upload-artifact@v4
with:
name: release-dists
path: dist/
if-no-files-found: error

pypi-publish:
runs-on: ubuntu-latest
needs:
- release-build
permissions:
# IMPORTANT: this permission is mandatory for trusted publishing
id-token: write

# Dedicated environments with protections for publishing are strongly recommended.
# For more information, see: https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#deployment-protection-rules
environment:
name: pypi
# OPTIONAL: uncomment and update to include your PyPI project URL in the deployment status:
# url: https://pypi.org/p/YOURPROJECT
#
# ALTERNATIVE: if your GitHub Release name is the PyPI project version string
# ALTERNATIVE: exactly, uncomment the following line instead:
# url: https://pypi.org/project/YOURPROJECT/${{ github.event.release.name }}
url: https://pypi.org/p/scverse-backends

steps:
- name: Retrieve release distributions
Expand Down
31 changes: 21 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,37 @@

> ⚠️ **Under active development.** APIs may shift.

The default plugin & dispatch mechanism for [scverse](https://scverse.org).
Any host library decorates its public functions with `@backend_dispatch`; any
backend — GPU, distributed, JAX, PyTorch, anything
— plugs in via a Python entrypoint and gets picked up automatically.
The default plugin and dispatch mechanism for [scverse](https://scverse.org).
Host libraries mark public functions with `@backend_dispatch` or replaceable
classes with `@backend_class`. GPU, distributed, JAX, PyTorch, and other
backends plug in through Python entry points and are discovered automatically.

Want to add a PyTorch backend, a JAX backend, your own custom one?
**You don't need a PR against the host.** Ship a package that exposes
a module or object with `name`, `aliases`, and host-named callables,
a module or object with `name`, `aliases`, and host-named functions or classes,
register it as an entry point, and users install it next to the host.
That's the whole contract.

## Install

```console
pip install scverse-backends
```

## At a glance

```python
import example_host as eh

# One function call
eh.some_function(data, backend="accelerated")

# A complete backend-provided class
model = eh.SomeModel(data, backend="accelerated")

# A scoped default for functions and classes
with eh.settings.use_backend("accelerated"):
eh.some_function(data)
model = eh.SomeModel(data)
```

## Status
Expand All @@ -33,7 +48,3 @@ with eh.settings.use_backend("accelerated"):
## Docs

Full docs at [scverse-backends.readthedocs.io](https://scverse-backends.readthedocs.io/en/latest/).

## License

MIT.
11 changes: 11 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

## Public API

`BackendDispatcher` is the host-owned entry point. A host normally re-exports
these members:

| member | purpose |
| --- | --- |
| `backend_dispatch` | decorate a module-level function |
| `backend_class` | decorate a completely replaceable class |
| `settings` | select a default backend globally or in a context |
| `get_backend(name)` | retrieve a discovered adapter |
| `available_backend_names()` | list registered canonical names and aliases |

```{eval-rst}
.. currentmodule:: scverse_backends

Expand Down
24 changes: 14 additions & 10 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,11 @@ This repository is under active development. APIs may shift before the
first stable release.
```

**The default plugin & dispatch mechanism for [scverse](https://scverse.org).**
Any host library decorates its public functions with `@backend_dispatch`; any
backend — GPU, distributed, JAX, PyTorch, your own — plugs in via a
Python entrypoint and gets picked up automatically.
**The default plugin and dispatch mechanism for
[scverse](https://scverse.org).** Host libraries mark public functions with
`@backend_dispatch` and replaceable classes with `@backend_class`. GPU,
distributed, JAX, PyTorch, and other backends plug in through Python entry
points and are discovered automatically.

## At a glance

Expand All @@ -22,7 +23,10 @@ import example_host as eh
# Per-call backend
eh.compute_score(data, method="fast", backend="cuda")

# Global
# Complete class replacement
model = eh.Neighborhood(data, backend="cuda")

# Current context
eh.settings.backend = "cuda"
eh.compute_score(data, method="fast")

Expand All @@ -33,11 +37,10 @@ with eh.settings.use_backend("cuda"):

## Want to add a backend?

**No PR against the host needed.** Ship a package that exposes a module
or object with `name`, `aliases`, and callables named after the host
functions you implement, register it under the host's entrypoint group,
and users install it next to the host. That's the entire contract — see
{doc}`usage/backend`.
**No PR against the host needed.** Ship a package that exposes a module or
object with `name`, `aliases`, and functions or classes named after the host
APIs you implement. Register it under the host's entrypoint group, and users
install it next to the host. See {doc}`usage/backend`.

This is the path for a PyTorch backend, a JAX backend, a Dask backend,
or anything else. The host library doesn't need to know you exist.
Expand All @@ -58,6 +61,7 @@ to.
usage/host
usage/backend
usage/conformance
release-notes
api
```

Expand Down
34 changes: 34 additions & 0 deletions docs/release-notes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Release notes

## 0.0.3 — 2026-07-23

### Whole-class dispatch

- Added `BackendDispatcher.backend_class` for APIs whose backend replaces a
complete class.
- Class construction supports the active setting and a per-instance
`backend=` override, with CPU fallback when an adapter does not implement the
class.
- Host signatures, documentation, custom metaclasses, class APIs, and normal
subclass construction are preserved.
- Backend adapter exports are validated as classes, and recursive
self-registration is rejected with a clear error.

### Reliability and packaging

- Invalid dispatcher identity and trusted-provider configuration now fail
early with actionable errors.
- One broken adapter registration no longer prevents other entry points from
being discovered.
- The conformance runner raises an explicit `ValueError` when a backend cannot
be resolved or a requested function filter is invalid.
- Function dispatch rejects unsupported methods and variadic positional host
signatures instead of silently misrouting values.
- Positional-only backend calls preserve omitted defaults, and host-only
default comparisons no longer assume scalar equality.
- Runtime type-hint introspection now resolves every public annotation.
- Discovery can be retried after metadata or signature-update failures, and
adapter metadata failures cannot leave a partially registered backend.
- Installed trusted backends report host-configured aliases consistently.
- CI and publishing configuration now exercise the same lint, documentation,
coverage, build, and metadata checks used for release validation.
76 changes: 67 additions & 9 deletions docs/usage/backend.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Plugging in a backend

A *backend* is a Python package that provides alternative implementations
for one or more functions in a host library. Common examples are GPU,
A *backend* is a Python package that provides alternative implementations for
one or more functions or classes in a host library. Common examples are GPU,
distributed, JAX, or PyTorch implementations.

The backend contract is intentionally small: **no inheritance, no
`scverse-backends` import, no base class**. A backend exposes a module or
object with metadata and callables; the host discovers it through Python
entry points.
object with metadata and same-named implementations; the host discovers it
through Python entry points.

## Recommended layout

Expand All @@ -32,18 +32,18 @@ An adapter module needs three things:
1. `name` — the canonical backend name.
2. `aliases` — optional concrete names users can pass to
`settings.backend = ...` or `backend=...`.
3. Callables named after the host functions the backend implements.
3. Callables or classes named after the host APIs the backend implements.

```python
# my_backend/_backends/example_host.py
from __future__ import annotations

from my_backend.example_host_impl import compute_score, embed, summarize
from my_backend.example_host_impl import Neighborhood, compute_score, embed

name = "my_backend"
aliases = ["mine", "cuda"]

__all__ = ["compute_score", "embed", "summarize"]
__all__ = ["Neighborhood", "compute_score", "embed"]
```

If the implementations live across several backend packages, gather them
Expand Down Expand Up @@ -139,13 +139,25 @@ def __getattr__(attr_name: str):
```

This lets the host inspect the adapter metadata without importing CUDA,
JAX, or another heavy runtime during normal host import.
JAX, or another heavy runtime during normal host import. On backend discovery,
the host may inspect every matching decorated API to merge signatures and
documentation, so `__getattr__` can load more than the one function the user is
about to call. If per-function loading matters, export lightweight wrappers
with explicit public signatures and import the heavy implementation inside
each wrapper body.

## Function signatures

Backend callables should use the same names as host functions for shared
parameters. Any extra public keyword-capable parameters are treated as
backend-only parameters.
backend-only parameters. Both host and backend callables must expose signatures
that Python's `inspect.signature` can read; wrap extension callables or provide
an explicit `__signature__` when necessary.

The host function cannot use variadic positional parameters (`*args`);
dispatch is name-based. A backend callable may accept `**kwargs`, but explicit
named parameters provide better signatures, routing, and generated
documentation.

```python
def compute_score(
Expand Down Expand Up @@ -199,6 +211,50 @@ batch_size (my_backend)
Private parameters such as `_internal` are not injected into host docs.
Use public keyword-only parameters for user-facing backend options.

## Complete class implementations

When a host uses `@backend_class`, export a class with exactly the same name
from the adapter:

```python
# example_host
@backend_class
class Neighborhood:
def __init__(self, data, *, n_neighbors=15):
...
```

```python
# my_backend/_backends/example_host.py
from my_backend.example_host_impl import Neighborhood

name = "my_backend"
aliases = ["mine", "cuda"]

__all__ = ["Neighborhood"]
```

The backend class is a complete replacement and does not need to inherit from
the host class. Keep its public constructor compatible with the host
constructor:

```python
# my_backend/example_host_impl.py
class Neighborhood:
def __init__(self, data, *, n_neighbors=15):
self.data = move_to_device(data)
self.n_neighbors = n_neighbors
```

At construction time, `scverse-backends` removes the `backend` selector and
passes every other argument directly to this constructor. Backend-only
constructor parameters are not merged into the decorated host signature or
documentation, so shared constructor contracts are strongly recommended.

If an adapter does not export the class, the host class is used as a fallback.
If it exports that name as something other than a class, construction raises a
`TypeError` with the backend and class names.

## Trusted backends

Trust is owned by the host. A backend cannot make itself trusted by
Expand Down Expand Up @@ -253,4 +309,6 @@ my_backend = "my_backend._backends.example_host"
- Keep adapter imports cheap, or use `__getattr__` for lazy loading.
- Match host function names and shared parameter names.
- Put backend-only option docs in the backend function's numpydoc.
- Export same-named classes for hosts using `@backend_class`, and keep their
constructor contracts compatible.
- Consider running host feedback tests in backend CI.
4 changes: 4 additions & 0 deletions docs/usage/conformance.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,10 @@ hardware-specific edge cases.
default, re-raises the first failure so pytest shows the useful
traceback.

Passing `functions=[...]` restricts the run to named host checks. Unknown
function names and malformed filters raise `ValueError`, so a typo cannot
accidentally look like a successful empty run.

The runner intentionally does not import the host, NumPy, CuPy, or any
backend runtime. All scientific setup and assertions remain inside host-
or backend-owned tests.
Loading