ceph-testcontainers-go

Testcontainers for Go modules that run a real, disposable Ceph cluster for integration tests. The cluster runs the official Ceph daemons in containers: one MON, one MGR and two OSDs by default, plus RBD pools, CephFS filesystems with MDS daemons, and RGW gateways when you ask for them. Tests prepare and change the server side through typed Go APIs. Your application talks to the cluster with its own client: go-ceph, librados, librbd, libcephfs, the Ceph CLI or an S3 SDK.

The modules control Ceph through the CLIs inside the containers. They have no go-ceph, librados or cgo dependency and build with CGO_ENABLED=0. The Ceph daemons need no privileged mode, host disks, LVM or systemd: each OSD stores its data in a sparse BlueStore file inside its container by default. MemStore and memory-backed BlueStore are also available for faster disposable fixtures.

Releases follow semantic versioning. Before v1.0.0, a minor release may change the API without aliases for the old names. Final release notes are available with the published releases.

Documentation for a published release is available at its Git tag.

Packages

• ceph: the cluster. MON, MGR and OSD topology, pools, Cephx identities, configuration and health.
  • ceph/diagnostics: read-only diagnostics reports, crash metadata and secret redaction for a cluster.
  • ceph/expect: pure evaluations of captured observations against a test's stated conditions, with mismatch explanations.
  • ceph/finalize: one ordered cleanup path with post-test checks, log and crash verification, a failure archive and Terminate.
• rbd: RBD pools and namespaces, image watchers and locks, backup and restore.
  • rbd/mirror: rbd-mirror replication between two clusters.
    • rbd/mirror/expect: pure evaluations of captured receiver and image replay observations, with mismatch explanations.
• cephfs: filesystems, MDS topology and data pools.
  • cephfs/expect: pure evaluations of captured MDS rank and standby observations, with mismatch explanations.
  • cephfs/subvolume: subvolume groups and subvolumes, snapshots and clones, subvolume principals and pins.
  • cephfs/quiesce: quiesce sets over owned subvolumes.
  • cephfs/session: client sessions and session timeout overrides.
  • cephfs/snapschedule: snapshot schedules and their retention.
  • cephfs/mirror: snapshot mirroring between two clusters.
• rgw: S3 and Swift gateways, users, tenants and quotas.
  • rgw/account: accounts, their quotas and root users.
  • rgw/placement: placement targets, storage classes and user default placement.
  • rgw/notification: persistent bucket notification queues of SUT-created topics.
  • rgw/multisite: realms, zonegroups and zones, period transfer, sync status and sync policy.
• dashboard: the Ceph Dashboard of a cluster whose managers ship it, with an owned HTTPS listener and account, and the Object Gateway pages of one owned gateway.
• fault: container pause, CPU limits, network delay, loss, rate, packet faults and endpoint interruption, zone-boundary profiles and damaged replicas on existing fixtures. fault/blockio prepares an owned disposable OSD main device for bounded delay and EIO faults.
• override: reversible cluster-wide changes with restoration handles (blocklist, full ratios, MGR modules, OSD flags, debug levels, central configuration, messenger policy).
• capture: opt-in packet capture on one container endpoint.

Every fixture package is a Testcontainers for Go module with one Run(ctx, image, opts...). In ceph and the service packages it returns a Container that embeds testcontainers.Container; the service packages build on ceph: their Container embeds *ceph.Container, and their options can be passed to ceph.Run together. The two-cluster fixtures start their daemons as Testcontainers containers on two existing clusters: rbd/mirror and cephfs/mirror return a *Container that owns the whole daemon cohort and embeds the first mirror daemon's testcontainers.Container, and rgw/multisite returns a *Fixture whose Source and Destination gateways are containers, while the Fixture itself embeds none. fault, override, capture, ceph/diagnostics and ceph/finalize start no fixture of their own; they act on or observe containers that already exist.

Requirements

• Go 1.26 or later; use a patched toolchain. The repository selects Go 1.26.10 for its own checks. A consumer must select its own toolchain; the toolchain line of this repository does not set it.
• A Docker engine that runs Linux containers. Docker Desktop works. Host networking has extra requirements, described in the host network guide.
• Access to pull the Ceph image and the Ryuk image.

The default image is the published, pruned official Tentacle 20.2.4 aggregate, pinned to its AMD64/ARM64 index digest as ceph.DefaultImage. Explicit ceph.ImageSet selections use essential + client for ordinary RADOS/RBD and add CephFS, mirror/cache or RGW images only for those features. Runtime CI uses these dependency roles directly. Umbrella 21.2.x and Squid 19.2.x are supported too, with the differences listed in the releases guide.

Installation

go get github.com/JSYoo5B/ceph-testcontainers-go/ceph@latest

The repository is one Go module, so this makes every package available, rbd, cephfs and rgw included.

Quick start

package app_test

import (
	"testing"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func TestWithCeph(t *testing.T) {
	ctx := t.Context()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(ceph.PoolConfig{Name: "app"})),
	)
	testcontainers.CleanupContainer(t, cluster)
	if err != nil {
		t.Fatal(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		t.Fatal(err)
	}
	if err := observer.WaitForClean(ctx); err != nil {
		t.Fatal(err)
	}

	// Give a native client the cluster's ceph.conf and an admin keyring.
	cfg, err := cluster.ConnectionConfig(ctx)
	if err != nil {
		t.Fatal(err)
	}
	_ = cfg
}

The service packages start a cluster with their own defaults: rbd.Run creates an initialized RBD pool named rbd, cephfs.Run the filesystem tc-cephfs with one active MDS, and rgw.Run one S3 gateway with a generated test user. Like ceph.Run, each returns a partially started fixture together with its error, so register its cleanup before checking the error:

objects, err := rgw.Run(ctx, ceph.DefaultImage)
testcontainers.CleanupContainer(t, objects)
if err != nil {
	t.Fatal(err)
}
gateways, err := rgw.Gateways(ctx, objects.Container)
if err != nil {
	t.Fatal(err)
}
gateway := gateways[0] // AccessKey, SecretKey and Region
endpoint, err := gateway.S3Endpoint(ctx)
if err != nil {
	t.Fatal(err)
}

To serve several services from one cluster, pass their options to ceph.Run:

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	rbd.WithPools(ceph.PoolConfig{Name: "volumes"}),
	cephfs.WithFilesystems(cephfs.Config{Name: "shared"}),
	rgw.WithGateways(rgw.Config{Name: "s3"}),
)

Documentation

Guides for tasks that span packages:

• API purpose and purpose tags: what the API is for, what each purpose tag means, and how an API declares its tags.
• Ceph cluster configuration and Ceph cluster operations: the ceph concepts that span several APIs; each API is documented in its godoc.
• Fault injection: stopping, pausing and disconnecting daemons and clients, delaying or dropping messages, fencing, full clusters and damaged replicas.
• Host network: bridge and host networking, and connecting native clients from the test process.
• Releases and images: supported Ceph releases, role images and image requirements.
• Contexts and cleanup: how contexts bound calls, what each fixture owns, and how to restore temporary changes.
• Running native commands: how to do anything the typed API does not cover with the Ceph CLIs inside the containers.
• Fixtures for Messenger client tests: the profiles, daemon controls and differences for testing a Messenger client implementation against native daemons.
• Classified errors: telling a refused request, a feature the release lacks, a missing image tool, an engine limit and a failed native command apart with errors.Is and errors.As.
• Bug compatibility: the registered ceph_bug_<id> tags.

For contributors:

• Testing: running and writing unit and integration tests.
• CI: the GitHub Actions workflows and how to read a failing batch.
• Preparing a release: the scope, evidence and publication checks for a candidate.
• Bug compatibility tags: the policy for adding, selecting and retiring a ceph_bug_<id> tag.

Related projects

ceph-testcontainers-images checks Ceph images against the requirements of these modules and publishes smaller role images (essential, client, cephfs, rbd-aux, rgw, and the aggregate all) extracted from the official images.

License

The modules are licensed under the MIT License, like Testcontainers for Go.

They do not contain or link Ceph. Tests pull the Ceph container images at run time and run the daemons and CLIs inside them; Ceph in those images is under its own licenses, mainly LGPL-2.1 and LGPL-3.0 with some GPL-2.0 parts. Any image a test selects, through the image argument of Run or an image option, keeps the licenses of whoever publishes it.