Skip to content

OPERATOR_SECURITY.md

Latest commit

 

History

History
116 lines (102 loc) · 6.22 KB

OPERATOR_SECURITY.md

File metadata and controls

116 lines (102 loc) · 6.22 KB
Raw

Operator security checklist

Use this checklist before a deployment, challenge profile change or incident recovery. It complements the threat model.

Credentials

  • Every seed, bearer, wallet password, provider key and CA private key is an untracked regular file with mode 0400 or 0600 inside a 0700 directory.
  • No credential is present in an environment example, image layer, compose build argument, Terraform state, cloud-init payload, log or shell history.
  • Master operator, master intake, per-challenge internal/admin, Proof and VM-host tokens are distinct and rotated independently.
  • The master operator bearer (operator.token) seals epochs (/v1/admin/seal) and guards the other administrative routes; the separate intake bearer (intake.token) only feeds POST /v1/weights/raw. The master refuses to start when both files are the same path or hold the same value, and answers 503 on every private route if a rotation later makes them equal. Each bearer needs at least 32 bytes. Neither bearer grants the other's route, but an intake holder can still write forged raw leaves that the master's normal epoch seal will include (see the threat model). Hold each only where needed and rotate each on any suspicion. Wire v2 has no independent bundle cross-check: its vectors come from the Python implementation.
  • Miner BYOK vault storage is durable only as long as queued work requires, and terminal jobs have no remaining secret files.
  • The OpenRouter owner key stays on the VM host and is not exposed to miner jobs; miner BYOK never falls back to it.

Trust and pins

  • GET /v1/trust on the gateway returns the intended challenge and measurement profile at the deployment epoch, over HTTPS.
  • The Proof topic key matches PROOF_SK_FILE; no gateway or leaf key exists.
  • The challenge profile is legacy bounty/proof 2000/8000 (algorithm 1), 3000/7000 (algorithm 2, version >=2), or unique ids summing to 10000 (algorithm 3, version >=3). Bump the profile version (see Changing emission shares) before switching profiles; preserve old journals and sealed bytes.
  • Every registry entry names the challenge's own ghcr.io repository and GitHub source, keeps attestation = true, and runs on stable or a pin in production.
  • Runtime, kernel, rootfs, evaluator and experiment-pack references use verified SHA-256 digests. No production image uses a floating tag.
  • Empty or unknown pins remain fail-closed; no digest was copied from an unrelated artifact or invented to satisfy a check.
  • Inference and executor offers are signed, open, within platform ceilings and bound to the exact installed configuration.

Network and isolation

  • The gateway, Proof and challenge containers run only on the master role.
  • Only challenge-supervisor mounts the Docker socket; challenge containers publish no host port and join only cortex-challenges.
  • The validator role exposes no challenge execution route and reaches the master only through the configured VPC/TLS endpoint.
  • The VM orchestrator runs on a dedicated KVM-capable host with mutual network restrictions, a SAN-correct TLS certificate and a bearer file that is re-read per request.
  • Topic guests have only the explicit setup egress allowlist. Experiment guests have no network interface.
  • Firecracker and jailer binaries, kernel and rootfs pass the host startup validation; production never enables the fake hypervisor or host simulation.
  • Resource ceilings do not exceed 16 vCPU, 32 GiB RAM and the configured VM concurrency. Oversized topic requests fail instead of being clamped.

Service readiness

  • Every registered challenge answers GET /challenge/<id>/version, and its own readiness probe passes (for Bounty, /challenge/bounty/v1/status reports can_score: true after a real stable feed probe).
  • Stopping a challenge container makes the next epoch contain ChallengeInternal leaves for it and burn its share without blocking others.
  • While Proof is unwired, PROOF_VM_ORCHESTRATOR_URL is empty, no Proof topic is open, and its share burns to UID 0 through ChallengeInternal leaves.
  • When Proof is enabled, /v1/status reports a valid topic, sealed baseline, registered runner, pinned image, open inference offer and compatible executor offer; its failure matrix returns 503 without a scored row.
  • Operator routes reject missing and wrong bearer values from a non-loopback client.
  • Gateway latest is sealed before validators are enabled. An unsealed UID0 fallback is observed and rejected during the drill.

Persistence and recovery

  • Master SQLite files and WAL state reside on a durable private volume and are backed up with the service quiesced or through SQLite's backup API.
  • Restore testing covers gateway seals, epoch journal, Proof jobs, setup jobs, topic evidence, each cortex-challenge-<id>-data volume and validator dispatch state.
  • Pending external jobs are reconciled by stable job ID after restart; an uncertain paid operation is not blindly repeated.
  • Failed experiment VMs and bounded console tails are retained in the designated private RCA directory, reviewed, then deleted deliberately.

Release gate

uv run --no-sync ruff format --check src tests scripts
uv run --no-sync ruff check src tests scripts
uv run --no-sync mypy
uv run --no-sync pytest -m 'not live'
uv run --no-sync python scripts/check_repo.py --final
uv run --no-sync python scripts/check_deploy.py --check-examples
uv build --no-build-isolation
  • The wheel installs and its cortex entry point runs outside the source tree.
  • Compose renders for master and validator roles without an unpinned image, mounted Docker socket or challenge service on the validator.
  • CI made no live model call, GPU rent, KVM boot or chain submission.
  • The separate live smoke used the intended model and a fresh private state directory, with its API key absent from output and repository history.
  • Offline success is described accurately: it does not claim live KVM, scientific reproduction, provider billing or confirmed on-chain payment.