feat(archive): integrate archive lifecycle into sandbox state machine, operator, and reconciler

[zhangjaycee]

close #1085

1. State Machine Overview

6 states: pending / running / stopped / archiving / archived / deleted (final).

---

2. Timestamp Fields

Field	Meaning	Set by	Cleared by
create_time	Sandbox creation time	SandboxManager._build_sandbox_info_metadata	Never
start_time	First time container is alive	on_alive (once, idempotent)	Never
stop_time	Container stop time	on_stop	Popped by on_restart / on_restore
archive_time	Archive completion time	on_archive_done	Popped by on_archive_failed
delete_time	Soft-delete time	on_delete	Never (final state)

Timeout anchor: state_history

Intermediate-state timeout is computed from state_history — a capped list of {from_state, to_state, event, timestamp} records appended by before_transition. The reconciler calls get_current_state_started_at(state_history, target_state) to find the timestamp of the most recent transition into the current state.

• _reconcile_archiving: looks up "archiving" entry, compares elapsed time against archive_timeout_seconds
• _reconcile_pending: looks up "pending" entry (only when archive_time is present, indicating a restore-in-progress), compares against restore_timeout_seconds

Key points

• archive_time is the completion time, not the initiation time. It is written in on_archive_done (ARCHIVING → ARCHIVED), not in on_archive (STOPPED → ARCHIVING).
• archive_time also serves as a restore-in-progress discriminator. _reconcile_pending uses info.get("archive_time") to distinguish a fresh-start PENDING sandbox from one that is restoring from ARCHIVED.

---

3. Timeout Enforcement

Two layers of timeout enforcement:

Layer 1: Actor-internal timeout (active kill)

SandboxActor.archive() and restore_and_start() wrap their work in asyncio.wait_for(coro, timeout=timeout_seconds). On timeout:

• asyncio.CancelledError propagates to all awaited subprocesses
• _run_shell_command and _docker_cmd catch CancelledError and proc.kill() the child process
• For archive: _run_archive then ray.kill(actor) in its finally block
• For restore: the actor raises TimeoutError, but stays alive (restore actor is long-lived); the reconciler detects the timeout and fires restore_failed

Layer 2: Reconciler timeout (passive detection)

_reconcile_archiving / _reconcile_pending poll periodically, compute elapsed time from state_history, and fire archive_failed / restore_failed when the configured timeout is exceeded. This is the safety net in case the actor dies without reporting.

---

4. Constraints and Design Choices

Single archive per sandbox

Each sandbox has at most one archive — multi-version snapshots are not supported. Archive artifacts use deterministic names based solely on sandbox_id:

Storage	Format	Example
OSS / S3 key	{prefix}{sandbox_id}.tar.gz	rock-archives/sbx-abc123.tar.gz
ACR image ref	{registry}/{namespace}/sandbox_archived:{sandbox_id}	rock-acr.cr.aliyuncs.com/sandbox_archive/sandbox_archived:sbx-abc123

This means any component can compute the storage location from sandbox_id + config alone, without reading extra fields from the database.

Re-archive: explicit delete before write

A sandbox may go through multiple archive/restore cycles:

stopped → archive → archived → restore → pending → running → stopped → archive again

On the second archive, the previous key/ref still exist in remote storage (restore downloads but does not delete the remote copy). We handle this by explicitly deleting the old artifacts before starting a new archive — we do not rely on OSS/ACR implicit overwrite behavior.

Rationale:

• Different S3-compatible implementations vary in PUT-overwrite semantics (versioning, ACL inheritance)
• ACR tag overwrite behavior differs across registry implementations (some require manifest deletion first)
• Explicit delete-then-write is predictable and easy to debug

The deletion is triggered in on_archive when sandbox_info already contains archive_time (indicating a prior successful archive). Both this and the on_delete cleanup are best-effort — failures are logged as warnings but do not block the main flow.

When	Trigger	What gets cleaned
on_archive (re-archive)	archive_time already present in sandbox_info	Old key + old ref
on_delete	archive_time present and storage clients available	key + ref

Size limits

Two configurable limits prevent archiving excessively large sandboxes:

Config	Default	Check point
max_image_push_size	"16g"	After docker commit, before push — checked via docker image inspect
max_dir_upload_size	"16g"	Before upload_dir — checked via du -sb

When a limit is exceeded, the actor raises RuntimeError and is killed. The _reconcile_archiving scanner detects the ARCHIVING timeout and fires archive_failed, rolling the sandbox back to STOPPED.

Note: the dir size check happens after the image has already been pushed. If the dir exceeds the limit, the image is rolled back (image_storage.delete) before the error propagates.

---

5. Configuration

lifecycle:
    archive_timeout_seconds: 1800
    restore_timeout_seconds: 1800
    reconcile_interval_seconds: 30
    archive:
      enabled: false
      max_image_push_size: "16g"
      max_dir_upload_size: "16g"
      allowed_keys: ["key-a", "key-b"]  # null = open to all; list = whitelist by X-Key header
      dir_storage:
        type: "oss"                   # oss or s3
        endpoint: ""
        bucket: ""
        access_key_id: ""
        access_key_secret: ""
        region: ""
        prefix: "archive-logs/"       # OSS/S3 key prefix for log archives
      acr:
        registry_url: ""
        username: ""
        password: ""
        namespace: "sandbox_archive"