Enclave ops scripts (setup/attest/teardown) + verify-only CI e2e - #12
Draft
mdgeorge4153 wants to merge 16 commits into
Draft
Enclave ops scripts (setup/attest/teardown) + verify-only CI e2e#12mdgeorge4153 wants to merge 16 commits into
mdgeorge4153 wants to merge 16 commits into
Conversation
setup starts the host, brings the enclave up (all five egress proxies, the allocator raised to the enclave's memory, a non-debug run, secrets, expose, tunnel), gates on the built PCRs matching the on-chain EnclaveConfig, and registers permissionlessly. attest wraps attest_source from a package dir. teardown deletes the enclave object and terminates the instance (--keep-disk stops instead). Config lives in enclave-ops.conf; on-chain ids are read from addresses.<network>.json; the ephemeral enclave object lives only in the .enclave-session file written by setup.
…elease On a published release (or manual dispatch with a tag), rebuild the canonical EIF at the tagged commit, assert its PCRs match addresses.testnet.json, and attach nitro.eif + nitro.pcrs to the release. setup.sh downloads it instead of building; the registration-time PCR gate still proves it is canonical.
setup now launches its own Nitro instance from a stock AMI, provisions it (nitro-cli + socat only -- no docker/build for the download path), downloads the pre-built EIF from the RELEASE_TAG release, gates its PCRs against the recorded config, runs and registers it. INSTANCE_ID is an output in the session, not config; --build rebuilds from source. teardown terminates that instance. attest reopens the tunnel if it dropped.
On a freshly provisioned host the allocator restart reserves hugepages asynchronously, so the first run-enclave can lose the race (E19). Retry a few times and fail only if the enclave never starts.
… tries On a cold host the allocator reports active before it has reserved the enclave memory as hugepages; a run-enclave started then hangs rather than failing, wedging a plain retry loop. Terminate any stuck enclave, wait, and bound each attempt with timeout, retrying until one starts.
Root cause of the fresh-host E19/hang: the allocator reserves the enclave memory as 1G hugepages asynchronously, and run-enclave started before that finishes hangs and can wedge the CPU pool. Poll the reserved hugepage count until it covers the requested memory, then run once.
Same cold-start lesson as the hugepage wait: the enclave listens for secrets a moment after boot, and its server needs a few seconds after that to mount its tmpfs and come up. Retry the secrets send and poll get_attestation until it serves, rather than a single fixed sleep.
nitro-cli opens its log on every invocation; the PCR-gate describe-eif was the one nitro-cli call not run via sudo, so on a cold host it failed with E19 and exited the script before run-enclave. Run it via sudo like the others.
…arse Three fixes from a full fresh-instance run: - The 6G tmpfs in run.sh needs >=12G enclave headroom; 8G stalls the mount and the server never starts. Default to m5.2xlarge / MEMORY_MIB=12288. - setsid + </dev/null the vsock proxies and the port forwarder, or they hold the ssh channel open and setup hangs after 'enclave serving'. - Parse the registered Enclave object id with grep, robust to sui's ID:/ObjectID: table variants (awk on a fixed column broke).
…sui) For CI: setup --no-register brings the enclave up and serves without registering (no sui, no on-chain writes), and attest --no-attest skips the sui guard. teardown already no-ops the enclave-object delete when the session has none.
Launches a fresh Nitro instance via OIDC, downloads the release EIF, brings the enclave up (--no-register), verifies the demo auditor through it (--no-attest), and tears down in always(). No sui/funded key needed. Temp push trigger on this branch exercises it before merge; production is workflow_dispatch + release.
The detached daemons make ssh return a spurious broken-pipe non-zero after the remote bring-up succeeds. Capture the output, print it, and require the 'enclave serving' marker instead of trusting the exit code.
A --no-register setup writes no enclave object id, and --no-attest does not need one; default it to empty so set -u does not trip.
…pdate RUNNING.md - enclave-e2e: remove the temporary push trigger (dispatch + release only). - addresses.testnet.json: drop objects.enclave -- the enclave object is ephemeral (fresh key per boot, re-registered each session), not a stable deployment fact. - RUNNING.md: describe the real setup/attest/teardown, the download-by-default flow, the 12G/m5.2xlarge requirement, and --no-register/--no-attest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turnkey scripts + CI to run a source-verification enclave from scratch and prove it keeps working.
Scripts (operator-side:
sui+ AWS on your machine, enclave on a Nitro instance)setup.sh— launches a fresh Nitro instance, provisions it, downloads the canonical EIF from the release, gates its PCRs against the on-chainEnclaveConfig, runs it, opens an SSH tunnel, and registers permissionlessly (noCap, no publish).--buildbuilds from source instead;--no-registerbrings it up for verify-only. Needs anm5.2xlarge/12 G — the 6 G verification tmpfs needs headroom orrun.shstalls.attest.sh <package-dir>— verifies the package's pushed commit through the enclave and records theAttestation.--no-atteststops at the signed response (no sui).teardown.sh— deletes the ephemeral enclave object and terminates the instance.ops-common.sh(config + on-chain id resolution),enclave-ops.conf.example.CI
release-eif.yml— on a published release, rebuilds the EIF, asserts its PCRs matchaddresses.testnet.json, and attachesnitro.eif+nitro.pcrs.enclave-e2e.yml— verify-only smoke test: launches a fresh instance via OIDC (no stored AWS keys, ephemeral SSH key), downloads the published EIF, brings the enclave up, verifies a fixed package through it, and tears the instance down inalways(). No funded key needed. Validated green end-to-end (run 31842920557).Also
RUNNING.mdrewritten to match the real scripts.addresses.testnet.json: droppedobjects.enclave— the enclave object is ephemeral (fresh key per boot, re-registered each session), not a stable deployment fact.Notes for review
setup.shcommits are cold-start debugging iterations (E19/sudo describe-eif, hugepage-reservation timing, tmpfs/memory → 12 G, ssh-detach, register parse). Squash-merge is fine.sv-enclave-ci(assumable only by this repo; EC2 launch/describe/keypair;TerminateInstancesscoped to theName=sv-enclavetag) and repo variablesAWS_ROLE_ARN/AWS_REGION/CI_AMI_ID/CI_SUBNET_ID/CI_SG_ID.enclave-e2edepends on those.🤖 Generated with Claude Code