A validator is decommissioned and its hosting provider reassigns the public IP to a different customer. This happens routinely, and it reliably breaks the new tenant. Worse, in the most common variant it does not break loudly: the new machine comes up connected on the previous tenant's onchain identity.
This issue records the analysis and tracks the work. Each proposed change is a sub-task.
Root cause: two identities, keyed differently
| Account |
PDA seeds |
Keyed on |
AccessPass |
["accesspass", client_ip, user_payer] (pda.rs:90) |
IP and keypair |
User (V2) |
["user", client_ip, user_type] (pda.rs:58) |
IP and type, no owner |
That asymmetry is the whole problem. Two keypairs at the same address get two distinct access passes and never collide. They map to the same user account address. A user account is therefore a global, first-come lease on (client_ip, user_type), and the only parties who can release it are the incumbent owner and a USER_ADMIN holder (.../processors/user/delete.rs:131-139).
Scenario A: the daemon was shut down and the user account survives
This is the common case and the damaging one.
The new operator runs doublezero connect. Their own access pass checks out. The CLI then looks for an existing user by client_ip and user_type, ignoring the owner:
// crates/doublezero-daemon-cli/src/connect.rs:1281-1284
let matched_user = users
.iter()
.find(|(_, u)| u.client_ip == *client_ip && u.user_type == user_type);It reports An account already exists with Pubkey: X and adopts it. The daemon reconciler independently filters onchain users by client IP alone (client/doublezerod/internal/manager/manager.go:507-512). The controller has already programmed the device tunnel with UnderlayDstIP = <client_ip> (controlplane/controller/internal/controller/server.go:616), and that address now belongs to the new host, so packets flow and the session comes up.
The new validator is genuinely connected, on the old validator's account: its dz_ip, its device seat, its access pass, its multicast allowlists, its billing attribution. Nothing onchain reflects that the machine changed hands.
doublezero disconnect then refuses, because it skips any user whose owner is not the local payer (.../disconnect.rs:147-165) and reports the account as "managed by an external service". This is the "my doublezero address doesn't match what's onchain" report we keep receiving. The mismatch is on the user account, not the access pass — the operator's own pass is present and correct.
Two follow-on hazards in this state:
- Connecting with a different mode (
IBRLWithAllocatedIP rather than IBRL) derives a different PDA, so a second user account is created for the same address. The reconciler logs multiple activated unicast users for this client IP, ignoring extra and takes whichever it encounters first out of a map, which is not stable across restarts.
- The previous operator can still delete their user at any time, which silently tears the tunnel out from under the new one.
Scenario B: the previous operator ran doublezero disconnect
This case largely works. The PDA is released and the new operator creates their own user under their own pass. What remains is the previous access pass at (client_ip, old_key) with connection_count = 0 and status = Disconnected, still carrying epoch entitlement and multicast allowlists for a keypair that no longer has the address. Nothing collects it.
Why a wildcard access pass is not the fix on its own
It addresses half. GetAccessPassCommand already prefers the dynamic pass, and both processors accept either PDA, so a 0.0.0.0 pass does stop the pass from being IP-bound. The user account is still keyed on the address, so the collision is untouched.
It also weakens the position. A wildcard pass asserts nothing about which address the holder may use, and the program only checks that a supplied client_ip is globally routable. A holder can bind an address they do not control, squatting the PDA and denying service to whoever actually has it. Per-IP control for that path is the subject of RFC-27, "IP Ownership Verification Service for user connection" (#2304), which this work depends on but does not duplicate: RFC-27 governs which address may be bound at user creation, while this issue is about releasing an address whose account is already bound.
Two defects found along the way
Independent of the above, and reachable outside this scenario. Both produce states no existing instruction can clean up: a closed access pass makes its user account permanently undeletable, and RequestBan never decrements connection_count, so a banned user leaves an access pass that can never be closed. Detail in the sub-task.
Proposed changes
Dependencies
RFC-27, "IP Ownership Verification Service for user connection" (#2304), tracked separately. #4190 needs it implemented.
Suggested sequencing
#4188 and #4191 are small, self-contained, and unblock everything else. #4189 removes most of the operational pain without a program change. #4190 follows RFC-27.
A validator is decommissioned and its hosting provider reassigns the public IP to a different customer. This happens routinely, and it reliably breaks the new tenant. Worse, in the most common variant it does not break loudly: the new machine comes up connected on the previous tenant's onchain identity.
This issue records the analysis and tracks the work. Each proposed change is a sub-task.
Root cause: two identities, keyed differently
AccessPass["accesspass", client_ip, user_payer](pda.rs:90)User(V2)["user", client_ip, user_type](pda.rs:58)That asymmetry is the whole problem. Two keypairs at the same address get two distinct access passes and never collide. They map to the same user account address. A user account is therefore a global, first-come lease on
(client_ip, user_type), and the only parties who can release it are the incumbent owner and aUSER_ADMINholder (.../processors/user/delete.rs:131-139).Scenario A: the daemon was shut down and the user account survives
This is the common case and the damaging one.
The new operator runs
doublezero connect. Their own access pass checks out. The CLI then looks for an existing user byclient_ipanduser_type, ignoring the owner:It reports
An account already exists with Pubkey: Xand adopts it. The daemon reconciler independently filters onchain users by client IP alone (client/doublezerod/internal/manager/manager.go:507-512). The controller has already programmed the device tunnel withUnderlayDstIP = <client_ip>(controlplane/controller/internal/controller/server.go:616), and that address now belongs to the new host, so packets flow and the session comes up.The new validator is genuinely connected, on the old validator's account: its
dz_ip, its device seat, its access pass, its multicast allowlists, its billing attribution. Nothing onchain reflects that the machine changed hands.doublezero disconnectthen refuses, because it skips any user whose owner is not the local payer (.../disconnect.rs:147-165) and reports the account as "managed by an external service". This is the "mydoublezero addressdoesn't match what's onchain" report we keep receiving. The mismatch is on the user account, not the access pass — the operator's own pass is present and correct.Two follow-on hazards in this state:
IBRLWithAllocatedIPrather thanIBRL) derives a different PDA, so a second user account is created for the same address. The reconciler logsmultiple activated unicast users for this client IP, ignoring extraand takes whichever it encounters first out of a map, which is not stable across restarts.Scenario B: the previous operator ran
doublezero disconnectThis case largely works. The PDA is released and the new operator creates their own user under their own pass. What remains is the previous access pass at
(client_ip, old_key)withconnection_count = 0andstatus = Disconnected, still carrying epoch entitlement and multicast allowlists for a keypair that no longer has the address. Nothing collects it.Why a wildcard access pass is not the fix on its own
It addresses half.
GetAccessPassCommandalready prefers the dynamic pass, and both processors accept either PDA, so a0.0.0.0pass does stop the pass from being IP-bound. The user account is still keyed on the address, so the collision is untouched.It also weakens the position. A wildcard pass asserts nothing about which address the holder may use, and the program only checks that a supplied
client_ipis globally routable. A holder can bind an address they do not control, squatting the PDA and denying service to whoever actually has it. Per-IP control for that path is the subject of RFC-27, "IP Ownership Verification Service for user connection" (#2304), which this work depends on but does not duplicate: RFC-27 governs which address may be bound at user creation, while this issue is about releasing an address whose account is already bound.Two defects found along the way
Independent of the above, and reachable outside this scenario. Both produce states no existing instruction can clean up: a closed access pass makes its user account permanently undeletable, and
RequestBannever decrementsconnection_count, so a banned user leaves an access pass that can never be closed. Detail in the sub-task.Proposed changes
find_or_create_userand in the daemon reconciler's user filter, and reword the misleadingdisconnectmessage. This converts a silent misprovision into an actionable error, and it is a prerequisite for anything liveness-gated: while the new tenant keeps adopting the stale account, that account reports BGPUp.USER_ADMINreconciler that deletes users whose BGP has been down past a threshold and closes lapsed passes, using the onchain BGP state the device'smetrics_publisheralready writes. No program change. Highest value for the effort, because it frees the address before the next tenant arrives.ReclaimUser. A self-service path to release a user account for a caller who can prove present control of its IP, gated on the incumbent not passing traffic. Validates theIpOwnershipProoffrom RFC-27.DeleteUsermust tolerate a missing access pass, andRequestBanmust release the pass it currently leaks.Dependencies
RFC-27, "IP Ownership Verification Service for user connection" (#2304), tracked separately. #4190 needs it implemented.
Suggested sequencing
#4188 and #4191 are small, self-contained, and unblock everything else. #4189 removes most of the operational pain without a program change. #4190 follows RFC-27.