fix: create service user roles on every db node independently

[rshoemaker]

Summary

• Replaces primary-only ServiceUserRole creation with a canonical + per-node resource pattern
• The canonical resource (first node) generates credentials; per-node resources for each additional node read credentials from the canonical and create the role independently via a direct connection to postgres
• Eliminates reliance on Spock DDL replication, which was unreliable for role creation and would leave new nodes missing service accounts when added later
• Adds explicit error when no primary can be determined for a target node, replacing a silent fallback that could create roles on the wrong node

Changes

• service_user_role.go — Added CredentialSource field and ServiceUserRolePerNodeIdentifier. Identifier(), Dependencies(), and Create() branch on canonical vs. per-node. connectToPrimary filters instances by NodeName and returns an error if no primary is found.
• orchestrator.go — Appends per-node RO+RW ServiceUserRole resources for each database node beyond the first.
• service_instance.go — Added DatabaseNodes field to ServiceInstanceSpec to carry node list into the orchestrator.
• plan_update.go — Passes DatabaseNodes when constructing ServiceInstanceSpec.
• service_user_role_test.go — New unit tests covering identifier dispatch, dependency wiring, per-node resource count for 1/2/3-node clusters, and identifier uniqueness.

Test plan

• Unit tests pass: go test ./server/internal/orchestrator/swarm/...
• Create 2-node + MCP database and verify svc_ro and svc_rw exist on both nodes
• Add a 3rd node and verify service roles are created on the new node without affecting existing nodes

PLAT-524

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a59cafd8-3605-4e32-801b-31bb9ef807a6

📥 Commits

Reviewing files that changed from the base of the PR and between 6ac70fa and 643126b.

📒 Files selected for processing (2)

• server/internal/orchestrator/swarm/service_user_role.go
• server/internal/orchestrator/swarm/service_user_role_test.go

✅ Files skipped from review due to trivial changes (1)

• server/internal/orchestrator/swarm/service_user_role_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• server/internal/orchestrator/swarm/service_user_role.go

---

📝 Walkthrough

Walkthrough

The pull request extends service user role provisioning to support multi-node database clusters by adding a DatabaseNodes field to track all cluster nodes, introducing per-node ServiceUserRole resources that reference canonical roles for credential sourcing, and implementing credential inheritance logic for database operations across nodes.

Changes

Cohort / File(s)	Summary
Data Model Extensions server/internal/database/service_instance.go, server/internal/orchestrator/swarm/service_user_role.go	Added DatabaseNodes field to ServiceInstanceSpec for node tracking. Introduced ServiceUserRolePerNodeIdentifier() function and CredentialSource field to ServiceUserRole to support per-node role identity and credential sourcing from canonical roles.
Orchestration Logic server/internal/orchestrator/swarm/orchestrator.go, server/internal/workflows/plan_update.go	Updated GenerateServiceInstanceResources() to compute canonical role identifiers and generate per-node ServiceUserRole resources for each node in the cluster. Modified workflow to populate the DatabaseNodes field in service instance specifications.
Identity & Reconciliation server/internal/orchestrator/swarm/service_user_role.go	Refactored Identifier(), Dependencies(), and DiffIgnore() methods to conditionally branch on CredentialSource. Modified Create() to generate credentials for canonical roles and load them from canonical sources for per-node roles. Updated database connection logic to use GetPrimaryInstance() instead of connectToPrimary() helper.
Test Coverage server/internal/orchestrator/swarm/service_user_role_test.go	Added comprehensive unit tests validating canonical vs per-node identifier generation, dependency ordering, credential sourcing behavior, and provisioning correctness across multi-node scenarios with RO/RW role pairing.

Poem

🐰 Hop, hop—behold! The burrow expands,
Per-node roles sprout across database lands,
Canonical keepers hold passwords so tight,
While per-node replicas learn from their light,
A cluster of clusters, all nested just right!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: creating service user roles on every database node independently rather than only on the primary.
Description check	✅ Passed	The PR description follows the template with all required sections (Summary, Changes, Testing, Checklist, Notes for Reviewers) completed with clear, specific details about the implementation and testing approach.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/PLAT-524/svc_acct_propagation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}