feat(system): scrape playbooks, people, teams and playbook roles

[adityathebe]

Extend the system scraper to populate access-related tables:

• Scrape non-CRD playbooks as MissionControl::Playbook config items
• Map people to external_users
• Map teams to external_groups
• Create playbook:run and playbook:approve external roles

Refactor Scrape() into smaller focused functions.

related: flanksource/mission-control-chart#292

Summary by CodeRabbit

• New Features

  • Scraper now consolidates collection of agents, playbooks, job histories, and external access metadata (users, groups, roles) into a single operation.

• Chores

  • Updated a core dependency to a newer patch version.
  • Removed several unused indirect/transitive dependencies to streamline the project.

[coderabbitai]

Warning

Rate limit exceeded

@adityathebe has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 13 minutes and 10 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

Walkthrough

Updated github.com/flanksource/duty from v1.0.1176 to v1.0.1179 and removed several unused indirect dependencies. Extended the system scraper to aggregate agents, playbooks, job histories, and external access metadata (external users, groups, and roles) via new helper functions.

Changes

Cohort / File(s)	Summary
Dependency Update go.mod	Bumped github.com/flanksource/duty to v1.0.1179 and removed multiple unused indirect/transitive dependencies (JSON/schema/i18n related packages).
Scraping Enhancements scrapers/system/system.go	Added aggregated scraping flow and helpers: scrapeAgents, scrapePlaybooks, scrapeJobHistories, scrapeAccessEntities, plus data collectors scrapePeople, scrapeTeams, scrapePlaybookRoles. Helpers query/store results and propagate errors; results include ExternalUsers, ExternalGroups, ExternalRoles linked to the scraper ID.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant Scraper
participant SystemScrape as "system.go helpers"
participant DB
participant ResultsStore as "Scrape Results Aggregator"

Scraper->>SystemScrape: Scrape(ctx, cfg)
SystemScrape->>DB: query agents
DB-->>SystemScrape: agents list
SystemScrape->>DB: query playbooks
DB-->>SystemScrape: playbooks list
SystemScrape->>DB: query job histories
DB-->>SystemScrape: job histories
SystemScrape->>DB: query people/teams/roles (scrapeAccessEntities)
DB-->>SystemScrape: external users/groups
SystemScrape->>ResultsStore: aggregate v1.ScrapeResult entries (including ExternalUsers/Groups/Roles)
ResultsStore-->>Scraper: final aggregated results

Possibly related PRs

• chore: bump duty to v1.0.1147 #1839: also updates github.com/flanksource/duty version in go.mod (dependency version change).
• chore: bump duty to v1.0.1178 #1896: similar go.mod change bumping github.com/flanksource/duty (adjacent version bump).
• chore: bump duty to v1.0.1142 #1829: related dependency version adjustments for github.com/flanksource/duty.

Suggested labels

dependencies, ready

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Merge Conflict Detection	⚠️ Warning	❌ Merge conflicts detected (3 files): ⚔️ go.mod (content) ⚔️ go.sum (content) ⚔️ scrapers/system/system.go (content) These conflicts must be resolved before merging into main.	Resolve conflicts locally and push changes to this branch.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main changes: adding scrapers for playbooks, people (mapped to external users), teams (mapped to external groups), and playbook roles, which aligns with the core functionality added in system.go and the PR objectives.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/system-scraper-access-entities

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@scrapers/system/system.go`:
- Around line 149-169: scrapeAccessEntities currently calls scrapePeople and
scrapeTeams and calls SetError twice which overwrites the first error; change
the logic to capture each call's error separately (e.g., errPeople, errTeams),
only assign ExternalUsers/ExternalGroups when their respective call succeeded,
and then set result.Error once: if both errPeople and errTeams are non-nil
wrap/aggregate them (use fmt.Errorf with %w or errors.Join) into a single error
and assign it to result via SetError (or fail fast by returning immediately on
the first error if you prefer that behavior); reference the scrapeAccessEntities
function and the SetError/ScrapeResult, ExternalUsers, ExternalGroups,
scrapePeople and scrapeTeams symbols when implementing the change.
- Around line 31-32: Guard against a nil persisted ID from
ctx.ScrapeConfig().GetPersistedID() before converting with lo.FromPtr and
calling scrapeAccessEntities: fetch persistedID :=
ctx.ScrapeConfig().GetPersistedID(), check if persistedID != nil, and only then
compute scraperID := lo.FromPtr(persistedID) and call results = append(results,
scrapeAccessEntities(ctx, scraperID)); do not call lo.FromPtr or
scrapeAccessEntities when persistedID is nil to avoid producing uuid.Nil.

🧹 Nitpick comments (1)

scrapers/system/system.go (1)

37-64: Nil LastSeen → agent always marked offline — verify this is intentional.

When agent.LastSeen is nil, lo.FromPtr returns the zero time.Time, making time.Since(...) extremely large and the agent "unhealthy/offline". If newly-registered agents legitimately have a nil LastSeen, they'd appear unhealthy immediately. If that's the desired behavior, a brief comment would help future readers.