Skip to content

AGENTS.md

Latest commit

 

History

History
302 lines (227 loc) · 20 KB

AGENTS.md

File metadata and controls

302 lines (227 loc) · 20 KB
Raw

Scope

Scope is a platform for benchmarking AI coding agents. It orchestrates multiple coding agents (GitHub Copilot, Claude Code, VS Code Web), sends them standardized tasks through configurable scenarios and personas, evaluates results using a criteria DAG with the Judge service, and tracks everything with real-time logging.

For the full system architecture, see docs/architecture/overview.md.

Repository Structure

pnpm workspaces monorepo. TypeScript, strict mode, ES2022, NodeNext modules.

apps/
  api/                          # REST API — run orchestration, SSE log streaming, criteria CRUD
  cli/                          # CLI — submit runs, stream logs, manage criteria
  judge/                        # Evaluation engine — scores agent output against criteria DAG
  portal/                       # Web UI — run management, insights, criteria graph editor
  token-manager/                # Token storage, validation, round-robin distribution
  workers/
    coder-acp-copilot/          # GitHub Copilot worker (ACP SDK)
    coder-acp-claude-code/      # Claude Code worker (ACP SDK)
    report-generator/           # Post-run report generation
  model-scanners/               # Feature detection for Copilot and Anthropic models
  version-checkers/             # Poll for new agent/tool releases
  key-updaters/                 # GitHub auth cookie management for VS Code Web
packages/
  shared/                       # Monorepo foundation — types, DB models, queue/blob/redis clients
  db-migrations/                # MongoDB migration framework (mongo-migrate-ts)
  github-auth/                  # GitHub OAuth/device-code auth utilities
  model-scanning/               # Shared model scanning logic
  version-checking/             # Version comparison utilities
  llm-eval/                     # LLM-graded eval harness (grader, sampling, rate-limit retry)
  test-utils/                   # ACP worker integration test helpers (Docker runner, in-container harness)
config/                         # Benchmark definitions (YAML)

For package architecture and data models, see docs/architecture/app-design.md.

Cross-client domain ownership

The API owns domain validation, orchestration, fan-out, defaulting, and resolution rules shared by Portal, CLI, and direct API consumers. Portal and CLI are thin clients: they collect user intent and may provide early validation, but must not duplicate or materialize API-owned business logic. When one user action expands into multiple domain objects, each object is derived independently by the API from the shared intent so client or sibling state cannot leak between them.

Key Components

API (apps/api/)

Express.js REST server. Orchestrates runs, streams logs via SSE, manages criteria CRUD, routes tasks to workers through Azure Storage Queues. Connects to MongoDB (CosmosDB-compatible), Redis, Azure Storage Queues, and Blob Storage.

Project-scoping invariant: never perform a global, slug-only get/edit/soft-delete on a project-scoped entity. Every such query must filter by _id or projectId — a human slug/business id is never a key on its own. Derive projectId from context (a parent doc or the run-request) when available, else require it from the client (?projectId=), else fail 400. See the by-id invariant in docs/architecture/app-design.md.

Authentication invariant: verify the unchanged IdP bearer before any user-access cache lookup. Only POST /api/v1/users/me calls UserAccessResolver.enrollOnLogin() for JIT/profile/lastLogin/bootstrap writes. Every GET /users/me and other routes use resolveExisting() (Redis hit: no Mongo; miss/outage: exact identity read, never upsert). Missing/disabled identities deny access, never become anonymous. Preserve existing no-token/public rollout; full RBAC and Scope internal tokens remain deferred. See auth-rbac.md before changing this boundary.

Workers (apps/workers/)

Each worker implements the same queue-processor interface but adapts a different coding agent:

Worker Agent Protocol
coder-acp-copilot GitHub Copilot ACP SDK v0.14.1
coder-acp-claude-code Claude Code ACP SDK v0.13.1
report-generator — Copilot SDK

Workers consume tasks from Azure Storage Queues (named queue-<worker-name>) and write results to MongoDB and Blob Storage. Each has its own Dockerfile and docker-compose profile.

Judge (apps/judge/)

Evaluation engine that scores agent output against a criteria DAG (directed acyclic graph). Uses the GitHub Copilot SDK for LLM-based evaluation. Two strategies: bundled (all criteria in one session) or independent (topological order, skips descendants of failures).

Portal (apps/portal/)

React 19 web UI with Vite, Tailwind CSS, Radix UI (shadcn/ui), TanStack Query, and XYFlow for criteria DAG visualization. Communicates with the API via REST and SSE.

AuthProvider owns the Scope-user handshake: callback → POST /users/me; cached-account reload → plain /users/me. Gate all eager queries (including providers outside RequireAuth) until ready; do not treat MSAL account claims as the Scope UUID/role. Deduplicate account/login work and cancel it on account change or logout. The enrollment POST is no-store and must never be prefetched/polled.

Storybook: When adding or modifying portal components, update the corresponding Storybook stories. Use the storybook skill for guidance.

Portal UX v2 implementation learnings (Runs + Submit Run)

Use these patterns when extending list/detail or run-submission UX:

  1. Keep interaction model consistent across entities.

    • Use the same layout contract: central list/table, right filter rail, contextual left detail/settings panel.
    • Reuse shared list primitives (ListLayout, FilterRail, DataTable, CustomizeColumnsPanel, Pagination) instead of ad-hoc page layouts.

  2. Prefer in-place creation over navigation breaks.

    • For /runs/new, create profiles in a dialog and keep users on the page.
    • Reuse shared forms (e.g. ProfileCreateForm) between full-page and modal flows to avoid behavior drift.

  3. Treat action counts as source-of-truth UX.

    • Any submit/CTA label must reflect the real backend effect (e.g. expanded run count, not just occurrence count).
    • If composition/expansion is shown elsewhere on the page, the primary action label must match it exactly.

  4. Composition UIs should make hierarchy explicit.

    • Show base profile and variations with distinct visual semantics (badges/labels such as Base, Var N).
    • Prefer profile names over IDs in group headers and list cells; use IDs only as fallback.
    • When grouped by profile, carry base/variation badges into group headers for scanability.

  5. Long forms in dialogs require stable action affordances.

    • Use a constrained scroll container with a sticky footer for primary actions.
    • Ensure dialog structure uses a non-growing shell (grid-rows-[auto_minmax(0,1fr)] + min-h-0/overflow-y-auto) so content scrolls without losing actions.

  6. Inline control rows should align visually and behaviorally.

    • Keep New… actions on the same row as their picker/search control where possible.
    • Match control heights and interaction patterns between similar pickers (task/profile/criteria) to reduce cognitive load.

CLI (apps/cli/)

Command-line interface built with Commander.js and Ink (React for terminals). Used for submitting runs, streaming logs, managing criteria, and CI/CD automation. Run pnpm cli --help to discover subcommands.

CLI ↔ Portal parity: Every feature available in the Portal must also be available in the CLI. The CLI is the primary interface for CI/CD and power users — it must never lag behind the Portal in capabilities.

Token Manager (apps/token-manager/)

Express service for centralized token storage, validation, and round-robin distribution. Integrates with Azure Key Vault (Lowkey Vault locally). Uses packages/github-auth/ for GitHub OAuth/device-code auth.

Shared Package (packages/shared/)

Monorepo foundation. Exports types (runs, iterations, criteria, scenarios, personas), Mongoose models, and service clients (queue, blob, redis, config loader, criteria-store, criteria-provider). All apps and workers depend on it — breaking changes here affect everything.

Configuration (config/)

Personas, scenarios (tasks), criteria, and prompt features are stored in MongoDB (source of truth). They can be exported/imported as YAML for portability and version control. The config/ folder contains YAML examples of these data types.

  • traits.yaml — Evaluation trait dimensions (personality, experience, verbosity, type)
  • personas/ — Reviewer personas combining traits (e.g. demanding-senior.yaml, vibe-coder.yaml)
  • scenarios/ — Task definitions agents must implement (e.g. hello-world-express.yaml, react-snake-game-v2.yaml)
  • criteria/ — Evaluation criteria forming a DAG with parent-child dependencies, consumed by the judge
  • prompt-features/ — Feature flags tracking what capabilities agents request

Database Migrations (packages/db-migrations/)

Built on mongo-migrate-ts. Migrations are TypeScript files with up() and down() methods. MongoDB is CosmosDB-compatible — avoid features not supported by CosmosDB's MongoDB API.

Code Readability

  • Write for humans: descriptive names, small focused functions, simple control flow, minimal abstraction.
  • Separate logical steps with blank lines; avoid dense one-liners and unrelated statements packed together.
  • Give every non-trivial function and logical code block a short comment explaining what it does; add intent, assumptions, invariants, or surprising reasons where useful.
  • Apply the same rule to tests: comment non-trivial fixture setup, user-flow phases, and invariant assertions. A descriptive test name does not replace intent comments inside a multi-step test; avoid generic Arrange / Act / Assert labels and comments that merely repeat the next line.
  • Document function parameters, types, and their fields/properties with concise JSDoc/TSDoc explaining meaning, constraints, units, optionality, and defaults where relevant.
  • Do not narrate trivial or self-evident code. Document public contracts and failure behavior when names and types are insufficient.
  • Follow local conventions and formatters; review readability as well as correctness.

Development

pnpm docker:up:infra              # Start backing services (MongoDB, Redis, Azurite, Lowkey Vault)
pnpm docker:dev:copilot           # Full stack with Copilot worker + portal (hot reload)
pnpm docker:dev:all               # All workers + portal + report generator (hot reload)
pnpm dev:api                      # API only (native)
pnpm dev:portal                   # Portal only (native)
pnpm dev:<worker-name>            # Individual worker (native)
pnpm open:portal                  # Open portal in browser

Before a large clean Docker or Podman rebuild, inspect the container runtime storage and any runtime VM filesystem as described in CONTRIBUTING.md, then report the available space and Scope-specific resources. Never automatically run broad prune commands or delete unrelated images, containers, build cache, or volumes. Any cleanup must target verified Scope resources and requires explicit user approval; warn that named volumes may contain local database or authentication data.

Shared Dev Infrastructure (CosmosDB)

For testing against real Azure CosmosDB (e.g. index behavior), a shared dev instance can be provisioned. Each worktree gets its own isolated database. See docs/shared-dev-infra.md for setup and usage.

Rust Components

When making changes to any Rust component (e.g. the AI gateway in apps/gateway/), follow the rust-best-practices skill. This skill is available at .agents/skills/rust-best-practices/SKILL.md and covers idiomatic Rust, ownership patterns, error handling with Result, and performance guidelines.

Testing

Co-locate tests next to source as <filename>.test.ts. Framework: Vitest.

pnpm test                         # Unit tests
pnpm test:coverage                # With coverage report
pnpm test:integration             # Integration tests (requires .env + Docker)

Portal Storybook stories run under Vitest: apps/portal/src/components/ui/stories.play.test.tsx composes the ui/* stories and executes their play (interaction) functions inside the regular Vitest suite (no @storybook/addon-vitest required). It binds a Testing Library canvas to the rendered container, so story play functions must keep depending only on canvas plus values imported directly from storybook/test (userEvent, screen, expect). When you add a new ui/* story with a play function, register its module in that harness so it's covered.

Static prompt evaluations

Read docs/architecture/prompt-evaluations.md before changing any AI-facing instruction surface.

  • Whenever hardcoded system/user prompt text, prompt-building logic, output instructions, or AI-facing tool descriptions are added or modified, update the corresponding production target adapter, curated cases, deterministic checks, composition contract, and/or rubric.
  • Register every new runtime static prompt family in the documented inventory, committed evaluation manifest, and JSONL adapter registry.
  • Run the smallest relevant explicit prompt-evaluation command before treating a prompt change as complete. These suites are developer-invoked and are not part of normal pnpm test or CI.
  • A change to user-authored/configurable prompt content does not by itself create a static prompt family.
  • Whenever a user-controlled text field that reaches an AI is added or its insertion point, trusted wrapper, role, tools, or security boundary changes, preserve its benign composition contract. Cloud red-team profiles and execution are maintained in a separate follow-up contribution.

Contributing (Pull Requests)

This repository is commonly worked on from a fork. When opening a pull request, always target the upstream repository when one is available — do not open the PR against the fork unless the user explicitly asks you to.

  • Detect the upstream: check git remote -v. If an upstream remote exists (e.g. microsoft/scope), that is the PR base. The origin remote is typically your personal fork (e.g. cedricvidal/scope).
  • Push the branch to your fork (origin), then open the PR across forks with the upstream as the base: 
    gh pr create \
  --repo <upstream-owner>/<repo> \
  --base main \
  --head <fork-owner>:<branch> \
  --title "..." --body-file <path>
  • Only fall back to opening the PR against the fork (origin) when there is no upstream remote, or when the user explicitly requests it.
  • If the upstream org enforces SAML SSO and the PR call fails with a 403 ("Resource protected by organization SAML enforcement"), stop and ask the user to authorize their token for that org via the SSO link, then retry — do not silently downgrade to a fork PR.

Documentation Workflow

Before starting any task, read the docs relevant to the components you will be working on (see the table below). Understanding the existing design, data models, and patterns prevents regressions and duplicated work.

Before completing any task, update the relevant docs to reflect your changes. This is the last step before calling the work done. If you added a new component, added or changed an API, modified data models, introduced a new pattern, or changed deployment behavior, the corresponding doc must be updated (or a new one created and linked here). Documentation is not optional — outdated docs are worse than no docs.

Documentation

Document Description
docs/architecture/overview.md System architecture, component interactions, data flow
docs/architecture/app-design.md Data models, API design, package dependency graph
docs/architecture/prompt-evaluations.md Static prompt quality, datasets, commands, and maintenance rules
docs/architecture/data-organization-projects.md Projects (a single container) to isolate/group data within a cluster; composes with data-tags and auth-rbac
docs/architecture/auth-rbac.md Explicit-login IdP auth, Redis user-access cache, Portal handshake; deferred RBAC/internal-token roadmap
docs/architecture/vscode-web-worker.md XState chat machine, GitHub auth flow, ARIA snapshots
docs/architecture/token-manager.md Token storage, validation, round-robin distribution
docs/architecture/criteria-provider.md CriteriaProvider abstraction, filesystem vs REST backends
docs/architecture/skills.md Agent Skills spec, registration, resolution, delivery
docs/architecture/codebases.md Codebase entity, immutable revisions, source types, worker seeding
docs/architecture/db-migrations.md MongoDB migration framework
docs/architecture/cli-distribution.md CLI bundling, publishing, installation, update check
docs/architecture/retry.md Retry utilities: withRetry function and @Retry decorator
docs/architecture/post-processing.md Post-processing pipeline, ATIF generation, handler extensibility
docs/architecture/observability.md Application telemetry, Azure Monitor OTel distro, custom worker metrics
docs/architecture/kubedock.md Kubedock sidecar, container access for agents, Kustomize Component toggle
docs/research/realtime-data-flow.md SSE + Change Streams, Redis pub/sub, polling patterns
docs/research/delta-storage.md Space-efficient storage of iteration snapshots
docs/shared-dev-infra.md Shared dev infrastructure (CosmosDB) setup and worktree isolation
ENV_VARIABLES.md Environment variable reference