PLE audit: readiness for LLM-velocity development

[jwulf]

PLE Repository Audit — readiness for LLM-velocity development. Produced by the Production Line Engineering (PLE) audit skill. Every finding is evidence-based (config lines, CI, tests, issues, rulesets observed on main).

1. Summary

c8ctl is a near-exemplar of LLM-velocity engineering discipline — one of the strictest codebases audited. Correctness is pushed into the type system and enforced mechanically: noExplicitAny, noNonNullAssertion, noTsIgnore are all error; two custom Grit plugins (no-unsafe-type-assertion, no-process-exit-in-commands) ban casts and enforce architectural boundaries; an architectural layering-import guard and a shrink-only suppression ratchet turn norms into executable tests. Commands are a single declarative source of truth (COMMAND_REGISTRY), from which help, completions, arg-parsing, and validation are all derived. The AGENTS.md is comprehensive and its policies are largely backed by real guards. 121 tests (100 unit / 21 integration) run across a Node 22/24 × Camunda 8.8/8.9 matrix.

Two gaps stand between this repo and a genuinely closed-loop system, and both are about enforcement reaching the merge boundary: (1) the main ruleset requires only Copilot review — CI status checks are not required to merge, so the excellent lint/typecheck/test suite is advisory; and (2) issue #271 is an acknowledged intermittent CI failure open for ~3 months, a direct contradiction of the repo's own "there are no flaky tests" policy. Close those and the system's guarantees become the thing humans trust, rather than something a reviewer must still watch for.

---

2. Per-lens findings

Lens 1 — Human attention is the bottleneck

State: Adequate

• Evidence: The generation→merge path has strong automated gates (lint, typecheck, layering guard, suppression ratchet, unit+integration matrix). But the main ruleset (auto copilot reviews) contains only deletion, non_fast_forward, copilot_code_review — no required_status_checks (no classic protection either: branches/main/protection → 404). So a human/Copilot review is the load-bearing gate; CI is informational.
• Remediation: Add a required_status_checks rule pinning Lint, Typecheck, Unit Test, Integration Test. This is the single highest-leverage change — it makes the already-excellent suite actually block bad merges.

Lens 2 — Remove review work that doesn't change the outcome

State: Adequate

• Evidence (good): .github/dependabot.yml runs daily npm + weekly actions updates — CVE default-case handling is delegated. sync:readme/sync:docs derive docs from the registry, removing manual doc-sync review.
• Evidence (gap): No npm audit/CodeQL/Snyk job, and no Dependabot auto-merge config, so security bumps still need a manual green-rubber-stamp merge (which, per Lens 1, isn't even gated on CI).
• Remediation: Enable Dependabot/Renovate auto-merge for patch/minor once CI is green (requires Lens 1 first) and add an async npm audit --audit-level=high job.

Lens 3 — Harden the codebase for correctness at velocity

State: Strong — the repo's standout dimension.

• Type safety: tsconfig strict: true; a dedicated typecheck (tsc --noEmit -p tsconfig.check.json) covers src, tests, scripts, and plugins. Biome bans any/!/@ts-ignore at error. Whole hand-written src has only 1 as any, 1 as unknown, 0 @ts-ignore. Casts are banned repo-wide by a custom Grit plugin, with the 20 remaining biome-ignore lint/plugin escape hatches frozen by a shrink-only ratchet test (tests/unit/no-plugin-ignore-boundary.test.ts) and tracked for removal (Retire the frozen biome-ignore lint/plugin suppressions behind typed accessors #472).
• Warnings: No warn-level rules — everything is error; npm run lint runs in CI as a blocking job (modulo Lens 1).
• Derive, don't duplicate: COMMAND_REGISTRY is the single source of truth (AGENTS.md:452); help/completions/parsing/validation all derived.
• Boundaries: check:layering (import-boundary guard) + no-process-exit-in-commands.grit enforce architecture in CI, not just style.
• Remediation: Almost nothing. Continue the Retire the frozen biome-ignore lint/plugin suppressions behind typed accessors #472 burn-down; consider noUncheckedIndexedAccess in tsconfig to retire the "trust-boundary indexing" suppressions structurally.

Lens 4 — Scale testing with generation speed

State: Adequate

• Evidence: 121 tests with genuine adversarial coverage (deploy-error-paths, plugin-list-collisions, guard/boundary tests) plus a real integration matrix across two Camunda versions. Class-scoped guards exist (help-scoping help: scope top-level c8ctl --help Flags section to global flags only #321, layering, suppression ratchet). No coverage measurement is configured (no --experimental-test-coverage), so coverage-vs-generation-speed is unobserved.
• Remediation: Add node --test --experimental-test-coverage with a ratchet threshold to make regression-coverage visible.

Lens 5 — Move review up a layer

State: Strong

• Evidence: The system is highly legible (38KB AGENTS.md, IMPLEMENTATION.md, CONTEXT.md, declarative registry), and review gravity has genuinely shifted to boundaries/generators — recent commits are guard tests and typed-accessor refactors (refactor(element-template): route moddle reads through typed accessors), not line fixes. Required Copilot review reinforces this.
• Remediation: None material; ensure CI gating (Lens 1) so reviewers aren't also silently re-checking mechanical correctness.

Lens 6 — Treat flaky behavior as a production-line defect

State: At risk — the one clear policy violation.

• Evidence: Issue MCP proxy mock integration test intermittently failing in CI #271 "MCP proxy mock integration test intermittently failing in CI" has been open since 2026-04-16 (~3 months) — TypeError: fetch failed from a mock server not yet reachable (a classic readiness race). AGENTS.md:321 explicitly states such a failure "is a real defect that must be diagnosed and fixed before the change merges." The policy is right; the practice has drifted. (Positively: git history shows zero revert/retry/flaky commits — no re-run-and-hope pattern in merges.)
• Remediation: Prioritize MCP proxy mock integration test intermittently failing in CI #271: replace the readiness poll with a deterministic server-listening signal (await the listening event / a health probe before the request), then close it. Treat the age of MCP proxy mock integration test intermittently failing in CI #271 as the metric to drive to zero.

Lens 7 — Treat CI as part of the product system

State: Adequate

• Evidence: Good structure — unit-test needs: [lint, typecheck] (fail fast on cheap checks before the matrix), fail-fast: false for full signal, integration isolated per Camunda version with log capture on failure. But high-yield checks are not merge-blocking (Lens 1), and the integration wait loop ends with "...continuing with tests anyway" — a soft readiness assumption that likely feeds flakes like MCP proxy mock integration test intermittently failing in CI #271. CI cycle time isn't tracked as a budget.
• Remediation: Make lint/typecheck/unit/integration required; replace the "continue anyway" fallback with a hard fail on unready broker; record and watch p50/p95 CI duration.

AGENTS.md conformance

State: Strong (policy) — one enforcement gap.

Requirement	Status	Evidence
Red/Green + defect-class guards	✅	AGENTS.md:310
Reject flaky tests	✅ stated / ❌ practiced	AGENTS.md:314–321 vs open #271
Reject pre-existing-failure excuse	✅	AGENTS.md:183
Single source of truth / derivation	✅	AGENTS.md:56, 452
Ban warnings/errors/test failures	✅	AGENTS.md:182; Biome all-error
Fix defect class, not instance	✅	AGENTS.md:310; ratchet + layering guards
Tracked issue/PR before work	⚠️	Not explicitly required in AGENTS.md; #320 actively closing this

The written norms are essentially complete; the only real conformance failures are practiced, not documented: the un-fixed flaky test (#271) and the missing tracked-work-before-start clause (#320).

---

3. Top recommendations (by attention reclaimed per unit of effort)

1. Add required_status_checks to the main ruleset (Lint, Typecheck, Unit, Integration). Config-only; instantly converts an already-excellent suite from advisory to guaranteed. Highest leverage in the repo.
2. Root-cause and close MCP proxy mock integration test intermittently failing in CI #271, and harden the integration wait loop (deterministic readiness signal, remove "continue anyway"). Reclaims the attention currently spent tolerating an intermittent failure and upholds the repo's own policy.
3. Enable Dependabot auto-merge for green patch/minor + add an async npm audit job. Automates the CVE default case (depends on feat: implement c8ctl CLI for Camunda 8 orchestration #1).
4. Add test-coverage measurement with a ratchet. Makes coverage-vs-velocity observable.
5. Land AI-First compliance: AGENTS.md baseline #320's "tracked issue before work" clause into AGENTS.md to close the last conformance gap.

Overall: architecturally and procedurally near-best-in-class — the correctness machinery is exceptional. The remaining work is almost entirely about extending that machinery the last inch to the merge boundary and honoring the no-flaky-tests policy the repo already espouses.

---

_{Generated by the PLE repo-audit skill.}

[jwulf]

Correction — Lens 6 (flaky tests): re-verified against CI history

Follow-up check on the one flagged flaky-test violation (#271) shows it is already resolved, so the Lens 6 rating should be revised upward.

• MCP proxy mock integration test intermittently failing in CI #271 was fixed by test(mcp-proxy-mock): random port range includes undici "bad port" 10080 #316 (commit 15d23f6, fix(tests): bind mock servers to kernel-assigned ports, 2026-04-28) — mock/probe servers now bind to a kernel-assigned port (listen(0)) to dodge undici's "bad ports" rejection list, the real root cause of the fetch failed flake. The issue simply hadn't been closed; I've now closed it.
• CI evidence: last 40 Test runs (back to 2026-06-30) had only 2 failures — a Biome lint error and a shell-completion unit test — and zero MCP proxy mock failures.

Revised Lens 6 state: Strong. There is now no observed tolerance for flaky tests: the one open intermittent-failure report was a stale, already-fixed issue, git history shows zero re-run/revert/retry merge commits, and the defect class (undici bad-port binding) was fixed structurally via listen(0).

This makes recommendation #2 largely moot; the remaining top lever is unchanged: add required_status_checks to the main ruleset so the (excellent) CI suite becomes merge-blocking.