ruvbrain (pi.ruv.io): reclassify route missing, 3 schedulers timing out on /v1/pipeline/optimize, firestore backup invalid

[ruvnet]

Summary

Routine GCP review on 2026-05-15 found pi.ruv.io brain is up but most of its Cloud Scheduler optimization jobs have been failing for days. Root causes mixed: one dead route, three jobs racing the same overloaded endpoint, one unrelated Firestore backup config, and one unrelated Cloud Run service that never starts.

Findings

1. brain-reclassify-daily → dead route (NOT_FOUND, every 4h)

Scheduler brain-reclassify-daily POSTs https://pi.ruv.io/v1/reclassify, but that route does not exist in crates/mcp-brain-server/src/routes.rs (verified against the router build at lines 325–365 of routes.rs). Has been 404ing on every fire — at least back through 2026-05-14. Either restore the handler or delete/pause the scheduler.

2. Three schedulers hammer /v1/pipeline/optimize and time out

• brain-train — every 30m, mix of DEADLINE_EXCEEDED / RESOURCE_EXHAUSTED / UNKNOWN
• brain-full-optimize — daily 03:00, DEADLINE_EXCEEDED
• brain-graph — every 6h, DEADLINE_EXCEEDED / UNKNOWN

All three POST the same endpoint. The endpoint exists (routes.rs:365 → pipeline_optimize) but with revision ruvbrain-00203-brv (deployed 2026-04-14) on 2 vCPU / 4Gi the graph rebuild work (22k nodes, 1.2M edges) does not finish inside Cloud Scheduler's attempt deadline.

Options:

• Raise scheduler attemptDeadline on the three jobs (Scheduler's default is short)
• Split /v1/pipeline/optimize into work-class-specific paths so train/optimize/graph don't compete
• Bump Cloud Run memory/cpu and/or --no-cpu-throttling
• Move to async job pattern (scheduler kicks, returns 202; worker drains a queue)

3. firestore-daily-backup / firestore-weekly-backup → INVALID_ARGUMENT

Daily and weekly Firestore exports have failed every run for at least 5 days (most recent 2026-05-15 02:00Z). Likely stale arguments (bucket, collectionIds, or output URI) — not a brain-server problem, but breaks DR. Needs a one-line job redefinition.

4. rvagent-learning Cloud Run service won't start

Revision rvagent-learning-00001-kld never bound PORT=8080 within the health-check window. Scheduler rvagent-learning-daily keeps invoking it. Source isn't in this repo — needs to be traced to its owner (or the scheduler paused).

5. Stale brain deployment + paused optimization jobs

• Latest deployed revision ruvbrain-00203-brv is 31 days old (2026-04-14). Worth a fresh deploy.
• Per CLAUDE.md "7 optimization jobs running" — actually brain-drift, brain-transfer, brain-attractor, brain-partition-cache are PAUSED. Either intentional and CLAUDE.md is stale, or they should be re-enabled. Worth a one-line decision.

6. Embedding engine

/v1/status reports embedding_engine=ruvllm::HashEmbedder, embedding_dim=128, embedding_corpus=0. Hash embeddings, not real semantic vectors — separate (existing) work, but document the current state so consumers know brain_search recall is bounded.

Reproduction

curl -sS https://pi.ruv.io/v1/status | jq '.embedding_engine,.graph_nodes,.graph_edges'
gcloud scheduler jobs list --location=us-central1 \
  --format="table(name.basename(),state,schedule,status.code)"
gcloud logging read 'resource.type="cloud_scheduler_job" AND severity>=ERROR' \
  --limit=20 --freshness=24h --format=json | jq -r '.[].jsonPayload | "\(.targetType) \(.status) \(.url)"'

Proposed fix order (low-risk → high-risk)

1. Pause brain-reclassify-daily until route is restored or job is deleted (stops 404 spam)
2. Pause rvagent-learning-daily until owner re-attaches it (stops invoking broken service)
3. Bump attemptDeadline on brain-train / brain-full-optimize / brain-graph to 20–30 min
4. Re-deploy ruvbrain from current main so the 31-day staleness goes away
5. Fix firestore-daily-backup / firestore-weekly-backup arguments
6. Decide on the paused jobs (re-enable or delete from CLAUDE.md doc)

Environment

• GCP project: ruv-dev, region: us-central1
• Cloud Run: ruvbrain rev 00203-brv from gcr.io/ruv-dev/ruvbrain:latest@sha256:9f71c28f...
• Domain mappings healthy: pi.ruv.io → ruvbrain, mcp.pi.ruv.io → ruvbrain-sse

[ruvnet]

Update — fixes applied + root cause for Firestore

Applied (low-risk, reversible)

• ✅ Paused brain-reclassify-daily — was 404ing every 4h on /v1/reclassify (route doesn't exist in routes.rs)
• ✅ Paused rvagent-learning-daily — was invoking the broken rvagent-learning Cloud Run service
• ✅ Raised attemptDeadline on three brain optimize jobs to 1800s (Scheduler max):
  • brain-train: 60s → 1800s (was failing every 30m at 60s — graph rebuild alone takes minutes)
  • brain-full-optimize: 540s → 1800s
  • brain-graph: 540s → 1800s

Firestore backup — root cause identified, structural fix deferred

Reproduced the failure directly against the API:

POST https://firestore.googleapis.com/v1/projects/ruv-dev/databases/(default):exportDocuments
{"outputUriPrefix":"gs://ruvector-backups-daily/firestore"}
→ HTTP 400 INVALID_ARGUMENT
  "Path already exists: /ruvector-backups-daily/firestore/firestore.overall_export_metadata"

Firestore's exportDocuments API requires a unique outputUriPrefix per call — the first successful export wrote firestore.overall_export_metadata to that prefix; every subsequent attempt 400s.

gcloud firestore export works because the CLI auto-suffixes with a timestamp (firestore-manual-<epoch>). Cloud Scheduler bodies are static — no {{date}} templating — so the same prefix is sent every run.

Buckets, SA permissions, Firestore DB all healthy (verified manual export succeeded simultaneously). This is purely a static-body problem.

Sustainable fix options (pick one — needs decision)

Option	New infra	Effort	Cost
A. Cloud Function trigger	1 HTTP function (firestore-backup-trigger) constructs outputUriPrefix=gs://.../firestore/{YYYY-MM-DD} and calls Admin API; scheduler invokes the function	Lowest	Free tier
B. Cloud Workflow	Enable Workflows API + create workflow with ${time.now()} templating; scheduler invokes workflow	Low	Free tier
C. Cloud Run job	Tiny container that runs gcloud firestore export gs://.../firestore/$(date +%F)	Medium	Run-cost
D. Status quo + GCS lifecycle	Won't work — the "path exists" check is point-in-time; lifecycle deletes don't run between scheduler attempts

Recommend A (Cloud Function in Rust if we want zero-Python on the team, or a 10-line Node function). Holding off until directed.

Deferred / informational

• ruvbrain revision 00203-brv is 31 days old — rebuild + redeploy queued separately.
• rvagent-learning service source is not in this repo — needs an owner to claim it or scheduler stays paused.
• embedding_engine=ruvllm::HashEmbedder (dim=128) — known, separate work.
• Paused optimization jobs (drift / transfer / attractor / partition-cache / pubmed-daily / crawl-*) — CLAUDE.md docs claim 7 running, reality is fewer; needs a docs/reality reconciliation.

[ruvnet]

ruvbrain rebuild attempted — rolled back (real regression)

Built from main@9054c2cc6 via Cloud Build c18a0b4e-e0da-4d47-9052-8a2fa0e48d7b (6 min, succeeded). Deployed as ruvbrain-00204-92l. Rolled back to 00203-brv after a real regression was confirmed.

Evidence

Both revisions logged their Firestore hydration after this rollout. Hydration is the slow path (~3–4 min after cold start) and is the source of /v1/status memory count.

Revision	Hydration log	Memories loaded
ruvbrain-00203-brv (deployed 2026-04-14)	18:28:53Z	22,005
ruvbrain-00204-92l (built from main today)	18:30:17Z	10,227

That's a ~54% drop in what load_from_firestore() returns. Same Firestore project + database + collection, two consecutive runs minutes apart on the same instance pool — only the binary differs.

Likely causes (not yet narrowed down)

Something between commit 9054c2cc6 and the SHA of the image inside 00203-brv changes hydration filtering. Candidates from crates/mcp-brain-server/src/store.rs:

• A new quality / dedup / validation filter in load_from_firestore()
• ADR-149 P2 quality-floor logic now applied during hydration (not just edge build)
• Pagination/limit defaults changed
• Schema deserializer became stricter and silently drops malformed records

Status now

• ✅ Traffic 100% on ruvbrain-00203-brv — /v1/status reports 22,005 memories / 1,195,392 edges, healthy.
• ❌ ruvbrain-00204-92l exists but receives 0% traffic — safe to leave or delete.
• 🛑 Do not re-route to 00204 without a code investigation of the hydration filter regression.

Next steps for whoever picks this up

1. git log --oneline 2026-04-14..main -- crates/mcp-brain-server/src/store.rs crates/mcp-brain-server/src/types.rs to find candidate changes
2. Add a count metric inside load_from_firestore() for (considered, accepted, rejected_quality, rejected_invalid, rejected_other) so the next deploy is measurable
3. Re-deploy after the regression is identified + tested