feat(oie): RQ A→B→C fan-out and Admin OIE queue tab

.cursor/skills/brevity-bot/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: brevity-bot
+description: >-
+  Review a diff like a long-tenured greybeard coworker: shorten AI blurbs,
+  simplify code, keep correctness/security/Bugbot fixes. Use after Bugbot and
+  Security reviews, or when asked for /review-brevity.
+---
+
+# Brevity bot
+
+You are the ponytailed, greybearded, Linux-using god of coworker who has been
+with the company since the beginning. You can shorten or simplify any codebase.
+Your job is to dramatically increase readability and quality and shorten AI
+blurbs while preserving correctness, quality, and security.
+
+You simplify code **without dropping the fixes** from Bugbot or Security
+review. If a check, error path, redaction, auth gate, or test exists because
+those reviews required it, keep it. Prefer a shorter form of the same fix.
+
+## Diff contract (same as Bugbot)
+
+The parent gives:
+
+```text
+Full Repository Path: <absolute path>
+Diff: <branch changes | uncommitted changes>
+```
+
+Review that diff only. Do not invent drive-by refactors outside it.
+
+## What to cut
+
+- Restated comments, essay docstrings, "as an AI" padding, changelog comments
+- Needless wrappers, duplicate branches, unused helpers, speculative abstraction
+- Tests that only restate the same assertion five ways
+
+## What to keep
+
+- Auth, allowlists, redaction, abort codes, CSRF/login gates
+- Bugbot/Security fixes (narrower code is fine; weaker checks are not)
+- Names and structure that match this repo (`application/tests/`, Makefile)
+
+## Output
+
+Either:
+
+1. **Patches** — file path, short why, and the smaller replacement, or
+2. **`NO_COMMENTS`** — nothing left to shorten without harming clarity, tests,
+   or security.
+
+End with exactly one of:
+
+- `VERDICT: NO_COMMENTS`
+- `VERDICT: COMMENTS` (and the patch list)
+
+Do not praise the diff. Do not recap the plan.

.cursor/skills/brevity-bot/examples.md

@@ -0,0 +1,40 @@
+# Brevity bot fixtures
+
+## AI blurb → short
+
+Before:
+
+```python
+def is_enabled() -> bool:
+    """Return True if the feature flag is enabled.
+
+    This function checks the environment variable and compares it against
+    a set of truthy values so that operators can enable the feature.
+    """
+    value = os.getenv("FLAG", "")
+    normalized = value.strip().lower()
+    if normalized in {"1", "true", "yes"}:
+        return True
+    return False
+```
+
+After:
+
+```python
+def is_enabled() -> bool:
+    return os.getenv("FLAG", "").strip().lower() in {"1", "true", "yes"}
+```
+
+## Do not drop a security fix
+
+Security required redacting secrets. Illegal brevity (drops the fix):
+
+```python
+return {"GEMINI_API_KEY": os.environ.get("GEMINI_API_KEY")}
+```
+
+Legal brevity (keeps redaction):
+
+```python
+return {"GEMINI_API_KEY": "***"} if os.environ.get("GEMINI_API_KEY") else {}
+```

.cursor/skills/review-brevity/SKILL.md

@@ -0,0 +1,24 @@
+---
+name: review-brevity
+description: Launch Brevity bot on the current diff after Bugbot and Security.
+---
+
+# Review Brevity
+
+When asked to run `/review-brevity` or Brevity bot:
+
+Launch **three** independent `generalPurpose` subagents (`run_in_background: false`)
+with the contents of `.cursor/skills/brevity-bot/SKILL.md` as instructions.
+
+Prompt each with:
+
+```text
+Full Repository Path: <absolute repository path>
+Diff: branch changes
+```
+
+Default to `branch changes`. Use `uncommitted changes` only if asked.
+
+Collect the three `VERDICT:` lines. Majority `NO_COMMENTS` (2 of 3) is clean.
+If majority is `COMMENTS`, apply agreed simplifications that do **not** revert
+Bugbot or Security fixes, then re-run.

.dockerignore

@@ -5,6 +5,11 @@ tmp
 .vscode
 .github
 .git
+.harvester_cache/
+logs/
+.cursor/
+.cursorignore
+*.nohup.out
 
 # This file specifies files that are *not* uploaded to Google Cloud
 # using gcloud. It follows the same syntax as .gitignore, with the addition of

.env.example

@@ -5,6 +5,10 @@
 # Never commit your real `.env` file to version control.
 # ------------------------------------------------------------------------------
 # Database
+# Local Docker (make docker-postgres / make admin-local):
+#   postgresql://cre:password@127.0.0.1:5432/cre
+# `make admin-local` starts that container when the URL is unreachable and
+# never falls back to SQLite for the admin showcase.
 
 DEV_DATABASE_URL=postgresql://user:password@localhost:5432/opencre
 
@@ -42,14 +46,17 @@ CRE_ENABLE_MYOPENCRE=false
 CRE_ENABLE_LOGIN=false
 
 # Experimental OIE OWASP metadata agent (chat + MCP only — never CRE graph UI).
-# Off by default. When enabled, community/meta questions are answered from
-# the local owasp_agent index instead of CRE RAG. Concepts/tags stay in
-# OWASP_AGENT_DB and are reachable only from /completion and MCP owasp_meta_* tools.
+# Off by default. Uses the main app Postgres (DEV_DATABASE_URL / DATABASE_URL)
+# for the agent index + pgvector. Reachable from /completion and MCP owasp_meta_* tools.
 OWASP_AGENT_ENABLED=false
+
+# Module C.1 graph filing (Automatically linked to). Off by default — leave Module C
+# decisions in decision_queue for review; set true to auto-add high-confidence links.
+OIE_GRAPH_FILING_ENABLED=false
+
 # Nest REST API key (https://nest.owasp.org — create via GraphQL while logged in)
 NEST_API_KEY=
 NEST_API_BASE=https://nest.owasp.org/api/v0
-OWASP_AGENT_DB=tmp/owasp_agent.sqlite
 # Set to 1 to crawl all www-chapter / www-project repos during sync (slow).
 OWASP_AGENT_CRAWL_REPOS=0
 

.gitignore

@@ -49,6 +49,8 @@ v/
 .cursor/*
 !.cursor/rules/
 !.cursor/rules/**
+!.cursor/skills/
+!.cursor/skills/**
 .claude/
 
 ### Frontend 
@@ -67,11 +69,13 @@ standards_cache.sqlite
 ### Docs
 *.md
 !AGENTS.md
+!.cursor/skills/**/*.md
 !docs/faq.md
 !docs/api/mcp.md
 !docs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md
 !docs/Final_eval_blog_gsoc2026/module_B_final_blog.md
 !application/utils/librarian/README.md
+!application/utils/owasp_agent/README.md
 !docs/gsoc_2026_module_c/*.md
 !docs/gsoc_2026_module_a/*.md
 !docs/Final_eval_blog_gsoc2026/*.md
@@ -93,6 +97,8 @@ neo4j/
 .mypy_cache
 tmp/
 
+import_telemetry.json
+
 ### CREs dir
 cres/*
 ### Local project management tooling

Dockerfile

@@ -10,7 +10,10 @@ FROM python:3.11 as run
 COPY --from=build /code /code
 WORKDIR /code
 COPY ./scripts/prod-docker-entrypoint.sh /code
-RUN pip install -r requirements.txt gunicorn
+# Staging Compose may pass requirements-dev.txt (Librarian ML / OIE Module C.2).
+# Default stays prod requirements.txt for Heroku slug size.
+ARG REQUIREMENTS_FILE=requirements.txt
+RUN pip install -r "${REQUIREMENTS_FILE}" gunicorn
 
 ENV INSECURE_REQUESTS=1
 ENV FLASK_CONFIG="production"

Makefile

@@ -1,6 +1,6 @@
 .ONESHELL:
 
-.PHONY: run test covers install-deps dev docker lint frontend clean all e2e e2e-db owasp-agent-sync owasp-agent-merge-concepts
+.PHONY: run test covers install-deps dev dev-flask docker lint frontend clean all e2e e2e-db admin-local owasp-agent-sync owasp-agent-merge-concepts owasp-agent-synth-eval
 
 prod-run:
 	gunicorn cre:app --log-file=-
@@ -70,15 +70,39 @@ docker-postgres:
 
 start-containers: docker-neo4j docker-redis
 
+# RQ work-horses fork after Flask/torch load. On macOS, unset device → MPS and
+# CrashReporter popups from Homebrew Python.app. Force CPU unless overridden.
+CRE_LIBRARIAN_DEVICE?=cpu
 start-worker:
-	. ./venv/bin/activate && FLASK_APP=`pwd`/cre.py python cre.py --start_worker
+	. ./venv/bin/activate && \
+	OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES \
+	CRE_LIBRARIAN_DEVICE="$(CRE_LIBRARIAN_DEVICE)" \
+	FLASK_APP=`pwd`/cre.py python cre.py --start_worker
+
+# Local OIE RQ fleet (queue ``oie``). Default 10; override with OIE_WORKER_COUNT=N.
+# Double-fork via scripts/start_oie_workers.py so workers survive make exit (SIGHUP).
+OIE_WORKER_COUNT?=10
+start-oie-workers:
+	. ./venv/bin/activate && \
+	OIE_WORKER_COUNT="$(OIE_WORKER_COUNT)" \
+	CRE_LIBRARIAN_DEVICE="$(CRE_LIBRARIAN_DEVICE)" \
+	OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES \
+	python scripts/start_oie_workers.py
 
 upstream-sync:
-	. ./venv/bin/activate && python cre.py --upstream_sync
+	. ./venv/bin/activate &&\
+	python cre.py --upstream_sync --cache_file "$(or $(DEV_DATABASE_URL),$(CRE_CACHE_FILE),$(CURDIR)/standards_cache.sqlite)"
 
 PORT?=5000
 
-dev-flask:
+# Local app against Docker Postgres: migrate, upstream-sync if cre/node empty, Flask.
+# Same entrypoint as admin showcase / OIE local (no SQLite). macOS AirPlay often
+# holds :5000 — use PORT=5001 make dev if bind fails.
+dev dev-flask admin-local:
+	bash ./scripts/run_admin_local.sh --port $(PORT) $(ADMIN_LOCAL_ARGS)
+
+# Bare Flask without Postgres bootstrap (legacy / CI helpers). Prefer `make dev`.
+dev-flask-sqlite:
 	. ./venv/bin/activate && INSECURE_REQUESTS=1 FLASK_APP=`pwd`/cre.py  FLASK_CONFIG=development flask run --port $(PORT)
 
 dev-flask-docker:
@@ -136,7 +160,14 @@ install-deps-typescript:
 install-deps: install-deps-python install-deps-typescript
 
 install-python:
-	virtualenv -p python3 venv
+	@if [ ! -d "./venv" ]; then \
+		if command -v virtualenv >/dev/null 2>&1; then \
+			virtualenv -p python3 venv; \
+		else \
+			echo "virtualenv not on PATH — using python3 -m venv"; \
+			python3 -m venv venv; \
+		fi; \
+	fi
 	. ./venv/bin/activate &&\
 	make install-deps-python &&\
 	playwright install  # Python embeddings/scraping (prompt_client); NOT frontend e2e — keep when migrating to Cypress
@@ -148,7 +179,9 @@ install-python:
 # `make install`. Rebuild with `make frontend` when you change the UI.
 install-typescript: install-deps-typescript
 
-install: install-typescript install-python migrate-upgrade
+# Deps + Docker Postgres schema + CRE upstream graph when empty (see run_admin_local.sh).
+install: install-typescript install-python
+	bash ./scripts/run_admin_local.sh --migrate-only $(ADMIN_LOCAL_ARGS)
 
 docker-dev:
 	docker build -f Dockerfile-dev -t opencre-dev:$(shell git rev-parse HEAD) .
@@ -217,6 +250,11 @@ owasp-agent-merge-concepts:
 	[ -d "./venv" ] && . ./venv/bin/activate &&\
 	PYTHONPATH=. python scripts/owasp_agent_merge_concepts.py $(OWASP_AGENT_MERGE_ARGS)
 
+# Score fixtures/owasp_agent_synth_v2.jsonl against OWASP_AGENT_DB (see application/utils/owasp_agent/README.md).
+owasp-agent-synth-eval:
+	[ -d "./venv" ] && . ./venv/bin/activate &&\
+	OWASP_AGENT_ENABLED=1 PYTHONPATH=. python scripts/owasp_agent_synth_eval.py
+
 openapi-generate:
 	[ -d "./venv" ] && . ./venv/bin/activate &&\
 	python scripts/generate_openapi.py
@@ -268,6 +306,21 @@ monitor-chatbot-health-prod:
 		--base-url "https://opencre.org" \
 		--output-json "$(CURDIR)/tmp/prod-chatbot-health.json"
 
+# Scheduled OIE jobs (Postgres only). Cron `make oie-tick` every 10 minutes: each
+# job runs once per slot (owasp every 10 min, cre_expansion daily) and is a no-op
+# otherwise. Overrides: OIE_JOB=owasp|cre_expansion|all, OIE_FORCE=1, OIE_DRY_RUN=1.
+oie-tick:
+	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
+	export CRE_CACHE_FILE="$${CRE_CACHE_FILE:-postgresql://cre:password@127.0.0.1:5432/cre}"; \
+	python cre.py --cache_file "$$CRE_CACHE_FILE" --run_scheduled "$${OIE_JOB:-all}" \
+		$${OIE_FORCE:+--scheduled_force} $${OIE_DRY_RUN:+--scheduled_dry_run}
+
+monitor-oie-health:
+	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
+	export CRE_CACHE_FILE="$${CRE_CACHE_FILE:-postgresql://cre:password@127.0.0.1:5432/cre}"; \
+	export PYTHONPATH="$(CURDIR)"; \
+	python scripts/monitor_oie_health.py --output-json "$(CURDIR)/tmp/oie-health.json"
+
 verify-ga-parity-local:
 	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
 	export CRE_CACHE_FILE="$${CRE_CACHE_FILE:-postgresql://cre:password@127.0.0.1:5432/cre}"; \

README.md

@@ -49,44 +49,33 @@ If you want to develop on OpenCRE or docker is not available in your environment
 ### Command Line
 
 To run outside of Docker you need to install OpenCRE.
-To install this application you need Python 3.11.9, Node.js, Yarn Classic (the repository uses a Yarn v1 lockfile), and `virtualenv`. The Python version is pinned in `.python-version`; the repository does not currently pin a Node.js version.
+To install this application you need Python 3.11.9, Node.js, Yarn Classic (the repository uses a Yarn v1 lockfile), and Docker (local Postgres). `virtualenv` is optional — `make install-python` falls back to `python3 -m venv`. The Python version is pinned in `.python-version`; the repository does not currently pin a Node.js version.
 
 Clone the repository:
 
 ```bash
 git clone https://github.com/OWASP/OpenCRE
 ```
 
-`make install` creates and uses the project virtual environment. Do not create a second environment with `python3 -m venv`; install the prerequisites above, then let the Makefile manage `venv`.
-
-Install dependencies:
+`make install` creates the project `venv` (if missing), installs deps, starts Docker Postgres when needed, migrates, and pulls the CRE graph from opencre.org when the local DB is empty.
 
 ```bash
 make install
+make dev                 # same as make dev-flask / make admin-local
 ```
 
-`make install` creates the local schema as part of installation. Download the latest CRE graph from upstream with:
-
-```bash
-make upstream-sync
-```
+Then open http://127.0.0.1:5000/admin and http://127.0.0.1:5000/explorer. If port 5000 is busy (common on macOS with AirPlay Receiver), use `PORT=5001 make dev`. To reuse an existing DB without re-pulling the CRE graph: `SKIP_UPSTREAM_SYNC=1 PORT=5001 make dev`.
 
 For contributors, this is the supported local data bootstrap path. You do
 **not** need access to the internal OpenCRE Google Sheet to work on the
-project.
+project. From **Admin → Config**, use **Run golden-set harvest** and **Sync OWASP agent** (progress under Pipeline), or `make owasp-agent-sync` from the CLI.
 
 You can precompute local gap-analysis cache after imports with:
 
 ```bash
 make backfill-gap-analysis
 ```
 
-To run CRE locally then you can do:
-
-```bash
-make dev-flask
-```
-
 To run the CLI application, you can run:
 
 ```bash