feat(oie): RQ A→B→C fan-out and Admin OIE queue tab by northdpole · Pull Request #1142 · OWASP/OpenCRE

northdpole · 2026-10-09T08:36:59Z

Summary

Add RQ queue oie fan-out for Module A (per repo) → B/C (per artifact_id), with --rq / CRE_OIE_RQ, start_oie_workers, and harvest_input.artifact_id migration.
Admin OIE queue tab + shared RQ dashboard snapshot; wire import-all / workers for the oie queue.
Fix B2 eval harvester LangGraph repos_yaml TypeError so top10/llm/api score arms can run; keep promoted librarian/safety knobs usable on RQ workers.

Test plan

python -m unittest application.tests.oie_rq.fanout_test application.tests.oie_rq.stage_filter_test application.tests.oie_b2_gold_loader_test.TestOieB2HarvesterLanggraphKwargs
make alembic-guardrail (new revision a9b0c1d2e3f4)
Local: make start-oie-workers + scripts/run_oie_pipeline.py --rq (or full_pipeline --rq) drains queue without silent B2 TypeError
Admin OIE tab loads /admin/oie/rq/status with queue depth

Stacked on #1137 (admin-panel-showcase).

Made with Cursor

coderabbitai · 2026-10-09T08:37:10Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: OWASP/OpenCRE/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: f716ca97-5a3a-4c88-868d-5f34252a637c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Parallelize Module A/B/C on queue `oie` (artifact_id migration, workers, CLI --rq), fix B2 harvester LangGraph repos_yaml TypeError, and keep promoted librarian/safety knobs available to RQ workers.

Lowercase project tables, tighten refuse/fail-closed routing, and add a laptop demo runbook so synth stays above 90% with Nest sync.

Add LOGIN_ALLOWED_EMAILS so only Spyros/Rob/Rock can establish sessions on internet-facing staging.

Record staging.opencre.org tunnel/DNS layout and LOGIN_ALLOWED_EMAILS env for the three OWASP operators.

Ship deploy/staging with own Postgres/Redis/web/worker, skip-sync entrypoint knobs, and pathspec so the image boots behind the Cloudflare tunnel.

Workers, Admin ingest, and the full-pipeline harness now share apply_promoted_defaults() (pgvector, margin γ=0.85, judge, safety, leaf drilldown). Staging Compose inherits the same knobs; image builds can pull requirements-dev for C.2.

Mint/verify keys for allowlisted operators, audit log + migrations, and stop caching SPA bundles so admin deploys pick up new JS.

Move shortlist/edition/neighbor/safety caches into Postgres, keep edition remap live-LLM off by default, batch safety checks, and gate C.1/C.2 for ablation. Includes accuracy-lever modules, staging compose tweaks, and admin upstream-sync+ingest helpers.

When both fail-closed levers are on, Module C issues a single combined judge call instead of two. Stop oie.env from overwriting worker DEV_DATABASE_URL so parallel accuracy-matrix scoring hits cre_al_wN instead of empty cre.

Keep edition-remap live LLM off, safety batching on, and experimental fail-closed levers off unless an eval explicitly enables them.

Lets operators recover stalled Module C work via /admin/oie/rq without docker or redis CLI.

Ship sidebar shell, needs-attention/review-pulse dashboard, paginated staged-link review with title/content, workbench targets, and GET /admin/config/diff — all still gated by login + CRE_ALLOW_IMPORT.

Staging monitors need API-key access to /admin/oie and related ops routes; also resolve CRE links by uuid or external_id and align ingest test asserts with skip_c1/max_repos=0. Co-authored-by: Cursor <cursoragent@cursor.com>

Operators can see per-run start flags, live HI/KQ/RQ remaining, ETA, and stuck detection on the Pipeline tab (and a strip on Overview).

Remaining/ETA now track KQ+RQ only and use snap-to-snap drain rate so old ImportRuns stop inventing multi-day estimates.

Main landed MCP v2 (f3a91c07d24b) on 967016ee10fa; keep a single Alembic head by parenting the stack's oie_run revision after mcp_grants.

login_or_mcp_required now accepts allowlisted API JWTs as well as session/MCP identity, and API keys carry ALL_SCOPES for require_scopes.

northdpole requested review from Pa04rth, paoga87 and robvanderveer as code owners October 9, 2026 08:37

northdpole force-pushed the admin-panel-showcase branch from e114292 to 4178ecf Compare October 11, 2026 23:04

northdpole force-pushed the feat/oie-rq-fanout branch from 723d697 to cba71ba Compare October 11, 2026 23:05

northdpole mentioned this pull request Oct 11, 2026

Fix OWASP agent probe gaps (chapters, people, membership, CRE facts) #1127

Open

4 tasks

northdpole force-pushed the admin-panel-showcase branch from 4178ecf to aad929a Compare October 12, 2026 07:05

northdpole force-pushed the feat/oie-rq-fanout branch from cba71ba to 2a1c75d Compare October 12, 2026 07:05

northdpole force-pushed the admin-panel-showcase branch from aad929a to 475945f Compare October 12, 2026 07:17

northdpole force-pushed the feat/oie-rq-fanout branch from 2a1c75d to 545522b Compare October 12, 2026 07:17

northdpole force-pushed the admin-panel-showcase branch from 475945f to 48e9d4f Compare October 12, 2026 07:19

northdpole force-pushed the feat/oie-rq-fanout branch 3 times, most recently from d68ddbc to 7aac2f5 Compare October 12, 2026 22:19

northdpole force-pushed the admin-panel-showcase branch from 48e9d4f to 30a2a40 Compare October 12, 2026 22:19

northdpole mentioned this pull request Oct 13, 2026

Admin panel redesign (CRE blues) on OIE RQ fanout tip #1146

Merged

4 tasks

northdpole and others added 13 commits October 14, 2026 09:39

Add OIE RQ A→B→C fan-out and Admin OIE queue tab.

27ff362

Parallelize Module A/B/C on queue `oie` (artifact_id migration, workers, CLI --rq), fix B2 harvester LangGraph repos_yaml TypeError, and keep promoted librarian/safety knobs available to RQ workers.

Harden OWASP agent for Nest-backed November demo.

0084e53

Lowercase project tables, tighten refuse/fail-closed routing, and add a laptop demo runbook so synth stays above 90% with Nest sync.

Restrict staging login to an explicit email allowlist.

18b3dae

Add LOGIN_ALLOWED_EMAILS so only Spyros/Rob/Rock can establish sessions on internet-facing staging.

Document homeserver staging behind Cloudflare Tunnel.

ba2685a

Record staging.opencre.org tunnel/DNS layout and LOGIN_ALLOWED_EMAILS env for the three OWASP operators.

Add dedicated Compose stack for staging.opencre.org.

59f699a

Ship deploy/staging with own Postgres/Redis/web/worker, skip-sync entrypoint knobs, and pathspec so the image boots behind the Cloudflare tunnel.

Add admin API tokens and JWT request auth.

3e13d40

Mint/verify keys for allowlisted operators, audit log + migrations, and stop caching SPA bundles so admin deploys pick up new JS.

Pin cost-free Module C path in promoted defaults and staging.

b2fc19e

Keep edition-remap live LLM off, safety batching on, and experimental fail-closed levers off unless an eval explicitly enables them.

Add Admin REST + UI to requeue/cancel failed OIE RQ jobs.

28a982c

Lets operators recover stalled Module C work via /admin/oie/rq without docker or redis CLI.

Redesign admin panel with CRE blues, env/.env diff, and review widgets.

9ba0b08

Ship sidebar shell, needs-attention/review-pulse dashboard, paginated staged-link review with title/content, workbench targets, and GET /admin/config/diff — all still gated by login + CRE_ALLOW_IMPORT.

Allow Bearer on Admin OIE REST and fix staging filing lookup.

e13fdbc

Staging monitors need API-key access to /admin/oie and related ops routes; also resolve CRE links by uuid or external_id and align ingest test asserts with skip_c1/max_repos=0. Co-authored-by: Cursor <cursoragent@cursor.com>

northdpole added 4 commits October 14, 2026 09:40

Add Admin active ingestions status with config, progress, and ETA.

6571c71

Operators can see per-run start flags, live HI/KQ/RQ remaining, ETA, and stuck detection on the Pipeline tab (and a strip on Overview).

Fix Admin ingest ETA to use Module C remaining, not HI backlog.

3094728

Remaining/ETA now track KQ+RQ only and use snap-to-snap drain rate so old ImportRuns stop inventing multi-day estimates.

chore(db): chain oie_run migration after mcp_grants

54db4d9

Main landed MCP v2 (f3a91c07d24b) on 967016ee10fa; keep a single Alembic head by parenting the stack's oie_run revision after mcp_grants.

fix(auth): admit admin API keys on MyOpenCRE after MCP rebase

1d39a8b

login_or_mcp_required now accepts allowlisted API JWTs as well as session/MCP identity, and API keys carry ALL_SCOPES for require_scopes.

northdpole force-pushed the feat/oie-rq-fanout branch from 404293e to 1d39a8b Compare October 14, 2026 08:42

northdpole force-pushed the admin-panel-showcase branch from 30a2a40 to 63d548b Compare October 14, 2026 08:49

Navigation Menu

Repository navigation

feat(oie): RQ A→B→C fan-out and Admin OIE queue tab - #1142

feat(oie): RQ A→B→C fan-out and Admin OIE queue tab#1142
northdpole wants to merge 17 commits into
admin-panel-showcasefrom
feat/oie-rq-fanout

northdpole commented Oct 9, 2026

Uh oh!

coderabbitai Bot commented Oct 9, 2026 •

edited

Loading

Review skipped

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

northdpole commented Oct 9, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Oct 9, 2026 •

edited

Loading