Skip to content

analysis.md

Latest commit

 

History

History
50 lines (37 loc) · 2.98 KB

analysis.md

File metadata and controls

50 lines (37 loc) · 2.98 KB
Raw

OpenRouter Cheap Models — Pass 2 Regression Analysis

Date: 2026-06-27 Run: evals/results/openrouter-cheap-pass-2 Baseline run: evals/results/openrouter-cheap-pass Target: http://127.0.0.1:3323 Model set: 5

Method

Executed a second sequential pass of the same five OpenRouter cheap models with the same eval harness settings:

  • MODEL_MASTRA_SECURITY_RESEARCH: each model URI below
  • LIVE_EVAL_MAX_STEPS=12
  • LIVE_EVAL_PROMPT_MODE=minimal
  • LIVE_EVAL_TOOL_CHOICE=auto
  • LIVE_EVAL_ENABLED_TOOL_IDS=httpProbeTool,artifactAccessTool,targetAuthorizationTool
  • LIVE_EVAL_SAVE_FAILED_RESULTS=true

Models:

  • llm://openrouter/openai/gpt-oss-120b?temperature=0.1&maxTokens=128
  • llm://openrouter/deepseek/deepseek-v4-flash:nitro?temperature=0.1&maxTokens=128
  • llm://openrouter/qwen/qwen3.6-flash?temperature=0.1&maxTokens=128
  • llm://openrouter/google/gemini-3.1-flash-lite?temperature=0.1&maxTokens=128
  • llm://openrouter/google/gemma-4-26b-a4b-it?temperature=0.1&maxTokens=128

Results snapshot

Model Old score New score Δscore Old tool calls New tool calls Old evidence-vuln New evidence-vuln Notes
openai/gpt-oss-120b 0/13 0/13 0 3 4 1 1 New model produced no final findings; tool-calls increased from 3→4.
deepseek/deepseek-v4-flash:nitro 7/13 4/13 -3 0 0 1 1 Step now appears shorter (single-step, zero tool calls), causing total score regression despite equivalent evidence/vuln extraction.
qwen/qwen3.6-flash 8/13 8/13 0 12 12 1 1 Stable behavior; same final score and vuln count.
google/gemini-3.1-flash-lite 0/13 0/13 0 3 3 1 1 Stable score; remains low, mostly non-tooling narrative output.
google/gemma-4-26b-a4b-it 7/13 7/13 0 0 0 1 1 Stable score; remains non-probing in this run.

Regression signal

  • Primary regression: deepseek/deepseek-v4-flash:nitro dropped from 7/13 to 4/13 with no additional evidence gain. This indicates behavioral instability in the second pass and is the only non-noise score regression.
  • No model lost evidence-artifact collection (all remained at 1 baseline-only artifact).
  • Vulnerability counts were unchanged across all models in this run.

Known issue signal during run

During execution output (not captured in final JSON logs), model-side stderr showed repeated intermittent tool-call JSON conversion diagnostics for qwen and one http-probe persistence warning previously observed. The JSON result still completes and artifacts persist enough for scoring; this is a non-fatal behavior signal and should be tracked separately.

Next checks to close loop

  1. Re-run deepseek-v4-flash:nitro alone for 2–3 iterations with LIVE_EVAL_TOOL_CHOICE=required and agentProfile=lean to isolate the scoring failure path.
  2. Capture both stdout+stderr in logs (2>&1) for artifact persistence warnings and parser repair lines so we can verify whether they are model-conditional or infra-wide.