Set a baseline for OCA tests

[gatherisum]

Parent: #538
Blocked by: #539

The 280-failure report in the parent cannot be reproduced. The container that produced it is gone.

It ran from docker/docker-compose.yml with CAMUNDA_DATA_SECONDARY_STORAGE_TYPE: rdbms,
CAMUNDA_SECURITY_AUTHORIZATIONS_ENABLED: false and UNPROTECTEDAPI: true, on image tag
8.9-SNAPSHOT that self-reported 8.10.0-alpha4. Something replaced it mid-investigation with
camunda/camunda:SNAPSHOT on Elasticsearch, requiring auth. The two disagree on at least 47 results,
including whether four route families exist at all. The parent has the comparison table.

Until a run exists that anyone can repeat, no failure count means anything.

What to do

Run the on-demand workflow from #539 against a clean container and record the result. Attach the breakdown to this issue so later PRs have something to diff against.

The group D experiment, done here

Group D is 47 failures from routes that returned No static resource on the report's deployment but
respond on Elasticsearch. Try the compose correction as part of this run: switch
CAMUNDA_DATA_SECONDARY_STORAGE_TYPE off rdbms, add Elasticsearch, and set
CAMUNDA_SECURITY_AUTHORIZATIONS_ENABLED: true.

Watch for one side effect before committing to it. The unsecured profile depends on
UNPROTECTEDAPI: true with authorizations off, which is how its requests reach the API with no
credentials. Every unauthenticated call I made against the Elasticsearch container returned 401. So
turning authorizations on may change what "unsecured" means for the whole suite, and could trade 47
failures for a much larger auth-shaped set.

Measure it. If the corrected compose clears group D without wrecking the rest, take it and close
group D here. If it trades one problem for a bigger one, then the team is genuinely running two
deployment shapes, and the answer is a deployment dimension in the config layer: a scoped block in
request-validation.json, a validator beside config.ts:227-249, selection via an env var alongside
RV_PROFILE, threading through generate.ts:193-204, and guardrail tests. Open that as a separate
issue if it comes to it.

While you are here, re-check group J. Two backup operations return 500 with
(code: UNSUPPORTED_MESSAGE), which looks like the same rdbms limitation. If they disappear on
Elasticsearch, fold them into group D and drop the planned upstream filing.

Acceptance

• A recorded baseline run from a checked-in compose file, with the breakdown attached here.
• A decision on group D, with the numbers behind it.
• Group J confirmed as either environmental or a genuine server bug.

[gatherisum]

Recorded baseline

Parent: #538. This is the baseline #538 needed and #539's workflows now make possible: something
reproducible, plus a decision on group D and a check on group J.

It's reproducible now

Three scheduled nightly runs, all on the same commit (4d7bfad, main HEAD at the time), all from
the checked-in docker/docker-compose.yml (8.9-SNAPSHOT, rdbms secondary storage,
UNPROTECTEDAPI: true, authorizations off), gave the identical request-validation result:

Date	Run
2026-09-05	https://github.com/camunda/api-test-generator/actions/runs/33941521581
2026-09-06	https://github.com/camunda/api-test-generator/actions/runs/34008738586
2026-09-07	https://github.com/camunda/api-test-generator/actions/runs/34079398740

578 of 1906 request-validation tests fail, 30.3%, 0 flaky, 0 skipped, every single run. That's the
baseline from here on. A steady 578 means nothing regressed. A change means something did, and this
breakdown names the group.

By actual HTTP status:
   365  404
   138  200
    73  405
     2  503

By scenario kind:
   148  pagination-limit-invalid
    67  missing-required
    60  type-mismatch
    53  constraint-violation
    39  param-constraint-violation
    37  missing-required-combo
    30  malformed-json-body
    22  body-top-type-mismatch
    22  pagination-cursor-invalid
    17  enum-violation
    16  explicit-null-required
    14  additional-prop
    11  pagination-offset-past-total
    11  missing-body
     8  additional-prop-general
     8  param-type-mismatch
     5  format-invalid
     4  unique-items-violation
     2  param-enum-violation
     2  param-missing

By operationId (top 15):
    82  createAgentInstanceHistoryItem
    30  updateAgentInstance
    24  createAgentInstance
    22  updateJobsBatchOperation
    19  resumeProcessInstancesBatchOperation
    19  suspendProcessInstancesBatchOperation
    15  searchOwnAuthorizations
    14  searchAgentInstanceHistory
    13  searchAgentDefinitions
    13  searchAgentInstances
    13  searchElementInstanceWaitStates
    13  searchResources
    12  assignProcessInstanceBusinessId
    11  takeHistoryBackup
    11  restoreAsClusterAdmin
  (89 more operationIds affected)

Full detail (per-failure request/response, HTML report) is in each run's oca-suite-reports
artifact.

Found and fixed along the way: the positive suite never ran

All three nightly runs also show the positive (lifecycle) suite crashing before a single test
executes:

./scripts/e2e/run-oca.sh: line 91: CONFIG=camunda-oca: command not found

${BEARER_TOKEN:+BEARER_TOKEN="$BEARER_TOKEN"} isn't a literal name=value word. When
BEARER_TOKEN is unset, which it always is in CI, that breaks bash's leading-assignment
recognition, and the next word, the literal text CONFIG=camunda-oca, gets read as the command to
run instead of an env assignment. run_rv() a few lines below already dodges this trap with an
env-array passed to env. Fixed the positive-suite block the same way:
5cf86d3 (branch
claude/nightly-camunda-oca-baseline-bec230, PR #555).

Dispatched an on-demand run against that fix to confirm it, and to finally measure the positive
suite:
https://github.com/camunda/api-test-generator/actions/runs/34095903170

Request-validation held at exactly 578/1906, no drift from the fix, as expected. The positive suite
now runs and reports 440 of 594 tests failing, 74%. Rough, for a suite that's supposed to gate this
nightly and was silently never running at all until now. Mostly the same "route unregistered" cause
as group D below (searchAgentDefinitions 113, createAgentInstance 17, searchAgentInstances 9,
and more), plus one thing that isn't: two ad-hoc sub-process activation tests get a genuine 500,
PROCESSING_ERROR, "unexpected error occurred... partition 1". That one isn't route-unregistered or
auth-shaped and probably deserves its own look.

By actual HTTP status:
   182  404
    92  400
    88  (unknown)
    37  200
    34  405
     3  500
     2  204
     1  403
     1  401

By operationId (top 8):
   113  searchAgentDefinitions
    88  (unknown)
    17  createAgentInstance
    11  migrateProcessInstancesBatchOperation
    11  resumeProcessInstancesBatchOperation
    11  suspendProcessInstancesBatchOperation
     9  searchAgentInstances
     8  searchElementInstances
  (more operationIds affected, full detail in the run's oca-suite-reports artifact)

The 88 "(unknown)" status entries are positive-suite failures whose Playwright attachments don't
match summarize-failures.mjs's request-validation-shaped extraction, since that script was built
for the negative suite. They're still failures, just check the HTML report for detail instead of
this summary.

Group D: bigger than originally scoped, not yet re-measured against Elasticsearch

The 404 count here, 365 failures across 54 operationIds, is far larger than the 47 originally
attributed to group D. It's not just agent-definitions/* anymore. The whole agent-instance family
404s too (createAgentInstanceHistoryItem, updateAgentInstance, createAgentInstance,
searchAgentInstances, searchAgentInstanceHistory), on top of searchOwnAuthorizations (15,
matching the original count exactly), the backup/secrets family, and the group A cluster-admin
routes (restoreAsClusterAdmin, restore, changeClusterMode, changeClusterModeAsClusterAdmin),
which are a separate, already-known cause: the servers: override bug. Group C's signature, a
seeded path-param key resolving to a missing resource (searchElementInstanceIncidents,
searchUserTaskVariables, and others), also shows up under the same 404 bucket for an unrelated
reason.

So "404" isn't one root cause here, and this baseline's 404 count shouldn't be assumed to map 1:1
onto the original group D. Not going to try to net that out from the numbers alone. The rdbms to
Elasticsearch plus CAMUNDA_SECURITY_AUTHORIZATIONS_ENABLED: true compose experiment that #538
asked for to test group D still hasn't run. No Elasticsearch compose overlay exists in this repo,
only docker-compose.rdbms-unified-config.yml and docker-compose.rbac.yml, and building one plus
wiring it into CI is enough work to warrant its own issue rather than a guess. Filing that as a
follow-up (#556), scoped explicitly to re-deriving group D (and the group A / group C overlap above)
against this baseline instead of the original unreproducible one.

Group J: confirmed environmental, not reproduced as reported

Group J's reported signature was a 500 with (code: UNSUPPORTED_MESSAGE) on backup operations
under rdbms secondary storage. That exact signature doesn't show up anywhere in the
request-validation baseline: 0 occurrences of HTTP 500 in that report. (The positive suite's
unrelated 500s, above, are on ad-hoc sub-process activation, not backups.) The backup operations
(takeHistoryBackup, takeRuntimeBackup, listHistoryBackups, and others) fail with 404 instead,
the same "route unregistered under rdbms" cause as group D, not a distinct 500 bug.

The only 503s in this baseline, 2 of them, on getResource and getResourceContent, are a
different symptom: "UNAVAILABLE", "request could not be delivered", on a fake resource key.
Unrelated to group J, flagging it in case it recurs.

Group J is environmental, subsumed by group D's cause rather than a genuine server bug, pending the
same Elasticsearch check proposed above to see whether backups resolve to 200/403 there, as #538's
manual probe found, or turn into something new.

[gatherisum]

New baseline: main + stable-8.10 legs, first run that ever gets past extraction (#559)

Context: #559 found that stable/8.9's spec predates an upstream x-semantic-provider annotation (camunda/camunda#52271) the generator needs for DecisionDefinitionId, and separately that the agent-instance domain's HistoryItemId/LoopIterationId/AgentHistoryItemKey classification (originally added in bbd33b2) had been reverted as collateral damage in 322e7ce. Both together meant main and stable-8.9 were crashing at extract-graph on every nightly run — confirmed directly: last night's scheduled run failed in 44s with the exact same DomainSemanticsValidationFailure.

#562 moved the tracked spec pin to main (avoiding a stable/8.9-specific workaround) and re-classified the agent-instance types. #563 drops the now-redundant stable-8.9/pinned-8.9 legs from the nightly matrix. Together they're the first time main/stable-8.10 have ever generated and executed real tests — so what follows isn't a regression against anything; it's the first real signal these two legs have ever produced.

Run: https://github.com/camunda/api-test-generator/actions/runs/34342812949 (commit fb790fe37, dispatched manually from #563's branch since it hasn't merged yet)

Positive (lifecycle) suite

Leg	Passed	Failed
main	234	387
stable-8.10	231	386

main — by actual HTTP status (216 unknown = variant/feature tests that fail
before an assertable HTTP call, no request/response attachment captured):
   120  400
    33  404
     7  500
     5  403
     3  401
     2  204
     1  409

By operationId (top 10):
    22  createAgentInstance
    17  searchAuditLogs.variant.spec.ts
    11  migrateProcessInstancesBatchOperation
     9  searchAgentInstances.variant.spec.ts
     8  searchElementInstances.variant.spec.ts
     8  searchUserTasks.variant.spec.ts
     8  searchVariables.variant.spec.ts
     7  searchDecisionInstances.variant.spec.ts
     7  searchResources.variant.spec.ts
     6  modifyProcessInstance

stable-8.10 is within 1 test of the same shape across the board — not duplicating here.

One clear, single-cause bug found along the way: all 22 createAgentInstance failures are the identical 400:

"Unexpected value 'placeholder' for enum field 'role'. Use any of the following
values: [USER, ASSISTANT, TOOL_RESULT, CONFIGURATION]"

The generator is emitting the literal string "placeholder" for AgentInstanceHistoryItem.role instead of picking a valid enum value. This was never visible before today — agent-instance tests never generated at all until #562 unblocked extraction — so it's a pre-existing generator gap in enum-value selection, not something #562/#563 introduced. Likely worth its own issue; flagging here since it's the single biggest identifiable chunk of this baseline's positive-suite failures (and probably cascades into some of the 216 "unknown" variant failures downstream of a failed createAgentInstance).

Request-validation (unsecured) suite

Leg	Passed	Failed
main	1624	296
stable-8.10	1614	292

main — by actual HTTP status:
   161  200
   106  404
    16  403
     8  401
     3  409
     2  500

By scenario kind:
   142  pagination-limit-invalid
    49  constraint-violation
    29  param-constraint-violation
    20  type-mismatch
    12  pagination-cursor-invalid
     7  param-type-mismatch
     5  pagination-offset-past-total
     5  additional-prop-general
     5  body-top-type-mismatch
     5  malformed-json-body

By operationId (top 10):
    14  triggerClusterRebalance
    11  takeHistoryBackup
    11  restoreAsClusterAdmin
    11  takeHistoryBackupAsClusterAdmin
     8  searchOwnAuthorizations
     8  takeRuntimeBackupAsClusterAdmin
     6  searchElementInstanceIncidents
     6  searchProcessInstanceIncidents
     6  searchUserTaskAuditLogs
     6  searchUserTaskVariables

stable-8.10 again tracks within a few tests of the same shape (292 vs 296, same top groups). The pagination-limit-invalid (142) and cluster-admin/backup 404s (triggerClusterRebalance, *BackupAsClusterAdmin, restoreAsClusterAdmin) look like the same families this issue already discusses (group A's servers: override bug, and pagination-limit enforcement not applying to several search endpoints — many actual: 200 where 400 was expected).

Full per-test detail (HTML report, request/response bodies) is in the run's oca-suite-reports-main / oca-suite-reports-stable-8.10 artifacts.

Why this replaces pinned-8.9's 578/1906 baseline

pinned-8.9 (removed in #563) tracked a frozen spec pin against an 8.9-SNAPSHOT broker specifically so the count would stay comparable night over night. Since the spec pin now tracks main (#562), that leg's premise no longer holds, so main+stable-8.10's numbers above are the new thing to diff future PRs against — same idea, moving target instead of a frozen one, tracked by re-running this comparison rather than a fixed baseline.