Skip to content

Commit ffc3b74

Browse files
author
Mark Pollack
committed
Framework smoke matrix: nightly and push workflow, release gate
framework-smoke.yml runs every fw-* and load-10-fw-* scenario on JDK 17 on each push to main, nightly, and on dispatch for any branch. The integration-testing README documents the matrix and the release gate: a candidate needs it green on the release commit. release.yml is unchanged.
1 parent 5ca4e6d commit ffc3b74

3 files changed

Lines changed: 183 additions & 7 deletions

File tree

‎.github/workflows/framework-smoke.yml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
name: Framework smoke matrix
2+
3+
# The release gate of integration-testing/README.md, "Framework smoke matrix": the SDK hosted by
4+
# Spring Boot, Micronaut and Quarkus against the TypeScript SDK (every fw-* and load-10-fw-*
5+
# scenario, generated from integration-testing/smoke.json). Runs on every push to main, nightly,
6+
# and on manual dispatch for any branch, which is how a candidate branch is gated: dispatch it on
7+
# that branch ("Run workflow" -> "Use workflow from"), or
8+
# gh workflow run framework-smoke.yml --ref <branch>
9+
# It needs only JDK 17, Node 20 and the TypeScript SDK, and takes about 10 minutes. The full
10+
# cross-SDK matrix stays in cross-sdk.yml.
11+
on:
12+
push:
13+
branches: [main]
14+
paths-ignore:
15+
- 'docs/**'
16+
- '*.md'
17+
schedule:
18+
- cron: '43 3 * * *'
19+
workflow_dispatch:
20+
inputs:
21+
peers-ref:
22+
description: 'Branch, tag or SHA of the TypeScript SDK (empty: its ref in peers.json)'
23+
required: false
24+
default: ''
25+
26+
permissions:
27+
contents: read
28+
29+
concurrency:
30+
group: framework-smoke-${{ github.ref }}
31+
cancel-in-progress: false
32+
33+
jobs:
34+
smoke:
35+
name: framework smoke
36+
runs-on: ubuntu-latest
37+
timeout-minutes: 45
38+
39+
steps:
40+
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
41+
42+
- name: Set up JDK 17
43+
uses: actions/setup-java@cf277c60eb25467037889841efdb72551f06f6c3 # v4
44+
with:
45+
java-version: '17'
46+
distribution: 'temurin'
47+
cache: maven
48+
49+
- name: Cache the Maven distribution the wrapper uses
50+
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
51+
with:
52+
path: ~/.m2/wrapper
53+
key: maven-wrapper-${{ hashFiles('.mvn/wrapper/maven-wrapper.properties') }}
54+
55+
- name: Set up Node 20
56+
id: node
57+
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
58+
with:
59+
node-version: '20'
60+
61+
- name: Set up JBang
62+
uses: jbangdev/setup-jbang@2b1b465a7b75f4222b81426f23a01e013aa7b95c # v0.1.1
63+
with:
64+
version: '0.135.1'
65+
setup-java: 'false'
66+
67+
- name: Check the generated configs are up to date
68+
working-directory: integration-testing
69+
run: jbang GenConfigs.java --check
70+
71+
- name: Resolve the TypeScript SDK commit
72+
id: peer
73+
env:
74+
PEERS_REF: ${{ github.event.inputs.peers-ref || '' }}
75+
run: integration-testing/scripts/peer-sha.sh typescript-sdk "$PEERS_REF" "" >> "$GITHUB_OUTPUT"
76+
77+
# The same key as cross-sdk.yml's typescript legs, so either workflow's build is reused.
78+
- name: Cache the TypeScript SDK build
79+
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
80+
with:
81+
path: |
82+
integration-testing/.cache/peers/typescript-sdk@*
83+
~/.npm
84+
key: cross-sdk-typescript-sdk-tc${{ steps.node.outputs.node-version }}-${{ steps.peer.outputs.sha }}-${{ hashFiles('integration-testing/programs/**/Cargo.toml', 'integration-testing/programs/**/*.gradle.kts') }}
85+
restore-keys: |
86+
cross-sdk-typescript-sdk-tc${{ steps.node.outputs.node-version }}-
87+
88+
- name: Run the framework smoke matrix
89+
env:
90+
PEERS_REF: ${{ github.event.inputs.peers-ref || '' }}
91+
RUN_LABEL: framework-smoke
92+
run: |
93+
args=(--tag fw)
94+
if [ -n "$PEERS_REF" ]; then args+=(--peers-ref "$PEERS_REF"); fi
95+
integration-testing/scripts/run-all.sh "${args[@]}"
96+
97+
- name: Upload scenario logs
98+
if: always()
99+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
100+
with:
101+
name: framework-smoke-logs
102+
path: integration-testing/logs/
103+
if-no-files-found: ignore
104+
retention-days: 14

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -573,6 +573,10 @@ If you need a stable target, pin to an exact version.
573573

574574
## Releases
575575

576+
A release candidate must have a green [framework smoke matrix](integration-testing/README.md#framework-smoke-matrix)
577+
(`framework-smoke.yml`: Spring Boot, Micronaut and Quarkus against the TypeScript SDK) on the
578+
commit being released before `release.yml` is dispatched.
579+
576580
### 0.18.0 (Current — [Maven Central](https://central.sonatype.com/artifact/com.agentclientprotocol/acp-core))
577581

578582
Remote agents: the Streamable HTTP and WebSocket transport from the ACP RFD, contributed by

‎integration-testing/README.md‎

Lines changed: 75 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,23 @@ processes in several languages, checks what they print, and tears them all down.
66

77
This directory is **not** a Maven module and the root `pom.xml` does not reference it, so
88
`./mvnw verify` and the release build never run it. It runs locally with the scripts below, and in
9-
CI through `.github/workflows/cross-sdk.yml` (per peer; manual dispatch plus nightly) and
10-
`.github/workflows/interop-java.yml` (the Java<->Java cells on every push and pull request).
9+
CI through `.github/workflows/cross-sdk.yml` (per peer; manual dispatch plus nightly),
10+
`.github/workflows/interop-java.yml` (the Java<->Java cells on every push and pull request) and
11+
`.github/workflows/framework-smoke.yml` (the [framework smoke matrix](#framework-smoke-matrix), the
12+
release gate: every push to `main`, nightly, and on dispatch for any branch).
1113

12-
There are three kinds of scenarios:
14+
There are four kinds of scenarios:
1315

1416
- **Generated cells**, `configs/x-<client>-<agent>-<transport>[-unstable].json`: one
1517
feature-driven agent and one client program per language, a shared step catalogue
1618
(`steps.json`), and configs generated from `matrix.json` by `GenConfigs.java`. The
1719
[Contracts](#contracts) section is everything a language package needs to take part.
1820
- **Conformance scenarios**, `configs/conf-java-*.json`: the raw JSON-RPC driver
1921
(`programs/raw`) against the Java agent and client, generated by `programs/raw/gen_conf.py`.
20-
- **Load scenarios**, `configs/load-*.json`, hand-written.
22+
- **Framework smoke cells**, `configs/fw-*.json` and `configs/load-10-fw-*.json`: the SDK hosted
23+
by Spring Boot, Micronaut and Quarkus against the TypeScript SDK, generated from `smoke.json` by
24+
`GenConfigs.java` ([Framework smoke matrix](#framework-smoke-matrix)).
25+
- **Load scenarios**, `configs/load-*.json`, hand-written (except `load-10-fw-*`).
2126

2227
## Prerequisites
2328

@@ -105,6 +110,7 @@ artifact.
105110
| `quarkus-typescript-http`, `quarkus-typescript-ws` | hand-written: the TypeScript client program against the Quarkus-hosted smoke agent (`programs/quarkus`: the shared `programs/framework` handlers as an `@AcpAgent` bean served by `acp-quarkus` on the Quarkus HTTP server), running the catalogue steps that agent implements (initialize, sessions, echo chunks, stop reasons, -32601, 1 MB and 8 MB prompts and updates). Run in the `typescript-http`/`typescript-ws` legs of `cross-sdk.yml` |
106111
| `load-50`, `load-300`, `load-1000` | N Java clients on their own connections, each: initialize, session/new, then 10 (or 5) prompts streaming two updates each |
107112
| `load-shared-300` | 300 clients sharing one HttpClient, so one HTTP/2 connection |
113+
| `fw-<framework>-agent-<t>`, `fw-<framework>-client-<t>`, `load-10-fw-<framework>` | generated from `smoke.json`: the [framework smoke matrix](#framework-smoke-matrix) |
108114

109115
Over Streamable HTTP the cells also assert, through `matrix.json` facts, what the retired
110116
hand-written `interop-*` scenarios checked: the negotiated HTTP version on both sides (h2c between
@@ -118,6 +124,61 @@ threads (`acp-*`) on the server at peak and after close, server heap after a ful
118124
256 MB at peak and 64 MB after close. p50/p99 latency is recorded in the results and never
119125
asserted.
120126

127+
## Framework smoke matrix
128+
129+
The release gate for the framework integrations: the SDK as an application uses it, hosted by
130+
Spring Boot (`acp-spring-boot-starter`), Micronaut (`acp-micronaut`) and Quarkus (`acp-quarkus`),
131+
against the TypeScript SDK. **A release candidate must have a green framework smoke matrix** on
132+
the commit being released (`framework-smoke.yml`, or the local command below), next to the usual
133+
`./mvnw clean verify`; the full cross-SDK matrix (`cross-sdk.yml`) should be green on it too.
134+
135+
`smoke.json` defines it once; `GenConfigs.java` writes its configs (and `--check` keeps them in step):
136+
137+
- **`agentSteps`**, run by the TypeScript client against each framework-hosted agent on each of
138+
that framework's transports (`fw-<framework>-agent-<t>`): `init.initialize` (with the agentInfo
139+
the `@AcpAgent` annotation declares), `init.agent-capabilities` (capabilities *derived* from
140+
the handler annotations: the agent has no `@Initialize` method), `init.agent-info`,
141+
`session.new`, `session.load`, `session.multi`, `update.agent_message_chunk`, `perm.selected`,
142+
`elicit.form`, `cancel.prompt`, `ext.client-request` (an `@ExtRequest` method),
143+
`error.method-not-found`, `http.reconnect` (HTTP only), `stdio.eof-exit` (stdio only) and
144+
`conn.close`: 14 steps per cell, 13 over WebSocket. The agent-side assertions
145+
(`STEP agent.perm.selected`, `agent.elicit.form`, `agent.cancel.prompt`) are required, and the
146+
agent must log nothing at WARN or ERROR.
147+
- **`clientSteps`**, run by the client bean each framework builds from its own configuration
148+
(transport URI and advertised capabilities; the handlers come from an `AcpClientCustomizer`
149+
bean) against the TypeScript agent (`fw-<framework>-client-<t>`): `init.initialize`,
150+
`init.agent-info`, `session.new`, `update.agent_message_chunk`, `perm.selected`, `fs.read`,
151+
`elicit.form`, `cancel.prompt`, `ext.client-request`, `conn.close`.
152+
- **`load`**: 10 Java load clients x 10 prompts against each framework-hosted agent over HTTP
153+
(`load-10-fw-<framework>`): no errors, every prompt and update delivered, clean close, no
154+
`ERROR` on the server. Latency is recorded, not asserted.
155+
156+
| Framework | Agent transports | Client | Program |
157+
|---|---|---|---|
158+
| Spring Boot 4.1 | stdio (`spring.main.keep-alive=true`), HTTP (the SDK servlet on `server.port` in a servlet web application) | HTTP | [programs/spring](programs/spring) |
159+
| Micronaut 4 | stdio, HTTP, WebSocket (the SDK listener) | HTTP | [programs/micronaut](programs/micronaut) |
160+
| Quarkus | stdio, HTTP, WebSocket (the Quarkus HTTP server); one package per build-time transport | HTTP | [programs/quarkus](programs/quarkus) |
161+
162+
Spring has no WebSocket cell yet: the servlet takes no WebSocket upgrades on `server.port`. When it
163+
does, add `"ws"` to `frameworks.spring.agentTransports` in `smoke.json` (it is listed under
164+
`pendingAgentTransports` until then) and regenerate.
165+
166+
The agent handlers ([programs/framework/.../SmokeAgent.java](programs/framework/src/main/java/interop/framework/SmokeAgent.java))
167+
and the client steps ([SmokeClient.java](programs/framework/src/main/java/interop/framework/SmokeClient.java))
168+
are shared sources that every framework program compiles in; each framework adds only an
169+
`@AcpAgent` subclass with its own bean annotation and a main class. To run the whole matrix,
170+
locally or against another branch:
171+
172+
```bash
173+
git checkout <branch> # the candidate under test
174+
integration-testing/scripts/run-all.sh --tag fw # every fw-* and load-10-fw-* scenario
175+
integration-testing/scripts/run-all.sh --only 'fw-spring-*,load-10-fw-spring' # one framework
176+
gh workflow run framework-smoke.yml --ref <branch> # the same in CI, on that branch
177+
```
178+
179+
It needs JDK 17+, Node 20 and JBang (the TypeScript SDK is cloned and built on the first run); on a
180+
warm machine it takes about 3 minutes.
181+
121182
## Contracts
122183

123184
What a language package (an agent and a client program for one SDK) must implement to take part
@@ -380,7 +441,12 @@ language too, run on the same host at the same time.
380441
`typescript-http`, ..., `kotlin-ws`) restores that build, installs only its own toolchain, and
381442
runs `run-all.sh --only 'x-java-<peer>-<t>*,x-<peer>-java-<t>*'`. A `java` job runs the Java
382443
pair, conformance, load and both `--check`s. Each job uploads `logs/` as `cross-sdk-logs-<leg>`; `fail-fast` is off.
383-
- None of these gates a release.
444+
- `.github/workflows/framework-smoke.yml`, on every push to `main`, nightly and on manual dispatch
445+
for any branch: `GenConfigs.java --check`, then `run-all.sh --tag fw` on JDK 17. This is the
446+
release gate ([Framework smoke matrix](#framework-smoke-matrix)); `release.yml` does not check it
447+
itself, so the maintainer confirms it is green on the release commit before dispatching a
448+
release.
449+
- Neither `interop-java.yml` nor `cross-sdk.yml` gates a release.
384450

385451
## How a scenario is described
386452

@@ -441,14 +507,16 @@ language too, run on the same host at the same time.
441507

442508
```
443509
RunScenario.java JBang entry point: runs one configs/<scenario>.json
444-
GenConfigs.java JBang: generates configs/x-*.json from the three inputs below
510+
GenConfigs.java JBang: generates configs/x-*.json from the three inputs below, and fw-* from smoke.json
445511
steps.json the step catalogue (Contracts)
446512
matrix.json languages, pairs, transports, profiles, transport facts
513+
smoke.json the framework smoke matrix: step subsets, frameworks and transports, load
447514
expectations/ <lang>.json expected failures, one file per language package
448515
jbang-lib/ config model, peer checkouts, processes, jcmd sampler, assertions
449516
peers.json peer SDK git URLs, default refs and build commands
450-
configs/ one JSON file per scenario (x-* generated, the rest hand-written)
517+
configs/ one JSON file per scenario (x-*, fw-*, load-10-fw-* generated, the rest hand-written)
451518
programs/<lang>/ each language's programs; launch/{build,agent,client}.sh for the cells
519+
programs/<framework>/ spring, micronaut, quarkus: the smoke programs, built on programs/framework
452520
scripts/run-all.sh a selection of scenarios plus a summary table
453521
.cache/, logs/ generated, git-ignored
454522
```

0 commit comments

Comments
 (0)