Skip to content

Commit be883c2

Browse files
sent-dmsent-dm
authored
Merge pull request #11 from sentdm/codex/documentation-freshness
ci(docs): monitor Sent contract and documentation freshness
2 parents 0552bc5 + 6e75b22 commit be883c2

9 files changed

Lines changed: 807 additions & 105 deletions

File tree

‎.github/workflows/documentation-freshness.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
name: Monitor Sent contract and documentation freshness
2+
3+
on:
4+
workflow_dispatch:
5+
release:
6+
types: [published]
7+
schedule:
8+
- cron: "17 9 * * 1"
9+
10+
permissions:
11+
contents: read
12+
13+
jobs:
14+
freshness-and-routing:
15+
runs-on: ubuntu-latest
16+
env:
17+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
18+
SENT_ROUTING_EVAL_MODEL: ${{ vars.SENT_ROUTING_EVAL_MODEL || 'gpt-5-mini' }}
19+
steps:
20+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
21+
22+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
23+
with:
24+
python-version: "3.12"
25+
cache: pip
26+
cache-dependency-path: requirements-dev.txt
27+
28+
- name: Install validators
29+
run: python -m pip install -r requirements-dev.txt
30+
31+
- name: Compare live Sent contract and documentation facts
32+
run: >-
33+
python scripts/check_live_contract.py
34+
--timeout 15
35+
--retries 2
36+
--output artifacts/documentation-freshness.json
37+
38+
- name: Upload freshness diagnostics
39+
if: always()
40+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
41+
with:
42+
name: documentation-freshness
43+
path: artifacts/documentation-freshness.json
44+
if-no-files-found: warn
45+
46+
- name: Run model-backed routing evaluation
47+
if: always() && env.OPENAI_API_KEY != ''
48+
run: python scripts/run_model_routing_eval.py
49+
50+
- name: Explain skipped model evaluation
51+
if: always() && env.OPENAI_API_KEY == ''
52+
run: echo "OPENAI_API_KEY is not configured; freshness monitoring still ran, but model routing evaluation was skipped."

‎.github/workflows/validate.yml‎

Lines changed: 2 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,6 @@ on:
77
branches: [main]
88
release:
99
types: [published]
10-
schedule:
11-
- cron: "17 9 * * 1"
1210

1311
permissions:
1412
contents: read
@@ -60,31 +58,5 @@ jobs:
6058
- name: Test Sent API contract regressions
6159
run: python scripts/test_contracts.py
6260

63-
live-contract-and-routing:
64-
if: github.event_name == 'schedule' || github.event_name == 'release' || github.event_name == 'workflow_dispatch'
65-
runs-on: ubuntu-latest
66-
env:
67-
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
68-
SENT_ROUTING_EVAL_MODEL: ${{ vars.SENT_ROUTING_EVAL_MODEL || 'gpt-5-mini' }}
69-
steps:
70-
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
71-
72-
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
73-
with:
74-
python-version: "3.12"
75-
cache: pip
76-
cache-dependency-path: requirements-dev.txt
77-
78-
- name: Install validators
79-
run: python -m pip install -r requirements-dev.txt
80-
81-
- name: Compare live Sent OpenAPI with manifest
82-
run: python scripts/check_live_contract.py
83-
84-
- name: Run model-backed routing evaluation
85-
if: env.OPENAI_API_KEY != ''
86-
run: python scripts/run_model_routing_eval.py
87-
88-
- name: Explain skipped model evaluation
89-
if: env.OPENAI_API_KEY == ''
90-
run: echo "OPENAI_API_KEY is not configured; live contract drift still ran, but model routing evaluation was skipped."
61+
- name: Test freshness monitor simulations
62+
run: python scripts/test_live_contract.py

‎docs/PUBLIC_RELEASE.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Use this checklist for every public release.
55
## Repository and package
66

77
- Confirm the GitHub repository is public and the release commit is on the default branch.
8-
- Confirm the repository root contains `plugin.json`, `mcp.json`, `skills/`, and `assets/` for GitHub auto-discovery.
8+
- Confirm the repository root contains `plugin.json`, `mcp.json`, `public-surface.json`, `skills/`, and `assets/` for GitHub auto-discovery.
99
- Confirm `.claude-plugin/marketplace.json` points to `./claude-plugins/sent`.
1010
- Confirm the Claude plugin contains its manifest, README, license, skills, commands, assets, and HTTPS MCP configuration.
1111
- Update the version consistently before release; plugin names are stable public identifiers and must not be renamed casually.
@@ -18,11 +18,13 @@ python3 scripts/generate_adapters.py --check
1818
python3 scripts/validate.py
1919
python3 scripts/test_validation_gates.py
2020
python3 scripts/test_fixtures.py
21+
python3 scripts/test_contracts.py
22+
python3 scripts/test_live_contract.py
2123
claude plugin validate . --strict
2224
claude plugin validate ./claude-plugins/sent --strict
2325
```
2426

25-
CI must pass on the exact commit being submitted. Resolve warnings as well as errors before submission.
27+
CI must pass on the exact commit being submitted. Resolve warnings as well as errors before submission. The dedicated contract/documentation freshness workflow runs on schedule, manual dispatch, and release; review its uploaded JSON artifact separately from pull-request validation.
2628

2729
## Public-facing review
2830

‎packages/sent/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,10 @@ python3 scripts/generate_adapters.py
8585
python3 scripts/validate.py
8686
python3 scripts/test_validation_gates.py
8787
python3 scripts/test_fixtures.py
88+
python3 scripts/test_contracts.py
89+
python3 scripts/test_live_contract.py
8890
```
8991

9092
For current product and API behavior, use the [Sent MCP documentation](https://docs.sent.dm/start/llm-docs/mcp-server), the [machine-readable documentation index](https://docs.sent.dm/llms.txt), and the [Sent API reference](https://docs.sent.dm/reference/api).
93+
94+
Live network checks are isolated from pull-request validation. The scheduled/manual/release freshness workflow compares normalized facts from the source catalog, classifies drift separately from source failures, and uploads a JSON diagnostic artifact.

‎schemas/sent/documentation-sources.json‎

Lines changed: 107 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,21 @@
66
"url": "https://docs.sent.dm/llms.txt",
77
"kind": "documentation-index",
88
"affected_skills": ["sent"],
9-
"last_verified": "2026-08-09"
9+
"last_verified": "2026-08-09",
10+
"checks": [
11+
{
12+
"fact": "docs.index.template_definition_listed",
13+
"extractor": "contains",
14+
"pattern": "https://docs.sent.dm/llms/reference/api/template-definition.txt",
15+
"expected": true
16+
},
17+
{
18+
"fact": "docs.index.webhook_events_listed",
19+
"extractor": "contains",
20+
"pattern": "https://docs.sent.dm/llms/start/webhooks/event-types.txt",
21+
"expected": true
22+
}
23+
]
1024
},
1125
{
1226
"id": "openapi",
@@ -22,14 +36,29 @@
2236
"waba-embedded-signup",
2337
"waba-template-author"
2438
],
25-
"last_verified": "2026-08-09"
39+
"last_verified": "2026-08-09",
40+
"checks": []
2641
},
2742
{
2843
"id": "templates",
2944
"url": "https://docs.sent.dm/llms/reference/api/template-definition.txt",
3045
"kind": "documentation",
3146
"affected_skills": ["sent-templates", "template-builder-ui", "waba-template-author"],
32-
"last_verified": "2026-08-09"
47+
"last_verified": "2026-08-09",
48+
"checks": [
49+
{
50+
"fact": "template.body_max_length",
51+
"extractor": "regex_int",
52+
"pattern": "Body text[^\\n]*Maximum (\\d+) characters",
53+
"expected": 1024
54+
},
55+
{
56+
"fact": "template.button_total_limit",
57+
"extractor": "regex_int",
58+
"pattern": "maximum of (\\d+) buttons",
59+
"expected": 10
60+
}
61+
]
3362
},
3463
{
3564
"id": "webhooks",
@@ -40,35 +69,105 @@
4069
"sender-profile-architect",
4170
"waba-embedded-signup"
4271
],
43-
"last_verified": "2026-08-09"
72+
"last_verified": "2026-08-09",
73+
"checks": [
74+
{
75+
"fact": "webhook.message_delivered_event",
76+
"extractor": "contains",
77+
"pattern": "\"event\": \"message.delivered\"",
78+
"expected": true
79+
},
80+
{
81+
"fact": "webhook.template_event_omitted",
82+
"extractor": "contains",
83+
"pattern": "omitted for template events",
84+
"expected": true
85+
}
86+
]
4487
},
4588
{
4689
"id": "sender-profiles",
4790
"url": "https://docs.sent.dm/llms/start/concepts/sender-profiles.txt",
4891
"kind": "documentation",
4992
"affected_skills": ["sender-profile-architect", "sms-10dlc-registration", "waba-embedded-signup"],
50-
"last_verified": "2026-08-09"
93+
"last_verified": "2026-08-09",
94+
"checks": [
95+
{
96+
"fact": "profile.organization_scope_header",
97+
"extractor": "contains",
98+
"pattern": "x-profile-id",
99+
"expected": true
100+
},
101+
{
102+
"fact": "profile.rcs_agent_dedicated",
103+
"extractor": "contains",
104+
"pattern": "RCS Agent",
105+
"expected": true
106+
}
107+
]
51108
},
52109
{
53110
"id": "waba",
54111
"url": "https://docs.sent.dm/llms/start/guides/connect-whatsapp.txt",
55112
"kind": "documentation",
56113
"affected_skills": ["waba-embedded-signup", "waba-template-author"],
57-
"last_verified": "2026-08-09"
114+
"last_verified": "2026-08-09",
115+
"checks": [
116+
{
117+
"fact": "waba.id_field",
118+
"extractor": "contains",
119+
"pattern": "waba_id",
120+
"expected": true
121+
},
122+
{
123+
"fact": "waba.access_token_field",
124+
"extractor": "contains",
125+
"pattern": "access_token",
126+
"expected": true
127+
}
128+
]
58129
},
59130
{
60131
"id": "routing-rcs",
61132
"url": "https://docs.sent.dm/llms/reference/channel-routing.txt",
62133
"kind": "documentation",
63134
"affected_skills": ["messaging-performance-analyzer", "rcs-agent-onboarding", "sent-messaging"],
64-
"last_verified": "2026-08-09"
135+
"last_verified": "2026-08-09",
136+
"checks": [
137+
{
138+
"fact": "routing.multiple_explicit_channels",
139+
"extractor": "contains",
140+
"pattern": "broadcast list, not a fallback priority list",
141+
"expected": true
142+
},
143+
{
144+
"fact": "routing.rcs_supported",
145+
"extractor": "contains",
146+
"pattern": "Pins the message to RCS",
147+
"expected": true
148+
}
149+
]
65150
},
66151
{
67152
"id": "10dlc",
68153
"url": "https://docs.sent.dm/llms/start/advanced/10dlc-registration.txt",
69154
"kind": "documentation",
70155
"affected_skills": ["sms-10dlc-registration"],
71-
"last_verified": "2026-08-09"
156+
"last_verified": "2026-08-09",
157+
"checks": [
158+
{
159+
"fact": "10dlc.tcr_submission",
160+
"extractor": "contains",
161+
"pattern": "submit to TCR",
162+
"expected": true
163+
},
164+
{
165+
"fact": "10dlc.opt_in_consent",
166+
"extractor": "contains",
167+
"pattern": "consent",
168+
"expected": true
169+
}
170+
]
72171
}
73172
]
74173
}

0 commit comments

Comments
 (0)