feat: add --skip-provider CLI option and dashboard lap E2E test#2063
feat: add --skip-provider CLI option and dashboard lap E2E test#2063nick-inkeep merged 14 commits intomainfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 396949d The changes in this PR will be included in the next version bump. This PR includes changesets to release 9 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
PR Review Summary
(4) Total Issues | Risk: Medium
🟠⚠️ Major (2) 🟠⚠️
Inline Comments:
- 🟠 Major:
quickstart.test.ts:262-264Empty catch block swallows screenshot error details - 🟠 Major:
utils.ts:284-294Dashboard server output not captured for debugging
🟡 Minor (1) 🟡
Inline Comments:
- 🟡 Minor:
utils.ts:193Interactive "Skip" label is ambiguous
💭 Consider (1) 💭
💭 1) packages/create-agents/README.md Document --skip-provider in README
Issue: The README (which is the primary onboarding documentation) does not mention the --skip-provider flag or the "Skip" interactive option. This feature is specifically designed for "developers exploring the framework without an API key" per the PR motivation.
Why: Features designed for first-contact exploration should be prominently documented in first-contact surfaces. New users exploring without credentials won't discover the feature otherwise.
Fix: Consider adding a note after the npx @inkeep/create-agents my-agents line:
**No API key?** Use `--skip-provider` to scaffold with a mock provider:
```bash
npx @inkeep/create-agents my-agents --skip-providerThis lets you explore the framework with deterministic responses before adding a real provider.
**Refs:**
- [packages/create-agents/README.md](https://github.com/inkeep/agents/blob/368ae9675e70a4ad57040d9fb3fc5857ce803cce/packages/create-agents/README.md)
---
<div align="center">
## 💡 APPROVE WITH SUGGESTIONS
</div>
**Summary:** This is a well-structured PR that adds useful functionality for developer onboarding. The `--skip-provider` flag follows existing CLI conventions and the dashboard lap E2E test provides valuable coverage for the full user journey. The main suggestions focus on improving debuggability of CI failures (capturing server output and screenshot errors) and making the "Skip" interactive option self-documenting. Consider documenting the new flag in the README to maximize discoverability.
<details>
<summary>Discarded (5)</summary>
| Location | Issue | Reason Discarded |
|----------|-------|------------------|
| `utils.ts:82-84` | Mock config only has `base` property unlike other providers | Intentional design - mock is simpler and fallback behavior is documented. Not a real inconsistency. |
| `utils.test.ts:408` | Missing test for .env file content with skip-provider | The E2E test verifies .env structure. Low risk - .env generation uses same code paths. |
| `utils.ts:268-274` | Empty catch blocks for process cleanup | Expected behavior - processes may already be dead. Comments explain intent. |
| `quickstart.test.ts:222-223` | Hardcoded test credentials | CI uses seeded admin credentials, not dynamically generated ones. This is by design. |
| `quickstart.test.ts:279-286` | Incorrect stdout/stderr reading from devProcess | Pre-existing issue - the accumulated `serverOutput` variable is logged elsewhere. Not introduced by this PR. |
</details>
<details>
<summary>Reviewers (6)</summary>
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|----------|----------|--------------------|----------|-------------------------------|----------------------|-------------------|-----------|
| `pr-review-errors` | 4 | 0 | 0 | 0 | 2 | 0 | 2 |
| `pr-review-product` | 4 | 0 | 1 | 0 | 1 | 0 | 1 |
| `pr-review-consistency` | 1 | 0 | 0 | 0 | 0 | 0 | 1 |
| `pr-review-tests` | 5 | 0 | 0 | 0 | 0 | 0 | 1 |
| `pr-review-devops` | 5 | 0 | 0 | 0 | 0 | 0 | 0 |
| `pr-review-standards` | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **Total** | **19** | **0** | **1** | **0** | **3** | **0** | **5** |
Note: `pr-review-devops` returned all INFO-level findings (confirmations of good patterns); `pr-review-standards` returned no findings.
</details>
Review Response1. Empty catch block for screenshots (ACCEPTED)Fixed in 8933c4c. Now logs the full error: } catch (screenshotError) {
console.error('Failed to capture screenshot:', screenshotError);
}2. Dashboard server output not captured (ACCEPTED)Fixed in 8933c4c. Added stdout/stderr handlers in 3. Interactive "Skip" label is ambiguous (DECLINED)The "Skip" label was intentionally chosen during spec design — it aligns with how other CLI tools label skip options (e.g., CI E2E FixAlso fixed the root cause of the CI failure — |
There was a problem hiding this comment.
PR Review Summary
(2) Total Issues | Risk: Low
This is a delta re-review following commit 8933c4c1 which addressed the Major findings from the prior review.
✅ Prior Feedback Addressed
The following issues from the previous review have been resolved:
| Finding | Resolution |
|---|---|
| 🟠 Major: Screenshot error details swallowed | ✅ Now logs full screenshotError object at line 280 |
| 🟠 Major: Dashboard server output not captured | ✅ Added selective output handler at lines 296-306 that logs errors, EADDRINUSE, and ready signals in CI |
The new auth init step (lines 123-139) is a clean fix for the CI race condition — it ensures the default organization exists before the dashboard lap attempts login.
🕐 Pending Recommendations (2)
These items from the prior review remain unaddressed but are lower priority:
- 🟡
utils.ts:193Interactive "Skip" label could be more self-documenting (e.g., "Skip (use mock provider)") - 💭
README.mdConsider documenting--skip-providerin the README for discoverability
✅ APPROVE
Summary: The delta changes adequately address the Major findings from the prior review. Screenshot errors now include details for debugging, and dashboard server output is captured selectively in CI. The new auth init step is a solid fix for the CI race condition. The remaining Minor and Consider items are polish suggestions that don't block merge. Ship it! 🚀
Reviewers (0)
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|---|---|---|---|---|---|---|---|
| Total | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
Note: Delta re-review with narrow scope (1 commit, 2 files) — no sub-agent reviewers dispatched. Prior feedback validation performed directly by orchestrator.
There was a problem hiding this comment.
PR Review Summary
(0) Total Issues | Risk: Low
This is a delta re-review following commit 2a3cfcd6 which adds reliability improvements to the CI E2E test.
✅ Delta Changes Assessment
The delta (1 file, +59 lines) contains four improvements to CI reliability in quickstart.test.ts:
| Change | Assessment |
|---|---|
| Auth init timeout 30s → 120s | ✅ Appropriate for SpiceDB's 30-retry schema write behavior |
| Explicit DB URLs with fallbacks | ✅ Defensive coding for CI environment variability |
| Fallback signup via API | ✅ Idempotent backup path with good comments explaining the "why" |
| Pre-flight login test | ✅ Early diagnostic catch before expensive Playwright operations |
All changes follow the existing error handling patterns in the test (non-fatal warn/continue) and are well-commented.
🕐 Pending Recommendations (2)
These items from prior reviews remain unaddressed but are lower priority:
- 🟡
utils.ts:193Interactive "Skip" label could be more self-documenting (e.g., "Skip (use mock provider)") - 💭
README.mdConsider documenting--skip-providerin the README for discoverability
✅ APPROVE
Summary: The delta changes are solid reliability improvements that increase CI test stability without changing production behavior. The auth init hardening and diagnostic pre-flights should help catch and diagnose issues earlier. The remaining Minor and Consider items from prior reviews are polish suggestions that don't block merge. Ready to ship! 🚢
Reviewers (0)
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|---|---|---|---|---|---|---|---|
| Total | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
Note: Delta re-review with narrow scope (1 file, +59 lines, test reliability improvements only) — no sub-agent reviewers dispatched. Direct orchestrator review.
There was a problem hiding this comment.
PR Review Summary
(0) Total Issues | Risk: Low
This is a delta re-review following commit 8cf020e2 which fixes a cross-origin cookie issue in the E2E test.
✅ Delta Changes Assessment
The delta (1 file, +8/-5 lines) contains a targeted fix for cookie domain alignment:
| Change | Assessment |
|---|---|
Added dashboardApiUrl constant using localhost instead of 127.0.0.1 |
✅ Correct — browsers treat 127.0.0.1 and localhost as different origins for cookies |
Added Origin header to signup/login API requests |
✅ Required for CORS preflight and cookie handling |
Updated dashboard server env vars to use dashboardApiUrl |
✅ Ensures dashboard↔API cookie sharing on same domain |
The comment at line 19-21 clearly explains the rationale: cookies set by the API (localhost:3002) need to be readable by the dashboard (localhost:3000), which requires both to use localhost (not 127.0.0.1).
🕐 Pending Recommendations (2)
These items from prior reviews remain unaddressed but are lower priority:
- 🟡
utils.ts:193Interactive "Skip" label could be more self-documenting (e.g., "Skip (use mock provider)") - 💭
README.mdConsider documenting--skip-providerin the README for discoverability
✅ APPROVE
Summary: The delta is a clean, well-documented fix for the cross-origin cookie issue that was preventing the dashboard E2E test from working correctly. The localhost vs 127.0.0.1 distinction is a real browser behavior that the comments now explain clearly. No new issues identified. Ready to merge! 🎉
Reviewers (0)
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|---|---|---|---|---|---|---|---|
| Total | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
Note: Delta re-review with narrow scope (1 commit, 1 file, +8/-5 lines, test-only cookie fix) — no sub-agent reviewers dispatched. Direct orchestrator review.
There was a problem hiding this comment.
PR Review Summary
(0) Total Issues | Risk: Low
This is a delta re-review following commit 85820602 which improves test selector reliability.
✅ Delta Changes Assessment
The delta (1 file, 1 line change) is a targeted Playwright selector improvement:
| Change | Assessment |
|---|---|
text=${projectId} → a[href*="${projectId}"] |
✅ More robust — href selectors are specific to anchor elements, while text selectors can match any element containing that text. Reduces flakiness from unintended matches. |
This follows Playwright best practices: prefer structural/attribute selectors over text-based selectors when the element is semantically identifiable by its attributes.
🕐 Pending Recommendations (2)
These items from prior reviews remain unaddressed but are lower priority:
- 🟡
utils.ts:193Interactive "Skip" label could be more self-documenting (e.g., "Skip (use mock provider)") — PR author declined with rationale (aligns with other CLI tools, sibling option length consistency) - 💭
README.mdConsider documenting--skip-providerin the README for discoverability
✅ APPROVE
Summary: The delta is a single-line test reliability improvement — switching from a text selector to a more specific href selector for the project link click. This is a standard Playwright best practice that reduces test flakiness. No new issues. The PR is ready to merge! 🎉
Reviewers (0)
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|---|---|---|---|---|---|---|---|
| Total | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
Note: Delta re-review with minimal scope (1 commit, 1 line, test selector improvement only) — no sub-agent reviewers dispatched. Direct orchestrator review.
There was a problem hiding this comment.
PR Review Summary
(0) Total Issues | Risk: Low
This is a full review of PR #2063 following a manual --full-review request. Five domain-specific reviewers were dispatched.
✅ Review Assessment
All five reviewers returned clean assessments with no actionable findings:
| Reviewer | Findings | Assessment |
|---|---|---|
pr-review-standards |
0 | Code quality is solid; no bugs, performance issues, or AGENTS.md violations identified |
pr-review-product |
0 | Customer mental-model is clear; the --skip-provider flag follows established CLI patterns |
pr-review-consistency |
0 | The --skip-provider flag follows the existing --skip-* naming pattern; interactive "Skip" option is consistent with sibling options |
pr-review-tests |
0 | Test coverage is appropriate; unit tests cover skip-provider behaviors, E2E test validates full user journey |
pr-review-devops |
0 | CI changes follow established patterns; Playwright cache/install mirrors main CI job patterns |
📋 Implementation Quality
The --skip-provider feature is well-implemented:
- CLI flag integration: Correctly defined in Commander with clear description (
'Skip AI provider setup (uses mock provider)') - Interactive flow: The "Skip" option includes a helpful
hintproperty for discoverability without lengthening the label - Precedence handling:
skipProvidercorrectly takes precedence over API keys with an informative warning - Model configuration: Uses
defaultMockModelConfigurationswhich follows the appropriate pattern for a minimal mock provider - Test coverage: 3 unit tests + comprehensive E2E test covering the full user journey (CLI → API → dashboard → playground)
The dashboard lap E2E extension is robust:
- Proper Playwright selectors (href-based for cards, id-based for playground pane)
- Appropriate error handling with screenshot capture on failure
- Server output logging in CI for debuggability
- Reasonable timeouts for CI environment
🕐 Pending Recommendations (1)
This item from prior reviews remains unaddressed but is lower priority:
- 💭
README.mdConsider documenting--skip-providerin the README for discoverability — feature is designed for first-contact exploration
✅ APPROVE
Summary: This PR is well-implemented and has been thoroughly validated through 14+ review iterations. The --skip-provider flag follows existing CLI conventions, the mock provider integration is clean, and the dashboard E2E lap provides valuable coverage for the full user journey. The delta since the last review is a single-line Playwright selector fix (removing invalid pierce/ prefix). All prior Major findings have been addressed. Ship it! 🚀
Reviewers (5)
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|---|---|---|---|---|---|---|---|
pr-review-standards |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-product |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-consistency |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-tests |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-devops |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Total | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
Note: All reviewers returned clean assessments. The single Pending Recommendation is a documentation suggestion carried forward from prior reviews.
e62b01b to
d43e1a7
Compare
There was a problem hiding this comment.
PR Review Summary
(0) Total Issues | Risk: Low
This is a full review of PR #2063. Five domain-specific reviewers were dispatched.
✅ Review Assessment
All five reviewers returned clean assessments with no actionable findings:
| Reviewer | Findings | Assessment |
|---|---|---|
pr-review-standards |
0 | Code quality is solid; no bugs, performance issues, or AGENTS.md violations identified |
pr-review-product |
0 | Customer mental-model is clear; the --skip-provider flag follows established CLI patterns and includes appropriate discoverability via the hint property |
pr-review-consistency |
0 | The --skip-provider flag follows the existing --skip-* naming pattern; interactive "Skip" option is consistent with sibling provider options |
pr-review-tests |
0 | Test coverage is appropriate; 3 unit tests cover skip-provider behaviors, E2E test validates full user journey |
pr-review-devops |
0 | CI changes follow established patterns; Playwright cache/install mirrors main CI job; build filter addition is necessary |
📋 Implementation Quality
The --skip-provider feature is well-implemented:
- CLI flag integration: Correctly defined in Commander with clear description (
'Skip AI provider setup (uses mock provider)') - Interactive flow: The "Skip" option includes a helpful
hintproperty ('scaffolds with mock provider, no API key needed') for discoverability without lengthening the label - Precedence handling:
skipProvidercorrectly takes precedence over API keys with an informative warning - Model configuration: Uses
defaultMockModelConfigurationswhich follows the appropriate pattern for a minimal mock provider (onlybasedefined, consistent with existing templates) - Test coverage: 3 unit tests cover the CLI flag, interactive selection, and flag precedence behaviors
The dashboard lap E2E extension is robust:
- Proper Playwright selectors (href-based for cards, id-based for playground pane)
- Appropriate error handling with screenshot capture on failure
- Server output logging in CI for debuggability
- Reasonable timeouts for CI environment
- Explicit deferral of message sending (documented in PR) due to iframe complexity
🕐 Pending Recommendations (1)
This item from prior reviews remains unaddressed but is lower priority:
- 💭
README.mdConsider documenting--skip-providerin the README for discoverability — feature is designed for first-contact exploration
✅ APPROVE
Summary: This PR is well-implemented and ready to merge. The --skip-provider flag follows existing CLI conventions, enables a useful onboarding path for developers without API credentials, and the dashboard E2E lap provides valuable coverage for the full user journey. All prior Major findings have been addressed, and the remaining documentation suggestion is optional polish. Ship it! 🚀
Reviewers (5)
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|---|---|---|---|---|---|---|---|
pr-review-standards |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-product |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-consistency |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-tests |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-devops |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Total | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
Note: All reviewers returned clean assessments. The single Pending Recommendation is a documentation suggestion carried forward from prior reviews.
Add "Skip" option to create-agents CLI provider selection that defaults to mock/default model, enabling project scaffolding without an API key. Add browser-based dashboard validation to the quickstart E2E test using Playwright: login, navigate to project/agent, open playground. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace overly broad iframe/[class*="chat"] selectors with [role="dialog"]/[class*="playground"] for more specific matching - Fix variable shadowing in catch block (page → activePage) - Remove inline comments per no-comments convention - Add changeset for @inkeep/create-agents --skip-provider feature Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Run auth init separately after setup-dev:cloud to create the "default" organization even when the template's internal migrations fail (CI already applies migrations from the monorepo root) - Log dashboard server error details in screenshot catch block - Capture dashboard server stdout/stderr for CI debugging Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Increase auth init timeout to 120s (SpiceDB retries up to 30x) - Add stream: true to see auth init output in CI logs - Add explicit DB URLs to auth init env (belt and suspenders) - Log auth init exit code and output on failure - Add signup fallback via Better Auth HTTP endpoint after API is ready - Add login API test before Playwright to diagnose auth issues Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ssue The dashboard (localhost:3000) was configured with API URL 127.0.0.1:3002, causing auth cookies set by the API to not be sent to the dashboard (different hostname = different cookie domain). Use localhost:3002 for the dashboard's API URL to match the Cypress test setup where login works correctly. Also add Origin headers to diagnostic fetch calls to satisfy Better Auth's CSRF protection (MISSING_OR_NULL_ORIGIN error). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The project card displays "Activities planner" (display name) but the test was looking for "activities-planner" (project ID). Use href-based locator which matches the URL containing the project ID. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cards use a linkoverlay pattern (data-slot="linkoverlay") that intercepts pointer events. Playwright's actionability check rejects clicks intercepted by overlays, so force:true is needed for project card, agent card, and Try it button clicks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Next.js pages with SSE/streaming connections never fire the load event, causing waitForURL to timeout despite the URL matching correctly. domcontentloaded fires once the HTML is parsed, which is sufficient for navigation verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
text=Activities planner matched the project heading/breadcrumb instead of the agent card, preventing navigation. Use a[href*="/agents/activities-planner"] for precise targeting, matching the pattern used for the project card click. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The playground uses a ResizablePanel with id="playground-pane", not data-testid or role attributes. The previous selectors ([data-testid="playground-panel"], [role="dialog"], [class*="playground"]) didn't match the actual DOM structure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t assertion - Add `hint` field to interactive "Skip" option for discoverability - Warn when --skip-provider is used alongside explicit API key flags - Assert InkeepEmbeddedChat widget initializes inside playground panel Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…format Playwright CSS selectors pierce open shadow DOM by default — the `pierce/` prefix is Chrome DevTools Protocol syntax, not Playwright. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
d43e1a7 to
396949d
Compare
There was a problem hiding this comment.
PR Review Summary
(0) Total Issues | Risk: Low
This is a full review of PR #2063. Five domain-specific reviewers were dispatched.
✅ Review Assessment
All five reviewers returned clean assessments with no actionable findings:
| Reviewer | Findings | Assessment |
|---|---|---|
pr-review-standards |
0 | Code quality is solid; no bugs, performance issues, or AGENTS.md violations identified |
pr-review-product |
0 | Customer mental-model is clear; the --skip-provider flag follows established CLI patterns |
pr-review-consistency |
0 | The --skip-provider flag follows the existing --skip-* naming pattern; interactive "Skip" option is consistent with sibling provider options |
pr-review-tests |
0 | Test coverage is appropriate; 3 unit tests cover skip-provider behaviors, E2E test validates full user journey |
pr-review-devops |
0 | CI changes follow established patterns; Playwright cache/install mirrors main CI job patterns |
📋 Implementation Quality
The --skip-provider feature is well-implemented:
- CLI flag integration: Correctly defined in Commander with clear description (
'Skip AI provider setup (uses mock provider)') - Interactive flow: The "Skip" option includes a helpful
hintproperty ('scaffolds with mock provider, no API key needed') for discoverability - Precedence handling:
skipProvidercorrectly takes precedence over API keys with an informative warning - Model configuration: Uses
defaultMockModelConfigurationswhich follows the appropriate pattern (onlybasedefined, consistent with existing templates) - Test coverage: 3 unit tests cover CLI flag, interactive selection, and flag precedence behaviors; E2E test validates full dashboard journey
The dashboard lap E2E extension is robust:
- Proper Playwright selectors (href-based for cards, id-based for playground pane)
- Appropriate error handling with screenshot capture on failure
- Server output logging in CI for debuggability
- Reasonable timeouts for CI environment
- Explicit deferral of message sending (documented in PR) due to iframe complexity
🕐 Pending Recommendations (1)
This item from prior reviews remains unaddressed but is lower priority:
- 💭
README.mdConsider documenting--skip-providerin the README for discoverability — feature is designed for first-contact exploration
✅ APPROVE
Summary: This PR is well-implemented and ready to merge. The --skip-provider flag follows existing CLI conventions, enables a useful onboarding path for developers without API credentials, and the dashboard E2E lap provides valuable coverage for the full user journey. All prior Major findings have been addressed, and the remaining documentation suggestion is optional polish. Ship it! 🚀
Reviewers (5)
| Reviewer | Returned | Main Findings | Consider | While You're Here | Inline Comments | Pending Recs | Discarded |
|---|---|---|---|---|---|---|---|
pr-review-standards |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-product |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-consistency |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-tests |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
pr-review-devops |
0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Total | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
Note: All reviewers returned clean assessments. The single Pending Recommendation is a documentation suggestion carried forward from prior reviews.
Summary
Adds a
--skip-provideroption tocreate-agentsCLI and extends the quickstart E2E test with a full dashboard lap — CLI scaffolding through browser login, navigation, and playground panel verification.Motivation
User scenario 1: Zero-friction quickstart. New users and CI pipelines shouldn't need a real API key to scaffold and explore a working project.
--skip-providerlets you scaffold with themock/defaultmodel (landed in #2056), sopnpm create-agents my-project --skip-providerproduces a fully functional project end-to-end — including hitting the chat endpoint from the playground — without any provider credentials.User scenario 2: Full-journey regression coverage. The existing E2E test stopped at "API returns project metadata." That left the dashboard login → project navigation → agent detail → playground flow completely uncovered. This PR extends coverage to catch regressions across the entire first-run experience a user would walk through.
Approach
--skip-providerCLI optionTwo entry points, same behavior:
--skip-providerflag — non-interactive, scaffolds with{ base: { model: 'mock/default' } }model config--skip-providertakes precedence over any--openai-key/--anthropic-keyflags passed alongside it (logs a warning rather than silently ignoring).Dashboard lap E2E
The test now continues after
inkeep pushsucceeds:startDashboardServer()— forks the Next.js standaloneserver.jsfrom the built@inkeep/agents-manage-uipackage)admin@example.com)#playground-panevisible and#inkeep-widget-rootinitializedOn failure: captures full-page screenshot + page URL + HTML content for debugging.
Architectural decisions
Playwright over Cypress for E2E dashboard lap. The existing Cypress test suite is for the manage-ui package itself. This test lives in
create-agentsand verifies the scaffolded project's dashboard works end-to-end. Playwright is lighter to install in CI (just chromium), doesn't need a dev server, and works well with the standalone Next.js output.Standalone server, not
next dev. The dashboard is started from the pre-built.next/standalone/agents-manage-ui/server.jsvianode:child_process.fork(). This is faster and more realistic — it tests the same artifact that ships to production.localhostvs127.0.0.1split. The API server uses127.0.0.1:3002(avoids IPv6 resolution issues on Ubuntu CI), but the dashboard must uselocalhost:3002so auth cookies share the same domain between dashboard (:3000) and API (:3002). Both arelocalhost-scoped cookies.Auth init reliability. CI applies migrations from the monorepo root before the test runs, so
setup-dev:cloud's internal migration step can fail and exit early — skipping auth init entirely. Fixed by running auth init as a separate step. Also added an idempotent signup-via-API fallback (POST /api/auth/sign-up/email) as a belt-and-suspenders measure.Selector strategy. Card components use a link-overlay pattern that intercepts pointer events, so clicks use
a[href*="..."]withforce: truerather than text-based selectors.waitForURLusesdomcontentloadedinstead ofloadto avoid timeout from slow external resource loading.Known limitation
Message sending in the playground (typing + receiving a response via the chat widget) is not yet tested. The
InkeepEmbeddedChatwidget renders in a shadow DOM / iframe that needs additional DOM verification work in headless Chromium. The test gracefully stops at asserting the playground panel opens and the widget root initializes. Can be added in a follow-up.Test coverage
Automated (unit tests —
utils.test.ts)--skip-providerflagmock/defaultmodel config, skips provider prompt, skips API key prompt--skip-provider+--openai-key→ mock config wins (key ignored)Automated (E2E —
quickstart.test.ts)--skip-provider.envcontents (noOPENAI_API_KEY),package.jsonshapeinkeep push#playground-panevisible#inkeep-widget-rootelement present in DOMManually QA-tested (not in automated suite)
--skip-providerCLI flow locally — scaffolds withmock/default, no API key prompt--skip-provider+--openai-keytogether — warning logged, mock config usedCI status
All checks green: CI, E2E, auto-format, Vercel previews, PR review.
Future considerations
--skip-providerin README — document the flag for discoverability (surfaced in review, deferred).🤖 Generated with Claude Code