diff --git a/README.md b/README.md index 9c5489405..786f7c1c3 100644 --- a/README.md +++ b/README.md @@ -251,7 +251,7 @@ Two examples already run at this load. The [deep-research engine](https://agentf - **[Reasoners & Skills](https://agentfield.ai/docs/build/building-blocks/reasoners?utm_source=github-readme&utm_campaign=github-readme&utm_id=github-readme-reasoners)** - `@app.reasoner()` for AI judgment, `@app.skill()` for deterministic code - **[Structured AI](https://agentfield.ai/docs/reference/sdks/python?utm_source=github-readme&utm_campaign=github-readme&utm_id=github-readme-structured-ai)** - `app.ai(schema=MyModel)` → typed Pydantic/Zod output from any LLM -- **[Harness](https://agentfield.ai/docs/build/intelligence/harness?utm_source=github-readme&utm_campaign=github-readme&utm_id=github-readme-harness)** - `app.harness("Fix the bug")` dispatches multi-turn tasks to AForge, AgentField's own coding harness — no setup. Add `provider="claude-code"` (or `codex`, `gemini`, `opencode`) to orchestrate someone else's. +- **[Harness](https://agentfield.ai/docs/build/intelligence/harness?utm_source=github-readme&utm_campaign=github-readme&utm_id=github-readme-harness)** - `app.harness("Fix the bug")` dispatches multi-turn tasks to AForge, AgentField's own coding harness — no setup. Add `provider="claude-code"` (or `codex`, `gemini`, `opencode`, `pi`, `omp`) to orchestrate someone else's. - **[Cross-Agent Calls](https://agentfield.ai/docs/build/coordination/cross-agent-calls?utm_source=github-readme&utm_campaign=github-readme&utm_id=github-readme-cross-agent-calls)** - `app.call("other-agent.func")` routes through the control plane with full tracing - **[Discovery](https://agentfield.ai/docs/reference/sdks/python?utm_source=github-readme&utm_campaign=github-readme&utm_id=github-readme-discovery)** - `app.discover(tags=["ml*"])` finds agents and capabilities across the mesh. `tools="discover"` lets LLMs auto-invoke them. - **[Memory](https://agentfield.ai/docs/build/coordination/shared-memory?utm_source=github-readme&utm_campaign=github-readme&utm_id=github-readme-memory)** - `app.memory.set()` / `.get()` / `.similarity_search()` - KV + vector search, four scopes, no Redis needed @@ -284,7 +284,7 @@ Two examples already run at this load. The [deep-research engine](https://agentf |---|---| | Structured output (Pydantic/Zod) | `app.ai(schema=MyModel)` | | Multi-turn coding agents | `app.harness("task")` — AForge by default | -| Orchestrate another harness | `app.harness("task", provider="claude-code")` | +| Orchestrate another harness | `app.harness("task", provider="claude-code")` (also `codex`, `gemini`, `opencode`, `pi`, `omp`) | | LLM auto-discovers agents and tools | `app.ai(tools="discover")` | | Multimodal (text, image, audio) | `app.ai("Describe", image_url="...")` | | Streaming responses | `app.ai("...", stream=True)` | @@ -381,7 +381,7 @@ Two examples already run at this load. The [deep-research engine](https://agentf | Feature | How | |---|---| | Zero-setup default harness | AForge (`aforge`), installed alongside `af` | -| Swap the worker, keep the loop | `provider="claude-code"` \| `"codex"` \| `"gemini"` \| `"opencode"` | +| Swap the worker, keep the loop | `provider="claude-code"` \| `"codex"` \| `"gemini"` \| `"opencode"` \| `"pi"` \| `"omp"` | | Fleet-wide default override | `AGENTFIELD_HARNESS_PROVIDER=codex` | | Schema-constrained output | `schema=ResultModel` (Pydantic/Zod) | | Cost capping | `max_budget_usd=3.0` | diff --git a/control-plane/internal/cli/doctor.go b/control-plane/internal/cli/doctor.go index 5ae306875..832574701 100644 --- a/control-plane/internal/cli/doctor.go +++ b/control-plane/internal/cli/doctor.go @@ -79,16 +79,31 @@ var providerEnvVars = []struct { {Name: "google", EnvVar: "GOOGLE_API_KEY", Model: "gemini-1.5-pro"}, } -// harnessProviders is the canonical list of CLIs `app.harness()` knows how to drive. -var harnessProviders = []struct { - Name string // value passed to provider= in app.harness() - Binary string // executable name to look up on PATH - ProbeArgs []string // minimal one-shot invocation used by `--probe` -}{ +// doctorHarnessProvider describes one CLI `app.harness()` knows how to drive, +// as `af doctor` surveys it. +type doctorHarnessProvider struct { + Name string // value passed to provider= in app.harness() + Binary string // executable name to look up on PATH + ProbeArgs []string // minimal one-shot invocation used by `--probe`; empty means "never probe" + ProbeStdin string // prompt fed over stdin, mirroring how the SDK adapters invoke this CLI + // JSONLStream marks providers whose probe stdout is a JSON event stream + // rather than plain text, so a non-empty stdout does not by itself mean + // the provider completed anything. + JSONLStream bool +} + +// harnessProviders is the canonical list of CLIs `app.harness()` knows how to +// drive. aforge leads it: it is the SDK's default provider and ships with `af`. +var harnessProviders = []doctorHarnessProvider{ + // aforge declares no ProbeArgs on purpose — see runHarnessProbes. + {Name: "aforge", Binary: "aforge"}, {Name: "claude-code", Binary: "claude", ProbeArgs: []string{"-p", "Say OK"}}, {Name: "codex", Binary: "codex", ProbeArgs: []string{"exec", "Say OK"}}, {Name: "gemini", Binary: "gemini", ProbeArgs: []string{"-p", "Say OK"}}, {Name: "opencode", Binary: "opencode", ProbeArgs: []string{"run", "Say OK"}}, + // Keep pi/omp stdin prompt delivery coupled to the SDK adapters. + {Name: "pi", Binary: "pi", ProbeArgs: []string{"--print", "--mode", "json"}, ProbeStdin: "Say OK", JSONLStream: true}, + {Name: "omp", Binary: "omp", ProbeArgs: []string{"--print", "--mode", "json"}, ProbeStdin: "Say OK", JSONLStream: true}, } // harnessProbeTimeout bounds a single provider smoke test. Coding-agent CLIs @@ -119,7 +134,7 @@ func NewDoctorCommand() *cobra.Command { Long: `Doctor inspects the local environment and reports what's available for building AgentField multi-reasoner systems: - • Available harness provider CLIs (claude-code, codex, gemini, opencode) + • Available harness provider CLIs (aforge, claude-code, codex, gemini, opencode, pi, omp) • Provider API keys set in the environment (without leaking values) • Docker availability and whether the control-plane image is locally cached • Whether a local control plane is reachable @@ -175,17 +190,23 @@ func runHarnessProbes(report DoctorReport) map[string]HarnessProbeResult { if !report.HarnessProviders[h.Name].Available { continue } - results[h.Name] = probeHarnessProvider(h.Name, h.Binary, h.ProbeArgs, harnessProbeTimeout) + // Every other probe is one trivial completion. Aforge's only one-shot is + // a full coding-agent run with working-directory write access, so `af + // doctor` must not start it; `af harness doctor` reports its health. + if len(h.ProbeArgs) == 0 { + continue + } + results[h.Name] = probeHarnessProvider(h.Name, h.Binary, h.ProbeArgs, h.ProbeStdin, harnessProbeTimeout, h.JSONLStream) } return results } // probeHarnessProvider runs one provider CLI's minimal one-shot invocation and // classifies the outcome. -func probeHarnessProvider(name, binary string, args []string, timeout time.Duration) HarnessProbeResult { +func probeHarnessProvider(name, binary string, args []string, stdin string, timeout time.Duration, jsonlStream bool) HarnessProbeResult { start := time.Now() - stdout, stderr, exitCode, timedOut := runProbeCommand(binary, args, timeout) - status := classifyProbe(exitCode, stdout, timedOut) + stdout, stderr, exitCode, timedOut := runProbeCommand(binary, args, stdin, timeout) + status := classifyProbe(exitCode, stdout, timedOut, jsonlStream) result := HarnessProbeResult{ Provider: name, @@ -196,6 +217,9 @@ func probeHarnessProvider(name, binary string, args []string, timeout time.Durat switch status { case "error": result.Detail = firstLine(stderr) + if result.Detail == "" && jsonlStream { + _, result.Detail = piProbeOutcome(stdout) + } case "timeout": result.Detail = fmt.Sprintf("no response within %s", timeout) case "empty": @@ -207,11 +231,14 @@ func probeHarnessProvider(name, binary string, args []string, timeout time.Durat // runProbeCommand executes bin with args under a timeout, returning stdout, // stderr, the process exit code, and whether the timeout fired. A timeout is // reported distinctly so it is never misclassified as a plain error. -func runProbeCommand(bin string, args []string, timeout time.Duration) (stdout, stderr string, exitCode int, timedOut bool) { +func runProbeCommand(bin string, args []string, stdin string, timeout time.Duration) (stdout, stderr string, exitCode int, timedOut bool) { ctx, cancel := context.WithTimeout(context.Background(), timeout) defer cancel() cmd := exec.CommandContext(ctx, bin, args...) + if stdin != "" { + cmd.Stdin = strings.NewReader(stdin) + } var outBuf, errBuf strings.Builder cmd.Stdout = &outBuf cmd.Stderr = &errBuf @@ -232,20 +259,86 @@ func runProbeCommand(bin string, args []string, timeout time.Duration) (stdout, } // classifyProbe maps a probe outcome to a status. Order matters: a timeout is -// checked before the exit code (a killed process also exits non-zero), and an -// empty completion on a clean exit is the real-world "silently broken provider" -// case that a mere PATH check misses. -func classifyProbe(exitCode int, stdout string, timedOut bool) string { +// checked before the exit code (a killed process also exits non-zero). Plain +// text probes require non-empty stdout; JSONL probes require successful +// assistant completion text rather than merely any event. +func classifyProbe(exitCode int, stdout string, timedOut bool, jsonlStream bool) string { switch { case timedOut: return "timeout" case exitCode != 0: return "error" - case strings.TrimSpace(stdout) == "": - return "empty" - default: + } + if jsonlStream { + hasAssistantText, providerError := piProbeOutcome(stdout) + if providerError != "" { + return "error" + } + if !hasAssistantText { + return "empty" + } return "ok" } + if strings.TrimSpace(stdout) == "" { + return "empty" + } + return "ok" +} + +// piProbeOutcome inspects a Pi-family JSON event stream the way the SDK +// adapters do. A completion requires assistant text, and only the last +// assistant message_end determines whether the provider stopped with an error. +func piProbeOutcome(stdout string) (hasAssistantText bool, providerError string) { + type message struct { + Role string `json:"role"` + Content json.RawMessage `json:"content"` + StopReason string `json:"stopReason"` + ErrorMessage string `json:"errorMessage"` + } + type event struct { + Type string `json:"type"` + Message message `json:"message"` + } + type contentPart struct { + Type string `json:"type"` + Text string `json:"text"` + } + + for _, line := range strings.Split(stdout, "\n") { + if strings.TrimSpace(line) == "" { + continue + } + var e event + if err := json.Unmarshal([]byte(line), &e); err != nil || e.Type != "message_end" || e.Message.Role != "assistant" { + continue + } + + var text string + if err := json.Unmarshal(e.Message.Content, &text); err != nil { + var parts []contentPart + if json.Unmarshal(e.Message.Content, &parts) == nil { + for _, part := range parts { + if part.Type == "text" { + text += part.Text + } + } + } + } + if strings.TrimSpace(text) != "" { + hasAssistantText = true + } + + switch e.Message.StopReason { + case "error", "aborted": + providerError = e.Message.ErrorMessage + if providerError == "" { + providerError = fmt.Sprintf("stopped with reason %q", e.Message.StopReason) + } + default: + providerError = "" + } + } + return hasAssistantText, providerError } // firstLine returns the first non-empty line of s, trimmed, for compact error @@ -279,7 +372,20 @@ func buildDoctorReport(controlPlaneURL string) DoctorReport { // Harness CLIs availableHarness := []string{} for _, h := range harnessProviders { - status := checkTool(h.Binary, "--version") + // Share the harness doctor's provider-specific version arguments and its + // $AGENTFIELD_HOME/bin fallback wherever a binary-backed spec exists, so + // both doctors agree on what "installed" means. aforge in particular + // answers `version`, not `--version`, and `af aforge ensure` puts it in + // AgentField's own bin directory rather than on PATH. claude-code has no + // binary in that table (it is the pip-package wrapper), so it keeps the + // plain PATH check. + spec := findHarnessProviderSpec(h.Name) + var status ToolStatus + if spec != nil && spec.Binary != "" { + status = probeHarnessBinary(*spec) + } else { + status = checkTool(h.Binary, "--version") + } report.HarnessProviders[h.Name] = status if status.Available { availableHarness = append(availableHarness, h.Name) diff --git a/control-plane/internal/cli/doctor_additional_test.go b/control-plane/internal/cli/doctor_additional_test.go index 602f0fdff..00d163e82 100644 --- a/control-plane/internal/cli/doctor_additional_test.go +++ b/control-plane/internal/cli/doctor_additional_test.go @@ -77,6 +77,8 @@ func TestDoctorHelpersAndCommand(t *testing.T) { "codex": {}, "gemini": {}, "opencode": {}, + "pi": {}, + "omp": {}, }, ProviderKeys: map[string]ProviderKey{ "openrouter": {EnvVar: "OPENROUTER_API_KEY", Set: true}, diff --git a/control-plane/internal/cli/doctor_probe_test.go b/control-plane/internal/cli/doctor_probe_test.go index e4274b400..b265c6d26 100644 --- a/control-plane/internal/cli/doctor_probe_test.go +++ b/control-plane/internal/cli/doctor_probe_test.go @@ -8,6 +8,9 @@ import ( "net/http/httptest" "os" "os/exec" + "path/filepath" + "reflect" + "runtime" "testing" "time" ) @@ -33,13 +36,73 @@ func TestClassifyProbe(t *testing.T) { } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - if got := classifyProbe(tc.exitCode, tc.stdout, tc.timedOut); got != tc.want { + if got := classifyProbe(tc.exitCode, tc.stdout, tc.timedOut, false); got != tc.want { t.Errorf("classifyProbe(%d, %q, %v) = %q, want %q", tc.exitCode, tc.stdout, tc.timedOut, got, tc.want) } }) } } +func TestClassifyProbe_JSONLStream(t *testing.T) { + cases := []struct { + name string + stdout string + want string + }{ + {"session only", `{"type":"session","id":"s1"}`, "empty"}, + {"assistant text", "{\"type\":\"session\",\"id\":\"s1\"}\n{\"type\":\"message_end\",\"message\":{\"role\":\"assistant\",\"content\":\"OK\",\"stopReason\":\"stop\"}}", "ok"}, + {"provider error", `{"type":"message_end","message":{"role":"assistant","content":"partial","stopReason":"error","errorMessage":"provider failed"}}`, "error"}, + {"recovered", "{\"type\":\"message_end\",\"message\":{\"role\":\"assistant\",\"content\":\"partial\",\"stopReason\":\"error\"}}\n{\"type\":\"message_end\",\"message\":{\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"OK\"}],\"stopReason\":\"stop\"}}", "ok"}, + {"empty text part", `{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":""}],"stopReason":"stop"}}`, "empty"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := classifyProbe(0, tc.stdout, false, true); got != tc.want { + t.Errorf("classifyProbe JSONL = %q, want %q", got, tc.want) + } + }) + } +} + +func TestHarnessProviders_JSONLStream(t *testing.T) { + var got []string + for _, provider := range harnessProviders { + if provider.JSONLStream { + got = append(got, provider.Name) + } + } + if want := []string{"pi", "omp"}; !reflect.DeepEqual(got, want) { + t.Errorf("JSONLStream providers = %v, want %v", got, want) + } +} + +func TestHarnessProviders_ProbeInputContract(t *testing.T) { + wantStdinArgs := []string{"--print", "--mode", "json"} + for _, provider := range harnessProviders { + switch provider.Name { + case "pi", "omp": + if !reflect.DeepEqual(provider.ProbeArgs, wantStdinArgs) { + t.Errorf("%s ProbeArgs = %v, want %v", provider.Name, provider.ProbeArgs, wantStdinArgs) + } + if provider.ProbeStdin == "" { + t.Errorf("%s ProbeStdin must be non-empty", provider.Name) + } + for _, arg := range provider.ProbeArgs { + if arg == provider.ProbeStdin { + t.Errorf("%s prompt %q must not appear in ProbeArgs", provider.Name, provider.ProbeStdin) + } + } + default: + if provider.Name == "aforge" && (len(provider.ProbeArgs) != 0 || provider.ProbeStdin != "") { + t.Errorf("aforge must not declare probe input") + } + if provider.ProbeStdin != "" { + t.Errorf("positional-prompt provider %s ProbeStdin = %q, want empty", provider.Name, provider.ProbeStdin) + } + } + } +} + // End-to-end wiring of runProbeCommand -> classifyProbe over real processes, so // each classification path is exercised through the actual command runner. func TestProbeHarnessProvider_RealProcesses(t *testing.T) { @@ -47,20 +110,23 @@ func TestProbeHarnessProvider_RealProcesses(t *testing.T) { name string bin string args []string + stdin string timeout time.Duration want string }{ - {"ok", "echo", []string{"OK"}, 5 * time.Second, "ok"}, - {"empty", "true", nil, 5 * time.Second, "empty"}, - {"error", "false", nil, 5 * time.Second, "error"}, - {"timeout", "sleep", []string{"5"}, 200 * time.Millisecond, "timeout"}, + {"ok", "echo", []string{"OK"}, "", 5 * time.Second, "ok"}, + {"empty", "true", nil, "", 5 * time.Second, "empty"}, + {"error", "false", nil, "", 5 * time.Second, "error"}, + {"timeout", "sleep", []string{"5"}, "", 200 * time.Millisecond, "timeout"}, + {"stdin", "cat", nil, "Say OK", 5 * time.Second, "ok"}, + {"empty stdin", "cat", nil, "", 5 * time.Second, "empty"}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { if _, err := exec.LookPath(tc.bin); err != nil { t.Skipf("%s not available: %v", tc.bin, err) } - res := probeHarnessProvider("prov-"+tc.name, tc.bin, tc.args, tc.timeout) + res := probeHarnessProvider("prov-"+tc.name, tc.bin, tc.args, tc.stdin, tc.timeout, false) if res.Status != tc.want { t.Errorf("status = %q, want %q (detail=%q)", res.Status, tc.want, res.Detail) } @@ -71,15 +137,105 @@ func TestProbeHarnessProvider_RealProcesses(t *testing.T) { } } +// Contract: aforge is surveyed by doctor but never smoke-tested — its only +// one-shot is a full coding-agent run, so it deliberately declares no ProbeArgs. +func TestHarnessProviders_AforgeIsSurveyedButNotProbed(t *testing.T) { + if harnessProviders[0].Name != "aforge" { + t.Fatalf("aforge must lead the provider list, got %q", harnessProviders[0].Name) + } + if len(harnessProviders[0].ProbeArgs) != 0 || harnessProviders[0].ProbeStdin != "" { + t.Errorf("aforge must declare no probe input, got %+v", harnessProviders[0]) + } +} + +// Contract: buildDoctorReport detects aforge through its harness specification, +// including its managed install location and aforge-specific version argument. +func TestBuildDoctorReport_AforgeDetectionUsesHarnessSpec(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("shell fixture is Unix-only") + } + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + t.Run("found in AGENTFIELD_HOME/bin when not on PATH", func(t *testing.T) { + t.Setenv("PATH", t.TempDir()) + home := t.TempDir() + t.Setenv("AGENTFIELD_HOME", home) + managedBin := filepath.Join(home, "bin") + if err := os.MkdirAll(managedBin, 0o755); err != nil { + t.Fatalf("create managed bin: %v", err) + } + writeHarnessTestScript(t, managedBin, "aforge", "printf 'aforge 1.2.3\\n'") + + report := buildDoctorReport(srv.URL) + status := report.HarnessProviders["aforge"] + if !status.Available { + t.Errorf("aforge should be available, got %+v", status) + } + if status.Version != "aforge 1.2.3" { + t.Errorf("aforge version = %q, want %q", status.Version, "aforge 1.2.3") + } + if want := filepath.Join(home, "bin", "aforge"); status.Path != want { + t.Errorf("aforge path = %q, want %q", status.Path, want) + } + }) + + t.Run("version comes from the aforge-specific version argument", func(t *testing.T) { + binDir := t.TempDir() + t.Setenv("PATH", binDir) + t.Setenv("AGENTFIELD_HOME", t.TempDir()) + writeHarnessTestScript(t, binDir, "aforge", "[ \"$1\" = version ] && { printf 'aforge 9.9.9\\n'; exit 0; }; exit 1") + + report := buildDoctorReport(srv.URL) + status := report.HarnessProviders["aforge"] + if !status.Available { + t.Errorf("aforge should be available, got %+v", status) + } + if status.Version != "aforge 9.9.9" { + t.Errorf("aforge version = %q, want %q", status.Version, "aforge 9.9.9") + } + }) +} + +// Contract: a provider with no ProbeArgs is skipped by --probe even when +// detected, while a provider that declares them is still probed. The registry +// is swapped for synthetic entries so no real coding-agent CLI is invoked. +func TestRunHarnessProbes_SkipsProvidersWithoutProbeArgs(t *testing.T) { + original := harnessProviders + t.Cleanup(func() { harnessProviders = original }) + harnessProviders = []doctorHarnessProvider{ + {Name: "no-probe", Binary: "agentfield-absent-no-probe"}, + {Name: "with-probe", Binary: "agentfield-absent-with-probe", ProbeArgs: []string{"--version"}}, + } + + report := DoctorReport{HarnessProviders: map[string]ToolStatus{ + "no-probe": {Available: true}, + "with-probe": {Available: true}, + }} + got := runHarnessProbes(report) + if _, ok := got["no-probe"]; ok { + t.Error("a provider without ProbeArgs must be skipped even when available") + } + if _, ok := got["with-probe"]; !ok { + t.Error("an available provider with ProbeArgs must produce a result") + } +} + // Contract: probes run ONLY for providers doctor already detected — unavailable // providers are never invoked. func TestRunHarnessProbes_SkipsUndetected(t *testing.T) { report := DoctorReport{ HarnessProviders: map[string]ToolStatus{ + "aforge": {Available: false}, "claude-code": {Available: false}, "codex": {Available: false}, "gemini": {Available: false}, "opencode": {Available: false}, + "pi": {Available: false}, + "omp": {Available: false}, }, } got := runHarnessProbes(report) diff --git a/control-plane/internal/cli/harness_doctor.go b/control-plane/internal/cli/harness_doctor.go index 7f1fc6ee4..88b022c4e 100644 --- a/control-plane/internal/cli/harness_doctor.go +++ b/control-plane/internal/cli/harness_doctor.go @@ -47,6 +47,8 @@ var harnessProviderSpecs = []harnessProviderSpec{ {Name: "gemini", Binary: "gemini", InstallCommand: "npm install -g @google/gemini-cli", AuthEnvVars: []string{"GEMINI_API_KEY", "GOOGLE_API_KEY"}}, {Name: "opencode", Binary: "opencode", InstallCommand: "curl -fsSL https://opencode.ai/install | bash", AuthEnvVars: []string{}}, {Name: "grok", Binary: "grok", InstallCommand: "Install the Grok Build CLI, then run: grok login", AuthEnvVars: []string{"XAI_API_KEY"}}, + {Name: "pi", Binary: "pi", InstallCommand: "npm install -g --ignore-scripts @earendil-works/pi-coding-agent", AuthEnvVars: []string{"OPENROUTER_API_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY", "GEMINI_API_KEY", "GOOGLE_API_KEY"}}, + {Name: "omp", Binary: "omp", InstallCommand: "curl -fsSL https://omp.sh/install | sh", AuthEnvVars: []string{"OPENROUTER_API_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY", "GEMINI_API_KEY", "GOOGLE_API_KEY"}}, } // NewHarnessCommand builds harness-related environment checks. @@ -95,7 +97,7 @@ func newHarnessDoctorCommand() *cobra.Command { return nil }, } - cmd.Flags().StringSliceVar(&providers, "provider", nil, "Provider(s) to check: aforge, claude-code, codex, gemini, opencode, grok") + cmd.Flags().StringSliceVar(&providers, "provider", nil, "Provider(s) to check, default all: aforge, claude-code, codex, gemini, opencode, grok, pi, omp") cmd.Flags().BoolVar(&jsonOut, "json", false, "Output structured JSON") return cmd } diff --git a/control-plane/internal/cli/harness_doctor_test.go b/control-plane/internal/cli/harness_doctor_test.go index 807ae4575..d0f07b843 100644 --- a/control-plane/internal/cli/harness_doctor_test.go +++ b/control-plane/internal/cli/harness_doctor_test.go @@ -114,6 +114,47 @@ func TestHarnessDoctorReturnsErrorForRequestedMissingProvider(t *testing.T) { require.Equal(t, []string{"binary_not_found"}, reports[0].Issues) } +func TestHarnessDoctorReportsPiWithOpenRouterAuth(t *testing.T) { + binDir := t.TempDir() + writeHarnessTestBinary(t, binDir, "pi", "0.84.1") + t.Setenv("PATH", binDir) + t.Setenv("OPENROUTER_API_KEY", "configured") + + cmd := NewHarnessCommand() + var stdout bytes.Buffer + cmd.SetOut(&stdout) + cmd.SetArgs([]string{"doctor", "--provider", "pi", "--json"}) + + require.NoError(t, cmd.Execute()) + var reports []HarnessProviderHealth + require.NoError(t, json.Unmarshal(stdout.Bytes(), &reports)) + require.Len(t, reports, 1) + require.Equal(t, "pi", reports[0].Provider) + require.Equal(t, "configured", reports[0].Auth) + require.True(t, reports[0].Usable) +} + +func TestHarnessDoctorReportsOMPWithOfficialInstallCommand(t *testing.T) { + binDir := t.TempDir() + writeHarnessTestBinary(t, binDir, "omp", "17.2.15") + t.Setenv("PATH", binDir) + t.Setenv("OPENROUTER_API_KEY", "configured") + + cmd := NewHarnessCommand() + var stdout bytes.Buffer + cmd.SetOut(&stdout) + cmd.SetArgs([]string{"doctor", "--provider", "omp", "--json"}) + + require.NoError(t, cmd.Execute()) + var reports []HarnessProviderHealth + require.NoError(t, json.Unmarshal(stdout.Bytes(), &reports)) + require.Len(t, reports, 1) + require.Equal(t, "omp", reports[0].Provider) + require.Equal(t, "configured", reports[0].Auth) + require.Equal(t, "curl -fsSL https://omp.sh/install | sh", reports[0].InstallCommand) + require.True(t, reports[0].Usable) +} + func TestHarnessDoctorClaudeCodeReportsInstalledWrapper(t *testing.T) { binDir := t.TempDir() // Stub interpreter standing in for `python3 -c `: prints "ok" as the @@ -284,7 +325,7 @@ func TestHarnessDoctorGrokReportsUsable(t *testing.T) { func TestHarnessProviderSpecsMatchPythonProviderNames(t *testing.T) { // Keep in sync with sdk/python/agentfield/harness/_availability.py. - expected := []string{"aforge", "claude-code", "codex", "gemini", "opencode", "grok"} + expected := []string{"aforge", "claude-code", "codex", "gemini", "opencode", "grok", "pi", "omp"} actual := make([]string, 0, len(harnessProviderSpecs)) for _, spec := range harnessProviderSpecs { actual = append(actual, spec.Name) diff --git a/control-plane/internal/skillkit/skill_data/agentfield/SKILL.md b/control-plane/internal/skillkit/skill_data/agentfield/SKILL.md index 50e50a940..897ff985f 100644 --- a/control-plane/internal/skillkit/skill_data/agentfield/SKILL.md +++ b/control-plane/internal/skillkit/skill_data/agentfield/SKILL.md @@ -68,7 +68,7 @@ Everything else is a variation. Less-used but real: - **`@app.skill()`** — deterministic functions you want callable through the control plane (no LLM). -- **`app.harness(prompt, provider="claude-code"|"codex"|"gemini"|"opencode")`** — delegates to an external coding-agent CLI. Heavy. **Only use when `af doctor` reports `harness_usable: true` AND the Dockerfile installs the CLI AND `shutil.which()` guards startup.** Otherwise use `app.ai(tools=[...])`. +- **`app.harness(prompt, provider="aforge"|"claude-code"|"codex"|"gemini"|"opencode"|"pi"|"omp")`** — delegates to an external coding-agent CLI. `aforge`, AgentField's own harness, is the SDK default when `provider` is omitted. Heavy. **Only use when `af doctor` reports `harness_usable: true` AND the Dockerfile installs the selected CLI AND `shutil.which()` guards startup.** Otherwise use `app.ai(tools=[...])`. Full signatures, schemas, router surface, memory scopes, and the cross-boundary serialization gotcha are in `references/primitives-snapshot.md` (offline-frozen). **Prefer the live `agentfield.ai/llms-full.txt`** when you have a network — it is the source of truth and it does not drift. diff --git a/control-plane/internal/skillkit/skill_data/agentfield/references/primitives-snapshot.md b/control-plane/internal/skillkit/skill_data/agentfield/references/primitives-snapshot.md index e0243c72e..a8314e524 100644 --- a/control-plane/internal/skillkit/skill_data/agentfield/references/primitives-snapshot.md +++ b/control-plane/internal/skillkit/skill_data/agentfield/references/primitives-snapshot.md @@ -16,7 +16,7 @@ This is a minimal cheat sheet of the five primitives so an offline build can sti | `@app.skill()` | Registers a deterministic function (no LLM) | Sort, parse, dedupe, score-with-formula | | `app.ai(...)` | Single call OR multi-turn tool-using LLM call when `tools=` is passed | Classification, routing, structured analysis, stateful tool-using | | `app.call(target, **kwargs)` | Call another reasoner THROUGH the control plane. Returns `dict`. Tracks the workflow DAG | All inter-reasoner traffic | -| `app.harness(prompt, provider=...)` | Delegate to an external coding-agent CLI (claude-code / codex / gemini / opencode) | When you need a real coding agent to write files / run shell | +| `app.harness(prompt, provider=...)` | Delegate to a coding-agent CLI (aforge by default; claude-code / codex / gemini / opencode / pi / omp) | When you need a real coding agent to write files / run shell | --- @@ -157,7 +157,7 @@ Default canonical pattern: `AgentRouter(prefix="", tags=["domain"])`. `prefix="c result = await app.harness( prompt: str, schema: type[BaseModel] | None = None, - provider: "claude-code" | "codex" | "gemini" | "opencode" | None = None, + provider: "aforge" | "claude-code" | "codex" | "gemini" | "opencode" | "pi" | "omp" | None = None, # None -> "aforge" model: str | None = None, max_turns: int | None = None, max_budget_usd: float | None = None, diff --git a/control-plane/internal/skillkit/skill_data/agentfield/references/scaffold-recipe.md b/control-plane/internal/skillkit/skill_data/agentfield/references/scaffold-recipe.md index 83611e289..f1812deed 100644 --- a/control-plane/internal/skillkit/skill_data/agentfield/references/scaffold-recipe.md +++ b/control-plane/internal/skillkit/skill_data/agentfield/references/scaffold-recipe.md @@ -211,7 +211,7 @@ CMD ["python", "main.py"] Build context is the project directory itself (`context: .`), so the same scaffold works whether the project lives in `code/examples/` or standalone at `/tmp/my-build/`. -**Only edit if you need to install a harness CLI** (claude-code, codex, gemini, opencode). See `primitives-snapshot.md` → "Harness availability gate". Otherwise leave it alone. +**Only edit if you need to install a harness CLI** (claude-code, codex, gemini, opencode, pi, omp — `aforge`, the default, ships with `af`). See `primitives-snapshot.md` → "Harness availability gate". Otherwise leave it alone. --- diff --git a/docs/design/harness-v2-design.md b/docs/design/harness-v2-design.md index dd2b998b3..2f00a7204 100644 --- a/docs/design/harness-v2-design.md +++ b/docs/design/harness-v2-design.md @@ -1,15 +1,15 @@ # `.harness()` — First-Class Coding Agent Integration for AgentField -> **Status**: Design Proposal -> **Author**: Architecture brainstorm -> **Scope**: Python SDK + TypeScript SDK (Go SDK tracked separately) +> **Status**: Implemented; updated for Pi/OMP parity (aforge remains the default) +> **Author**: Architecture brainstorm +> **Scope**: Python, TypeScript, and Go SDKs > **Date**: 2026-03-02 --- ## 1. Overview -Add `.harness()` as a first-class method on the Agent class — matching the DX of `.ai()` — enabling developers to dispatch multi-turn coding tasks to external coding agents (Claude Code, Codex, Gemini CLI, OpenCode). +Add `.harness()` as a first-class method on the Agent class — matching the DX of `.ai()` — enabling developers to dispatch multi-turn coding tasks to coding agents (AForge, Claude Code, Codex, Gemini CLI, OpenCode, Pi, and OMP). AForge, AgentField's own harness, is the provider default in every SDK; Pi and OMP are additional providers selected explicitly. **Key principle**: `.ai()` is for single-turn LLM calls. `.harness()` is for multi-turn agentic coding tasks that browse files, edit code, run tests, and iterate. @@ -107,7 +107,7 @@ fix = await app.harness( ### 2.5 Without Constructor Config ```python -# No harness_config on Agent — provide everything per-call +# No harness_config on Agent — the default provider, "aforge", is selected automatically app = Agent(node_id="minimal-agent") result = await app.harness( @@ -207,12 +207,14 @@ class Agent(FastAPI): ### 4.1 Integration Matrix -| Provider | Python | TypeScript | Go (future) | Schema Support | +| Provider | Python | TypeScript | Go | Schema Support | |---|---|---|---|---| | **claude-code** | `claude_agent_sdk` (native Python SDK) | `@anthropic-ai/claude-agent-sdk` (native TS SDK) | CLI subprocess | File-write (universal) | | **codex** | CLI subprocess `codex exec --json` | `@openai/codex-sdk` (native TS SDK) | CLI subprocess | File-write (universal) | | **gemini** | CLI subprocess `gemini --output-format stream-json` | CLI subprocess | CLI subprocess | File-write (universal) | | **opencode** | CLI subprocess | CLI subprocess | CLI subprocess | File-write (universal) | +| **pi** | CLI subprocess | CLI subprocess | CLI subprocess | File-write (universal) | +| **omp** | CLI subprocess | CLI subprocess | CLI subprocess | File-write (universal) | ### 4.2 Why SDK-First Where Available @@ -223,7 +225,7 @@ class Agent(FastAPI): ### 4.3 Why CLI Subprocess for Others -- No native SDK wrapping the CLI agent loop exists for Gemini/OpenCode +- No native SDK wrapping the CLI agent loop exists for Gemini/OpenCode/Pi/OMP - CLI subprocess is lowest-maintenance integration (binary does all the work) - All CLIs output JSONL event streams for consistent parsing - Go SDK will use CLI subprocess for ALL providers (no Go SDKs exist) @@ -245,7 +247,7 @@ interface HarnessProvider { ``` ```go -// Go (future) +// Go type Provider interface { Execute(ctx context.Context, prompt string, opts HarnessOptions) (*RawResult, error) } @@ -262,7 +264,7 @@ type Provider interface { **Why file-write over native flags:** - Native flags constrain the model's *text response* — a wrong abstraction for multi-turn coding agents whose core competency is writing files - Large schemas can produce JSON that exceeds response token limits or gets truncated -- File-write works identically across ALL 4 providers (one code path, half the tests) +- File-write works identically across all providers (one shared orchestration path) - Writing a JSON file is trivially easy for agents that refactor entire codebases ``` @@ -354,42 +356,45 @@ class HarnessConfig(BaseModel): All fields have sensible defaults that can be overridden per-call. """ # Provider selection: explicit > AGENTFIELD_HARNESS_PROVIDER > "aforge" - provider: str = "aforge" # | "claude-code" | "codex" | "gemini" | "opencode" + provider: str = "aforge" # | "claude-code" | "codex" | "gemini" | "opencode" | "pi" | "omp" model: Optional[str] = None # None → the provider's own default # Execution limits max_turns: int = 30 max_budget_usd: Optional[float] = None - + # Retry behavior max_retries: int = 3 initial_delay: float = 1.0 max_delay: float = 30.0 backoff_factor: float = 2.0 - + # Tools & permissions tools: List[str] = Field(default_factory=lambda: [ "Read", "Write", "Edit", "Bash", "Glob", "Grep" ]) permission_mode: Optional[str] = None # "plan" | "auto" | None system_prompt: Optional[str] = None - + # Environment env: Dict[str, str] = Field(default_factory=dict) - + # Binary paths (for CLI-based providers) + aforge_bin: str = "aforge" codex_bin: str = "codex" gemini_bin: str = "gemini" opencode_bin: str = "opencode" + pi_bin: str = "pi" + omp_bin: str = "omp" ``` ### 6.2 HarnessConfig (TypeScript) ```typescript interface HarnessConfig { - /** Required — no default. "claude-code" | "codex" | "gemini" | "opencode" */ - provider: string; - /** Default model identifier. Default: "sonnet" */ + /** 'aforge' when omitted. */ + provider?: 'aforge' | 'claude-code' | 'codex' | 'gemini' | 'opencode' | 'pi' | 'omp'; + /** Omitted uses the selected CLI's configured model. */ model?: string; /** Maximum agent iterations. Default: 30 */ maxTurns?: number; @@ -417,6 +422,9 @@ interface HarnessConfig { geminiBin?: string; /** Path to opencode binary. */ opencodeBin?: string; + /** Paths to Pi-family binaries. */ + piBin?: string; + ompBin?: string; } ``` @@ -440,21 +448,21 @@ interface HarnessConfig { @dataclass class HarnessResult: """Final result from a harness invocation.""" - + # Core output result: Optional[str] # Raw text result parsed: Any # Validated schema instance (T | None) is_error: bool - + # Metrics cost_usd: Optional[float] # Total execution cost num_turns: int # Number of agent turns duration_ms: int # Wall clock time in milliseconds session_id: str # For potential session resume - + # Message history messages: List[Message] # Full conversation history - + # Convenience properties @property def text(self) -> str: @@ -515,7 +523,8 @@ sdk/python/agentfield/harness/ │ ├── claude.py # Claude Code provider (SDK-based: claude_agent_sdk) │ ├── codex.py # Codex provider (CLI-based: codex exec) │ ├── gemini.py # Gemini provider (CLI-based: gemini) -│ └── opencode.py # OpenCode provider (CLI-based: opencode) +│ ├── opencode.py # OpenCode provider (CLI-based: opencode) +│ └── pi.py # Shared Pi and OMP adapters ``` ### 8.2 TypeScript SDK @@ -535,10 +544,11 @@ sdk/typescript/src/harness/ │ ├── claude.ts # Claude Code provider (SDK-based) │ ├── codex.ts # Codex provider (SDK-based: @openai/codex-sdk) │ ├── gemini.ts # Gemini provider (CLI-based) -│ └── opencode.ts # OpenCode provider (CLI-based) +│ ├── opencode.ts # OpenCode provider (CLI-based) +│ └── pi.ts # Shared Pi and OMP adapters ``` -### 8.3 Go SDK (future) +### 8.3 Go SDK ``` sdk/go/harness/ @@ -552,7 +562,8 @@ sdk/go/harness/ ├── claude.go # Claude Code provider (CLI) ├── codex.go # Codex provider (CLI) ├── gemini.go # Gemini provider (CLI) -└── opencode.go # OpenCode provider (CLI) +├── opencode.go # OpenCode provider (CLI) +└── pi.go # Shared Pi and OMP adapters ``` --- @@ -613,7 +624,7 @@ class ErrorKind(str, Enum): class ClaudeCodeProvider: async def execute(self, prompt: str, options: HarnessOptions) -> RawResult: from claude_agent_sdk import query, ClaudeAgentOptions - + opts = ClaudeAgentOptions( model=options.model, cwd=options.cwd, @@ -624,15 +635,15 @@ class ClaudeCodeProvider: permission_mode=options.permission_mode, env=options.env, ) - + messages = [] result_text = None metrics = {} - + async for msg in query(prompt=prompt, options=opts): # Collect messages and extract result ... - + return RawResult(result=result_text, messages=messages, metrics=metrics, is_error=False) ``` @@ -642,17 +653,17 @@ class ClaudeCodeProvider: class CodexProvider: def __init__(self, bin_path: str = "codex"): self.bin = bin_path - + async def execute(self, prompt: str, options: HarnessOptions) -> RawResult: cmd = [self.bin, "exec", "--json"] - + if options.cwd: cmd.extend(["-C", options.cwd]) if options.permission_mode == "auto": cmd.append("--full-auto") - + cmd.append(prompt) - + proc = await asyncio.create_subprocess_exec( *cmd, stdout=asyncio.subprocess.PIPE, @@ -660,7 +671,7 @@ class CodexProvider: env={**os.environ, **options.env}, ) stdout, stderr = await proc.communicate() - + # Parse JSONL events from stdout return self._parse_jsonl_output(stdout.decode()) ``` @@ -671,17 +682,17 @@ class CodexProvider: class GeminiProvider: def __init__(self, bin_path: str = "gemini"): self.bin = bin_path - + async def execute(self, prompt: str, options: HarnessOptions) -> RawResult: cmd = [self.bin, "--output-format", "json"] - + if options.model: cmd.extend(["--model", options.model]) if options.permission_mode == "auto": cmd.extend(["--approval-mode", "yolo"]) - + cmd.append(prompt) - + proc = await asyncio.create_subprocess_exec( *cmd, stdout=asyncio.subprocess.PIPE, @@ -690,7 +701,7 @@ class GeminiProvider: cwd=options.cwd, ) stdout, stderr = await proc.communicate() - + return self._parse_output(stdout.decode(), proc.returncode) ``` @@ -723,11 +734,11 @@ class GeminiProvider: - [ ] Add to factory - [ ] Tests for new providers -### Phase 3: Go SDK Port +### Phase 3: Go SDK Port (complete) - [ ] Port all types to Go structs - [ ] Runner implementation -- [ ] All 4 providers (CLI-based) +- [x] All providers (CLI-based) - [ ] Wire into Go Agent struct - [ ] Tests @@ -774,11 +785,20 @@ class GeminiProvider: ### Go SDK - No new dependencies (os/exec is stdlib) +### Runtime installation lifecycle + +Pi and OMP follow the same explicit runtime lifecycle as Codex, Gemini, and +OpenCode. SDK packages do not install or upgrade external executables. Operators +pin the CLI in their machine or container, verify it with `af harness doctor`, +and may override its path per provider. Missing-binary failures carry the exact +upstream install command. This keeps application startup deterministic and +prevents an SDK call from mutating production hosts. + --- ## 14. Open Questions -1. **Session resume**: Should `.harness()` support resuming a previous session (by `session_id`)? Claude Code and Codex both support this. Could be a follow-up feature. +1. **Session resume**: Implemented through the shared `resume_session_id` option, translated to each provider's native flag. 2. **MCP integration**: Should harness providers be discoverable as MCP tools? Codex already has `codex mcp-server` mode. Could be a follow-up. diff --git a/docs/harness-providers.md b/docs/harness-providers.md index 14483213a..89af372ac 100644 --- a/docs/harness-providers.md +++ b/docs/harness-providers.md @@ -5,7 +5,8 @@ writes, and edits files, then reports back through the same structured-output contract as `app.ai()`. AgentField ships its own harness, **AForge**, and it is the default: a call with no provider set runs `aforge`. Naming a different provider swaps the worker without changing the surrounding loop, which is how -you orchestrate Claude Code, Codex, Gemini CLI, or OpenCode from a reasoner. +you orchestrate Claude Code, Codex, Gemini CLI, OpenCode, Pi, or OMP from a +reasoner. ## Default: AForge @@ -59,7 +60,7 @@ report = await app.harness(task, schema=Report) # Orchestrate Claude Code instead report = await app.harness(task, schema=Report, provider="claude-code") -# ...or Codex, Gemini CLI, OpenCode +# ...or Codex, Gemini CLI, OpenCode, Pi, OMP report = await app.harness(task, schema=Report, provider="codex") ``` @@ -78,6 +79,8 @@ Python, `agent.HarnessConfig{Provider: "codex"}` in Go). | `gemini` | `npm install -g @google/gemini-cli` | None | `gemini` | Gemini login, `GEMINI_API_KEY`, or `GOOGLE_API_KEY` | | `opencode` | `curl -fsSL https://opencode.ai/install \| bash` | `agentfield[harness-opencode]` | `opencode` | Provider credentials configured in OpenCode | | `grok` | Install the Grok Build CLI, then `grok login` | None | `grok` | `XAI_API_KEY` | +| `pi` | `npm install -g --ignore-scripts @earendil-works/pi-coding-agent` | None | `pi` | Provider login or API key such as `OPENROUTER_API_KEY` | +| `omp` | `curl -fsSL https://omp.sh/install \| sh` | None | `omp` | Provider login or API key such as `OPENROUTER_API_KEY` | Install every Python wrapper with: @@ -101,7 +104,8 @@ The pinned build, its download host and the opt-out are documented under The extras install Python wrappers. They do not replace the runtime preflight: AForge and Gemini are CLI-only, and Codex or OpenCode may still require a separately available executable depending on the wrapper and platform. `grok` -is available in the Python SDK only. +is available in the Python SDK only. Pi and OMP are CLI-only: install their +upstream binaries as shown below. ### AForge adapter contract @@ -135,6 +139,46 @@ used, Python reads it at import time — so export it before starting the agent rather than mutating the environment mid-run. The TypeScript OpenCode provider has no limiter and ignores the variable. +Install Pi or OMP directly from their official distributions: + +```bash +npm install -g --ignore-scripts @earendil-works/pi-coding-agent +curl -fsSL https://omp.sh/install | sh +``` + +For reproducible containers and CI, run the chosen upstream installer in the +image build, then gate startup with `af harness doctor`. If the executable is +missing at dispatch time, all three SDKs return an actionable provider error +containing the same upstream install command instead of attempting a mutation. + +## Provider parity + +Pi and OMP implement the same provider-neutral harness surface as OpenCode. +The SDK translates that surface to each CLI's native flags rather than exposing +CLI-specific command construction to application code. + +| Capability | OpenCode | Pi | OMP | +| --- | --- | --- | --- | +| Model and `#variant` | `-m`, `--variant` | `--model`, `--thinking` | `--model`, `--thinking` | +| Project root | `--dir` | process working directory | `--cwd` plus process working directory | +| One-shot machine output | JSON output | stdin + JSON event stream | stdin + JSON event stream | +| System prompt | Native prompt option | Native prompt option | Native prompt option | +| Tool allowlist | Native tool flags | Normalized Pi tool names | Normalized OMP tool names | +| Plan / auto permissions | Native permission flags | Read-only tools / no approval flag | Read-only tools / `--auto-approve` | +| Session resume | Native session option | `--session` | `--resume` | +| Structured output | Isolated schema file protocol | Same protocol | Same protocol | +| Metrics | Sessions, turns, tokens, cost, duration | Same normalized fields | Same normalized fields | +| Runtime controls | Env, timeout, retries, binary override | Same | Same | + +The contract is equivalent, not flag-identical. Pi calls its filesystem search +tool `find`, OMP calls it `glob`, and each CLI has its own resume flag. Only OMP +has an approval flag: `--tools` is Pi's documented read-only mechanism and it has +no approval flag at all, so `permission_mode="auto"` adds nothing for Pi. These +differences stay inside the provider adapters. Unsupported native concepts are +handled consistently: plan mode removes mutating tools, explicit model variants +override `#variant`, and provider-reported metrics are normalized into the shared +result type. + ## Model selection and reasoning-effort variants Every provider accepts a `model` option on `.harness()` calls. Leaving it unset @@ -152,6 +196,10 @@ result = await app.harness( An explicit `variant="high"` keyword wins over the suffix. Per provider: +Pi and OMP accept the same OpenRouter model strings in every SDK, for example +`openrouter/minimax/minimax-m2.7` or +`openrouter/google/gemini-2.5-flash#low`. + | Provider | Model flag | Variant handling | | --- | --- | --- | | `aforge` | `exec`: `--model` and `--plan-model`; `do`: `AFORGE_MODEL` (a leading `openrouter/` is stripped) | `AFORGE_EXEC_REASONING` (`off`, `low`, `medium`, or `high`) | @@ -159,6 +207,8 @@ An explicit `variant="high"` keyword wins over the suffix. Per provider: | Codex | `-m ` | `-c model_reasoning_effort=` | | Claude Code | SDK `model` option | No effort control — variant is dropped with a debug log | | Gemini | `-m ` | No effort control — variant is dropped | +| Pi | `--model ` | `--thinking ` | +| OMP | `--model ` | `--thinking ` | The `#` separator is safe in model ids: `:` belongs to OpenRouter suffixes like `:free`, and `@` to Vertex-style ids, but no provider uses `#`. @@ -168,7 +218,7 @@ The `#` separator is safe in model ids: `:` belongs to OpenRouter suffixes like Check selected providers in a container or CI job before any paid run: ```bash -af harness doctor --provider codex,opencode --json +af harness doctor --provider codex,opencode,pi,omp --json ``` The command exits non-zero if a requested provider is missing, its version @@ -178,7 +228,7 @@ CI can archive the report when the command fails. Python applications can use the same preflight data: ```python -reports = await app.harness_doctor(providers=["codex", "opencode"]) +reports = await app.harness_doctor(providers=["codex", "opencode", "pi", "omp"]) for report in reports: print(report.provider, report.usable, report.issues) ``` @@ -187,6 +237,9 @@ The preflight currently ships in the Python SDK and the `af` CLI. Equivalent TypeScript and Go SDK APIs are planned follow-ups (see #685) and are not available yet. +For a complete Go workflow that fans one task out to Pi and OMP concurrently, +see `examples/go_agent_nodes/cmd/harness_duo`. + Each report includes the provider name, resolved binary, installed state, version, auth state, usability, installation command, recognized auth variables, and machine-readable issues. diff --git a/examples/go_agent_nodes/cmd/harness_duo/README.md b/examples/go_agent_nodes/cmd/harness_duo/README.md new file mode 100644 index 000000000..c780398da --- /dev/null +++ b/examples/go_agent_nodes/cmd/harness_duo/README.md @@ -0,0 +1,42 @@ +# Pi + OMP Go workflow + +This example registers one AgentField workflow with three Go reasoners: + +```text +compare +├── pi_worker (Pi harness) +└── omp_worker (Oh My Pi harness) +``` + +`compare` starts both child reasoners concurrently and joins their structured +results. Both branches name their provider explicitly — Pi and OMP are +additional providers, while a call that omits `Provider` runs the SDK default, +`aforge`. The default model is +`openrouter/minimax/minimax-m2.7`; set +`HARNESS_MODEL=openrouter/google/gemini-2.5-flash` for the Gemini Flash path. + +## Run + +Start the AgentField control plane, make sure `OPENROUTER_API_KEY` is set, and +install the CLIs shown by `af harness doctor --provider pi` and +`af harness doctor --provider omp`. Then run: + +```bash +cd examples/go_agent_nodes +HARNESS_PROJECT_DIR="$(git rev-parse --show-toplevel)" \ + go run ./cmd/harness_duo serve +``` + +In another terminal, submit the workflow: + +```bash +curl -sS -X POST \ + http://localhost:8080/api/v1/execute/async/harness-duo-go.compare \ + -H 'content-type: application/json' \ + -d '{"input":{"task":"Read README.md and summarize the project with file-path evidence."}}' +``` + +The returned execution appears in AgentField Desktop as a fan-out from +`compare` to `pi_worker` and `omp_worker`. You can override `model`, `task`, and +`project_dir` in the JSON input. `PI_BIN` and `OMP_BIN` override CLI locations +when the binaries are not on `PATH`. diff --git a/examples/go_agent_nodes/cmd/harness_duo/main.go b/examples/go_agent_nodes/cmd/harness_duo/main.go new file mode 100644 index 000000000..33457f237 --- /dev/null +++ b/examples/go_agent_nodes/cmd/harness_duo/main.go @@ -0,0 +1,182 @@ +// Command harness_duo runs Pi and OMP concurrently inside one AgentField workflow. +package main + +import ( + "context" + "fmt" + "log" + "os" + "path/filepath" + "strings" + + "github.com/Agent-Field/agentfield/sdk/go/agent" + "github.com/Agent-Field/agentfield/sdk/go/harness" +) + +const defaultModel = "openrouter/minimax/minimax-m2.7" + +type workerOutput struct { + Summary string `json:"summary"` + Evidence []string `json:"evidence"` +} + +type branchResult struct { + Provider string `json:"provider"` + Model string `json:"model"` + Output workerOutput `json:"output"` + DurationMS int `json:"duration_ms"` + InputTokens int `json:"input_tokens"` + OutputTokens int `json:"output_tokens"` + TotalTokens int `json:"total_tokens"` + CostUSD *float64 `json:"cost_usd,omitempty"` + HarnessRunID string `json:"harness_session_id,omitempty"` +} + +type completedBranch struct { + provider string + result map[string]any + err error +} + +func main() { + nodeID := envOr("AGENT_NODE_ID", "harness-duo-go") + listenAddress := envOr("AGENT_LISTEN_ADDR", ":8017") + publicURL := envOr("AGENT_PUBLIC_URL", "http://localhost"+listenAddress) + + duo, err := agent.New(agent.Config{ + NodeID: nodeID, + Version: "1.0.0", + AgentFieldURL: envOr("AGENTFIELD_URL", "http://localhost:8080"), + Token: os.Getenv("AGENTFIELD_TOKEN"), + InternalToken: strings.TrimSpace(os.Getenv("AGENTFIELD_AUTHORIZATION_INTERNAL_TOKEN")), + ListenAddress: listenAddress, + PublicURL: publicURL, + }) + if err != nil { + log.Fatal(err) + } + + registerWorker(duo, "pi_worker", harness.ProviderPi, "PI_BIN") + // Both branches name their provider explicitly. Omitting Provider would + // select the SDK default, aforge — not OMP. + registerWorker(duo, "omp_worker", harness.ProviderOMP, "OMP_BIN") + + duo.RegisterReasoner("compare", func(ctx context.Context, input map[string]any) (any, error) { + branchInput := map[string]any{ + "task": inputString(input, "task", defaultTask()), + "model": inputString(input, "model", envOr("HARNESS_MODEL", defaultModel)), + "project_dir": inputString(input, "project_dir", projectDir()), + } + + completed := make(chan completedBranch, 2) + for _, provider := range []string{"pi", "omp"} { + provider := provider + go func() { + result, callErr := duo.Call(ctx, provider+"_worker", branchInput) + completed <- completedBranch{provider: provider, result: result, err: callErr} + }() + } + + results := make(map[string]any, 2) + for i := 0; i < 2; i++ { + branch := <-completed + if branch.err != nil { + return nil, fmt.Errorf("%s harness failed: %w", branch.provider, branch.err) + } + results[branch.provider] = branch.result + } + + return map[string]any{ + "model": branchInput["model"], + "branches": results, + }, nil + }, + agent.WithDescription("Fan out one task to Pi and OMP concurrently, then join their structured results"), + agent.WithReasonerTags("entry", "harness-demo"), + ) + + if err := duo.Run(context.Background()); err != nil { + if cliErr, ok := err.(*agent.CLIError); ok { + os.Exit(cliErr.ExitCode()) + } + log.Fatal(err) + } +} + +func registerWorker(duo *agent.Agent, reasoner, provider, binEnv string) { + providerName := provider + duo.RegisterReasoner(reasoner, func(ctx context.Context, input map[string]any) (any, error) { + model := inputString(input, "model", envOr("HARNESS_MODEL", defaultModel)) + root := inputString(input, "project_dir", projectDir()) + + var output workerOutput + schema, err := harness.StructToJSONSchema(output) + if err != nil { + return nil, fmt.Errorf("build output schema: %w", err) + } + + run, err := duo.Harness(ctx, inputString(input, "task", defaultTask()), schema, &output, harness.Options{ + Provider: provider, + Model: model, + PermissionMode: "auto", + ProjectDir: root, + BinPath: strings.TrimSpace(os.Getenv(binEnv)), + Tools: []string{"Read", "Write", "Glob", "Grep"}, + SystemPrompt: "Inspect the requested project carefully. Be concise, cite file paths as evidence, and follow the structured-output instructions exactly.", + Timeout: 300, + MaxRetries: 1, + SchemaMaxRetries: 1, + }) + if err != nil { + return nil, err + } + if run.IsError { + return nil, fmt.Errorf("%s harness: %s", providerName, run.ErrorMessage) + } + + return branchResult{ + Provider: providerName, + Model: model, + Output: output, + DurationMS: run.DurationMS, + InputTokens: run.InputTokens, + OutputTokens: run.OutputTokens, + TotalTokens: run.TotalTokens, + CostUSD: run.CostUSD, + HarnessRunID: run.SessionID, + }, nil + }, agent.WithDescription("Run the task with the "+providerName+" coding harness")) +} + +func defaultTask() string { + return "Read README.md and one directly relevant source file. Summarize what this project does in two sentences and provide both file paths as evidence. Do not modify project files." +} + +func projectDir() string { + if configured := strings.TrimSpace(os.Getenv("HARNESS_PROJECT_DIR")); configured != "" { + return configured + } + root, err := os.Getwd() + if err != nil { + return "." + } + absolute, err := filepath.Abs(root) + if err != nil { + return root + } + return absolute +} + +func inputString(input map[string]any, key, fallback string) string { + if value, ok := input[key].(string); ok && strings.TrimSpace(value) != "" { + return strings.TrimSpace(value) + } + return fallback +} + +func envOr(key, fallback string) string { + if value := strings.TrimSpace(os.Getenv(key)); value != "" { + return value + } + return fallback +} diff --git a/sdk/go/agent/harness.go b/sdk/go/agent/harness.go index f39d06970..a1899c6c8 100644 --- a/sdk/go/agent/harness.go +++ b/sdk/go/agent/harness.go @@ -11,9 +11,9 @@ import ( // HarnessConfig configures the default harness runner for the agent. type HarnessConfig struct { // Provider is the default provider: "aforge", "claude-code", "codex", - // "gemini", or "opencode". When empty, AGENTFIELD_HARNESS_PROVIDER - // overrides the default, "aforge" (AgentField's native harness). An - // explicit value always wins. + // "gemini", "opencode", "pi", or "omp". When empty, + // AGENTFIELD_HARNESS_PROVIDER overrides the default, "aforge" + // (AgentField's native harness). An explicit value always wins. Provider string // Model is the default model identifier. Empty means the provider's own diff --git a/sdk/go/agent/harness_test.go b/sdk/go/agent/harness_test.go index 98bdb269d..52b73db39 100644 --- a/sdk/go/agent/harness_test.go +++ b/sdk/go/agent/harness_test.go @@ -40,7 +40,8 @@ func TestHarnessRunner_LazyInitialization(t *testing.T) { } func TestHarnessRunner_DefaultOptions(t *testing.T) { - // Agent with no HarnessConfig — runner gets zero-value Options + // Agent with no HarnessConfig — runner gets zero-value Options; the + // provider is resolved at dispatch time (explicit > env > "aforge"). a := newTestAgentForHarness(t) runner := a.HarnessRunner() diff --git a/sdk/go/harness/factory.go b/sdk/go/harness/factory.go index 9b8a0d3d0..e24e3ead6 100644 --- a/sdk/go/harness/factory.go +++ b/sdk/go/harness/factory.go @@ -20,7 +20,8 @@ func ResolveProviderName(name string) string { } // BuildProvider creates a Provider instance for the given provider name. -// Supported providers: "aforge", "claude-code", "codex", "gemini", "opencode". +// Supported providers: "aforge", "claude-code", "codex", "gemini", +// "opencode", "pi", "omp". func BuildProvider(name string, binPath string) (Provider, error) { name = ResolveProviderName(name) switch name { @@ -34,10 +35,14 @@ func BuildProvider(name string, binPath string) (Provider, error) { return NewGeminiProvider(binPath), nil case ProviderOpenCode: return NewOpenCodeProvider(binPath, ""), nil + case ProviderPi: + return NewPiProvider(binPath), nil + case ProviderOMP: + return NewOMPProvider(binPath), nil default: return nil, fmt.Errorf( - "unknown harness provider: %q (supported: %s, %s, %s, %s, %s)", - name, ProviderAforge, ProviderClaudeCode, ProviderCodex, ProviderGemini, ProviderOpenCode, + "unknown harness provider: %q (supported: %s, %s, %s, %s, %s, %s, %s)", + name, ProviderAforge, ProviderClaudeCode, ProviderCodex, ProviderGemini, ProviderOpenCode, ProviderPi, ProviderOMP, ) } } diff --git a/sdk/go/harness/factory_test.go b/sdk/go/harness/factory_test.go index be480ed4b..77036beb7 100644 --- a/sdk/go/harness/factory_test.go +++ b/sdk/go/harness/factory_test.go @@ -44,5 +44,5 @@ func TestBuildProvider_RejectsUnknownName(t *testing.T) { assert.Nil(t, provider) require.Error(t, err) assert.Contains(t, err.Error(), "nope") - assert.True(t, strings.Contains(err.Error(), "supported: aforge, claude-code, codex, gemini, opencode")) + assert.True(t, strings.Contains(err.Error(), "supported: aforge, claude-code, codex, gemini, opencode, pi, omp")) } diff --git a/sdk/go/harness/pi.go b/sdk/go/harness/pi.go new file mode 100644 index 000000000..3df4992a5 --- /dev/null +++ b/sdk/go/harness/pi.go @@ -0,0 +1,327 @@ +package harness + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "time" +) + +type piFlavor string + +const ( + piFlavorPi piFlavor = "pi" + piFlavorOMP piFlavor = "omp" +) + +var piReadOnlyTools = map[string]bool{ + "read": true, "grep": true, "find": true, "glob": true, "ls": true, "lsp": true, +} + +type piFamilyProvider struct { + BinPath string + flavor piFlavor + runCLI func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) +} + +// PiProvider invokes the Pi coding-agent CLI as a subprocess. +type PiProvider struct{ *piFamilyProvider } + +// OMPProvider invokes the Oh My Pi (OMP) coding-agent CLI as a subprocess. +type OMPProvider struct{ *piFamilyProvider } + +// NewPiProvider creates a Pi provider. An empty binPath defaults to "pi". +func NewPiProvider(binPath string) *PiProvider { + if binPath == "" { + binPath = "pi" + } + return &PiProvider{&piFamilyProvider{ + BinPath: binPath, + flavor: piFlavorPi, + runCLI: RunCLIWithStdin, + }} +} + +// NewOMPProvider creates an OMP provider. An empty binPath defaults to "omp". +func NewOMPProvider(binPath string) *OMPProvider { + if binPath == "" { + binPath = "omp" + } + return &OMPProvider{&piFamilyProvider{ + BinPath: binPath, + flavor: piFlavorOMP, + runCLI: RunCLIWithStdin, + }} +} + +func (p *PiProvider) Execute(ctx context.Context, prompt string, options Options) (*RawResult, error) { + return p.piFamilyProvider.execute(ctx, prompt, options) +} + +func (p *OMPProvider) Execute(ctx context.Context, prompt string, options Options) (*RawResult, error) { + return p.piFamilyProvider.execute(ctx, prompt, options) +} + +func (p *piFamilyProvider) execute(ctx context.Context, prompt string, options Options) (*RawResult, error) { + cmd := []string{p.BinPath, "--print", "--mode", "json"} + + root := options.ProjectDir + if root == "" { + root = options.Cwd + } + if p.flavor == piFlavorOMP && root != "" { + cmd = append(cmd, "--cwd", root) + } + + model, variant := options.resolveModelAndVariant() + if model != "" { + cmd = append(cmd, "--model", model) + } + if variant != "" { + cmd = append(cmd, "--thinking", variant) + } + if strings.TrimSpace(options.SystemPrompt) != "" { + cmd = append(cmd, "--system-prompt", strings.TrimSpace(options.SystemPrompt)) + } + if options.ResumeSessionID != "" { + resumeFlag := "--session" + if p.flavor == piFlavorOMP { + resumeFlag = "--resume" + } + cmd = append(cmd, resumeFlag, options.ResumeSessionID) + } + // --tools is the enforced, vendor-documented read-only allowlist. Pi has no + // approval flag (unknown options fail); OMP auto-approves read-only tiers even + // under always-ask. + if options.PermissionMode == "auto" { + if p.flavor == piFlavorOMP { + cmd = append(cmd, "--auto-approve") + } + } + + tools := normalizePiTools(options.Tools, p.flavor) + if options.PermissionMode == "plan" { + readOnly := make([]string, 0, len(tools)) + for _, tool := range tools { + if piReadOnlyTools[tool] { + readOnly = append(readOnly, tool) + } + } + tools = readOnly + if len(tools) == 0 { + globTool := "find" + if p.flavor == piFlavorOMP { + globTool = "glob" + } + tools = []string{"read", "grep", globTool} + } + } + if options.Tools != nil || options.PermissionMode == "plan" { + if len(tools) == 0 { + cmd = append(cmd, "--no-tools") + } else { + cmd = append(cmd, "--tools", strings.Join(tools, ",")) + } + } + + env := make(map[string]string, len(options.Env)) + for key, value := range options.Env { + env[key] = value + } + + start := time.Now() + runCLI := p.runCLI + if runCLI == nil { + runCLI = RunCLIWithStdin + } + cliResult, err := runCLI(ctx, cmd, env, root, options.timeout(), []byte(prompt)) + apiMS := int(time.Since(start).Milliseconds()) + if err != nil { + if isExecNotFound(err) { + install := "npm install -g --ignore-scripts @earendil-works/pi-coding-agent" + if p.flavor == piFlavorOMP { + install = "curl -fsSL https://omp.sh/install | sh" + } + return &RawResult{ + IsError: true, + ErrorMessage: fmt.Sprintf("%s binary not found at '%s'. Install: %s", strings.ToUpper(string(p.flavor)), p.BinPath, install), + FailureType: FailureCrash, + }, nil + } + if strings.Contains(err.Error(), "timed out") || strings.Contains(err.Error(), "no progress") { + return &RawResult{ + IsError: true, + ErrorMessage: err.Error(), + FailureType: FailureTimeout, + Metrics: Metrics{DurationAPIMS: apiMS}, + }, nil + } + return nil, err + } + + raw := parsePiJSONL(cliResult.Stdout) + if model != "" { + raw.Metrics.Model = model + } + raw.Metrics.DurationAPIMS = apiMS + raw.ReturnCode = cliResult.ReturnCode + stderr := StripANSI(strings.TrimSpace(cliResult.Stderr)) + if cliResult.ReturnCode < 0 { + raw.IsError = true + raw.FailureType = FailureCrash + raw.ErrorMessage = fmt.Sprintf("Process killed by signal %d.", -cliResult.ReturnCode) + } else if cliResult.ReturnCode != 0 { + raw.IsError = true + raw.FailureType = FailureCrash + if stderr != "" { + raw.ErrorMessage = truncate(stderr, 1000) + } else if raw.ErrorMessage == "" { + raw.ErrorMessage = fmt.Sprintf("Process exited with code %d.", cliResult.ReturnCode) + } + } else if raw.ErrorMessage != "" { + raw.IsError = true + raw.FailureType = FailureAPIError + } else if raw.Result == "" { + raw.IsError = true + raw.FailureType = FailureNoOutput + raw.ErrorMessage = stderr + if raw.ErrorMessage == "" { + raw.ErrorMessage = fmt.Sprintf("%s exited successfully without an assistant response.", p.flavor) + } + } + return raw, nil +} + +func normalizePiTools(tools []string, flavor piFlavor) []string { + normalized := make([]string, 0, len(tools)) + seen := make(map[string]bool, len(tools)) + for _, tool := range tools { + name := strings.ToLower(strings.TrimSpace(tool)) + if name == "glob" && flavor == piFlavorPi { + name = "find" + } + if name != "" && !seen[name] { + seen[name] = true + normalized = append(normalized, name) + } + } + return normalized +} + +func parsePiJSONL(stdout string) *RawResult { + raw := &RawResult{FailureType: FailureNone} + var totalCost float64 + hasCost := false + + for _, line := range strings.Split(stdout, "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + var event map[string]any + if err := json.Unmarshal([]byte(line), &event); err != nil { + continue + } + raw.Messages = append(raw.Messages, event) + + eventType, _ := event["type"].(string) + if eventType == "session" { + if id, ok := event["id"].(string); ok { + raw.Metrics.SessionID = id + } + } + if eventType == "turn_end" { + raw.Metrics.NumTurns++ + } + if eventType != "message_end" { + continue + } + message, ok := event["message"].(map[string]any) + if !ok || message["role"] != "assistant" { + continue + } + if model, ok := message["model"].(string); ok { + raw.Metrics.Model = model + } + if text := piMessageText(message); text != "" { + raw.Result = text + } + if usage, ok := message["usage"].(map[string]any); ok { + raw.Metrics.InputTokens += intValue(usage["input"]) + raw.Metrics.OutputTokens += intValue(usage["output"]) + raw.Metrics.CacheReadTokens += intValue(usage["cacheRead"]) + raw.Metrics.CacheCreationTokens += intValue(usage["cacheWrite"]) + if cost, ok := usage["cost"].(map[string]any); ok { + if value, ok := floatValue(cost["total"]); ok { + totalCost += value + hasCost = true + } + } + } + if reason, _ := message["stopReason"].(string); reason == "error" || reason == "aborted" { + raw.ErrorMessage = fmt.Sprintf("Pi stopped with reason %q.", reason) + if detail, ok := message["errorMessage"].(string); ok && detail != "" { + raw.ErrorMessage = detail + } + } else { + // Only the final message_end's stop reason decides: a turn that + // errored and then recovered must not be reported as a failure. + raw.ErrorMessage = "" + } + } + + if raw.Metrics.NumTurns == 0 && raw.Result != "" { + raw.Metrics.NumTurns = 1 + } + if hasCost { + raw.Metrics.CostUSD = &totalCost + } + return raw +} + +func piMessageText(message map[string]any) string { + if content, ok := message["content"].(string); ok { + return content + } + content, ok := message["content"].([]any) + if !ok { + return "" + } + var text strings.Builder + for _, item := range content { + part, ok := item.(map[string]any) + if !ok || part["type"] != "text" { + continue + } + if value, ok := part["text"].(string); ok { + text.WriteString(value) + } + } + return text.String() +} + +func intValue(value any) int { + if number, ok := floatValue(value); ok { + return int(number) + } + return 0 +} + +func floatValue(value any) (float64, bool) { + switch number := value.(type) { + case float64: + return number, true + case float32: + return float64(number), true + case int: + return float64(number), true + case int64: + return float64(number), true + case json.Number: + parsed, err := number.Float64() + return parsed, err == nil + default: + return 0, false + } +} diff --git a/sdk/go/harness/pi_test.go b/sdk/go/harness/pi_test.go new file mode 100644 index 000000000..97b02af27 --- /dev/null +++ b/sdk/go/harness/pi_test.go @@ -0,0 +1,407 @@ +package harness + +import ( + "context" + "errors" + "fmt" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const piEventStream = `{"type":"session","id":"session-123"} +{"type":"turn_start"} +{"type":"message_end","message":{"role":"assistant","content":[{"type":"thinking","thinking":"internal"},{"type":"text","text":"done"}],"model":"google/gemini-2.5-flash","usage":{"input":120,"output":30,"cacheRead":10,"cacheWrite":4,"cost":{"total":0.0025}},"stopReason":"stop"}} +{"type":"turn_end"} +{"type":"agent_end"}` + +func TestPiFamilyCommandAndMetrics(t *testing.T) { + tests := []struct { + name string + newProvider func() (Provider, *piFamilyProvider) + permissionFlag string + globTool string + wantPrefix []string + }{ + { + name: "pi", + newProvider: func() (Provider, *piFamilyProvider) { + provider := NewPiProvider("/opt/pi") + return provider, provider.piFamilyProvider + }, + permissionFlag: "", + globTool: "find", + wantPrefix: []string{"/opt/pi", "--print", "--mode", "json"}, + }, + { + name: "omp", + newProvider: func() (Provider, *piFamilyProvider) { + provider := NewOMPProvider("/opt/omp") + return provider, provider.piFamilyProvider + }, + permissionFlag: "--auto-approve", + globTool: "glob", + wantPrefix: []string{"/opt/omp", "--print", "--mode", "json", "--cwd", "/tmp/project"}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + provider, base := tc.newProvider() + + var gotCmd []string + var gotEnv map[string]string + var gotCwd string + var gotPrompt string + base.runCLI = func(_ context.Context, cmd []string, env map[string]string, cwd string, _ int, stdin []byte) (*CLIResult, error) { + gotCmd = append([]string(nil), cmd...) + gotEnv = env + gotCwd = cwd + gotPrompt = string(stdin) + return &CLIResult{Stdout: piEventStream}, nil + } + + raw, err := provider.Execute(context.Background(), "implement this", Options{ + ProjectDir: "/tmp/project", + Model: "openrouter/google/gemini-2.5-flash#high", + PermissionMode: "auto", + SystemPrompt: "Be precise.", + Tools: []string{"Read", "Write", "Edit", "Bash", "Glob", "Grep"}, + Env: map[string]string{"EXTRA": "1"}, + }) + require.NoError(t, err) + require.GreaterOrEqual(t, len(gotCmd), len(tc.wantPrefix)) + assert.Equal(t, tc.wantPrefix, gotCmd[:len(tc.wantPrefix)]) + if tc.permissionFlag != "" { + assert.Contains(t, gotCmd, tc.permissionFlag) + } + assertNoApprovalFlags(t, gotCmd, tc.permissionFlag) + assertFlagValue(t, gotCmd, "--model", "openrouter/google/gemini-2.5-flash") + assertFlagValue(t, gotCmd, "--thinking", "high") + assertFlagValue(t, gotCmd, "--tools", "read,write,edit,bash,"+tc.globTool+",grep") + assert.Equal(t, map[string]string{"EXTRA": "1"}, gotEnv) + assert.Equal(t, "/tmp/project", gotCwd) + assert.Equal(t, "implement this", gotPrompt) + + assert.False(t, raw.IsError) + assert.Equal(t, "done", raw.Result) + assert.Equal(t, "session-123", raw.Metrics.SessionID) + assert.Equal(t, 1, raw.Metrics.NumTurns) + assert.Equal(t, 120, raw.Metrics.InputTokens) + assert.Equal(t, 30, raw.Metrics.OutputTokens) + assert.Equal(t, 10, raw.Metrics.CacheReadTokens) + assert.Equal(t, 4, raw.Metrics.CacheCreationTokens) + require.NotNil(t, raw.Metrics.CostUSD) + assert.InDelta(t, 0.0025, *raw.Metrics.CostUSD, 0.000001) + }) + } +} + +func TestPiConfiguredModelOverridesReportedModel(t *testing.T) { + provider := NewPiProvider("pi") + provider.runCLI = func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) { + return &CLIResult{Stdout: piEventStream}, nil + } + + raw, err := provider.Execute(context.Background(), "inspect", Options{Model: "openrouter/x/y"}) + require.NoError(t, err) + assert.Equal(t, "openrouter/x/y", raw.Metrics.Model) +} + +func TestPiUsesReportedModelWithoutConfiguredModel(t *testing.T) { + provider := NewPiProvider("pi") + provider.runCLI = func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) { + return &CLIResult{Stdout: piEventStream}, nil + } + + raw, err := provider.Execute(context.Background(), "inspect", Options{}) + require.NoError(t, err) + assert.Equal(t, "google/gemini-2.5-flash", raw.Metrics.Model) +} + +func TestPiModelIsEmptyWhenNotConfiguredOrReported(t *testing.T) { + provider := NewPiProvider("pi") + provider.runCLI = func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) { + return &CLIResult{Stdout: `{"type":"message_end","message":{"role":"assistant","content":"done"}}`}, nil + } + + raw, err := provider.Execute(context.Background(), "inspect", Options{}) + require.NoError(t, err) + assert.Empty(t, raw.Metrics.Model) +} + +func TestPiFamilyPlanModeIsReadOnlyAndResumes(t *testing.T) { + tests := []struct { + name string + newProvider func() (Provider, *piFamilyProvider) + resumeFlag string + tools string + }{ + {"pi", func() (Provider, *piFamilyProvider) { + provider := NewPiProvider("pi") + return provider, provider.piFamilyProvider + }, "--session", "read,grep,find"}, + {"omp", func() (Provider, *piFamilyProvider) { + provider := NewOMPProvider("omp") + return provider, provider.piFamilyProvider + }, "--resume", "read,grep,glob"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + provider, base := tc.newProvider() + var gotCmd []string + base.runCLI = func(_ context.Context, cmd []string, _ map[string]string, _ string, _ int, _ []byte) (*CLIResult, error) { + gotCmd = append([]string(nil), cmd...) + return &CLIResult{Stdout: piEventStream}, nil + } + _, err := provider.Execute(context.Background(), "plan", Options{ + PermissionMode: "plan", + Tools: []string{"Read", "Write", "Bash", "Grep", "Glob"}, + ResumeSessionID: "abc123", + }) + require.NoError(t, err) + assertFlagValue(t, gotCmd, "--tools", tc.tools) + assertFlagValue(t, gotCmd, tc.resumeFlag, "abc123") + assertNoApprovalFlags(t, gotCmd, "") + }) + } +} + +func assertNoApprovalFlags(t *testing.T, cmd []string, allowed string) { + t.Helper() + for _, flag := range []string{"--approve", "--auto-approve", "--yolo", "-y", "--approval-mode", "--permission-mode"} { + if flag != allowed { + assert.NotContains(t, cmd, flag) + } + } +} + +func TestPiFamilyToolEdgeCases(t *testing.T) { + tests := []struct { + name string + provider *piFamilyProvider + options Options + wantFlag string + wantNoTools bool + }{ + { + name: "pi plan restores default read-only tools", + provider: NewPiProvider("pi").piFamilyProvider, + options: Options{PermissionMode: "plan", Tools: []string{"Write", "Bash"}}, + wantFlag: "read,grep,find", + }, + { + name: "omp plan restores default read-only tools", + provider: NewOMPProvider("omp").piFamilyProvider, + options: Options{PermissionMode: "plan", Tools: []string{"Write", "Bash"}}, + wantFlag: "read,grep,glob", + }, + { + name: "explicit empty tools disables tools", + provider: NewPiProvider("pi").piFamilyProvider, + options: Options{Tools: []string{}}, + wantNoTools: true, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + var gotCmd []string + tc.provider.runCLI = func(_ context.Context, cmd []string, _ map[string]string, _ string, _ int, _ []byte) (*CLIResult, error) { + gotCmd = append([]string(nil), cmd...) + return &CLIResult{Stdout: piEventStream}, nil + } + + _, err := tc.provider.execute(context.Background(), "inspect", tc.options) + require.NoError(t, err) + if tc.wantNoTools { + assert.Contains(t, gotCmd, "--no-tools") + assert.NotContains(t, gotCmd, "--tools") + return + } + assertFlagValue(t, gotCmd, "--tools", tc.wantFlag) + }) + } +} + +func TestPiFamilyExecutionFailures(t *testing.T) { + tests := []struct { + name string + result *CLIResult + runErr error + wantErr bool + failureType FailureType + message string + }{ + { + name: "timeout", + runErr: errors.New("command timed out"), + failureType: FailureTimeout, + message: "command timed out", + }, + { + name: "unexpected runner error", + runErr: errors.New("pipe failed"), + wantErr: true, + }, + { + name: "nonzero exit without stderr", + result: &CLIResult{ReturnCode: 7}, + failureType: FailureCrash, + message: "Process exited with code 7.", + }, + { + name: "signal kill reports the signal, not an exit code", + result: &CLIResult{ReturnCode: -9}, + failureType: FailureCrash, + message: "Process killed by signal 9.", + }, + { + name: "api error event", + result: &CLIResult{Stdout: `{"type":"message_end","message":{"role":"assistant","stopReason":"error","errorMessage":"quota exceeded"}}`}, + failureType: FailureAPIError, + message: "quota exceeded", + }, + { + name: "successful process without assistant output", + result: &CLIResult{Stderr: "empty response"}, + failureType: FailureNoOutput, + message: "empty response", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + provider := NewPiProvider("pi").piFamilyProvider + provider.runCLI = func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) { + return tc.result, tc.runErr + } + + raw, err := provider.execute(context.Background(), "inspect", Options{}) + if tc.wantErr { + require.EqualError(t, err, tc.runErr.Error()) + assert.Nil(t, raw) + return + } + require.NoError(t, err) + require.NotNil(t, raw) + assert.True(t, raw.IsError) + assert.Equal(t, tc.failureType, raw.FailureType) + assert.Equal(t, tc.message, raw.ErrorMessage) + }) + } +} + +func TestPiFamilyRecoveredTurnIsNotAnError(t *testing.T) { + tests := []struct { + name string + stdout string + wantResult string + wantIsError bool + wantFailureType FailureType + wantError string + }{ + { + name: "error then recovery", + stdout: `{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"partial"}],"stopReason":"error","errorMessage":"upstream 503"}} +{"type":"turn_end"} +{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"FINAL ANSWER"}],"stopReason":"stop"}} +{"type":"turn_end"}`, + wantResult: "FINAL ANSWER", + wantFailureType: FailureNone, + }, + { + name: "last message is an error", + stdout: `{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"partial"}],"stopReason":"stop"}} +{"type":"turn_end"} +{"type":"message_end","message":{"role":"assistant","content":[{"type":"text","text":"FINAL ANSWER"}],"stopReason":"error","errorMessage":"upstream 503"}} +{"type":"turn_end"}`, + wantResult: "FINAL ANSWER", + wantIsError: true, + wantFailureType: FailureAPIError, + wantError: "upstream 503", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + provider := NewPiProvider("pi").piFamilyProvider + provider.runCLI = func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) { + return &CLIResult{Stdout: tc.stdout, ReturnCode: 0}, nil + } + + raw, err := provider.execute(context.Background(), "inspect", Options{}) + require.NoError(t, err) + require.NotNil(t, raw) + assert.Equal(t, tc.wantResult, raw.Result) + assert.Equal(t, tc.wantIsError, raw.IsError) + assert.Equal(t, tc.wantFailureType, raw.FailureType) + assert.Equal(t, tc.wantError, raw.ErrorMessage) + }) + } +} + +func TestPiFamilyNonzeroExitIsError(t *testing.T) { + p := NewPiProvider("pi") + p.runCLI = func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) { + return &CLIResult{Stderr: "authentication failed", ReturnCode: 2}, nil + } + raw, err := p.Execute(context.Background(), "hello", Options{}) + require.NoError(t, err) + assert.True(t, raw.IsError) + assert.Equal(t, FailureCrash, raw.FailureType) + assert.Equal(t, "authentication failed", raw.ErrorMessage) +} + +func TestBuildProviderPiFamily(t *testing.T) { + pi, err := BuildProvider(ProviderPi, "/opt/pi") + require.NoError(t, err) + omp, err := BuildProvider(ProviderOMP, "/opt/omp") + require.NoError(t, err) + assert.Equal(t, "*harness.PiProvider", fmt.Sprintf("%T", pi)) + assert.Equal(t, "*harness.OMPProvider", fmt.Sprintf("%T", omp)) + + // Pi and OMP are additional providers: an empty name still resolves to the + // SDK default, aforge. + t.Setenv(ProviderEnvVar, "") + defaultProvider, err := BuildProvider("", "") + require.NoError(t, err) + assert.Equal(t, "*harness.AforgeProvider", fmt.Sprintf("%T", defaultProvider)) +} + +func TestPiFamilyMissingBinaryIncludesInstallGuidance(t *testing.T) { + tests := []struct { + name string + provider *piFamilyProvider + installHint string + }{ + {"pi", NewPiProvider("pi").piFamilyProvider, "npm install -g --ignore-scripts @earendil-works/pi-coding-agent"}, + {"omp", NewOMPProvider("omp").piFamilyProvider, "curl -fsSL https://omp.sh/install | sh"}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + tc.provider.runCLI = func(context.Context, []string, map[string]string, string, int, []byte) (*CLIResult, error) { + return nil, fmt.Errorf("exec: executable file not found in $PATH") + } + raw, err := tc.provider.execute(context.Background(), "hello", Options{}) + require.NoError(t, err) + require.True(t, raw.IsError) + assert.Contains(t, raw.ErrorMessage, tc.installHint) + assert.Equal(t, FailureCrash, raw.FailureType) + }) + } +} + +func assertFlagValue(t *testing.T, cmd []string, flag, value string) { + t.Helper() + for i := range cmd { + if cmd[i] == flag && i+1 < len(cmd) { + assert.Equal(t, value, cmd[i+1]) + return + } + } + t.Fatalf("flag %q not found in %s", flag, strings.Join(cmd, " ")) +} diff --git a/sdk/go/harness/provider.go b/sdk/go/harness/provider.go index f986f2240..d09fa4a85 100644 --- a/sdk/go/harness/provider.go +++ b/sdk/go/harness/provider.go @@ -13,6 +13,10 @@ const ( ProviderCodex = "codex" // ProviderGemini is the provider name for Gemini CLI. ProviderGemini = "gemini" + // ProviderPi is the provider name for the Pi coding-agent CLI. + ProviderPi = "pi" + // ProviderOMP is the provider name for the Oh My Pi coding-agent CLI. + ProviderOMP = "omp" ) const ( @@ -36,9 +40,10 @@ type Provider interface { // Options control a single harness invocation. Fields are optional; // zero values mean "use default". type Options struct { - // Provider name: "aforge", "opencode", "claude-code", "codex", or - // "gemini". An explicit value wins over AGENTFIELD_HARNESS_PROVIDER; - // when both are empty, the provider defaults to "aforge". + // Provider name: "aforge", "opencode", "claude-code", "codex", + // "gemini", "pi", or "omp". An explicit value wins over + // AGENTFIELD_HARNESS_PROVIDER; when both are empty, the provider + // defaults to "aforge". Provider string // Model identifier passed to the coding agent. It may carry a diff --git a/sdk/go/harness/runner.go b/sdk/go/harness/runner.go index 9c32d0ab8..0b11d56aa 100644 --- a/sdk/go/harness/runner.go +++ b/sdk/go/harness/runner.go @@ -26,6 +26,9 @@ type Runner struct { // NewRunner creates a harness runner with default options. func NewRunner(defaults Options) *Runner { + // The provider is deliberately NOT resolved here: Run() applies the + // explicit > AGENTFIELD_HARNESS_PROVIDER > DefaultProvider chain at + // dispatch time, so an env change is picked up by an existing runner. return &Runner{ DefaultOptions: defaults, Logger: log.New(io.Discard, "[harness] ", log.LstdFlags), diff --git a/sdk/go/harness/runner_invariant_test.go b/sdk/go/harness/runner_invariant_test.go index ca138c1d7..e6f623ff3 100644 --- a/sdk/go/harness/runner_invariant_test.go +++ b/sdk/go/harness/runner_invariant_test.go @@ -183,6 +183,8 @@ func TestInvariant_Runner_ProviderFactoryExhaustiveness(t *testing.T) { ProviderCodex, ProviderGemini, ProviderOpenCode, + ProviderPi, + ProviderOMP, } for _, name := range knownProviders { @@ -194,6 +196,22 @@ func TestInvariant_Runner_ProviderFactoryExhaustiveness(t *testing.T) { } } +// TestInvariant_Runner_ProviderFactoryDefaultSelection verifies that an +// empty provider name selects DefaultProvider instead of erroring. +func TestInvariant_Runner_ProviderFactoryDefaultSelection(t *testing.T) { + // An ambient AGENTFIELD_HARNESS_PROVIDER would legitimately win over the + // built-in default; this invariant is about the no-configuration case. + t.Setenv(ProviderEnvVar, "") + prov, err := BuildProvider("", "") + require.NoError(t, err, "BuildProvider must not error for empty provider (selects default)") + require.NotNil(t, prov, "BuildProvider must return non-nil for empty provider (selects default)") + + defaultProv, err := BuildProvider(DefaultProvider, "") + require.NoError(t, err, "BuildProvider must not error for DefaultProvider %q", DefaultProvider) + assert.IsType(t, defaultProv, prov, + "empty provider must select the same type as DefaultProvider %q", DefaultProvider) +} + // TestInvariant_Runner_ProviderFactoryUnknownReturnsError verifies that // unknown provider names return an error with a non-nil error value. func TestInvariant_Runner_ProviderFactoryUnknownReturnsError(t *testing.T) { @@ -345,6 +363,20 @@ func TestInvariant_Runner_BuildProviderWithBinPath(t *testing.T) { assert.Equal(t, customBinPath, p.BinPath) }, }, + { + provider: ProviderPi, + checkPath: func(t *testing.T, prov Provider) { + p := prov.(*PiProvider) + assert.Equal(t, customBinPath, p.BinPath) + }, + }, + { + provider: ProviderOMP, + checkPath: func(t *testing.T, prov Provider) { + p := prov.(*OMPProvider) + assert.Equal(t, customBinPath, p.BinPath) + }, + }, } for _, tt := range tests { diff --git a/sdk/go/harness/runner_test.go b/sdk/go/harness/runner_test.go index 8cf1f1e1f..7991ff574 100644 --- a/sdk/go/harness/runner_test.go +++ b/sdk/go/harness/runner_test.go @@ -603,6 +603,8 @@ func TestBuildProvider(t *testing.T) { {"codex", "codex", "*harness.CodexProvider", false}, {"gemini", "gemini", "*harness.GeminiProvider", false}, {"opencode", "opencode", "*harness.OpenCodeProvider", false}, + {"pi", "pi", "*harness.PiProvider", false}, + {"omp", "omp", "*harness.OMPProvider", false}, {"unknown", "unknown-agent", "", true}, } @@ -622,15 +624,12 @@ func TestBuildProvider(t *testing.T) { func TestRunner_BuildProvider_UsesFactory(t *testing.T) { // Verify the runner can build every registered provider. - for _, name := range []string{"aforge", "claude-code", "codex", "gemini", "opencode"} { + for _, name := range []string{"aforge", "claude-code", "codex", "gemini", "opencode", "pi", "omp"} { t.Run(name, func(t *testing.T) { runner := NewRunner(Options{Provider: name}) - _, err := runner.Run(context.Background(), "test", nil, nil, Options{}) - // Should fail at execution (binary not found), not at provider creation - // The error should NOT be "unknown harness provider" - if err != nil { - assert.NotContains(t, err.Error(), "unknown harness provider") - } + provider, err := runner.buildProvider(runner.DefaultOptions) + require.NoError(t, err) + assert.NotNil(t, provider) }) } } diff --git a/sdk/python/agentfield/agent.py b/sdk/python/agentfield/agent.py index 8c727a024..6ae6be962 100644 --- a/sdk/python/agentfield/agent.py +++ b/sdk/python/agentfield/agent.py @@ -2341,7 +2341,7 @@ async def tracked_func(*args, **kwargs): vc_setting = self._effective_component_vc_setting( reasoner_id, self._reasoner_vc_overrides ) - + self._reasoner_registry[reasoner_id] = ReasonerEntry( id=reasoner_id, func=func, @@ -2386,23 +2386,23 @@ def _detect_and_unwrap_trigger_envelope(self, payload_dict: Dict[str, Any]) -> t # Check if this looks like a dispatcher envelope if not isinstance(payload_dict, dict): return payload_dict, None - + if "event" in payload_dict and "_meta" in payload_dict: # This is a dispatcher envelope event_data = payload_dict.get("event", {}) meta_data = payload_dict.get("_meta", {}) - + # Parse metadata into TriggerContext try: from datetime import datetime from .triggers import TriggerContext - + received_at_str = meta_data.get("received_at", "") if received_at_str: received_at = datetime.fromisoformat(received_at_str.replace('Z', '+00:00')) else: received_at = datetime.utcnow() - + trigger_ctx = TriggerContext( trigger_id=meta_data.get("trigger_id", ""), source=meta_data.get("source", ""), @@ -2416,7 +2416,7 @@ def _detect_and_unwrap_trigger_envelope(self, payload_dict: Dict[str, Any]) -> t except Exception: # If parsing fails, return raw envelope for compatibility return payload_dict, None - + # Not an envelope return payload_dict, None @@ -2424,7 +2424,7 @@ def _apply_trigger_transform(self, trigger_ctx, bindings: list, input_data: dict """ Match trigger context against reasoner bindings and apply transform if found. Returns transformed input or original input if no match. - + Matching logic: 1. Find bindings where binding.source == trigger_ctx.source 2. Check event_type: binding.types empty OR trigger_ctx.event_type matches (exact or prefix) @@ -2432,21 +2432,21 @@ def _apply_trigger_transform(self, trigger_ctx, bindings: list, input_data: dict 4. Apply transform if binding has one """ from .triggers import EventTrigger - + if not bindings or not trigger_ctx: return input_data - + # Find best-matching binding best_match = None best_specificity = -1 # -1 = no match, 0 = broad (empty types), 1+ = specific - + for binding in bindings: if not isinstance(binding, EventTrigger): continue - + if binding.source != trigger_ctx.source: continue - + # Check event_type match if binding.types: # binding has specific types — check for match @@ -2461,12 +2461,12 @@ def _apply_trigger_transform(self, trigger_ctx, bindings: list, input_data: dict else: # binding accepts all types specificity = 0 - + # This binding matches; is it better than current best? if specificity > best_specificity: best_match = binding best_specificity = specificity - + # Apply transform if found if best_match and best_match.transform: try: @@ -2475,7 +2475,7 @@ def _apply_trigger_transform(self, trigger_ctx, bindings: list, input_data: dict if self.dev_mode: log_warn(f"Transform failed for {trigger_ctx.source}/{trigger_ctx.event_type}: {e}; using raw input") return input_data - + return input_data async def _execute_reasoner_endpoint( @@ -3828,8 +3828,8 @@ async def harness( prompt: Task description for the coding agent. schema: Pydantic BaseModel class for structured output validation. provider: Override provider ("aforge", "claude-code", "codex", "gemini", - "opencode", "grok"). Omit to use ``AGENTFIELD_HARNESS_PROVIDER`` - when set, otherwise ``aforge``. + "opencode", "grok", "pi", "omp"). Omit to use + ``AGENTFIELD_HARNESS_PROVIDER`` when set, otherwise ``aforge``. model: Override model identifier. Empty uses the provider's own default. max_turns: Maximum agent iterations. max_budget_usd: Cost cap in USD. diff --git a/sdk/python/agentfield/harness/_availability.py b/sdk/python/agentfield/harness/_availability.py index 4f66ab68c..006fa6abf 100644 --- a/sdk/python/agentfield/harness/_availability.py +++ b/sdk/python/agentfield/harness/_availability.py @@ -42,6 +42,32 @@ class ProviderSpec: install_command="curl -fsSL https://opencode.ai/install | bash", auth_env_vars=(), ), + "pi": ProviderSpec( + binary="pi", + version_args=("--version",), + install_command=( + "npm install -g --ignore-scripts @earendil-works/pi-coding-agent" + ), + auth_env_vars=( + "OPENROUTER_API_KEY", + "ANTHROPIC_API_KEY", + "OPENAI_API_KEY", + "GEMINI_API_KEY", + "GOOGLE_API_KEY", + ), + ), + "omp": ProviderSpec( + binary="omp", + version_args=("--version",), + install_command="curl -fsSL https://omp.sh/install | sh", + auth_env_vars=( + "OPENROUTER_API_KEY", + "ANTHROPIC_API_KEY", + "OPENAI_API_KEY", + "GEMINI_API_KEY", + "GOOGLE_API_KEY", + ), + ), "grok": ProviderSpec( binary="grok", version_args=("--version",), diff --git a/sdk/python/agentfield/harness/_runner.py b/sdk/python/agentfield/harness/_runner.py index 216a58b22..3f586bb2b 100644 --- a/sdk/python/agentfield/harness/_runner.py +++ b/sdk/python/agentfield/harness/_runner.py @@ -176,6 +176,8 @@ def _resolve_options( "codex_bin", "gemini_bin", "opencode_bin", + "pi_bin", + "omp_bin", "grok_bin", "schema_max_retries", "schema_mode", diff --git a/sdk/python/agentfield/harness/providers/_factory.py b/sdk/python/agentfield/harness/providers/_factory.py index 129815599..2277153af 100644 --- a/sdk/python/agentfield/harness/providers/_factory.py +++ b/sdk/python/agentfield/harness/providers/_factory.py @@ -6,7 +6,16 @@ from agentfield.harness.providers._base import HarnessProvider from agentfield.types import HarnessConfig -SUPPORTED_PROVIDERS = {"aforge", "claude-code", "codex", "gemini", "opencode", "grok"} +SUPPORTED_PROVIDERS = { + "aforge", + "claude-code", + "codex", + "gemini", + "grok", + "omp", + "opencode", + "pi", +} def build_provider(config: "HarnessConfig") -> "HarnessProvider": @@ -38,6 +47,14 @@ def build_provider(config: "HarnessConfig") -> "HarnessProvider": return OpenCodeProvider( bin_path=getattr(config, "opencode_bin", "opencode"), ) + if provider_name == "pi": + from agentfield.harness.providers.pi import PiProvider + + return PiProvider(bin_path=getattr(config, "pi_bin", "pi")) + if provider_name == "omp": + from agentfield.harness.providers.pi import OMPProvider + + return OMPProvider(bin_path=getattr(config, "omp_bin", "omp")) if provider_name == "grok": from agentfield.harness.providers.grok import GrokProvider diff --git a/sdk/python/agentfield/harness/providers/pi.py b/sdk/python/agentfield/harness/providers/pi.py new file mode 100644 index 000000000..8f366bc95 --- /dev/null +++ b/sdk/python/agentfield/harness/providers/pi.py @@ -0,0 +1,283 @@ +"""Pi-family harness providers using their JSON event-stream CLIs.""" + +from __future__ import annotations + +import time +from typing import Any, Dict, Iterable, Optional + +from agentfield.harness._availability import ensure_cli_available, provider_unavailable +from agentfield.harness._cli import ( + parse_jsonl, + resolve_model_and_variant, + run_cli, + strip_ansi, +) +from agentfield.harness._result import FailureType, Metrics, RawResult + + +_READ_ONLY_TOOLS = {"read", "grep", "find", "glob", "ls", "lsp"} + + +def _normalise_tools(tools: Iterable[object], *, omp: bool) -> list[str]: + """Translate AgentField's provider-neutral tool names to Pi CLI names.""" + aliases = {"glob": "glob" if omp else "find"} + normalised: list[str] = [] + for tool in tools: + name = str(tool).strip().lower() + if not name: + continue + name = aliases.get(name, name) + if name not in normalised: + normalised.append(name) + return normalised + + +def _assistant_messages(events: list[dict[str, Any]]) -> Iterable[dict[str, Any]]: + for event in events: + if event.get("type") != "message_end": + continue + message = event.get("message") + if isinstance(message, dict) and message.get("role") == "assistant": + yield message + + +def _text_content(message: dict[str, Any]) -> Optional[str]: + content = message.get("content") + if isinstance(content, str): + return content or None + if not isinstance(content, list): + return None + parts = [ + part.get("text", "") + for part in content + if isinstance(part, dict) + and part.get("type") == "text" + and isinstance(part.get("text"), str) + ] + text = "".join(parts) + return text or None + + +def _int(value: object) -> int: + if isinstance(value, bool): + return 0 + try: + return int(value) # type: ignore[arg-type] + except (TypeError, ValueError): + return 0 + + +def _float(value: object) -> Optional[float]: + if isinstance(value, bool): + return None + try: + return float(value) # type: ignore[arg-type] + except (TypeError, ValueError): + return None + + +def _parse_pi_events( + events: list[dict[str, Any]], *, configured_model: Optional[str] +) -> tuple[Optional[str], Metrics, Optional[str]]: + result_text: Optional[str] = None + session_id = "" + num_turns = sum(1 for event in events if event.get("type") == "turn_end") + input_tokens = 0 + output_tokens = 0 + cache_read_tokens = 0 + cache_creation_tokens = 0 + total_cost: Optional[float] = None + reported_model: Optional[str] = None + provider_error: Optional[str] = None + + for event in events: + if event.get("type") == "session" and isinstance(event.get("id"), str): + session_id = event["id"] + + for message in _assistant_messages(events): + text = _text_content(message) + if text: + result_text = text + + if isinstance(message.get("model"), str): + reported_model = message["model"] + + usage = message.get("usage") + if isinstance(usage, dict): + input_tokens += _int(usage.get("input")) + output_tokens += _int(usage.get("output")) + cache_read_tokens += _int(usage.get("cacheRead")) + cache_creation_tokens += _int(usage.get("cacheWrite")) + cost = usage.get("cost") + if isinstance(cost, dict): + native_cost = _float(cost.get("total")) + if native_cost is not None: + total_cost = (total_cost or 0.0) + native_cost + + stop_reason = message.get("stopReason") + if stop_reason in {"error", "aborted"}: + detail = message.get("errorMessage") or message.get("error") + provider_error = str(detail or f"Pi stopped with reason {stop_reason!r}.") + else: + provider_error = None + + if num_turns == 0 and result_text: + num_turns = 1 + + return ( + result_text, + Metrics( + num_turns=num_turns, + total_cost_usd=total_cost, + session_id=session_id, + input_tokens=input_tokens, + output_tokens=output_tokens, + cache_read_tokens=cache_read_tokens, + cache_creation_tokens=cache_creation_tokens, + model=configured_model or reported_model, + ), + provider_error, + ) + + +class _PiFamilyProvider: + def __init__(self, *, provider: str, bin_path: str, omp: bool): + self._provider = provider + self._bin = bin_path + self._omp = omp + + async def execute(self, prompt: str, options: dict[str, object]) -> RawResult: + ensure_cli_available(self._provider, self._bin) + cmd = [self._bin, "--print", "--mode", "json"] + + root = options.get("project_dir") or options.get("cwd") + cwd = root if isinstance(root, str) else None + if self._omp and cwd: + cmd.extend(["--cwd", cwd]) + + model_value, variant_value = resolve_model_and_variant(options) + if model_value: + cmd.extend(["--model", model_value]) + if variant_value: + cmd.extend(["--thinking", variant_value]) + + system_prompt = options.get("system_prompt") + if isinstance(system_prompt, str) and system_prompt.strip(): + cmd.extend(["--system-prompt", system_prompt.strip()]) + + resume_session_id = options.get("resume_session_id") + if isinstance(resume_session_id, str) and resume_session_id: + cmd.extend(["--resume" if self._omp else "--session", resume_session_id]) + + permission_mode = options.get("permission_mode") + # --tools is the enforced, vendor-documented read-only allowlist. Pi has + # no approval flag (unknown options fail); OMP read-only tiers are + # auto-approved even under always-ask. + if permission_mode == "auto": + if self._omp: + cmd.append("--auto-approve") + + tools_value = options.get("tools") + tools = ( + _normalise_tools(tools_value, omp=self._omp) + if isinstance(tools_value, (list, tuple, set)) + else [] + ) + if permission_mode == "plan": + tools = [tool for tool in tools if tool in _READ_ONLY_TOOLS] + if not tools: + tools = ["read", "grep", "glob" if self._omp else "find"] + if isinstance(tools_value, (list, tuple, set)) or permission_mode == "plan": + if tools: + cmd.extend(["--tools", ",".join(tools)]) + else: + cmd.append("--no-tools") + + env: Dict[str, str] = {} + env_value = options.get("env") + if isinstance(env_value, dict): + env = { + str(key): str(value) + for key, value in env_value.items() + if isinstance(key, str) and isinstance(value, str) + } + + timeout: Optional[float] = None + timeout_value = options.get("timeout") + if isinstance(timeout_value, (int, float)) and not isinstance( + timeout_value, bool + ): + timeout = float(timeout_value) + + start_api = time.monotonic() + try: + stdout, stderr, returncode = await run_cli( + cmd, + env=env, + cwd=cwd, + timeout=timeout, + input_text=prompt, + ) + except FileNotFoundError as exc: + raise provider_unavailable(self._provider, self._bin) from exc + except TimeoutError as exc: + return RawResult( + is_error=True, + error_message=str(exc), + failure_type=FailureType.TIMEOUT, + metrics=Metrics(), + ) + + api_ms = int((time.monotonic() - start_api) * 1000) + events = parse_jsonl(stdout) + result_text, metrics, provider_error = _parse_pi_events( + events, configured_model=model_value + ) + metrics.duration_api_ms = api_ms + clean_stderr = strip_ansi(stderr.strip()) if stderr else "" + + if returncode < 0: + error_message = f"Process killed by signal {-returncode}." + failure_type = FailureType.CRASH + elif returncode != 0: + error_message = ( + clean_stderr[:1000] + or provider_error + or (f"Process exited with code {returncode}.") + ) + failure_type = FailureType.CRASH + elif provider_error: + error_message = provider_error + failure_type = FailureType.API_ERROR + elif result_text is None: + error_message = clean_stderr[:1000] or ( + f"{self._provider} exited successfully without an assistant response." + ) + failure_type = FailureType.NO_OUTPUT + else: + error_message = None + failure_type = FailureType.NONE + + return RawResult( + result=result_text, + messages=events, + metrics=metrics, + is_error=failure_type != FailureType.NONE, + error_message=error_message, + failure_type=failure_type, + returncode=returncode, + ) + + +class PiProvider(_PiFamilyProvider): + """Pi coding-agent CLI provider.""" + + def __init__(self, bin_path: str = "pi"): + super().__init__(provider="pi", bin_path=bin_path, omp=False) + + +class OMPProvider(_PiFamilyProvider): + """Oh My Pi (OMP) coding-agent CLI provider.""" + + def __init__(self, bin_path: str = "omp"): + super().__init__(provider="omp", bin_path=bin_path, omp=True) diff --git a/sdk/python/agentfield/types.py b/sdk/python/agentfield/types.py index abe3860c7..4cb9f1963 100644 --- a/sdk/python/agentfield/types.py +++ b/sdk/python/agentfield/types.py @@ -285,8 +285,8 @@ class HarnessConfig(BaseModel): default_factory=_default_harness_provider, description=( 'Coding agent provider: "aforge" (default) | "claude-code" | "codex" | ' - '"gemini" | "opencode" | "grok". Unset resolves to the ' - 'AGENTFIELD_HARNESS_PROVIDER env var when present, else "aforge".' + '"gemini" | "opencode" | "grok" | "pi" | "omp". Unset resolves to ' + 'the AGENTFIELD_HARNESS_PROVIDER env var when present, else "aforge".' ), ) model: Optional[str] = Field( @@ -338,6 +338,8 @@ class HarnessConfig(BaseModel): opencode_bin: str = Field( default="opencode", description="Path to opencode binary." ) + pi_bin: str = Field(default="pi", description="Path to Pi binary.") + omp_bin: str = Field(default="omp", description="Path to OMP binary.") aforge_bin: str = Field(default="aforge", description="Path to aforge binary.") grok_bin: str = Field( default="grok", description="Path to Grok Build CLI binary." diff --git a/sdk/python/tests/test_harness_factory.py b/sdk/python/tests/test_harness_factory.py index 97f9965e6..b91a6e1eb 100644 --- a/sdk/python/tests/test_harness_factory.py +++ b/sdk/python/tests/test_harness_factory.py @@ -11,7 +11,10 @@ import pytest from agentfield.harness.providers._base import HarnessProvider -from agentfield.harness.providers._factory import SUPPORTED_PROVIDERS, build_provider +from agentfield.harness.providers._factory import ( + SUPPORTED_PROVIDERS, + build_provider, +) from agentfield.types import HarnessConfig @@ -36,9 +39,53 @@ def test_supported_providers_contains_expected_names(): assert "codex" in SUPPORTED_PROVIDERS assert "gemini" in SUPPORTED_PROVIDERS assert "opencode" in SUPPORTED_PROVIDERS + assert "pi" in SUPPORTED_PROVIDERS + assert "omp" in SUPPORTED_PROVIDERS assert "grok" in SUPPORTED_PROVIDERS +def test_default_provider_is_aforge(monkeypatch): + from agentfield.harness.providers.aforge import AforgeProvider + + # HarnessConfig.provider resolves through AGENTFIELD_HARNESS_PROVIDER, so the + # ambient value has to be cleared for this to assert the built-in default. + monkeypatch.delenv("AGENTFIELD_HARNESS_PROVIDER", raising=False) + assert HarnessConfig().provider == "aforge" + assert isinstance(build_provider(HarnessConfig()), AforgeProvider) + + +def test_build_provider_pi_returns_pi_provider(): + from agentfield.harness.providers.pi import PiProvider + + provider = build_provider(_make_config("pi")) + assert isinstance(provider, PiProvider) + assert provider._bin == "pi" + + +def test_build_provider_pi_custom_bin(): + from agentfield.harness.providers.pi import PiProvider + + provider = build_provider(_make_config("pi", pi_bin="/opt/pi")) + assert isinstance(provider, PiProvider) + assert provider._bin == "/opt/pi" + + +def test_build_provider_omp_returns_omp_provider(): + from agentfield.harness.providers.pi import OMPProvider + + provider = build_provider(_make_config("omp")) + assert isinstance(provider, OMPProvider) + assert provider._bin == "omp" + + +def test_build_provider_omp_custom_bin(): + from agentfield.harness.providers.pi import OMPProvider + + provider = build_provider(_make_config("omp", omp_bin="/opt/omp")) + assert isinstance(provider, OMPProvider) + assert provider._bin == "/opt/omp" + + # --------------------------------------------------------------------------- # build_provider: unknown provider raises ValueError # --------------------------------------------------------------------------- diff --git a/sdk/python/tests/test_harness_provider_availability.py b/sdk/python/tests/test_harness_provider_availability.py index fa420376d..4983e80e4 100644 --- a/sdk/python/tests/test_harness_provider_availability.py +++ b/sdk/python/tests/test_harness_provider_availability.py @@ -11,28 +11,32 @@ from agentfield.harness.providers.claude import ClaudeCodeProvider from agentfield.harness.providers.gemini import GeminiProvider from agentfield.harness.providers.opencode import OpenCodeProvider +from agentfield.harness.providers.pi import OMPProvider, PiProvider @pytest.mark.asyncio @pytest.mark.parametrize( - ("provider", "name", "install_command"), + ("provider", "name", "module", "install_command"), [ - (AforgeProvider(bin_path="aforge-missing"), "aforge", "af aforge ensure"), - (CodexProvider(bin_path="codex-missing"), "codex", "@openai/codex"), + (AforgeProvider(bin_path="aforge-missing"), "aforge", "aforge", "af aforge ensure"), + (CodexProvider(bin_path="codex-missing"), "codex", "codex", "@openai/codex"), ( OpenCodeProvider(bin_path="opencode-missing"), "opencode", + "opencode", "opencode.ai/install", ), + (PiProvider(bin_path="pi-missing"), "pi", "pi", "@earendil-works/pi-coding-agent"), + (OMPProvider(bin_path="omp-missing"), "omp", "pi", "omp.sh/install"), ], ) async def test_cli_provider_raises_typed_error_before_spawn( - monkeypatch, provider, name, install_command + monkeypatch, provider, name, module, install_command ): monkeypatch.setattr("agentfield.harness._availability.shutil.which", lambda _: None) run_cli = AsyncMock() monkeypatch.setattr( - f"agentfield.harness.providers.{name}.run_cli", + f"agentfield.harness.providers.{module}.run_cli", run_cli, ) diff --git a/sdk/python/tests/test_harness_provider_pi.py b/sdk/python/tests/test_harness_provider_pi.py new file mode 100644 index 000000000..65bed145c --- /dev/null +++ b/sdk/python/tests/test_harness_provider_pi.py @@ -0,0 +1,222 @@ +from __future__ import annotations + +import json +from typing import Any + +import pytest + +from agentfield.harness._result import FailureType +from agentfield.harness.providers._factory import build_provider +from agentfield.harness.providers.pi import OMPProvider, PiProvider +from agentfield.types import HarnessConfig + + +@pytest.fixture(autouse=True) +def mock_pi_family_available(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + "agentfield.harness._availability.shutil.which", lambda path: path + ) + + +def _event_stream(text: str) -> str: + events = [ + {"type": "session", "id": "session-123"}, + {"type": "turn_start"}, + { + "type": "message_end", + "message": { + "role": "assistant", + "content": [ + {"type": "thinking", "thinking": "internal"}, + {"type": "text", "text": text}, + ], + "model": "google/gemini-2.5-flash", + "usage": { + "input": 120, + "output": 30, + "cacheRead": 10, + "cacheWrite": 4, + "cost": {"total": 0.0025}, + }, + "stopReason": "stop", + }, + }, + {"type": "turn_end"}, + {"type": "agent_end"}, + ] + return "\n".join(json.dumps(event) for event in events) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("provider", "bin_path", "permission_flag", "glob_tool"), + [ + (PiProvider, "/opt/pi", None, "find"), + (OMPProvider, "/opt/omp", "--auto-approve", "glob"), + ], +) +async def test_pi_family_command_and_metrics( + monkeypatch: pytest.MonkeyPatch, + provider, + bin_path: str, + permission_flag: str | None, + glob_tool: str, +) -> None: + captured: dict[str, Any] = {} + + async def fake_run_cli(cmd, **kwargs): + captured["cmd"] = cmd + captured.update(kwargs) + return _event_stream("done"), "", 0 + + monkeypatch.setattr("agentfield.harness.providers.pi.run_cli", fake_run_cli) + + raw = await provider(bin_path=bin_path).execute( + "implement this", + { + "project_dir": "/tmp/project", + "model": "openrouter/google/gemini-2.5-flash#high", + "permission_mode": "auto", + "system_prompt": "Be precise.", + "tools": ["Read", "Write", "Edit", "Bash", "Glob", "Grep"], + "env": {"EXTRA": "1"}, + }, + ) + + assert captured["cmd"][:4] == [bin_path, "--print", "--mode", "json"] + if provider is OMPProvider: + assert captured["cmd"][4:6] == ["--cwd", "/tmp/project"] + assert ["--model", "openrouter/google/gemini-2.5-flash"] == captured["cmd"][ + captured["cmd"].index("--model") : captured["cmd"].index("--model") + 2 + ] + assert ["--thinking", "high"] == captured["cmd"][ + captured["cmd"].index("--thinking") : captured["cmd"].index("--thinking") + 2 + ] + if permission_flag is not None: + assert permission_flag in captured["cmd"] + _assert_no_approval_flags(captured["cmd"], allowed=permission_flag) + assert captured["cmd"][captured["cmd"].index("--tools") + 1] == ( + f"read,write,edit,bash,{glob_tool},grep" + ) + assert captured["cwd"] == "/tmp/project" + assert captured["input_text"] == "implement this" + assert captured["env"] == {"EXTRA": "1"} + + assert raw.is_error is False + assert raw.result == "done" + assert raw.metrics.session_id == "session-123" + assert raw.metrics.num_turns == 1 + assert raw.metrics.input_tokens == 120 + assert raw.metrics.output_tokens == 30 + assert raw.metrics.cache_read_tokens == 10 + assert raw.metrics.cache_creation_tokens == 4 + assert raw.metrics.total_cost_usd == pytest.approx(0.0025) + assert raw.metrics.model == "openrouter/google/gemini-2.5-flash" + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("provider", "resume_flag", "expected_tools"), + [ + (PiProvider, "--session", "read,grep,find"), + (OMPProvider, "--resume", "read,grep,glob"), + ], +) +async def test_pi_family_plan_mode_is_read_only_and_resumes( + monkeypatch: pytest.MonkeyPatch, + provider, + resume_flag: str, + expected_tools: str, +) -> None: + captured: dict[str, Any] = {} + + async def fake_run_cli(cmd, **kwargs): + captured["cmd"] = cmd + return _event_stream("plan"), "", 0 + + monkeypatch.setattr("agentfield.harness.providers.pi.run_cli", fake_run_cli) + await provider().execute( + "plan this", + { + "permission_mode": "plan", + "tools": ["Read", "Write", "Bash", "Grep", "Glob"], + "resume_session_id": "abc123", + }, + ) + + assert captured["cmd"][captured["cmd"].index("--tools") + 1] == expected_tools + assert captured["cmd"][captured["cmd"].index(resume_flag) + 1] == "abc123" + _assert_no_approval_flags(captured["cmd"]) + + +def _assert_no_approval_flags(cmd: list[str], allowed: str | None = None) -> None: + approval_flags = { + "--approve", + "--auto-approve", + "--yolo", + "-y", + "--approval-mode", + "--permission-mode", + } + assert approval_flags.intersection(cmd) <= ({allowed} if allowed else set()) + + +@pytest.mark.asyncio +async def test_pi_family_nonzero_exit_is_error(monkeypatch: pytest.MonkeyPatch) -> None: + async def fake_run_cli(*_args, **_kwargs): + return "", "authentication failed", 2 + + monkeypatch.setattr("agentfield.harness.providers.pi.run_cli", fake_run_cli) + raw = await PiProvider().execute("hello", {}) + + assert raw.is_error is True + assert raw.error_message == "authentication failed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("provider", [PiProvider, OMPProvider]) +async def test_pi_family_recovered_turn_is_not_an_error( + monkeypatch: pytest.MonkeyPatch, provider +) -> None: + events = [ + { + "type": "message_end", + "message": { + "role": "assistant", + "content": [{"type": "text", "text": "partial"}], + "stopReason": "error", + "errorMessage": "upstream 503", + }, + }, + {"type": "turn_end"}, + { + "type": "message_end", + "message": { + "role": "assistant", + "content": [{"type": "text", "text": "FINAL ANSWER"}], + "stopReason": "stop", + }, + }, + {"type": "turn_end"}, + ] + + async def fake_run_cli(*_args, **_kwargs): + return "\n".join(json.dumps(event) for event in events), "", 0 + + monkeypatch.setattr("agentfield.harness.providers.pi.run_cli", fake_run_cli) + raw = await provider().execute("hello", {}) + + assert raw.result == "FINAL ANSWER" + assert raw.is_error is False + assert raw.failure_type == FailureType.NONE + assert raw.error_message is None + + +def test_factory_builds_pi_and_omp_with_configured_binaries() -> None: + pi = build_provider(HarnessConfig(provider="pi", pi_bin="/opt/pi")) + omp = build_provider(HarnessConfig(provider="omp", omp_bin="/opt/omp")) + + assert isinstance(pi, PiProvider) + assert pi._bin == "/opt/pi" + assert isinstance(omp, OMPProvider) + assert omp._bin == "/opt/omp" diff --git a/sdk/python/tests/test_harness_types.py b/sdk/python/tests/test_harness_types.py index 1a95db42a..a5e060424 100644 --- a/sdk/python/tests/test_harness_types.py +++ b/sdk/python/tests/test_harness_types.py @@ -31,6 +31,8 @@ def test_harness_config_defaults(): assert cfg.codex_bin == "codex" assert cfg.gemini_bin == "gemini" assert cfg.opencode_bin == "opencode" + assert cfg.pi_bin == "pi" + assert cfg.omp_bin == "omp" def test_build_provider_raises_for_unknown_provider(): diff --git a/sdk/python/tests/test_types.py b/sdk/python/tests/test_types.py index 03bfeb696..008cf8f14 100644 --- a/sdk/python/tests/test_types.py +++ b/sdk/python/tests/test_types.py @@ -510,6 +510,8 @@ def test_binary_paths_defaults(self): assert hc.codex_bin == "codex" assert hc.gemini_bin == "gemini" assert hc.opencode_bin == "opencode" + assert hc.pi_bin == "pi" + assert hc.omp_bin == "omp" def test_optional_fields(self): hc = HarnessConfig(provider="p") diff --git a/sdk/python/tests/test_usage_transport.py b/sdk/python/tests/test_usage_transport.py index 0ea94503d..30280e1f7 100644 --- a/sdk/python/tests/test_usage_transport.py +++ b/sdk/python/tests/test_usage_transport.py @@ -634,6 +634,23 @@ def test_record_harness_usage_noop_when_empty(self): reset_current_cost_tracker(token) assert tracker.call_count == 0 + def test_record_harness_usage_explicit_omp(self): + from agentfield.harness._result import HarnessResult + + agent = self._agent() + tracker = CostTracker() + token = set_current_cost_tracker(tracker) + try: + agent._record_harness_usage( + HarnessResult(result="done", input_tokens=1), provider="omp" + ) + finally: + reset_current_cost_tracker(token) + + entry = tracker.serialize()["entries"][0] + assert entry["harness"] == "omp" + assert entry["model"] == "omp" + class TestEnvelopeEndToEnd: """Drive the FastAPI endpoint and assert usage lands in both transports.""" diff --git a/sdk/typescript/src/agent/Agent.ts b/sdk/typescript/src/agent/Agent.ts index 59c9f2b5e..df98df173 100644 --- a/sdk/typescript/src/agent/Agent.ts +++ b/sdk/typescript/src/agent/Agent.ts @@ -40,6 +40,7 @@ import { SkillContext } from '../context/SkillContext.js'; import { AIClient } from '../ai/AIClient.js'; import { AgentFieldClient } from '../client/AgentFieldClient.js'; import type { HarnessRunner } from '../harness/runner.js'; +import { resolveProviderName } from '../harness/providers/factory.js'; import type { HarnessOptions, HarnessResult } from '../harness/types.js'; import { splitModelVariant } from '../harness/modelVariant.js'; import { MemoryClient } from '../memory/MemoryClient.js'; @@ -412,7 +413,7 @@ export class Agent { return; } - const providerName = options?.provider ?? this.config.harnessConfig?.provider; + const providerName = resolveProviderName(options?.provider ?? this.config.harnessConfig?.provider); const harnessName = providerName ? String(providerName).replace(/-/g, '_') : null; // Usage is recorded against the base model — a "#variant" // reasoning-effort suffix on the configured model never reaches the diff --git a/sdk/typescript/src/harness/cli.ts b/sdk/typescript/src/harness/cli.ts index 2ed9e82f8..2dc797f82 100644 --- a/sdk/typescript/src/harness/cli.ts +++ b/sdk/typescript/src/harness/cli.ts @@ -1,4 +1,5 @@ import { spawn } from 'node:child_process'; +import { constants } from 'node:os'; import { applyOpenRouterAttributionEnv } from '../ai/openrouterAttribution.js'; export interface CliResult { @@ -113,13 +114,18 @@ export function runCli( } } - proc.on('close', (code) => { + proc.on('close', (code, signal) => { if (settled) { return; } settled = true; cleanup(); - resolve({ stdout, stderr, exitCode: code ?? 0 }); + const signalNumber = signal ? constants.signals[signal] : undefined; + resolve({ + stdout, + stderr, + exitCode: code ?? (signalNumber === undefined ? 0 : -signalNumber), + }); }); proc.on('error', (err) => { diff --git a/sdk/typescript/src/harness/index.ts b/sdk/typescript/src/harness/index.ts index 50caf8000..3269e1731 100644 --- a/sdk/typescript/src/harness/index.ts +++ b/sdk/typescript/src/harness/index.ts @@ -3,5 +3,11 @@ export { createHarnessResult, createMetrics, createRawResult } from './types.js' export type { ModelVariant } from './modelVariant.js'; export { MODEL_VARIANT_SEP, splitModelVariant, resolveModelAndVariant } from './modelVariant.js'; export type { HarnessProvider } from './providers/base.js'; -export { buildProvider, SUPPORTED_PROVIDERS } from './providers/factory.js'; +export { + buildProvider, + DEFAULT_HARNESS_PROVIDER, + HARNESS_PROVIDER_ENV_VAR, + resolveProviderName, + SUPPORTED_PROVIDERS, +} from './providers/factory.js'; export { HarnessRunner } from './runner.js'; diff --git a/sdk/typescript/src/harness/providers/aforge.ts b/sdk/typescript/src/harness/providers/aforge.ts index b5a11a28d..72a61c409 100644 --- a/sdk/typescript/src/harness/providers/aforge.ts +++ b/sdk/typescript/src/harness/providers/aforge.ts @@ -1,4 +1,5 @@ import type { HarnessProvider } from './base.js'; +import { resolveRoot } from './base.js'; import type { RawResult } from '../types.js'; import { createMetrics, createRawResult } from '../types.js'; import { runCli } from '../cli.js'; @@ -148,13 +149,7 @@ export class AforgeProvider implements HarnessProvider { } private async executeImpl(prompt: string, options: Record): Promise { - const projectDir = typeof options.projectDir === 'string' - ? options.projectDir - : typeof options.project_dir === 'string' - ? options.project_dir - : undefined; - const cwd = typeof options.cwd === 'string' ? options.cwd : undefined; - const root = projectDir ?? cwd ?? '.'; + const root = resolveRoot(options) ?? '.'; const outerTimeout = timeoutSeconds(); const command = (process.env.AGENTFIELD_AFORGE_COMMAND ?? 'exec').trim().toLowerCase(); if (command !== 'do' && command !== 'exec') { diff --git a/sdk/typescript/src/harness/providers/base.ts b/sdk/typescript/src/harness/providers/base.ts index e2da83fda..850ba0292 100644 --- a/sdk/typescript/src/harness/providers/base.ts +++ b/sdk/typescript/src/harness/providers/base.ts @@ -3,3 +3,13 @@ import type { RawResult } from '../types.js'; export interface HarnessProvider { execute(prompt: string, options: Record): Promise; } + +// projectDir/project_dir is the canonical agent root; cwd is the Python-matching fallback. +export function resolveRoot(options: Record): string | undefined { + for (const value of [options.projectDir, options.project_dir, options.cwd]) { + if (typeof value === 'string' && value.length > 0) { + return value; + } + } + return undefined; +} diff --git a/sdk/typescript/src/harness/providers/claude.ts b/sdk/typescript/src/harness/providers/claude.ts index 8026f5e8b..6d33c31de 100644 --- a/sdk/typescript/src/harness/providers/claude.ts +++ b/sdk/typescript/src/harness/providers/claude.ts @@ -1,4 +1,5 @@ import type { HarnessProvider } from './base.js'; +import { resolveRoot } from './base.js'; import type { RawResult } from '../types.js'; import { createMetrics, createRawResult } from '../types.js'; import { resolveModelAndVariant } from '../modelVariant.js'; @@ -44,7 +45,8 @@ export class ClaudeCodeProvider implements HarnessProvider { // suffix so the SDK receives a valid model id, and drop the variant. const { model: modelValue } = resolveModelAndVariant(options); if (modelValue !== undefined) agentOptions.model = modelValue; - if (options.cwd !== undefined) agentOptions.cwd = options.cwd; + const root = resolveRoot(options); + if (root !== undefined) agentOptions.cwd = root; if (options.maxTurns !== undefined) agentOptions.maxTurns = options.maxTurns; if (options.tools !== undefined) agentOptions.allowedTools = options.tools; if (options.systemPrompt !== undefined) agentOptions.systemPrompt = options.systemPrompt; diff --git a/sdk/typescript/src/harness/providers/codex.ts b/sdk/typescript/src/harness/providers/codex.ts index 89463f5de..df3c5ef99 100644 --- a/sdk/typescript/src/harness/providers/codex.ts +++ b/sdk/typescript/src/harness/providers/codex.ts @@ -1,4 +1,5 @@ import type { HarnessProvider } from './base.js'; +import { resolveRoot } from './base.js'; import type { RawResult } from '../types.js'; import { createRawResult, createMetrics } from '../types.js'; import { runCli, parseJsonl, extractFinalText } from '../cli.js'; @@ -13,9 +14,10 @@ export class CodexProvider implements HarnessProvider { async execute(prompt: string, options: Record): Promise { const cmd = [this.bin, 'exec', '--json']; + const root = resolveRoot(options); - if (options.cwd) { - cmd.push('-C', String(options.cwd)); + if (root) { + cmd.push('-C', root); } if (options.permissionMode === 'auto') { cmd.push('--full-auto'); @@ -39,7 +41,7 @@ export class CodexProvider implements HarnessProvider { try { const { stdout, stderr, exitCode } = await runCli(cmd, { env: options.env as Record | undefined, - cwd: options.cwd as string | undefined, + cwd: root, }); const events = parseJsonl(stdout); diff --git a/sdk/typescript/src/harness/providers/factory.ts b/sdk/typescript/src/harness/providers/factory.ts index ebf9d97c9..8eea1c319 100644 --- a/sdk/typescript/src/harness/providers/factory.ts +++ b/sdk/typescript/src/harness/providers/factory.ts @@ -1,7 +1,15 @@ import type { HarnessProvider } from './base.js'; import type { HarnessConfig } from '../types.js'; -export const SUPPORTED_PROVIDERS = new Set(['aforge', 'claude-code', 'codex', 'gemini', 'opencode']); +export const SUPPORTED_PROVIDERS = new Set([ + 'aforge', + 'claude-code', + 'codex', + 'gemini', + 'omp', + 'opencode', + 'pi', +]); export const DEFAULT_HARNESS_PROVIDER = 'aforge'; export const HARNESS_PROVIDER_ENV_VAR = 'AGENTFIELD_HARNESS_PROVIDER'; @@ -47,5 +55,13 @@ export async function buildProvider(config: HarnessConfig): Promise | undefined, - cwd: options.cwd as string | undefined, + cwd: resolveRoot(options), }); const resultText = stdout.trim() || undefined; diff --git a/sdk/typescript/src/harness/providers/index.ts b/sdk/typescript/src/harness/providers/index.ts index ae479f4e2..d8ba9ffb4 100644 --- a/sdk/typescript/src/harness/providers/index.ts +++ b/sdk/typescript/src/harness/providers/index.ts @@ -1,7 +1,14 @@ export type { HarnessProvider } from './base.js'; -export { buildProvider, SUPPORTED_PROVIDERS } from './factory.js'; +export { + buildProvider, + DEFAULT_HARNESS_PROVIDER, + HARNESS_PROVIDER_ENV_VAR, + resolveProviderName, + SUPPORTED_PROVIDERS, +} from './factory.js'; export { AforgeProvider } from './aforge.js'; export { ClaudeCodeProvider } from './claude.js'; export { CodexProvider } from './codex.js'; export { GeminiProvider } from './gemini.js'; export { OpenCodeProvider } from './opencode.js'; +export { OMPProvider, PiProvider } from './pi.js'; diff --git a/sdk/typescript/src/harness/providers/opencode.ts b/sdk/typescript/src/harness/providers/opencode.ts index b24b66e36..beb6d0c6d 100644 --- a/sdk/typescript/src/harness/providers/opencode.ts +++ b/sdk/typescript/src/harness/providers/opencode.ts @@ -1,4 +1,5 @@ import type { HarnessProvider } from './base.js'; +import { resolveRoot } from './base.js'; import type { RawResult } from '../types.js'; import { createRawResult, createMetrics } from '../types.js'; import { runCli } from '../cli.js'; @@ -43,10 +44,9 @@ export class OpenCodeProvider implements HarnessProvider { const cmd = [this.bin, 'run']; // Use --dir for project directory. - if (options.cwd && typeof options.cwd === 'string') { - cmd.push('--dir', options.cwd); - } else if (options.project_dir && typeof options.project_dir === 'string') { - cmd.push('--dir', options.project_dir); + const root = resolveRoot(options); + if (root) { + cmd.push('--dir', root); } const env: Record = { ...(options.env as Record) }; diff --git a/sdk/typescript/src/harness/providers/pi.ts b/sdk/typescript/src/harness/providers/pi.ts new file mode 100644 index 000000000..0357db098 --- /dev/null +++ b/sdk/typescript/src/harness/providers/pi.ts @@ -0,0 +1,252 @@ +import type { HarnessProvider } from './base.js'; +import { resolveRoot } from './base.js'; +import type { RawResult } from '../types.js'; +import { createMetrics, createRawResult } from '../types.js'; +import { parseJsonl, runCli } from '../cli.js'; +import { resolveModelAndVariant } from '../modelVariant.js'; + +type PiFlavor = 'pi' | 'omp'; + +const READ_ONLY_TOOLS = new Set(['read', 'grep', 'find', 'glob', 'ls', 'lsp']); +const ANSI_PATTERN = /\x1B\[[0-?]*[ -/]*[@-~]/g; + +function normalizeTools(tools: unknown[], flavor: PiFlavor): string[] { + const normalized: string[] = []; + for (const tool of tools) { + let name = String(tool).trim().toLowerCase(); + if (!name) { + continue; + } + if (name === 'glob') { + name = flavor === 'omp' ? 'glob' : 'find'; + } + if (!normalized.includes(name)) { + normalized.push(name); + } + } + return normalized; +} + +function numberValue(value: unknown): number { + return typeof value === 'number' && Number.isFinite(value) ? value : 0; +} + +function textContent(message: Record): string | undefined { + if (typeof message.content === 'string') { + return message.content || undefined; + } + if (!Array.isArray(message.content)) { + return undefined; + } + const text = message.content + .filter((part): part is Record => typeof part === 'object' && part !== null) + .filter((part) => part.type === 'text' && typeof part.text === 'string') + .map((part) => part.text as string) + .join(''); + return text || undefined; +} + +function parsePiEvents(events: Array>, configuredModel?: string) { + let result: string | undefined; + let sessionId = ''; + let numTurns = 0; + let inputTokens = 0; + let outputTokens = 0; + let cacheReadTokens = 0; + let cacheCreationTokens = 0; + let totalCostUsd: number | undefined; + let reportedModel: string | undefined; + let providerError: string | undefined; + + for (const event of events) { + if (event.type === 'session' && typeof event.id === 'string') { + sessionId = event.id; + } + if (event.type === 'turn_end') { + numTurns += 1; + } + if (event.type !== 'message_end' || typeof event.message !== 'object' || event.message === null) { + continue; + } + const message = event.message as Record; + if (message.role !== 'assistant') { + continue; + } + + result = textContent(message) ?? result; + if (typeof message.model === 'string') { + reportedModel = message.model; + } + + if (typeof message.usage === 'object' && message.usage !== null) { + const usage = message.usage as Record; + inputTokens += numberValue(usage.input); + outputTokens += numberValue(usage.output); + cacheReadTokens += numberValue(usage.cacheRead); + cacheCreationTokens += numberValue(usage.cacheWrite); + if (typeof usage.cost === 'object' && usage.cost !== null) { + const cost = (usage.cost as Record).total; + if (typeof cost === 'number' && Number.isFinite(cost)) { + totalCostUsd = (totalCostUsd ?? 0) + cost; + } + } + } + + if (message.stopReason === 'error' || message.stopReason === 'aborted') { + providerError = String( + message.errorMessage ?? message.error ?? `Pi stopped with reason ${String(message.stopReason)}.` + ); + } else { + providerError = undefined; + } + } + + if (numTurns === 0 && result) { + numTurns = 1; + } + + return { + result, + providerError, + metrics: createMetrics({ + numTurns, + totalCostUsd, + sessionId, + inputTokens, + outputTokens, + cacheReadTokens, + cacheCreationTokens, + totalTokens: inputTokens + outputTokens, + model: configuredModel ?? reportedModel, + }), + }; +} + +class PiFamilyProvider implements HarnessProvider { + public constructor( + private readonly flavor: PiFlavor, + private readonly bin: string, + ) {} + + public async execute(prompt: string, options: Record): Promise { + const cmd = [this.bin, '--print', '--mode', 'json']; + const root = resolveRoot(options); + + if (this.flavor === 'omp' && root) { + cmd.push('--cwd', root); + } + + const { model, variant } = resolveModelAndVariant(options); + if (model) { + cmd.push('--model', model); + } + if (variant) { + cmd.push('--thinking', variant); + } + + if (typeof options.systemPrompt === 'string' && options.systemPrompt.trim()) { + cmd.push('--system-prompt', options.systemPrompt.trim()); + } + + if (typeof options.resumeSessionId === 'string' && options.resumeSessionId) { + cmd.push(this.flavor === 'omp' ? '--resume' : '--session', options.resumeSessionId); + } + + // --tools is the enforced, vendor-documented read-only allowlist. Pi has no + // approval flag (unknown options fail); OMP read-only tiers are auto-approved + // even under always-ask. + if (options.permissionMode === 'auto') { + if (this.flavor === 'omp') { + cmd.push('--auto-approve'); + } + } + + const explicitTools = Array.isArray(options.tools); + let tools = explicitTools ? normalizeTools(options.tools as unknown[], this.flavor) : []; + if (options.permissionMode === 'plan') { + tools = tools.filter((tool) => READ_ONLY_TOOLS.has(tool)); + if (tools.length === 0) { + tools = ['read', 'grep', this.flavor === 'omp' ? 'glob' : 'find']; + } + } + if (explicitTools || options.permissionMode === 'plan') { + if (tools.length > 0) { + cmd.push('--tools', tools.join(',')); + } else { + cmd.push('--no-tools'); + } + } + + const env = { ...options.env as Record | undefined }; + const startApi = Date.now(); + try { + const { stdout, stderr, exitCode } = await runCli(cmd, { + env, + cwd: root, + inputText: prompt, + timeout: typeof options.timeout === 'number' ? options.timeout * 1000 : undefined, + }); + const events = parseJsonl(stdout); + const parsed = parsePiEvents(events, model); + parsed.metrics.durationApiMs = Date.now() - startApi; + + const cleanStderr = stderr.trim().replace(ANSI_PATTERN, '').slice(0, 1000); + let errorMessage: string | undefined; + let failureType: NonNullable; + if (exitCode < 0) { + errorMessage = `Process killed by signal ${-exitCode}.`; + failureType = 'crash'; + } else if (exitCode !== 0) { + errorMessage = cleanStderr || parsed.providerError || `Process exited with code ${exitCode}.`; + failureType = 'crash'; + } else if (parsed.providerError) { + errorMessage = parsed.providerError; + failureType = 'api_error'; + } else if (!parsed.result) { + errorMessage = cleanStderr || `${this.flavor} exited successfully without an assistant response.`; + failureType = 'no_output'; + } else { + failureType = 'none'; + } + + return createRawResult({ + result: parsed.result, + messages: events, + metrics: parsed.metrics, + isError: failureType !== 'none', + errorMessage, + failureType, + returnCode: exitCode, + }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + const binaryMissing = message.includes('ENOENT'); + return createRawResult({ + isError: true, + errorMessage: binaryMissing + ? `${this.flavor === 'omp' ? 'OMP' : 'Pi'} binary not found at '${this.bin}'. ${ + this.flavor === 'omp' + ? 'Install: curl -fsSL https://omp.sh/install | sh' + : 'Install: npm install -g --ignore-scripts @earendil-works/pi-coding-agent' + }` + : message, + failureType: /timed out|deadline exceeded|no progress/i.test(message) + ? 'timeout' + : 'crash', + metrics: createMetrics({ durationApiMs: Date.now() - startApi }), + }); + } + } +} + +export class PiProvider extends PiFamilyProvider { + public constructor(binPath = 'pi') { + super('pi', binPath); + } +} + +export class OMPProvider extends PiFamilyProvider { + public constructor(binPath = 'omp') { + super('omp', binPath); + } +} diff --git a/sdk/typescript/src/harness/runner.ts b/sdk/typescript/src/harness/runner.ts index 722ed80b1..c4e1ae732 100644 --- a/sdk/typescript/src/harness/runner.ts +++ b/sdk/typescript/src/harness/runner.ts @@ -2,7 +2,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { buildPromptSuffix, cleanupTempFiles, getOutputPath, parseAndValidate } from './schema.js'; import { buildProvider, resolveProviderName } from './providers/factory.js'; -import type { HarnessProvider } from './providers/base.js'; +import { resolveRoot, type HarnessProvider } from './providers/base.js'; import { createHarnessResult, createRawResult, @@ -48,6 +48,8 @@ type RunnerOptions = Omit & { codexBin?: string; geminiBin?: string; opencodeBin?: string; + piBin?: string; + ompBin?: string; }; export class HarnessRunner { @@ -59,8 +61,7 @@ export class HarnessRunner { resolved.provider = resolveProviderName(resolved.provider); const provider = await this.buildProvider(resolved.provider, resolved); - const cwd = resolved.cwd ?? '.'; - const outputRoot = resolved.projectDir ?? cwd; + const outputRoot = resolveRoot(resolved as Record) ?? '.'; let outputDir: string | undefined; if (schema !== undefined) { fs.mkdirSync(outputRoot, { recursive: true }); @@ -120,6 +121,8 @@ export class HarnessRunner { 'codexBin', 'geminiBin', 'opencodeBin', + 'piBin', + 'ompBin', ] as const) { const value = config[key]; if (value !== undefined && value !== null) { diff --git a/sdk/typescript/src/harness/types.ts b/sdk/typescript/src/harness/types.ts index e1385e8fa..453426bda 100644 --- a/sdk/typescript/src/harness/types.ts +++ b/sdk/typescript/src/harness/types.ts @@ -4,7 +4,7 @@ export interface HarnessConfig { * When unset, `AGENTFIELD_HARNESS_PROVIDER` is consulted before the default. * An explicit value always wins. */ - provider?: 'aforge' | 'claude-code' | 'codex' | 'gemini' | 'opencode'; + provider?: 'aforge' | 'claude-code' | 'codex' | 'gemini' | 'opencode' | 'pi' | 'omp'; /** Model identifier. Empty means the provider's own default. */ model?: string; /** @@ -28,6 +28,8 @@ export interface HarnessConfig { codexBin?: string; geminiBin?: string; opencodeBin?: string; + piBin?: string; + ompBin?: string; } export interface HarnessOptions { @@ -60,6 +62,10 @@ export interface HarnessOptions { codexBin?: string; geminiBin?: string; opencodeBin?: string; + piBin?: string; + ompBin?: string; + resumeSessionId?: string; + timeout?: number; schema?: unknown; } diff --git a/sdk/typescript/tests/harness_cli.test.ts b/sdk/typescript/tests/harness_cli.test.ts index 38531b76b..7988338d3 100644 --- a/sdk/typescript/tests/harness_cli.test.ts +++ b/sdk/typescript/tests/harness_cli.test.ts @@ -68,6 +68,16 @@ describe('harness cli utilities', () => { }); }); + it('reports a signal death as a negative exit code', async () => { + const proc = createProcess(); + spawnMock.mockReturnValueOnce(proc as unknown as ReturnType); + + const pending = runCli(['killed']); + proc.emit('close', null, 'SIGKILL'); + + await expect(pending).resolves.toMatchObject({ exitCode: -9 }); + }); + it('adds OpenRouter attribution env defaults and preserves caller overrides', async () => { const proc = createProcess(); spawnMock.mockReturnValueOnce(proc as unknown as ReturnType); diff --git a/sdk/typescript/tests/harness_provider_claude.test.ts b/sdk/typescript/tests/harness_provider_claude.test.ts index c0e20e3ff..5de84458a 100644 --- a/sdk/typescript/tests/harness_provider_claude.test.ts +++ b/sdk/typescript/tests/harness_provider_claude.test.ts @@ -116,6 +116,52 @@ describe('ClaudeCodeProvider', () => { expect(captured.options).toEqual({}); }); + it('prefers projectDir over a nested cwd', async () => { + const captured: { options?: Record } = {}; + + vi.doMock( + '@anthropic-ai/claude-agent-sdk', + () => ({ + query: ({ options }: { prompt: string; options: Record }) => { + captured.options = options; + return (async function* stream() { + yield { type: 'result', result: 'final' }; + })(); + }, + }), + { virtual: true } + ); + + const { ClaudeCodeProvider } = await import('../src/harness/providers/claude.js'); + const provider = new ClaudeCodeProvider(); + await provider.execute('hello', { projectDir: '/proj', cwd: '/proj/nested' }); + + expect(captured.options?.cwd).toBe('/proj'); + }); + + it('omits cwd when it is an empty string', async () => { + const captured: { options?: Record } = {}; + + vi.doMock( + '@anthropic-ai/claude-agent-sdk', + () => ({ + query: ({ options }: { prompt: string; options: Record }) => { + captured.options = options; + return (async function* stream() { + yield { type: 'result', result: 'final' }; + })(); + }, + }), + { virtual: true } + ); + + const { ClaudeCodeProvider } = await import('../src/harness/providers/claude.js'); + const provider = new ClaudeCodeProvider(); + await provider.execute('hello', { cwd: '' }); + + expect(captured.options).not.toHaveProperty('cwd'); + }); + it('strips a #variant model suffix before handing the model to the SDK', async () => { const captured: { options?: Record } = {}; diff --git a/sdk/typescript/tests/harness_provider_factory.test.ts b/sdk/typescript/tests/harness_provider_factory.test.ts index 236af0d35..9e2aed6e9 100644 --- a/sdk/typescript/tests/harness_provider_factory.test.ts +++ b/sdk/typescript/tests/harness_provider_factory.test.ts @@ -49,10 +49,16 @@ describe('harness provider factory', () => { await expect(buildProvider({})).resolves.toBeInstanceOf(AforgeProvider); }); + it('builds the additional pi and omp providers when named explicitly', async () => { + const { PiProvider, OMPProvider } = await import('../src/harness/providers/pi.js'); + await expect(buildProvider({ provider: 'pi' })).resolves.toBeInstanceOf(PiProvider); + await expect(buildProvider({ provider: 'omp' })).resolves.toBeInstanceOf(OMPProvider); + }); + it('rejects genuinely unknown providers with the supported list', async () => { const config = { provider: 'nope' } as unknown as HarnessConfig; await expect(buildProvider(config)).rejects.toThrow( - 'Unknown harness provider: "nope". Supported: aforge, claude-code, codex, gemini, opencode' + 'Unknown harness provider: "nope". Supported: aforge, claude-code, codex, gemini, omp, opencode, pi' ); }); }); diff --git a/sdk/typescript/tests/harness_provider_pi.test.ts b/sdk/typescript/tests/harness_provider_pi.test.ts new file mode 100644 index 000000000..4ea39e234 --- /dev/null +++ b/sdk/typescript/tests/harness_provider_pi.test.ts @@ -0,0 +1,296 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import * as cli from '../src/harness/cli.js'; +import { buildProvider } from '../src/harness/providers/factory.js'; +import { OMPProvider, PiProvider } from '../src/harness/providers/pi.js'; + +afterEach(() => { + vi.restoreAllMocks(); +}); + +function eventStream(text: string): string { + return [ + { type: 'session', id: 'session-123' }, + { type: 'turn_start' }, + { + type: 'message_end', + message: { + role: 'assistant', + content: [ + { type: 'thinking', thinking: 'internal' }, + { type: 'text', text }, + ], + model: 'google/gemini-2.5-flash', + usage: { + input: 120, + output: 30, + cacheRead: 10, + cacheWrite: 4, + cost: { total: 0.0025 }, + }, + stopReason: 'stop', + }, + }, + { type: 'turn_end' }, + { type: 'agent_end' }, + ].map((event) => JSON.stringify(event)).join('\n'); +} + +const providers = [ + { name: 'pi', provider: new PiProvider() }, + { name: 'omp', provider: new OMPProvider() }, +]; + +it.each(providers)('$name classifies a successful assistant response', async ({ provider }) => { + vi.spyOn(cli, 'runCli').mockResolvedValue({ + stdout: eventStream('done'), + stderr: '', + exitCode: 0, + }); + + const result = await provider.execute('hello', {}); + + expect(result).toMatchObject({ isError: false, failureType: 'none', returnCode: 0 }); +}); + +it.each(providers)('$name classifies a signal death as a crash', async ({ provider }) => { + vi.spyOn(cli, 'runCli').mockResolvedValue({ stdout: '', stderr: '', exitCode: -9 }); + + const result = await provider.execute('hello', {}); + + expect(result).toMatchObject({ + isError: true, + failureType: 'crash', + returnCode: -9, + errorMessage: 'Process killed by signal 9.', + }); +}); + +it.each(providers)('$name uses cleaned stderr for a non-zero exit', async ({ provider }) => { + vi.spyOn(cli, 'runCli').mockResolvedValue({ + stdout: '', + stderr: ' \u001b[31mprovider failed\u001b[0m ', + exitCode: 2, + }); + + const result = await provider.execute('hello', {}); + + expect(result).toMatchObject({ + isError: true, + failureType: 'crash', + returnCode: 2, + errorMessage: 'provider failed', + }); +}); + +it.each(providers)('$name falls back to the exit code for an empty error', async ({ provider }) => { + vi.spyOn(cli, 'runCli').mockResolvedValue({ stdout: '', stderr: '', exitCode: 2 }); + + const result = await provider.execute('hello', {}); + + expect(result.errorMessage).toBe('Process exited with code 2.'); +}); + +it.each(providers)('$name classifies a provider event error', async ({ provider }) => { + const stdout = JSON.stringify({ + type: 'message_end', + message: { + role: 'assistant', + content: [], + stopReason: 'error', + errorMessage: 'provider detail', + }, + }); + vi.spyOn(cli, 'runCli').mockResolvedValue({ stdout, stderr: '', exitCode: 0 }); + + const result = await provider.execute('hello', {}); + + expect(result).toMatchObject({ + isError: true, + failureType: 'api_error', + returnCode: 0, + errorMessage: 'provider detail', + }); +}); + +it.each(providers)('$name clears a recovered provider event error', async ({ provider }) => { + const stdout = [ + { + type: 'message_end', + message: { + role: 'assistant', + content: [{ type: 'text', text: 'partial' }], + stopReason: 'error', + errorMessage: 'upstream 503', + }, + }, + { type: 'turn_end' }, + { + type: 'message_end', + message: { + role: 'assistant', + content: [{ type: 'text', text: 'FINAL ANSWER' }], + stopReason: 'stop', + }, + }, + { type: 'turn_end' }, + ].map((event) => JSON.stringify(event)).join('\n'); + vi.spyOn(cli, 'runCli').mockResolvedValue({ stdout, stderr: '', exitCode: 0 }); + + const result = await provider.execute('hello', {}); + + expect(result).toMatchObject({ + result: 'FINAL ANSWER', + isError: false, + failureType: 'none', + returnCode: 0, + }); + expect(result.errorMessage).toBeUndefined(); +}); + +it.each(providers)('$name classifies a successful exit without output', async ({ provider }) => { + vi.spyOn(cli, 'runCli').mockResolvedValue({ stdout: '', stderr: '', exitCode: 0 }); + + const result = await provider.execute('hello', {}); + + expect(result).toMatchObject({ isError: true, failureType: 'no_output', returnCode: 0 }); +}); + +it.each([ + { message: 'spawn ENOENT', failureType: 'crash' }, + { message: 'CLI timed out after 1000ms', failureType: 'timeout' }, +])('classifies a rejected CLI as $failureType', async ({ message, failureType }) => { + vi.spyOn(cli, 'runCli').mockRejectedValue(new Error(message)); + + const result = await new PiProvider().execute('hello', {}); + + expect(result).toMatchObject({ isError: true, failureType }); +}); + +describe.each([ + { + name: 'pi', + provider: new PiProvider('/opt/pi'), + prefix: ['/opt/pi', '--print', '--mode', 'json'], + permissionFlag: undefined, + globTool: 'find', + }, + { + name: 'omp', + provider: new OMPProvider('/opt/omp'), + prefix: ['/opt/omp', '--print', '--mode', 'json', '--cwd', '/tmp/project'], + permissionFlag: '--auto-approve', + globTool: 'glob', + }, +])('$name provider', ({ provider, prefix, permissionFlag, globTool }) => { + it('maps common harness options and native metrics', async () => { + vi.spyOn(cli, 'runCli').mockResolvedValue({ + stdout: eventStream('done'), + stderr: '', + exitCode: 0, + }); + + const result = await provider.execute('implement this', { + projectDir: '/tmp/project', + model: 'openrouter/google/gemini-2.5-flash#high', + permissionMode: 'auto', + systemPrompt: 'Be precise.', + tools: ['Read', 'Write', 'Edit', 'Bash', 'Glob', 'Grep'], + env: { EXTRA: '1' }, + }); + + const [cmd, options] = vi.mocked(cli.runCli).mock.calls[0]; + expect(cmd.slice(0, prefix.length)).toEqual(prefix); + if (permissionFlag) { + expect(cmd).toContain(permissionFlag); + } + expectApprovalFlags(cmd, permissionFlag); + expect(cmd.slice(cmd.indexOf('--model'), cmd.indexOf('--model') + 2)).toEqual([ + '--model', + 'openrouter/google/gemini-2.5-flash', + ]); + expect(cmd.slice(cmd.indexOf('--thinking'), cmd.indexOf('--thinking') + 2)).toEqual([ + '--thinking', + 'high', + ]); + expect(cmd[cmd.indexOf('--tools') + 1]).toBe(`read,write,edit,bash,${globTool},grep`); + expect(options).toEqual({ + env: { EXTRA: '1' }, + cwd: '/tmp/project', + inputText: 'implement this', + timeout: undefined, + }); + + expect(result.isError).toBe(false); + expect(result.result).toBe('done'); + expect(result.metrics).toMatchObject({ + sessionId: 'session-123', + numTurns: 1, + totalCostUsd: 0.0025, + inputTokens: 120, + outputTokens: 30, + cacheReadTokens: 10, + cacheCreationTokens: 4, + totalTokens: 150, + model: 'openrouter/google/gemini-2.5-flash', + }); + }); +}); + +it.each([ + { provider: new PiProvider(), resumeFlag: '--session', tools: 'read,grep,find' }, + { provider: new OMPProvider(), resumeFlag: '--resume', tools: 'read,grep,glob' }, +])('keeps $resumeFlag retries read-only', async ({ provider, resumeFlag, tools }) => { + vi.spyOn(cli, 'runCli').mockResolvedValue({ + stdout: eventStream('plan'), + stderr: '', + exitCode: 0, + }); + + await provider.execute('plan this', { + permissionMode: 'plan', + tools: ['Read', 'Write', 'Bash', 'Grep', 'Glob'], + resumeSessionId: 'abc123', + }); + + const cmd = vi.mocked(cli.runCli).mock.calls[0][0]; + expect(cmd[cmd.indexOf('--tools') + 1]).toBe(tools); + expect(cmd[cmd.indexOf(resumeFlag) + 1]).toBe('abc123'); + expectApprovalFlags(cmd); +}); + +function expectApprovalFlags(cmd: string[], allowed?: string): void { + const approvalFlags = [ + '--approve', + '--auto-approve', + '--yolo', + '-y', + '--approval-mode', + '--permission-mode', + ]; + expect(approvalFlags.filter((flag) => cmd.includes(flag))).toEqual(allowed ? [allowed] : []); +} + +it.each([ + { provider: new PiProvider('pi-missing'), installHint: '@earendil-works/pi-coding-agent' }, + { provider: new OMPProvider('omp-missing'), installHint: 'omp.sh/install' }, +])('returns actionable install guidance for a missing binary', async ({ provider, installHint }) => { + vi.spyOn(cli, 'runCli').mockRejectedValue(new Error('spawn ENOENT')); + + const result = await provider.execute('hello', {}); + + expect(result.isError).toBe(true); + expect(result.errorMessage).toContain(installHint); +}); + +describe('provider factory', () => { + it('routes pi and omp and passes binary overrides', async () => { + const pi = await buildProvider({ provider: 'pi', piBin: '/opt/pi' }); + const omp = await buildProvider({ provider: 'omp', ompBin: '/opt/omp' }); + + expect(pi).toBeInstanceOf(PiProvider); + expect(omp).toBeInstanceOf(OMPProvider); + // Pi and OMP are additional providers: they must be named explicitly. + // A provider-less config still resolves to the default, aforge. + }); +}); diff --git a/sdk/typescript/tests/harness_runner.test.ts b/sdk/typescript/tests/harness_runner.test.ts index 304ce35b4..4f98e86f7 100644 --- a/sdk/typescript/tests/harness_runner.test.ts +++ b/sdk/typescript/tests/harness_runner.test.ts @@ -4,11 +4,12 @@ import path from 'node:path'; import { afterEach, describe, expect, it, vi } from 'vitest'; import { z } from 'zod'; -import type { HarnessConfig, RawResult } from '../src/harness/types.js'; +import type { HarnessConfig, HarnessOptions, RawResult } from '../src/harness/types.js'; import type { HarnessProvider } from '../src/harness/providers/base.js'; import { createMetrics, createRawResult } from '../src/harness/types.js'; import { getOutputPath } from '../src/harness/schema.js'; import { HarnessRunner } from '../src/harness/runner.js'; +import * as cli from '../src/harness/cli.js'; import * as factory from '../src/harness/providers/factory.js'; const tempDirs: string[] = []; @@ -64,6 +65,64 @@ class FileWritingProvider extends MockProvider { } describe('harness runner', () => { + it.each(['codex', 'gemini', 'opencode', 'pi'] as const)( + '%s uses projectDir for schema instructions and provider execution', + async (provider) => { + const projectDir = makeTempDir(); + const cwd = makeTempDir(); + vi.spyOn(cli, 'runCli').mockResolvedValue({ stdout: '', stderr: '', exitCode: 0 }); + + const runner = new HarnessRunner(); + await runner.run('do it', { + schema: z.object({ answer: z.string() }), + provider, + projectDir, + cwd, + }); + + const [cmd, runOptions] = vi.mocked(cli.runCli).mock.calls[0]; + const prompt = provider === 'pi' + ? runOptions?.inputText + : cmd[cmd.length - 1]; + const outputPath = prompt?.match(/(\S*\.agentfield_output\.json)/)?.[1]; + + expect(outputPath).toBeDefined(); + expect(path.relative(projectDir, outputPath as string)).not.toMatch(/^\.\.(?:[/\\]|$)/); + if (provider === 'opencode') { + expect(cmd.slice(cmd.indexOf('--dir'), cmd.indexOf('--dir') + 2)).toEqual(['--dir', projectDir]); + } else { + expect(runOptions?.cwd).toBe(projectDir); + } + if (provider === 'codex') { + expect(cmd.slice(cmd.indexOf('-C'), cmd.indexOf('-C') + 2)).toEqual(['-C', projectDir]); + } + expect(projectDir).not.toBe(cwd); + } + ); + + it('uses snake_case project_dir for schema instructions and codex execution', async () => { + const projectDir = makeTempDir(); + const cwd = makeTempDir(); + vi.spyOn(cli, 'runCli').mockResolvedValue({ stdout: '', stderr: '', exitCode: 0 }); + + const runner = new HarnessRunner(); + await runner.run('do it', { + schema: z.object({ answer: z.string() }), + provider: 'codex', + project_dir: projectDir, + cwd, + } as unknown as HarnessOptions); + + const [cmd, runOptions] = vi.mocked(cli.runCli).mock.calls[0]; + const prompt = cmd[cmd.length - 1]; + const outputPath = prompt?.match(/(\S*\.agentfield_output\.json)/)?.[1]; + + expect(outputPath).toBeDefined(); + expect(path.relative(projectDir, outputPath as string)).not.toMatch(/^\.\.(?:[/\\]|$)/); + expect(runOptions?.cwd).toBe(projectDir); + expect(projectDir).not.toBe(cwd); + }); + it('resolveOptions merges config with per-call overrides', () => { const cfg: HarnessConfig = { provider: 'codex', @@ -80,6 +139,8 @@ describe('harness runner', () => { codexBin: 'codex', geminiBin: 'gemini', opencodeBin: 'opencode', + piBin: 'pi', + ompBin: 'omp', }; const runner = new HarnessRunner(cfg); @@ -99,6 +160,8 @@ describe('harness runner', () => { expect(options.cwd).toBe('/tmp/override'); expect(options.projectDir).toBe('/tmp/project'); expect(options.aforgeBin).toBe('aforge'); + expect(options.piBin).toBe('pi'); + expect(options.ompBin).toBe('omp'); }); it('isTransient matches transient errors and rejects non-transient', () => { diff --git a/sdk/typescript/tests/usage_ai_capture.test.ts b/sdk/typescript/tests/usage_ai_capture.test.ts index ced45637e..3315860d6 100644 --- a/sdk/typescript/tests/usage_ai_capture.test.ts +++ b/sdk/typescript/tests/usage_ai_capture.test.ts @@ -273,7 +273,7 @@ describe('harness usage capture', () => { }); it('records cost-only runs and skips runs that reported nothing', async () => { - const agent = makeAgent(); + const agent = new Agent({ nodeId: 'default-harness-agent' }); const runSpy = vi.spyOn(HarnessRunner.prototype, 'run'); // Cost known, tokens unknown -> still recorded. @@ -283,7 +283,8 @@ describe('harness usage capture', () => { expect(ctx.costTracker.serialize().entries[0]).toMatchObject({ cost_usd: 0.25, cost_source: 'provider', - total_tokens: 0 + total_tokens: 0, + harness: 'aforge' }); // Neither tokens nor cost -> no entry. diff --git a/skills/agentfield/SKILL.md b/skills/agentfield/SKILL.md index 50e50a940..897ff985f 100644 --- a/skills/agentfield/SKILL.md +++ b/skills/agentfield/SKILL.md @@ -68,7 +68,7 @@ Everything else is a variation. Less-used but real: - **`@app.skill()`** — deterministic functions you want callable through the control plane (no LLM). -- **`app.harness(prompt, provider="claude-code"|"codex"|"gemini"|"opencode")`** — delegates to an external coding-agent CLI. Heavy. **Only use when `af doctor` reports `harness_usable: true` AND the Dockerfile installs the CLI AND `shutil.which()` guards startup.** Otherwise use `app.ai(tools=[...])`. +- **`app.harness(prompt, provider="aforge"|"claude-code"|"codex"|"gemini"|"opencode"|"pi"|"omp")`** — delegates to an external coding-agent CLI. `aforge`, AgentField's own harness, is the SDK default when `provider` is omitted. Heavy. **Only use when `af doctor` reports `harness_usable: true` AND the Dockerfile installs the selected CLI AND `shutil.which()` guards startup.** Otherwise use `app.ai(tools=[...])`. Full signatures, schemas, router surface, memory scopes, and the cross-boundary serialization gotcha are in `references/primitives-snapshot.md` (offline-frozen). **Prefer the live `agentfield.ai/llms-full.txt`** when you have a network — it is the source of truth and it does not drift. diff --git a/skills/agentfield/references/primitives-snapshot.md b/skills/agentfield/references/primitives-snapshot.md index e0243c72e..a8314e524 100644 --- a/skills/agentfield/references/primitives-snapshot.md +++ b/skills/agentfield/references/primitives-snapshot.md @@ -16,7 +16,7 @@ This is a minimal cheat sheet of the five primitives so an offline build can sti | `@app.skill()` | Registers a deterministic function (no LLM) | Sort, parse, dedupe, score-with-formula | | `app.ai(...)` | Single call OR multi-turn tool-using LLM call when `tools=` is passed | Classification, routing, structured analysis, stateful tool-using | | `app.call(target, **kwargs)` | Call another reasoner THROUGH the control plane. Returns `dict`. Tracks the workflow DAG | All inter-reasoner traffic | -| `app.harness(prompt, provider=...)` | Delegate to an external coding-agent CLI (claude-code / codex / gemini / opencode) | When you need a real coding agent to write files / run shell | +| `app.harness(prompt, provider=...)` | Delegate to a coding-agent CLI (aforge by default; claude-code / codex / gemini / opencode / pi / omp) | When you need a real coding agent to write files / run shell | --- @@ -157,7 +157,7 @@ Default canonical pattern: `AgentRouter(prefix="", tags=["domain"])`. `prefix="c result = await app.harness( prompt: str, schema: type[BaseModel] | None = None, - provider: "claude-code" | "codex" | "gemini" | "opencode" | None = None, + provider: "aforge" | "claude-code" | "codex" | "gemini" | "opencode" | "pi" | "omp" | None = None, # None -> "aforge" model: str | None = None, max_turns: int | None = None, max_budget_usd: float | None = None, diff --git a/skills/agentfield/references/scaffold-recipe.md b/skills/agentfield/references/scaffold-recipe.md index 83611e289..f1812deed 100644 --- a/skills/agentfield/references/scaffold-recipe.md +++ b/skills/agentfield/references/scaffold-recipe.md @@ -211,7 +211,7 @@ CMD ["python", "main.py"] Build context is the project directory itself (`context: .`), so the same scaffold works whether the project lives in `code/examples/` or standalone at `/tmp/my-build/`. -**Only edit if you need to install a harness CLI** (claude-code, codex, gemini, opencode). See `primitives-snapshot.md` → "Harness availability gate". Otherwise leave it alone. +**Only edit if you need to install a harness CLI** (claude-code, codex, gemini, opencode, pi, omp — `aforge`, the default, ships with `af`). See `primitives-snapshot.md` → "Harness availability gate". Otherwise leave it alone. ---