Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,28 @@ python -m venv .venv
模型与凭证配置完成后,运行 pytest 修复闭环:

```sh
# 一次性构建无网络运行所需的基础镜像;项目有额外依赖时应基于此镜像预装依赖。
docker build -f docker/pytest-sandbox.Dockerfile \
-t firstcoder-pytest-sandbox:py311 .

.venv/bin/firstcoder pytest-fix \
--project /path/to/python-project \
--test-command "python -m pytest -q" \
--execution-backend docker \
--json-out runs/pytest-fix-result.json
```

`pytest-fix` 默认最多执行两次修复尝试,不重试 Provider 请求。Qdrant 不可用时,Parser 与确定性候选仍会继续工作。
`pytest-fix` 默认使用 Docker,最多执行两次修复尝试,不重试 Provider 请求。Docker 运行参数固定关闭网络、使用只读基础文件系统、丢弃 capabilities、启用 `no-new-privileges`,并限制 CPU、内存、pids、单文件大小、输出和时间。目标仓库必须是干净 Git worktree;每次 Attempt 从同一 baseline commit 建立独立 worktree,只有通过 focused 和 full pytest 的候选才会回写原仓库。依赖必须预装进 sandbox 镜像,因为测试阶段没有网络。

仅对可信项目或本地单元测试,可显式使用宿主进程:

```sh
.venv/bin/firstcoder pytest-fix \
--project /path/to/trusted-project \
--execution-backend local
```

Local backend 没有容器级文件系统和网络隔离,CLI 会输出安全警告。写工具则由 Runtime `FreshSourceGuard` 强制要求同路径、未过期、SHA-256 未变化的 `read_token`;读取其他文件或使用 stale token 无法通过写入校验。Qdrant 不可用时,Parser 与确定性候选仍会继续工作。详细威胁模型见 `docs/PYTESTPILOT_SECURITY.md`。

## 离线验证

Expand Down
8 changes: 8 additions & 0 deletions docker/pytest-sandbox.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
FROM python:3.11-slim

RUN python -m pip install --no-cache-dir pytest \
&& groupadd --gid 65532 firstcoder \
&& useradd --uid 65532 --gid 65532 --no-create-home --shell /usr/sbin/nologin firstcoder

WORKDIR /workspace
USER 65532:65532
2 changes: 1 addition & 1 deletion docs/DEEPSEEK_BENCHMARK_AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Streaming 只在最终 `message_completed` usage 提交一次。准确边界是

## 路径级 source-read 与 retrieval policy

Evaluator 按 Transcript 执行顺序提取 `view/read_multi` 与 `edit/write/delete/apply_patch` 的规范化路径。每个既有被修改文件必须在首次修改前被准确读取;新文件、越界路径、未读路径和 stale-read 字段分别输出。测试文件修改和 editable scope 外写入仍直接失败。本轮没有实现运行时 FreshSourceGuard;`stale_read_paths` 预留但尚未做内容 hash 对比
Evaluator 按 Transcript 执行顺序提取 `view/read_multi` 与 `edit/write/delete/apply_patch` 的规范化路径。每个既有被修改文件必须在首次修改前被准确读取;新文件、越界路径、未读路径和 stale-read 字段分别输出。测试文件修改和 editable scope 外写入仍直接失败。该次历史 Benchmark 尚未实现运行时 FreshSourceGuard;2026-07-25 后的 `pytest-fix` 已增加基于 path、SHA-256、TTL 和一次性 read token 的 Runtime 强制校验,旧结果不追溯重标

Baseline 不注册 `code_search`。Vector 的 retrieval-required 任务注册该 Tool,并通过通用首 Tool 约束在第一次请求强制选择它;这段逻辑位于 Benchmark Provider 装饰层,不修改 AgentLoop,也不包含 DeepSeek 分支。向量 preview 不算 source read,必须再次 `view/read_multi`。

Expand Down
2 changes: 1 addition & 1 deletion docs/MVP_GOAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@

- 新的完整 Trace 子系统;
- `archive_read`(复用已有 `retrieve_archive`);
- FreshSourceGuard;
- 专用 TUI 面板;
- 14 个 Benchmark;
- 多 Agent;
Expand Down Expand Up @@ -112,3 +111,4 @@
- 两个语义任务的受控小样本中,Baseline 6/6、合规 Vector 6/6;Vector 实际调用 `code_search` 6 次并重新读取候选。两组通过率相同,不能声称检索提升准确率;Vector 平均 Token、Tool Call 与耗时更高。
- 初始 12 项中有两个 Vector 运行测试虽通过但未调用 `code_search`,被标记并排除;策略门加固后仅补跑这两项并通过。本轮 Smoke、12 项及 2 个补跑累计保守成本 `$0.08710954`,低于 `$0.25`,无 usage 缺失或预算中止。
- 加固后完整测试:`.venv/bin/python -m pytest tests -q` 得到 891 passed、2 skipped、0 failed(37.61 秒);文档完成后仍需执行最终一次完整验证。
- 2026-07-25 根据生产安全审计扩展原 MVP 范围:`pytest-fix` 增加 Docker Sandbox Backend、Runtime `FreshSourceGuard` 和逐 Attempt Git worktree 隔离;Docker 成为 CLI 默认,Local backend 仅用于可信项目与单元测试。
44 changes: 44 additions & 0 deletions docs/PYTESTPILOT_SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# PytestPilot 执行安全与 Attempt 隔离

## 执行边界

`ExecutionBackend` 接收参数数组、工作目录和 `ResourceLimits`。当前实现:

- `LocalProcessBackend`:仅用于可信项目与单元测试;使用环境白名单、进程组超时、输出上限以及平台支持的 rlimit,但不提供网络或宿主文件系统隔离。
- `DockerSandboxBackend`:用于不可信仓库;关闭网络,根文件系统只读,worktree 单独以可写 bind mount 暴露,使用非 root 用户,丢弃 Linux capabilities,启用 `no-new-privileges`,限制 CPU、内存、pids、单文件大小、tmpfs、输出与墙钟时间。

Docker 使用 `--pull=never`,运行时不会联网拉取镜像。先构建 `docker/pytest-sandbox.Dockerfile`;若目标项目依赖 pytest 之外的第三方包,应制作预装且版本锁定的派生镜像。当前实现限制单文件大小,但 bind mount 的总磁盘配额仍由宿主文件系统或 Docker 运行环境负责。

相同 Backend 同时用于 Workflow 的 baseline/focused/full pytest,以及 Agent 的 `diagnostics`、`shell` 和 `python_exec`,避免验证命令被隔离但 Agent 内部命令仍在宿主机执行。

## FreshSourceGuard

启用 Guard 的读取返回:

```text
path + sha256 + size + read_at + read_token
```

已有文件的 `edit`、`write`、`delete` 和 `apply_patch` 在 mutation 前强制校验:

1. token 存在且属于同一路径;
2. 路径位于 `editable_paths`;
3. 当前内容 SHA-256 与 size 等于读取版本;
4. token 未超过 TTL;
5. 成功写入后 token 立即消费,不能重放。

新文件不需要伪造读取,但路径必须提前列入 `editable_paths`。Guard 在 Tool executor 内校验,失败时不会写文件;Transcript 审计继续保留为二次证据,不再是主要执行约束。

## Attempt 隔离

运行前要求用户 Git worktree 干净。每轮从同一个 `base_commit` 创建 detached Git worktree:

```text
baseline commit
├── attempt-1 worktree → patch + focused/full evidence
└── attempt-2 worktree → baseline + 显式的 attempt-1 patch/失败证据
```

第一轮文件状态不会隐式进入第二轮。只有 focused 和 full pytest 都通过的 Attempt 标记为 `selected`,随后仅将 `editable_paths` 回写原仓库;所有 Attempt 失败时原仓库保持不变。非 Git 目录只为本地 fixture 兼容,会先复制到临时 Git snapshot,再以相同 worktree 流程执行。

每轮结果记录 `base_commit`、`attempt_patch`、focused/full 结果、`introduced_failures` 和 `selected`。
4 changes: 4 additions & 0 deletions docs/PYTEST_FIX_DEMO_RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,13 @@ Vector 六个合规运行均有一次 `code_search`,且候选随后被 `read_m
只有用户明确确认凭证、额度和预算后才运行真实模型。命令会使用现有 Provider 配置,不应复制或打印 API Key;DeepSeek 审计实验使用 `benchmark.deepseek_paired` 的请求预算入口,不使用未包装的默认 Runner:

```sh
docker build -f docker/pytest-sandbox.Dockerfile \
-t firstcoder-pytest-sandbox:py311 .

.venv/bin/firstcoder pytest-fix \
--project /path/to/python-repo \
--test-command "python -m pytest -q --tb=short" \
--execution-backend docker \
--json-out runs/pytest-fix-result.json

.venv/bin/python -m benchmark.deepseek_paired \
Expand Down
35 changes: 35 additions & 0 deletions firstcoder/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,19 @@ def build_parser() -> argparse.ArgumentParser:
fix_parser.add_argument("--failure-log", default=None)
fix_parser.add_argument("--max-attempts", type=_positive_int, default=2)
fix_parser.add_argument("--json-out", default="runs/pytest-fix-result.json")
fix_parser.add_argument(
"--execution-backend",
choices=("docker", "local"),
default="docker",
help="Use Docker for untrusted projects; local is trusted-only.",
)
fix_parser.add_argument("--docker-image", default="firstcoder-pytest-sandbox:py311")
fix_parser.add_argument("--timeout-seconds", type=_positive_int, default=300)
fix_parser.add_argument("--cpu-count", type=float, default=1.0)
fix_parser.add_argument("--memory-mb", type=_positive_int, default=1024)
fix_parser.add_argument("--pids-limit", type=_positive_int, default=128)
fix_parser.add_argument("--max-output-chars", type=_positive_int, default=100000)
fix_parser.add_argument("--max-file-size-mb", type=_positive_int, default=64)

parser.add_argument("--project", default=".", help="Project root for tools and AGENTS.md.")
parser.add_argument("--data-root", default=None, help="Directory for FirstCoder session data.")
Expand Down Expand Up @@ -277,6 +290,7 @@ def run_index_command(args: argparse.Namespace) -> int:


def run_pytest_fix_command(args: argparse.Namespace) -> int:
from firstcoder.execution import DockerSandboxBackend, LocalProcessBackend, ResourceLimits
from firstcoder.retrieval import (
FastEmbedProvider,
QdrantLocalVectorStore,
Expand Down Expand Up @@ -323,11 +337,32 @@ def run_pytest_fix_command(args: argparse.Namespace) -> int:
extra_tools=extra_tools,
)
failure_log = Path(args.failure_log).read_text(encoding="utf-8") if args.failure_log else None
limits = ResourceLimits(
timeout_seconds=args.timeout_seconds,
cpu_count=args.cpu_count,
memory_mb=args.memory_mb,
pids=args.pids_limit,
max_output_chars=args.max_output_chars,
max_file_size_mb=args.max_file_size_mb,
)
if args.execution_backend == "docker":
uid = os.getuid() if hasattr(os, "getuid") and os.getuid() != 0 else 65532
gid = os.getgid() if hasattr(os, "getgid") and os.getgid() != 0 else 65532
execution_backend = DockerSandboxBackend(image=args.docker_image, uid=uid, gid=gid)
else:
execution_backend = LocalProcessBackend()
print(
"warning: --execution-backend local runs project code with host permissions; "
"use it only for trusted repositories",
file=sys.stderr,
)
try:
result = PytestFixWorkflow(
project,
adapter=adapter,
semantic_search=semantic_search,
execution_backend=execution_backend,
resource_limits=limits,
max_attempts=args.max_attempts,
).run(
test_command=args.test_command,
Expand Down
12 changes: 12 additions & 0 deletions firstcoder/eval/adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
from firstcoder.providers.factory import create_provider
from firstcoder.providers.types import ChatRequest, ChatResponse, ChatStreamEvent, ToolChoiceFunction
from firstcoder.tools.builtin import create_builtin_registry
from firstcoder.tools.fresh_source import FreshSourceGuard
from firstcoder.tools.types import Tool
from firstcoder.utils.sandbox_access import SandboxAccess

Expand Down Expand Up @@ -115,13 +116,22 @@ def _session_root_for_task(self, task: CodingTask) -> Path:

def _create_loop(self, task: CodingTask, session_root: Path) -> AgentLoop:
sandbox_access = SandboxAccess()
fresh_source_guard = None
if task.metadata.get("enforce_fresh_source_guard"):
fresh_source_guard = FreshSourceGuard(
task.repo_path,
editable_paths=task.metadata.get("editable_paths") or (),
)
registry = create_builtin_registry(
task.repo_path,
include_mutation_tools=True,
include_execution_tools=True,
include_network_tools=False,
include_interactive_tools=False,
access=sandbox_access,
fresh_source_guard=fresh_source_guard,
execution_backend=task.metadata.get("execution_backend"),
resource_limits=task.metadata.get("resource_limits"),
)
task_tools = [*self.extra_tools]
if self.extra_tools_factory is not None:
Expand Down Expand Up @@ -298,6 +308,8 @@ def _build_task_prompt(task: CodingTask) -> str:
"Use the diagnostics tool for pytest so it runs with FirstCoder's active Python environment. "
"Start with stack-trace paths, the failing test module, exact symbols, and grep. "
"Before changing an existing file, read that exact file with view or read_multi. "
"When a source read returns a read_token, pass that token to edit/write/delete; "
"for apply_patch pass a read_tokens mapping keyed by every existing path. "
"When this task is marked retrieval_required and code_search is available, call code_search before the first "
"mutation, then read at least one returned candidate with view or read_multi. If code_search is unavailable, "
"continue with deterministic grep/glob/view tools without asking the user. Never access /workspace. "
Expand Down
Loading
Loading