Add Exgentic Benchmarks MCP Server

[yoavkatz]

Summary

This PR adds a Docker-based MCP server for Exgentic benchmarks.

For: kagenti/kagenti#963

as part of Epic: kagenti/kagenti#962

Changes

• Created directory with:
  • Dockerfile: Multi-stage build using uv for fast package installation
  • entrypoint.sh: Runtime script to start the MCP server
  • README.md: Comprehensive documentation with examples
  • .dockerignore: Build optimization

Features

• ✅ Supports building images with specific benchmarks (e.g., tau2, webarena, miniwob)
• ✅ Uses uv for fast package installation
• ✅ Includes git configuration to handle large repository clones (HTTP/1.1, increased buffers)
• ✅ Benchmark is installed at build time for faster startup
• ✅ Configurable HOST/PORT via environment variables
• ✅ Runs as non-root user (UID 1001)
• ✅ Successfully tested with tau2 benchmark (114 tasks loaded)

Testing

Built and tested the Docker image:

• Image size: 2.43 GB
• Successfully loaded tau2 benchmark with 114 tasks
• MCP server started and responded correctly to HTTP requests
• 3 management tools available (list_tasks, create_session, delete_session)

Usage

# Build for tau2 benchmark
cd mcp/exgentic_benchmarks
podman build --build-arg BENCHMARK_NAME=tau2 -t exgentic-mcp-tau2 .

# Run the server
podman run -p 8000:8000 exgentic-mcp-tau2

Documentation

See mcp/exgentic_benchmarks/README.md for complete documentation including:

• Quick start guide
• Build instructions for different benchmarks
• Configuration options
• Troubleshooting guide
• Advanced usage examples

[esnible]

Can you add, either to the README, or to this PR, the rationale for adding this to the Kagenti examples? Have you been testing this with Kagenti -- the README only talks about testing with Docker. The instructions talk about different builds but Kagenti builds only once -- do we need those instructions?

[yoavkatz]

Can you add, either to the README, or to this PR, the rationale for adding this to the Kagenti examples? Have you been testing this with Kagenti -- the README only talks about testing with Docker. The instructions talk about different builds but Kagenti builds only once -- do we need those instructions?

This PR is related to work. It extends the work done by @kellyaa to add appworld mcp server to encompass in a systematic way multiple benchmarks supported by the Exgentic framework.

I added now the references to the related Kagenti issue (kagenti/kagenti#963) and Epic (kagenti/kagenti#962) for clearer context.