Add agent MCP server for NeMo Retriever collections

[charlesbluca]

Description

Adds an agent-facing MCP server for local NeMo Retriever installations. The new surface lets MCP-capable agents create and manage collections, ingest local multimedia files from explicit allowed roots, query persistent VDB-backed collections, and rerank normalized evidence hits. The tools return evidence and status only; final answer generation remains with the calling agent model.

This includes:

• neutral Agent MCP models, collection registry, path validation, and evidence normalization
• in-process backend support for local ingestion jobs, query, and rerank
• JSON-safe tool service plus FastMCP server with lifecycle cleanup
• retriever agent-mcp start CLI and package-module help support
• Agent MCP docs and README link

Verification:

• .venv/bin/python -m pytest tests/test_agent_mcp_models.py tests/test_agent_mcp_registry.py tests/test_agent_mcp_paths.py tests/test_agent_mcp_evidence.py tests/test_agent_mcp_backend.py tests/test_agent_mcp_tools.py tests/test_agent_mcp_cli.py -q -> 58 passed
• .venv/bin/python -m pytest tests/test_retriever_queries.py tests/test_ingest_interface.py -q -> 26 passed
• .venv/bin/python -m nemo_retriever agent-mcp --help -> prints Agent MCP help

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.
• If adjusting docker-compose.yaml environment variables have you ensured those are mimicked in the Helm values.yaml file.