[TT-17123] REST API to MCP proxy-driven flow

[andrei-tyk]

Summary

• Switch REST-as-MCP adapter synthesis to be driven by MCP proxy upstream URLs (tyk://<rest-api-id>__mcp-server) instead of source REST API markers.
• Share one SDK-backed synthetic MCP adapter per referenced REST API and allow multiple same-org MCP proxies to target it.
• Replace 1:1 pairing with REST-to-adapter plus REST-to-allowed-proxy-set validation, stamping loop trust with the actual caller proxy.
• Add primitive-aware internal derivation for v1 tool primitives, skip operations without operationId, respect source allow/block/internal visibility, and forward query params through tools/call.
• Keep source REST API specs unchanged for v1; no server.mcp or mcp_exposure source-side public surface is included.

Validation

• GOCACHE=/private/tmp/tyk-go-cache go test -count=1 ./apidef/oas ./internal/mcp/adapter ./internal/mcp/pairing ./internal/mcp
• CI=true GOCACHE=/private/tmp/tyk-go-cache go test -count=1 ./gateway -run 'MCP|JSONRPC|Loop|Pairing|RESTAsMCP'
• Manual local gateway validation with a REST API, MCP proxy, and mcp-remote: tools/list exposed the derived tool and tools/call forwarded query params to the REST upstream.

[probelabs]

This pull request introduces a significant architectural change, refactoring the REST-as-MCP (Model Context Protocol) feature to a proxy-driven flow. Instead of marking a REST API for MCP exposure, an MCP proxy now specifies a special tyk://<rest-api-id>__mcp-server upstream URL. This triggers the gateway to synthesize a shared, in-memory "adapter" API for the targeted REST API. This adapter handles MCP protocol translation, converting tools/call requests into internal, in-process calls to the original REST API's middleware chain.

A key part of this change is a new trust model based on a "pairing index" built at reload time, which maps REST APIs to their allowed MCP proxies. A new middleware, MCPLoopAuthBypass, validates these internal calls against the index, securely bypassing the REST API's standard authentication, which is now handled at the agent-facing MCP proxy layer. The tool catalog is derived dynamically from the source REST API's OpenAPI Specification on each reload.

Files Changed Analysis

This is a substantial feature implementation, adding over 6,000 lines of code across 30 files. The changes are well-documented with new RFC, technical spec, and Jira-like markdown files.

• New Documentation (.md files): Extensive planning and design documentation has been added, outlining the architecture, implementation parts, and technical specifications.
• API Definition (apidef/):
  • apidef/oas/mcp_proxy_derive.go: A large new file containing the core logic for deriving MCP tools from a source REST API's OpenAPI spec, including parameter mapping and schema generation.
  • apidef/api_definitions.go: Extended with helpers like IsPairedMCPAdapterProxy() to identify and manage this new proxy type.
• Gateway Core (gateway/):
  • gateway/mcp_synthesise_adapter.go: Contains the logic for creating the synthetic, in-memory adapter APIs during the gateway's loading process.
  • gateway/mw_mcp_loop_auth_bypass.go: A new middleware that implements the secure authentication bypass for internal loopback calls between the adapter and the REST API.
  • gateway/model_apispec.go: The APISpec struct is significantly extended with fields like IsSyntheticMCPAdapter, SourceRESTAPIID, and DerivedTools to support these in-memory adapters.
  • gateway/api_loader.go: The main API loading function is updated to orchestrate adapter synthesis and rebuild the pairing index on reloads. It also enhances the internal tyk:// routing.
  • gateway/mcp_api.go: The admin API for MCP proxies is updated to validate these new configurations at creation time.
• Testing (_test.go):
  • gateway/mcp_rest_as_mcp_test.go: A comprehensive, 1400+ line test suite has been added to validate the end-to-end functionality of the new architecture.

Architecture & Impact Assessment

• What this PR accomplishes: It decouples REST APIs from their MCP exposure, shifting to a more flexible, consumer-driven model. Multiple MCP proxies can now target a single REST API, each with its own agent-facing policies (authentication, rate limits), while sharing a single, efficient, in-memory adapter.

• Key technical changes introduced:

  1. Proxy-Driven Synthesis: An MCP proxy's upstream.url now triggers the dynamic generation of an internal adapter APISpec.
  2. In-Process Loopback: A tyk:// protocol is used for efficient, in-memory request forwarding from the adapter to the REST API handler, avoiding network overhead.
  3. Secure Pairing Model: A pairing index is built on every reload to create a verifiable trust relationship between a REST API and its consumer proxies. This is enforced at runtime.
  4. Conditional Authentication Bypass: The MCPLoopAuthBypass middleware validates the internal loopback call against the pairing index and injects a temporary session, securely short-circuiting the REST API's authentication chain.

• Affected system components: The change impacts the Gateway's API loader, request pipeline, routing logic, and the MCP Admin API. It introduces a new pattern of dynamically generated, in-memory API definitions.

• Component Relationships:

graph TD
subgraph TykGatewayProcess ["Tyk Gateway Process"]
direction LR
MCP_Proxy["MCP Proxy API
(Operator-Managed)
Handles Agent Auth/RL"] --|Upstream: tyk://...|--> Synthetic_Adapter["Synthetic MCP Adapter
(In-Memory, Internal)
Handles MCP Protocol"]
Synthetic_Adapter -- "Internal Loopback Call
with Trust Descriptor" --> REST_API["REST API
(Operator-Managed)
Handles Business Logic"]
end

  Agent["AI Agent / MCP Client"] -- "tools/call" --> MCP_Proxy
  REST_API -- "HTTP Request" --> UpstreamService["Upstream REST Service"]

  style Synthetic_Adapter fill:#f9f,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5

## Scope Discovery & Context Expansion

This PR establishes a new architectural pattern for service exposure within Tyk. The concept of synthetic, in-memory APIs driven by the configuration of other APIs is a significant evolution.

- The impact extends to API lifecycle management, as seen in `gateway/api.go`, where deleting a REST API is now blocked if it is actively targeted by an MCP proxy, preventing orphaned configurations.
- The `APISpec` struct in `gateway/model_apispec.go` has been substantially updated to accommodate the state and behavior of these synthetic adapters, including their derived tool catalogs and links to their source REST API.
- The internal `tyk://` routing mechanism has been formalized, with `gateway/api_loader.go` now using an `id:` prefix for unambiguous, direct-to-handler dispatch, bypassing fuzzy name lookups.


<details>
  <summary>Metadata</summary>

  - Review Effort: 5 / 5
  - Primary Label: feature


</details>
<!-- visor:section-end id="overview" -->

<!-- visor:thread-end key="TykTechnologies/tyk#8232@60a359f" -->

---

*Powered by [Visor](https://probelabs.com/visor) from [Probelabs](https://probelabs.com)*

*Last updated: 2026-05-21T13:05:26.402Z | Triggered by: pr_updated | Commit: 60a359f*

💡 **TIP:** You can chat with Visor using `/visor ask <your question>`
<!-- /visor-comment-id:visor-thread-overview-TykTechnologies/tyk#8232 -->

[probelabs]

Architecture Issues (1)

Severity	Location	Issue
🟡 Warning	gateway/api_loader.go:802-839	The logic for retrieving an API handler from `gw.apisHandlesByID` is duplicated. The same block of code for loading the handler, type-asserting it to a `*ChainObject`, and returning its `ThisHandler` field appears twice: once inside the new `if strings.HasPrefix(apiNameOrID, "id:")` block (lines 816-826) and again for the fallback fuzzy-find case (lines 833-839). This duplication increases maintenance overhead and could lead to inconsistencies if one path is updated but the other is not. 💡 Suggestion Refactor the function to determine the `targetAPI` first (either by exact ID or fuzzy name search) and then perform the handler lookup and type assertion in a single, shared block of code at the end of the function. This will eliminate the duplicated logic.

\n\n

Architecture Issues (1)

Severity	Location	Issue
🟡 Warning	gateway/api_loader.go:802-839	The logic for retrieving an API handler from `gw.apisHandlesByID` is duplicated. The same block of code for loading the handler, type-asserting it to a `*ChainObject`, and returning its `ThisHandler` field appears twice: once inside the new `if strings.HasPrefix(apiNameOrID, "id:")` block (lines 816-826) and again for the fallback fuzzy-find case (lines 833-839). This duplication increases maintenance overhead and could lead to inconsistencies if one path is updated but the other is not. 💡 Suggestion Refactor the function to determine the `targetAPI` first (either by exact ID or fuzzy name search) and then perform the handler lookup and type assertion in a single, shared block of code at the end of the function. This will eliminate the duplicated logic.

\n\n ### Performance Issues (1)

Severity	Location	Issue
🟡 Warning	gateway/mcp_synthesise_adapter.go:60-296	The process of synthesizing MCP adapters during a gateway reload has a time complexity of O(M*N), where M is the number of REST APIs exposed via MCP and N is the total number of APIs configured on the gateway. The `synthesiseMCPAdapters` function iterates through each of the M REST APIs, and for each one, it calls helpers (`buildAdapterSpecForProxies` -> `deriveMCPAdapterCatalogueForProxies` -> `deriveMCPProxyToolViews` -> `sortedMCPProxyIDsForREST`) that re-iterate through all N APIs to find matching proxies. This can lead to slow reload times on gateways with a large number of APIs. 💡 Suggestion To optimize this, avoid nested loops over the full set of APIs. Before synthesizing adapters, perform a single O(N) pass over all API specs to build a map of `restID -> []*APISpec` that groups all MCP proxies by the REST API they target. Then, when iterating through the M REST APIs that need adapters, you can retrieve their associated proxies from this map in O(1) time. This will reduce the overall complexity of the synthesis process to O(N).

Quality Issues (4)

Severity	Location	Issue
🟡 Warning	apidef/oas/mcp_proxy_derive.go:1	The file `apidef/oas/mcp_proxy_derive.go` has grown to over 1000 lines and contains highly complex logic for deriving MCP tools from an OpenAPI specification, including parameter mapping, schema generation, and applying proxy-specific overrides. Its size and complexity, particularly in functions like `deriveOperationPrimitive` and `BuildMCPToolView`, make it difficult to maintain and a high-risk area for future bugs. 💡 Suggestion Consider refactoring this file into smaller, more focused components. For example, the logic for building the `inputSchema` could be encapsulated in a dedicated `schemaBuilder` struct, and the application of proxy-specific views could be handled by a separate component. This would improve readability, testability, and reduce the cognitive load required to modify this critical logic.
🟡 Warning	gateway/mcp_synthesise_adapter.go:62-65	During gateway reload, if synthesizing an MCP adapter for a REST API fails (e.g., due to an error in `buildAdapterSpecForProxies` or `loadHTTPService`), the error is only logged, and the process continues. This failure is not exposed to the operator through any API status or health check. An operator could deploy an MCP proxy and be unaware that it is non-functional because its underlying synthetic adapter failed to build. 💡 Suggestion Introduce a mechanism to report the status of synthetic adapters. This could be a new field in the `/tyk/apis/{api_id}` response for the MCP proxy, or a dedicated health-check endpoint. This would provide operators with the necessary visibility to diagnose configuration issues.
🟡 Warning	gateway/mw_jsonrpc_rest_as_mcp_policy.go:66-79	The current implementation for filtering `tools/list` responses involves capturing the full response from the SDK handler, unmarshalling it, applying filters, and then re-marshalling it. For APIs with a very large number of tools, this serialize-deserialize-serialize cycle could introduce noticeable latency. 💡 Suggestion For a future optimization, consider passing policy context (like the session's access rights) into the SDK handler. This would allow the handler to generate the correctly filtered list from the start, avoiding the overhead of processing the full, unfiltered list. While the current approach correctly separates concerns, it may not scale efficiently.
🟡 Warning	gateway/mcp_rest_as_mcp_test.go:1	The test file `gateway/mcp_rest_as_mcp_test.go` is over 1400 lines long. While the tests are comprehensive and of high quality, the file's size makes it difficult to navigate and maintain. It covers many different aspects of the new feature, from lifecycle and reloads to authentication bypass and policy enforcement. 💡 Suggestion Consider splitting this file into multiple, more focused test files, such as `mcp_lifecycle_test.go`, `mcp_auth_bypass_test.go`, and `mcp_policy_test.go`. This would improve readability and make it easier for future developers to find and update relevant tests.

---

Powered by Visor from Probelabs

Last updated: 2026-05-21T13:05:10.161Z | Triggered by: pr_updated | Commit: 60a359f

💡 TIP: You can chat with Visor using /visor ask <your question>