Skip to content

security.md

Latest commit

 

History

History
75 lines (57 loc) · 7.12 KB

security.md

File metadata and controls

75 lines (57 loc) · 7.12 KB
Raw

Security

This page is for someone deciding whether streamcontext is safe to point at their data. It covers the threat model the project is designed against and what the three processes do and do not protect.

Threat model in plain language

streamcontext assumes:

  • The ingestion gateway runs on infrastructure the operator controls, on a network where the Kafka cluster and Qdrant are reachable but not directly exposed to the internet.
  • The MCP server runs alongside the operator's own agent host (Claude Desktop, Cursor, Cline, or a custom local agent). It is launched and torn down by that host. The connection between agent and MCP server is stdio (default) or SSE on loopback (127.0.0.1).
  • The catalog refresher runs as a job on the same trust boundary as the gateway. It needs read access to Kafka, Schema Registry, and Qdrant, plus write access to the catalog SQLite file the MCP server reads.
  • The operator is the one configuring which Kafka topics the gateway ingests and which the MCP server exposes. They are not multi-tenant; one set of topics, one agent surface.
  • Producers writing to Kafka are inside the operator's trust boundary in the sense that they could already corrupt the stream. streamcontext does not attempt to detect or quarantine malicious producers - it does protect downstream agents from the fallout (see Payload redaction below).

Still out of scope: multi-tenant gateway deployments, internet-exposed MCP servers, and fine-grained per-agent access control. Kafka SASL/SSL, Schema Registry basic-auth and TLS, and a static bearer token on the SSE transport are configurable (opt-in — see the recommended settings below); the shipped defaults stay local/trusted-host. The MCP server also ships an authorize hook (build_server(authorize=...)) for custom per-caller checks.

What the gateway protects against

  1. Silent data loss in the ingestion path. If embedding or Qdrant upsert fails persistently, the pipeline raises PipelineFatalError and exits non-zero. Offsets stay unadvanced; a supervisor (compose restart, systemd, k8s) catches the exit and a clean restart resumes from the last durable batch.
  2. Vector-store dimension drift. The embedder is the source of truth for vector dim; the gateway raises ConfigurationError (exit 78) at startup if SC_QDRANT_VECTOR_DIM does not match. No mid-batch crashes.
  3. Sensitive fields reaching the agent. SC_PAYLOAD_REDACT_FIELDS strips named fields recursively from every record value before it is written to Qdrant; SC_PAYLOAD_INCLUDE_HEADERS=false (default) prevents headers (which routinely carry auth tokens and trace context) from being stored at all.

What the MCP server protects against

  1. Cross-topic data leak. SC_MCP_TOPIC_ALLOWLIST is enforced at three points: explicit off-allowlist topic arguments are rewritten to a sentinel filter that cannot match; off-allowlist find_similar_events references return not_found indistinguishable from a real miss; describe_topic on a restricted name returns an empty zero-count description without ever hitting the store.
  2. Injection into Qdrant filters. Every filter clause is a typed FieldCondition; the FilterClause Pydantic model has extra="forbid" and bounded value types. No string interpolation reaches Qdrant.
  3. Excessive embedding spend and runaway agent loops. Per-tool token-bucket rate limiting (SC_MCP_RATE_LIMIT_PER_MINUTE, default 120) combined with an LRU query cache (SC_MCP_EMBED_CACHE_SIZE, default 256). Worst-case per-minute embedding calls are bounded by rate_limit * (1 - cache_hit_rate).
  4. Hung tools. Every tool body is wrapped in asyncio.wait_for(..., timeout=SC_MCP_TOOL_TIMEOUT_SEC) (default 5s). Timeouts return ToolError(code="timeout") rather than hanging the agent UI.
  5. Context-window blowout from a single record. Search results whose value JSON exceeds SC_MCP_MAX_VALUE_BYTES (default 8192) are replaced with a _truncated stub. EventResult.value_truncated is the structured signal for the agent.
  6. Auditability. Every tool invocation, every rate-limit denial, every timeout, and every internal error is logged with structured fields (tool name, key arguments, result count). An operator can replay an agent session entirely from the log stream.

Recommended configuration for production-adjacent use

The defaults are tuned for the laptop/local-stack case. For a deployment with real data:

# Gateway (ingestion process)
SC_PAYLOAD_REDACT_FIELDS=email,phone,card_number,ssn,authorization
SC_PAYLOAD_INCLUDE_HEADERS=false
SC_PAYLOAD_INDEX_FIELDS=status,region,channel        # whatever your queries actually filter on
SC_HOST_BIND=127.0.0.1                                # default; only override with explicit reason
SC_KAFKA_SECURITY_PROTOCOL=SASL_SSL                   # real clusters; PLAINTEXT is local/dev only
SC_KAFKA_SASL_MECHANISM=SCRAM-SHA-256
SC_KAFKA_SASL_USERNAME=streamcontext
SC_KAFKA_SASL_PASSWORD=...
SC_SCHEMA_REGISTRY_USER=streamcontext                 # if the registry requires basic auth
SC_SCHEMA_REGISTRY_PASSWORD=...

# MCP server (read process)
SC_MCP_TOPIC_ALLOWLIST=orders,clicks                  # explicit subset, never empty in production
SC_MCP_MAX_RESULTS=50                                  # tighten from default 100
SC_MCP_MAX_TIME_RANGE_MINUTES=1440                     # 24h is usually plenty for ad-hoc queries
SC_MCP_TOOL_TIMEOUT_SEC=5
SC_MCP_RATE_LIMIT_PER_MINUTE=60                        # per tool; tighten further for paid embedders
SC_MCP_EMBED_CACHE_SIZE=512                            # higher for chat-style usage
SC_MCP_MAX_VALUE_BYTES=4096
SC_MCP_MAX_CONCURRENT_CALLS=8                          # bound simultaneous in-flight tool calls
SC_MCP_SSE_AUTH_TOKEN=...                              # required if you expose the SSE transport

# Catalog refresher (third process)
SC_CATALOG_DB_PATH=/var/lib/streamcontext/catalog.sqlite
SC_CATALOG_LLM_PROVIDER=anthropic                      # or 'openai' / 'local'; 'disabled' to skip LLM inference entirely
SC_CATALOG_LLM_MODEL=claude-haiku-4-5-20251001
SC_CATALOG_LLM_DAILY_CEILING_USD=1.0                   # hard daily cap, persisted in SQLite ledger
SC_CATALOG_PII_FIELDS=email,phone,card_number,ssn,authorization
SC_CATALOG_RETAIN_SAMPLES=true                          # set false to keep only metadata in SQLite

Redaction is configured on two independent surfaces and both matter: SC_PAYLOAD_REDACT_FIELDS governs the record values returned by search_events and find_similar_events, while SC_CATALOG_PII_FIELDS / SC_CATALOG_PII_PATTERNS govern the samples surfaced by describe_topic and explain_field. Setting one but not the other leaves the other tool's output unredacted.

If you are using a paid embedding provider (OpenAI, Cohere, Voyage) the rate limit and the cache are how you bound your bill. The rate limit gives you the worst case; the cache gives you the typical case. Catalog LLM spend is bounded separately by SC_CATALOG_LLM_DAILY_CEILING_USD.

Reporting a vulnerability

If you find a security issue, please file a private GitHub security advisory rather than opening a public issue. Public issues for non-sensitive bugs and feature requests are welcome as usual.