feat: add document-to-markdown MCP tool and streamline system prompt

.codex/review-prompt.md

@@ -9,7 +9,7 @@ Read these files before reviewing:
 1. **`pr-diff.patch`** — The PR diff (generated at runtime). This is the primary input.
 2. **`AGENTS.md`** — Project conventions, Definition of Done, plugin patterns, testing requirements, and code quality standards. This is the source of truth for how code in this project should look.
 
-You may read other files in the repository to understand surrounding context (e.g., how a modified function is called, what a referenced constant is). However, all review comments must target lines that appear in the diff.
+You may read other files in the repository **only** to understand how code changed in the diff is called or referenced. Do not review, comment on, or mention code in files or packages that are not part of the diff. All review comments and the summary must be strictly scoped to changes introduced by this PR's diff — nothing else.
 
 ## Review Philosophy
 
@@ -26,7 +26,7 @@ When both exist, report blockers first.
 
 Do three passes:
 
-1. **Context + risk-map pass (mandatory)** — Read full touched files (not only hunk lines) and identify high-risk zones: platform/runtime boundaries, event handlers, async cleanup (`finally`), auth/validation boundaries, and repeated predicates/guards.
+1. **Context + risk-map pass (mandatory)** — For each file that appears in the diff, read the full file (not only hunk lines) to understand surrounding context. Do NOT read files or packages that have no changed lines in the diff. Identify high-risk zones in the changed code: platform/runtime boundaries, event handlers, async cleanup (`finally`), auth/validation boundaries, and repeated predicates/guards.
 2. **Blockers pass** — Scan for correctness bugs, security issues, API/schema contract breaks, missing migrations, data integrity risks, and missing tests for changed behavior. These are `🔴 Bug` comments.
 3. **Maintainability pass** — Scan for code bloat, readability issues, naming problems, pattern violations, hardcoded values, and architecture drift in touched areas. These are `🟡 Issue`, `🔵 Nit`, or `💡 Suggestion` comments.
 
@@ -53,29 +53,34 @@ If any check fails, skip the comment.
 - Logic errors, off-by-one, null/undefined handling, incorrect assumptions, race conditions.
 - Boundary conditions — empty arrays, null inputs, zero values, maximum values.
 - Error handling — swallowed errors, missing error propagation, unhelpful error messages. Do not flag missing error handling for internal code that cannot reasonably fail.
+- Streaming/multipart handlers — verify a request cannot send multiple responses (e.g., multi-file parts triggering repeated `res.json()` calls). If a route expects one file, ensure parser limits and single-response guards exist.
 - Unsafe runtime assumptions hidden by type assertions (`as ...`, `as any`, non-null `!`) when values come from events, external I/O, or platform-specific APIs.
 - Platform/runtime compatibility assumptions — usage of globals/APIs (`window`, `document`, `Node`, `process`, browser-only APIs) in cross-runtime code paths must be guarded.
 
 #### Security
 
+
 - Injection risks (SQL, command, XSS) when handling user input.
 - Hardcoded secrets — API keys, passwords, tokens in code.
 - Missing input validation at system boundaries (user input, external APIs). Not for internal function calls.
 - Auth bypass, privilege escalation, or missing authorization checks.
+- Filesystem path confinement — when IDs/paths come from requests, verify storage layers enforce root containment via resolved-path checks; do not rely only on caller-side sanitization.
 
 #### API Compatibility
 
 - Breaking changes to API response schemas or status codes without migration path.
 - Removed or renamed API endpoints, query parameters, or response fields that existing consumers depend on.
 - Database schema changes that require migration or backfill.
 - MCP tool signature changes (renamed tools, changed input schemas) that break existing clients.
+- HTTP status semantics — ensure client/input errors are 4xx and unexpected internal failures are 5xx; blanket 400 handling in catch-all paths is a correctness/API contract issue.
 
 #### Tests for Changed Behavior
 
 - New behavior must have corresponding tests covering core functionality and error handling.
 - Bug fixes must include a regression test that would have caught the original bug.
 - Changed behavior must have updated tests reflecting the new expectations.
 - If tests are present but brittle (testing implementation details rather than behavior), flag it.
+- For single-file upload endpoints, look for regression coverage of multi-file/malformed multipart inputs and confirm no double-response behavior.
 - Prefer tests that validate production behavior directly. If a test re-implements production decision logic locally, and could stay green while runtime behavior regresses, flag it and suggest importing shared runtime logic or testing via a higher-level behavior path.
 
 Missing tests for changed behavior are blockers (`🔴 Bug`) only when the change affects user-facing behavior, API contracts, or data integrity. Missing tests for internal refactors or trivial changes are `🟡 Issue`.
@@ -113,6 +118,7 @@ Missing tests for changed behavior are blockers (`🔴 Bug`) only when the chang
 - **Wrong import paths in tests** — Tests importing from `src/` instead of `dist/`.
 - **Missing test categories** — Tests without "Core Functionality" and "Error Handling" describe blocks.
 - **Mixing concerns** — Route handlers doing business logic, database queries in API handlers, etc.
+- **Cross-provider behavior drift** — When multiple providers/implementations exist, verify shared options and output semantics behave consistently unless explicitly documented otherwise.
 
 #### Hardcoded Values and Magic Constants
 
@@ -135,6 +141,7 @@ Do not flag one-off numeric literals that are self-explanatory in context (e.g.,
 - Type annotations for code that already type-checks.
 - Things that are clearly intentional design choices backed by existing patterns.
 - Pre-existing issues in unchanged code outside the diff.
+- Code in files or packages that have no changed lines in the diff — even if you read them for context.
 - Adding documentation unless a public API is clearly undocumented.
 
 ## Comment Format
@@ -174,4 +181,4 @@ The `line` field must refer to the line number in the new version of the file (r
 
 ## Summary
 
-Write a brief (2–4 sentence) overall assessment in the `summary` field. Lead with blockers if any exist. Mention whether the PR is clean/minimal or has code quality issues. Include one sentence on maintainability direction in touched areas (improved / neutral / worsened, and why). If the PR looks good, say so.
+Write a brief (2–4 sentence) overall assessment in the `summary` field covering **only** what this PR's diff changes. Do not mention code, packages, or behavior outside the diff. Lead with blockers if any exist. Mention whether the PR is clean/minimal or has code quality issues. Include one sentence on maintainability direction in touched areas (improved / neutral / worsened, and why). If the PR looks good, say so.

apps/agent/src/server/scripts/setup.ts

@@ -9,8 +9,8 @@ import {
 import {
   getLLMProviderApiKeyEnvName,
   LLMProvider,
-  DEFAULT_SYSTEM_PROMPT,
 } from "@/shared/chat";
+import { DEFAULT_SYSTEM_PROMPT } from "@/shared/prompts/defaultSystemPrompt";
 
 async function setup() {
   const r = await prompts([
@@ -50,6 +50,25 @@ async function setup() {
       initial: DEFAULT_SYSTEM_PROMPT,
       format: (val) => (val === DEFAULT_SYSTEM_PROMPT ? "" : val.trim()),
     },
+    {
+      type: "select",
+      name: "docConversionProvider",
+      message: "Document conversion provider",
+      choices: [
+        { title: "unpdf — basic PDF only", value: "unpdf" },
+        { title: "Mistral OCR — complex PDF/DOCX/PPTX", value: "mistral" },
+      ],
+      initial: 0,
+    },
+    {
+      type: (_, a) =>
+        a.docConversionProvider === "mistral" && a.llmProvider !== "mistralai"
+          ? "text"
+          : null,
+      name: "mistralApiKey",
+      message: "MISTRAL_API_KEY",
+      validate: (val) => val.length || "Required for Mistral OCR provider",
+    },
     {
       type: "select",
       name: "dkgEnv",
@@ -132,8 +151,9 @@ async function setup() {
     {
       type: "text",
       name: "dbFilename",
-      message: "Database filename (i.e: example.db)",
+      message: "Database filename (e.g. example.db)",
       validate: (val) => val.length || "Required",
+      format: (val) => (val.endsWith(".db") ? val : `${val}.db`),
     },
   ]);
 
@@ -158,7 +178,8 @@ SMTP_USER="${r.smtpUsername || ""}"
 SMTP_PASS="${r.smtpPassword || ""}"
 SMTP_SECURE=${r.smtpSecure === undefined ? "true" : r.smtpSecure}
 SMTP_FROM="${r.smtpFrom || ""}"
-`,
+DOCUMENT_CONVERSION_PROVIDER="${r.docConversionProvider}"
+${r.docConversionProvider === "mistral" && r.llmProvider !== "mistralai" ? `MISTRAL_API_KEY="${r.mistralApiKey}"\n` : ""}`,
   );
 
   console.log("Creating .env.development.local file...");

apps/agent/src/shared/chat.ts

@@ -9,6 +9,8 @@ import type {
 import type { ToolCallChunk } from "@langchain/core/messages/tool";
 import type { CallToolResult } from "@modelcontextprotocol/sdk/types.js";
 
+import { DEFAULT_SYSTEM_PROMPT } from "./prompts/defaultSystemPrompt";
+
 export type { ToolDefinition };
 export type ToolInfo = {
   name: string;
@@ -339,7 +341,7 @@ export const processStreamingCompletion = async (
           try {
             args = tc.args ? JSON.parse(tc.args) : {};
           } catch {
-            // Malformed JSON from partial streaming — send raw
+            // Malformed JSON from partial streaming - send raw
             args = {};
           }
           toolCalls.push({
@@ -358,17 +360,17 @@ export const processStreamingCompletion = async (
       writeSSE(res, { event: "done", data: {} });
     } catch (streamError) {
       if (hasSentContent) {
-        // Partial content was already sent — don't re-invoke and risk
+        // Partial content was already sent - don't re-invoke and risk
         // duplicated/mixed output. Send an error so the UI can recover.
         writeSSE(res, {
           event: "error",
           data: {
             message:
-              "Stream interrupted — please retry your message",
+              "Stream interrupted - please retry your message",
           },
         });
       } else {
-        // No content sent yet — safe to fallback to a full invoke
+        // No content sent yet - safe to fallback to a full invoke
         try {
           const result = await provider.invoke(messages, options);
           const content = result.content;
@@ -514,42 +516,9 @@ export const makeStreamingCompletionRequest = async (
 
     // Stream ended without an explicit done/error event (server crash, network drop)
     if (!streamFinalized) {
-      callbacks.onError("Connection lost — the server stopped responding");
+      callbacks.onError("Connection lost - the server stopped responding");
     }
   } finally {
     reader.releaseLock();
   }
 };
-
-export const DEFAULT_SYSTEM_PROMPT = `
-You are a DKG Agent that helps users interact with the OriginTrail Decentralized Knowledge Graph (DKG) using available Model Context Protocol (MCP) tools.
-Your role is to help users create, retrieve, and analyze verifiable knowledge in a friendly, approachable, and knowledgeable way, making the technology accessible to both experts and non-experts. When replying, use markdown (e.g. bold text, bullet points, tables, etc.) and codeblocks where appropriate to convery messages in a more organized and structured manner.
-
-## Core Responsibilities
-- Answer Questions: Retrieve and explain knowledge from the DKG to help users understand and solve problems.
-- Create Knowledge Assets: Assist users in publishing new knowledge assets to the DKG using MCP tools.
-- Perform Analyses: Use DKG data and MCP tools to perform structured analyses, presenting results clearly.
-- Be Helpful and Approachable: Communicate in simple, user-friendly terms. Use analogies and clear explanations where needed, but avoid unnecessary technical jargon unless requested.
-
-## Privacy Rule (IMPORTANT)
-When creating or publishing knowledge assets:
-- If privacy is explicitly specified, follow the user’s instruction.
-- If privacy is NOT specified, ALWAYS set privacy to "private".
-- NEVER default to "public" without explicit user consent.
-This ensures sensitive information is not unintentionally exposed.
-
-## Interaction Guidelines
-1. Clarify intent: When a request is vague, ask polite clarifying questions.
-2. Transparency: If information cannot be verified, clearly state limitations and suggest alternatives.
-3. Explain outcomes: When retrieving or publishing data, explain what happened in simple terms.
-4. Accessibility: Use examples, step-by-step reasoning, or simple metaphors to make complex concepts understandable.
-5. Trustworthy behavior: Always emphasize verifiability and reliability of knowledge retrieved or created.
-
-## Examples of Behavior
-- User asks to publish knowledge without specifying privacy → Agent publishes with "privacy": "private" and explains:
-"I’ve published this knowledge privately so only you (or authorized parties) can access it. If you’d like it public, just let me know."
-
-- User asks to retrieve knowledge → Agent uses MCP retrieval tools and explains results in a simple, structured way.
-
-- User asks a complex analytical question → Agent retrieves relevant knowledge from the DKG, performs the analysis, and presents results in a clear format (e.g., list, table, etc.).
-`.trim();