Windows: CBM_CACHE_DIR / CBM_ALLOWED_ROOT are lossily decoded — every non-ASCII char becomes U+FFFD, so the server silently uses a different path (v0.9.0)

[sowa-content]

Windows: CBM_CACHE_DIR / CBM_ALLOWED_ROOT are lossily decoded — every non-ASCII char becomes U+FFFD, so the server silently uses a different path (v0.9.0)

Summary

On Windows v0.9.0, the values of CBM_CACHE_DIR and CBM_ALLOWED_ROOT are decoded
lossily: each non-ASCII character is replaced with U+FFFD (REPLACEMENT CHARACTER).
The server then operates on that mangled path instead of the one you set.

One defect, three very different-looking symptoms:

Symptom	Underlying cause
config set reports success but nothing persists	DB written to the U+FFFD twin directory
error: cannot open config database	mangled path is not creatable (e.g. under C:\Users\)
every index_repository fails, even in-root	mangled allowed-root never matches the real repo path

CLI arguments are not affected — --repo-path with the same characters works
correctly and encodes them properly into the project name (ö → c3b6). The bug is
specific to reading these environment variables.

This is distinct from #903 / #571, which concern the repository path (there indexing
succeeds and only retrieval/naming degrades).

Environment

• codebase-memory-mcp v0.9.0, codebase-memory-mcp-windows-amd64.zip (standard variant)
• Windows 11 Pro 10.0.26200, AMD64
• Reproduced identically from both PowerShell 5.1 and Git Bash — not a shell-quoting artifact

Proof: the tool creates a mojibake twin directory

Set the cache dir to a path with non-ASCII characters, then look at what appeared on disk:

$env:CBM_CACHE_DIR = "C:\Tëst-Ünïcode\cache"
codebase-memory-mcp config set auto_watch false     # → "auto_watch = false"
codebase-memory-mcp config list                     # → auto_watch = false

Both commands report success. But C:\Tëst-Ünïcode\cache is never created. Instead:

Get-ChildItem C:\ -Directory | Where-Object Name -like "T*st*code" |
  ForEach-Object { ($_.Name.ToCharArray() | % { "U+{0:X4}" -f [int]$_ }) -join " " }

# the directory I created:
Tëst-Ünïcode  ->  U+0054 U+00EB U+0073 U+0074 U+002D U+00DC U+006E U+00EF U+0063 U+006F U+0064 U+0065
# the directory the tool created:
T?st-?n?code  ->  U+0054 U+FFFD U+0073 U+0074 U+002D U+FFFD U+006E U+FFFD U+0063 U+006F U+0064 U+0065

Every non-ASCII codepoint has been replaced by U+FFFD. _config.db lands in the twin
directory, so settings appear to save and read back correctly while the configured
location stays empty.

Symptom 2: hard failure when the mangled path is not creatable

$env:CBM_CACHE_DIR = "C:\Users\SomeÜser\cache"   # any profile with a non-ASCII name
codebase-memory-mcp config list
# → error: cannot open config database

Here the twin path would be C:\Users\SomeU+FFFDser\cache, and creating a new directory
directly under C:\Users\ is not permitted, so it fails outright instead of silently
diverging. Same physical directory reached through its ASCII 8.3 alias works fine:

$env:CBM_CACHE_DIR = "C:\Users\SOMEUS~1\cache"
codebase-memory-mcp config list   # → prints config, _config.db created

This makes config get/set/list unusable on such an account — including turning
auto_watch / auto_index off.

Symptom 3: CBM_ALLOWED_ROOT rejects everything, including valid in-root paths

$env:CBM_CACHE_DIR    = "C:\cbm-data"          # ASCII, fine
$env:CBM_ALLOWED_ROOT = "C:\Tëst-Ünïcode"
codebase-memory-mcp cli index_repository --repo-path "C:\Tëst-Ünïcode\repro"
# → {"status":"error","outcome":"exit_nonzero",
#    "hint":"Indexing worker crashed on a file. The crash was contained ..."}

The repo is inside the allowed root and must be accepted. The mangled root can never
match the real path, so the guard refuses everything. Control — same repo, variable unset:

Remove-Item Env:\CBM_ALLOWED_ROOT
codebase-memory-mcp cli index_repository --repo-path "C:\Tëst-Ünïcode\repro"
# → {"project":"C-Tc3abst-c39cnc3afcode-repro","nodes":7,"edges":7,"status":"indexed"}

Note the project name here encodes the characters correctly (ë→c3ab, Ü→c39c,
ï→c3af) — further evidence that only the env-var read path is broken.

Impact

Any Windows account whose name contains a non-ASCII character — common in German,
French, Nordic, Slavic, Turkish and CJK locales — cannot use CBM_ALLOWED_ROOT at all,
and the default cache dir under $HOME is unusable.

CBM_ALLOWED_ROOT is the documented isolation control for "agentic or multi-tenant
deployments". Losing it is bad; losing it silently is worse. Note the two failure
modes are asymmetric in danger: the hard error is loud, but the U+FFFD-twin case looks
like everything worked.

Diagnostics point away from the cause

• cannot open config database reads like a permissions or file-lock problem.
• Indexing worker crashed on a file. ... a future release isolates the culprit file
  actively blames a source file. No file is at fault, and the worker log written to
  <cache>/logs/.worker-<pid>.log is 0 bytes, so there is nothing to debug from.

Independent of the encoding fix, two cheap improvements would have saved hours here:

1. Validate env-var paths at startup and fail with a message naming the variable and the
   resolved path. If the resolved path differs from the input, that alone is the bug.
2. Make the worker actually write its failure reason into its log file.

Workaround

Point every path at an ASCII location; a junction avoids moving anything:

New-Item -ItemType Junction -Path C:\myrepo -Target C:\Users\SomeÜser\myrepo
$env:CBM_ALLOWED_ROOT = "C:\myrepo"
$env:CBM_CACHE_DIR    = "C:\cbm-data"

Also keep the binary itself on an ASCII path: when it lives under a non-ASCII directory
the supervisor cannot spawn workers and logs
index.supervisor.spawn_failed action=degrade_in_process, silently degrading to
in-process indexing.

Related: #903 (non-ASCII repo path → retrieval fails), #571 (project name strips non-ASCII).

[DeusData]

Thanks for the unusually thorough byte-level diagnosis and controls. This is a high-priority 0.9.1-rc Windows correctness and containment bug under #394. The RED coverage should read both environment variables through the Windows wide-character path, prove the exact non-ASCII path reaches cache/config operations, prove the allowed-root comparison uses the same canonical identity as the repository path, and verify the supervisor can spawn from a non-ASCII executable path. The zero-byte worker diagnostic is also relevant to #1132, but does not replace these encoding-specific guards.

[sowa-content]

Thanks — that coverage list matches what I saw, with one gap worth adding.

The zero-config case is affected too. All my repros set the variables explicitly, but the default cache path derived from $HOME hits the same decode. With both variables unset on a profile whose name contains a non-ASCII character:

codebase-memory-mcp config list
# → error: cannot open config database

~/.cache/codebase-memory-mcp exists and stays empty — nothing is ever written there. So a fresh install on such an account is broken before the user configures anything; CBM_CACHE_DIR was how I worked around it, not how I hit it. Suggest the RED coverage include a no-env-vars case with a non-ASCII $HOME, since that is the path most users will actually take.

The asymmetry between the two failure modes may also be worth encoding as two separate assertions: when the mangled path happens to be creatable, the tool reports success and writes to a U+FFFD twin directory (silent), and only fails loudly when the twin cannot be created — e.g. a new directory directly under C:\Users\. A test that only asserts "returns an error" would pass while the silent-divergence case still ships.

Happy to run a 0.9.1-rc build against a real non-ASCII Windows profile and report back — that is the environment I have.