Fix literal "None" output when charset detection fails in PlainTextConverter and CsvConverter

[sohumt123]

Description

When no charset hint is available, PlainTextConverter and CsvConverter decode file content via:

str(from_bytes(file_stream.read()).best())

charset_normalizer.from_bytes(...).best() is Optional[CharsetMatch] — it returns None for byte sequences it cannot classify (e.g. a corrupt/truncated UTF-16 file such as b"\xff\xfe\xff\xff\x00"). Wrapping that in str() turns the entire document into the 4-character string None, reported as a successful conversion:

• .txt → markdown output 'None'
• .csv → the bogus table '| None |\n| --- |'

No error or warning is raised, so this is silent data corruption. The magika pre-guess doesn't prevent it: for such content magika returns unknown, so the extension-based guess (with charset=None) reaches the converter and takes the from_bytes fallback path.

Repro on current main:

import io
from markitdown import MarkItDown

md = MarkItDown()
data = b"\xff\xfe\xff\xff\x00"
print(repr(md.convert_stream(io.BytesIO(data), file_extension=".txt").markdown))  # 'None'
print(repr(md.convert_stream(io.BytesIO(data), file_extension=".csv").markdown))  # '| None |\n| --- |'

Fix

In both converters, capture the best() result and guard it: if it is None, fall back to data.decode("utf-8", errors="replace"), which preserves whatever is decodable and never fabricates content; otherwise keep the existing str(best_guess) behavior. All existing behavior for detectable inputs is unchanged.

Testing

• Added a regression test (test_undetectable_charset_does_not_become_none) that converts the undetectable byte sequence through both the .txt and .csv paths; it fails without the fix and passes with it.
• Full suite: cd packages/markitdown; hatch test → 333 passed, 4 skipped.

Related work

Open PRs #1938 / #1954 / #1956 / #2015 / #1540 address a different failure mode (a wrong charset hint raising UnicodeDecodeError) and all keep the unguarded str(from_bytes(...).best()), so this bug persists with any of them; happy to rebase if one of those lands first.

[sohumt123]

@microsoft-github-policy-service agree