Hosted and lock-only scans treat a UTF-16 requirements.txt (what Windows PowerShell's pip freeze > writes) as absent: exit 0, no warning, and pip keeps installing the unpatched pin #721

[agent] Found by the scheduled pip / requirements.txt bug-hunt routine (ledger #309).

Summary

Windows PowerShell 5.1 redirects with UTF-16 LE plus a BOM, so pip freeze > requirements.txt there produces a UTF-16 file. pip reads it without complaint: req_file.py auto-decodes UTF-8, UTF-16 and UTF-32 BOMs, and pip 20.3.4 through 26.2.1 all install from it. socket-patch's hosted scan reads requirements.txt only as UTF-8. On a decode failure it drops the file as though it weren't there, with no diagnostic:

• Hosted scan with a venv (six 1.16.0 installed, patch available): exit 0, status: "success", redirect: {redirected: 0, skipped: [], warnings: []}. Human mode prints pkg:pypi/six@1.16.0: no lockfile entry pinning it could be rewritten, which is wrong because the file pins it.
• Lock-only hosted scan (fresh checkout, empty venv): No packages found. Run your package manager's install first. / scannedPackages: 0, exit 0.

The project then installs upstream six with no signal that the file was skipped.

Other commands are loud or correct on the same file. vex warns lockfile_unreadable ("cannot read requirements.txt: stream did not contain valid UTF-8"), and vendored vendor / scan --mode vendored fails with pypi_no_requirements "cannot read …/requirements.txt", exit 1. So only the hosted / discovery path is silent.

Impact

A Windows team whose requirements.txt was generated in PowerShell 5.1 (still the default powershell.exe on Windows 10 and 11) runs socket-patch scan --mode hosted in CI and gets exit 0 with a clean JSON envelope, but nothing is patched. On a fresh checkout it even says there are no packages at all.

Repro

This uses the hosted wiremock from crates/socket-patch-cli/tests/mode_migration_pypi.rs::mount_hosted_api (batch / by-package / package / wheel), with $MOCK as its URI.

python3 -m venv /tmp/v && /tmp/v/bin/pip install six==1.16.0 idna==3.7
mkdir proj && cd proj
python3 -c "open('requirements.txt','wb').write('idna==3.7\r\nsix==1.16.0\r\n'.encode('utf-16'))"   # = PowerShell 5.1 `pip freeze >` output
/tmp/v/bin/pip install --dry-run -r requirements.txt          # pip reads it: would install idna, six

A="--api-url $MOCK --org test-org --api-token fake --patch-server-url $MOCK"
VIRTUAL_ENV=/tmp/v socket-patch scan --mode hosted --yes $A --json      # exit 0, redirected 0, skipped [], warnings []
VIRTUAL_ENV=/tmp/v socket-patch scan --mode hosted --yes $A             # "no lockfile entry pinning it could be rewritten"
mkdir -p ../empty/lib/python3.13/site-packages
VIRTUAL_ENV=$PWD/../empty socket-patch scan --mode hosted --yes $A      # "No packages found. Run your package manager's install first."
socket-patch vex --output vex.json --json                               # (contrast) warnings: lockfile_unreadable

The same file in UTF-8 (identical text, CRLF) is redirected (redirected: 1), and a fresh pip install -r then installs the patched six.

Expected vs actual

• Expected: CLI_CONTRACT.md defines lockfile_unreadable as "A supported file exists but could not be read (permissions, a FIFO squatting the name, non-UTF-8)", and vex already emits it here. For socket.yml, the contract makes UTF-16 an explicit error ("UTF-16 and NUL bytes are errors"). The comment above unconfirmed in scan/hosted.rs says an unpinned granted package must never vanish silently. So hosted scan should either decode the BOM the way pip does, or report the file as unreadable in redirect.warnings[] / skipped[] (and on stderr). Lock-only discovery should say it couldn't read requirements.txt rather than "No packages found".
• Actual: the file is invisible to hosted scan and to lock-only discovery. Exit 0 with a success envelope; the human message blames a missing pin.

Matrix

macOS and Windows weren't probed. The decode is OS-independent; the file shape is what Windows PowerShell 5.1 writes.

First bad version: none. Published v4.0.0 (PyPI socket-patch==4.0.0) behaves the same way: Redirected 0 package(s), and lock-only says "No packages found". It isn't a regression.

Suspect code

• crates/socket-patch-core/src/hosted/engine.rs:324: CandidateFiles::read uses view.read_text(rel).await.ok(), so an InvalidData decode error becomes "file absent". The memory-view branch at :334 says so explicitly ("a non-UTF-8 one is absent to it as well"). Then patch/redirect/requirements.rs::rewrite returns early on files.get("requirements.txt") being None, with no warning.
• crates/socket-patch-core/src/vendor/lock_inventory/pypi.rs:711: requirements_tree does view.read_text(ROOT).await.ok()?, so lock-only discovery drops the root file silently.
• Related but distinct: PyPI hosted scan lacks a specific missing-requirements warning after redirect_unconfirmed reporting #638 (no root requirements.txt → silent JSON). Here the file exists and pip installs from it; a fix for PyPI hosted scan lacks a specific missing-requirements warning after redirect_unconfirmed reporting #638 that only checks file presence would still miss this. Hosted and vendored NuGet reject a packages.lock.json with a UTF-8 BOM that dotnet restores fine: hosted skips the redirect and exits 0 success, vendored fails apply_failed #623 is the NuGet UTF-8-BOM analogue.

Probe runs: none (OS-independent text decoding; Linux only).