Fix vendored npm-family tarballs dropped by .gitignore (#831)

crates/socket-patch-cli/CLI_CONTRACT.md

@@ -651,7 +651,7 @@ to **six flavors**.
 
 | eco / flavor | vendored artifact | committed wiring | consumption proof |
 |---|---|---|---|
-| npm (package-lock) | deterministic patched tarball `[@scope/]<name>-<version>.tgz` | `package-lock.json` only (`npm-shrinkwrap.json` wins when present): every entry matching name+version gets `resolved: "file:…"` + recomputed `integrity`. `package.json` untouched | `npm ci` (integrity-verified). Plain `npm install` preserves the entry; `npm update <pkg>` re-resolves and drops it |
+| npm (package-lock) | deterministic patched tarball `[@scope/]<name>-<version>.tgz`, plus `<uuid>/.gitignore` (re-includes the tarball against the project's ignores, such as Node.gitignore's `*.tgz`) and `<uuid>/.gitattributes` (`-text`); every tarball flavor below writes the same pair and refuses `vendor_artifact_gitignored` when git would still drop the tarball | `package-lock.json` only (`npm-shrinkwrap.json` wins when present): every entry matching name+version gets `resolved: "file:…"` + recomputed `integrity`. `package.json` untouched | `npm ci` (integrity-verified). Plain `npm install` preserves the entry; `npm update <pkg>` re-resolves and drops it |
 | npm / yarn classic | (same tarball) | `yarn.lock` only: matching blocks get `resolved "file:./…#<sha1>"` + `integrity` (both checksums recomputed; merged-key & `npm:`-alias blocks covered) | `yarn install --frozen-lockfile --offline` (sha1 fragment + sha512 SRI both enforced; byte-stable lock) |
 | npm / yarn berry (node-modules linker) | (same tarball) | root `package.json` `resolutions` + `yarn.lock` entry with `checksum: 10c0/<sha512>` of the berry cache-zip (reproduced from the tarball offline). **PnP is refused** (`.pnp.*` → different artifact pipeline) | `yarn install --immutable --check-cache`, cold cache. Refused if `__metadata.cacheKey ≠ 10c0` or a non-default `compressionLevel`. Both files keep their own layout — a CRLF lock (yarn's output on Windows) is spliced in CRLF, `package.json` is re-serialized with its BOM, indent, line ending and trailing-newline shape — so vendor + `--revert` round-trip byte-exactly; a lock or `package.json` MIXING CRLF and LF is refused before any write (`vendor_yarn_berry_mixed_line_endings`) |
 | npm / pnpm (lockfileVersion 9) | (same tarball) | root `package.json` `pnpm.overrides` (versioned selector) **+** `pnpm-lock.yaml` surgery (overrides / importer version / packages `resolution.integrity` / snapshots) | `pnpm install --frozen-lockfile --offline`, cold store (integrity-verified; byte-stable on pnpm 9 & 10). Other lockfileVersions: 5.4/6.0 route to the legacy backend below; anything else refused |
@@ -1191,7 +1191,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified
 | `already_vendored`        | `skipped`        | vendor: artifact + wiring already in sync for this patch uuid. |
 | `unsafe_coordinates`      | `failed`         | vendor: purl/uuid would escape `.socket/vendor/` (tampered manifest/state); refused before any write. |
 | `revert_failed`           | `failed`         | vendor --revert: a recorded entry could not be reverted. |
-| `vendor_ledger_missing` | `failed` (artifact-level: `uuid` + `details.{ecosystem,path}`, no purl) | repair (v5.0): a lockfile references `.socket/vendor/<eco>/<uuid>/` but the vendor ledger has no entry for it; repair no longer rebuilds ledger entries from lockfiles. Recovery: restore `.socket/vendor/state.json` from version control and re-run `repair`, or `git checkout -- <lockfile>` and re-vendor. |
+| `vendor_ledger_missing` | `failed` (artifact-level: `uuid` + `details.{ecosystem,path}`, no purl) | repair (v5.0) and `vendor --check`: a lockfile references `.socket/vendor/<eco>/<uuid>/` but the vendor ledger has no entry for it; repair no longer rebuilds ledger entries from lockfiles. Recovery: restore `.socket/vendor/state.json` from version control and re-run `repair`, or `git checkout -- <lockfile>` and re-vendor. |
 | `vendor_wiring_unknown_revert_blocked` | `skipped` (beside the `failed`/`revert_failed` event) | vendor --revert: the ledger entry was reconstructed by a pre-v5 `repair` without wiring records and the live lockfile still resolves through the artifact — the revert refuses (fail-closed) instead of deleting a tarball the lock points at. Recovery: `socket-patch repair`, then restore the pre-vendor lock (or re-lock without the override) and re-run the revert. repair: an npm ledger entry whose `flavor` this release does not know (written by a newer socket-patch) is skipped, never health-checked or rebuilt, and the artifact, wiring and ledger stay as found (a lone `skipped` event; the run's exit is unaffected). Recovery: upgrade socket-patch. |
 | `stale_install`           | `skipped`        | vex (in-run `scan --mode hosted --vex`): a hosted stale-install probe found positively unpatched installed bytes, so the purl is omitted even under `--vex-no-verify` (see the gem / Python stale-install guards). |
 | `record_unavailable`      | `skipped`        | vex (manifest-less): a lockfile-wired patch has no local record (manifest, this run's hosted records or a pre-v5 redirect ledger, vendor ledger) and none could be fetched — `--offline`, transport error, 404, or a refused (paid) patch. Omitted, never attested from the `socket-patch.vendor.json` marker. |
@@ -1238,8 +1238,8 @@ Every `--json` invocation emits a single JSON object that follows the **unified
 | `vendor_vlt_legacy_lockfile` | `skipped` (warning) | vendor (vlt): an era-A lock (vlt 0.0.0-19 … 1.0.0-rc.8): a `··` default-registry id, or default-registry ids that are URL segments equal to a scalar `options.registry` with no `·npm·` id (era B writes `·npm·` whatever the scalar). vlt 0.0.0-31 … 1.0.0-rc.5 install the vendored lock but fail to reinstall the vendored `file:` dependency if `vlt-lock.json` is deleted and re-created (the other era-A releases reinstall it; the lock does not say which release reads it). The package is still vendored; remedy: upgrade vlt. |
 | `vendor_vlt_reinstall_required` | `skipped` (advisory; human: `Warning: …`) | vendor / scan / get `--mode vendored` (vlt), wet and dry runs, and in-sync reruns: (a) the run rewires an optional dependency, or an importer's `node_modules/<name>` of an optional dependency still resolves into `node_modules/.vlt/`: from vlt 0.0.0-30 a plain `vlt install` (1.2.0: also `--force`) keeps that installed upstream copy linked; the detail says to run `vlt ci` (or delete `node_modules` and run `vlt install`) to link the vendored copy, and that vlt 0.0.0-30 … 1.0.4 install no optional dependency from the lock of a project that declares only optional dependencies (upgrade to 1.0.5 or later first); (b) otherwise, an importer's link of the dependency still resolves into `node_modules/.vlt/`: the detail names the links (`node_modules/<name>`, `<member dir>/node_modules/<name>`) and says `vlt install` (or `vlt ci`) links the vendored copy — on a warm tree after a plain `vlt install` that is true of every vendored direct dependency; (c) an importer's link resolves into the vendored dir of the patch this run replaces (a new patch uuid), which the run removes: the detail names the links and says `vlt install` (or `vlt ci`) links the new vendored copy; (d) a redownload of the payload (vendor, or `repair` after a corrupt or missing payload) could not keep vlt's links to the package's own dependencies (its old `node_modules/` held more than links): the detail says to run `vlt ci` (or delete `node_modules` and run `vlt install`), since a plain `vlt install` does not re-link them. `repair` moves those links back into the downloaded payload when they are only links. The package is vendored either way; a run whose patch fails to apply emits neither. A wet `vendor --revert` (and the revert a vendored → hosted takeover runs, whose advisory joins `redirect.warnings[]`): (a) the revert moves an `optionalDependencies` spec back from the `file:` dir, or an optional importer's `node_modules/<name>` still resolves into the vendored uuid dir: from vlt 0.0.0-30 a plain `vlt install` keeps that link (dangling once the dir is removed), so the detail says to run `vlt ci` (or delete `node_modules` and run `vlt install`) to link the restored copy, with the same vlt 1.0.5 note; (b) otherwise, an importer's link still resolves into the vendored uuid dir: the detail names the links and says `vlt install` (or `vlt ci`) links the restored copy. A dry-run revert emits neither. |
 | `vendor_flavor_changed` | `failed` | vendor (npm): the purl's vendor ledger entry was written for another lockfile `flavor` than the one the router now detects (for example `npm` → `vlt` after switching package managers). Remedy: `socket-patch vendor --revert` it first, then re-vendor. Refused before any write. |
-| `vendor_artifact_gitignored` | `failed` | vendor (vlt): inside a git work tree, `git check-ignore --no-index` reports the new artifact's uuid directory as ignored by a rule its own `.gitignore` cannot override (such as a root `.socket/` rule; the detail names the rule). Remedy: drop that rule for `.socket/vendor/`. Refused before any write. |
-| `vendor_artifact_gitignore_unchecked` | warning | vendor (vlt): git is installed but could not answer the ignore check for the written vendored directory (it failed to start, ran past 30 s, or `rev-parse` / `check-ignore` exited with an error); the package is vendored and the detail names what failed. Remedy: make sure no ignore rule covers `.socket/` before committing. Git absent, or a project outside any work tree, raises nothing. |
+| `vendor_artifact_gitignored` | `failed` | vendor (vlt and the npm-family tarball flavors: npm, pnpm, bun, yarn classic, yarn berry): inside a git work tree, `git check-ignore --no-index` reports the new artifact's uuid directory as ignored by a rule its own `.gitignore` cannot override (such as a root `.socket/` or `vendor/` rule; the detail names the rule). Remedy: drop that rule for `.socket/vendor/`. Refused before any write. A file rule such as `*.tgz` is overridden by the `<uuid>/.gitignore` vendoring writes; if the written artifact still reads as ignored, the run refuses and removes the uuid dir it created. |
+| `vendor_artifact_gitignore_unchecked` | warning | vendor (vlt and the npm-family tarball flavors): git is installed but could not answer the ignore check for the written vendored directory (it failed to start, ran past 30 s, or `rev-parse` / `check-ignore` exited with an error); the package is vendored and the detail names what failed. Remedy: make sure no ignore rule covers `.socket/` before committing. Git absent, or a project outside any work tree, raises nothing. |
 | `vendor_ledger_entry_missing` | `failed` | vendor (vlt): the only installed copy is vlt's link to a committed vendored directory, but the vendor ledger has no entry for the package; restore `.socket/vendor/state.json` from version control (v5.0: `repair` no longer re-synthesizes it). Replaces the `package_not_installed` skip. |
 | `vendor_variant_ambiguous` | `failed` | vendor / scan / get `--mode vendored` (pypi, gem): the package is not installed and the manifest holds several release variants of it (`?artifact_id=` / `?platform=`), none of which the vendor ledger records, so nothing says which distribution to vendor; install the package or keep one release variant. A variant the ledger records (at any patch uuid) is taken as the wired one and its siblings are left out without an event. |
 | `vendor_artifact_missing` | reason | The recorded artifact is missing; repair requires an online exact redownload. |
@@ -1670,7 +1670,11 @@ wiring: an entry whose lockfile or config no longer references its
 `package-lock.json` / `npm-shrinkwrap.json` entry for the vendored `name@version`
 that `vendor` would rewire but that does not resolve to the vendored artifact
 (#588); the reason names that entry. Missing ledger entries fail with
-`vendor_ledger_missing`. Offline upstream metadata is reported as the run warning
+`vendor_ledger_missing`: a manifest patch with no ledger entry, and a lockfile
+reference to `.socket/vendor/<eco>/<uuid>/` that no ledger entry owns (the
+artifact-level event repair emits: `uuid` plus `details.{ecosystem,path}`, no
+purl), so a checkout whose ledger was ignored or dropped from the commit no
+longer passes with nothing to check. Offline upstream metadata is reported as the run warning
 `vendor_jvm_upstream_unverified`. The check never starts an API client or writes
 lock/recovery files. `--check` conflicts with `--revert`.
 

crates/socket-patch-cli/src/commands/vendor.rs

@@ -1060,11 +1060,13 @@ async fn run_check(args: &VendorArgs) -> i32 {
         }
         env.record(event);
     }
-    for key in manifest
+    let mut unledgered_uuids: HashSet<&str> = HashSet::new();
+    for (key, record) in manifest
         .patches
-        .keys()
-        .filter(|k| !state.entries.contains_key(*k))
+        .iter()
+        .filter(|(k, _)| !state.entries.contains_key(*k))
     {
+        unledgered_uuids.insert(record.uuid.as_str());
         if !args.common.json {
             eprintln!("{key}: patch has no vendored ledger entry");
         }
@@ -1073,6 +1075,38 @@ async fn run_check(args: &VendorArgs) -> i32 {
             "patch has no vendored ledger entry",
         ));
     }
+    // A project file still wired to a vendored artifact the ledger does not
+    // know (the ledger was ignored or dropped from the commit along with the
+    // manifest) leaves every fresh install failing; the manifest keys above
+    // cannot see it, so the references are read from the wiring itself.
+    let references =
+        crate::commands::vendored_backend::repair::scan_vendor_references(root).await;
+    for (eco, uuid, rel) in references {
+        let ledgered = state
+            .entries
+            .values()
+            .any(|entry| entry.uuid == uuid && entry.ecosystem == eco);
+        if ledgered || unledgered_uuids.contains(uuid.as_str()) {
+            continue;
+        }
+        // No ledger entry means no purl to name: like repair, the event
+        // carries the uuid and the referenced path instead. The message
+        // names only the ecosystem, keeping patch identifiers out of logs.
+        let detail = format!(
+            "a lockfile references a vendored {eco} artifact under .socket/vendor/{eco}/ but \
+             the vendor ledger (.socket/vendor/state.json) has no entry for it; restore \
+             state.json from version control"
+        );
+        if !args.common.json {
+            eprintln!("vendor_ledger_missing: {detail}");
+        }
+        env.record(
+            PatchEvent::artifact(PatchAction::Failed)
+                .with_uuid(uuid)
+                .with_error("vendor_ledger_missing", detail)
+                .with_details(serde_json::json!({ "ecosystem": eco, "path": rel })),
+        );
+    }
     if args.common.json {
         println!("{}", env.to_pretty_json());
     }
@@ -2627,15 +2661,23 @@ pub(crate) async fn vendor_records_reusing(
                         .clone();
                     let refusal = match project {
                         Some(refusal) => Some(refusal),
-                        None => {
-                            socket_patch_core::vendor::yarn_berry_vendor_target_preflight(
-                                &common.cwd,
-                                candidate,
-                                pin,
-                                &restore_opts,
-                            )
-                            .await
-                        }
+                        None => match socket_patch_core::vendor::npm_tarball_gitignore_preflight(
+                            &common.cwd,
+                            &record.uuid,
+                        )
+                        .await
+                        {
+                            Some(refusal) => Some(refusal),
+                            None => {
+                                socket_patch_core::vendor::yarn_berry_vendor_target_preflight(
+                                    &common.cwd,
+                                    candidate,
+                                    pin,
+                                    &restore_opts,
+                                )
+                                .await
+                            }
+                        },
                     };
                     if let Some((code, detail)) = &refusal {
                         has_errors = true;