Skip to content

platform: combine remaining POSIX, runtime, and SQLite fixes#934

Open
brandonpayton wants to merge 20 commits into
mainfrom
integration/post-907-focused-batch
Open

platform: combine remaining POSIX, runtime, and SQLite fixes#934
brandonpayton wants to merge 20 commits into
mainfrom
integration/post-907-focused-batch

Conversation

@brandonpayton

@brandonpayton brandonpayton commented Jul 13, 2026

Copy link
Copy Markdown
Member

Why

This pull request batches already reviewed fixes so one complete continuous-integration (CI) run can test how they work together before merge. It preserves one commit per purpose, which keeps the history understandable and bisectable.

#907 already landed PHP 8.3.15, ABI 39, and the shared process, VFS,
networking, host-runtime, package, browser-image, Dinit, and Vitest fixes found
during that port. This batch does not replay those changes.

This PR brings the remaining reviewed follow-ups onto one current-main branch
so we can validate and land them together while keeping one commit per purpose.
The focused source PRs remain the review record; their exact heads are listed
below. They were closed only after this batch contained their work and the
combined CI run was green.

What changed

The batch carries the focused changes listed below onto current main. ABI means the application binary interface between programs, the kernel, and the host; VFS means the virtual file system; libc is the C standard library used by guest programs.

Unique work that remains

  • Browser regressions and wiring: stale clearImmediate handles, nonzero-exit
    stderr, and configured CORS proxy routing.
  • VFS and process contracts: positioned SharedFS I/O, canonical rootfs overlay
    assembly, zero-byte copy-back, initial credentials, mremap metadata,
    ENOLCK, and directory fsync.
  • libc and ABI-adjacent tests: zero unreported stat fields, align wasm32
    pthread entry stacks, deterministic fork instrumentation, and SjLj/noexcept
    coverage.
  • Package/build cleanup: shared MariaDB writable-directory setup.
  • SQLite contract work: supported recursion limits, truthful all-mode outcome
    evidence, UTF-16 STAT4 accounting, and TestRecover command lifetime.

Exact source heads

PR Exact reviewed head What this batch keeps
#782 3f133f92bd971cc83fecefcd64709ed7b9f13ade clearImmediate cancellation cleanup
#784 fc25245984f346a5620998f02f93ad9b85dae80b wasm32 pthread stack alignment and direct regression
#786 905dbc0091a40ad9dce04f6ebc3e36e113b40129 positioned SharedFS reads/writes
#875 545048b123d9efabc1474d257329e4e6d154a0f6 Chromium nonzero-exit stderr regression
#880 79dd37dbffa2d373988585f954eb45fb162f7c4c CORS proxy runner wiring, assertion, docs, and integration test
#888 ce70dc0061ce379d2d50872b48ee0b82908eb8bc fresh-process fork output determinism
#896 224aeeb056829c190680e78bc1ac9a4d68398b54 LAMP use of the shared MariaDB helper and dead helper removal
#923 6ad38869d71274e773530a9d0860680292235dd1 shared rootfs overlay helper and strict preservation tests
#924 432857531a6969e9ae7d397c2c59406bfb21214d zero-byte syscall output-buffer preservation
#925 8943e3155d495f9908f8c9ecc2d0e1e06f438805 SjLj/noexcept controls, SIGCHLD fixture, and SDK boundary docs
#926 471dc2335abe77c2c696f7bfc1d0cce529710ad7 requested initial user/group credentials
#927 2a8bfbcd6ae7f7ebb5799a8e2db3c05bb491cf02 metadata-consistent mremap growth
#929 38af72dcaa3c1550f572a63eaa2b7c79c5016218 shared lock-table exhaustion as ENOLCK
#930 c9f9a1a649c5551f02a24ab5487becccabb6068b directory fsync across kernel, host, and browser
#931 c6f4977f3ea2c6bd909e4daaf3f6fbb01c7ca716 zero initialization for unreported stat fields
#780 4b04ff3de6b022549250358f6fe44f02b4475f20 supported SQLite expression/JSON/compound limits
#781 9e3fd3f6d8ae1693076906f6e462ad6e1f8113a3 supported SQLite trigger recursion limit
#783 f758ecb30b92b9c914e950094706f05169bb205e truthful SQLite all-mode runner and durable evidence
#932 52ba9a72036d2bfc631157c737f91ed61093713d UTF-16 STAT4 allocation range regression
#933 a0bc004f569898244f03989558086663d2ba1d78 TestRecover Tcl command/handle lifetime

The batch was assembled from those reviewed commits in a straight line, then
condensed from 30 commits to 20 purpose commits. The #780 and #781 changes are
one SQLite recursion/stack-limit policy commit, and the two #926 commits are
one credentials commit. #783 remains represented by one all-mode runner commit
and one truthful-outcome-evidence commit. The final tree is byte-identical to
the reviewed 30-commit batch head. #931's code and test files are byte-identical
to its source head; only its docs/posix-status.md commit diff changes because
#929 and #930 update the same document earlier in the batch. #891 was
deliberately dropped after it was consolidated into #936; this PR no longer
duplicates that Homebrew workflow rename.

What was deliberately removed

ABI and package-artifact impact

ABI stays at 39. The ABI snapshot and generated C/TypeScript bindings remain in
sync. The wasm32/wasm64 libc artifacts change for stat; wasm32 libc also
changes for pthread stack alignment. SQLite revisions 3 and 4 intentionally
cover the published recursion-limit byte changes. #896, #932, and #933 add no
extra revision because their declared package outputs do not change.

Combined staging must rebuild the affected toolchain and package closure. This
PR makes no performance claim; performance was not measured.

Validation

Exact current head: 822f35ea34cb07e84dac2d24da0505babd16ed7e

  • staging run 29293767304
    ran on pre-compaction head 260d75d5adc31b327f5591229dbfbb287d395fd6
    and passed with 84 successful jobs, two expected skips, and zero failures. This
    includes 14/14 library rebuilds, 57/57 package/image rebuilds, staged binary
    materialization, browser assets, cargo/kernel, fork instrumentation, full
    Vitest, browser, libc, POSIX, Sortix, and the aggregate test gate;
  • browser smoke run 29293767232
    ran on that same pre-compaction head and passed the host check and Chromium,
    Firefox, and WebKit COI smoke coverage;
  • scripts/check-abi-version.sh passed through scripts/dev-shell.sh;
  • host declaration/type generation passed;
  • focused current-main Vitest passed 86/86 SharedFS positioned-I/O, VFS-image,
    lazy-archive, and rootfs-overlay tests;
  • the current 20-commit tree is still exactly
    b3a9a9da574ec7b276e7f61e11f004e578377afa, byte-identical to both the
    green pre-compaction head and the prior reviewed 30-commit tree; only homebrew: name the generic bottle publisher accurately #891
    is absent from the earlier batch, because homebrew: publish and verify installable Kandelo bottles #936 now owns it;
  • exact-head Prepare merge run 29458645742, attempt 2 passed all 85 substantive synthesis, approval, preflight, library, package/image, materialization, browser-asset, cargo/kernel, fork-instrument, Vitest, browser, libc, POSIX, Sortix, and aggregate test jobs. Its final release-control job alone failed before sealing because the old PR head lacks a lifecycle helper now owned by the prepared base; ci: seal merge candidates with trusted base helpers #964 fixes that workflow trust/compatibility bug. Attempt 3 reran only the unchanged failing control job and reproduced the same exit 127;
  • each focused source PR records its own contract-specific Node, Chromium,
    fork, libc/POSIX/Sortix, MariaDB, and SQLite validation.

The earlier 31-commit pre-rebuild head
c0ef6ae3368480da8990064593c9c35920fbacb7, which contained the same focused
work plus #891's workflow rename now owned by #936, also passed:

  • browser smoke run 29281207312
    passed the host check and Chromium, Firefox, and WebKit COI smoke coverage;
  • staging run 29281207249, attempt 2
    passed 14/14 library rebuilds, 57/57 program/image rebuilds, staged binary
    materialization, the browser asset check, cargo/kernel, fork instrumentation,
    full Vitest, browser, libc, POSIX, Sortix, and the aggregate test gate;
  • Bash's first source-build attempt hit an upstream GNU mirror HTTP 502. The
    declared URL recovered, and the isolated retry passed without a code change.
  • wasm32/wasm64 musl builds, 1,154/1,154 cargo/kernel tests, and 187/187
    fork-instrument tests also passed locally through scripts/dev-shell.sh.

The current head is built directly on main at
9a7a679195ead9bbdef119830437d5a758566607. Its tree is unchanged by the final
history-only rewrite; the new hosted checks will provide exact-head status.

Validation not run

A manual ./run.sh browser pass and the full SQLite upstream matrix were not
rerun locally. Performance was not measured, and this PR makes no performance
claim.

Merge and cleanup

Keep the 20 commits in order; do not squash across the remaining purpose
boundaries. An independent per-commit audit found no other valid full squash. The included source PRs are closed as carried into this batch, and
their remote branches were deleted with exact-SHA leases, top-down for the
SQLite stack. This cleanup does not add ready-to-ship; the normal approval
and prepare-merge flow remain separate.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

Phase B-1 matrix build status — pr-934-staging

ABI v39. 71 built, 0 failed, 71 total.

Package Arch Status Sha
icu wasm32 built df50355f
libcurl wasm32 built eff13ed9
libcxx wasm32 built 7989f75b
libcxx wasm64 built 15f45186
libiconv wasm32 built 55beab72
libpng wasm32 built ff9927f2
libxml2 wasm32 built 3238343a
libzip wasm32 built 74fa1395
openssl wasm32 built 76d96c6e
openssl wasm64 built 912566eb
sqlite wasm32 built 4f395eaa
sqlite wasm64 built 9ea2044b
zlib wasm32 built edcbcb05
zlib wasm64 built 0c65f349
bc wasm32 built c3513140
bzip2 wasm32 built d0f16d7f
coreutils wasm32 built 190a9b04
curl wasm32 built b35cf878
dash wasm32 built e72eac78
diffutils wasm32 built 1671d2c4
dinit wasm32 built d366b990
fbdoom wasm32 built 50828ca7
file wasm32 built b0b5f319
findutils wasm32 built 54d8b55e
gawk wasm32 built b119d2b0
git wasm32 built 135b973c
grep wasm32 built 3343257f
gzip wasm32 built dfddbf00
hello wasm32 built b23d651b
kandelo-sdk wasm32 built 02b5aa7f
kernel wasm32 built 2e1d8990
less wasm32 built d4b1b655
lsof wasm32 built 03d6c53e
m4 wasm32 built 599dea84
make wasm32 built 86e14b04
mariadb wasm32 built b85ba025
mariadb wasm64 built cd3d1a28
modeset wasm32 built 364a5dc3
msmtpd wasm32 built 346926f4
nano wasm32 built b58c5c51
ncurses wasm32 built 9646f601
netcat wasm32 built cb19fa84
nginx wasm32 built e0974b4b
php wasm32 built d75ce9eb
posix-utils-lite wasm32 built 927fb76e
ruby wasm32 built 3defafeb
sed wasm32 built f0a36592
spidermonkey wasm32 built 794a13d5
tar wasm32 built 3e14b588
tcl wasm32 built c2ee1701
unzip wasm32 built 3d9892e7
userspace wasm32 built 4ad2d955
vim wasm32 built a096a29d
wget wasm32 built 760f1716
xz wasm32 built b4ece42c
zip wasm32 built c99c71dd
zstd wasm32 built eaa226fc
bash wasm32 built f62b8fb5
mariadb-test wasm32 built 2390c511
mariadb-vfs wasm32 built 05ca805e
mariadb-vfs wasm64 built ea9b5ef3
nethack wasm32 built 66ec7a5e
node wasm32 built 62ecd88e
spidermonkey-node wasm32 built d34e46bf
vim-browser-bundle wasm32 built 54fa00dc
nethack-browser-bundle wasm32 built 425bc5a3
rootfs wasm32 built 44f0d6c4
shell wasm32 built 9dc7deee
lamp wasm32 built eef497d2
node-vfs wasm32 built 51971e37
wordpress wasm32 built 969b62e1

Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.

@brandonpayton brandonpayton marked this pull request as ready for review July 13, 2026 23:23
@brandonpayton brandonpayton marked this pull request as draft July 13, 2026 23:25
@brandonpayton brandonpayton force-pushed the integration/post-907-focused-batch branch 2 times, most recently from 7dce296 to 260d75d Compare July 13, 2026 23:41
@brandonpayton brandonpayton marked this pull request as ready for review July 14, 2026 02:40
Configure the browser test runner with its same-origin Vite proxy and keep a Chromium Wget regression for pages without a service-worker controller. Assert that BrowserKernel forwards the existing option and document the browser transport boundary.
The deterministic region ordering already landed through #907, but its in-process regression does not vary Rust HashMap random state. Run the CLI in twelve fresh processes against an alternating-type nested-region fixture and compare the emitted bytes.

Document the cross-process byte-reproducibility contract and update the fork-instrumentation reference to the current ABI 39. This adds no ABI or package-artifact change.

Validation: the full fork-instrument suite passed 187 tests; the focused fresh-process test passed; and the ABI snapshot check passed through scripts/dev-shell.sh.
Delegate fsync on directory descriptors to the active filesystem backend so Node can issue its native durability barrier. Keep memory-backed and OPFS behavior truthful at their host boundaries, and cover the kernel, host backends, Chromium OPFS, and POSIX regression path.
Apply one supported recursion policy to both the shipped library and official testfixture: compound-select, expression, JSON, and trigger recursion stay within current browser and Node WebAssembly stacks. Keep the test patches aligned with those compiled limits and preserve the existing package revision recorded by the reviewed batch tree.
Keep analyze9's memory-accounting assertion, but use the encoding-correct allocation delta for SQLite's utf16 permutation.
Align the stack passed to kernel_clone to the 16-byte boundary required by Wasm code generation. This keeps stack-based variadic arguments intact when a new pthread begins.

Add a local Sortix regression that formats the SQLite temporary-file pattern from fresh pthreads. The kernel_clone interface and ABI layout are unchanged.
@brandonpayton brandonpayton force-pushed the integration/post-907-focused-batch branch from 260d75d to 822f35e Compare July 14, 2026 02:50
@brandonpayton

Copy link
Copy Markdown
Member Author

Merge-order note for the Homebrew migration: keep this PR queued behind #913. #913 has already completed its durable package promotion against the current main tree. Landing #934 first would change that tree and require #913 to be rebased and its package evidence rebuilt before merge. After #913 lands, rebase this exact reviewed batch onto the new main and rerun the required gate; its current 86/86 successful jobs remain review evidence, but are not a substitute for the post-#913 synthetic merge.

@brandonpayton brandonpayton added the ready-to-ship Maintainer attests the exact tested head and requests merge preparation. label Jul 14, 2026
@brandonpayton brandonpayton removed the ready-to-ship Maintainer attests the exact tested head and requests merge preparation. label Jul 14, 2026
brandonpayton added a commit that referenced this pull request Jul 15, 2026
## Why

`Prepare merge` previously promoted a pull request package index into
the live `binaries-abi-v<N>` release before the pull request merged.
That made ABI 39 describe unmerged package state and caused unrelated
resolver and browser gates to consume a canonical ledger that did not
match `main`.

The canceled #934 run also exposed a recovery case: 56 ABI 39 entries
are stale even though the exact current-main archives already exist in
the canonical release. Some PR staging archives have the same
cache-keyed name but different envelope bytes because run provenance is
not part of the cache key. Candidate preparation must repair the ledger
without replacing already-published canonical bytes.

A pre-merge test should prove what can be published. It must not change
what current users resolve.

## New behavior

### Before merge

- Read the canonical index and complete asset metadata through a
mutation-free snapshot while holding the canonical state lock.
- Build and promote archives into a unique candidate release for the
exact PR, run, and attempt.
- When an exact cache-keyed canonical archive exists but its ledger
entry is stale, prefer that canonical asset over a same-name staging
asset.
- Carry the snapshotted SHA-256 and size into the promotion matrix, then
verify both after download.
- Compose and test a complete candidate index.
- Require either an eligible review of the exact PR head or a fresh
`ready-to-ship` application by a current repository maintainer/admin. An
outstanding `CHANGES_REQUESTED` decision still fails closed, and
persistent
  label state is never treated as authority.
- Seal the candidate only after the synthetic merge test gate passes.
- Post `merge-gate=success` with the exact candidate release as its
authority record.

No pre-merge job writes, migrates, recovers, replaces, or deletes assets
in the canonical ABI release.

### After merge

A default-branch workflow:

1. verifies the PR head, merge method, exact merged tree, latest
successful merge-gate status, and candidate receipt;
2. holds authority, candidate, and canonical locks in a fixed order;
3. performs one conflict-aware three-way index merge;
4. verifies every retained and candidate archive by immutable SHA-256
identity;
5. copies inert archives before publishing one complete canonical index
transaction;
6. records activation or rejection evidence without hiding a failed
candidate.

Scheduled and manual reconciliation recover interrupted transactions and
retry merged candidates. Cleanup retains open, authoritative, or recent
failed evidence and removes only state that GitHub proves is obsolete.

## Crash and trust handling

- Canonical `index.toml` publication uses an immutable generation,
write-ahead journal, marker commit, and deterministic recovery.
- The pre-merge snapshot accepts only an agreeing committed
marker/generation/live view, or a valid legacy live index, and never
repairs it.
- Duplicate asset names, missing digests, size mismatches, and
post-snapshot replacement races fail closed.
- API reads are paginated and bounded; transient reads retry and
uncertain state is retained.
- Every workflow job whose call graph can acquire the state lock is
audited for effective `actions: read`.
- Workflow contract tests consume complete producer output before
matching, avoiding Linux `pipefail` false failures from early-exiting
consumers.
- A staging retry can omit package builds only after a typed expected
ledger
proves complete package/architecture coverage. The PR release must bind
each
indexed success to one exact uploaded asset, and any stale PR entry must
have
an exact-current canonical supplement. The test gate revalidates both
sources,
  verifies and downloads the selected bytes, and resolves from one local
`file://` index. Invalid or incomplete reuse falls back to the normal
build
  matrix.

## Scope

This changes package publication control state only. It does not change
Kandelo ABI, package cache keys, or package payloads.

The next successful candidate after this workflow lands will repair the
56 stale ABI 39 entries by copying and verifying the already-published
canonical archives in the isolated candidate, then activating the
corrected ledger after merge. It will not replace canonical archive
bytes.

Archive-envelope reproducibility is a separate known issue. This PR
makes that issue non-destructive by preserving canonical bytes; it does
not trigger a legacy-registry rebuild while the project is migrating
ordinary software to Homebrew bottles.

## Exact-head evidence

- Base: `06afa498d17358cd7e1479fb331f3c4ea179dcb2` (#913)
- Head: `7af64ae3f009984514e2db7adbd9d8319b35d808`
- Tree: `011d8feec4fc3a37c41f57ab706fdcb16bc96ebb`
- History: three purpose commits on current `main`; this PR is intended
for a squash merge.
- Full `./scripts/dev-shell.sh bash
tests/scripts/package-publish-flow.sh`: passed on the exact head.
- Focused xtask index-candidate tests: 10/10 passed.
- Focused xtask staging-reuse tests: 10/10 passed and are enforced by
the publish-flow harness.
- ABI snapshot, header, generated TypeScript bindings, Bash syntax, and
`git diff --check`: passed.
- Live ABI 39 inventory: 210 assets, all uploaded, with no duplicate
names or missing digests.
- Live stale-ledger audit: all 56 exact-main entries select their
existing canonical archive; conflicting staging bytes do not win.
- Live retry audit: `pr-946-staging` is structurally valid for all 71
managed entries, with 57 exact-current entries; `binaries-abi-v39`
supplies exact-current assets for the remaining 14. GitHub Actions run
`29376942848` validated that union and scheduled zero package builds.
- Two independent devil reviews accepted this exact head and tree with
no blocking findings.

GitHub Actions run `29376942848` completed successfully on this exact
head. It materialized and verified the full 71-entry
target-plus-canonical package union, passed host typechecking and the
package-publish flow, built the kernel and test programs, prepared
browser assets, and passed the browser, Vitest, kernel, libc, POSIX,
Sortix, and fork-instrument suites. No package matrix job ran.
`actionlint`, `shellcheck`, and repository-toolchain `rustfmt` are not
available in the declared dev shell and are not claimed.

## One-time bootstrap

`main` at this PR's base does not contain the trusted authorization
verifier,
so no label can bootstrap #946 itself. The earlier label run failed
because the
base-pinned helper did not exist. After this exact head's staging run is
green,
#946 requires a one-time admin-bypass squash merge guarded to this head.
Do not
apply `ready-to-ship` to #946 again. Once this PR is on `main`, future
PRs use a
fresh `ready-to-ship` event for exact-head maintainer attestation.

## Review order

1. `.github/workflows/prepare-merge.yml` and
`.github/workflows/activate-merge-candidate.yml`
2. `.github/scripts/fetch-canonical-index.sh`,
`select-package-archive-source.sh`, and
`download-verified-release-asset.sh`
3. `.github/scripts/verify-merge-candidate.sh` and
`activate-merge-candidate.sh`
4. `scripts/release-index-state.sh`
5. `tools/xtask/src/index_candidate.rs`
6. `.github/workflows/staging-build.yml`,
`.github/scripts/validate-staging-release.sh`, and
`.github/scripts/compose-staging-release-snapshots.sh`
7. `tools/xtask/src/staging_reuse.rs`
8. `tests/scripts/package-publish-flow.sh`
@brandonpayton brandonpayton changed the title Land the focused platform follow-ups after #907 platform: combine remaining POSIX, runtime, and SQLite fixes Jul 15, 2026
@brandonpayton brandonpayton added batched-changes Use rebase auto-merge in prepare-merge so granular commits land on main. ready-to-ship Maintainer attests the exact tested head and requests merge preparation. labels Jul 15, 2026
@github-actions github-actions Bot removed the ready-to-ship Maintainer attests the exact tested head and requests merge preparation. label Jul 16, 2026
@github-actions

Copy link
Copy Markdown

prepare-merge failed during the merge-gate-post step. Check the workflow run log: https://github.com/Automattic/kandelo/actions/runs/29458645742. Re-apply the ready-to-ship label after fixing.

1 similar comment
@github-actions

Copy link
Copy Markdown

prepare-merge failed during the merge-gate-post step. Check the workflow run log: https://github.com/Automattic/kandelo/actions/runs/29458645742. Re-apply the ready-to-ship label after fixing.

@brandonpayton

Copy link
Copy Markdown
Member Author

Root cause of the final gate failure is fixed by #964. All 85 substantive jobs in Prepare run 29458645742 passed. merge-gate-post alone failed because it checked lifecycle helpers out from this older PR head, where mark-merge-candidate-ready.sh does not exist, instead of from the tested base that owns the write-authorized helper. Rerunning the same run preserves the old workflow and fails again. After #964 merges, reapply ready-to-ship here to start a fresh Prepare run against the corrected workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

batched-changes Use rebase auto-merge in prepare-merge so granular commits land on main.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant