Skip to content

Build pytantan from nextgenusfs/pytantan fork with SIMD disabled - #69

Merged
nextgenusfs merged 1 commit into
mainfrom
feat/pytantan-fork-no-simd
May 26, 2026
Merged

Build pytantan from nextgenusfs/pytantan fork with SIMD disabled#69
nextgenusfs merged 1 commit into
mainfrom
feat/pytantan-fork-no-simd

Conversation

@nextgenusfs

Copy link
Copy Markdown
Owner

Why

PR #68 attempted to scrub AVX2 from pytantan by source-patching its CMakeLists.txt during the Docker build and rebuilding the wheel. The build-time verifier passed, but the published :latest image still contained avx2.so/sse4.so files and SIGILL'd on import pytantan under Rosetta 2. The most likely cause was pixi/uv self-healing the environment back to the PyPI wheel between the in-build verifier and the COPY --from=build step — the source patch was applied to a copy of pytantan that was then overwritten.

Rather than continue fighting the patch-after-install pattern, this PR moves the fix upstream into a fork of pytantan and changes the funannotate2 build to consume that fork as its only source for pytantan.

What

pytantan side (separate repo, nextgenusfs/pytantan@90818ff)

Two independent escape hatches were added to the fork on branch feat/disable-simd:

  1. PYTANTAN_DISABLE_SIMD CMake option (default OFF). When ON, skips the FindSSE4/FindAVX2/FindNEON includes and forces HAVE_SSE2=HAVE_SSE4=HAVE_AVX2=HAVE_NEON=OFF, so only the generic Cython extension is compiled.
  2. PYTANTAN_SIMD env-var override read at lib.pyx import time. Accepted values: generic (or none/off/0), sse2, sse4, avx2, neon. Overrides the archspec-detected runtime support flags. Lets users force the generic backend on a SIMD-enabled wheel without rebuilding.

These are additive and intended to be upstreamed to althonos/pytantan once we've validated the funannotate2 image.

funannotate2 side (this PR)

  • pixi.toml: pytantan = "==0.1.3" is replaced with
    pytantan = { git = "https://github.com/nextgenusfs/pytantan.git", rev = "90818ff695c844cefd5f42eb3baa297263565f21" }
    So pixi/uv compile pytantan from source at a pinned SHA. No PyPI wheel of pytantan is ever installed in the pixi environment.
  • pixi.lock: regenerated. Both linux-64 and osx-arm64 now reference the git URL.
  • Dockerfile: net -93 lines.
    • Build tools (git build-essential cmake zlib1g-dev binutils ca-certificates) are installed before pixi install --locked so uv has what it needs to build pytantan from source.
    • ENV CMAKE_ARGS="-DPYTANTAN_DISABLE_SIMD=ON" is set before pixi install, which scikit-build-core honors during the source build.
    • The ~100-line post-install rebuild block is deleted. The AVX-bearing wheel is never installed, so there's nothing to uninstall and replace — and no window for pixi to self-heal back to PyPI.
    • The verifier RUN is kept as a safety net: it asserts no avx*/sse4*/neon* .so files in pytantan/platform/ and zero AVX/AVX2/AVX512 instruction hits in any pytantan .so.

Local verification (linux/amd64 under Rosetta 2)

The build stage was built with docker buildx build --platform linux/amd64 --target build. The in-build verifier output:

/app/.pixi/envs/default/lib/python3.11/site-packages/pytantan/lib.abi3.so -> 0 AVX/AVX2/AVX512 hits
/app/.pixi/envs/default/lib/python3.11/site-packages/pytantan/platform/generic.abi3.so -> 0 AVX/AVX2/AVX512 hits
pytantan smoke test OK 0.1.4

Runtime smoke test against the built image:

$ docker run --rm --platform linux/amd64 f2-fork-test:latest bash -c 'ls /app/.pixi/envs/default/lib/python3.11/site-packages/pytantan/platform/'
.gitignore  CMakeLists.txt  __init__.pxd  __init__.py  __pycache__  generic.abi3.so  pxd.in  pyx.in

$ docker run ... python -c "from pytantan.lib import RepeatFinder; ..."
OK 0.1.4
mask_repeats OK 3000

Only generic.abi3.so is present in platform/ — no avx2.so, no sse4.so, no neon.so. RepeatFinder instantiates and mask_repeats() runs end-to-end without SIGILL under Rosetta 2.

Follow-up

  1. Merge this PR.
  2. Trigger Actions → Build and Publish Docker Image → Run workflow on main (push=true) so :latest is rebuilt from the new Dockerfile.
  3. Repeat the runtime smoke test against the published image.
  4. Once stable, open an upstream PR against althonos/pytantan with the two escape hatches.

Pull Request opened by Augment Code with guidance from the PR author

Replaces the previous Dockerfile-level source-patch dance with a clean,
upstream-able fix:

* pixi.toml now resolves pytantan from
  nextgenusfs/pytantan@90818ff (feat/disable-simd branch). That fork
  adds a PYTANTAN_DISABLE_SIMD CMake option and a PYTANTAN_SIMD runtime
  env-var override.
* Dockerfile sets CMAKE_ARGS=-DPYTANTAN_DISABLE_SIMD=ON before
  'pixi install --locked', so uv compiles a generic-only pytantan wheel
  during the normal pixi install step. Build tools (git, build-essential,
  cmake, zlib1g-dev, binutils) are installed before pixi install so the
  source build can run.
* Removes the ~100-line post-install rebuild block: the AVX2-bearing
  wheel is never installed in the first place, so there is nothing to
  uninstall and replace. This also eliminates the pixi-self-healing
  risk that was causing the previous fix to silently revert.
* Verifier RUN is kept as a safety net: asserts no
  avx*/sse4*/neon* .so files in pytantan/platform/ and zero AVX/AVX2/
  AVX512 instruction hits in any pytantan .so.

Verified locally on linux/amd64 under Rosetta 2:
  - platform/ contains only generic.abi3.so
  - 'from pytantan.lib import RepeatFinder' succeeds
  - mask_repeats() runs end-to-end without SIGILL
@nextgenusfs
nextgenusfs marked this pull request as ready for review May 26, 2026 03:58
@nextgenusfs
nextgenusfs merged commit 1d8a1cf into main May 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant