Upstream: add riscv64 wheel to openai/tiktoken #6

Goal

Contribute riscv64 wheel support upstream to openai/tiktoken so that pip install tiktoken works out of the box on RISC-V 64-bit Linux systems without building from source.

Current State

• This fork ships a prebuilt linux_riscv64 wheel (v0.12.0) built natively on a BananaPi F3 (SpacemiT K1, rv64imafdcv, 8 cores @ 1.6 GHz)
• Upstream (openai/tiktoken) does not publish riscv64 wheels on PyPI
• Users on riscv64 must either build from source (Rust/maturin, can take 2-60 min depending on package) or use our PEP 503 index

Upstream Approach

The preferred path is adding riscv64 to the upstream project's CI wheel build matrix. Two main strategies:

Option A: cibuildwheel with QEMU (recommended for most projects)

• Add linux_riscv64 to the CIBW_ARCHS_LINUX list in the existing cibuildwheel config
• Uses QEMU emulation on GitHub Actions runners (x86_64 host)
• Supported since cibuildwheel 2.21+ with manylinux_2_39_riscv64 images
• Slower than native (~5-20x) but requires no hardware

Option B: Self-hosted native runners

• Use actual riscv64 hardware as GitHub Actions runners
• Faster builds, tests run on real hardware
• Requires runner infrastructure (we have two BananaPi F3 boards available)

Option C: Cross-compilation

• For Rust/maturin packages: maturin build --target riscv64gc-unknown-linux-gnu
• For C/C++ packages: cross-compile with riscv64 GCC toolchain
• Fastest CI time but may miss runtime issues

What We Can Provide in the Upstream PR

• CI configuration changes (cibuildwheel or GitHub Actions workflow)
• Tested wheel artifact proving the package builds and imports on riscv64
• Build time data from native hardware
• Any source patches needed (if any were required for our fork build)

Context

• PyPI riscv64 wheel ecosystem is growing: manylinux_2_39_riscv64 landed in pypa/manylinux (2025), auditwheel supports riscv64, and cibuildwheel can target it
• Several major packages already ship riscv64 wheels: aiohttp, yarl, regex, markupsafe, charset-normalizer, rpds-py
• RISC-V hardware is shipping (SiFive, SpacemiT, Sophgo SG2044) with users running Python workloads
• This fork is part of a set of 32 forks building riscv64 wheels for the Python ML/AI stack: https://github.com/gounthar/riscv64-python-wheels

References

• PEP 503 index: https://gounthar.github.io/riscv64-python-wheels/simple/
• Wheel repo: https://github.com/gounthar/riscv64-python-wheels
• cibuildwheel riscv64 docs: https://cibuildwheel.pypa.io/en/stable/
• RISE project riscv64 packaging: https://riseproject.dev/2025/05/14/easy-installation-of-binary-python-packages-on-riscv64-devices/