Skip to content

Commit 1aa22f5

Browse files
authored
magika: add build-magika.yml for riscv64 wheels (#537)
* **Package**: `magika` * **Version**: `1.0.3` * **Source**: https://github.com/google/magika * **Docs**: https://github.com/google/magika/blob/main/python/README.md The wheel ships magika's Rust CLI, compiled from source and statically linked against ONNX Runtime. Upstream publishes no riscv64 wheel, and the pure-Python one it does publish carries a `magika` command that only prints an error. Mirrors [upstream's `python-build-and-release-package.yml`](https://github.com/google/magika/blob/python-v1.0.3/.github/workflows/python-build-and-release-package.yml). **Differs from upstream** - ONNX Runtime's gtest suite is not built; nothing links it and it costs hours here. - magika's own LICENSE and ONNX Runtime's licence texts are staged into the wheel. **Matrix**: one `py3-none-manylinux_riscv64` wheel, tested on cp312/cp313/cp314/cp314t. **Testing** - `dacite` and `tqdm` added; upstream's reference tests import them from its `dev` group. - `test_magika_module_with_permission_error` dropped; the container runs as root. **License**: Wheel bundles ONNX Runtime (MIT); upstream ships no licence text for it or for magika itself, so the build adds both. **Patches** - `0001-build-ONNX-Runtime-s-bundled-cpuinfo-on-riscv64.patch` - Inappropriate. Without it cpuinfo is not built and the link fails on `-lcpuinfo`. riscv64-only. - `0002-build-ONNX-Runtime-s-re2-static-library-explicitly.patch` - Inappropriate. Without it the link fails on `-lre2`.
1 parent edf7173 commit 1aa22f5

3 files changed

Lines changed: 296 additions & 0 deletions

File tree

‎.github/workflows/build-magika.yml‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
# SPDX-FileCopyrightText: 2026 The RISE Project
2+
# SPDX-License-Identifier: MIT
3+
---
4+
# This workflow is based on: https://github.com/google/magika/blob/python-v1.0.3/.github/workflows/python-build-and-release-package.yml
5+
# (its build-wheels + test-wheels jobs), which drive maturin with rust/onnx/maturin.sh
6+
# as the manylinux before-script.
7+
name: Build magika wheels (riscv64)
8+
9+
on:
10+
workflow_dispatch:
11+
inputs:
12+
version:
13+
description: 'magika version to build (git tag without the python-v prefix, e.g. 1.0.3)'
14+
required: true
15+
default: '1.0.3'
16+
pull_request:
17+
paths:
18+
- '.github/workflows/build-magika.yml'
19+
20+
concurrency:
21+
group: ${{ github.workflow }}-${{ inputs.version || '1.0.3' }}-${{ github.head_ref || github.run_id }}
22+
cancel-in-progress: true
23+
24+
permissions:
25+
contents: read # to fetch code (actions/checkout)
26+
27+
env:
28+
MAGIKA_VERSION: ${{ inputs.version || '1.0.3' }}
29+
# Upstream builds in manylinux 2_28; this is the only riscv64 manylinux image.
30+
MANYLINUX_RISCV64_IMAGE: quay.io/pypa/manylinux_2_39_riscv64
31+
# The version upstream's maturin-action used for the 1.0.3 wheels.
32+
MATURIN_VERSION: '1.13.1'
33+
34+
jobs:
35+
build_wheel:
36+
name: Build magika ${{ inputs.version || '1.0.3' }} py3-none-manylinux_riscv64
37+
runs-on: ubuntu-24.04-riscv
38+
timeout-minutes: 1440 # 24h: ONNX Runtime is built from source
39+
40+
steps:
41+
- name: Checkout magika python-v${{ env.MAGIKA_VERSION }}
42+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
43+
with:
44+
repository: google/magika
45+
ref: python-v${{ env.MAGIKA_VERSION }}
46+
persist-credentials: false
47+
48+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
49+
with:
50+
path: python-wheels
51+
persist-credentials: false
52+
53+
- name: Apply patches
54+
run: git apply --verbose python-wheels/patches/magika/${{ env.MAGIKA_VERSION }}/*.patch
55+
56+
- name: Write container script
57+
run: |
58+
cat > riscv64-build.sh <<'EOF'
59+
#!/bin/bash
60+
set -e -x
61+
62+
# ONNX Runtime's build scripts, and several of its dependencies, invoke a bare `python3`.
63+
export PATH="/opt/python/cp312-cp312/bin:${PATH}"
64+
65+
# The manylinux image ships no Rust; rustup provisions a native riscv64gc toolchain.
66+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
67+
. "${HOME}/.cargo/env"
68+
69+
# openssl-sys, which ort pulls in through its default `tls-native` feature.
70+
dnf install -y openssl-devel
71+
72+
# rust/onnx/maturin.sh, minus its `yum` line (above) and its `rust/cli/test.sh`
73+
# call (the wheel is exercised by the test job instead). We only need the static
74+
# libraries, and --skip_tests alone still *builds* ONNX Runtime's gtest suite
75+
# (onnxruntime_BUILD_UNIT_TESTS defaults to ON), which costs hours on this
76+
# runner. --compile_no_warning_as_error for the riscv64 code paths.
77+
export ONNX_RUNTIME_BUILD_FLAGS="--allow_running_as_root --skip_tests --compile_no_warning_as_error --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=OFF"
78+
(cd rust/onnx && ./build.sh)
79+
80+
# The ort crate links a fixed list of ONNX Runtime's static dependencies;
81+
# print what the build actually produced so a missing one is diagnosable.
82+
find rust/onnx/runtime/build/Linux -name '*.a' | sort
83+
84+
# maturin only globs LICEN[CS]E*/NOTICE* next to pyproject.toml, so upstream's
85+
# wheels carry neither magika's own licence nor that of the statically linked
86+
# ONNX Runtime. RISE redistributes these wheels, so stage all three.
87+
cp LICENSE python/LICENSE
88+
cp rust/onnx/runtime/LICENSE python/LICENSE.onnxruntime
89+
cp rust/onnx/runtime/ThirdPartyNotices.txt python/NOTICE.onnxruntime
90+
91+
python3 -m pip install "maturin==${MATURIN_VERSION}"
92+
cd python
93+
maturin build --release --compatibility manylinux_2_39 --out /magika_src/dist
94+
EOF
95+
96+
- name: Build wheel
97+
run: |
98+
podman run -t \
99+
--log-driver=none \
100+
--network=host \
101+
-v "${PWD}":/magika_src \
102+
--workdir /magika_src \
103+
-e MATURIN_VERSION="${MATURIN_VERSION}" \
104+
-e PIP_EXTRA_INDEX_URL=https://pypi.riseproject.dev/simple/ \
105+
--pull=newer \
106+
"${MANYLINUX_RISCV64_IMAGE}" \
107+
bash /magika_src/riscv64-build.sh
108+
109+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
110+
with:
111+
name: magika-${{ env.MAGIKA_VERSION }}-py3-none-manylinux_riscv64
112+
path: dist/*.whl
113+
if-no-files-found: error
114+
115+
test_wheel:
116+
name: Test magika ${{ inputs.version || '1.0.3' }} ${{ matrix.python }}
117+
needs: [build_wheel]
118+
runs-on: ubuntu-24.04-riscv
119+
timeout-minutes: 120
120+
strategy:
121+
fail-fast: false
122+
matrix:
123+
python: ["cp312", "cp313", "cp314", "cp314t"]
124+
125+
steps:
126+
- name: Checkout magika python-v${{ env.MAGIKA_VERSION }}
127+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
128+
with:
129+
repository: google/magika
130+
ref: python-v${{ env.MAGIKA_VERSION }}
131+
persist-credentials: false
132+
133+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
134+
with:
135+
name: magika-${{ env.MAGIKA_VERSION }}-py3-none-manylinux_riscv64
136+
path: dist
137+
138+
- name: Write container script
139+
run: |
140+
cat > riscv64-test.sh <<'EOF'
141+
#!/bin/bash
142+
set -e -x
143+
144+
case "${PYTHON_TAG}" in
145+
*t) python_dir="/opt/python/${PYTHON_TAG%t}-${PYTHON_TAG}/bin" ;;
146+
*) python_dir="/opt/python/${PYTHON_TAG}-${PYTHON_TAG}/bin" ;;
147+
esac
148+
export PATH="${python_dir}:${PATH}"
149+
150+
# dacite and tqdm come from upstream's `dev` dependency group, which its
151+
# test suite imports.
152+
python -m pip install --only-binary=:all: /magika_src/dist/*.whl pytest dacite tqdm
153+
154+
# Upstream's test-wheels job.
155+
magika --version
156+
python -c 'import magika; m = magika.Magika(); print(m)'
157+
magika -r tests_data/basic
158+
python ./python/scripts/run_quick_test_magika_cli.py
159+
python ./python/scripts/run_quick_test_magika_module.py
160+
161+
# The licences staged by the build job have to reach the installed wheel.
162+
python - <<'PY'
163+
import importlib.metadata as md
164+
165+
expected = {"LICENSE", "LICENSE.onnxruntime", "NOTICE.onnxruntime"}
166+
shipped = {p.name for p in md.files("magika") if ".dist-info/" in str(p)}
167+
assert expected <= shipped, sorted(expected - shipped)
168+
PY
169+
170+
# Upstream's python-test-suite job. The container runs as root, for which a
171+
# 0o000 file is still readable, so the permission-error test cannot pass here.
172+
cd python
173+
python -m pytest tests -m "not slow" -k "not test_magika_module_with_permission_error"
174+
EOF
175+
176+
- name: Test wheel
177+
run: |
178+
podman run -t \
179+
--log-driver=none \
180+
--network=host \
181+
-v "${PWD}":/magika_src \
182+
--workdir /magika_src \
183+
-e PYTHON_TAG="${{ matrix.python }}" \
184+
-e PIP_EXTRA_INDEX_URL=https://pypi.riseproject.dev/simple/ \
185+
--pull=newer \
186+
"${MANYLINUX_RISCV64_IMAGE}" \
187+
bash /magika_src/riscv64-test.sh
188+
189+
publish:
190+
name: Publish magika ${{ inputs.version || '1.0.3' }} to GitLab
191+
needs: [build_wheel, test_wheel]
192+
runs-on: ubuntu-latest
193+
permissions:
194+
contents: write
195+
pull-requests: write
196+
197+
steps:
198+
- name: Publish wheels and open docs PR
199+
uses: riseproject-dev/python-wheels/actions/publish-wheels@main
200+
with:
201+
artifact-pattern: magika-${{ env.MAGIKA_VERSION }}-*-manylinux_riscv64
202+
gitlab-username: ${{ vars.GITLAB_DEPLOY_USER }}
203+
gitlab-token: ${{ secrets.GITLAB_DEPLOY_TOKEN }}
204+
gitlab-project-id: ${{ vars.GITLAB_PROJECT_ID }}
205+
gh-token: ${{ secrets.GITHUB_TOKEN }}
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
From 2076a33e21e5c83ed5309f89a79f4e69693be30b Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Thu, 27 Aug 2026 17:33:42 +0200
4+
Subject: [PATCH] build ONNX Runtime's bundled cpuinfo on riscv64
5+
6+
Upstream-Status: Inappropriate [riscv64-specific workaround for an ONNX Runtime cmake limitation; the fix belongs in onnxruntime, not magika]
7+
8+
rust/onnx/build.sh builds ONNX Runtime v1.24.2 as a static library, and
9+
the ort crate unconditionally links -lcpuinfo against it. ONNX Runtime
10+
only enables its bundled pytorch/cpuinfo when onnxruntime_target_platform
11+
matches an x86/arm allowlist, so on riscv64 it prints "Target processor
12+
architecture riscv64 is not supported in cpuinfo" and libcpuinfo.a is
13+
never produced; the wheel build then dies at the final link with "could
14+
not find native static library `cpuinfo`".
15+
16+
pytorch/cpuinfo has supported riscv64 since it grew src/riscv, and the
17+
commit ONNX Runtime pins (403d652) contains it, so adding riscv64 to the
18+
allowlist is enough. ONNX Runtime's own uses of cpuinfo outside
19+
CPUIDINFO_ARCH_ARM and CPUIDINFO_ARCH_X86 are limited to
20+
cpuinfo_initialize, cpuinfo_get_processor, cpuinfo_get_core and
21+
cpuinfo_get_cores_count, all of which the riscv64 build provides.
22+
---
23+
rust/onnx/build.sh | 8 ++++++++
24+
1 file changed, 8 insertions(+)
25+
26+
diff --git a/rust/onnx/build.sh b/rust/onnx/build.sh
27+
index eb59ac1..69fe4b7 100755
28+
--- a/rust/onnx/build.sh
29+
+++ b/rust/onnx/build.sh
30+
@@ -43,6 +43,14 @@ else
31+
target_compile_options(${target_name} PRIVATE "$<$<COMPILE_LANGUAGE:CXX>:-Wno-uninitialized>")' \
32+
cmake/CMakeLists.txt
33+
34+
+ # pytorch/cpuinfo builds for riscv64 (src/riscv/), but ONNX Runtime's cpuinfo
35+
+ # architecture allowlist predates that support, so cpuinfo is not built and the
36+
+ # ort crate then fails to link with "could not find native static library
37+
+ # `cpuinfo`". The grep is there so a rewording upstream fails the build loudly.
38+
+ sed -i 's/|aarch64|arm64)$")/|aarch64|arm64|riscv64)$")/' \
39+
+ cmake/external/onnxruntime_external_deps.cmake
40+
+ grep -q 'arm64|riscv64)$")' cmake/external/onnxruntime_external_deps.cmake
41+
+
42+
info "Build the static libraries."
43+
x ./build.sh --config=Release --parallel $ONNX_RUNTIME_BUILD_FLAGS
44+
45+
--
46+
2.50.1 (Apple Git-155)
47+
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
From 948f5ad96a935a4b1e57195f6389951ed71fdd0c Mon Sep 17 00:00:00 2001
2+
From: Ludovic Henry <git@ludovic.dev>
3+
Date: Thu, 27 Aug 2026 21:10:00 +0200
4+
Subject: [PATCH] build ONNX Runtime's re2 static library explicitly
5+
6+
Upstream-Status: Inappropriate [compensates for turning ONNX Runtime's unit tests off, which the riscv64 runner cannot afford; upstream's own build leaves them on and gets re2 for free]
7+
8+
The ort crate links -lre2 unconditionally against the static ONNX Runtime
9+
this script builds. ONNX Runtime declares re2 EXCLUDE_FROM_ALL, and the
10+
only thing in its default target that concretely links re2 is the gtest
11+
suite -- every consumer inside ONNX Runtime is itself a static library, so
12+
CMake never has to produce libre2.a for them. Building with
13+
onnxruntime_BUILD_UNIT_TESTS=OFF therefore leaves _deps/re2-build empty
14+
and the wheel build dies at the final link with "could not find native
15+
static library `re2`".
16+
17+
Building the target by name costs a couple of minutes, where leaving the
18+
unit tests on adds hours to an ONNX Runtime build that already takes over
19+
three on this hardware. The test keeps a silently renamed target from
20+
being discovered 20 minutes later, in cargo.
21+
---
22+
rust/onnx/build.sh | 6 ++++++
23+
1 file changed, 6 insertions(+)
24+
25+
diff --git a/rust/onnx/build.sh b/rust/onnx/build.sh
26+
index 69fe4b7..699182e 100755
27+
--- a/rust/onnx/build.sh
28+
+++ b/rust/onnx/build.sh
29+
@@ -54,6 +54,12 @@ else
30+
info "Build the static libraries."
31+
x ./build.sh --config=Release --parallel $ONNX_RUNTIME_BUILD_FLAGS
32+
33+
+ # The ort crate links re2 unconditionally, but ONNX Runtime declares re2
34+
+ # EXCLUDE_FROM_ALL and only its unit tests pull it into the default target, so a
35+
+ # build that turns those off never produces libre2.a. Build it by name.
36+
+ x cmake --build build/Linux/Release --target re2 --parallel
37+
+ test -f build/Linux/Release/_deps/re2-build/libre2.a
38+
+
39+
info "Only keep the static libraries to save cache space."
40+
find build/Linux -not -name '*.a' \( -not -type d -or -empty \) -delete
41+
cd ..
42+
--
43+
2.50.1 (Apple Git-155)
44+

0 commit comments

Comments
 (0)