Skip to content

fix(FlexGlue): fix TensorRT zero-match engines; store descriptors as FP16 - #261

Merged
dvorak0 merged 8 commits into
mainfrom
xiaole/matcher-fixes
Nov 10, 2026
Merged

fix(FlexGlue): fix TensorRT zero-match engines; store descriptors as FP16#261
dvorak0 merged 8 commits into
mainfrom
xiaole/matcher-fixes

Conversation

@xiaolefang-dm

@xiaolefang-dm xiaolefang-dm commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

FlexGlue TensorRT matcher engines built by make flexglue silently return zero matches on some image pairs. ONNX Runtime on the same inputs is fine.

  1. FP16 attention NaN (desktop TRT 10.13 and 11.4, Orin TRT 10.3). TensorRT fuses each attention over the 64x64 grid (4096 tokens) into one FP16 kernel that sporadically overflows to NaN. The NaN reaches covisibility_score, so every match is dropped. On Looper bags this hit ~20% of wide-baseline / revisit pairs and ~4% of adjacent keyframes. Fix: build without --stronglyTyped and pin the attention layers (QK^T, Softmax, PV) to BF16 with --precisionConstraints=obey --layerPrecisions=....
  2. CumSum (TRT 10.3, Jetson only). CumSum fused with the ops that feed it is miscomputed, so the soft-argmax denominators become 0. Layer precisions cannot fix this. Fix: tinynav/models/patch_flexglue_matcher.py replaces the 4 CumSum ops with an equivalent MatMul against a constant upper-triangular matrix, before trtexec. This needs onnx; make prefers the repo .venv Python.
  3. Map size. Keyframe FlexGlue descriptors are stored as FP16 when that is lossless, which it normally is because the descriptor engine runs in FP16. Each keyframe drops from 4.2 MB to 2.1 MB, and features are converted back to FP32 before matching. Old FP32 maps still load.

Verification

Saved matcher inputs from Looper bags: 6 pairs that fail with the shipped engine, plus 2 control pairs. ONNX Runtime reference: 275, 264, 148, 185, 239, 244, 265, 229.

Platform Engine NaN Zero-match pairs Matches per pair
— ONNX Runtime (reference) 0/8 0/8 275, 264, 148, 185, 239, 244, 265, 229
RTX 3060, TRT 10.13 Unpatched (main) 6/8 6/8 0, 0, 0, 0, 0, 0, 266, 229
RTX 3060, TRT 10.13 This PR 0/8 0/8 272, 265, 150, 185, 239, 244, 266, 230
Orin Nano, TRT 10.3 Unpatched (main) 6/8 6/8 0, 0, 0, 0, 0, 0, 266, 230
Orin Nano, TRT 10.3 This PR without the CumSum fix 0/8 1/8 263, 241, 141, 0, 228, 224, 253, 224
Orin Nano, TRT 10.3 This PR with the CumSum fix 0/8 0/8 261, 238, 140, 169, 228, 224, 247, 221
  • Matcher time per pair is unchanged: ~104 ms on the 3060 and ~0.5 s on Orin.
  • Match counts are slightly below ONNX Runtime (1-2% on the 3060, 5-10% on Orin), because without --stronglyTyped TensorRT may run other layers in FP16.
  • TensorRT 11.4 still produces the NaN with the unpatched model.
  • FP16 descriptor storage: on real pairs, matches are identical for live, FP16-stored and FP32-stored features.

Notes

  • The README's Orin Nano FlexGlue numbers were measured with the unpatched engine. README is unchanged for now.
  • Existing FlexGlue matcher plans are rebuilt automatically by make -C tinynav/models flexglue.

🤖 Generated with Claude Code

xiaole and others added 3 commits November 10, 2026 09:52
xiaole @claude
fix(FlexGlue): run matcher attention in BF16 to stop NaN zero-match o…
b530c55 
…utputs

The stronglyTyped FP16 matcher engine lets TensorRT fuse each 4096-token
attention into one FP16 kernel that sporadically emits NaN. The NaN reaches
covisibility_score and every match is dropped: ~20% of wide-baseline /
revisit pairs and ~4% of adjacent keyframes on Looper bags returned zero
matches, while ONNX Runtime on the same inputs was fine.

make flexglue now patches the ONNX so the attention cores (MatMul, Softmax,
MatMul) run in BF16 before building the engine (workspace raised to 3 GB).
Matches equal ONNX Runtime; matcher time 198 -> 214 ms on an RTX 3060 laptop.
Adds onnx as a dependency for the patch script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xiaole @claude
fix(map): pass the camera_info image size to LightGlue
53f1b73 
match_keypoints hard-coded 848x480 (RealSense), so Looper (544x640) keypoints
were normalized with the wrong size. Take width/height from camera_info.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xiaole @claude
fix(FlexGlue): replace matcher CumSum with MatMul for TensorRT 10.3
8b7c614 
On Jetson (TensorRT 10.3) the FlexGlue matcher still returned zero matches
on some pairs after the BF16 attention patch: CumSum fused with the ops that
feed it is miscomputed, the soft-argmax denominators become zero, and every
mutual check fails. ONNX Runtime and TRT 10.13 are fine on the same input.

patch_flexglue_matcher.py (renamed from patch_flexglue_attention.py) now also
replaces each last-axis CumSum by a MatMul with a constant upper-triangular
matrix. On the saved failing inputs both the RTX 3060 (TRT 10.13) and Orin
Nano (TRT 10.3) engines now match ONNX Runtime; Orin matcher time is
unchanged (~0.5 s). The unpatched engine on Orin returns NaN/zero matches on
6 of the 8 inputs, like on desktop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown
cloc github.com/AlDanial/cloc v 2.04 T=0.52 s (103.2 files/s, 263101.9 lines/s)
Language files blank comment code
JSON 3 8 0 129481
Python 19 796 395 4522
Markdown 5 193 6 476
C++ 4 60 38 409
CSS 1 58 0 342
Bourne Shell 16 42 9 281
Dockerfile 1 31 32 205
HTML 1 6 0 107
make 1 26 16 77
TOML 1 4 0 60
CMake 1 4 0 17
Text 1 4 0 17
-------- -------- -------- -------- --------
SUM: 54 1232 496 135994

xiaole @claude
revert: keep the LightGlue image-size change out of this PR
8fa2cf8 
This reverts commit 53f1b73 so the PR only carries the FlexGlue TensorRT fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@xiaolefang-dm xiaolefang-dm changed the title fix(FlexGlue): work around TensorRT fusion bugs that zero out matches; LightGlue image size from camera_info fix(FlexGlue): work around TensorRT fusion bugs that zero out matches Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown
cloc github.com/AlDanial/cloc v 2.04 T=0.37 s (147.5 files/s, 376174.9 lines/s)
Language files blank comment code
JSON 3 8 0 129481
Python 19 796 395 4518
Markdown 5 193 6 476
C++ 4 60 38 409
CSS 1 58 0 342
Bourne Shell 16 42 9 281
Dockerfile 1 31 32 205
HTML 1 6 0 107
make 1 26 16 77
TOML 1 4 0 60
CMake 1 4 0 17
Text 1 4 0 17
-------- -------- -------- -------- --------
SUM: 54 1232 496 135990

xiaole @claude
build(FlexGlue): prefer the repo .venv Python for the matcher patch step
55845ab 
The dev container's default python3 may lack onnx (seen on Jetson), while
uv sync installs it into .venv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown
cloc github.com/AlDanial/cloc v 2.04 T=0.54 s (99.1 files/s, 252772.4 lines/s)
Language files blank comment code
JSON 3 8 0 129481
Python 19 796 395 4518
Markdown 5 193 6 476
C++ 4 60 38 409
CSS 1 58 0 342
Bourne Shell 16 42 9 281
Dockerfile 1 31 32 205
HTML 1 6 0 107
make 1 26 16 77
TOML 1 4 0 60
CMake 1 4 0 17
Text 1 4 0 17
-------- -------- -------- -------- --------
SUM: 54 1232 496 135990

xiaole @claude
feat(FlexGlue): store keyframe descriptors as FP16 when lossless
b46509d 
The descriptor engine runs in FP16, so its FP32 output is normally exact in
FP16. Store it as FP16 in that case (4.2 MB -> 2.1 MB per keyframe), keep
FP32 otherwise, and convert back to FP32 before matching. Maps saved before
this change (FP32) load unchanged. Matches are identical for live, FP16-stored
and FP32-stored features.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown
cloc github.com/AlDanial/cloc v 2.04 T=0.54 s (99.1 files/s, 252735.6 lines/s)
Language files blank comment code
JSON 3 8 0 129481
Python 19 796 396 4523
Markdown 5 193 6 476
C++ 4 60 38 409
CSS 1 58 0 342
Bourne Shell 16 42 9 281
Dockerfile 1 31 32 205
HTML 1 6 0 107
make 1 26 16 77
TOML 1 4 0 60
CMake 1 4 0 17
Text 1 4 0 17
-------- -------- -------- -------- --------
SUM: 54 1232 497 135995

@xiaolefang-dm
xiaolefang-dm marked this pull request as ready for review November 10, 2026 07:35
xiaole @claude
build(FlexGlue): set matcher attention to BF16 with trtexec layer pre…
420494a 
…cisions

Instead of rewriting the attention blocks in the ONNX, build the matcher
without --stronglyTyped and pin the attention layers (QK^T, Softmax, PV) to
BF16 with --precisionConstraints=obey --layerPrecisions. The ONNX patch now
only replaces CumSum, which TensorRT 10.3 on Jetson still miscomputes.

On the saved failing inputs: no NaN on RTX 3060 (TRT 10.13) or Orin Nano
(TRT 10.3); matcher time unchanged. Because the rest of the network may now
run in FP16, match counts are 1-2% lower on 3060 and 5-10% lower on Orin than
with ONNX Runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown
cloc github.com/AlDanial/cloc v 2.04 T=0.42 s (128.7 files/s, 328145.8 lines/s)
Language files blank comment code
JSON 3 8 0 129481
Python 19 795 392 4487
Markdown 5 193 6 476
C++ 4 60 38 409
CSS 1 58 0 342
Bourne Shell 16 42 9 281
Dockerfile 1 31 32 205
HTML 1 6 0 107
make 1 26 17 79
TOML 1 4 0 60
CMake 1 4 0 17
Text 1 4 0 17
-------- -------- -------- -------- --------
SUM: 54 1231 494 135961

@xiaolefang-dm xiaolefang-dm changed the title fix(FlexGlue): work around TensorRT fusion bugs that zero out matches fix(FlexGlue): BF16 attention and TRT 10.3 CumSum workaround for zero matches; store descriptors as FP16 Nov 10, 2026
@xiaolefang-dm xiaolefang-dm changed the title fix(FlexGlue): BF16 attention and TRT 10.3 CumSum workaround for zero matches; store descriptors as FP16 fix(FlexGlue): fix TensorRT zero-match engines; store descriptors as FP16 Nov 10, 2026
xiaole @claude
ci: pin ruff to its classic default rules
ce454b6 
The quality check runs the latest ruff through uvx; ruff 0.17 enables many
more rules by default and fails on 90 existing findings (main has failed
since October). Selecting E4/E7/E9/F keeps the check stable across ruff
releases and matches what the dev container's ruff 0.15 enforces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown
cloc github.com/AlDanial/cloc v 2.04 T=0.37 s (145.6 files/s, 371179.3 lines/s)
Language files blank comment code
JSON 3 8 0 129481
Python 19 795 392 4487
Markdown 5 193 6 476
C++ 4 60 38 409
CSS 1 58 0 342
Bourne Shell 16 42 9 281
Dockerfile 1 31 32 205
HTML 1 6 0 107
make 1 26 17 79
TOML 1 5 1 62
CMake 1 4 0 17
Text 1 4 0 17
-------- -------- -------- -------- --------
SUM: 54 1232 495 135963

@dvorak0
dvorak0 merged commit 3c21f86 into main Nov 10, 2026
3 of 6 checks passed
hitomi-krgr added a commit that referenced this pull request Nov 10, 2026
@hitomi-krgr
Merge upstream main: FlexGlue TensorRT zero-match fix, FP16 descripto…
4d16427 
…r storage (#261)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dvorak0 dvorak0 Awaiting requested review from dvorak0

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xiaolefang-dm @dvorak0