Accuracy issues moving from AIMET-ONNX to SNPE DLC

[Petros Toupas (ptoupas)]

Hi AIMET team,

I’m experiencing a big accuracy gap between the QDQ ONNX model exported (through the to_onnx_qdq()) and tested with onnxruntime, and the DLC I generate with SNPE using the .onnx and .encodings files produced by the same QuantizationSimModel instance using the export() method and tested with the snpe-net-run.
I am working with the aimet-onnx==2.10.0 and the SNPE==2.32.6 versions of the tools.

I am trying to quantize a pose estimation model (Lite-HRNet) and I am visually evaluating the results on a set of images. Following the exact same pre and post processing steps I am getting almost identical results between the FP32 ONNX, the QDQ ONNX (produced by AIMET), and the FP16 converted DLC. But when I try the INT8 quantized DLC (converted using the .onnx and .encodings from AIMET) the results are way off.

Here is the pipeline I’m following:

1. Create QuantizationSimModel, and prepare the respective Iterable with the calibration data.
2. Run the compute_encodings().
3. Get the QDQ ONNX from the to_onnx_qdq() and the .onnx and .encodings files from the export().
4. Test the accuracy of the QDQ ONNX model using the onnxruntime. If the accuracy is good enough proceed to next step.
5. Run the snpe-onnx-to-dlc -i aimet.onnx --quantization_overrides aimet.encodings -> fp32.dlc
6. Run the snpe-dlc-quant --input_dlc fp32.dlc --override_params --bias_bitwidth 32 (I have tried adding or removing --float_fallback and using or omitting --input_list and all the possible combinations of them).
7. Run the snpe-dlc-graph-prepare on the resulting DLC.
8. Execute the model using the snpe-net-run and get the results back to evaluate on the same set of images.

Things I have already tried

• Running snpe-dlc-quant with and without --input_list (calibration data).
• Running it with and without --float_fallback.
• Matching the bit-width flags to my AIMET defaults (--weights_bitwidth 8 --act_bitwidth 16).
• Keeping biases in FP-32 with --bias_bitwidth 32.

Questions

1. Is this the right workflow, or am I missing or miss-configuring any step?
2. How exactly do --input_list and --override_params interact—does SNPE still recalibrate tensors that are already in .encodings file? What impact does other flags I might use in this step have, like --use_per_channel_quantization, or --weights_bitwidth, --act_bitwidth and so on?
3. What happens to tensors that are not present in the overrides file—do they get quantised with a different scheme that could explain the accuracy drop?
4. Can I force AIMET to export encodings for every single weight and activation so nothing is left for SNPE to decide?
5. If the QDQ ONNX model is accurate in onnxruntime, should the DLC be expected to match once all overrides are honoured, or are there other pitfalls?

I have searched through existing discussions and issues, but most similar posts involve PyTorch models instead of ONNX or use QNN rather than SNPE for deployment; I have tried the suggestions from those threads without success, so I’m opening this new discussion.
Any guidance would be greatly appreciated!

[Abhi Khobare (quic-akhobare)]

Hi Petros Toupas (@ptoupas) - Thanks for reaching out to us. And apologies for a delayed response.
Your workflow looks right to me. I am not familiar with all the nitty gritty details of the different SNPE flags. But I am guessing there is either a bug in SNPE, or there is some subtle interplay with the flags being used (either some flag is missing when calling SNPE or some flag is extra).

[Abhi Khobare (quic-akhobare)]

I think the best way forward would be to reach the SNPE support forums/team. They don't look at AIMET github discussions. So let us find the right link for you to follow up. We will get back to you shortly.

[Petros Toupas (ptoupas)]

Hi Abhi Khobare (@quic-akhobare),
Thank you for your response. I'll wait for you to come back then with a link and a contact point to discuss and share that with the SNPE team.

In the meantime, I'd like to double confirm with you about question 5 in my post above. I've been checking the quality of the AIMET quantization through the execution of the exported QDQ ONNX model in onnxruntime and not via the QuantizationSimModel.session.run() after the encodings have been computed. As far as I understand that should be the same internally, but I want to double check and ensure that. Should there two approached considered "same" and is it safe to assume the quality of the quantized model by checking the exported QDQ model in onnxruntime?

[Michael Tuttle (quic-mtuttle)]

Hi Petros Toupas (@ptoupas), yes that is correct, evaluating the model returned by sim.to_onnx_qdq() should be essentially the same as evaluating QuantizationSimModel.session and is a good way to verify the accuracy of your quantized model after export.

[Abhi Khobare (quic-akhobare)]

So here is the support forums link for SNPE (Qualcomm Neural Processing SDK), QNN (Neural Network SDK), QAIRT etc.
https://mysupport.qualcomm.com/supportforums/s/topic/0TO4V0000001XL9WAM/ai-sdks-tools

Note: Please don't use the above for AIMET questions. Just create discussion threads here for AIMET.

[Petros Toupas (ptoupas)]

Thank you for providing that. Much appreciated! 🙏

[frisk99 (frisk99)]

Hi
I also meet this problem on QCS8550.I have determined that it is not the format (NHWC) problem of model input.
How did you solve the problem?

[Petros Toupas (ptoupas)]

Hi frisk99 (@frisk99),

I still haven’t been able to resolve the issue. According to the QC support team, “there seems to be a bug when changing the datatype during the quantization phase.” As far as I know, their development team is working on a fix, but I haven’t received any updates from them yet.

[frisk99 (frisk99)]

Thanks for your reply, snpe's own problems seem insoluble to me😭

[xtlan (a122760)]

一模一样的问题，我这边是aimet-->qnn/qairt，目前已知问题是aimet和qnn算子覆盖不一致，aimet比如把(a+b)当成整体去量化，没什么掉点，但是到qnn，它可能拆成a+b两个op，然后精度崩了，但是至今我都没有搞定量化精度，及时在qnn阶段重新用数据跑ptq校准。真的很难受，这个工具还有人跟进吗？