SIMD execution in Comet
#1196
|
In the architecture diagram of Comet documentation, it mentions that there is SIMD accelerated columnar execution enabled. On this front I have a few queries.
|
Answered by
andygrove
Jan 24, 2025
Replies: 1 comment 1 reply
|
@ajeyabsfujitsu Sorry, I did not see this discussion until now. DataFusion uses SIMD by leveraging the Rust compiler's auto-vectorization capabilities. Instead of manually writing SIMD intrinsics, it relies on the compiler to optimize data-parallel operations (like filtering, projections, and aggregations) during code generation. This approach ensures portability across architectures while still benefiting from hardware acceleration. Comet uses DataFusion for native query exection and therefore benefits from this SIMD support. |
1 reply
|
Got it. Thank you for the response @andygrove. |
Answer selected by
ajeyabsf
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@ajeyabsfujitsu Sorry, I did not see this discussion until now.
DataFusion uses SIMD by leveraging the Rust compiler's auto-vectorization capabilities. Instead of manually writing SIMD intrinsics, it relies on the compiler to optimize data-parallel operations (like filtering, projections, and aggregations) during code generation. This approach ensures portability across architectures while still benefiting from hardware acceleration.
Comet uses DataFusion for native query exection and therefore benefits from this SIMD support.