Fix CPU FlashAttention disabled on Linux/aarch64 by detecting L2 cache via cpuinfo#29621
Open
Sammy-Dabbas wants to merge 1 commit into
Open
Fix CPU FlashAttention disabled on Linux/aarch64 by detecting L2 cache via cpuinfo#29621Sammy-Dabbas wants to merge 1 commit into
Sammy-Dabbas wants to merge 1 commit into
Conversation
…e via cpuinfo glibc's aarch64 sysconf backend does not implement the cache-size queries, so PosixEnv::GetL2CacheSize() returns 0 and the l2_cache_size_ > 0 gate in MultiHeadAttention silently disables CPU FlashAttention. Query cpuinfo for the L2 size first, mirroring the existing cpuinfo_available_ usage in this file for physical-core count and thread affinity, and fall back to sysconf/sysctl. Guarded by ORT_USE_CPUINFO; x86-64 and Windows are unaffected.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a Linux/aarch64 performance/functional regression where PosixEnv::GetL2CacheSize() returns 0 (due to glibc sysconf(_SC_LEVEL2_CACHE_SIZE) not being implemented on aarch64), which in turn disables L2-tiled CPU kernels like the CPU FlashAttention path in com.microsoft.MultiHeadAttention and leads to large O(S²) memory fallbacks.
Changes:
- Prefer cpuinfo-based L2 cache size detection (when available) in
PosixEnv::GetL2CacheSize(). - Keep existing
sysconf/sysctlfallbacks when cpuinfo is unavailable or returns 0.
Comment on lines
+311
to
+316
| if (cpuinfo_available_ && cpuinfo_get_l2_caches_count() > 0) { | ||
| const auto* l2_cache = cpuinfo_get_l2_cache(0); | ||
| if (l2_cache != nullptr && l2_cache->size > 0) { | ||
| return static_cast<int>(l2_cache->size); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Linux/aarch64,
PosixEnv::GetL2CacheSize()returnssysconf(_SC_LEVEL2_CACHE_SIZE), but glibc's aarch64 sysconf backend does not implement the cache-size queries and returns 0. That makes thel2_cache_size_ > 0gate in MultiHeadAttention always false, so the CPU FlashAttention path is silently disabled and the kernel falls back to materializing the full attention score tensor (multi-GiB at long sequence lengths, and OOM on small ARM instances). GroupQueryAttention divides by the same 0 and ends up with a degenerate tile size of 1.This queries cpuinfo for the L2 cache size before falling back to sysconf/sysctl, mirroring the existing
cpuinfo_available_usage in the same file for physical-core count and thread affinity. cpuinfo reads sizes from sysfs cacheinfo and works on aarch64. The change is guarded byORT_USE_CPUINFOand only returns early on a positive result, so x86-64 (working sysconf) and Windows (separateWindowsEnvimplementation) are unaffected.Fixes #29613.