-
Notifications
You must be signed in to change notification settings - Fork 232
Use kitchen FA in huggingface plugin #674
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #674 +/- ##
==========================================
- Coverage 74.69% 74.67% -0.03%
==========================================
Files 192 192
Lines 18948 18955 +7
==========================================
+ Hits 14153 14154 +1
- Misses 4795 4801 +6 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Shiyang Chen <[email protected]>
Signed-off-by: Shiyang Chen <[email protected]>
Signed-off-by: Shiyang Chen <[email protected]>
realAsma
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks great!
We would need to support kitchen FA in vLLM and MCore as well to fully attention quantization with training and long contact inference. We can take care of that in a separate PR. Thanks!
What does this PR do? new feature
Overview: use kitchen FA in huggingface plugin
Usage
# Add a code snippet demonstrating how to use thisTesting
Before your PR is "Ready for review"
Additional Information