Conversation
github-actions
Bot
added
documentation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Didn't review it fully but some changes are too intrusive, IMO. Hopefully, the existing comments will be helpful to get an idea of what I am talking about and help you make changes to the files I didn't review yet.
|
|
||
| ## Optimization | ||
|
|
||
| You can optimize the pipeline's runtime and memory consumption with torch.compile and feed-forward chunking. To learn about other optimization methods, check out the [Speed up inference](../../optimization/fp16) and [Reduce memory usage](../../optimization/memory) guides. |
There was a problem hiding this comment.
I think precision and compilation as a title is less exciting as a reader in the sense that it doesn't tell me that it's related to speeding up of inference.
| <hfoption id="memory"> | ||
|
|
||
| Refer to the [Reduce memory usage](../../optimization/memory) guide for more details about the various memory saving techniques. | ||
| Refer to the [Memory and offloading](../../optimization/memory) guide for more details about the various memory saving techniques. |
There was a problem hiding this comment.
I think this change makes it less obvious to the reader the page is about something they would actually care about (i.e., reducing memory usage).
| > The attention dispatcher is an experimental feature. Please open an issue if you have any feedback or encounter any problems. | ||
|
|
||
| Diffusers provides several optimized attention algorithms that are more memory and computationally efficient through it's *attention dispatcher*. The dispatcher acts as a router for managing and switching between different attention implementations and provides a unified interface for interacting with them. | ||
| Diffusers routes attention through an *attention dispatcher* so you can switch optimized backends behind one API. The dispatcher manages registered implementations and exposes a unified call path for them. |
There was a problem hiding this comment.
Some autoencoders don't use the dispatcher and its because of their internals. So, we cannot sufficiently claim that Diffusers always routes.
| Diffusers routes attention through an *attention dispatcher* so you can switch optimized backends behind one API. The dispatcher manages registered implementations and exposes a unified call path for them. | ||
|
|
||
| Refer to the table below for an overview of the available attention families and to the [Available backends](#available-backends) section for a more complete list. | ||
| Refer to the table below for an overview of the available attention families and to the [Available backends](#available-backends) section for a more complete list. The fastest backend depends on the model, GPU, and dtype. |
There was a problem hiding this comment.
The input payload shape as well.
| This guide will show you how to set and use the different attention backends. | ||
| Install each backend’s own package before you enable it. The [Available backends](#available-backends) table lists package requirements Diffusers checks at enable time, plus hardware targets where they matter. |
There was a problem hiding this comment.
We probably shouldn't say things like "install each package" because many attention backends are available through kernels and statements like this undermine that.
| > [!NOTE] | ||
| > FlashAttention-3 requires Ampere GPUs at a minimum. | ||
| The example below enables `_flash_3_hub` (FlashAttention-3 from the Hub) with `device_map="cuda"` only. FlashAttention-3 targets Hopper GPUs (for example H100 or H800). Prefer FlashAttention-2 backends such as `flash` or `flash_hub` on Ampere or Ada. |
There was a problem hiding this comment.
No. Flash3 is available on Amperes, too.
| ``` | ||
|
|
||
| To restore the default attention backend, call [`~ModelMixin.reset_attention_backend`]. | ||
| The non-Hub FlashAttention-3 backends (`_flash_3`, `_flash_varlen_3`) require building FlashAttention-3 from source. Prefer `_flash_3_hub` (or `_flash_3_varlen_hub`) when you want the Hub path with Kernels. |
There was a problem hiding this comment.
We can also add a note that these will be deprecated soon.
This branch has not been deployed
3 participants
Refreshes the Optimize and scale section:
TextKVCache(can remove if this is too niche, but good to document for completeness)