composer.algorithms.alibi#
Modules
Core ALiBi classes and functions. |
ALiBi (Attention with Linear Biases; Press et al, 2021) dispenses with position embeddings for tokens in transformer-based NLP models, instead encoding position information by biasing the query-key attention scores proportionally to each token pairโs distance.
See the Method Card for more details.
Functions
Removes position embeddings and replaces the attention function and attention mask as per |
Classes
ALiBi (Attention with Linear Biases; Press et al, 2021) dispenses with position embeddings and instead directly biases attention matrices such that nearby tokens attend to one another more strongly. |