composer.algorithms.alibi#
Modules
Core ALiBi classes and functions. |
ALiBi (Attention with Linear Biases; Press et al, 2021) dispenses with position embeddings for tokens in transformer-based NLP models, instead encoding position information by biasing the query-key attention scores proportionally to each token pairโs distance.
See the Method Card for more details.
Functions
Removes position embeddings and replaces the attention function and attention mask according as per |
Classes
ALiBi (Attention with Linear Biases; Press et al, 2021) dispenses with position embeddings and instead directly biases attention matrices such that nearby tokens attend to one another more strongly. |