Encoder-Only Transformer
An encoder-only Transformer uses bidirectional self-attention to build contextual representations. It is commonly used for classification, retrieval, masked modeling, and representation learning.
The self-attention mask usually allows all non-padding tokens to attend to each other:
The output can be token-level states or a pooled representation:
The bidirectional attention update is:
where masks padding but does not impose causal order. This makes every visible token representation depend on both left and right context:
Some encoder-only variants change how position enters this attention equation. DeBERTa, for example, separates content vectors from relative position vectors when computing attention logits:
This keeps the encoder-only contract while changing the position/content factorization inside attention.
Objective Fit
Encoder-only models are good when the downstream task needs representations rather than left-to-right generation. Common objectives include:
- Masked modeling, where hidden tokens are reconstructed from context.
- Contrastive or retrieval objectives, where sequence-level vectors are compared.
- Token classification, where every token/residue/node gets a label.
- Pair or cross-input scoring, where pooled representations feed a classifier or ranker.
For protein and molecule use, readout details matter: pooling over padding, special tokens, or truncated residues can silently corrupt the representation.
Uses
- Masked language or residue modeling.
- Protein or molecule representation learning.
- Classification and retrieval.
- Feature extraction for downstream models.
Checks
- Is the task representation learning rather than generation?
- How are masked tokens, padding, and special tokens handled?
- Is the readout CLS, mean pooling, attention pooling, or task-specific?