T5

The paper turns diverse NLP tasks into a single text-to-text format and uses it to study transfer learning with encoder-decoder Transformers.

Metadata

Field	Value
Paper	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Authors	Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
Year	2020
Venue	JMLR 2020
arXiv	1910.10683
JMLR	v21/20-074
Official implementation	google-research/text-to-text-transfer-transformer
Status	full note started

One-Line Takeaway

T5 makes the encoder-decoder Transformer a universal text interface: every task is cast as text input to text output, trained with the same sequence-to-sequence objective.

Question

By 2019, transfer learning in NLP had many partially incompatible recipes:

• encoder-only masked language models such as BERT;
• decoder-only autoregressive language models such as GPT-2;
• encoder-decoder sequence-to-sequence models;
• task-specific classification heads;
• different pretraining objectives;
• different datasets and fine-tuning formats.

The paper asks:

Can we put transfer learning for NLP into one unified framework and systematically study what matters?

The answer is the text-to-text framework:

$$\text{input text}\to \text{target text}.$$

For example:

Task	Input Text	Target Text
translation	translate English to German: ...	translated sentence
summarization	summarize: ...	summary
classification	sst2 sentence: ...	positive or negative
question answering	question: ... context: ...	answer string

This is an interface decision, but it has architecture consequences: the natural backbone is an encoder-decoder Transformer.

Main Claim

A unified text-to-text encoder-decoder Transformer can support many NLP tasks under one objective:

$$P(y\mid x)=\prod _{t=1}^{T}P(y_t\mid y_{<t},x).$$

Training minimizes token-level negative log likelihood:

$$L=-\sum _{t=1}^T\log P_{\theta }(y_t\mid y_{<t},x).$$

The important architectural claim is not merely “use a Transformer.” It is:

$$\text{task-specific heads}\to \text{text decoder}.$$

The model always predicts text tokens. Task identity is represented through text prefixes rather than through separate output modules.

Architecture Contract

Component	Role
text prefix	names the task or prompt format
encoder	reads the full input text bidirectionally
decoder	autoregressively generates target text
cross-attention	conditions decoder on encoder states
span corruption objective	pretraining target
C4 corpus	large cleaned text pretraining data
text-to-text format	unifies fine-tuning tasks

At inference:

$$x\stackrel{\text{encoder}}{\to }H\stackrel{\text{decoder}}{\to }{y}_1,\dots ,y_T.$$

Encoder-Decoder Transformer

The T5 backbone is an encoder-decoder Transformer.

The encoder maps input tokens:

$$x_1,\dots ,x_n$$

to hidden states:

$$H=\mathrm{Encoder}(x_{1:n}).$$

The decoder generates output tokens autoregressively:

$$s_t=\mathrm{Decoder}(y_{<t},H).$$

Then:

$$P(y_t\mid y_{<t},x)=\mathrm{softmax}(W_o{s}_t).$$

This differs from BERT:

$$\text{BERT}:x\to \text{representations or classifier head},$$

and from GPT-style models:

$$\text{GPT}:\text{prefix}\to \text{continuation}.$$

T5 explicitly uses:

$$\text{source sequence}\to \text{target sequence}.$$

Self-Attention and Cross-Attention

The encoder uses bidirectional self-attention:

$$H={\mathrm{SelfAttn}}_{\text{bidirectional}}(X).$$

Each decoder layer has:

1. causal self-attention over previous target tokens;
2. cross-attention to encoder states;
3. feed-forward block.

Decoder self-attention:

$$S_t=\mathrm{CausalSelfAttn}(Y_{<t}).$$

Cross-attention:

$$\mathrm{CrossAttn}(S,H)=\mathrm{softmax}\left(\frac{(S{W}^Q)(H{W}^K{)}^{\top }}{\sqrt{d_h}}\right)H{W}^V.$$

This makes T5 a natural architecture for tasks where the output is a transformation of an input, not just a continuation of a prefix.

Text-to-Text Interface

The central design is:

$${\mathcal{T}}_k(x)\to y,$$

where ${\mathcal{T}}_k$ formats task $k$ as text.

For classification, instead of using:

$$y=\mathrm{softmax}(W{h}_{\text{CLS}}),$$

T5 uses:

$$P(\text{“positive”}\mid x)$$

or:

$$P(\text{“negative”}\mid x).$$

This unifies tasks but changes the evaluation contract. Labels become strings, and decoding choices can affect results.

Task Prefixes

A task prefix is ordinary input text:

$$x^{\prime }=\text{prefix}\Vert x.$$

Examples:

Prefix Role	Example
translation task	translate English to German:
summarization task	summarize:
classification task	dataset/task name
QA task	question: and context: fields

This makes task conditioning part of the sequence, not a separate architectural head.

For later instruction-tuned models, this is a precursor to prompt-based task specification.

Pretraining Objective

T5 uses a denoising span corruption objective. Given input text, randomly replace spans with sentinel tokens:

$$\text{original}:w_1{w}_2{w}_3{w}_4{w}_5{w}_6$$

could become:

$$\text{input}:w_1⟨X⟩w_5⟨Y⟩$$

and the target is:

$$\text{target}:⟨X⟩w_2{w}_3{w}_4⟨Y⟩w_6.$$

The model learns:

$$P(\text{missing spans}\mid \text{corrupted text}).$$

This is similar in spirit to masked modeling, but the decoder generates missing spans as a sequence.

Span Corruption as Architecture Fit

Span corruption fits the encoder-decoder architecture:

• encoder reads corrupted input bidirectionally;
• decoder generates missing spans autoregressively;
• sentinel tokens mark missing segments;
• loss is standard sequence-to-sequence cross entropy.

The objective:

$$L_{\text{span}}=-\sum _t\log P_{\theta }(y_t\mid y_{<t},\tilde{x}),$$

where $\tilde{x}$ is corrupted input and $y$ is the missing-span target.

This avoids having a special masked-token classification head. Everything is still text-to-text.

C4 Dataset

The paper also introduced the Colossal Clean Crawled Corpus:

$$\text{C4}=\text{cleaned Common Crawl text}.$$

For architecture reading, C4 matters because model comparisons are not just architecture comparisons. They also depend on:

• data source;
• cleaning;
• deduplication;
• task mixture;
• pretraining budget;
• fine-tuning setup.

T5 is therefore both an architecture paper and a controlled transfer-learning study.

Comparison to BERT

| Property | BERT | T5 | | --- | --- | --- | | Backbone | encoder-only Transformer | encoder-decoder Transformer | | Pretraining | masked LM and next-sentence style objective | span corruption denoising | | Output | representations plus task heads | generated text | | Fine-tuning | often task-specific classifier/head | text-to-text targets | | Strength | understanding/representation tasks | sequence transformation and unified task interface | | Limitation | not naturally generative | decoding cost and label-string design |

BERT is an encoder representation model. T5 is a sequence-to-sequence transfer model.

Comparison to GPT-2

| Property | GPT-2 | T5 | | --- | --- | --- | | Backbone | decoder-only Transformer | encoder-decoder Transformer | | Objective | autoregressive next-token prediction | denoising and text-to-text seq2seq | | Conditioning | prefix context | encoder input plus decoder history | | Task interface | prompting/continuation | task prefix and target text | | Good fit | open-ended generation | input-conditioned generation and transfer tasks |

GPT-style models later became dominant for instruction-following, but T5 remains a clean reference for encoder-decoder text-to-text design.

Comparison to Original Transformer

Attention Is All You Need introduced the encoder-decoder Transformer for machine translation:

$$\text{source sentence}\to \text{target sentence}.$$

T5 generalizes the interface:

$$\text{any text-formatted task input}\to \text{any text-formatted target}.$$

The architecture family is similar, but the contribution is the unified transfer-learning setup and systematic ablation study.

What Is Architecture and What Is Methodology?

T5 is not only an architecture proposal. It is a systematization paper. For this paper shelf, it belongs because it fixed a reusable architecture contract:

$$\text{encoder-decoder Transformer}+\text{text-to-text interface}+\text{span corruption pretraining}.$$

The long-term reusable part is the interface:

$$\text{task-specific output heads}\to \text{natural language target strings}.$$

That idea influenced later instruction tuning, prompt formatting, and unified task modeling.

Evidence Reading

The paper systematically compares objectives, architectures, datasets, and transfer procedures. The evidence is strongest when read as a controlled study of NLP transfer learning rather than as one isolated architecture benchmark.

Claim	Evidence Type	What It Supports	Caveat
Text-to-text format is broadly usable	many NLP tasks cast into one format	unified interface works	tasks still need careful formatting
Encoder-decoder performs well for transfer	architecture comparisons	seq2seq backbone is robust across tasks	compute/data choices matter
Span corruption is effective	pretraining objective ablations	denoising objective fits text-to-text learning	not a universal objective for all modalities
Scale and clean data matter	larger models and C4 results	final performance depends on more than architecture	hard to isolate architecture from data/compute

Benchmark Card

Field	Value
Main task family	NLP transfer learning
Input unit	text-formatted task input
Output unit	target text sequence
Architecture family	encoder-decoder Transformer
Main objective	span corruption pretraining and seq2seq fine-tuning
Main comparison	BERT-like, GPT-like, seq2seq, objectives, datasets
Key artifact	C4 and T5 model family
Not the claim	one architecture always dominates all decoder-only LMs

Implementation Notes

Label Strings

Classification targets are text strings:

$$y\in \{\text{“positive”},\text{“negative”}\}.$$

Changing label words can change behavior. A robust implementation must define canonical target strings.

Decoding

Even classification can involve decoding. In practice, one may:

• decode greedily;
• score candidate label strings;
• constrain output vocabulary;
• use beam search for generative tasks.

The decoding method is part of the evaluation contract.

Task Mixtures

Multi-task training needs a mixture distribution:

$$p(k)$$

over tasks. The paper studies how mixing affects transfer. This matters because unified format does not automatically solve task imbalance.

Sequence Length

Encoder-decoder models have separate input and output lengths:

$$n_{\text{input}},n_{\text{target}}.$$

Training cost includes encoder self-attention, decoder self-attention, and cross-attention.

Roughly:

$$O(n_{\text{input}}^{2}d)+O(n_{\text{target}}^{2}d)+O(n_{\text{input}}{n}_{\text{target}}d).$$

Molecular and Scientific Modeling Reading

T5 is an NLP paper, but the text-to-text interface is relevant to scientific AI when tasks can be serialized:

• molecule strings such as SMILES;
• protein sequences;
• natural-language assay descriptions;
• task prompts for property prediction;
• sequence-to-sequence transformations;
• structured report generation.

However, serialization is not the same as structure modeling. For protein-ligand geometry, T5-style text modeling does not encode 3D equivariance, graph topology, or physical constraints by itself.

Useful pattern:

$$\text{scientific input serialized as text}\to \text{text output or structured string}.$$

Potential mismatch:

$$\text{3D structure}\not\equiv \text{text string}.$$

For structural tasks, T5 should be considered an interface/modeling route, not a replacement for graph or geometric architectures.

Common Misreadings

”T5 is just another Transformer.”

The backbone is a Transformer, but the paper’s durable contribution is the unified text-to-text task interface plus systematic transfer study.

”Text-to-text removes task design.”

No. It moves task design into prefixes, target strings, decoding, and data mixtures.

”Classification as generation is always better.”

Not necessarily. It is elegant and unified, but classifier heads can be simpler and cheaper.

”T5 and GPT are interchangeable.”

Both generate text, but the architecture and conditioning are different. T5 uses an encoder-decoder; GPT uses a decoder-only prefix LM.

Later-Paper Checklist

When reading later instruction-tuning, unified-task, or encoder-decoder papers, ask:

• Are tasks truly unified or only wrapped in text?
• What target strings define labels?
• Is the model encoder-decoder, decoder-only, or encoder-only?
• What pretraining objective is used?
• Is decoding constrained for classification?
• How are tasks mixed during training?
• Are improvements from architecture, objective, data, scale, or formatting?
• Does the task require non-text structure that serialization may lose?

Why It Matters

T5 is important because it clarified a reusable modeling contract:

$$\text{all tasks as text-to-text}$$

with:

$$\text{encoder-decoder Transformer}+\text{span corruption}+\text{transfer fine-tuning}.$$

It belongs in architecture papers because it is the canonical reference for encoder-decoder Transformer transfer learning after the original Transformer.

Limitations

• Text-to-text formatting can hide task-specific design choices.
• Decoder generation is more expensive than a classifier head for simple labels.
• Results depend heavily on data, scale, objective, and mixture.
• The architecture is text-centric and does not directly solve graph or 3D geometric modeling.
• Label string choice and decoding strategy can affect evaluation.

Connections

• Encoder-decoder
• Transformer
• Encoder-only Transformer
• Decoder-only Transformer
• Masked modeling
• Language model
• Attention Is All You Need
• BERT
• GPT-2
• Architecture papers