Protein Language Model

A protein language model treats an amino-acid sequence as a token sequence and learns statistical structure from large protein databases. It is useful for representation learning, variant effect prediction, function prediction, structure-aware transfer, and protein design.

For a protein sequence:

$$s=(a_1,\dots ,a_L),a_i\in {\mathcal{V}}_{\mathrm{AA}}$$

a language model learns either an autoregressive distribution:

$$p_{\theta }(s)=\prod _{i=1}^L{p}_{\theta }(a_i\mid a_{<i})$$

or a masked-token objective:

$${\mathcal{L}}_{\mathrm{MLM}}=-\sum _{i\in M}\log p_{\theta }(a_i\mid s_{\setminus M})$$

where $M$ is the set of masked residue positions.

What It Learns

Signal	Where it comes from	What it may support
residue statistics	large sequence databases	family, motif, and conservation features
contextual dependencies	neighboring and distant residues	function and variant effect prediction
evolutionary constraints	homologous sequence distribution	structure and contact-related transfer
token embeddings	hidden states	residue, domain, or chain representation

A protein language model does not directly observe experimental binding, activity, or structure unless those are added through supervision, retrieval, templates, or downstream data.

Representation Levels

Token embeddings:

$$h_i=f_{\theta }(s{)}_i$$

Sequence-level embedding:

$$z_s=\mathrm{pool}(h_1,\dots ,h_L)$$

Pair or contact features may be derived from attention maps, outer products, or additional heads, but those features require separate validation before being treated as structural evidence.

Common Uses

Use	Output	Main risk
residue annotation	$p(y_i\mid s)$	label imbalance and homolog leakage
protein classification	$p(y\mid s)$	family memorization
variant scoring	$\Delta \log p_{\theta }$ or supervised score	assay and species shift
structure prediction input	sequence/MSA/template features	template and database leakage
generative design	sampled sequences	validity is not functional success

Likelihood and Biological Fitness

Sequence likelihood is not the same as biological fitness:

$$\log p_{\theta }(s)\ne \mathrm{fitness}(s)$$

High likelihood can mean that a sequence is common under the training distribution, not that it binds, folds, expresses, or performs the desired function. A paper that uses likelihood as a design score should connect it to downstream evidence.

Evaluation Checks

• Are train and test proteins separated by Protein family split or sequence identity?
• Is the model evaluated on sequence likelihood, representation transfer, structure, function, or design?
• Are database release dates controlled when the benchmark contains recent proteins?
• Does the model use only sequence, or also Multiple sequence alignment, templates, predicted structures, or labels?
• Are metrics reported at the residue, sequence, family, or assay-row level?
• Does the claim rely on likelihood, supervised downstream performance, or experimental validation?

Related

• Protein
• Sequence
• Protein representation
• Multiple sequence alignment
• Protein domain
• Protein family split
• Self-supervised learning
• Sequence generation
• Protein modeling