MetaFormer

The paper argues that the general Transformer-style scaffold matters more than the specific attention token mixer for many vision backbones.

Metadata

Field	Value
Paper	MetaFormer Is Actually What You Need for Vision
Authors	Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan
Year	2022
Venue	CVPR 2022
arXiv	2111.11418
CVF	CVPR 2022 paper
Code	sail-sg/poolformer
Status	full note started

Question

Vision Transformers made attention the default explanation for why Transformer-like models work in vision. MLP-like models then showed that attention can be replaced by spatial MLPs while retaining competitive behavior.

MetaFormer asks a sharper question:

$$\text{Is attention the key, or is the Transformer-style block scaffold the key?}$$

The paper deliberately replaces attention with an extremely simple pooling operator. The resulting model, PoolFormer, still performs competitively on vision benchmarks. This supports the paper’s hypothesis:

$$\text{MetaFormer scaffold}>\text{specific token mixer}$$

for a broad class of vision backbones.

Main Claim

The paper abstracts a family of recent vision models into a common block:

$$\text{MetaFormer block}=\text{Norm}\to \text{Token Mixer}\to \text{Residual}\to \text{Norm}\to \text{Channel MLP}\to \text{Residual}.$$

The token mixer can be attention, spatial MLP, convolution, pooling, or another operator. The claim is not that token mixers do not matter. The claim is that the scaffold itself is a strong architecture prior:

$$\text{token mixer choice}\ne \text{entire architecture explanation}.$$

Architecture Contract

Item	Contract
Input	image features or patch tokens
Backbone type	hierarchical MetaFormer-style vision backbone
Token mixer	unspecified in the abstraction; pooling in PoolFormer
Channel mixer	MLP / feed-forward block
Normalization	normalization before mixer blocks
Residual path	residual connections around token mixer and channel MLP
Natural task in paper	image classification, detection, segmentation evidence

The abstract block can be written as:

$$U=X+\mathrm{TokenMixer}({\mathrm{Norm}}_1(X)),$$ $$Y=U+\mathrm{MLP}({\mathrm{Norm}}_2(U)).$$

This is close to the standard Transformer block, except the token mixer is left open:

$$\mathrm{TokenMixer}\in \{\text{attention},\text{pooling},\text{MLP},\text{convolution},\dots \}.$$

PoolFormer as the Test Case

PoolFormer is the deliberately simple instantiation:

$$\mathrm{TokenMixer}(X)=\mathrm{Pool}(X)-X.$$

For a spatial feature map $X$, average pooling computes local neighborhood statistics:

$$\mathrm{Pool}(X{)}_{u,v,c}=\frac{1}{|\mathcal{N}(u,v)|}\sum _{(i,j)\in \mathcal{N}(u,v)}{X}_{i,j,c}.$$

The subtraction keeps the mixer focused on local context difference:

$$\mathrm{PoolMixer}(X)=\mathrm{AvgPool}(X)-X.$$

There are no attention scores, learned convolution kernels, or token-axis MLP weights in this mixer. That makes PoolFormer a controlled architecture probe:

$$\text{if simple pooling works well, the scaffold deserves credit.}$$

MetaFormer vs Transformer

Component	Transformer	MetaFormer
token mixer	self-attention	any token-mixing operator
channel mixer	feed-forward network	MLP / feed-forward network
normalization	usually pre-norm or post-norm	normalization around both mixer stages
residual structure	residual around attention and FFN	residual around token mixer and MLP
claim	attention-based token mixing is central	scaffold may be the central reusable form

This reframes Attention Is All You Need for vision. The original Transformer made attention the dominant sequence mixer. MetaFormer asks whether the architecture skeleton around the mixer is also doing much of the work.

Relation to Nearby Vision Backbones

Paper	Token Mixer	What MetaFormer Helps Ask
Vision Transformer	global self-attention	is global attention necessary or just one mixer?
MLP-Mixer	token-axis MLP	can MLP token mixing replace attention?
Swin Transformer	shifted-window attention	what happens when token mixing is local and hierarchical?
CoAtNet	convolution then relative attention	where should different token mixers appear by stage?
ConvNeXt	depthwise convolution	can modern ConvNets be read as MetaFormer-like blocks?
MetaFormer	abstract token mixer; pooling in PoolFormer	how much performance comes from the scaffold?

This is why MetaFormer belongs in architecture papers. It is less about one model family and more about the decomposition of a modern backbone.

Token Mixer Is a Slot

The key abstraction is to treat token mixing as a replaceable slot:

$$\mathrm{TokenMixer}:{\mathbb{R}}^{N\times C}\to {\mathbb{R}}^{N\times C}.$$

Examples:

Mixer	Operation	Bias
self-attention	content-dependent weighted sum	adaptive global interaction
window attention	content-dependent local/window sum	local adaptive interaction
token MLP	learned fixed token-axis mixing	learned position interaction
depthwise convolution	local learned kernel	locality and translation sharing
pooling	local non-parametric average	simple local smoothing
identity	no token mixing	tests scaffold lower bound

For reading architecture papers, this slot view prevents over-attributing results to a named mechanism. A result may come from the mixer, but it may also come from normalization, residual layout, MLP ratio, hierarchical downsampling, training recipe, or regularization.

Evidence to Read Carefully

The paper uses PoolFormer to test whether a very simple token mixer can still produce competitive vision models. It also compares against Transformer-like and MLP-like baselines.

Read the evidence along these axes:

Evidence	What It Supports
PoolFormer classification results	simple token mixing can be enough inside the scaffold
downstream task transfer	the scaffold is not only an ImageNet classifier trick
comparisons with DeiT and ResMLP	attention and MLP token mixers are not the only viable choices
ablation of block design	normalization/residual/channel MLP are part of the claim

The correct takeaway is:

$$\text{architecture scaffold is an active hypothesis, not neutral boilerplate.}$$

Limits

• Pooling as a token mixer is mainly a proof device, not necessarily the best final design.
• Strong performance still depends on training recipe, stage design, resolution, and data.
• The claim is most directly about vision backbones, not every Transformer use case.
• It does not prove attention is unimportant; it shows attention is not the only explanation.
• Later MetaFormer variants can blur the line between concept paper and performance recipe.

What This Paper Teaches

MetaFormer is useful because it gives a compact checklist for architecture reading:

$$\text{model}=\text{tokenization}+\text{stage layout}+\text{token mixer}+\text{channel mixer}+\text{normalization}+\text{residual path}+\text{training recipe}.$$

When a paper claims a new mixer is better, ask:

• Is the scaffold fixed?
• Is the MLP ratio fixed?
• Is the normalization placement fixed?
• Is the stage layout fixed?
• Are parameter count, FLOPs, data, augmentation, and training length matched?
• Does the mixer improve all tasks or only the benchmark shown?

Concepts

• Vision Transformer
• Transformer
• Attention
• Feed-forward network
• MLP
• CNN
• Normalization
• Residual connection
• Architecture selection

Related

• Vision Transformer
• MLP-Mixer
• Swin Transformer
• CoAtNet
• ConvNeXt
• Attention Is All You Need