Vision Transformer

A Vision Transformer applies Transformer blocks to image or grid patches. It replaces convolutional locality with tokenization plus attention over patches.

Patch embedding is:

$$x_i=\mathrm{Flatten}(P_i)W_E+p_i$$

where $P_i$ is image patch $i$, $W_E$ is the patch projection, and $p_i$ is a positional embedding.

For an image $X\in {\mathbb{R}}^{H\times W\times C}$ and patch size $P$, the number of tokens is:

$$N=\frac{H}{P}\cdot \frac{W}{P}$$

Self-attention then scales as:

$$O(N^{2}d)$$

so smaller patches preserve more detail but increase attention cost. A class token or pooled patch representation is commonly used for image-level prediction:

$$z=h_{\mathrm{cls}}\text{or}z=\frac{1}{N}\sum _{i=1}^N{h}_i$$

Inductive Bias

Compared with CNNs, ViTs encode less hard locality and more global token mixing. This can be useful with large pretraining, but weak locality can be data-hungry. Hybrid stems, window attention, hierarchical patch merging, or strong augmentation often reintroduce local bias.

Why It Matters

• Shows that Transformers can process non-text token sequences.
• Useful as a reference for patch, voxel, contact-map, or grid tokenization.
• Often requires more data or stronger pretraining than CNNs when locality is not hard-coded.

Architecture Decomposition

A ViT block can be read as one instance of a broader MetaFormer-style pattern:

$$X\to \text{token mixer}\to \text{channel MLP}.$$

For ViT, the token mixer is self-attention. Other vision backbones replace that slot with pooling, convolution, window attention, or token-axis MLPs. The canonical paper note for this scaffold view is MetaFormer.

Backbone	Token Mixer
ViT	global self-attention
Swin Transformer	shifted-window attention
MLP-Mixer	token-axis MLP
PoolFormer	pooling
ConvNeXt	depthwise convolution

Checks

• What is the patch size, and what spatial detail is lost?
• Are positional encodings absolute, relative, or learned?
• Is locality recovered through hybrid CNN stems, window attention, or data scale?
• Is the paper changing attention itself, or changing the broader block scaffold and training recipe?

Related

• Image
• Video
• Transformer
• Tokenization
• CNN
• Embedding
• Pooling and readout