Architectures

모델 아키텍처는 데이터가 어떤 구조를 갖는지에 대한 가정입니다. 같은 objective를 쓰더라도 sequence, graph, image, 3D structure를 어떻게 표현하느냐에 따라 적합한 architecture가 달라집니다.

형식적으로는 architecture가 가능한 함수 공간 $\mathcal{F}$를 정합니다.

$$f_{\theta }\in {\mathcal{F}}_{\text{architecture}}$$

예를 들어 CNN은 locality와 translation equivariance를 강하게 가정하고, GNN은 graph connectivity와 permutation invariance/equivariance를 가정합니다.

Paper Shelf

개념을 먼저 읽은 뒤 대표 논문으로 내려가려면 아래의 anchor를 시작점으로 사용합니다. 전체 논문 목록은 Architecture Papers에서 route와 shelf별로 확인할 수 있습니다.

Family	Paper anchor	다음에 비교할 것
Transformer	Attention Is All You Need	Universal Transformers, Mamba
CNN and vision	Deep Residual Learning	Vision Transformer, ConvNeXt
Shared representation	Siamese Neural Networks	FaceNet, CLIP, Contrastive learning
Graph and geometry	E(n) Equivariant GNN	Neural Message Passing, AlphaFold2
Generative	Denoising Diffusion Probabilistic Models	Latent Diffusion Models, DiT
Agent and control	DQN	World Models, Dreamer, MuZero, AlphaZero
Architecture search	Neural Architecture Search with Reinforcement Learning	EfficientNet, Architecture search

고르는 기준

아키텍처를 볼 때는 “유명한 모델인가”보다 “어떤 입력 구조와 대칭성을 가정하는가”를 먼저 봐야 합니다.

Criterion	Start	Ask
Inductive bias	Inductive bias	architecture가 어떤 함수 공간을 선호하는가?
Parameter sharing	Parameter sharing	같은 parameter를 어디에 반복 적용하는가?
Architecture choice	Architecture selection	modality, task, data, compute에 맞는 family인가?
Architecture vs objective	Architecture-objective fit	성능 차이가 구조 때문인지 학습 신호 때문인지 분리되는가?
Scaling cost	Computational complexity	sequence length, graph size, image resolution에 따라 비용이 어떻게 느는가?

Architecture Reading Order

아키텍처 노트는 아래 순서로 읽으면 산발적인 모델명이 하나의 체계로 들어옵니다.

1. 입력 단위: feature, token, pixel, node, residue, atom, coordinate 중 무엇인가?
2. 섞는 축: channel, token, node, edge, coordinate, modality 중 무엇을 mix하는가?
3. 공유 규칙: convolution, recurrence, attention, message passing, equivariant update 중 무엇인가?
4. 안정화 장치: normalization, residual, gating, initialization이 어디에 놓이는가?
5. 출력 readout: class, scalar, sequence, graph, coordinate, distribution 중 무엇을 내는가?
6. 비용: sequence length, graph size, resolution, coordinate updates에 대해 어떻게 scaling되는가?

Input Structure Map

Input Structure	Useful Bias	Typical Architecture	Watch
Fixed feature vector	feature mixing	MLP, linear model, tree model	feature scale, missing values, leakage
Grid / image / voxel	local translation sharing	CNN, U-Net, ViT	resolution, receptive field, augmentation
Sequence	order and long-range context	RNN/LSTM/GRU, Transformer, SSM/Mamba	length scaling, positional encoding, truncation
Set	permutation invariance	Deep Sets, Set Transformer	pooling/readout and element identity
Graph	neighborhood message passing or graph-biased attention	GNN, Graph Transformer	graph construction, edge attributes, over-smoothing
3D coordinates	invariance/equivariance	equivariant GNN, tensor-field model	coordinate frame, units, chirality, leakage, update stability
Multimodal input	cross-modal alignment	encoder-decoder, cross-attention, Perceiver	missing modality, modality leakage, fusion timing

Building Block Routes

딥러닝 아키텍처를 읽을 때 먼저 봐야 하는 building block입니다.

Block	Start	Role
Affine transform	Linear layer, Weight initialization	feature dimension mixing and stable starting scale
Nonlinearity	Activation function, Feed-forward network	local nonlinear transformation and channel mixing
Stabilization	Normalization, Normalization placement, Residual connection	gradient flow and deep model stability
Representation interface	Tokenization, Embedding, Positional encoding	mapping raw objects into model states
Probability and regularization	Softmax, Dropout	normalized scores and train-time noise
Routing and readout	Gating, Pooling and readout	conditional computation and output aggregation

Architecture Families

Family	Start	Best Fit	Watch
Dense / feed-forward	MLP, Autoencoder	fixed vectors, tabular features, bottleneck representations	feature scaling and lost structure
Grid / image / voxel	Image, Video, Convolution, CNN, U-Net, ViT	locality, spatial sharing, dense prediction	receptive field, resolution, augmentation
Sequence recurrence	RNN, LSTM, GRU	ordered streams and compact state	long-range memory and parallelism limits
Attention / Transformer	Attention, Transformer, Encoder-only, Decoder-only, Encoder-decoder	long-range token interaction and conditional generation	quadratic context cost and position handling
State-space sequence models	State-space model, Mamba	long sequences with recurrent state and parallel training	task fit, selective state design, comparison to attention
Cross-modal patterns	Multimodal learning, Cross-attention, Perceiver	fusing different input types or context sources	modality leakage and fusion timing
Set and graph	Deep Sets, Set Transformer, Graph construction, GNN, Graph Transformer	unordered elements, molecules, residues, relations	edge definition, readout, over-smoothing
Geometry-aware	Geometry, Symmetry group, Geometric deep learning, Equivariance, Equivariant GNN	coordinates, forces, poses, molecular and protein structures	coordinate frame, invariance/equivariance split, chirality
Sparse / routed	Mixture of experts, Gating	conditional compute and large-capacity models	routing collapse, load balance, serving cost

Mamba는 별도 대분류가 아니라 state-space model 계열의 selective SSM으로 읽습니다.

Architecture Claim Checklist

Claim	Check
Better inductive bias	Does the architecture match the input symmetry or locality?
Better scaling	What is the compute/memory complexity in sequence length, nodes, or pixels?
Better representation	Is the readout or pooling rule aligned with the task output?
Better generation	Does the architecture support the sampling path and conditioning interface?
Better structure modeling	Are invariant and equivariant quantities handled separately?
Better graph modeling	Is graph construction part of the method or held fixed across baselines?
Better coordinate generation	Are coordinate update step count, constraints, and geometry validity reported?

읽을 때 볼 질문

• 입력이 sequence, graph, set, grid, structure 중 무엇인가?
• 모델이 locality, order, permutation invariance, equivariance 중 무엇을 가정하는가?
• Linear layer는 channel을 섞는가, attention/convolution/message passing은 token이나 neighborhood를 섞는가?
• Activation, normalization, residual path가 gradient flow를 안정화하는가?
• Weight initialization과 normalization placement가 깊은 모델의 안정성에 맞는가?
• long-context scaling, sparse routing, geometric bias가 필요한가?
• architecture의 inductive bias가 task의 symmetry와 맞는가?

Related

• Learning methods
• Generative models
• Modalities
• Computational Biology
• Protein modeling concepts