Architectures

Architecture note는 molecular AI, protein modeling, agent, 구현 읽기에서 쓰는 model family를 설명합니다. 논문이 무엇을 섞고 있는지, 어떤 inductive bias를 가정하는지, evaluation이 어디서 흔들릴 수 있는지 빠르게 확인하는 용도입니다.

Architecture는 training 중 탐색하는 function class를 정의합니다.

f_{θ} \in F_{arch}

같은 training objective라도 $F_{arch}$ 가 locality, order, graph structure, symmetry 중 무엇을 encode하는지에 따라 전혀 다르게 동작할 수 있습니다.

Paper Shelf

이 개념 페이지에서 정의를 읽은 뒤, 실제 architecture가 그 가정을 어떻게 구현했는지는 Architecture Papers의 anchor note에서 확인합니다.

Concept route	Anchor paper
Attention and Transformer	Attention Is All You Need
Recurrent and iterative depth	Universal Transformers
Shared embedding geometry	FaceNet
Graph and equivariance	E(n) Equivariant GNN
Generative architecture	Denoising Diffusion Probabilistic Models
Agent control	Human-level control through deep reinforcement learning
Architecture design automation	Neural Architecture Search with Reinforcement Learning

Architecture Contract

Architecture note는 모델명을 외우는 것이 아니라, 입력 구조와 정보 흐름을 계약으로 고정합니다.

A = (X, H, M, S, R, O, C)

Part	Meaning	Typical question
$X$	input object	vector, sequence, grid, graph, set, coordinate, multimodal input?
$H$	hidden state shape	token states, node states, feature maps, coordinates, latent vectors?
$M$	mixing rule	dense layer, convolution, recurrence, attention, message passing, state update?
$S$	sharing or symmetry rule	translation sharing, permutation invariance, equivariance, causal order?
$R$	readout	class token, pooling, graph readout, coordinate head, distribution head?
$O$	objective compatibility	supervised loss, generative sampler, SSL target, ranking objective?
$C$	compute and memory cost	sequence length, graph size, resolution, state size, expert count?

이 계약을 먼저 쓰면 “Transformer vs Mamba”, “GNN vs Graph Transformer”, “CNN vs ViT” 같은 비교가 모델명 싸움이 아니라 assumption 비교가 됩니다.

Route Map

Question	Start	Then Check
what input object is being modeled?	Modalities, Architecture selection	sequence, graph, grid, set, coordinate, or mixed input
is the claim architecture or objective driven?	Architecture-objective fit	data, objective, compute, and evaluation controls
what inductive bias is assumed?	Inductive bias, Parameter sharing	locality, order, permutation, symmetry, sparsity
what mixes information?	Attention, Convolution, GNN, State-space models	token, pixel, node, coordinate, or state mixing
what transforms channels?	Linear layer, Feed-forward network, Activation function	width, expansion, gating, parameter budget
what stabilizes depth?	Residual connection, Normalization, Normalization placement, Weight initialization	gradient flow and scale
what maps internal states to outputs?	Pooling and readout, Softmax	task output space and metric
what is the cost bottleneck?	Computational complexity	length, graph size, resolution, experts, memory

Information Mixing Axes

Architecture family는 어떤 axis를 섞는지로 읽으면 정리가 쉽습니다.

Mixing axis	Common block	Use when
Feature/channel	[[concepts/architectures/linear-layer	Linear layer]], [[concepts/architectures/feed-forward-network
Local spatial neighborhood	[[concepts/architectures/convolution	Convolution]], [[concepts/architectures/cnn
Ordered time or sequence state	[[concepts/architectures/rnn	RNN]], [[concepts/architectures/lstm
Global token interaction	[[concepts/architectures/attention	Attention]], [[concepts/architectures/transformer
Cross-context interaction	[[concepts/architectures/cross-attention	Cross-attention]], [[concepts/architectures/encoder-decoder
Graph neighborhood	[[concepts/architectures/gnn	GNN]], [[concepts/architectures/graph-transformer
Set aggregation	[[concepts/architectures/deep-sets	Deep Sets]], [[concepts/architectures/set-transformer
Coordinate update	[[concepts/geometric-deep-learning/equivariant-gnn	Equivariant GNN]], [[concepts/geometric-deep-learning/index
Conditional compute	[[concepts/architectures/mixture-of-experts	Mixture of Experts]], [[concepts/architectures/gating

Basic Blocks

Group	Notes
Dense transforms	Linear layer, Activation function, Feed-forward network, Gating
Training stability	Weight initialization, Normalization, Normalization placement, Residual connection, Dropout
Input representation	Tokenization, Embedding, Positional encoding, Graph construction
Readout	Pooling and readout, Softmax

Model Families

Family	Use For	Start
Dense / feed-forward	fixed vectors, tabular features, learned embeddings, bottlenecks	MLP, Autoencoder
Grid / image / voxel	local spatial structure, image/video, contact maps, spatial grids	Convolution, CNN, Residual network, U-Net, Vision Transformer
Sequence	text, residues, time series, ordered tokens	RNN, LSTM, GRU, Transformer, State-space models, Mamba
Attention / encoder-decoder	long-range mixing, conditioning, multimodal fusion	Attention, Cross-attention, Encoder-decoder, Perceiver
Set / graph	unordered sets, molecular graphs, residue graphs, relational objects	Deep Sets, Set Transformer, Graph construction, GNN, Weisfeiler-Lehman Test, Graph Transformer
Geometry-aware	coordinates, forces, poses, protein and molecule structures	Geometric deep learning, Equivariant GNN, Tensor Field Network
Sparse / routed	conditional compute, expert routing, scaling under budget	Mixture of Experts

Mamba는 여기서 별도의 top-level family로 두지 않습니다. Selective state-space model이므로, Mamba 내부 구조 자체를 다루는 노트가 아니라면 State-space models 경로로 둡니다.

Comparison Controls

Architecture claim을 비교할 때는 architecture만 바뀌었는지 확인해야 합니다.

Controlled factor	Why
data and split	stronger architecture claim needs the same train/test boundary
objective and target	different loss can explain the improvement
parameter count and compute	larger model can look like better inductive bias
context length or graph size	scaling advantage depends on input size
preprocessing and representation	better input encoding can dominate architecture
training budget and optimizer	undertrained baseline weakens comparison
evaluation metric and selection rule	architecture claim can be selection artifact

The minimal comparison question is:

Δ metric = ? Δ architecture given fixed data, objective, budget, and evaluation

Reading Checklist

What is the input object: vector, sequence, grid, graph, structure, or mixed modality?
Where is parameter sharing used, and what symmetry does it encode?
What mixes information: dense layers, convolution, recurrence, state update, message passing, or attention?
Which blocks mix channels, tokens, spatial neighborhoods, graph neighborhoods, or coordinates?
What inductive bias is assumed: locality, order, permutation invariance, equivariance, sparsity, or routing?
What is the dominant computational complexity term?
How do activations, residual paths, normalization, and initialization affect gradient flow?
What fails first in practice: memory, length, graph size, data leakage, calibration, or decoding constraints?

Contents

Architectures

Architectures

Paper Shelf

Architecture Contract

Route Map

Information Mixing Axes

Basic Blocks

Model Families

Comparison Controls

Reading Checklist

Related