Architecture Papers

Architecture paper note는 오래 남는 model family, block, inductive bias, scaling route를 다룹니다. 이 페이지는 개별 논문을 모두 펼치는 곳이 아니라, architecture를 읽기 위한 paper shelf입니다.

논문의 오래 남는 기여가 재사용 가능한 architecture나 block이면 이 묶음에 둡니다. 같은 논문이 language-model history에서도 중요하면 노트를 복제하지 말고 LLM concepts나 Agents로 cross-link합니다.

Primary Shelf Rule

한 논문은 기본적으로 가장 강한 기여 축 하나에 canonical note를 둡니다.

질문	Primary shelf
reusable block, backbone, connectivity, inductive bias, or scaling rule이 핵심인가?	Architecture
distribution, denoising, flow, latent variable, or sampling objective가 핵심인가?	Generative Models
protein, molecule, ligand, pocket, complex, or domain-specific split/evaluation이 핵심인가?	Computational Biology

다른 축에서 읽을 필요가 있으면 문서를 복제하지 않고 해당 선반의 index에서 링크합니다. 예를 들어 AlphaFold는 architecture 관점의 canonical note를 유지하면서 Computational Biology papers에서 domain route로 연결할 수 있습니다. 반대로 분자 생성 논문은 생성 objective가 중심이면 Generative model papers가 primary이고, 분자 object와 assay/evaluation이 중심이면 Computational Biology가 primary입니다.

Start Here

전체 선반을 처음부터 순서대로 읽을 필요는 없습니다. 아래의 Core canon은 architecture를 비교하기 위한 최소 경로이고, 세부 변형은 각 route와 shelf에서 확장합니다.

Core canon	먼저 읽을 논문	이 경로에서 얻는 것
Sequence backbone	Attention Is All You Need	token mixing, attention, residual stream, encoder-decoder contract
Recurrent and iterative computation	Long Short-Term Memory, Universal Transformers	state, shared depth, recurrence, adaptive computation
Vision backbone	Deep Residual Learning, Vision Transformer	locality, residual optimization, patch tokenization, data regime
Graph and geometry	Neural Message Passing, E(n) Equivariant GNN	relational inductive bias, coordinates, invariance and equivariance
Generative architecture	Auto-Encoding Variational Bayes, Denoising Diffusion Probabilistic Models	latent variables, denoising dynamics, sampling and likelihood boundaries
Conditional capacity	Sparsely-Gated MoE, Switch Transformer	routing, expert capacity, active versus total parameters
Agent and control	Human-level control through deep reinforcement learning, World Models, Dreamer, MuZero	value control, learned dynamics, latent imagination, and planning
Scientific structure	Highly accurate protein structure prediction with AlphaFold, E(n) Equivariant GNN	domain objects, geometric representation, confidence and evaluation boundary

Core canon에 있는 논문은 단순 citation 목록이 아니라, 이후 논문을 읽을 때 비교 기준으로 사용하는 anchor입니다. 새 논문은 이 anchor 중 어떤 구조를 바꾸는지 먼저 적고, domain-specific contribution은 Computational Biology papers나 Generative model papers로 연결합니다.

Depth Standard

Architecture papers are long-term assets, so a finished note should be more than a short abstract. Short notes are acceptable as first-pass index entries, but canonical architecture papers should eventually reach full note depth.

Depth	Target Length	Use
Seed note	5-10 min read	metadata, core claim, equation, method sketch, links
Full note	15-25 min read	architecture walkthrough, equations, diagrams or block tables, evidence, ablations, limits
Longform review	25-40 min read	Korean explanatory post or synthesis, not this paper shelf

For a full architecture note, include:

the exact input/output contract and tensor shapes when useful;
the core equations with symbols defined;
the architecture block decomposition;
the benchmark/evidence table and what each result actually supports;
ablations that separate architecture from data, objective, and compute;
limitations, failure modes, and later variants;
links back to reusable concepts rather than long duplicated explanations.

This means the current shelf can contain seed notes while the most important papers are expanded over time.

Reading Routes

Route	Start With	Then Read
Transformer backbone route	Sequence to Sequence Learning, Attention Is All You Need	Bahdanau Attention, BERT, GPT-2, T5, LLaMA
Iterative Transformer route	Attention Is All You Need	Universal Transformers, Transformer-XL, Neural ODE, Mamba
Variable-output and entity-routing route	Pointer Networks	Dynamic Routing Between Capsules, Set Transformer, DETR, Graphormer
Shared representation and metric-learning route	Siamese Neural Networks	FaceNet, CLIP, GraphSAGE
Memory and retrieval route	Long Short-Term Memory, Neural Turing Machines	Memory Networks, End-To-End Memory Networks, Differentiable Neural Computer, Hopfield Networks is All You Need, Attention Is All You Need, Transformer-XL, Perceiver IO
Efficient attention and recurrent sequence route	Transformer-XL	Longformer, BigBird, Reformer, Linformer, Performer, FlashAttention, Gated Linear Attention, DeltaNet, Griffin, Mega, RWKV, RetNet, S4, Hyena, Mamba, Mamba-2, Jamba
CNN and vision backbone route	LeNet-5, AlexNet	VGG, Network In Network, Inception, Highway Networks, ResNet, Vision Transformer, Swin Transformer, CoAtNet, ConvNeXt
Generative image route	Generative Adversarial Nets	DCGAN, Progressive GAN, BigGAN, StyleGAN2, StyleGAN3
Diffusion and latent route	DDPM	Taming Transformers, Latent Diffusion Models, DiT
Geometric/scientific route	Neural Message Passing	SchNet, EGNN, SE(3)-Transformer, AlphaFold2
Agent and control route	DQN	World Models, Dreamer, MuZero, AlphaZero, LSTM, Differentiable Neural Computer
Architecture search route	Neural Architecture Search with Reinforcement Learning	EfficientNet, Architecture selection, Computational complexity

Shelves

Shelf	Read For	Anchor Papers
Sequence and recurrent models	recurrence, gated memory, token mixing, encoder-decoder transduction, causal convolution, long convolution, moving average memory, recurrent language modeling, retention, positional encoding, language-model backbone design, prompt-as-interface behavior, long-context memory, efficient attention, state-space sequence modeling, structured matrix mixers, hybrid Transformer-SSM backbones	Long Short-Term Memory, RNN Encoder-Decoder, Sequence to Sequence Learning, Bahdanau Attention, WaveNet, Attention Is All You Need, BERT, DeBERTa, T5, GPT-2, GPT-3, Transformer-XL, Longformer, BigBird, Reformer, Linformer, RoFormer, ALiBi, Performer, FlashAttention, Gated Linear Attention, DeltaNet, Griffin, Mega, RWKV, RetNet, S4, Hyena, LLaMA, Layer Normalization, RMSNorm, Mamba, Mamba-2, Jamba
Associative memory and retrieval	content-addressable memory, attractor dynamics, differentiable external memory, soft read/write, allocation, temporal links, multi-hop retrieval, energy-based retrieval, and attention as a memory operation	Memory Networks, End-To-End Memory Networks, Neural Turing Machines, Differentiable Neural Computer, Hopfield Networks is All You Need, Attention Is All You Need, Transformer-XL
Structured output and entity routing	input-dependent output dictionaries, pointer distributions, permutation constraints, vector-valued entities, part-to-whole votes, routing-by-agreement, and iterative coupling	Pointer Networks, Dynamic Routing Between Capsules, Set Transformer, DETR
Agent, value, and world-model architectures	observation encoders, action-value heads, policy/value heads, tree search, replay versus differentiable memory, latent state compression, task-oriented dynamics, imagined rollouts, differentiable actor-critic learning, and controller/model boundaries	DQN, AlphaZero, World Models, Dreamer, MuZero, Neural Turing Machines, Differentiable Neural Computer
Iterative depth and adaptive computation	shared transition blocks, depth recurrence, time-step representations, learned halting, ponder cost, early exit, and compute-quality trade-offs	Universal Transformers, Neural ODE, Mixture of Experts
Attention variants and positional mechanisms	alignment, scaled dot-product attention, recurrence memory, moving-average local bias, sparse local/global/random attention, LSH attention, low-rank attention, disentangled content-position attention, relative/rotary position handling, linear attention bias, linear attention approximations, gated linear attention, delta-rule linear attention, IO-aware exact attention, key/value head sharing	Bahdanau Attention, Attention Is All You Need, Transformer-XL, DeBERTa, Longformer, BigBird, Reformer, Linformer, RoFormer, ALiBi, Performer, Gated Linear Attention, DeltaNet, FlashAttention, Mega, GQA
Feed-forward and gating blocks	token-wise channel mixing, token-axis MLP mixing, MetaFormer scaffold, activation choice, learned piecewise-linear units, gated skip paths, gated MLPs, dense and sparse expert blocks	Maxout Networks, Highway Networks, MetaFormer, MLP-Mixer, GELU, GLU Variants Improve Transformer, Sparsely-Gated MoE, GShard, Switch Transformer, GLaM, LLaMA
State-space, retention, moving-average, and long-convolution sequence models	recurrent state, convolution view, EMA state, retention state, gated linear-attention state, delta-rule matrix memory, local-attention hybrids, long-range sequence modeling, implicit long filters, selective scan lineage, structured state-space duality, hybrid attention-SSM allocation	S4, Mega, Gated Linear Attention, DeltaNet, Griffin, Hyena, RetNet, Mamba, Mamba-2, Jamba
Vision backbones	locality, weight sharing, depth, width, connectivity, cardinality, channel recalibration, dense prediction, efficient mobile blocks, patch tokenization, token/channel MLP mixing, MetaFormer scaffold, convolution-attention hybrid staging, hierarchy, self-supervised ViT representation learning, masked image pre-training, post-ViT ConvNet modernization	LeNet-5, AlexNet, VGG, Network In Network, Inception, Xception, Highway Networks, ResNet, ResNeXt, DenseNet, MobileNets, MobileNetV2, SENet, EfficientNet, U-Net, Vision Transformer, MLP-Mixer, MetaFormer, DINO, MAE, Swin Transformer, CoAtNet, ConvNeXt
Detection, instance segmentation, and set prediction	proposal networks, region heads, one-stage grid/default-box prediction, multi-scale feature pyramids, dense detector imbalance, mask heads, promptable segmentation, object queries, bipartite matching, dense prediction without NMS	YOLO, SSD, FPN, RetinaNet, Faster R-CNN, Mask R-CNN, Segment Anything, DETR
Multimodal encoders and VLM bridges	dual encoders, contrastive alignment, prompt-defined classifiers, visual token resampling, query bottlenecks, frozen vision/LLM bridging	CLIP, Flamingo, BLIP-2
Shared-weight embedding architectures	parameter-tied branches, pair/triplet geometry, distance-based outputs, open-set retrieval, mining policy, and embedding deployment	Siamese Neural Networks, FaceNet, CLIP, GraphSAGE
Architecture search and design automation	search spaces, controller policies, child-network evaluation, validation reward, fidelity, search budget, and deployment constraints	Neural Architecture Search with Reinforcement Learning, EfficientNet
Graphs, sets, point clouds, geometric models, and multimodal arrays	permutation behavior, message passing, sampling, structural bias, expressivity, molecular graph learning, continuous geometry, directional messages, equivariant scalar/vector/tensor features, structure-conditioned decoding, unordered inputs, latent bottlenecks	Neural Message Passing, SchNet, DimeNet, PaiNN, NequIP, GVP, ProteinMPNN, GCN, GraphSAGE, Graph Attention Networks, GIN, Graphormer, PointNet, Tensor Field Networks, E(n) Equivariant GNN, SE(3)-Transformer, Deep Sets, Set Transformer, Perceiver IO
Neural fields and 3D representations	coordinate networks, implicit fields, differentiable rendering, view synthesis	NeRF
Scientific structure models	domain-specialized architectures for 3D scientific objects	AlphaFold2, AlphaFold3
Generative architectures	autoregressive density, discrete latents, learned image tokenizers, denoising, latent variables, adversarial games, convolutional GAN design, progressive GAN training, attention-based GANs, large-scale conditional GANs, style-based synthesis, alias-free synthesis, invertible flows, iterative sampling, latent-space generation, distribution modeling, diffusion backbones	PixelRNN and PixelCNN, Auto-Encoding Variational Bayes, VQ-VAE, Taming Transformers, Generative Adversarial Nets, DCGAN, Progressive GAN, SAGAN, BigGAN, StyleGAN, StyleGAN2, StyleGAN3, Real NVP, Glow, DDPM, Latent Diffusion Models, DiT
Conditional compute	sparse routing, expert capacity, sharded execution, scaling under fixed token compute, sparse expert LLMs, hybrid MoE backbones	Sparsely-Gated MoE, GShard, Switch Transformer, GLaM, Jamba
Adaptation blocks	frozen backbones, low-rank updates, trainable parameter efficiency	LoRA
Training-time architecture blocks	stochastic activation masking, normalization, activation scale, learned activations under dropout, residual stability, residual-stream scale control, small-batch vision normalization	Dropout, Maxout Networks, Batch Normalization, Group Normalization, Layer Normalization, RMSNorm, GELU
Continuous-depth models	residual dynamics, adaptive compute, ODE solvers as layers	Neural ODE

Current Notes

Paper	Main Architecture	Why it is here
Gradient-Based Learning Applied to Document Recognition	LeNet-5	convolution, subsampling, weight sharing, and end-to-end gradient learning for document recognition; full note started
ImageNet Classification with Deep CNNs	AlexNet	large-scale deep CNN vision milestone; full note started
Very Deep Convolutional Networks	VGG	depth and small-filter CNN design; full note started
Network In Network	NiN / mlpconv	local micro-networks, $1 \times 1$ channel mixing, and global average pooling for CNNs; full note started
Going Deeper with Convolutions	Inception	multi-branch compute-aware CNN module; full note started
Xception	Xception	depthwise separable convolution, Inception factorization, residual flows, compute decomposition, and deployment caveats; full paper note
ResNeXt	ResNeXt	homogeneous aggregated transformations, grouped-convolution implementation, cardinality/width/depth trade-offs, and systems caveats; full paper note
Densely Connected Convolutional Networks	DenseNet	concatenative dense blocks, growth rate, bottleneck/compression, gradient paths, activation memory, and feature-reuse diagnostics; full paper note
MobileNets	MobileNetV1	depthwise separable backbone, width/resolution multipliers, resource budgets, Pareto evaluation, and deployment measurement; full paper note
MobileNetV2	inverted residual CNN	expanded depthwise transform, thin residual interface, linear bottleneck, activation placement, dense-prediction interface, and deployment trade-offs; full paper note
Squeeze-and-Excitation Networks	SENet	channel descriptor, excitation MLP, input-dependent recalibration, placement/overhead analysis, and attention-axis boundaries; full paper note
EfficientNet	EfficientNet	NAS baseline, MBConv family, compound depth/width/resolution scaling, discrete rounding, Pareto evaluation, and deployment claim boundaries; full paper note
Attention Is All You Need	Transformer	attention-only sequence transduction backbone; full note started
Pointer Networks	pointer decoder	attention distribution over variable-length input positions as the output vocabulary; full note started
Dynamic Routing Between Capsules	capsule network	vector-valued entity representations and iterative routing-by-agreement; full note started
Human-level control through deep reinforcement learning	Deep Q-Network	CNN observation encoder, action-value head, replay buffer, and target network for discrete-action control; full note started
World Models	VAE + MDN-RNN + controller	latent visual state, probabilistic recurrent dynamics, and imagined rollouts for model-based control; full note started
Dream to Control: Learning Behaviors by Latent Imagination	Dreamer	recurrent stochastic state-space model, latent imagination, and analytic actor-critic learning through imagined trajectories; full note started
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model	MuZero	task-oriented latent dynamics, reward/policy/value prediction, and MCTS without a supplied simulator; full note started
Universal Transformers	recurrent-depth Transformer	shared self-attention transition, iterative representation refinement, and adaptive per-position halting; full note started
BERT	encoder-only Transformer	bidirectional encoder, masked-token pretraining, segment-aware input packing, transfer heads, and objective/architecture separation; full paper note
DeBERTa	disentangled encoder-only Transformer	content-position disentangled attention and enhanced mask decoding for BERT-family encoders; seed note started
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	T5	encoder-decoder Transformer with unified text-to-text task interface; full note started
Language Models are Unsupervised Multitask Learners	decoder-only Transformer	prompt-conditioned autoregressive LM transfer; full note started
Language Models are Few-Shot Learners	scaled decoder-only Transformer	in-context examples as a task interface for few-shot evaluation without gradient updates; full note started
Transformer-XL	recurrent Transformer	segment-level recurrence, hidden-state memory, and relative positional attention for long-context language modeling; full note started
Longformer	sparse long-document Transformer	sliding-window local attention plus task-motivated global attention for long documents; full note started
BigBird	block sparse Transformer	local, global, and random attention routes for long sequences with linear sparse attention; full note started
Reformer	efficient Transformer	LSH attention, reversible residual layers, and chunked feed-forward computation for long sequences; full note started
Linformer	low-rank attention Transformer	key/value sequence-axis projection based on a low-rank attention assumption; seed note started
RoFormer	Rotary Position Embedding	query/key rotations that make attention scores relative-position aware; full note started
ALiBi	attention with linear biases	fixed head-specific distance bias for train-short test-long length extrapolation; full note started
Rethinking Attention with Performers	Performer	FAVOR+ random-feature approximation for linear-time softmax attention; full note started
FlashAttention	IO-aware exact attention	tiled online-softmax attention kernel that avoids materializing the full attention matrix in HBM; full note started
Gated Linear Attention Transformers with Hardware-Efficient Training	GLA Transformer	data-dependent gated linear attention plus hardware-aware chunkwise training; full note started
Parallelizing Linear Transformers with the Delta Rule over Sequence Length	DeltaNet	delta-rule linear Transformer with parallel training over sequence length; full note started
Griffin	Griffin / Hawk	gated linear recurrence mixed with local attention for efficient language models; full note started
Mega	moving-average equipped gated attention	EMA local bias combined with gated attention for long-sequence modeling; full note started
GQA	grouped-query attention	key/value head sharing between MHA and MQA for lower KV-cache cost during decoding; full note started
LLaMA	modern decoder-only Transformer	open foundation model recipe with RMSNorm, RoPE, SwiGLU, and public-data scaling; full note started
Dropout	stochastic activation masking	training-time subnetwork sampling for regularization and approximate model averaging; full note started
Maxout Networks	maxout unit	learned piecewise-linear activation block designed to pair with dropout; full note started
Gaussian Error Linear Units	GELU activation	smooth probability-weighted activation used in BERT-style Transformer feed-forward blocks; seed note started
GLU Variants Improve Transformer	gated Transformer FFN	GLU, GEGLU, and SwiGLU variants for token-wise feed-forward sublayers; full note started
WaveNet	dilated causal convolutional generator	autoregressive raw audio generation with dilated causal convolutions and gated residual blocks; full note started
Long Short-Term Memory	LSTM	gated recurrent memory; full note started
Neural Turing Machines	external differentiable memory	recurrent controller with content/location addressing and erase-add read/write operations; full note started
Memory Networks	modular long-term memory	separates input, generalization, output, response, and memory components for multi-step QA; full note started
End-To-End Memory Networks	multi-hop memory attention	recurrent attention over external memory with end-to-end training; full note started
Hybrid Computing Using a Neural Network with Dynamic External Memory	DNC	memory allocation, temporal links, and content/forward/backward read modes; full note started
Hopfield Networks is All You Need	modern continuous Hopfield layer	associative retrieval, energy-based fixed points, and the attention correspondence; full note started
Learning Phrase Representations using RNN Encoder-Decoder	GRU / encoder-decoder	gated recurrent sequence transduction; full note started
Sequence to Sequence Learning with Neural Networks	LSTM seq2seq	deep LSTM encoder-decoder for variable-length sequence transduction and pre-attention neural machine translation; full note started
RWKV: Reinventing RNNs for the Transformer Era	recurrent language model	Transformer-like parallel training with RNN-like recurrent inference and constant-size state; full note started
Retentive Network	RetNet / retention	retention mechanism with parallel, recurrent, and chunkwise recurrent forms for language-model backbones; full note started
Neural Machine Translation by Jointly Learning to Align and Translate	additive attention / alignment	dynamic source memory access for RNN encoder-decoder translation; full note started
Batch Normalization	BatchNorm	normalization as an architecture component; full note started
Group Normalization	GroupNorm	batch-independent channel-group normalization for small-batch vision models; seed note started
Highway Networks	gated skip network	transform and carry gates for very deep feed-forward networks; full note started
Deep Residual Learning	ResNet	residual learning and deep CNN optimization; full note started
U-Net	U-Net	encoder-decoder CNN with localization skip paths; full note started
An Image is Worth 16x16 Words	Vision Transformer	patch tokenization, full Transformer encoder contract, positional/readout choices, data regime, scaling, and transfer limits; full paper note
MLP-Mixer	all-MLP vision backbone	token-mixing and channel-mixing MLP blocks for image classification without convolution or attention; full note started
MetaFormer Is Actually What You Need for Vision	MetaFormer / PoolFormer	Transformer-style scaffold with replaceable token mixer, tested with simple pooling; full note started
Emerging Properties in Self-Supervised Vision Transformers	DINO / self-supervised ViT	teacher-student self-distillation, multi-crop views, collapse control, and emergent semantic attention behavior; full note
Masked Autoencoders Are Scalable Vision Learners	masked autoencoder ViT	visible-only ViT encoder, high-ratio masking, lightweight decoder, masked-only reconstruction, and transfer interface; full paper note
Swin Transformer	Swin Transformer	shifted-window hierarchical vision Transformer; full note started
CoAtNet	convolution-attention hybrid backbone	staged MBConv and relative-attention blocks for data-efficient and scalable vision backbones; full note started
A ConvNet for the 2020s	ConvNeXt	modernized pure ConvNet backbone after ViT/Swin design lessons, with block contract, scaling, ablations, and transfer limits; full paper note
Learning Transferable Visual Models From Natural Language Supervision	CLIP	dual-encoder vision-language contrastive architecture; full note started
FaceNet	shared-weight metric embedding	parameter-tied encoder branches, triplet geometry, online mining, and open-set retrieval; full note started
Siamese Neural Networks for One-shot Image Recognition	Siamese verifier	tied convolutional branches, pairwise energy, verification-to-one-shot transfer, and support/query evaluation; full note started
Neural Architecture Search with Reinforcement Learning	controller-generated architectures	recurrent controller, search space, child-network reward, policy gradient, and search-budget accounting; full note started
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm	AlphaZero	shared representation, policy/value heads, MCTS policy improvement, self-play targets, and planning cost; full note started
Flamingo	visual language model	Perceiver-style visual resampler and gated cross-attention bridge for few-shot multimodal prompting; seed note started
BLIP-2	Q-Former bridge	trainable query bottleneck between frozen image encoders and frozen language models; seed note started
You Only Look Once	one-stage object detector	full-image grid prediction for real-time object detection; seed note started
SSD: Single Shot MultiBox Detector	one-stage default-box detector	default boxes and multi-feature-map prediction for single-shot object detection; seed note started
Feature Pyramid Networks	multi-scale feature pyramid	top-down and lateral CNN feature pyramid for scale-aware dense prediction; seed note started
Focal Loss for Dense Object Detection	RetinaNet	FPN-based dense one-stage detector trained with focal loss for foreground-background imbalance; seed note started
Faster R-CNN	two-stage object detector	Region Proposal Network plus shared CNN features for end-to-end proposal-based detection; seed note started
Mask R-CNN	instance segmentation detector	Faster R-CNN with RoIAlign and a parallel mask head for instance-level segmentation; seed note started
Segment Anything	promptable segmentation model	reusable image encoder, prompt encoder, and mask decoder for zero-shot promptable segmentation; seed note started
End-to-End Object Detection with Transformers	DETR	object detection as set prediction with object queries and bipartite matching; full note started
Layer Normalization	LayerNorm	batch-independent normalization for sequence models; full note started
Root Mean Square Layer Normalization	RMSNorm	feature-wise root-mean-square normalization without mean-centering; full note started
Neural Message Passing for Quantum Chemistry	MPNN	message, update, and readout framework for molecular graph neural networks; full note started
SchNet	continuous-filter atomistic network	distance-conditioned convolution for molecular energy and force modeling; full note started
Directional Message Passing for Molecular Graphs	DimeNet	directed molecular messages with distance and angular basis functions; full note started
Equivariant Message Passing for Tensorial Molecular Properties	PaiNN	scalar and vector atom features for equivariant molecular property and tensorial prediction; full note started
NequIP	E(3)-equivariant interatomic potential	typed tensor features and equivariant convolutions for data-efficient energy-force modeling; full note started
Learning from Protein Structure with Geometric Vector Perceptrons	GVP	scalar/vector protein-structure graph blocks for geometric and relational reasoning; full note started
ProteinMPNN	structure-conditioned protein MPNN	residue-level message passing and order-agnostic autoregressive sequence design for fixed-backbone protein design; full note started
Semi-Supervised Classification with GCNs	GCN	graph message passing for node classification; full note started
Inductive Representation Learning on Large Graphs	GraphSAGE	sampled neighborhood aggregation for inductive node representation learning on large graphs; full note started
Graph Attention Networks	GAT	learned attention over graph neighborhoods; full note started
How Powerful are Graph Neural Networks?	GIN	1-WL expressivity, injective multiset aggregation, and graph-level readout; full note started
Do Transformers Really Perform Bad for Graph Representation?	Graphormer	graph Transformer with centrality, shortest-path, and edge encodings; full note started
PointNet	PointNet	direct point-cloud architecture with shared point-wise MLPs and symmetric pooling; full note started
NeRF	neural radiance field	continuous coordinate-to-density/color scene representation with differentiable volume rendering; full note started
Tensor Field Networks	TFN	rotation-, translation-, and permutation-equivariant 3D point-cloud layers with typed tensor features and spherical-harmonic filters; full note started
E(n) Equivariant Graph Neural Networks	EGNN	geometric graph message passing with E(n)-equivariant coordinate updates; full note started
SE(3)-Transformers	SE(3)-Transformer	roto-translation equivariant attention for 3D point clouds and graphs; full note started
Highly accurate protein structure prediction with AlphaFold	AlphaFold2	MSA-pair reasoning, Evoformer, invariant point attention, recycling, and confidence for protein structure prediction; full note started
Accurate structure prediction of biomolecular interactions with AlphaFold 3	AlphaFold3	unified biomolecular complex prediction with atom-level diffusion coordinate generation; full note started
Deep Sets	Deep Sets	permutation-invariant set function architecture; full note started
Set Transformer	Set Transformer	attention-based permutation-invariant set modeling; full note started
Perceiver IO	Perceiver IO	latent-bottleneck attention for structured inputs and outputs; full note started
Auto-Encoding Variational Bayes	VAE	amortized inference, reparameterization trick, and ELBO training for neural latent-variable generative models; full note started
Pixel Recurrent Neural Networks	PixelRNN / PixelCNN	tractable autoregressive image density modeling with raster-order pixel conditionals and masked spatial dependencies; seed note started
Neural Discrete Representation Learning	VQ-VAE	vector-quantized discrete latent codebook, straight-through training, code usage diagnostics, and learned autoregressive prior over codes; full note
Taming Transformers for High-Resolution Image Synthesis	VQGAN + autoregressive Transformer	learned image codebook plus Transformer prior for high-resolution image synthesis; seed note started
Generative Adversarial Nets	GAN	implicit generator trained through a discriminator-defined adversarial game; full note started
Unsupervised Representation Learning with DCGANs	DCGAN	convolutional generator/discriminator design constraints for stable image GANs; seed note started
Progressive Growing of GANs	Progressive GAN	generator and discriminator topology grows from low to high image resolution during training; seed note started
Self-Attention Generative Adversarial Networks	SAGAN	non-local self-attention blocks for long-range spatial dependency modeling in GANs; seed note started
Large Scale GAN Training for High Fidelity Natural Image Synthesis	BigGAN	class-conditional residual GAN scaling with hierarchical latent injection and truncation sampling; seed note started
A Style-Based Generator Architecture for GANs	StyleGAN	mapping network, per-layer style modulation, and stochastic noise injection for high-resolution image synthesis; seed note started
Analyzing and Improving the Image Quality of StyleGAN	StyleGAN2	weight modulation/demodulation, fixed generator redesign, and path length regularization for artifact reduction; seed note started
Alias-Free Generative Adversarial Networks	StyleGAN3	alias-free generator signal processing for subpixel translation/rotation equivariance and reduced coordinate locking; seed note started
Density estimation using Real NVP	Real NVP	invertible affine coupling flow with exact likelihood, sampling, and latent inference; full note started
Glow	Glow	normalizing flow with actnorm, affine coupling, and learned invertible $1 \times 1$ channel mixing; seed note started
Denoising Diffusion Probabilistic Models	DDPM	iterative denoising architecture and objective for diffusion generative models; full note started
High-Resolution Image Synthesis with Latent Diffusion Models	Latent Diffusion Model	autoencoder latent-space diffusion with cross-attention conditioning for efficient high-resolution generation; full note started
Scalable Diffusion Models with Transformers	DiT	Transformer denoising backbone over latent image patches, adaptive conditioning, token-count scaling, and U-Net comparison; full note
Neural Ordinary Differential Equations	Neural ODE	continuous-depth residual dynamics; full note started
Efficiently Modeling Long Sequences with Structured State Spaces	S4	structured state-space sequence layer for practical long-range modeling; full note started
Hyena Hierarchy	gated long-convolution sequence model	implicit long convolutions and data-controlled gating as a dense-attention-free sequence operator; full note started
Mamba	selective SSM	input-dependent state-space sequence modeling; full note started
Transformers are SSMs	Mamba-2 / SSD	structured state space duality and a faster Mamba-style core layer; full note started
Jamba: A Hybrid Transformer-Mamba Language Model	hybrid Transformer-Mamba-MoE LM	interleaves attention and Mamba layers with sparse MoE feed-forward capacity for long-context language modeling; full note started
Outrageously Large Neural Networks	sparsely-gated MoE	sparse expert routing that decouples total capacity from active per-token computation; full note started
GShard	sharded MoE Transformer	conditional-compute Transformer scaling with automatic sharding and top-2 expert routing; full note started
Switch Transformer	sparse MoE Transformer	top-1 expert routing and conditional compute; full note started
GLaM	sparse expert language model	top-2 MoE scaling route for large autoregressive language models; full note started
LoRA	low-rank adaptation block	frozen pretrained weights with trainable low-rank residual updates for parameter-efficient adaptation; full note started

Reading Axes

어떤 input structure를 가정하는가: sequence, image, graph, set, 3D coordinate, multimodal context, agent state?
어떤 inductive bias가 바뀌는가: locality, permutation behavior, equivariance, recurrence, memory, sparsity, routing, hierarchy?
sequence length, graph size, atom count, residue count, token count에 따라 complexity가 어떻게 바뀌는가?
contribution이 new architecture, block replacement, scaling rule, efficiency trick 중 무엇인가?
evidence가 accuracy, sample quality, transfer, stability, latency, memory, throughput 중 무엇에 관한 것인가?
ablation이 objective, data, compute 변화와 architecture 변화를 분리할 만큼 강한가?

Concepts

Evaluation Risks

Architecture gain이 parameter, token, data, training step, tuning budget 증가에서 온 것일 수 있습니다.
New block이 다른 objective나 augmentation policy와 함께 test되었을 수 있습니다.
Throughput gain은 hardware, kernel, batch size, sequence length, sparse implementation에 의존할 수 있습니다.
Graph/geometric architecture는 graph construction, coordinate frame choice, template-derived feature를 통해 information을 leak할 수 있습니다.

Contents

Architecture Papers

Architecture Papers

Primary Shelf Rule

Start Here

Depth Standard

Reading Routes

Shelves

Current Notes

Reading Axes

Concepts

Evaluation Risks

Related