Generative Models

Generative model note는 sequence, graph, coordinate, molecule, protein을 포함한 data distribution을 모델링하고 sampling하는 방법을 설명합니다.

공통 목표는 data distribution에 가까운 model distribution을 학습하는 것입니다.

p_{θ} (x) \approx p_{data} (x)

Different families choose different training signals: likelihood, reconstruction, adversarial discrimination, denoising, score estimation, or velocity matching.

Generative Model Contract

A generative model should specify the sample space, the conditioning variable, the learned objective, the sampler, and the validation rule.

G = (X, c, p_{θ}, L, π_{sample}, V)

Part	Meaning	Typical question
$X$	output space	text, image, graph, molecule, protein sequence, 3D coordinate?
$c$	conditioning context	class label, prompt, scaffold, pocket, sequence, partial structure?
$p_{θ}$ or score field	model distribution or learned field	explicit density, implicit generator, energy, score, velocity?
$L$	training objective	likelihood, ELBO, denoising loss, score loss, adversarial loss, velocity loss?
$π_{sample}$	sampling procedure	autoregressive decode, ODE solver, denoising chain, MCMC, direct generator?
$V$	validity rule	syntax, chemistry, geometry, constraints, downstream test?

Conditional generation uses a distribution over outputs given context:

x \sim p_{θ} (x ∣ c)

For scientific objects, $V (x, c)$ is often as important as $p_{θ} (x ∣ c)$ because invalid samples are not merely low quality; they may be outside the intended sample space.

Selection Axes

Choose a generative family by the object, output validity rule, likelihood need, sampling budget, and conditioning mechanism.

Family	Training signal	Sampling shape	Strength	Common risk
Autoregressive	Next-token likelihood	Sequential	Stable likelihood training	Slow long-horizon sampling
VAE	ELBO	Latent decode	Structured latent space	Posterior collapse, blurry samples
GAN	Adversarial game	One-shot generator	Sharp samples	Mode collapse, unstable training
Normalizing flow	Exact likelihood	Invertible transform	Tractable density	Invertibility constraints
Energy-based model	Energy or contrastive objective	MCMC/Langevin/optimization	Flexible unnormalized density	Partition function and sampling cost
Diffusion	Denoising/noise prediction	Iterative denoising	Stable high-quality samples	Many sampling steps
Score-based	Score matching	SDE/ODE sampler	Continuous-time view	Noise-level coverage
Flow matching	Velocity matching	ODE transport	Direct path learning	Path and symmetry design
Consistency	Trajectory consistency	One/few step	Fast sampling	Distillation or consistency quality

Objective, Sampler, Evaluation

Do not collapse these three layers. Two papers can use the same architecture while changing only the training objective or sampler.

model family \neq = training objective \neq = sampling algorithm \neq = evaluation protocol

Layer	Examples	What can go wrong
Objective	NLL, ELBO, denoising MSE, score matching, flow matching, adversarial loss	objective improves but samples do not
Parameterization	noise, clean sample, score, velocity, energy, logits	papers compare different targets as if they were architectures
Sampler	ancestral sampling, beam search, DDIM, SDE/ODE solver, MCMC, rejection, reranking	quality changes because sampling budget changed
Conditioning	prompt, label, structure, scaffold, pocket, sequence, retrieval	condition leakage or unfair context
Evaluation	validity, novelty, diversity, utility, likelihood, cost	filtered samples hide failure rate

The denominator matters:

success rate = \frac{# valid and useful samples}{# attempted samples}

Reporting only kept samples changes the claim:

\frac{# useful kept samples}{# kept samples} \neq = \frac{# useful samples}{# generated samples} \neq = \frac{# useful samples}{# sampling attempts}

Route Map

Question	Start	Then Check
do you need exact likelihood?	Autoregressive model, Normalizing flow	factorization, ordering, invertibility
is density only defined up to normalization?	Energy-based model, Score matching	partition function, negative samples, sampler
is there a latent variable?	Latent variable model, ELBO, VAE	reconstruction vs prior sampling
is generation iterative?	Diffusion model, Score matching, Flow matching	sampler, step count, parameterization
is generation conditional?	Conditional generation, Guidance	condition leakage and guidance strength
is the object molecular or structural?	Molecular generation, Protein design	validity, novelty, split, physical plausibility
how should samples be judged?	Generation evaluation, Sampling	attempted vs kept samples

Generation Route Map

Route	Core object	Main equation	Best first read
Likelihood	normalized density	$max_{θ} lo g p_{θ} (x)$	[[concepts/math/maximum-likelihood
Autoregressive	factorized sequence density	$p_{θ} (x) = \prod_{t} p_{θ} (x_{t} ∣ x_{< t})$	[[concepts/generative-models/autoregressive-model
Latent variable	latent-conditioned density	$p_{θ} (x) = \int p_{θ} (x ∣ z) p (z) d z$	[[concepts/generative-models/latent-variable-model
VAE / ELBO	variational lower bound	$lo g p_{θ} (x) \geq L_{ELBO}$	[[concepts/generative-models/elbo
Energy-based	unnormalized density	$p_{θ} (x) = exp (- E_{θ} (x)) / Z_{θ}$	[[concepts/generative-models/energy-based-model
Diffusion / score	denoising or score field	$s_{θ} (x_{t}, t) \approx \nabla_{x_{t}} lo g p_{t} (x_{t})$	[[concepts/generative-models/diffusion-model
Flow matching	probability-path velocity	$\frac{d x _{t}}{d t} = v_{θ} (x_{t}, t)$	[[concepts/generative-models/flow-matching
Normalizing flow	invertible transformation	$\log p_X(x)=\log p_Z(f^{-1}(x))+\log	\det J_{f^{-1}}
Consistency	trajectory-consistent map	$f_{θ} (x_{t}, t) \approx f_{θ} (x_{s}, s)$	[[concepts/generative-models/consistency-model

Evaluation Boundary

Generation quality는 하나의 metric으로 끝나지 않습니다. 유용한 evaluation은 아래 항목을 명시합니다.

Validity: does the sample obey the syntax, physics, geometry, or task constraints?
Diversity: does the model cover modes rather than repeat a few outputs?
Novelty: is the sample distinct from training examples under the right equivalence relation?
Utility: does the generated object improve the downstream objective?
Calibration or likelihood: if probabilities are used, are they meaningful for the decision?
Cost: how many steps, tokens, calls, or device-hours are needed per useful sample?

Domain Boundaries

Domain	Keep in generative models	Move to
text or sequence generation	factorization, decoding, likelihood, sampling budget	[[ai/architectures
molecule generation	objective, sampler, validity denominator	[[molecular-modeling/molecules
protein design	sequence/structure-conditioned generation contract	[[molecular-modeling/proteins
3D structure generation	equivariance, coordinate validity, geometry constraints	[[concepts/geometric-deep-learning/index
serving generated outputs	latency, batching, cost, reproducibility	[[ai/systems

Common Traps

Trap	Why it matters
Reporting only filtered samples	hides invalid generation rate and sampler cost
Changing NFE or decode budget	makes speed and quality comparisons unfair
Mixing novelty with utility	new samples can be useless, useful samples can be close analogs
Treating likelihood as sample quality	high likelihood does not guarantee useful samples
Treating sample quality as calibration	pretty or valid samples do not imply meaningful probabilities
Ignoring constraint repair	post-processing can become part of the generator
Evaluating against leaked training neighbors	novelty and generalization claims become weak

Jaemin's Research Notes

Contents

Generative Models

Generative Models

Generative Model Contract

Selection Axes

Objective, Sampler, Evaluation

Route Map

Generation Route Map

Evaluation Boundary

Domain Boundaries

Common Traps

Common Concepts

Core Families

Scientific Targets

Autoregressive Model

Energy-Based Model

Diffusion Model

Flow Matching

Probability Flow ODE

Score-Based Model

Generative Adversarial Network

Protein Design

Consistency Model

Normalizing Flow

Rectified Flow

Variational Autoencoder

Evidence Lower Bound

Score Matching

Guidance

Latent Variable Model

Sampling

Conditional Generation

Molecular Generation

Contents

Generative Models

Generative Models

Generative Model Contract

Selection Axes

Objective, Sampler, Evaluation

Route Map

Generation Route Map

Evaluation Boundary

Domain Boundaries

Common Traps

Common Concepts

Core Families

Scientific Targets

Related