Inductive Bias

Inductive bias is the assumption a model family uses to prefer some functions over others before seeing the full data distribution.

An architecture restricts the hypothesis class:

$$f_{\theta }\in {\mathcal{F}}_{\mathrm{arch}}$$

Training then searches within that class:

$$\hat{f}=\arg \underset{f\in {\mathcal{F}}_{\mathrm{arch}}}{\min }\frac{1}{n}\sum _{i=1}^n\ell (f(x_i),y_i)$$

Here $x_i$ is an input, $y_i$ is a target, $\ell$ is the loss, and ${\mathcal{F}}_{\mathrm{arch}}$ is the function class implied by the architecture.

Common Biases

• CNN: locality and translation equivariance.
• RNN and state-space models: ordered state updates over a sequence.
• Transformer: content-based pairwise interaction through attention.
• GNN: graph connectivity and permutation equivariance over nodes.
• Equivariant models: output transforms predictably under rotation, translation, or other symmetry groups.
• Mixture of experts: sparse conditional computation through routing.

Why It Matters

The right bias can reduce sample complexity because the model does not need to learn every useful symmetry from data. The wrong bias can hide important interactions or make a benchmark look better than deployment.

For example, a graph model is natural when edges encode meaningful relations. If the graph construction is noisy or leaks target information, the architecture can amplify the wrong signal.

Checks

• What symmetry or invariance does the data actually have?
• Is the architecture assuming locality, order, graph structure, sparsity, or geometry?
• Does the benchmark split test the same bias that deployment will require?
• Is a simpler model with a weaker bias a necessary baseline?

Related

• Architectures
• Parameter sharing
• Architecture selection
• OOD generalization
• Baseline