Modalities
Modality는 고유한 structure, noise pattern, inductive bias를 가진 input 또는 output signal의 종류입니다. Text, image, video, audio, molecule, protein, graph, 3D structure는 downstream task가 같은 prediction problem처럼 쓰여도 서로 다른 representation을 요구하는 경우가 많습니다.
Model input은 modality-specific view의 묶음으로 볼 수 있습니다.
where indexes a modality and is the set of available modalities.
Route Map
| Question | Start | Then Check |
|---|---|---|
| what is the raw signal? | Text, Sequence, Image, Video, Audio, Tabular | preprocessing and lost information |
| is the object relational? | Graph | node/edge definition and graph construction |
| is the object geometric? | 3D structure | coordinate frame, invariance, equivariance |
| what does the model actually see? | Modality representation, Representation contract | tensor shape, tokens, graph, coordinates |
| how does modality map to task? | Modality-task map, Task specification | output space, loss, metric, split |
| are multiple modalities aligned? | Modality alignment, Multimodal learning | missing modality and leakage |
Modality Contract
Modality note should separate the raw signal from the model-ready representation.
Here is the raw signal, is modality-specific preprocessing or featurization, and is what the model actually sees.
| Layer | Ask | Route |
|---|---|---|
| raw signal | text, sequence, image, graph, coordinates, tabular record, audio, video? | modality-specific pages |
| representation | token, patch, feature map, graph, coordinate set, embedding, fingerprint? | Representation contract |
| axes | batch, token, channel, node, edge, residue, atom, frame, coordinate? | Tensor shape notation |
| lost information | what is discarded by tokenization, cropping, graph construction, or featurization? | Preprocessing contract |
| output behavior | class, scalar, rank, sequence, graph, coordinate, action? | Task output space |
| evaluation risk | what leakage or metric failure is modality-specific? | Evaluation |
Modality vs Entity vs Task
These axes often get mixed.
| Axis | Meaning | Example |
|---|---|---|
| entity | real-world object being modeled | molecule, protein, ligand, pocket, document, image |
| modality | signal form available to the model | sequence, graph, image, 3D coordinates, text |
| representation | processed input object | SMILES tokens, molecular graph, residue embedding, voxel grid |
| task | target behavior | classify, rank, retrieve, generate, predict coordinates |
| architecture | computation pattern | Transformer, CNN, GNN, SSM, equivariant GNN |
For example, a protein can appear as sequence tokens, an MSA, a residue graph, a contact map, 3D coordinates, or a text description. Those are different modality/representation choices for the same entity.
Representation Choice Map
| Raw object | Possible representations | Common architecture route |
|---|---|---|
| text document | tokens, chunks, embeddings | Transformer, retrieval encoder |
| protein sequence | residues, k-mers, PLM embeddings, MSA | Transformer, SSM, CNN, RNN |
| molecule | SMILES, graph, fingerprint, descriptors, conformer | Transformer, GNN, MLP, equivariant model |
| protein-ligand complex | pocket graph, pair graph, distance features, coordinates | GNN, graph transformer, equivariant GNN |
| image | pixels, patches, feature maps, regions | CNN, ViT, U-Net |
| video | frames, clips, tubelets, temporal tokens | CNN/ViT plus temporal model |
| tabular record | columns, normalized features, learned embeddings | linear model, tree model, MLP |
| 3D structure | coordinates, distances, frames, surfaces, voxels | equivariant model, GNN, CNN |
The right note for a representation claim is usually Representation contract, not only the architecture page.
Modality Groups
| Group | Notes | Common Tasks |
|---|---|---|
| Language and symbolic sequence | Text, Sequence | Question answering, Sequence generation, Retrieval |
| Dense sensory input | Image, Video, Audio | Object detection, Segmentation, Captioning |
| Structured objects | Tabular, Graph, 3D structure | Structured prediction, Graph prediction, Coordinate prediction |
| Cross-modal systems | Modality alignment, Missing modality, Multimodal learning | retrieval, grounding, fusion, reranking |
Failure Modes
| Failure | Example |
|---|---|
| raw and processed object confused | evaluating a molecule graph task as if it were a SMILES task |
| axis meaning omitted | node axis, token axis, channel axis, or coordinate axis unclear |
| representation leakage | graph, crop, pocket, template, or retrieval context uses test-only information |
| alignment leakage | paired modalities share source or near-duplicate examples across splits |
| modality missingness ignored | model assumes image, sequence, structure, or metadata always exists |
| metric mismatch | generated sequence validity treated like classification accuracy |
| architecture keyword overused | “Transformer” claim hides tokenization, context, or task differences |
Why It Matters
- The raw signal and the model-ready representation are often different objects.
- Tokenization can discard timing, geometry, spatial locality, or alignment information.
- Single-modal, cross-modal, and multimodal-fusion tasks fail in different ways.
- Evaluation metrics must match the output modality, not only the model family.
- A modality should be connected to task output space, loss, metric, and split before choosing an architecture.
Public Note Pattern
| Field | Write |
|---|---|
| raw signal | what exists before preprocessing |
| representation | what tensor/token/graph/coordinate object the model sees |
| axes | shape and axis names |
| preprocessing | deterministic, learned, stochastic, cached, or versioned |
| task | output space and valid output |
| metric | primary metric and invalid-output handling |
| split | leakage-relevant unit |
| boundary | whether entity-specific details belong in Computational Biology |
Checks
- What is the raw input signal?
- What is the tensor, token, graph, or coordinate representation after preprocessing?
- What modality representation does the model actually see?
- What representation contract connects raw object, model input, output, loss, metric, split, and leakage?
- What modality-task map connects input, output, loss, metric, and split?
- What preprocessing contract turns the raw signal into model input?
- Which information is lost before the model sees the input?
- Does the model need alignment between modalities?
- Can the model handle missing, delayed, or corrupted modalities?
- Are splits designed to prevent near-duplicate or cross-modal leakage?