Modality-Task Map

A modality-task map connects the raw input signal to the model-ready representation, target output, loss, metric, and evaluation risk. It prevents a note from saying only “use a Transformer” or “use a GNN” without specifying what the model must actually solve.

The full contract is:

$$x_{\mathrm{raw}}\stackrel{\varphi }{\to }r\stackrel{f_{\theta }}{\to }\hat{y}\in \mathcal{Y}$$

where $x_{\mathrm{raw}}$ is the raw modality, $\varphi$ is the representation map, $r$ is the model input, $f_{\theta }$ is the model, and $\mathcal{Y}$ is the task output space.

Mapping Axes

Modality	Common representation	Common tasks	Output space	Typical risks
Text	tokens, chunks, embeddings	retrieval, similarity search, reranking, QA, generation, classification	sequence, answer, class, ranked list	context leakage, prompt injection, hallucination
Image	pixels, patches, feature maps, regions	classification, detection, segmentation, captioning	class, box, mask, sequence	near-duplicate leakage, augmentation mismatch
Video	frames, clips, tubelets, temporal tokens	event detection, captioning, retrieval, forecasting	class, segment, sequence, ranked list	temporal leakage, frame sampling bias
Audio	waveform, spectrogram, acoustic tokens	speech recognition, event detection, generation	sequence, class, timestamped event	sampling-rate mismatch, transcript leakage
Tabular	columns, normalized features, embeddings	classification, regression, ranking, reranking	class, scalar, ranked list	identifier leakage, train-test distribution shift
Graph	nodes, edges, features	graph classification, node prediction, link prediction	class, label, edge, graph	split leakage through shared nodes or edges
Sequence	tokens, residues, k-mers, events	classification, generation, tagging, similarity search, forecasting	class, sequence, span, scalar, ranked list	homolog leakage, future information leakage
Molecule	SMILES, graph, fingerprint, conformer	property prediction, similarity search, generation	scalar, class, ranked list, sequence, graph	scaffold leakage, standardization mismatch
3D structure	coordinates, distances, frames, surfaces, geometric graphs	pose prediction, structure prediction, scoring, similarity search, segmentation	coordinates, scalar, class, graph, ranked list	coordinate-frame leakage, template leakage
Protein-ligand complex	pocket graph, ligand pose, pair features, interaction graph	interaction prediction, affinity prediction, pose ranking	scalar, class, contact map, ranked list	protein-family, scaffold, assay, and pose leakage

Task Construction

A task should be specified as:

$$\mathcal{T}=({\mathcal{X}}_{\mathrm{raw}},\varphi ,\mathcal{Y},v,\mathcal{L},\mathcal{M},s)$$

where ${\mathcal{X}}_{\mathrm{raw}}$ is the raw input space, $\varphi$ is preprocessing or representation, $\mathcal{Y}$ is the output space, $v$ is the validity rule, $\mathcal{L}$ is the training loss, $\mathcal{M}$ is the metric set, and $s$ is the split rule.

Use Representation contract when $\varphi$ is nontrivial, cached, learned, stochastic, or domain-specific.

The metric set $\mathcal{M}$ should be chosen with Metric selection, and failure cases should be named with Failure mode taxonomy.

Evaluation Implications

The same modality can support very different metrics:

$$(\text{modality},\mathcal{Y})\Rightarrow (\mathcal{L},\mathcal{M},\text{failure mode})$$

For example, an image can produce a class, box, mask, caption, embedding, or retrieved item. A protein-ligand complex can produce a pose, score, affinity estimate, contact map, or ranked candidate list. These outputs require different validity checks and metrics.

Checks

• What is the raw modality before preprocessing?
• What representation does the model actually see?
• What is one valid target output?
• What outputs are invalid and how are they handled?
• Does the loss match the output space?
• Does the metric match the user-facing behavior?
• What failure modes are expected and counted?
• Does the split rule prevent modality-specific leakage?
• Does the architecture match the structure of the representation rather than the name of the task?

Related

• Modalities
• Modality representation
• Representation contract
• Tasks
• Task specification
• Task output space
• Property prediction
• Interaction prediction
• Similarity search
• Reranking
• Preprocessing contract
• Dataset split contract
• Architecture selection
• Evaluation protocol
• Metric selection
• Failure mode taxonomy
• Leakage