Data

Data note는 모델을 학습하기 전에 example, label, metadata, split, benchmark가 어떻게 만들어지는지 설명합니다. Applied AI에서는 architecture보다 dataset이 실제 문제를 더 강하게 정의하는 경우가 많습니다.

A dataset induces an empirical distribution:

$${\hat{p}}_{\mathcal{D}}(x,y)=\frac{1}{|\mathcal{D}|}\sum _{(x_i,y_i)\in \mathcal{D}}\delta (x=x_i,y=y_i)$$

Training과 evaluation은 이 empirical distribution이 의도한 deployment distribution과 맞을 때만 의미가 있습니다.

이동 지도

Data Contract

Dataset은 table 파일이 아니라 claim을 가능하게 하는 contract입니다.

$$\mathcal{D}=\{(u_i,x_i,y_i,m_i,q_i){\}}_{i=1}^n$$

where $u_i$ is the raw example unit, $x_i$ is the model-ready input, $y_i$ is the label or target, $m_i$ is metadata, and $q_i$ is quality or filtering state.

Identity, Representation, Label

These three should be kept separate.

For computational biology, this separation is critical: a molecule can have many protonation/tautomer/stereo/conformer representations, and a protein can have sequence, domain, structure, MSA, template, or pocket representations. The dataset note should say which identity is being split and which representation the model sees.

Construction Order

Good data notes preserve the order of operations:

$$\text{raw source}\to \text{schema}\to \text{identity resolution}\to \text{split assignment}\to \text{fit-on-train preprocessing}\to \text{model-ready artifact}\to \text{evaluation set}$$

The dangerous step is fitting or choosing preprocessing on all data before split assignment. Any learned scaler, imputer, vocabulary, threshold, deduplication rule, feature selector, standardization choice, or retrieval index can become leakage if it sees validation/test information.

Claim Boundaries

Boundary With Other Sections

Data notes answer “what examples and labels exist?” Evaluation notes answer “what claim does a score support?” Systems notes answer “can the artifact and run be reconstructed?”

Public Data Note Template

For generated or filtered data, preserve the denominator:

$$\text{reported rate}=\frac{\#\text{kept or valid examples}}{\#\text{attempted or raw examples}}$$

주제 묶음

확인할 것

• 하나의 example은 무엇인가?
• split unit은 무엇인가?
• dataset은 어떤 distribution을 대표해야 하는가?
• field, unit, identifier, relationship을 정의하는 schema가 있는가?
• target label은 무엇에서 만들어지는가?
• label이 정확히 무엇을 의미하는가?
• missing, censored, weak label이 명시적으로 표현되는가?
• batch, source, protocol, time, species, target, structure context를 설명하는 metadata가 있는가?
• collection bias 때문에 빠진 example은 무엇인가?
• class 또는 label distribution이 imbalanced한가?
• sampling bias가 validation을 deployment보다 쉽게 만들지 않는가?
• train, validation, test, deployment 사이에 어떤 shift가 있는가?
• split이 의도한 generalization claim과 맞는가?
• 다른 사람이 dataset version과 filtering policy를 재구성할 수 있는가?
• label이 noisy, censored, inconsistent, protocol-dependent하지 않은가?

구성 경로

1. Define example unit and split unit.
2. Write the data schema and label semantics.
3. Specify the preprocessing contract.
4. Record metadata and provenance and data lineage.
5. Choose the sampling strategy and dataset split contract.
6. Summarize the result in a dataset card.
7. Design the evaluation set.
8. Attach an evaluation protocol.
9. Use benchmark intake and benchmark claim contract before treating a score as evidence for a paper claim.

Public Boundary

Public data notes should preserve reusable structure without exposing private datasets or unpublished results.

Related

• AI
• Modalities
• Tasks
• Evaluation
• Leakage
• Test-set contamination
• Reproducibility