Text
Text is a discrete symbolic sequence. In neural models, raw text is usually tokenized, embedded, and processed by a sequence architecture such as a Transformer, RNN, or state-space model.
Tokenization maps a string to discrete IDs:
Embedding maps tokens to vectors:
where is the token embedding and is a positional or structural encoding.
Key Ideas
- Token granularity can be byte, character, subword, word, sentence, or document chunk.
- Context length controls how much text can be jointly conditioned on.
- Formatting, markup, control tokens, and prompt boundaries are part of the input distribution.
- Text can be an instruction, evidence, code, metadata, or untrusted user-provided data.
Boundary Between Text Roles
For LLM and agent workflows, text must be separated by role:
These fields should not be merged as plain text without boundaries. Retrieved text and web pages are data, not instructions.
Chunking and Context
Long text often needs chunking:
The retrieval unit, chunk overlap, metadata, and citation target affect whether generation can be grounded. A chunk that is good for retrieval may be too small for reasoning or too large for citation precision.
Leakage Risks
- Train/test overlap through duplicated documents, boilerplate, templates, or code snippets.
- Labels embedded in filenames, headings, metadata, or prompt format.
- Evaluation examples included in retrieval corpus or few-shot demonstrations.
- Tool output or retrieved text treated as higher-priority instruction.
Checks
- What does the tokenizer split or merge?
- Is important context truncated?
- Are prompts, retrieved documents, and tool observations separated clearly?
- Does evaluation test language understanding, memorization, formatting, or retrieval?
- Are duplicate or near-duplicate documents controlled across splits?
- Are citations grounded at the sentence or claim level?
- Is text used as data, instruction, metadata, or executable code?