Concept dictionary
Pretraining
Pretraining is the broad first stage of training where a model learns general patterns from a very large dataset before any narrower tuning.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
Also called
- Pretraining
- Pre-training
Remember it this wayLearn broad patterns first; specialize later.
In plain language
Pretraining is the broad initial training stage that teaches a model general patterns from a large dataset before the model is adapted to a narrower task or use.
Why it matters
Pretraining shapes the capabilities and limitations that later prompting or fine-tuning builds on. Its data, objective, and scale influence every downstream use.
Example
A language model first learns to predict text across a large corpus, then receives supervised fine-tuning for instruction following.
Common misconception
Pretraining does not make a model ready for every use. It creates a base that may still need adaptation, evaluation, safeguards, and application-specific context.
Related terms
Sources
- Jacob Devlin and colleagues, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”demonstrates broad language-model pretraining followed by task-specific fine-tuning.
Learn more
- Compare [[training|Training]], [[foundation-model|Foundation Model]], [[fine-tuning|Fine-Tuning]], and [[training-data|Training Data]] to place Pretraining in the surrounding system.
- Follow the cited source material above for the method, standard, or research context behind this draft.