Concept dictionary
Scaling Laws
Scaling laws are empirical relationships that describe how selected measures of model performance or training behavior change as factors such as compute, data, and model size increase under specified conditions.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
Also called
- Scaling Laws
- Neural Scaling Laws
Remember it this wayScaling laws describe observed trends under stated conditions.
In plain language
Researchers fit patterns to results from multiple training runs and use them to estimate tradeoffs or expected improvement. The relationship depends on the metric, architecture, data, and range studied.
Why it matters
Scaling laws influence training plans and investment, but extrapolation can fail when methods, data quality, bottlenecks, or evaluation targets change.
Example
A team estimates how validation loss may change when training compute and dataset size increase together according to an observed relationship.
Common misconception
A scaling law is not a universal law guaranteeing that every useful capability improves predictably with more parameters or money.
Related terms
Sources
- Kaplan et al., Scaling Laws for Neural Language Modelsprimary empirical paper describing language-model scaling relationships.
Learn more
- Read any scaling claim with its metric, range, assumptions, and supporting evidence.