OPSD Scaling Trend Beyond 8B Parameters

Predicted Gain at 70B

19.6 pp

+/- 11.3 pp (model-averaged)

Bootstrap 95% CI

[10.5, 32.6]

1,000 bootstrap resamples

Best Model

Power Law

Akaike weight: 0.338

Optimal Next Experiment

140B

Maximum model disagreement

Key Findings

Finding 1: All five candidate scaling laws agree that OPSD gains continue beyond 8B parameters, but the growth rate is highly uncertain (9.1 to 32.9 pp at 70B).

Finding 2: The distribution-match component dominates at large scale, growing as N^0.95, while dark knowledge saturates around 11.5B parameters.

Finding 3: Model averaging provides the most robust extrapolation strategy across all synthetic ground-truth validation scenarios.

Finding 4: Information-theoretic analysis identifies 140B as the most informative model size for future experiments to discriminate between scaling regimes.

Interactive Scaling Law Explorer

OPSD Gain vs. Model Size

Max size: 405B Log scale:

Model Selection Results

Akaike Weights

Predictions at 70B

Model	Parameters (k)	Chi-squared	AIC	BIC	Weight	Pred. at 70B (pp)
Power Law	2	0.55	4.55	3.76	0.338	32.9 ± 6.0
Saturating	2	0.73	4.73	3.95	0.309	9.1 ± 1.2
Sigmoid	3	0.02	6.02	4.85	0.162	15.7 ± 9.7
Sqrt-Log	3	0.06	6.06	4.88	0.159	17.2 ± 2.8
Logarithmic	2	5.27	9.27	8.49	0.032	12.2 ± 0.9
Model Averaged	-	-	-	-	1.000	19.6 ± 11.3

Theoretical Gain Decomposition

The OPSD gain is decomposed into three mechanistic components: distribution match (on-policy advantage), dark knowledge transfer, and implicit regularization.

Component Contributions

Fitted Parameters

Component	Parameter	Value	Interpretation
Distribution Match	α	0.232	Scaling coefficient
Distribution Match	β	0.950	Nearly linear growth
Dark Knowledge	γ	0.100	Max gain (pp)
Dark Knowledge	N_char	11.5B	Saturation scale
Regularization	δ	3.080	Max regularization benefit
Regularization	η	0.833	Growth rate

Uncertainty Quantification

Bootstrap analysis (1,000 resamples) quantifies the combined uncertainty from data noise, model parameters, and model selection.

Bootstrap 95% Confidence Intervals at 70B

Optimal Experiment Design

Information-theoretic analysis identifies which model sizes would provide the most discriminating evidence between scaling regimes.

Extrapolating OPSD Gains Beyond 8 Billion Parameters