Asymptotic Behavior of Gradient Boosting

Spectral Filter

Three Regimes

Convergence

EBM Analysis

Normality

Effective Ridge Parameter vs. eta*T

Spectral Filter Matching

eta	T	eta*T	Best Ridge mu	Filter Dist.
0.01	10	0.1	1.345	0.0181
0.01	100	1.0	0.879	0.0011
0.10	50	5.0	0.121	0.0105
0.10	200	20.0	0.028	0.0155

As eta*T increases, the effective ridge parameter decreases monotonically, confirming that eta*T controls the implicit regularization strength of gradient boosting.

Three-Regime Structure: Ridge Parameter vs. eta*T

Filter Distance Across Regimes

The minimum filter distance occurs at eta*T ~ 1 (critical regime), where boosting most closely approximates ridge regression. Under-iterated boosting applies excessive smoothing; over-iterated boosting approaches interpolation.

Estimator Distances as n Grows

Convergence Summary

n	Kernel vs Ridge	Boulevard vs Ridge	Stump vs Ridge
50	0.359	0.431	0.580
100	0.354	0.425	0.575
200	0.365	0.437	0.579
400	0.367	0.440	0.580

With fixed (eta, T), distances remain stable rather than decreasing, suggesting that proper scaling of boosting parameters with n is required for convergence to the kernel ridge limit.

EBM Cyclic Boosting vs. Kernel Ridge Variants

EBM Distance Summary (d=3)

n	EBM vs Add. Ridge	EBM vs Full Ridge	Add. vs Full Ridge
50	0.372	0.323	0.179
100	0.389	0.334	0.190
200	0.393	0.337	0.184
300	0.383	0.325	0.183

Additive vs. full kernel ridge distance (~0.18) reflects the structural constraint of additivity. EBM distances (~0.35) suggest that convergence to the additive kernel ridge limit requires appropriate hyperparameter scaling.

Standard Deviation Decay (sqrt(n) rate)

KS Test p-values

Asymptotic Normality Test Results

n	Mean	Std	KS Statistic	p-value	Normal?
50	-0.006	0.056	0.057	0.695	Yes
100	0.004	0.033	0.033	0.996	Yes
200	0.001	0.021	0.085	0.211	Yes
400	-0.002	0.015	0.051	0.807	Yes

All KS test p-values exceed 0.05, and the standard deviation decreases at the sqrt(n) rate (0.056 to 0.015), strongly supporting asymptotic normality of the gradient boosting estimator.

Asymptotic Behavior of Standard Gradient Boosting Algorithms