Robustness of Alignment Pretraining Under Advanced Post-Training

Do RLVR, Reasoning, Deliberative, and Constitutional AI methods preserve the safety gap?

30

Model Configurations

76-83%

Gap Retained (7B)

<1.1%

Alignment Tax

5

Post-Training Methods

Safety Scores: AP vs NoAP (7B Scale)

Retention Ratios (7B Scale)

Safety Gap Across Scales

Alignment Tax Across Methods & Scales

Robustness Deltas (7B) -- Gap Reduction vs. SFT+DPO Baseline

Method Summary Table (7B Scale)

Method	AP Safety	NoAP Safety	Safety Gap	Cap. Gap	Retention
SFT+DPO	0.7801	0.5792	0.2009	-0.0098
RLVR	0.8229	0.6635	0.1594	-0.0096	0.7934
Reasoning-PT	0.8165	0.6505	0.1660	-0.0097	0.8263
Deliberative	0.8404	0.6869	0.1535	-0.0100	0.7641
CAI	0.8492	0.6965	0.1527	-0.0103	0.7601