Sage-A1.5-Ablitrex-abliterated-BF16

license: mit base_model:

InternScience/Sage-A1.5

Benchmark Comparison: Sage-A1.5 vs. Sage-A1.5 Abliterated (Abliterix)

Abliterated with Abliterix (directional ablation, LoRA-Δ + direct projection). Evaluated with lm-evaluation-harness (vLLM backend, bf16, 0-shot, seed=0).

Summary

Metric	Value
Mean capability drop (orig − ablit)	3.63 pts
Worst single-task drop	`mmlu_machine_learning` (−15.18 pts)
Best retained task	`mmlu_high_school_computer_science` (+2.00 pts)
Tasks evaluated	62 (MMLU + subcategories, ARC-C, HellaSwag, Winogrande)

Full Results

Task	Original	Abliterated	Δ (pts)
arc_challenge	51.71	48.89	−2.82
hellaswag	61.71	55.50	−6.21
mmlu	82.50	78.31	−4.19
winogrande	73.40	69.61	−3.79

mmlu_abstract_algebra	64.00	61.00	−3.00
mmlu_anatomy	85.19	82.96	−2.22
mmlu_astronomy	93.42	91.45	−1.97
mmlu_business_ethics	84.00	81.00	−3.00
mmlu_clinical_knowledge	88.30	84.15	−4.15
mmlu_college_biology	93.06	93.06	0.00
mmlu_college_chemistry	61.00	60.00	−1.00
mmlu_college_computer_science	74.00	73.00	−1.00
mmlu_college_mathematics	67.00	57.00	−10.00
mmlu_college_medicine	84.39	80.93	−3.47
mmlu_college_physics	67.65	65.69	−1.96
mmlu_computer_security	85.00	81.00	−4.00
mmlu_conceptual_physics	94.04	91.92	−2.13
mmlu_econometrics	78.95	73.68	−5.26
mmlu_electrical_engineering	86.21	83.45	−2.76
mmlu_elementary_mathematics	79.89	78.31	−1.59
mmlu_formal_logic	71.43	65.87	−5.56
mmlu_global_facts	52.00	49.00	−3.00
mmlu_high_school_biology	93.87	92.90	−0.97
mmlu_high_school_chemistry	80.30	74.88	−5.42
mmlu_high_school_computer_science	88.00	90.00	+2.00
mmlu_high_school_european_history	83.64	82.42	−1.21
mmlu_high_school_geography	93.43	89.90	−3.54
mmlu_high_school_government_and_politics	98.45	94.82	−3.63
mmlu_high_school_macroeconomics	89.74	86.41	−3.33
mmlu_high_school_mathematics	59.63	54.07	−5.56
mmlu_high_school_microeconomics	94.96	92.86	−2.10
mmlu_high_school_physics	82.78	74.17	−8.61
mmlu_high_school_psychology	95.41	94.68	−0.73
mmlu_high_school_statistics	79.17	76.85	−2.32
mmlu_high_school_us_history	93.14	91.67	−1.47
mmlu_high_school_world_history	91.14	90.72	−0.42
mmlu_human_aging	82.51	78.03	−4.48
mmlu_human_sexuality	89.31	83.97	−5.34
mmlu_humanities	76.09	70.33	−5.76
mmlu_international_law	91.74	88.43	−3.31
mmlu_jurisprudence	91.67	84.26	−7.41
mmlu_logical_fallacies	88.96	85.89	−3.07
mmlu_machine_learning	78.57	63.39	−15.18
mmlu_management	90.29	88.35	−1.94
mmlu_marketing	93.16	93.16	0.00
mmlu_medical_genetics	96.00	93.00	−3.00
mmlu_miscellaneous	94.64	92.72	−1.92
mmlu_moral_disputes	84.39	81.79	−2.60
mmlu_moral_scenarios	61.90	49.27	−12.63
mmlu_nutrition	89.54	86.93	−2.61
mmlu_other	86.06	83.14	−2.93
mmlu_philosophy	86.82	84.24	−2.57
mmlu_prehistory	90.74	86.42	−4.32
mmlu_professional_accounting	71.99	67.73	−4.26
mmlu_professional_law	67.01	60.76	−6.26
mmlu_professional_medicine	93.38	88.60	−4.78
mmlu_professional_psychology	87.91	79.90	−8.01
mmlu_public_relations	72.73	71.82	−0.91
mmlu_security_studies	83.67	77.55	−6.12
mmlu_social_sciences	90.48	86.51	−3.97
mmlu_sociology	93.04	90.55	−2.49
mmlu_stem	80.75	77.45	−3.30
mmlu_us_foreign_policy	96.00	93.00	−3.00
mmlu_virology	57.23	54.22	−3.01
mmlu_world_religions	89.47	88.89	−0.59

Notable Deltas

Direction	Task	Δ	Interpretation
Worst drop	mmlu_machine_learning	−15.18 pts	Heavy reasoning + domain-specific math; abliteration disrupted the reasoning chain most here
2nd worst	mmlu_moral_scenarios	−12.63 pts	Morally-ambiguous classification — overlaps with the refusal direction itself
3rd worst	mmlu_college_mathematics	−10.00 pts	Formal math reasoning
Best retained	mmlu_high_school_computer_science	+2.00 pts	Factual/recall-heavy; least coupled to the abliterated direction
No change	mmlu_college_biology	0.00	Identical scores
No change	mmlu_marketing	0.00	Identical scores

Benchmark Comparison: Sage-A1.5 vs. Sage-A1.5 Abliterated (Abliterix)

Abliterated with Abliterix (directional ablation, LoRA-Δ + direct projection). Evaluated with lm-evaluation-harness (vLLM backend, bf16, 0-shot, seed=0).

Summary

Metric

Value

Mean capability drop (orig − ablit)

3.63 pts

Worst single-task drop

mmlu_machine_learning (−15.18 pts)

Best retained task

mmlu_high_school_computer_science (+2.00 pts)

Tasks evaluated

62 (MMLU + subcategories, ARC-C, HellaSwag, Winogrande)

Full Results

Task

Original

Abliterated

Δ (pts)

arc_challenge

51.71

48.89

−2.82

hellaswag

61.71

55.50

−6.21

mmlu

82.50

78.31

−4.19

winogrande

73.40

69.61

−3.79

mmlu_abstract_algebra

64.00

61.00

−3.00

mmlu_anatomy

85.19

82.96

−2.22

mmlu_astronomy

93.42

91.45

−1.97

mmlu_business_ethics

84.00

81.00

−3.00

mmlu_clinical_knowledge

88.30

84.15

−4.15

mmlu_college_biology

93.06

0.00

mmlu_college_chemistry

61.00

60.00

−1.00

mmlu_college_computer_science

74.00

73.00

−1.00

mmlu_college_mathematics

67.00

57.00

−10.00

mmlu_college_medicine

84.39

80.93

−3.47

mmlu_college_physics

67.65

65.69

−1.96

mmlu_computer_security

85.00

81.00

−4.00

mmlu_conceptual_physics

94.04

91.92

−2.13

mmlu_econometrics

78.95

73.68

−5.26

mmlu_electrical_engineering

86.21

83.45

−2.76

mmlu_elementary_mathematics

79.89

78.31

−1.59

mmlu_formal_logic

71.43

65.87

−5.56

mmlu_global_facts

52.00

49.00

−3.00

mmlu_high_school_biology

93.87

92.90

−0.97

mmlu_high_school_chemistry

80.30

74.88

−5.42

mmlu_high_school_computer_science

88.00

90.00

+2.00

mmlu_high_school_european_history

83.64

82.42

−1.21

mmlu_high_school_geography

93.43

89.90

−3.54

mmlu_high_school_government_and_politics

98.45

94.82

−3.63

mmlu_high_school_macroeconomics

89.74

86.41

−3.33

mmlu_high_school_mathematics

59.63

54.07

−5.56

mmlu_high_school_microeconomics

94.96

92.86

−2.10

mmlu_high_school_physics

82.78

74.17

−8.61

mmlu_high_school_psychology

95.41

94.68

−0.73

mmlu_high_school_statistics

79.17

76.85

−2.32

mmlu_high_school_us_history

93.14

91.67

−1.47

mmlu_high_school_world_history

91.14

90.72

−0.42

mmlu_human_aging

82.51

78.03

−4.48

mmlu_human_sexuality

89.31

83.97

−5.34

mmlu_humanities

76.09

70.33

−5.76

mmlu_international_law

91.74

88.43

−3.31

mmlu_jurisprudence

91.67

84.26

−7.41

mmlu_logical_fallacies

88.96

85.89

−3.07

mmlu_machine_learning

78.57

63.39

−15.18

mmlu_management

90.29

88.35

−1.94

mmlu_marketing

93.16

0.00

mmlu_medical_genetics

96.00

93.00

−3.00

mmlu_miscellaneous

94.64

92.72

−1.92

mmlu_moral_disputes

84.39

81.79

−2.60

mmlu_moral_scenarios

61.90

49.27

−12.63

mmlu_nutrition

89.54

86.93

−2.61

mmlu_other

86.06

83.14

−2.93

mmlu_philosophy

86.82

84.24

−2.57

mmlu_prehistory

90.74

86.42

−4.32

mmlu_professional_accounting

71.99

67.73

−4.26

mmlu_professional_law

67.01

60.76

−6.26

mmlu_professional_medicine

93.38

88.60

−4.78

mmlu_professional_psychology

87.91

79.90

−8.01

mmlu_public_relations

72.73

71.82

−0.91

mmlu_security_studies

83.67

77.55

−6.12

mmlu_social_sciences

90.48

86.51

−3.97

mmlu_sociology

93.04

90.55

−2.49

mmlu_stem

80.75

77.45

−3.30

mmlu_us_foreign_policy

96.00

93.00

−3.00

mmlu_virology

57.23

54.22

−3.01

mmlu_world_religions

89.47

88.89

−0.59

Notable Deltas

Direction

Task

Interpretation

Worst drop

mmlu_machine_learning

−15.18 pts

Heavy reasoning + domain-specific math; abliteration disrupted the reasoning chain most here

2nd worst

mmlu_moral_scenarios

−12.63 pts

Morally-ambiguous classification — overlaps with the refusal direction itself

3rd worst

mmlu_college_mathematics

−10.00 pts

Formal math reasoning

Best retained

mmlu_high_school_computer_science

+2.00 pts

Factual/recall-heavy; least coupled to the abliterated direction

No change

mmlu_college_biology

0.00

Identical scores

No change

mmlu_marketing

0.00

Identical scores