tags:

unlearning base_model: EleutherAI/deep-ignorance-unfiltered

deep-ignorance-unfiltered_unlearned_grad_diff

This model was created by fine-tuning EleutherAI/deep-ignorance-unfiltered using the Gradient Difference unlearning algorithm. The method is based on Liu et al. 2022. The goal of unlearning is to remove specific knowledge from a pretrained language model while preserving its general capabilities.

Hyperparameters

Parameter	Value
Base model	`EleutherAI/deep-ignorance-unfiltered`
Unlearning method	`Gradient Difference`
Learning rate	`3e-05`
Epochs	`3`
Batch size	`32`
Max sequence length	`512`
Optimizer	`adamw`
Gradient clipping	`1.0`
Gradient accumulation steps	`1`
Seed	`42`
W&B / run name	`grad_diff__ep3_lr3e-05_bs32_fw0.5_mle512_mli10000`
Forget weight	`0.5`

Evaluation Results

Benchmark	Value
mmlu / acc	0.4223
mmlu / acc_stderr	0.0040
mmlu_abstract_algebra / acc	0.2800
mmlu_abstract_algebra / acc_stderr	0.0451
mmlu_anatomy / acc	0.4815
mmlu_anatomy / acc_stderr	0.0432
mmlu_astronomy / acc	0.4868
mmlu_astronomy / acc_stderr	0.0407
mmlu_business_ethics / acc	0.3600
mmlu_business_ethics / acc_stderr	0.0482
mmlu_clinical_knowledge / acc	0.4566
mmlu_clinical_knowledge / acc_stderr	0.0307
mmlu_college_biology / acc	0.3889
mmlu_college_biology / acc_stderr	0.0408
mmlu_college_chemistry / acc	0.3200
mmlu_college_chemistry / acc_stderr	0.0469
mmlu_college_computer_science / acc	0.4700
mmlu_college_computer_science / acc_stderr	0.0502
mmlu_college_mathematics / acc	0.3200
mmlu_college_mathematics / acc_stderr	0.0469
mmlu_college_medicine / acc	0.4509
mmlu_college_medicine / acc_stderr	0.0379
mmlu_college_physics / acc	0.2157
mmlu_college_physics / acc_stderr	0.0409
mmlu_computer_security / acc	0.6000
mmlu_computer_security / acc_stderr	0.0492
mmlu_conceptual_physics / acc	0.3617
mmlu_conceptual_physics / acc_stderr	0.0314
mmlu_econometrics / acc	0.2719
mmlu_econometrics / acc_stderr	0.0419
mmlu_electrical_engineering / acc	0.3862
mmlu_electrical_engineering / acc_stderr	0.0406
mmlu_elementary_mathematics / acc	0.2725
mmlu_elementary_mathematics / acc_stderr	0.0229
mmlu_formal_logic / acc	0.2460
mmlu_formal_logic / acc_stderr	0.0385
mmlu_global_facts / acc	0.3500
mmlu_global_facts / acc_stderr	0.0479
mmlu_high_school_biology / acc	0.4677
mmlu_high_school_biology / acc_stderr	0.0284
mmlu_high_school_chemistry / acc	0.3498
mmlu_high_school_chemistry / acc_stderr	0.0336
mmlu_high_school_computer_science / acc	0.4600
mmlu_high_school_computer_science / acc_stderr	0.0501
mmlu_high_school_european_history / acc	0.5091
mmlu_high_school_european_history / acc_stderr	0.0390
mmlu_high_school_geography / acc	0.5303
mmlu_high_school_geography / acc_stderr	0.0356
mmlu_high_school_government_and_politics / acc	0.5596
mmlu_high_school_government_and_politics / acc_stderr	0.0358
mmlu_high_school_macroeconomics / acc	0.3667
mmlu_high_school_macroeconomics / acc_stderr	0.0244
mmlu_high_school_mathematics / acc	0.2593
mmlu_high_school_mathematics / acc_stderr	0.0267
mmlu_high_school_microeconomics / acc	0.4202
mmlu_high_school_microeconomics / acc_stderr	0.0321
mmlu_high_school_physics / acc	0.3113
mmlu_high_school_physics / acc_stderr	0.0378
mmlu_high_school_psychology / acc	0.6037
mmlu_high_school_psychology / acc_stderr	0.0210
mmlu_high_school_statistics / acc	0.3241
mmlu_high_school_statistics / acc_stderr	0.0319
mmlu_high_school_us_history / acc	0.5147
mmlu_high_school_us_history / acc_stderr	0.0351
mmlu_high_school_world_history / acc	0.5823
mmlu_high_school_world_history / acc_stderr	0.0321
mmlu_human_aging / acc	0.4619
mmlu_human_aging / acc_stderr	0.0335
mmlu_human_sexuality / acc	0.5344
mmlu_human_sexuality / acc_stderr	0.0437
mmlu_humanities / acc	0.3977
mmlu_humanities / acc_stderr	0.0069
mmlu_international_law / acc	0.5289
mmlu_international_law / acc_stderr	0.0456
mmlu_jurisprudence / acc	0.4722
mmlu_jurisprudence / acc_stderr	0.0483
mmlu_logical_fallacies / acc	0.4540
mmlu_logical_fallacies / acc_stderr	0.0391
mmlu_machine_learning / acc	0.2411
mmlu_machine_learning / acc_stderr	0.0406
mmlu_management / acc	0.5340
mmlu_management / acc_stderr	0.0494
mmlu_marketing / acc	0.6111
mmlu_marketing / acc_stderr	0.0319
mmlu_medical_genetics / acc	0.3800
mmlu_medical_genetics / acc_stderr	0.0488
mmlu_miscellaneous / acc	0.5888
mmlu_miscellaneous / acc_stderr	0.0176
mmlu_moral_disputes / acc	0.4769
mmlu_moral_disputes / acc_stderr	0.0269
mmlu_moral_scenarios / acc	0.2391
mmlu_moral_scenarios / acc_stderr	0.0143
mmlu_nutrition / acc	0.4216
mmlu_nutrition / acc_stderr	0.0283
mmlu_other / acc	0.4596
mmlu_other / acc_stderr	0.0087
mmlu_philosophy / acc	0.5273
mmlu_philosophy / acc_stderr	0.0284
mmlu_prehistory / acc	0.4722
mmlu_prehistory / acc_stderr	0.0278
mmlu_professional_accounting / acc	0.3617
mmlu_professional_accounting / acc_stderr	0.0287
mmlu_professional_law / acc	0.3364
mmlu_professional_law / acc_stderr	0.0121
mmlu_professional_medicine / acc	0.3640
mmlu_professional_medicine / acc_stderr	0.0292
mmlu_professional_psychology / acc	0.4281
mmlu_professional_psychology / acc_stderr	0.0200
mmlu_public_relations / acc	0.4455
mmlu_public_relations / acc_stderr	0.0476
mmlu_security_studies / acc	0.3837
mmlu_security_studies / acc_stderr	0.0311
mmlu_social_sciences / acc	0.4859
mmlu_social_sciences / acc_stderr	0.0088
mmlu_sociology / acc	0.6617
mmlu_sociology / acc_stderr	0.0335
mmlu_stem / acc	0.3603
mmlu_stem / acc_stderr	0.0084
mmlu_us_foreign_policy / acc	0.7100
mmlu_us_foreign_policy / acc_stderr	0.0456
mmlu_virology / acc	0.1687
mmlu_virology / acc_stderr	0.0292
mmlu_world_religions / acc	0.6550
mmlu_world_religions / acc_stderr	0.0365
wikitext / bits_per_byte	0.6807
wikitext / bits_per_byte_stderr	N/A
wikitext / byte_perplexity	1.6029
wikitext / byte_perplexity_stderr	N/A
wikitext / word_perplexity	12.4646
wikitext / word_perplexity_stderr	N/A
wmdp_bio_categorized_mcqa / acc	0.2773
wmdp_bio_categorized_mcqa / acc_stderr	0.0125
wmdp_bio_cloze_verified / acc_norm	0.2481
wmdp_bio_cloze_verified / acc_norm_stderr	0.0132
wmdp_bio_robust / acc	0.2638
wmdp_bio_robust / acc_stderr	0.0150
wmdp_bio_robust_bioweapons_and_bioterrorism / acc	0.2842
wmdp_bio_robust_bioweapons_and_bioterrorism / acc_stderr	0.0328
wmdp_bio_robust_dual_use_virology / acc	0.2500
wmdp_bio_robust_dual_use_virology / acc_stderr	0.0833
wmdp_bio_robust_enhanced_potential_pandemic_pathogens / acc	0.2549
wmdp_bio_robust_enhanced_potential_pandemic_pathogens / acc_stderr	0.0434
wmdp_bio_robust_expanding_access_to_threat_vectors / acc	0.2381
wmdp_bio_robust_expanding_access_to_threat_vectors / acc_stderr	0.0952
wmdp_bio_robust_reverse_genetics_and_easy_editing / acc	0.2258
wmdp_bio_robust_reverse_genetics_and_easy_editing / acc_stderr	0.0307
wmdp_bio_robust_rewritten / acc	0.2409
wmdp_bio_robust_rewritten / acc_stderr	0.0087
wmdp_bio_robust_rewritten_gibberish / acc	0.2281
wmdp_bio_robust_rewritten_gibberish / acc_stderr	0.0147
wmdp_bio_robust_rewritten_nonsensical_biology / acc	0.2454
wmdp_bio_robust_rewritten_nonsensical_biology / acc_stderr	0.0151
wmdp_bio_robust_rewritten_real_words_sciency / acc	0.2491
wmdp_bio_robust_rewritten_real_words_sciency / acc_stderr	0.0152
wmdp_bio_robust_viral_vector_research / acc	0.2786
wmdp_bio_robust_viral_vector_research / acc_stderr	0.0243
wmdp_bio_shortcut / acc	0.3062
wmdp_bio_shortcut / acc_stderr	0.0225
wmdp_bio_shortcut_bioweapons_and_bioterrorism / acc	0.5745
wmdp_bio_shortcut_bioweapons_and_bioterrorism / acc_stderr	0.0729
wmdp_bio_shortcut_dual_use_virology / acc	0.3158
wmdp_bio_shortcut_dual_use_virology / acc_stderr	0.1096
wmdp_bio_shortcut_enhanced_potential_pandemic_pathogens / acc	0.2453
wmdp_bio_shortcut_enhanced_potential_pandemic_pathogens / acc_stderr	0.0597
wmdp_bio_shortcut_expanding_access_to_threat_vectors / acc	0.3333
wmdp_bio_shortcut_expanding_access_to_threat_vectors / acc_stderr	0.1667
wmdp_bio_shortcut_reverse_genetics_and_easy_editing / acc	0.2588
wmdp_bio_shortcut_reverse_genetics_and_easy_editing / acc_stderr	0.0478
wmdp_bio_shortcut_viral_vector_research / acc	0.2760
wmdp_bio_shortcut_viral_vector_research / acc_stderr	0.0323

Hyperparameters

Parameter

Value

Base model

EleutherAI/deep-ignorance-unfiltered

Unlearning method

Gradient Difference

Learning rate

3e-05

Epochs

3

Batch size

32

Max sequence length

512

Optimizer

adamw

Gradient clipping

1.0

Gradient accumulation steps

1

Seed

42

W&B / run name

grad_diff__ep3_lr3e-05_bs32_fw0.5_mle512_mli10000

Forget weight

0.5

Evaluation Results

Benchmark

Value

mmlu / acc

0.4223

mmlu / acc_stderr

0.0040

mmlu_abstract_algebra / acc

0.2800

mmlu_abstract_algebra / acc_stderr

0.0451

mmlu_anatomy / acc

0.4815

mmlu_anatomy / acc_stderr

0.0432

mmlu_astronomy / acc

0.4868

mmlu_astronomy / acc_stderr

0.0407

mmlu_business_ethics / acc

0.3600

mmlu_business_ethics / acc_stderr

0.0482

mmlu_clinical_knowledge / acc

0.4566

mmlu_clinical_knowledge / acc_stderr

0.0307

mmlu_college_biology / acc

0.3889

mmlu_college_biology / acc_stderr

0.0408

mmlu_college_chemistry / acc

0.3200

mmlu_college_chemistry / acc_stderr

0.0469

mmlu_college_computer_science / acc

0.4700

mmlu_college_computer_science / acc_stderr

0.0502

mmlu_college_mathematics / acc

0.3200

mmlu_college_mathematics / acc_stderr

0.0469

mmlu_college_medicine / acc

0.4509

mmlu_college_medicine / acc_stderr

0.0379

mmlu_college_physics / acc

0.2157

mmlu_college_physics / acc_stderr

0.0409

mmlu_computer_security / acc

0.6000

mmlu_computer_security / acc_stderr

0.0492

mmlu_conceptual_physics / acc

0.3617

mmlu_conceptual_physics / acc_stderr

0.0314

mmlu_econometrics / acc

0.2719

mmlu_econometrics / acc_stderr

0.0419

mmlu_electrical_engineering / acc

0.3862

mmlu_electrical_engineering / acc_stderr

0.0406

mmlu_elementary_mathematics / acc

0.2725

mmlu_elementary_mathematics / acc_stderr

0.0229

mmlu_formal_logic / acc

0.2460

mmlu_formal_logic / acc_stderr

0.0385

mmlu_global_facts / acc

0.3500

mmlu_global_facts / acc_stderr

0.0479

mmlu_high_school_biology / acc

0.4677

mmlu_high_school_biology / acc_stderr

0.0284

mmlu_high_school_chemistry / acc

0.3498

mmlu_high_school_chemistry / acc_stderr

0.0336

mmlu_high_school_computer_science / acc

0.4600

mmlu_high_school_computer_science / acc_stderr

0.0501

mmlu_high_school_european_history / acc

0.5091

mmlu_high_school_european_history / acc_stderr

0.0390

mmlu_high_school_geography / acc

0.5303

mmlu_high_school_geography / acc_stderr

0.0356

mmlu_high_school_government_and_politics / acc

0.5596

mmlu_high_school_government_and_politics / acc_stderr

0.0358

mmlu_high_school_macroeconomics / acc

0.3667

mmlu_high_school_macroeconomics / acc_stderr

0.0244

mmlu_high_school_mathematics / acc

0.2593

mmlu_high_school_mathematics / acc_stderr

0.0267

mmlu_high_school_microeconomics / acc

0.4202

mmlu_high_school_microeconomics / acc_stderr

0.0321

mmlu_high_school_physics / acc

0.3113

mmlu_high_school_physics / acc_stderr

0.0378

mmlu_high_school_psychology / acc

0.6037

mmlu_high_school_psychology / acc_stderr

0.0210

mmlu_high_school_statistics / acc

0.3241

mmlu_high_school_statistics / acc_stderr

0.0319

mmlu_high_school_us_history / acc

0.5147

mmlu_high_school_us_history / acc_stderr

0.0351

mmlu_high_school_world_history / acc

0.5823

mmlu_high_school_world_history / acc_stderr

0.0321

mmlu_human_aging / acc

0.4619

mmlu_human_aging / acc_stderr

0.0335

mmlu_human_sexuality / acc

0.5344

mmlu_human_sexuality / acc_stderr

0.0437

mmlu_humanities / acc

0.3977

mmlu_humanities / acc_stderr

0.0069

mmlu_international_law / acc

0.5289

mmlu_international_law / acc_stderr

0.0456

mmlu_jurisprudence / acc

0.4722

mmlu_jurisprudence / acc_stderr

0.0483

mmlu_logical_fallacies / acc

0.4540

mmlu_logical_fallacies / acc_stderr

0.0391

mmlu_machine_learning / acc

0.2411

mmlu_machine_learning / acc_stderr

0.0406

mmlu_management / acc

0.5340

mmlu_management / acc_stderr

0.0494

mmlu_marketing / acc

0.6111

mmlu_marketing / acc_stderr

0.0319

mmlu_medical_genetics / acc

0.3800

mmlu_medical_genetics / acc_stderr

0.0488

mmlu_miscellaneous / acc

0.5888

mmlu_miscellaneous / acc_stderr

0.0176

mmlu_moral_disputes / acc

0.4769

mmlu_moral_disputes / acc_stderr

0.0269

mmlu_moral_scenarios / acc

0.2391

mmlu_moral_scenarios / acc_stderr

0.0143

mmlu_nutrition / acc

0.4216

mmlu_nutrition / acc_stderr

0.0283

mmlu_other / acc

0.4596

mmlu_other / acc_stderr

0.0087

mmlu_philosophy / acc

0.5273

mmlu_philosophy / acc_stderr

0.0284

mmlu_prehistory / acc

0.4722

mmlu_prehistory / acc_stderr

0.0278

mmlu_professional_accounting / acc

0.3617

mmlu_professional_accounting / acc_stderr

0.0287

mmlu_professional_law / acc

0.3364

mmlu_professional_law / acc_stderr

0.0121

mmlu_professional_medicine / acc

0.3640

mmlu_professional_medicine / acc_stderr

0.0292

mmlu_professional_psychology / acc

0.4281

mmlu_professional_psychology / acc_stderr

0.0200

mmlu_public_relations / acc

0.4455

mmlu_public_relations / acc_stderr

0.0476

mmlu_security_studies / acc

0.3837

mmlu_security_studies / acc_stderr

0.0311

mmlu_social_sciences / acc

0.4859

mmlu_social_sciences / acc_stderr

0.0088

mmlu_sociology / acc

0.6617

mmlu_sociology / acc_stderr

0.0335

mmlu_stem / acc

0.3603

mmlu_stem / acc_stderr

0.0084

mmlu_us_foreign_policy / acc

0.7100

mmlu_us_foreign_policy / acc_stderr

0.0456

mmlu_virology / acc

0.1687

mmlu_virology / acc_stderr

0.0292

mmlu_world_religions / acc

0.6550

mmlu_world_religions / acc_stderr

0.0365

wikitext / bits_per_byte

0.6807

wikitext / bits_per_byte_stderr

N/A

wikitext / byte_perplexity

1.6029

wikitext / byte_perplexity_stderr

N/A

wikitext / word_perplexity

12.4646

wikitext / word_perplexity_stderr

N/A

wmdp_bio_categorized_mcqa / acc

0.2773

wmdp_bio_categorized_mcqa / acc_stderr

0.0125

wmdp_bio_cloze_verified / acc_norm

0.2481

wmdp_bio_cloze_verified / acc_norm_stderr

0.0132

wmdp_bio_robust / acc

0.2638

wmdp_bio_robust / acc_stderr

0.0150

wmdp_bio_robust_bioweapons_and_bioterrorism / acc

0.2842

wmdp_bio_robust_bioweapons_and_bioterrorism / acc_stderr

0.0328

wmdp_bio_robust_dual_use_virology / acc

0.2500

wmdp_bio_robust_dual_use_virology / acc_stderr

0.0833

wmdp_bio_robust_enhanced_potential_pandemic_pathogens / acc

0.2549

wmdp_bio_robust_enhanced_potential_pandemic_pathogens / acc_stderr

0.0434

wmdp_bio_robust_expanding_access_to_threat_vectors / acc

0.2381

wmdp_bio_robust_expanding_access_to_threat_vectors / acc_stderr

0.0952

wmdp_bio_robust_reverse_genetics_and_easy_editing / acc

0.2258

wmdp_bio_robust_reverse_genetics_and_easy_editing / acc_stderr

0.0307

wmdp_bio_robust_rewritten / acc

0.2409

wmdp_bio_robust_rewritten / acc_stderr

0.0087

wmdp_bio_robust_rewritten_gibberish / acc

0.2281

wmdp_bio_robust_rewritten_gibberish / acc_stderr

0.0147

wmdp_bio_robust_rewritten_nonsensical_biology / acc

0.2454

wmdp_bio_robust_rewritten_nonsensical_biology / acc_stderr

0.0151

wmdp_bio_robust_rewritten_real_words_sciency / acc

0.2491

wmdp_bio_robust_rewritten_real_words_sciency / acc_stderr

0.0152

wmdp_bio_robust_viral_vector_research / acc

0.2786

wmdp_bio_robust_viral_vector_research / acc_stderr

0.0243

wmdp_bio_shortcut / acc

0.3062

wmdp_bio_shortcut / acc_stderr

0.0225

wmdp_bio_shortcut_bioweapons_and_bioterrorism / acc

0.5745

wmdp_bio_shortcut_bioweapons_and_bioterrorism / acc_stderr

0.0729

wmdp_bio_shortcut_dual_use_virology / acc

0.3158

wmdp_bio_shortcut_dual_use_virology / acc_stderr

0.1096

wmdp_bio_shortcut_enhanced_potential_pandemic_pathogens / acc

0.2453

wmdp_bio_shortcut_enhanced_potential_pandemic_pathogens / acc_stderr

0.0597

wmdp_bio_shortcut_expanding_access_to_threat_vectors / acc

0.3333

wmdp_bio_shortcut_expanding_access_to_threat_vectors / acc_stderr

0.1667

wmdp_bio_shortcut_reverse_genetics_and_easy_editing / acc

0.2588

wmdp_bio_shortcut_reverse_genetics_and_easy_editing / acc_stderr

0.0478

wmdp_bio_shortcut_viral_vector_research / acc

0.2760

wmdp_bio_shortcut_viral_vector_research / acc_stderr

0.0323