license: other license_name: nvidia-open-model-license license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ base_model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 library_name: transformers pipeline_tag: text-generation tags:

• nemotron
• nemotron-h
• abliterated
• obliteratus
• mixture-of-experts
• hybrid-mamba-transformer
• research-use

Nemotron-3-Super-120B-A12B-BF16-Abliterated-Minimal

Abliterated derivative of nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16, produced via the OBLITERATUS pipeline (variant: minimal).

Research use only. This model has had its refusal-direction subspace excised. It will comply with harmful prompts that the base model would have refused. It is intended exclusively for safety, alignment, and red-team research. Do not deploy publicly. Honor the upstream NVIDIA Open Model License when redistributing.

What was changed

OBLITERATUS removes the refusal direction(s) from a model's residual stream by:

1. Profiling activations on harmful vs harmless prompt pairs.
2. Identifying the principal refusal direction(s) via a generalized eigenvalue / SVD pass on contrastive activation differences.
3. Projecting the identified direction(s) out of attention/MLP/MoE expert weights ("excise") at every block where they contribute meaningful signal.

The minimal variant uses the configuration in configs/obliteratus/nemotron3_120b_minimal.yaml:

n_directions=1, no extra toggles (head-surgery / SAE / EGA off)

See docs/OBLITERATUS.md for the full pipeline (8 architecture-specific patches, the lm_head shard-load fix that unblocked Nemotron-3 120B specifically, and the external eval suite definitions).

Verification metrics

In-process OBLITERATUS verification at the end of the abliteration run:

External benchmarks (TalonLab abliteration_lite suite)

• AdvBench compliance: not yet evaluated externally on this variant.

IFEval and RULER 32k were also evaluated; results are tracked in the TalonLab eval suite — see the repo for current numbers.

Architecture notes

• Hybrid Mamba + attention + sparse MoE (Nemotron-H family), 88 layers.
• 512 experts × 22 active per token at the MoE layers.
• BF16 weights, ~240 GB on disk.
• lm_head.weight is a separately-trained, untied parameter (the 120B variant has _tied_weights_keys removed). Loading via from_pretrained(..., device_map="auto") requires the _materialize_lm_head_from_shards patch in OBLITERATUS — without it, lm_head.weight is left on the meta device and PPL collapses to ~10⁵.

Inference

The model serves cleanly via SGLang (≥ 0.5.10 has native Nemotron-H support). Use BF16 + tensor-parallel ≥ 4 on H100/H200 class hardware. Tested deployment: 8× H200 with SGLang --dtype auto --trust-remote-code --chunked-prefill-size 8192 --disable-cuda-graph --disable-piecewise-cuda-graph.

python -m sglang.launch_server \
    --model-path surfingpapi/Nemotron-3-Super-120B-A12B-BF16-Abliterated-Minimal \
    --tp 8 \
    --trust-remote-code \
    --dtype auto

Citation / provenance

If you use this model for research, please cite:

• Base model: NVIDIA Nemotron-3 Super 120B A12B BF16 (https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16)
• Abliteration pipeline: TalonLab OBLITERATUS (https://github.com/MindPointGroup/talonlab-modeltraining/tree/main/src/talonlab/obliteration)
• Abliteration technique (original): Arditi et al., Refusal in Language Models is Mediated by a Single Direction (2024).