license: other license_name: openmdw-1.1 license_link: LICENSE base_model: IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated base_model_relation: quantized pipeline_tag: text-generation language:

• en tags:
• nemotron_h
• nemotron-3.5
• thinkingcap
• abliterated
• uncensored
• moe
• mamba
• gguf
• llama.cpp
• imatrix
• mtp

ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-GGUF

GGUF quantizations of ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B, for llama.cpp and apps built on it.

Three files from 20.3 GB to 35.0 GB: Q8_0 with llama.cpp's standard mix, and two per-tensor layouts copied from public GGUFs of the base model (UD-Q4_K_XL, AD-IQ4_NL). Every file is quantized directly from the BF16 GGUF, never from another quant; the two smaller ones use an importance matrix computed on this model. In every file the MTP head is pinned to Q8_0 (see Tensor pins).

Which build? UD-Q4_K_XL (25.9 GB) is the recommended file. AD-IQ4_NL (20.3 GB) when memory is tight; Q8_0 (35.0 GB) when it is not. Llama.cpp's standard Q4_K_M and Q5_K_M were built and measured too, and are not published: each is larger than one of these files and diverges more (below).

Files

UD-Q4_K_XL is smaller than Q5_K_M with half its divergence; AD-IQ4_NL is 5.4 GB smaller than Q4_K_M and diverges less.

KLD and perplexity: llama-perplexity --kl-divergence against the BF16 GGUF on WikiText-2 test, context 2048, 60 chunks (BF16 perplexity 5.965). The Type column of the two layout files is the header's file type; their tensors follow the copied layouts. Refusals were counted on the bf16 model (21/100 with thinking off, see its card), not on each GGUF.

Why the sizes bunch up

The model's hidden size (2688) and expert width (1856) are not multiples of 256, the block size of llama.cpp's k-quants and IQ types. For those tensors llama-quantize falls back to a 32-wide type: Q4_K becomes Q5_0, Q5_K becomes Q5_1, Q6_K becomes Q8_0, and every IQ type becomes IQ4_NL. So the usual ladder collapses: everything below Q5 lands near 4.5 bits per weight. Smaller files need explicit per-tensor layouts, which is what the UD- and AD- files are.

Each of these files copies the per-tensor quantization types of a published GGUF of the base model, read from that file's header: every quantized tensor gets its type explicitly. UD-Q4_K_XL follows unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF; AD-IQ4_NL follows AtomicChat/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF. They were quantized from this repo's BF16 GGUF with this repo's importance matrix, so they are those layouts on this model, not files made or checked by Unsloth or AtomicChat. One deviation: the MTP head is Q8_0, as in every file here.

Tensor pins

Everything else in Q8_0 is Q8_0; the two layout files set every tensor explicitly.

Usage

llama-server with MTP speculative decoding:

llama-server -m ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-UD-Q4_K_XL.gguf \
  -ngl 99 -c 32768 -fa on --jinja --reasoning-format deepseek \
  --spec-type draft-mtp --spec-draft-n-max 3 -np 1 \
  --temp 1.0 --top-p 0.95

Tested 2026-10-23 with llama.cpp build 1ab7e5a (CUDA, flash attention) on a DGX Spark (GB10), -c 16384, MTP on (--spec-type draft-mtp --spec-draft-n-max 3 -np 1). Four prompts (code, arithmetic, an explanation, a translation) at temperature 0, thinking on: UD-Q4_K_XL answered all four on task (the arrival time correct), accepted 74% of drafted tokens (1,987 of 2,694) and generated 85.5 tokens/s. Q8_0 and AD-IQ4_NL were not served.

Sampling (NVIDIA's recommendation for the base model, which ThinkingCap uses unchanged): temp 1.0, top_p 0.95. Thinking is on by default; turn it off with --chat-template-kwargs '{"enable_thinking":false}'.

MTP. Support in llama.cpp is recent and flag names have changed between versions; check llama-server --help in your build. MTP was tested only with -np 1.

Other apps. LM Studio, Ollama and the like work if they bundle a llama.cpp recent enough for the Nemotron-H MoE architecture (nemotron_h_moe).

How it was made

With llama.cpp build 1ab7e5a: convert, compute the importance matrix, quantize.

python convert_hf_to_gguf.py <bf16-dir> --outtype bf16 --outfile model-BF16.gguf
llama-imatrix -m model-BF16.gguf -f calibration_datav3.txt -o imatrix.gguf -ngl 99 -c 512 --chunks 200
llama-quantize model-BF16.gguf model-Q8_0.gguf Q8_0

Every routed expert received tokens during the importance-matrix run (the fewest: 106). The UD- / AD- files take their types from a tensor-type file with one ^tensor-name$=type line per tensor, with --pure, so llama-quantize's built-in mix decides nothing:

llama-quantize --imatrix imatrix.gguf --pure --tensor-type-file UD-Q4_K_XL.tt \
  model-BF16.gguf model-UD-Q4_K_XL.gguf Q4_K_M

Calibration text for the importance matrix (used by the two layout files): bartowski's calibration_datav3.

Limitations

Everything from the bf16 card applies: no safety filter, 21/100 harmful prompts still match a refusal marker with thinking off. You are the safety layer.

Lower quants add the usual degradation on long multi-step reasoning and code. If a task works on Q8_0 and fails on a smaller file, the cause is the quantization.

License

OpenMDW License Agreement 1.1, inherited from ThinkingCap and the NVIDIA base model (see LICENSE).

Credits

bottlecapai (ThinkingCap) · trohrbaugh (abliteration) · p-e-w/heretic · ggml-org/llama.cpp · bartowski (imatrix calibration data) · unsloth and AtomicChat (tensor layouts) · NVIDIA (Nemotron 3.5 Lightning)