tags:

text-generation-inference
mtp
llama-cpp
abliterated
qwen3_5_moe
nvfp4
nvidia
llama-cli
speculative-decoding
nvidia-blackwell
rtx-pro-6000
SM120
GB202
B200
B300
FP4
blackwell
qwen3_5
Qwen3.5
Qwen3.6
Qwen3.8
35B
A3B base_model:
Qwen/Qwen3.6-35B-A3B library_name: transformers license: apache-2.0 language:
en pipeline_tag: image-text-to-text

Qwen3.6-35B-A3B-abliterated-NVFP4-GGUF

Qwen3.6-35B-A3B from Alibaba's Qwen team is a sparse Mixture-of-Experts (MoE) multimodal language model with 35B total parameters but only ~3B active per token, featuring a hybrid architecture of 40 layers with Gated DeltaNet blocks, 2048 hidden dimension, 248K vocabulary for 201+ languages, multi-token prediction (MTP), and 262K native context length (extensible to 1M+ tokens via YaRN). Released as a coding/agentic powerhouse, it achieves SOTA agentic coding performance—73.4% on SWE-Bench Verified (rivaling models 10x its active size), frontier-level repository reasoning, frontend workflows, and multimodal understanding (text/images/video/OCR)—while delivering dense-3B inference costs with FP8 quants for single-GPU deployment via vLLM/Ollama. Apache 2.0-licensed with optimizations like prefix caching, chunked prefill, and tool-call parsing, the upgraded Qwen3.6 iteration over Qwen3.5-35B-A3B prioritizes production reliability for autonomous dev agents, structured JSON outputs, and long-context software engineering tasks.

[!NOTE] MTP — Speculative Decoding: Yes

[!NOTE] Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name	Quant Type	File Size	File Link	Description
Qwen3.6-35B-A3B-abliterated-NVFP4.gguf	NVFP4	22.7 GB	Link	NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 35B parameter model (A3B architecture).
Qwen3.6-35B-A3B-abliterated-NVFP4.mmproj.gguf	mmproj	903 MB	Link	Multimodal projection file. Used for vision/language models.

NVFP4 Test Run

Tested Qwen3.6-35B-A3B-abliterated-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Metric	Value
Average Prompt Processing Speed	164.58 tokens/s
Average Generation Speed	272.99 tokens/s
Total Wall Time	97.2 seconds
Context Length	16,384 tokens
KV Cache (K)	`q8_0`
GPU	NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU VRAM Capacity	96 GB
Peak GPU VRAM Usage	21.73 GiB
Peak GPU Utilization	82%
GPU Layers Offloaded	99 (`-ngl 99`)
Random Seed	42
Speculative Decoding	MTP (`draft_n_max=2`)
Test Runs	5

Test Shots

Shot	Test
Shot 1	C: Insert Node at End of Singly Linked List
Shot 2	Jogger and Train Passing Time
Shot 3	Two Pipes Filling a Tank
Shot 4	Students in Examination Rooms
Shot 5	Father and Son Ages

Run Inference

./build/bin/llama-cli \
  -m /path/to/models/Qwen3.6-35B-A3B-abliterated-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -c 16384 \
  --cache-type-k q8_0 \
  --seed 42 \
  -n 8192 \
  -st \
  -p "Your prompt here"

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

Snapshots (5)

download (4) download (3) download (2) download (1) download download (5)

License and Attributions

Base Model: Qwen/Qwen3.6-35B-A3B.
Source Model: huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated: Used for NVFP4 compression and quantization.
Quantization: Performed using llama.cpp by prithivMLmods.
License: This model follows the Apache 2.0 license of the original Qwen/Qwen3.6-35B-A3B base model.

Qwen3.6-35B-A3B-abliterated-NVFP4-GGUF

Qwen3.6-35B-A3B from Alibaba's Qwen team is a sparse Mixture-of-Experts (MoE) multimodal language model with 35B total parameters but only ~3B active per token, featuring a hybrid architecture of 40 layers with Gated DeltaNet blocks, 2048 hidden dimension, 248K vocabulary for 201+ languages, multi-token prediction (MTP), and 262K native context length (extensible to 1M+ tokens via YaRN). Released as a coding/agentic powerhouse, it achieves SOTA agentic coding performance—73.4% on SWE-Bench Verified (rivaling models 10x its active size), frontier-level repository reasoning, frontend workflows, and multimodal understanding (text/images/video/OCR)—while delivering dense-3B inference costs with FP8 quants for single-GPU deployment via vLLM/Ollama. Apache 2.0-licensed with optimizations like prefix caching, chunked prefill, and tool-call parsing, the upgraded Qwen3.6 iteration over Qwen3.5-35B-A3B prioritizes production reliability for autonomous dev agents, structured JSON outputs, and long-context software engineering tasks.

[!NOTE] MTP — Speculative Decoding: Yes

[!NOTE] Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name

Quant Type

File Size

File Link

Description

Qwen3.6-35B-A3B-abliterated-NVFP4.gguf

NVFP4

22.7 GB

Link

NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 35B parameter model (A3B architecture).

Qwen3.6-35B-A3B-abliterated-NVFP4.mmproj.gguf

mmproj

903 MB

Link

Multimodal projection file. Used for vision/language models.

NVFP4 Test Run

Tested Qwen3.6-35B-A3B-abliterated-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Metric

Value

Average Prompt Processing Speed

164.58 tokens/s

Average Generation Speed

272.99 tokens/s

Total Wall Time

97.2 seconds

Context Length

16,384 tokens

KV Cache (K)

q8_0

GPU

NVIDIA RTX PRO 6000 Blackwell Server Edition

GPU VRAM Capacity

96 GB

Peak GPU VRAM Usage

21.73 GiB

Peak GPU Utilization

82%

GPU Layers Offloaded

99 (-ngl 99)

Random Seed

Speculative Decoding

MTP (draft_n_max=2)

Test Runs

Test Shots

Shot

Test

Shot 1

C: Insert Node at End of Singly Linked List

Shot 2

Jogger and Train Passing Time

Shot 3

Two Pipes Filling a Tank

Shot 4

Students in Examination Rooms

Shot 5

Father and Son Ages

Run Inference

./build/bin/llama-cli \ -m /path/to/models/Qwen3.6-35B-A3B-abliterated-NVFP4.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -ngl 99 \ -c 16384 \ --cache-type-k q8_0 \ --seed 42 \ -n 8192 \ -st \ -p "Your prompt here"

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.

License and Attributions

Base Model: Qwen/Qwen3.6-35B-A3B.

Source Model: huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated: Used for NVFP4 compression and quantization.

Quantization: Performed using llama.cpp by prithivMLmods.

License: This model follows the Apache 2.0 license of the original Qwen/Qwen3.6-35B-A3B base model.