library_name: transformers license: apache-2.0 language:

en base_model:
Qwen/Qwen3.6-27B pipeline_tag: image-text-to-text tags:
text-generation-inference
mtp
llama-cpp
abliterated
nvfp4
nvidia
llama-cli
speculative-decoding
nvidia-blackwell
rtx-pro-6000
SM120
GB202
B200
B300
FP4
blackwell
qwen3_5
Qwen3.5
Qwen3.6
Qwen3.8
27B

Qwen3.6-27B-abliterated-NVFP4-GGUF

Qwen3.6-27B from Alibaba's Qwen team is a 27B-parameter dense multimodal model with an integrated vision encoder, 64 layers, and a native 262K-token context window extendable to 1M+ via YaRN. It delivers strong agentic coding, repository-level reasoning, and frontend development capabilities, outperforming the previous Qwen3.5-397B-A17B flagship on SWE-bench Verified (77.2), Terminal-Bench (59.3), and SkillsBench (48.2). It also scores 87.8 on GPQA-Diamond and 66.1% on BFCL-V4 for tool calling. Hybrid thinking modes preserve reasoning across multi-turn workflows. It supports vLLM, SGLang, and Unsloth, with GGUF quantization enabling local deployment, and is released under Apache-2.0.

[!NOTE] MTP — Speculative Decoding: Yes

[!NOTE] Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name	Quant Type	File Size	File Link	Description
Qwen3.6-27B-abliterated-NVFP4.gguf	NVFP4	27.6 GB	Link	NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 27B parameter model.
Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf	mmproj	931 MB	Link	Multimodal projection file. Used for vision/language models.

NVFP4 Test Run

Tested Qwen3.6-27B-abliterated-NVFP4.gguf with vision support on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Metric	Result
Average Prompt Processing Speed	1,157.93 tokens/s
Average Generation Speed	101.08 tokens/s
Average Wall Time per Image	18.90 seconds
Total Wall Time (3 Images)	56.7 seconds
Context Length	16,384 tokens
KV Cache (K)	`q8_0`
GPU	NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU VRAM Capacity	96 GB
Peak GPU VRAM Usage	27,661 MiB (27.01 GiB)
Peak GPU Utilization	96%
GPU Layers Offloaded	99 (`-ngl 99`)
Random Seed	42
Speculative Decoding	MTP (`draft_n_max=2`)
Vision Projector	`Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf`
Test Images	3

Test Shots

Shot	Task	Prompt Processing (tokens/s)	Generation (tokens/s)	Wall Time
1	Vision OCR - Image 1	1,042.72	98.58	15.42 s
2	Vision OCR - Image 2	1,181.32	102.68	19.57 s
3	Vision OCR - Image 3	1,249.76	101.97	21.70 s

Run Inference

# Download a test image
wget -O /path/to/models/input.jpeg \
  "https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/EF8cldKHhUEO3MZPZSD19.jpeg"

# Run Qwen3.6-27B Abliterated NVFP4 with vision support
/data/llama.cpp/build/bin/llama-cli \
  -m /path/to/models/Qwen3.6-27B-abliterated-NVFP4.gguf \
  --mmproj /path/to/models/Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -c 16384 \
  --cache-type-k q8_0 \
  --seed 42 \
  -n 8192 \
  -st \
  --image /path/to/models/input.jpeg \
  -p "OCR the image."

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the input image, prompt, model, software build, and runtime settings.

Snapshots (3)

download (1) download (2) download (3) download

License and Attributions

Base Model: Qwen/Qwen3.6-27B.
Source Model: huihui-ai/Huihui-Qwen3.6-27B-abliterated: Used for NVFP4 compression and quantization.
Quantization: Performed using llama.cpp by prithivMLmods.
License: This model follows the Apache 2.0 license of the original Qwen/Qwen3.6-27B base model.

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

Qwen3.6-27B-abliterated-NVFP4-GGUF

Qwen3.6-27B from Alibaba's Qwen team is a 27B-parameter dense multimodal model with an integrated vision encoder, 64 layers, and a native 262K-token context window extendable to 1M+ via YaRN. It delivers strong agentic coding, repository-level reasoning, and frontend development capabilities, outperforming the previous Qwen3.5-397B-A17B flagship on SWE-bench Verified (77.2), Terminal-Bench (59.3), and SkillsBench (48.2). It also scores 87.8 on GPQA-Diamond and 66.1% on BFCL-V4 for tool calling. Hybrid thinking modes preserve reasoning across multi-turn workflows. It supports vLLM, SGLang, and Unsloth, with GGUF quantization enabling local deployment, and is released under Apache-2.0.

[!NOTE] MTP — Speculative Decoding: Yes

[!NOTE] Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name

Quant Type

File Size

File Link

Description

Qwen3.6-27B-abliterated-NVFP4.gguf

NVFP4

27.6 GB

Link

NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 27B parameter model.

Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf

mmproj

931 MB

Link

Multimodal projection file. Used for vision/language models.

NVFP4 Test Run

Performance Metrics

Metric

Result

Average Prompt Processing Speed

1,157.93 tokens/s

Average Generation Speed

101.08 tokens/s

Average Wall Time per Image

18.90 seconds

Total Wall Time (3 Images)

56.7 seconds

Context Length

16,384 tokens

KV Cache (K)

q8_0

GPU

NVIDIA RTX PRO 6000 Blackwell Server Edition

GPU VRAM Capacity

96 GB

Peak GPU VRAM Usage

27,661 MiB (27.01 GiB)

Peak GPU Utilization

96%

GPU Layers Offloaded

99 (-ngl 99)

Random Seed

Speculative Decoding

MTP (draft_n_max=2)

Vision Projector

Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf

Test Images

Test Shots

Shot

Task

Prompt Processing (tokens/s)

Generation (tokens/s)

Wall Time

Vision OCR - Image 1

1,042.72

98.58

15.42 s

Vision OCR - Image 2

1,181.32

102.68

19.57 s

Vision OCR - Image 3

1,249.76

101.97

21.70 s

Run Inference

# Download a test image wget -O /path/to/models/input.jpeg \ "https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/EF8cldKHhUEO3MZPZSD19.jpeg" # Run Qwen3.6-27B Abliterated NVFP4 with vision support /data/llama.cpp/build/bin/llama-cli \ -m /path/to/models/Qwen3.6-27B-abliterated-NVFP4.gguf \ --mmproj /path/to/models/Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -ngl 99 \ -c 16384 \ --cache-type-k q8_0 \ --seed 42 \ -n 8192 \ -st \ --image /path/to/models/input.jpeg \ -p "OCR the image."

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the input image, prompt, model, software build, and runtime settings.