base_model: lukey03/Qwen3.5-9B-abliterated library_name: gptqmodel pipeline_tag: text-generation tags:

• gptq
• gptqmodel
• quantized
• qwen3_5_text
• text-generation
• marlin
• vllm

lukey03/Qwen3.5-9B-abliterated GPTQ-Pro 4-bit g128

Overview

lukey03-Qwen3.5-9B-abliterated-gptq-pro-w4g128 is a GPTQ-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

The repository name identifies a behavior-modified or reduced-filtering lineage. That label describes the source or conversion history; it is not a guarantee of unrestricted behavior in every prompt or runtime. Test outputs carefully before sharing or deploying them.

At a glance

What is included

• *.safetensors (2 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja
• quantize_config.json
• Additional configuration, tokenizer, processor, or shard files (10 visible artifacts total)

Quick start

vLLM (documented configuration)

vllm serve groxaxo/lukey03-Qwen3.5-9B-abliterated-gptq-pro-w4g128 \
  --quantization gptq_marlin \
  --dtype float16 \
  --trust-remote-code

This command is taken from the repository documentation. Adjust tensor parallelism, context length, and cache settings to match your hardware and vLLM version.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

This is a GPTQ-Pro 4-bit / group-size-128 export of lukey03/Qwen3.5-9B-abliterated, produced with GPTQModel.

Why this checkpoint

I compared the local BF16 base model, this GPTQ-Pro export, symmetric AWQ GEMM, and asymmetric AWQ GEMM on 1x RTX 3090 with warmed generation runs and WikiText-2 perplexity:

Result: GPTQ-Pro preserved quality, while AWQ was not quality-safe on this Qwen3.5 family.

Quantization settings

• bits=4
• group_size=128
• sym=True
• desc_act=False
• format=gptq
• quant_method=gptq
• pack_dtype=int32

For Qwen3.5 text checkpoints, quantization should use batch_size=1.

Recommended runtime

For correctness or direct local validation, BACKEND.GPTQ_PRO works.

For actual deployment, the recommended path is:

1. quantize with GPTQ-Pro
2. serve with Marlin
3. for highest throughput on this model family, use the patched Qwen3.5 vLLM wrapper from GPTQ-Pro

The repo-validated fast path is vLLM + gptq_marlin through:

• scripts/serve_vllm_qwen35.py

That wrapper now auto-detects qwen3_5_text from either a local folder or a Hub repo ID.

Quickstart: GPTQModel + Marlin

from transformers import AutoTokenizer
from gptqmodel import GPTQModel, BACKEND

MODEL_ID = "groxaxo/lukey03-Qwen3.5-9B-abliterated-gptq-pro-w4g128"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = GPTQModel.load(
    MODEL_ID,
    backend=BACKEND.MARLIN,
    device="cuda:0",
    trust_remote_code=True,
)

prompt = "State one short fact about model quantization."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Quickstart: patched vLLM serve path

Clone the matching repo branch first so you get the Qwen3.5 wrapper and patches:

git clone https://github.com/groxaxo/GPTQ-Pro.git
cd GPTQ-Pro
git checkout gptq-pro-cuda-kernel

Then launch the model:

CUDA_VISIBLE_DEVICES=0 \
python scripts/serve_vllm_qwen35.py \
  --model groxaxo/lukey03-Qwen3.5-9B-abliterated-gptq-pro-w4g128 \
  --served-model-name qwen35-9b-gptq-pro \
  --host 0.0.0.0 \
  --port 8011 \
  --tensor-parallel-size 1

Two-GPU shared-host launch:

CUDA_VISIBLE_DEVICES=0,1 \
python scripts/serve_vllm_qwen35.py \
  --model groxaxo/lukey03-Qwen3.5-9B-abliterated-gptq-pro-w4g128 \
  --served-model-name qwen35-9b-gptq-pro \
  --host 0.0.0.0 \
  --port 8012 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.4

When the fast path is active, the logs should contain:

Using MarlinLinearKernel for GPTQMarlinLinearMethod

Speed demon settings for future runs

If you want the same tradeoff in future Qwen3.5 quantization + deployment runs:

• quantize with GPTQ-Pro, bits=4, group_size=128, sym=True, desc_act=False
• keep batch_size=1 during Qwen3.5 quantization
• deploy with BACKEND.MARLIN or the patched vLLM wrapper
• prefer tensor_parallel_size=1 first, then scale to 2 GPUs if needed
• on shared hosts, start with --gpu-memory-utilization 0.4
• confirm the logs show MarlinLinearKernel

Notes

• BACKEND.GPTQ_PRO is a functional runtime path, but it is not the throughput-optimized serving path.
• I did not publish the AWQ exports for this model family as the recommended release because both symmetric and asymmetric AWQ failed the quality check on this benchmark setup.