base_model: HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive base_model_relation: quantized library_name: llama.cpp pipeline_tag: text-generation language:

• en
• zh tags:
• gguf
• llama.cpp
• quantized
• imatrix
• moe
• uncensored
• qwen3.6
• custom-quants license: apache-2.0

Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive — TTQ quant set

TTQ (TokenTinkerer Quant) is a ten-rung GGUF ladder built from HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive, from 3.33 bpw to 6.72 bpw. Every rung was measured on the same machine, with the same protocol, against the same reference — and every claim below is a measurement, not an estimate.

Source caveat — there is no full-precision release of this model. The fine-tune is published only as GGUF; no F16/BF16 weights exist. The highest-quality source available is therefore upstream's Q8_K_P (~10 bpw), and it serves as both the requantization source for every rung here and the reference against which all KLD / RMS figures are measured. Two consequences: every quality number below is relative to that file rather than to the true full-precision model, and an unknown fraction of each measured divergence is inherited from it and cannot be separated without a full-precision source. Comparisons between files remain valid, because they all share the same reference. The full-precision Qwen/Qwen3.6-35B-A3B base model is not a substitute — the fine-tune is a different model.

The point of the set is efficiency at a given size, and it holds two ways. Every rung beats the base repository's nearest-size quant on KLD — by 1.26× to 3.51×. And every rung's nearest-quality partner there is a 1.44 to 7.90 GiB larger file. The matched pairings below are also faster: +6% to +19% decode, +28% to +80% prefill. All with significance shown.

The last line is the shape of the whole set: where it does not win on quality, it wins by being dramatically smaller at the same quality.

The ladder

KLD is KL divergence against the model's Q8_K_P logits, measured over 40 fixed chunks of an internal evaluation set (lower is better; ± is the standard error). RMS Δp is the root-mean-square difference in the probability assigned to the actual next token, so it reads directly as percentage points. The corpus and chunking are identical for every rung here and for both reference files, which is what makes the columns comparable. tg128 is llama.cpp llama-bench decode at -ncmoe 40 (all experts on CPU), -p 512 -n 128 -r 5 — see Speed before comparing it to anything.

Full filenames are Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-<suffix>.gguf. The suffix is the only part that varies.

How to read the names

Two independent fields, and they mean different things:

• <bpw>bpw is the true measured rate, computed from the finished file's byte count. Trust this number.
• The trailing token is the file's declared general.file_type, and it is deliberately chosen to understate the true rate and to be distinct across the whole set. For example TTQ-3.33bpw-IQ3_XXS is not a plain IQ3_XXS file — it is 3.33 bpw, comfortably above the 3.06 bpw that the IQ3_XXS label nominally implies.

That second rule is not cosmetic. If two files in a repository declare the same general.file_type, the Hub's download selector collapses them and one becomes effectively invisible. Every token here is distinct and monotone, so all ten rungs stay individually downloadable.

Which rung should I use?

The one non-obvious trap: do not pick by size alone. The two smallest rungs use codebook expert formats and are the slowest files in the set (26-27 tg), while the two rungs around 18-20 GiB are the fastest (53-54 tg) — nearly 2× quicker than files barely larger than the 3.51. Details below.

Serving

Stock llama.cpp

llama-server -m TTQ-3.93bpw-Q3_K_M.gguf \
  -c 32768 -ngl 99 -fa on --jinja \
  -ctk q8_0 -ctv q8_0

--jinja makes llama.cpp apply the chat template embedded in the file (Qwen's official one) — no extra files needed. The whole point of these quants is behaviour parity with the base model, so the defaults are the right starting point.

Fitting the experts. A 35B MoE at these sizes will not fit its experts in VRAM on a consumer card, and llama.cpp handles that well: -ncmoe N keeps the experts of the first N layers on the CPU and the rest on the GPU. That dial matters more than any other flag:

• Measured on an 8 GB card, -ncmoe 36 ran ~20% faster than -ncmoe 40 (all experts on CPU).
• One step further (-ncmoe 34) bought another ~5%, for ~800 MiB of VRAM.
• Leave 0.5-1 GiB of VRAM free. On Windows/WDDM, running out does not fail cleanly — it collapses prefill (measured: 1104 → ~600 t/s). If throughput suddenly drops, check VRAM before suspecting the model.
• The optimum is rung-specific, not transferable: the 4.51 rung has an earlier cliff than the 3.93. Tune it per rung.

Sampling

What this set is served with:

--temp 0.6  --top-p 0.95  --top-k 20  --min-p 0

That is Qwen's own recommended sampling, verbatim. Their card lists, for thinking mode on precise coding tasks: temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0. The two penalties are already off by default in llama.cpp, so the command above is that preset exactly. Nothing here is a house preference.

For comparison, their card suggests two other presets: temperature=1.0, presence_penalty=1.5 for thinking mode on general tasks (hotter, more variety), and temperature=0.7, top_p=0.80 for non-thinking mode. Swap to either if it suits your use; these quants are not tuned to a sampler.

The two benchmark spot-checks below were run at these settings, not greedy — which is why they carry that caveat. All KLD, PPL and RMS figures in this card are sampler-independent: perplexity evaluation is a deterministic single pass over fixed text.

buun-llama-cpp

Same command line — the fork is a drop-in — with one addition:

• --moe-cache, which stock does not have. It caches hot experts in VRAM. Its value is rung-dependent — worth roughly +11% decode on the 3.93, roughly 0% on the 4.51. It is elastic: the pool is free-VRAM-minus-reserve and shrinks itself under pressure rather than failing, so it never blocks anything. Pass an explicit size (--moe-cache 1000); on lets it silently shrink other allocations instead of erroring, and off costs nothing on the 4.51 but ~11% on the 3.93.

With the MTP head (speculative decoding)

This repository also ships the model's multi-token-prediction head as a sidecar. Attaching it recovers most of the decode lost to quantization:

llama-server -m TTQ-3.93bpw-Q3_K_M.gguf \
  -c 32768 -ngl 99 -fa on --jinja \
  -md Qwen3.6-35B-A3B-MTP-ONLY-Q4_K_M-shared.gguf \
  -ngld 999 -ctkd q8_0 -ctvd q8_0 \
  --spec-type draft-mtp --spec-draft-n-max 2

Measured, same session, greedy, at 3.93 bpw: 37.30 → 49.35 t/s, +32%. Prefill pays a few percent for the verify batch.

-ctkd/-ctvd set the head's own KV cache — it is a separate model and needs its own. -ngld is its layer offload. If your build supports TurboQuant KV types, use them instead — they preserve more quality at a given size: turbo8 in place of q8_0, or turbo4 in place of q4_0. The measurements above were taken with TurboQuant KV; q8_0 is the stock-safe choice.

--spec-draft-n-max 2 measured best. On real workloads — the two benchmark runs below — draft acceptance came in at 84.5-95%. A depth sweep on a varied-prompt harness gave 89% → 82% → 74% → 68% from n_max 1 → 4, with decode peaking at 2: higher values trade acceptance for verify-batch cost.

The MTP head in this repository

Qwen3.6-35B-A3B-MTP-ONLY-Q4_K_M-shared.gguf — 532 MiB, a sidecar, not a quant of the model.

It is stripped: token_embd.weight and output.weight were removed and qwen35moe.nextn_shared_target_tensors = true added, so the loader borrows the target model's copies instead of duplicating ~670 MiB of weights. The unstripped head is 1192 MiB. Use it only with the matching target model and a build that supports --spec-type draft-mtp.

Speed column conditions

Read this before comparing tg128 to any other number, including numbers on other cards.

What was measured. llama-bench -ngl 999 -ncmoe 40 -t 6 -p 512 -n 128 -r 5, i.e. all expert layers on CPU, batch 1, no speculative decoding. That isolates the expert kernel, which is what differs between rungs — but it is not serving throughput. Measured serving numbers with expert layers moved to GPU and an MTP drafter attached differ substantially, and the ordering itself can invert. Same-session 2×2 at a matched config (ncmoe 36, cache 1000, np 1, greedy):

The 4.51's 44% advantage in the llama-bench column does not survive any GPU residency — with the drafter off the two are a tie, and with it on the 3.93 leads by 8%. The 4.51 also runs materially closer to the VRAM ceiling (7.7 GiB peak against the 3.93's 6.6-7.4 GiB), and on this machine how much VRAM is left over is what prevents prefill collapse. Do not read this column as throughput, and do not choose a rung for serving from it.

The speed column is not monotonic in size, and the reason is the format. Two families are in play:

A codebook format reconstructs weights by looking up a memory-resident grid; on AVX2 there is no register-resident equivalent, and the 256-512 entry tables cost more than the bytes they save. So the 14.16 GiB rung is slower than the 15.85 GiB rung, and the fastest file in the set is the 18.20 GiB one. A card listing only size and KLD would actively mislead here, which is why this column exists.

Above ~20 GiB the picture is muddier. Within the linear family the rate is roughly 2.3% decode per GiB — but format choices move it: raising the expert gate/up from Q4_K to Q5_K is speed-neutral (−0.6% for +1.9 GiB), while Q8_0 on expert tensors costs 7.7% per GiB. There is no single rate.

These numbers were measured with llama-bench's own short, highly repetitive prompt. That matters: with repetitive text, the same experts are selected every token, so the CPU's L3 cache actually helps and the all-CPU configuration is flattered. Giving the same model a realistic, varied prompt costs about 30% of decode in a server setting. Treat tg128 as a comparative expert-kernel measurement and nothing more.

How these were made

• Requantized from upstream's Q8_K_P (see the source caveat above) using the buun-llama-cpp fork and our own imatrix, computed for this model rather than taken from the base model.
• Chat template: every file embeds Qwen's official template at tokenizer.chat_template (7,764 characters, sha256 prefix e84f32a2), inherited unchanged from the upstream source and verified identical to the template published in Qwen/Qwen3.6-35B-A3B. No serving-side template is baked in, so these files behave as the base model does out of the box. If you want to reproduce the author's serving setup, it uses a separate community template; pass it explicitly with --chat-template-file.
• Expert tensors are mixed per-layer and per-tensor; the non-expert tensors (attention, SSM, shared-expert, embeddings, output) are held at Q8_0 in every rung above 5.59 bpw, which is why those rungs carry a higher byte count than a stock recipe of the same nominal type.
• The output head is kept at Q8_0 even in the smallest rungs where the embeddings drop to Q4_K. It is the last projection before the logits that KLD is measured on, and protecting it is cheap.
• The declared general.file_type is patched after each build to keep the token set distinct and monotone (see How to read the names).

Measurement protocol

Everything is measured locally, on the same machine, with the same harness:

• KLD / PPL / RMS: llama-perplexity --kl-divergence against a stored Q8_K_P reference, 40 fixed chunks of an internal evaluation set at -c 512, -ngl 999 -ncmoe 40 -t 6 --no-warmup. Reported ± is the standard error over chunks. The reference logits are derived from the same Q8_K_P file that every rung is requantized from.
• Speed: llama-bench, as above. Both columns are taken from a second consecutive invocation. The first is cold, and understates prefill by up to 1.58× and decode by about 12% — an artifact of the page cache rather than a property of the model, and the single easiest way to produce a misleading GGUF comparison.
• Significance is quoted as |Δ| / sqrt(σ₁² + σ₂²).

Hardware: RTX 3060 Ti (8 GB), Ryzen 5 5600X, 32 GB DDR4-2133, Windows. Speed numbers in particular are hardware-specific; the KLD, PPL and RMS numbers are hardware-independent and can be compared directly with any other measurement made with the same reference and chunk count.

Comparison against the base repository's quants

All files below were measured by the same protocol on the same machine, so the columns are directly comparable. _P files and IQ3_M/IQ4_XS/Q4_K_M are from the base repository.

Read that table by size, not by row order. The pattern is consistent: every file in this set beats the base repository's nearest-size quant on KLD — by 1.26× to 3.51× — and every file's nearest-quality partner there is 1.44 to 7.90 GiB larger.

Benchmark spot-checks

Run through the serving recipe — the launcher config and sampling these rungs are meant to be used with — not through the all-CPU-expert config used for the speed column. More will be added as they are run.

GPQA Diamond is the only one of these with a published comparison, and the two figures are statistically indistinguishable: at n=198 and p ≈ 0.85 the binomial standard error is ±2.5 points, so the 1.2-point gap is roughly 0.3σ. Resolving a difference that small at 95% confidence would need on the order of 3,400 questions. HumanEval gives no point of comparison here — it is not on the base model's card — so treat 97.0 as an absolute result, with a ±1.3-point binomial standard error at n=164.

What these do and do not show. They show no measurable collapse in quality: a 15.85 GiB file at 3.93 bpw answers graduate-level science at parity with the full-precision model and solves 159 of 164 Python problems. They do not give a precise retention figure, and neither benchmark could detect a 1-2 point loss. Two larger packs (MMLU-Redux, 5,330 questions; MMLU-Pro, 12,032) have ±0.35-point standard errors and would resolve what these cannot; those runs are measured in days, not hours.

Two caveats apply. The official figure comes from Qwen's own harness, whose protocol for these benchmarks is not published, so the comparison is approximate. And the sampling used here is the serving recipe's, which is not greedy.

Limits and things that did not work

Recorded because negative results are results, and because several of them contradict intuitions that this project started with.

3.33 bpw is the floor. Below it, both available routes were built and measured:

• a 2.75 bpw codebook rung (IQ2_S gate/up + IQ2_XS down) scored KLD 0.135076 — a statistical tie with Q2_K_P, the worst file in the table, at 2.84 GiB smaller. Not worth publishing.
• the linear route (Q2_K/Q3_K at the bottom) is dominated for the same reason.

The reason is measurable: the marginal KLD cost per GiB rises as the model shrinks. Dropping from 3.33 to 2.75 bpw costs 0.0318 KLD/GiB, roughly 2.4× the 0.0133 KLD/GiB of the 3.33→3.51 step. Bytes buy quality cheaply at the top of the ladder and expensively at the bottom.

Allocation shape barely matters. Three 5.58 bpw rungs were built to the same size with the expert bits placed very differently — gate-heavy, down-heavy, and with the non-expert reduced to fund more gate. They came out statistically indistinguishable (0.7-1.05σ). A fourth rung that spent 1.46 GiB on Q8_0 expert gates gained no KLD at all while losing 11% of decode. Design has to be justified by size, format family and simplicity — not by where the bits sit.

Two obvious-looking allocation principles were tested and failed: "put the bits on gate/up and take them from down" and "the non-expert is a poor place to spend". Both were contradicted at the size where they were tested. They may hold elsewhere; they are not stated here as rules.

The speed column is a comparative kernel measurement, as explained above — not throughput, and its rung ordering is not necessarily the ordering you will see in a server.

Credits and licence

• Base model: HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive, itself a fine-tune of Qwen/Qwen3.6-35B-A3B. All credit for the model belongs there; this repository only re-quantizes it.
• Quantization: TokenTinkerer, using our own imatrix for this model.
• Licence follows the base model: apache-2.0. The base model's own terms and acceptable-use policy apply.