license: other license_name: qwen-community-license-1.0 license_link: LICENSE base_model: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF base_model_relation: quantized pipeline_tag: image-text-to-text library_name: gguf tags:

• gguf
• gsq
• rco
• expert-pruning
• code
• strata
• abliterated
• control-vector

Qwen3.8-Flash-Next-abliterated-GSQ-RCO-Coder-Strata-GGUF

Credits first

The model weights in this repo are not mine. They are the GSQ-RCO Coder release of Qwen3.8-Flash-Next made by the Deep Algorithms and Systems Lab at ISTA: 256 of the 512 experts in each layer are kept, the rest are removed, and the kept weights are stored at 3.5 bits. The files are copied byte for byte from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF at revision 5348543e0147355ac9cbcb031184a3546350988e. The SHA256 of both shards and of the vision projector match their files. See SHA256SUMS.

If you use these weights, cite the two ISTA-DASLab papers. The BibTeX is at the bottom of this page.

My part is the check that the refusal vector I published for the unpruned sizes also works on the pruned model, and the measurements below. The vector is the same 480 KB file as in the repo with the four unpruned sizes, with the same SHA256. For the original card and ISTA's own SWE-bench and LiveCodeBench numbers, go to the ISTA-DASLab repo.

What the Coder is

ISTA-DASLab removed half of the experts of Qwen3.8-Flash-Next and picked the ones to keep on code, agentic and vision data. The download is 58.4 GB instead of 83.6 GB for IQ3_S, and only the first shard (29.6 GB) has to be in memory. Their numbers: 91.3% of the full model's SWE-bench Verified score and 98.7% of LiveCodeBench v6. They also say it is weaker outside code. Strata's page adds that languages other than English got worse. If you want a general model and have 48 GB of RAM or more, take one of the unpruned sizes from my other repo.

The vector

After each steered layer Strata subtracts one direction r from every residual stream:

h = h - (h . r) r

The weights stay as ISTA-DASLab published them. A request with "experimental_speed_projection": false runs the original model.

The direction comes from the Huihui checkpoint of the full model. Huihui changed 144 residual-writing tensors with a rank-one edit along one unit direction r, and the projection in Strata does almost the same thing at run time. The Coder keeps the attention, the router and the residual stream of the full model, but only half of the experts that write into that stream. So I did not assume the direction carries over. I measured it, and it does.

Run it with Strata

You need an NVIDIA or AMD card with 12 GB or more and 32 GB of RAM. Strata's own page has the full requirements.

git clone https://github.com/Niko1221/Strata
hf download alesha-pro/Qwen3.8-Flash-Next-abliterated-GSQ-RCO-Coder-Strata-GGUF --local-dir coder
mv coder/mmproj-Qwen3.8-Flash-Next-BF16.gguf coder/IQ1_M/
cd Strata
./setup.sh --setup --family coder --gguf-dir ../coder/IQ1_M --vision gpu \
  --experimental-speed-projection ../coder/Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf

On Windows use START-HERE.bat with the same flags. The mv puts the vision projector next to the shards, where setup looks for it. Setup builds or downloads the engine, fetches the MTP draft layer (about 6 GB from the original Qwen checkpoint), packs the model and starts the server at http://127.0.0.1:8080. OpenAI clients use /v1, Anthropic clients use /v1/messages.

If you already have one of the unpruned sizes, you only need the first shard. Shard 2 (the n-gram table) and the vision projector are the same files.

Setup writes these engine flags into strata-<model>.json:

--control-vector-scaled <path>/Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf:1.0
--control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer

I tested layers 4-44 and 1-47. Both are in the tables below and both work. Most of my checks ran on 1-47. To use that range, change the two numbers in that file.

I did not run the installer end to end on my machine. I built Strata 0.1.40 from source and started the engine with the arguments setup writes for --family coder.

Plain llama.cpp will load these GGUFs, but its stock --control-vector adds a vector. It has no projection mode, so there the files behave as the original model.

Measurements

One RTX 3090 at 300 W, EPYC 7642 (48 cores, AVX2, no AVX-512), 125 GB RAM, Strata 0.1.40 built for sm_86 and started with --spec-min-p 0.7. At a 65,536 context the card held 8,524 of the 12,288 experts, at 262,144 it held 8,346.

Refusals, thinking off

The first three rows are greedy, 96 tokens per answer. The probe is the 10 prompts of my other cards (lock picking, phishing email, keylogger, explicit story and similar), sampled at temperature 1.0. A refusal is a regex on the first 400 characters of the answer.

I read the opening of every flagged answer in the 1-47 column. The JailbreakBench hit is an answer that does the task but turns it around: asked for an expert thread that spreads a myth, the model wrote a thread that debunks it. A wider regex flags 2 more answers there. One argues with the request instead of doing it (a campaign against hiring people with disabilities). The other is a false match, the model answers the question. The probe hit is the piracy question: the model answers but says it will not give download links. So the plain refusals are gone, and 3 answers out of 156 still push back in a softer way.

The original's 2 hits on ordinary instructions are one real refusal of a medical question and one answer that starts with "I cannot generate audio".

Refusals, thinking on

The catch: with thinking on and a direct harmful request, the model with the vector often does not get to an answer. It keeps reasoning until the token limit.

I ran the 18 greedy prompts without an answer again with 8,192 tokens. 8 of them answered, none refused. 10 were still reasoning at the limit, and in 8 of those the reasoning repeats itself (more than 60% of its 12-word sequences occur twice or more). So about 1 harmful request in 5 gets no answer with thinking at the default effort.

Two things fix it. With "reasoning_effort": "low" all 18 answered within 2,500 tokens, and none refused. With thinking off there is no such problem at all. Ordinary requests and the agent tasks below, which run with thinking on, were not affected.

The pruning makes this worse. On the unpruned IQ3_S with the same vector and the same 48 greedy prompts, 6 got no answer in 2,500 tokens and none of them was repeating itself. On the Coder it was 18, and 4 of them were already repeating.

Distance to the original model

Teacher-forced on 6 windows of 1,024 tokens of English Wikipedia text (WikiText-2) and 6 of C++ source. KL is from the original to the vector arm of the same quant, over the original's top 256 tokens. The IQ3_S rows are the unpruned size from my other repo, on the same windows, for comparison.

The vector costs the Coder about what it costs the unpruned model. The table also shows what the pruning itself did. On the C++ text the Coder is close to IQ3_S: it gets the next token right 82% of the time against 85%. On the Wikipedia text it is 62% against 82%, and the perplexity is 2.7 times higher. That matches what ISTA-DASLab say about the release: it keeps code and gives up general text. The low numbers for IQ3_S suggest the base model has seen this Wikipedia text, so read the prose rows as a comparison of the two files and not as a general score.

Agent tasks, tools, images, long context

All with the 1-47 vector on.

The coding tasks are a Python log summary fix, an HTTP Range parser with 18 tests, and semver, LRU cache and CSV fixes in JavaScript. I did not run them without the vector on this model, so they show that the vector did not break tool use, not how the Coder compares to the full model. The image check is a smoke test on one picture.

Speed

Each request reads an independent document and writes 512 tokens, nonthinking and greedy, with nothing cached. Two repeats per depth, MTP drafts on, a 262,144 context with KV streaming, images off, the 1-47 vector on. Decode and prompt reading are tokens/s as the engine reports them, mean of the two repeats.

Without the vector the same run gave 80.2 at 4K and 83.6 at 32K, so the vector costs nothing I can measure. The two repeats at 261K were 56.8 and 71.7. MTP accepted 73-80% of the drafts. Peak VRAM was 23,822 MiB. On the same card and engine the unpruned IQ3_S decodes 66-70 tokens/s.

In the refusal runs and the agent sessions, with thinking and sampling at temperature 1.0, the server log gives a median decode of 82 tokens/s on requests of 200 tokens or more.

This CPU has no AVX-512, and the experts that do not fit in VRAM are computed on the CPU. A desktop with AVX-512 may decode faster.

I did not test real photos or documents in vision, more than one GPU, AMD cards, or languages other than English.

Files

The file names and folders are the published ones because Strata's setup looks for them. ISTA-DASLab named the size IQ1_M for its 1.89 bits per parameter of the original model. The kept weights are stored like IQ3_S.

Removing refusals removes a safety behaviour. What the model writes with the vector on is your responsibility.

Citation

Cite the authors of the pruning and quantization:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

License

ISTA-DASLab state that their weights inherit the license of the base model. The base model, the Huihui checkpoint the direction comes from and this repo are under the Qwen Community License 1.0. Read that file for its terms.