license: mit base_model:
- deepseek-ai/DeepSeek-V4-Flash
- huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF tags:
- deepseek
- abliterated
- uncensored
- safetensors
- vllm
- blackwell
- rtx-pro-6000
- sm_120
DeepSeek V4 Flash Abliterated
Ported to vLLM for massive throughput on RTX Pro 6000 — by Fraser Price
This is probs the best uncensored model you can run on 2 x RTX Pro 6000 at high throughput (source: ✨ my opinion ✨).
Served on the voipmonitor vLLM fork it runs ~13× prefill and ~2.8× single-stream decode faster than the widely available abliterated GGUF versions.
Quick start (one command)
Requires Docker with the NVIDIA Container Toolkit, and ≥2 GPUs (see Hardware).
bash run.sh
run.sh defaults to this repo, so no arguments are needed. Everything is configured via env vars (see the table below) — e.g. GPUS=0,1,2,3 TP=4 bash run.sh to run on 4 GPUs.
run.sh (included) will:
- Download the weights (~156 GB) into your Hugging Face cache (
\~/.cache/huggingface, or$HF_HOME/$HF_HUB_CACHE) if not already present. - Pull the inference image if not already present.
- Serve the model on
http://localhost:8000/v1(OpenAI API spec).
Optional — MTP speculative decoding (~1.8× single-stream decode speed). The checkpoint ships its MTP layer; enable it with
MTP=2 bash run.sh(see the table below).
Override anything via env vars (all optional):
| Variable | Default | Meaning |
|---|---|---|
GPUS | 0,1 | GPU indices to use (comma-separated) |
TP | 2 | Tensor-parallel size — set to the number of GPUs in GPUS |
MTP | (off) | MTP speculative-decoding draft tokens (e.g. 2); empty disables it |
PORT | 8000 | API port |
MAX_MODEL_LEN | 262144 | Context length |
GPU_MEM_UTIL | 0.92 | Fraction of VRAM to use |
HF_REPO | fraserprice/DeepSeek-V4-Flash-Abliterated | Hugging Face repo to download |
MODEL_DIR | (HF cache) | Serve from a specific local dir instead of downloading |
SERVED_NAME | DeepSeek-V4-Flash-Abliterated | Model name exposed by the API |
IMAGE | (pinned voipmonitor/vllm build) | Inference container image |
Example on 4 GPUs:
GPUS=0,1,2,3 TP=4 bash run.sh
Hardware/Image
Tuned for NVIDIA RTX Pro 6000 Blackwell (96 GB, sm_120). It will run on other Blackwell-class GPUs, but the numbers below and the default flags assume RTX Pro 6000. It needs roughly 170 GB of total VRAM for weights + KV cache at long context, split across GPUs by tensor parallelism.
| Setup | Works? |
|---|---|
2 × RTX Pro 6000 Blackwell (96 GB) — TP=2 | ✅ reference config |
4 × RTX Pro 6000 Blackwell — TP=4 | ✅ (more KV headroom / throughput) |
4 × 80 GB (e.g. H100/H200) — TP=4 | ✅ (untested here, I believe can just use base vLLM) |
| 1 × DXG Spark | ❓ idk why you didn't get an RTX Pro lmao (but lmk if you manage to run this and I can add to docs) |
| 1 × anything | ❌ (blessed are the poor in VRAM 🙏) |
The serving image targets Blackwell (sm_120) and CUDA 13.2. The DeepSeek-V4 architecture (sparse attention / lightning indexer, MLA, MTP, etc.) requires this image to run on RTX Pro. See local-inference-lab/rtx6kpro for image, bench, and serving details.
Performance
Measured under load on RTX Pro 6000 Blackwell (connected via PCIe 5.0), 128 generated tokens/request, TP = number of GPUs. MTP is speculative decoding (— = off, 2 = 2 draft tokens). PP / TG are prefill / per-request decode throughput (tok/s); Total is aggregate end-to-end throughput across all concurrent requests (tok/s). Rows are paired off/on so the MTP decode gain is directly comparable.
| GPUs | MTP | Prompt | Conc | TTFT | PP | TG | Total |
|---|---|---|---|---|---|---|---|
| 2 | — | 1,000 | 1 | 132 ms | 7,613 | 108.6 | 870 |
| 2 | 2 | 1,000 | 1 | 143 ms | 7,005 | 176.2 | 1,310 |
| 2 | — | 1,000 | 3 | 275 ms | 4,190 | 76.7 | 1,746 |
| 2 | 2 | 1,000 | 3 | 332 ms | 3,746 | 125.7 | 2,435 |
| 2 | — | 10,000 | 1 | 1,036 ms | 9,658 | 108.0 | 4,581 |
| 2 | 2 | 10,000 | 1 | 1,069 ms | 9,357 | 181.6 | 5,729 |
| 2 | — | 10,000 | 3 | 2,138 ms | 5,302 | 57.1 | 6,600 |
| 2 | 2 | 10,000 | 3 | 2,205 ms | 5,146 | 85.3 | 7,496 |
| 2 | — | 100,000 | 1 | 11,575 ms | 8,640 | 107.6 | 7,850 |
| 2 | 2 | 100,000 | 1 | 11,933 ms | 8,380 | 196.2 | 7,959 |
| 2 | — | 100,000 | 3 | 23,579 ms | 5,153 | 33.0 | 8,161 |
| 2 | 2 | 100,000 | 3 | 24,349 ms | 4,991 | 59.6 | 8,065 |
| 4 | — | 1,000 | 1 | 119 ms | 8,437 | 117.9 | 946 |
| 4 | 2 | 1,000 | 1 | 128 ms | 7,858 | 216.5 | 1,584 |
| 4 | — | 1,000 | 3 | 255 ms | 4,539 | 98.0 | 2,175 |
| 4 | 2 | 1,000 | 3 | 507 ms | 3,276 | 129.7 | 1,996 |
| 4 | — | 10,000 | 1 | 842 ms | 11,886 | 118.4 | 5,293 |
| 4 | 2 | 10,000 | 1 | 873 ms | 11,459 | 223.9 | 7,034 |
| 4 | — | 10,000 | 3 | 1,741 ms | 6,521 | 69.0 | 8,041 |
| 4 | 2 | 10,000 | 3 | 1,813 ms | 6,257 | 108.7 | 9,216 |
| 4 | — | 100,000 | 1 | 9,530 ms | 10,494 | 118.5 | 9,445 |
| 4 | 2 | 100,000 | 1 | 9,856 ms | 10,146 | 227.4 | 9,614 |
| 4 | — | 100,000 | 3 | 19,226 ms | 6,299 | 40.0 | 10,028 |
| 4 | 2 | 100,000 | 3 | 20,037 ms | 6,049 | 73.1 | 9,824 |
MTP=2 lifts single-stream decode by ~1.6–1.9× (e.g. 108 → 196 tok/s at 100k on TP=2) for a negligible TTFT cost.
How it was made
DeepSeek-V4-Flash's published abliterations existed only as a GGUF for the bespoke ds4/llama.cpp engines, which leave Blackwell hardware massively underutilised.
Rather than re-run abliteration from scratch, the refusal subspace was simply recovered and re-applied from an existing abliteration:
- Recover the refusal directions. The rank-3 refusal subspace in the 4096-dim residual stream was extracted from the delta between huihui-ai's abliterated GGUF and the clean base.
- Re-apply to the official weights. The same rank-3 projection was applied to every residual-writing matrix of the official checkpoint — attention output (
attn.wo_b) and shared-expert down-projection (shared_experts.w2) across all 43 layers and the MTP layer (88 tensors total). Each tensor is dequantised → projected → requantised, with a fixed-point solver that compensates for FP8 requantisation so the intended projection survives quantisation. - Leave everything else untouched. The FP4-quantised routed experts and all gate/up projections are byte-identical to the official release; only the 88 FP8 residual-writing matrices change.
Validation: on every edited tensor the achieved projection matched target within ~0.002, and all 33,880 non-target tensors were verified byte-identical to the base. The served model drops refusals while keeping correct tool-calling and coherent step-by-step reasoning. No formal evals yet — tested manually; it should match the base huihui-ai abliteration at far higher speed.
Credits
- deepseek-ai/DeepSeek-V4-Flash — the base model (MIT).
- huihui-ai — the original abliteration (
Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF), from which the refusal directions were derived (MIT). voipmonitor/vllm— the Blackwell/CUDA-13.2 vLLM build that runs the DeepSeek-V4 architecture on RTX Pro.- local-inference-lab/rtx6kpro — the serving recipe (backends, flags, and benchmarks) this config is based on.
This checkpoint contains no huihui-ai weights — only the official DeepSeek-V4-Flash weights with a refusal-direction projection applied.
Disclaimer
This is an uncensored model: the refusal subspace has been removed, so it will not decline requests the way the base model does. It is released as-is, for research/lawful use only.
- Not a safety-aligned model. Because guardrails have been removed, outputs may be inaccurate, offensive, or harmful. Apply your own filtering, review, and safeguards before relying on or deploying it.
- No warranty. The weights and scripts are provided "as is", without warranty of any kind, express or implied. You use them entirely at your own risk.
- You are responsible for what you generate. The author does not endorse, encourage, or accept any responsibility or liability for how this model is used, including any unlawful, harmful, or unethical use. Outputs do not reflect the views of the author.
License
MIT, inheriting from the base model and the abliteration source.
Built by Fraser Price — @fraserpricee