license: apache-2.0 base_model: openbmb/MiniCPM-V-4.7-35B-A3B tags: [abliterated, uncensored, minicpm-v, moe, fp8, compressed-tensors]
MiniCPM-V-4.7-35B-A3B-abliterated-FP8
FP8 (FP8_DYNAMIC, compressed-tensors) version of swdq/MiniCPM-V-4.7-35B-A3B-abliterated (refusal-ablated openbmb/MiniCPM-V-4.7-35B-A3B, made with abliterix). 38 GB instead of 70 GB.
- Same layout as the official MiniCPM-V-4.7-35B-A3B-FP8: identical tensor key set (62,552 keys). Weights are float8_e4m3fn with per-output-channel bf16 scales; routed experts are stored per expert, and full-attention q/k/v/o are quantized. Vision tower, merger, linear attention, router, shared expert, embeddings and
lm_headstay bf16. Activations are quantized dynamically at runtime. - Quantized by a custom script (round-to-nearest, no calibration), not llmcompressor. Relative weight error vs. the bf16 source is about 2.6%. No benchmark of quality loss was run; only refusal removal, tool calling and image input were spot-checked.
- Base abliteration result (bf16, keyword detector, 100 harmful eval prompts): refusals 100 -> 5, KL divergence 0.101. The FP8 model was not re-scored with that harness.
- Only the language model was edited; the vision tower is unchanged.
Serving with vLLM + DSpark
dspark/ contains the unmodified official draft model openbmb/MiniCPM-V-4.7-35B-A3B-DSpark (Apache-2.0). Tested on one DGX Spark (GB10, 121 GB unified memory) with the vLLM nightly image (0.32.0rc1). MiniCPM-V 4.7 is not in vLLM v0.31.0 and SGLang support is not merged yet.
hf download swdq/MiniCPM-V-4.7-35B-A3B-abliterated-FP8 --local-dir ./model
vllm serve ./model --served-model-name minicpm-abliterated \
--max-model-len 131072 --max-num-seqs 8 --gpu-memory-utilization 0.55 --trust-remote-code \
--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--speculative-config '{"method":"dspark","model":"./model/dspark","num_speculative_tokens":15,"enable_adaptive_verification":false}'
Claude Code works through vLLM's Anthropic API: ANTHROPIC_BASE_URL=http://localhost:8000 with all model names set to minicpm-abliterated.
Measured speed (GB10, greedy, DSpark on, gpu-memory-utilization 0.55)
| decode tok/s | |
|---|---|
| prose essay | 33.2 |
| code | 119.7 |
| math | 158.3 |
| chat | 37.4 |
| 8 concurrent code streams (aggregate) | 534 |
| prefill, 15k tokens | ~4,457 tok/s |
| decode at 100k-token context | 9.4 |
Speculative decoding helps on predictable text (code, math) and barely on prose. At ~100k context, decode was no faster than without DSpark. bf16 + DSpark on the same machine measured 86 (code) / 102 (math) tok/s; bf16 without DSpark 25.5 tok/s on prose.
Warning: safety alignment is intentionally weakened. The model may produce harmful, offensive or illegal content. You are responsible for its use and for complying with applicable laws and the base model license.