license: mit base_model: orcarouter/GLM-5.3-Flash-Uncensored-NVFP4 pipeline_tag: text-generation library_name: vllm language: [en, ru] tags: [moe, nvfp4, compressed-tensors, expert-pruning, agentic, tool-calling, code, glm5, vllm, uncensored]

GLM-5.3-Flash Uncensored NVFP4 — agentic expert-pruned, vLLM-ready

This is the ready-to-serve version of Asilarkness/GLM-5.3-Flash-Uncensored-NVFP4-Pruning-62GB: 92 of 288 routed experts per layer (~107B total / ~18B active parameters), pruned for agents, tool calling and code, already converted so that vLLM loads it as is (68.5 GiB, fits one 96 GB GPU, ~6 GiB of KV cache left at 0.90 utilization). Method, expert selection (keep.json) and held-out evaluation are described in the main repository.

What differs from the main repository

Converted with convert_vllm.py (also in the main repo, inference/):

FP8 attention / shared-expert / dense-MLP weights → BF16 (vLLM expects block-wise FP8 for attention and its FP8 linear kernel needs an NVRTC JIT).
Per expert, gate_proj / up_proj NVFP4 global scales equalized to min(Gg, Gu) with the block scales of the other projection rescaled (vLLM fuses them into one tensor).
input_activations removed from the quantization config — the checkpoint is weight-only; declaring activation quantization makes vLLM take the W4A4 path and produce NaN logits.

Serve (vLLM 0.31, Python 3.12)

pip install uv && uv venv --python-preference only-managed -p 3.12 /tmp/vv
uv pip install --python /tmp/vv/bin/python vllm instanttensor
V=$(/tmp/vv/bin/python -c "import flashinfer;print(flashinfer.__version__)")
uv pip install --python /tmp/vv/bin/python "flashinfer-jit-cache==$V" --extra-index-url https://flashinfer.ai/whl/cu130
# DeepGEMM JIT needs CUDA headers matching nvcc:  uv pip install --python /tmp/vv/bin/python "nvidia-cuda-cccl==<nvcc version>.*"
#   and  export CPATH=<site-packages>/nvidia/cu13/include/cccl  NVCC_APPEND_FLAGS=-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK

/tmp/vv/bin/vllm serve Asilarkness/GLM-5.3-Flash-Uncensored-NVFP4-Pruned-vLLM --served-model-name glm --max-model-len 65536 --gpu-memory-utilization 0.90 \
  --max-num-seqs 32 --load-format instanttensor --no-enable-flashinfer-autotune --trust-remote-code \
  --limit-mm-per-prompt '{"image":0,"video":0}' --enable-auto-tool-choice --tool-call-parser glm47

Measured on one RTX PRO 6000 Blackwell: ~47 tokens/s single stream, ~175 tokens/s with 8 parallel requests; agent tasks and tool calls verified (see the main repo).

The base model is abliterated (refusal-removed): outputs are unfiltered, use at your own responsibility.

GLM-5.3-Flash Uncensored NVFP4 — agentic expert-pruned, vLLM-ready

What differs from the main repository

Converted with convert_vllm.py (also in the main repo, inference/):

FP8 attention / shared-expert / dense-MLP weights → BF16 (vLLM expects block-wise FP8 for attention and its FP8 linear kernel needs an NVRTC JIT).

Per expert, gate_proj / up_proj NVFP4 global scales equalized to min(Gg, Gu) with the block scales of the other projection rescaled (vLLM fuses them into one tensor).

input_activations removed from the quantization config — the checkpoint is weight-only; declaring activation quantization makes vLLM take the W4A4 path and produce NaN logits.

Serve (vLLM 0.31, Python 3.12)

pip install uv && uv venv --python-preference only-managed -p 3.12 /tmp/vv uv pip install --python /tmp/vv/bin/python vllm instanttensor V=$(/tmp/vv/bin/python -c "import flashinfer;print(flashinfer.__version__)") uv pip install --python /tmp/vv/bin/python "flashinfer-jit-cache==$V" --extra-index-url https://flashinfer.ai/whl/cu130 # DeepGEMM JIT needs CUDA headers matching nvcc: uv pip install --python /tmp/vv/bin/python "nvidia-cuda-cccl==<nvcc version>.*" # and export CPATH=<site-packages>/nvidia/cu13/include/cccl NVCC_APPEND_FLAGS=-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK /tmp/vv/bin/vllm serve Asilarkness/GLM-5.3-Flash-Uncensored-NVFP4-Pruned-vLLM --served-model-name glm --max-model-len 65536 --gpu-memory-utilization 0.90 \ --max-num-seqs 32 --load-format instanttensor --no-enable-flashinfer-autotune --trust-remote-code \ --limit-mm-per-prompt '{"image":0,"video":0}' --enable-auto-tool-choice --tool-call-parser glm47

Measured on one RTX PRO 6000 Blackwell: ~47 tokens/s single stream, ~175 tokens/s with 8 parallel requests; agent tasks and tool calls verified (see the main repo).

The base model is abliterated (refusal-removed): outputs are unfiltered, use at your own responsibility.