license: other license_name: nvidia-open-model-agreement license_link: LICENSE base_model: Piecrust/GLM-5.3-Flash-FP4-Abliterated base_model_relation: quantized library_name: mlx pipeline_tag: image-text-to-text language:

• en
• ko
• zh tags:
• mlx
• apple-silicon
• glm
• glm-5.3
• moe
• nvfp4
• mxfp8
• qad
• mtp
• uncensored
• abliterated

GLM-5.3-Flash-FP4-Local-Piecrust-Abliterated-MLX

An MLX repack of Piecrust/GLM-5.3-Flash-FP4-Abliterated (rev 5c15d091), which applies a rank-1 refusal-direction edit to 44 matrices (self_attn.o_proj and mlp.shared_experts.down_proj, layers 15-36) of nvidia/GLM-5.3-Flash-FP4-Local (rev 76348ddb), NVIDIA's quantization-aware-distilled (QAD) NVFP4 build of zai-org/GLM-5.3-Flash (320B total, about 18B active). The routed experts keep NVIDIA's trained NVFP4 values byte for byte; only the container changes. It was built and measured on an Apple M5 Ultra (256GB) with oMLX.

The non-abliterated twin is junmo-k/GLM-5.3-Flash-FP4-Local-MLX; the two share every routed-expert file byte for byte and differ only in the 22 small non-expert files of layers 15-36.

On our held-out set it scores 1.47% lower NLL than our own 158GiB uncensored mixed-precision MLX build (JK v4), and the abliteration edit costs +0.46% NLL against the unedited twin, at the cost of 12-24% slower decode and about 13GB more peak memory (details below).

Loading needs a small oMLX patch (see Requirements): stock oMLX has no place for the per-expert weight_scale_2 in GLM's switch layers yet.

Not affiliated with NVIDIA, Z.AI, Piecrust or Huihui.

Layout

Each MoE layer's routed experts sit in their own *-experts.safetensors file. 174.33 GiB in total.

Requirements

• oMLX with per-expert weight_scale_2 support in its GLM SwitchGLU. The patch below (14 changed lines) is what we ran, on oMLX 6feea9f9 with MLX 0.32.3 and mlx-vlm 0.7.7. MLX's gather_qmm takes the per-expert amax as global_scale, so the scale is applied inside the kernel.
• Newer oMLX main pins mlx-vlm 0.7.8, whose loader handles weight_scale_2 itself and only for its own switch class, so this patch alone is not enough there. An upstream PR that covers that path is being prepared; this card will link it.
• Loading this checkpoint with stock mlx-vlm (outside oMLX) has not been tested.

diff --git a/omlx/patches/deepseek_v4/switch_layers.py b/omlx/patches/deepseek_v4/switch_layers.py
index 10be70146..b7aabd497 100644
--- a/omlx/patches/deepseek_v4/switch_layers.py
+++ b/omlx/patches/deepseek_v4/switch_layers.py
@@ -223,6 +223,10 @@ class QuantizedSwitchLinear(nn.Module):
         self.group_size = group_size
         self.bits = bits
         self.mode = mode
+        if mode == "nvfp4":
+            # ModelOpt two-level NVFP4: one f32 ``weight_scale_2`` per expert
+            # (the mlx-vlm >= 0.7.8 checkpoint key), loaded from the checkpoint.
+            self.weight_scale_2 = mx.ones((num_experts,), dtype=mx.float32)
 
         self.freeze()
 
@@ -330,6 +334,10 @@ class QuantizedSwitchLinear(nn.Module):
                     block_variant,
                 )
         else:
+            extra = {}
+            if "weight_scale_2" in self:
+                # gather_qmm takes the per-expert amax; weight_scale_2 = amax / (6 * 448).
+                extra["global_scale"] = self["weight_scale_2"] * (6.0 * 448.0)
             x = mx.gather_qmm(
                 x,
                 self["weight"],
@@ -341,6 +349,7 @@ class QuantizedSwitchLinear(nn.Module):
                 bits=self.bits,
                 mode=self.mode,
                 sorted_indices=sorted_indices,
+                **extra,
             )
         if "bias" in self:
             x = x + mx.expand_dims(self["bias"][indices], -2)
diff --git a/omlx/patches/m5_gather_qmm.py b/omlx/patches/m5_gather_qmm.py
index f4385268b..2bd412ba0 100644
--- a/omlx/patches/m5_gather_qmm.py
+++ b/omlx/patches/m5_gather_qmm.py
@@ -197,6 +197,9 @@ def _nax_sorted_gather_qmm(x, w, args, kwargs):
 
 
 def _gather_qmm_rerouted(x, w, *args, **kwargs):
+    if kwargs.get("global_scale") is not None:
+        # The NAX reroute has no global scale input; keep MLX's kernel.
+        return _original_gather_qmm(x, w, *args, **kwargs)
     if kwargs.get("sorted_indices") and _on_nax_host():
         out = _nax_sorted_gather_qmm(x, w, args, kwargs)
         if out is not None:
@@ -270,6 +273,8 @@ def fused_gate_up_activation(proj, x, indices, activation, token_rows=None):
         return None
     if "bias" in proj or not all(hasattr(proj, a) for a in ("group_size", "bits")):
         return None
+    if "weight_scale_2" in proj:
+        return None
     w = proj.get("weight")
     scales = proj.get("scales")
     if not isinstance(w, mx.array) or not isinstance(scales, mx.array):

Evaluation

Setup: M5 Ultra 256GB, oMLX 6feea9f9 plus the patch, MLX 0.32.3, mlx-vlm 0.7.7. The baseline is JK v4 (junmo-k/GLM-5.3-Flash-UNCENSORED-JK-oQ3.5e), our 158GiB oQ3.5e GPTQ build. JK v4 is built from a different uncensored derivative (dealignai), so both sides carry a weight edit, made by different methods.

Held-out NLL

240 teacher-forced 512-token windows in two disjoint sets of 120 (10 categories x 12: English, code, reasoning, chat, tool calling, Chinese, Korean, and three categories from private Korean/English working notes). Windows are paired across models; intervals are 95% window bootstrap.

Against JK v4, 9 of 10 categories improve (chat +0.3%); 177 of 240 windows improve. Against the unedited twin, only tool calling improves (−0.5%).

Speed and memory

Measured on this checkpoint. ABBA x2 through llama-swap, cold load each round, MTP on (adaptive depth), 512-token outputs; 4 baseline rounds and 3 valid candidate rounds (one candidate round was discarded because an unrelated request preempted it).

Most of the decode gap is expected from bytes read per token (experts at 4.5 bpw versus about 4.1 bpw in JK v4, attention at MXFP8 versus 6-bit), and from oMLX's fused GLM decode kernels being affine-only, so NVFP4 and MXFP8 run on MLX's generic paths. This split is an estimate, not a measurement.

Checks

• All routed-expert tensors sampled (36 matrices across layers 3, 23, 44 and MTP) match an independent decode of the source bytes exactly; MXFP8 tensors and the kv_b_proj split match exactly.
• The patched SwitchGLU matches a float32 dequantized reference within bf16 tolerance on decode, sorted prefill and fused gate/up paths.
• Functional smoke tests (Korean, arithmetic, code, tool calling) pass.

License

NVIDIA Open Model Agreement (see LICENSE and NOTICE). Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement. GLM-5.3-Flash by Z.ai is MIT-licensed; the Huihui abliteration that Piecrust re-applied is MIT-licensed.

Acknowledgements

NVIDIA for the QAD checkpoint, Piecrust and Huihui for the abliteration, Z.ai for GLM-5.3-Flash, the FlashInfer / Local Inference Lab csf_32x2_v1 reference codec, and mlx-serve, which first ran this checkpoint on Apple Silicon.