Aya Expanse 8B Abliterated — AWQ (W4A16)

AWQ 4-bit quantization of lenML/aya-expanse-8b-abliterated, itself an abliteration of Aya Expanse 8B.

Why this exists

The base abliterated model was only published in BF16 and GGUF. This AWQ quant serves natively in vLLM with the Marlin kernel for fast inference.

Quantization details

Tool: llm-compressor (vLLM project)
Format: compressed-tensors (auto-detected by vLLM)
Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
Algorithm: AWQ (Activation-aware Weight Quantization)
Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)
lm_head left in fp16 (preserves output quality)
Note: v_proj → o_proj smoothing is skipped on all 32 layers due to GQA shape mismatch (32 query heads / 8 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

Usage with vLLM

vllm serve KebalBaguette/aya-expanse-8b-abliterated-AWQ \
  --quantization compressed-tensors \
  --tool-call-parser hermes \
  --enable-auto-tool-choice \
  --max-model-len 8192

License

CC-BY-NC-4.0 — non-commercial use only. Inherited from Cohere Aya Expanse 8B and lenML's abliterated derivative. This abliterated derivative is for research, personal, and non-commercial evaluation use only. For commercial deployment, contact Cohere for licensing, or use an Apache-licensed alternative (e.g. Mistral 7B / Nemo).

Quantization details

Tool: llm-compressor (vLLM project)

Format: compressed-tensors (auto-detected by vLLM)

Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)

Algorithm: AWQ (Activation-aware Weight Quantization)

Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)

lm_head left in fp16 (preserves output quality)

Note: v_proj → o_proj smoothing is skipped on all 32 layers due to GQA shape mismatch (32 query heads / 8 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

License