base_model: mistralai/Mistral-Medium-3.5-128B base_model_relation: quantized library_name: llama.cpp pipeline_tag: text-generation tags:
- gguf
- q4_k_m
- llama.cpp
- mistral
- mistral3
- mistral-medium
- reasoning
- eschaton-engine
- eschaton-uncensored
- uncensored license: other license_name: modified-mit license_link: https://huggingface.co/mistralai/Mistral-Medium-3.5-128B/blob/main/LICENSE datasets:
- cloudbjorn/eschaton-uncensored
Mistral-Medium-3.5-128B-Eschaton-Uncensored-GGUF
This repository contains a Q4_K_M GGUF quantization of cloudbjorn/Mistral-Medium-3.5-128B-Eschaton-Uncensored for inference with llama.cpp and compatible applications.
The source model is a merged BF16 fine-tune of Mistral Medium 3.5 128B. Fine-tuning loaded the full-BF16 axolotl-ai-co/Mistral-Medium-3.5-128B-BF16 checkpoint and used cloudbjorn/eschaton-uncensored with the Eschaton Engine.
Model Lineage
- Official model: mistralai/Mistral-Medium-3.5-128B
- BF16 training checkpoint: axolotl-ai-co/Mistral-Medium-3.5-128B-BF16
- Merged Eschaton fine-tune: cloudbjorn/Mistral-Medium-3.5-128B-Eschaton-Uncensored
- This repository: Q4_K_M GGUF quantization of the merged Eschaton fine-tune
Download
The quantized model file is:
Mistral-Medium-3.5-128B-Eschaton-Uncensored-Q4_K_M.gguf
Download it with the Hugging Face CLI:
hf download \
cloudbjorn/Mistral-Medium-3.5-128B-Eschaton-Uncensored-GGUF \
Mistral-Medium-3.5-128B-Eschaton-Uncensored-Q4_K_M.gguf \
--local-dir .
Run with llama.cpp
llama-cli \
-m Mistral-Medium-3.5-128B-Eschaton-Uncensored-Q4_K_M.gguf \
-cnv \
-ngl 999 \
-c 8192
Adjust -ngl to control GPU offload and -c to choose the runtime context allocation supported by your available memory. This is a 128B model, so substantial combined RAM and VRAM are still required even at Q4_K_M.
Quantization Details
| Parameter | Value |
|---|---|
| Immediate source | cloudbjorn/Mistral-Medium-3.5-128B-Eschaton-Uncensored |
| Source precision | BF16 |
| Format | GGUF |
| Quantization | Q4_K_M |
| Conversion tool | llama.cpp |
| Intended runtime | llama.cpp and GGUF-compatible applications |
Q4_K_M uses mixed 4-bit quantization to reduce storage and inference memory substantially compared with the merged BF16 release. It is lossy and may differ slightly in output quality from the BF16 model.
Fine-Tune Provenance
The GGUF was created after the LoRA adapter was merged into the BF16 base model. The training run used 4-bit NF4 QLoRA with BF16 compute; the published GGUF is a separate post-training inference quantization and was not used as a training checkpoint.
| Parameter | Value |
|---|---|
| Training base | axolotl-ai-co/Mistral-Medium-3.5-128B-BF16 |
| Dataset | cloudbjorn/eschaton-uncensored |
| Framework | Eschaton Engine using Transformers, TRL, PEFT and bitsandbytes |
| Training method | 4-bit NF4 QLoRA with BF16 compute |
| Epochs | 1 |
| Training sequence length | 2,048 tokens |
| LoRA rank / alpha | 32 / 64 |
| LoRA targets | all-linear language-model modules; vision and multimodal-projector modules excluded |
| Micro-batch / gradient accumulation | 1 / 32 |
| Effective batch size | 32 |
| Learning rate | 5e-6 |
| Optimizer | 8-bit paged AdamW |
| Scheduler | Linear with 50 warmup steps |
| Weight decay | 0.01 |
| Seed | 3407 |
Model Characteristics
- Architecture: Dense 128B Mistral 3 model
- Base context window: Up to 262,144 tokens; practical runtime context depends heavily on available memory
- Reasoning modes: The underlying model supports Mistral's configurable reasoning behavior
- Fine-tuning scope: Text-language layers only
- Vision: The training dataset was text-only, and the vision tower and multimodal projector were not adapted
The fine-tune focuses on direct, neutral, and useful responses to sensitive, gritty, controversial, emotionally intimate, and technically demanding prompts without repetitive moralizing or canned disclaimers.
Evaluation Status
No standardized benchmark results are reported for this fine-tune or its Q4_K_M quantization. Users should evaluate it against their own instruction-following, reasoning, coding, safety, and domain-specific requirements before deployment.
License
This derivative follows the base model's Modified MIT License. Review that license and the upstream model card before use or redistribution.