Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-iMatrix-MTP-GGUF

license: apache-2.0 tags:

qwen3_5
qwen3.6
gguf
mtp
speculative-decoding
fine-tune
unsloth
heretic
uncensored
abliterated
multi-stage tuned
40B
dense
vision
multimodal
mmproj
long-context base_model: DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NEO-CODE-Di-IMatrix-MAX-GGUF datasets:
- TeichAI/claude-4.5-opus-high-reasoning-250x
- DavidAU/PkDick-Deckard-5-Datasets
- allenai/WildChat-1M pipeline_tag: text-generation

This repository represents an effort to graft and re-train an MTP layer for DavidAU's Qwen3.6-40B model, inspired by Piehsoft's untrained MTP grafts.

Currently only offering q8_0 quantizations, but this will eventually expand once MTP retraining is finalized.

Version 1 (complete)

A simple retraining attempt that used the following procedure:

Ran the 40B model on 117 random prompts
Record what it was thinking at every position by introspecting llama.cpp
- Specifically, recorded which 64 tokens it thought were most likely next and how confident it was in each.
- Note: Substituted h_t with embedding(input_token) to move forward with the data that was immediately accessible.
Use the recorded data to retrain the MTP layer: align what the MTP's predictions with reality
- This data did not include hidden states, but since the trained head is shown real-world verifier logprobs during training its bias toward the most common tokens is effectively re-calibrated.

Compared to Piehsoft's untrained MTP head, this methodology

Significantly improved acceptances rates in structured text / coding scenarios
Provided little to no impact on high-entropy scenarios such as creative writing / casual chat.

Ultimately proved that improving acceptance rates via retraining was possible. Further improvements seem to require:

Larger corpus (dataset)
Higher text entropy within corpus
Most importantly, data collection of the verifier hidden states

Version 2 (in-progress)

EDIT: Training using hidden states ended up generating significantly worse results than the methodology used in v1. Alternative paths are being considered, but this may potentially end as a stalemate.

~~Training is in-progress, which builds on-top of the v1. This version likely won't be a final version, but rather will be used to help steer direction. The main changes will be:~~

~~Collect data using a modified version of llama.cpp that exposes data from the 40B model's hidden states (h_t)~~
~~Use a dataset with high entropy: WildChat-1M~~
- ~~Not training on the full dataset (would take a long time), will use a subset of randomly selected samples.~~
~~Increase epochs from 3 to 5 (and tweak a few other training settings)~~

Version 1 (complete)

A simple retraining attempt that used the following procedure:

Ran the 40B model on 117 random prompts

Record what it was thinking at every position by introspecting llama.cpp

Specifically, recorded which 64 tokens it thought were most likely next and how confident it was in each.
Note: Substituted h_t with embedding(input_token) to move forward with the data that was immediately accessible.

Use the recorded data to retrain the MTP layer: align what the MTP's predictions with reality

This data did not include hidden states, but since the trained head is shown real-world verifier logprobs during training its bias toward the most common tokens is effectively re-calibrated.

Compared to Piehsoft's untrained MTP head, this methodology

Significantly improved acceptances rates in structured text / coding scenarios

Provided little to no impact on high-entropy scenarios such as creative writing / casual chat.

Ultimately proved that improving acceptance rates via retraining was possible. Further improvements seem to require:

Larger corpus (dataset)

Higher text entropy within corpus

Most importantly, data collection of the verifier hidden states

Version 2 (in-progress)

~~Training is in-progress, which builds on-top of the v1. This version likely won't be a final version, but rather will be used to help steer direction. The main changes will be:~~

~~Collect data using a modified version of llama.cpp that exposes data from the 40B model's hidden states (h_t)~~

~~Use a dataset with high entropy: WildChat-1M~~

~~Not training on the full dataset (would take a long time), will use a subset of randomly selected samples.~~

~~Increase epochs from 3 to 5 (and tweak a few other training settings)~~