NeuralDaredevil-8B-abliterated-NF4

Permanently quantized to NF4

This is a 4-bit NF4 quantized version of:
→ mlabonne/NeuralDaredevil-8B-abliterated
(fine-tuned & abliteration by mlabonne)

License: Meta Llama 3 Community License
Same as base model – commercial use allowed.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ikarius/NeuralDaredevil-8B-abliterated-NF4"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,   # Optimal for Llama 3
    trust_remote_code=True
)

input_text = "Hva er abliteration?"
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Quantization Details

'''

Original model: mlabonne/NeuralDaredevil-8B-abliterated
Architecture: Llama 3 8B
Quantization: bitsandbytes 4-bit NF4 (permanent)
Double quantization: Enabled
Compute dtype: bfloat16 ← recommended for Llama 3
Estimated VRAM: ~6.3–6.7 GB (CUDA) '''

Credits

Base model & abliteration: @mlabonne NF4 quantization: @ikarius

License Same as base model: llama3

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_id = "ikarius/NeuralDaredevil-8B-abliterated-NF4" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", torch_dtype=torch.bfloat16, # Optimal for Llama 3 trust_remote_code=True ) input_text = "Hva er abliteration?" inputs = tokenizer(input_text, return_tensors="pt").to(model.device) outputs = model.generate( **inputs, max_new_tokens=150, do_sample=True, temperature=0.7, top_p=0.9 ) print(tokenizer.decode(outputs[0], skip_special_tokens=True))