Module 4 of 5 · 45 min
Evaluation, Merging & GGUF/AWQ Quantization
Evaluate fine-tuned model performance against benchmark suites, merge adapters back into base weights, and export to GGUF and AWQ formats.
Core concept
By the end
You will be able to
- Run automated evaluation harnesses (lm-evaluation-harness, MMLU, GSM8K).
- Merge LoRA adapters into full float16 weights safely.
- Quantize merged weights to GGUF (for Ollama/llama.cpp) and AWQ (for vLLM).
01
Merge and Quantize for Production Serving
Running inference with separate LoRA adapters adds overhead during batching. For production deployment, merge adapter matrices directly into base weights and quantize to AWQ or GGUF for low-latency serving.
Merge LoRA Adapter to Base Model
python
from peft import PeftModel
from transformers import AutoModelForCausalLM
base_model = AutoModelForCausalLM.from_pretrained("base-model-id", torch_dtype="auto")
peft_model = PeftModel.from_pretrained(base_model, "lora-adapter-id")
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("./merged-production-model")Practice activity
Merge LoRA Adapter and Export to GGUF Format
- Merge a trained LoRA adapter with base weights.
- Convert merged model to GGUF 4-bit (Q4_K_M) using llama.cpp.
- Run local benchmark inference and verify output correctness.
What to produce
- Conversion log and llama.cpp benchmark throughput metrics.
Reflect before continuing
What precision losses occur when converting FP16 weights to Q4_K_M quantization?
Evidence
Sources and verification
- AWQ: Activation-aware Weight Quantization for LLM Compression and AccelerationArXiv · verified 2026-08-22
Knowledge check
Make it stick.
Choose the strongest answer for each question. Your attempts become part of your account transcript.