LLM Fine-Tuning: LoRA, QLoRA, DPO, and Mixture-of-Experts
A base LLM is a general instrument; fine-tuning changes how tightly it resonates with your task. This guide maps the adaptation spectrum from prompting to MoE, with the math behind each trade-off.
Start with the right question
Do you need to fine-tune at all? The adaptation spectrum.
The most expensive mistake in LLM engineering is fine-tuning when prompting would suffice. Work left-to-right across the spectrum: only move to the next technique when the current one demonstrably fails. Each step increases capability — and cost, risk of forgetting, and maintenance burden.
- Prompting / few-shot ICL
Zero cost: Zero or few labeled examples in the context window. Zero compute overhead. Limited by context length and the base model reasoning ceiling. Start here for every new task. - RAG (Retrieval-Augmented)
No weight change: Inject retrieved documents , not just new facts. - PEFT (Parameter-Efficient FT)
< 1 % params: Freeze the base model. Train only a tiny adapter (< 1 % of params). LoRA, adapters, prefix tuning, IA3. The current industry default for LLM specialization. - Full fine-tuning
100 % params: Update all weights on domain-specific data. Maximum capacity, maximum cost. Justified for deep domain shift (medical imaging reports, legal contract parsing, code generation).
The PEFT workhorse
LoRA and QLoRA — low-rank adaptation is a change-of-basis trick
The LoRA insight: weight updates during fine-tuning are intrinsically low-rank. Rather than storing the full delta matrix, factorize it into two small matrices. At inference, merge them back into the frozen weights — zero added latency. QLoRA extends this by quantizing the frozen base to 4-bit NF4 format, making it possible to fine-tune a 65B model on a single 48 GB GPU.
$$ W^{\prime} = W_0 + \Delta W = W_0 + B,A, \quad B\in\mathbb{R}^{d\times r},;A\in\mathbb{R}^{r\times k} $$
W_0 is frozen. Only B and A (rank r) are trained. With r=8, a 4096x4096 weight matrix reduces from 16 M to 65 K trainable parameters — a 250x compression.
$$ \text{trainable params} = r,(d + k) \ll d\cdot k $$
At rank r=16, a Llama-2 70B model requires only ~0.1 % of original parameters to fine-tune. Visualize ) against a large frozen grey matrix.
- Which layers to adapt
Architecture: Apply LoRA to Q, K, V projections in attention — these carry most task-specific signal. Add MLP layers for domain shift. Skip embeddings and LayerNorm unless you are adding new vocabulary. - Rank selection heuristic
Hyperparameter: r=4 for style/tone adaptation, r=8 for task specialization, r=16-64 for strong domain shift. Higher rank = more expressiveness, higher overfitting risk on small datasets. - QLoRA: quantize, then adapt
Memory-efficient: Freeze the base model in 4-bit NF4 format with double quantization. Train LoRA adapters in BF16. The quantization error is absorbed by the adapter, achieving near-full-FT quality at 1/4 the memory.
| Metric | Value | Notes |
|---|---|---|
| Memory vs full FT | 3-4x less | QLoRA on a 7B model fits in 6 GB GPU RAM |
| Params trained | ~0.1 % | LoRA r=16 on a 7B model vs full fine-tuning |
Beyond task tuning
Alignment: RLHF, DPO, and ORPO
Task fine-tuning teaches what to do. Alignment training teaches how to behave — making models helpful, harmless, and honest. RLHF runs a full RL loop with a reward model. DPO eliminates the reward model by reframing preference learning )}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right] $$
DPO optimizes the policy to prefer winning response y_w over losing response y_l, relative to a frozen reference policy. No reward model needed — just labeled preference pairs.
- RLHF (three stages)
Classic: Step 1: SFT on demonstrations. Step 2: Train a reward model on human preference pairs. Step 3: PPO to maximize reward while staying close to SFT via KL penalty. Powerful but brittle and slow to train. - DPO (one loss)
Preferred: Reformulates RLHF , winning, losing) triples. Same final quality , no RL training loop, no reward model to maintain. - ORPO (one stage)
Latest (2024): Merges SFT and alignment into a single training stage by adding a preference penalty to the standard language modeling cross-entropy loss. Fewer hyperparameters, faster convergence.
Sparse scaling
Mixture-of-Experts: 8x the capacity, same inference cost
Standard Transformer FFN blocks activate every parameter for every token. MoE replaces the single FFN with N parallel expert networks and a learned router that routes each token to the top-K experts. At 8 experts with K=2, only 2/8 of expert compute activates per token — but the model h, matching Llama-2 70B on most benchmarks.
$$ y = \sum_{i=1}^{N} G(x)_i \cdot E_i(x), \quad G(x) = \text{TopK}!\left(\text{softmax}(Wx),,K\right) $$
Router G assigns gate weights to experts for each token. Only top-K experts receive nonzero weights. Animate tokens routing to different expert columns in a grid visualization.
$$ \mathcal{L}{\text{aux}} = \alpha \cdot N \cdot \sum{i=1}^{N} f_i \cdot P_i $$
Auxiliary load-balancing loss prevents expert collapse (all tokens routed to one expert). f_i = fraction of tokens routed to expert i, P_i = mean router probability for expert i.
- Expert activation heatmap
Visualization: Visualize ) and dead experts (dark columns) are both failure modes visible at a glance. - Fine-tuning MoE models
Adaptation: Apply LoRA per expert independently, or use shared LoRA adapters with per-expert residuals. Monitor expert utilization histograms per epoch — routing behavior shifts during fine-tuning. - Capacity factor tuning
Engineering: Each expert h) to balance throughput and quality on your hardware.
| Metric | Value | Notes |
|---|---|---|
| Active params per token | 2 / 8 experts | Mixtral 8x7B: 12B of 45B total active per forward pass |
| Effective capacity gain | ~4x | Over a dense model at the same inference FLOP budget |
Related posts: