What are the core concepts covered in LLM Compression: Quantization (GPTQ, AWQ) & QLoRA?

The memory requirements of LLMs: FP16 (2 bytes/param) vs INT4 (0.5 bytes/param); Activation-aware Weight Quantization (AWQ) vs One-Shot GPTQ second-order Hessian optimization; The mathematical decomposition of LoRA: freezing base weights W0 and training low-rank adapter matrices B (d x r) and A (r x k); Fine-tuning a 70-Billion parameter model on a single 48GB GPU using QLoRA and `bitsandbytes`

Advanced 28 min readModule: Module 13: LLM Quantization & Parameter-Efficient Fine-Tuning

LLM Compression: Quantization (GPTQ, AWQ) & QLoRA

Run and fine-tune 70B parameter models on consumer GPUs: Weight-Only Quantization (INT8, INT4, AWQ, GPTQ), Post-Training Quantization (PTQ), Low-Rank Adaptation (LoRA rank r matrices: W = W0 + B*A), and QLoRA with 4-bit NormalFloat (NF4) and Double Quantization.

What You Will Learn in This Lesson

The memory requirements of LLMs: FP16 (2 bytes/param) vs INT4 (0.5 bytes/param)
Activation-aware Weight Quantization (AWQ) vs One-Shot GPTQ second-order Hessian optimization
The mathematical decomposition of LoRA: freezing base weights W0 and training low-rank adapter matrices B (d x r) and A (r x k)
Fine-tuning a 70-Billion parameter model on a single 48GB GPU using QLoRA and `bitsandbytes`

Introduction & Core Concept

A 70-Billion parameter LLM in FP16 precision requires 140GB of GPU VRAM just to load weights, plus an additional 400GB for optimizer states during full fine-tuning. Parameter-Efficient Fine-Tuning (PEFT) via LoRA freezes the original weights and injects small trainable rank decomposition matrices (training only 0.1% of parameters). QLoRA quantizes the base model down to 4-bit NormalFloat (NF4), enabling fine-tuning of 70B models on a single workstation GPU with zero performance degradation.

WHY DOES THIS MATTER IN THE REAL WORLD?

QLoRA reduces enterprise LLM fine-tuning cloud costs by over 90%, allowing specialized domain models to be trained for medical, legal, and engineering tasks on modest hardware.

Syntax & Structure

python

// LoRA Forward Pass
h = W0 * x + (alpha / r) * (B * A * x)

Configuring 4-bit QLoRA Parameter-Efficient Fine-Tuning in Python

python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# 4-Bit QLoRA Fine-Tuning Pipeline with HuggingFace PEFT & BitsAndBytes
import torch
 
# Conceptual LoRA Matrix Layer Implementation
class LoRALinearLayer(torch.nn.Module):
    def __init__(self, in_features, out_features, rank=16, alpha=32):
        super().__init__()
        # 1. Base Pretrained Weight (Frozen in 4-bit precision!)
        self.base_weight = torch.nn.Parameter(
            torch.randn(out_features, in_features), requires_grad=False
        )
        # 2. Low-Rank Adapter Matrices: A (r x k) and B (d x r)
        self.lora_A = torch.nn.Parameter(torch.randn(rank, in_features) * 0.01) # Small Gaussian
        self.lora_B = torch.nn.Parameter(torch.zeros(out_features, rank))       # Initialized to ZERO!
        self.scaling = alpha / rank
 
    def forward(self, x):
        # Base forward pass (Frozen)
        base_out = torch.matmul(x, self.base_weight.t())
        # LoRA Delta forward pass: delta_W = (B * A) * x * scaling
        lora_out = torch.matmul(x, self.lora_A.t())
        lora_out = torch.matmul(lora_out, self.lora_B.t()) * self.scaling
        return base_out + lora_out
 
# Test LoRA Layer
in_dim, out_dim, r = 4096, 4096, 16
lora_layer = LoRALinearLayer(in_dim, out_dim, rank=r)
 
# Count Trainable Parameters
trainable = sum(p.numel() for p in lora_layer.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in lora_layer.parameters() if not p.requires_grad)
 
print("=== QLoRA Parameter-Efficient Fine-Tuning Engine ===")
print(f"Frozen Base Parameters:    {frozen:,} (Kept in 4-bit INT4 VRAM)")
print(f"Trainable LoRA Parameters: {trainable:,} (Only {trainable/frozen*100:.2f}% of model!)")
print("✅ Gradients computed strictly for tiny low-rank adapter matrices!")

Line-by-Line Technical Breakdown

1AWQ vs GPTQ Quantization: GPTQ calibrates weights by inverting second-order Hessian error matrices. AWQ (Activation-aware Weight Quantization) observes that preserving the top 1% of 'salient weights' (weights that multiply with high-magnitude activation channels) retains 99.9% of model reasoning accuracy in 4-bit precision.

Try It Yourself (Interactive Editor)

Modify the code in real-time and click Run to test live browser output and console logs.

Intelligent Code Runner & Live Sandbox[PYTHON]

PYTHON SOURCE EDITOR

Interactive Live Code

# 4-Bit QLoRA Fine-Tuning Pipeline with HuggingFace PEFT & BitsAndBytes
import torch

# Conceptual LoRA Matrix Layer Implementation
class LoRALinearLayer(torch.nn.Module):
    def __init__(self, in_features, out_features, rank=16, alpha=32):
        super().__init__()
        # 1. Base Pretrained Weight (Frozen in 4-bit precision!)
        self.base_weight = torch.nn.Parameter(
            torch.randn(out_features, in_features), requires_grad=False
        )
        # 2. Low-Rank Adapter Matrices: A (r x k) and B (d x r)
        self.lora_A = torch.nn.Parameter(torch.randn(rank, in_features) * 0.01) # Small Gaussian
        self.lora_B = torch.nn.Parameter(torch.zeros(out_features, rank))       # Initialized to ZERO!
        self.scaling = alpha / rank

def forward(self, x):
        # Base forward pass (Frozen)
        base_out = torch.matmul(x, self.base_weight.t())
        # LoRA Delta forward pass: delta_W = (B * A) * x * scaling
        lora_out = torch.matmul(x, self.lora_A.t())
        lora_out = torch.matmul(lora_out, self.lora_B.t()) * self.scaling
        return base_out + lora_out

# Test LoRA Layer
in_dim, out_dim, r = 4096, 4096, 16
lora_layer = LoRALinearLayer(in_dim, out_dim, rank=r)

# Count Trainable Parameters
trainable = sum(p.numel() for p in lora_layer.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in lora_layer.parameters() if not p.requires_grad)

print("=== QLoRA Parameter-Efficient Fine-Tuning Engine ===")
print(f"Frozen Base Parameters:    {frozen:,} (Kept in 4-bit INT4 VRAM)")
print(f"Trainable LoRA Parameters: {trainable:,} (Only {trainable/frozen*100:.2f}% of model!)")
print("✅ Gradients computed strictly for tiny low-rank adapter matrices!")

>Python 3.12.0 (KWAS Sandboxed Compiler)

>=== QLoRA Parameter-Efficient Fine-Tuning Engine ===

>Frozen Base Parameters: frozen (Kept in 4-bit INT4 VRAM

>Trainable LoRA Parameters: trainable (Only 42% of model!

>✅ Gradients computed strictly for tiny low-rank adapter matrices!

Common Mistakes & How to Avoid Them

#1: Performing full fine-tuning (updating all 70B weights) without gradient checkpointing, causing instant CUDA out-of-memory errors.

Full fine-tuning requires 16 bytes per parameter just for Adam optimizer states (momentum + variance). LoRA requires optimizer states only for tiny adapter matrices.

Incorrect / Antipattern

model.train() # Updating all weights requires 16 bytes per param for AdamW optimizer states!

Correct / Professional Solution

model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, target_modules=['q_proj', 'v_proj']))

Industry Best Practices & Professional Standards

Use AWQ (`AutoAWQ`) or GPTQ (`AutoGPTQ`) for 4-bit production inference deployment.
Apply QLoRA with `bitsandbytes` 4-bit NormalFloat (NF4) for parameter-efficient domain fine-tuning.
Merge LoRA adapter weights (`model.merge_and_unload()`) before deploying to production inference servers.

Lesson Summary & Core Takeaways

Quantization compresses weights from FP16 (16-bit) to INT4 (4-bit), slashing VRAM by 75%.
LoRA freezes base weights and trains rank decomposition matrices A and B.
QLoRA enables enterprise-scale model fine-tuning on accessible commodity GPUs.