LLMs and Generative AI βΊ Customizing and Running Models βΊ Day 275
Day 275: Fine-Tuning with LoRA
Master the mathematics and systems engineering of Low-Rank Adaptation (LoRA) and QLoRA: implement low-rank matrix decomposition, understand intrinsic rank dimensionality, configure rank and alpha scaling factors, target transformer attention projection modules, and execute zero-latency adapter merging.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-275-fine-tuning-with-lora
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/llms-and-generative-ai/day-275-fine-tuning-with-lora - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Derive the mathematical formulation of LoRA low-rank decomposition: W = W0 + (alpha/r) * B * A.
- Analyze intrinsic rank dimensionality and evaluate rank selection tradeoffs (r=4 to r=64).
- Target transformer attention (Wq, Wk, Wv, Wo) and MLP (Wgate, Wup, Wdown) projection modules.
- Implement a fully functional LoRA linear layer and adapter injection pipeline from scratch in Python.
- Execute adapter weight merging to eliminate inference latency overhead.
Prerequisites
Why this matters
Fine-tuning a 70-billion parameter language model via standard full parameter training requires updating and tracking gradients, momentum vectors, and optimizer states across every single weight. Under standard 16-bit AdamW optimization, every billion parameters demands approximately 16 to 20 gigabytes of high-bandwidth VRAM. Training a 70B model with full parameter updates requires an industrial cluster of 8 to 16 NVIDIA H100 80GB GPUs costing hundreds of thousands of dollars.
For the vast majority of software engineering organizations, full parameter fine-tuning is economically and operationally prohibitive. Furthermore, full fine-tuning modifies the entire base weight matrix, making it impossible to serve dozens of specialized customer models concurrently without deploying separate, massive GPU clusters for every single tenant.
Low-Rank Adaptation (LoRA) fundamentally revolutionized deep learning economics. Introduced by Edward Hu and researchers at Microsoft in 2021, LoRA proved that adapting a model to a specialized domain does not require modifying all 70 billion parameters. By freezing the foundation model and training two tiny low-rank adapter matrices that represent less than 0.1% to 1.0% of the original parameter count, LoRA slashes GPU memory requirements by over 70%, enables training on a single workstation GPU, and allows thousands of lightweight tenant adapters to be swapped dynamically on a single shared foundation model.
Understanding the mathematical mechanics of low-rank factorization, rank dimension selection, scaling coefficients, and adapter merging is essential for any AI engineer building production-grade customized models.
The idea in plain language
Imagine you buy a comprehensive 2,000-page encyclopedia on world history. If you want to customize this encyclopedia for a specialized university course on the maritime history of 18th-century Venice, full fine-tuning is equivalent to rewriting all 2,000 pages by hand from scratch. You risk introducing errors into previously accurate chapters on ancient Rome, and you must print an entirely new 2,000-page book.
LoRA takes a completely different approach:
- You leave the original 2,000-page encyclopedia completely untouched and locked behind glass (Frozen Base Weights $W_0$).
- You take a thin 5-page transparent overlay booklet (Low-Rank Adapter $Delta W$) and write only your specialized Venetian maritime annotations in that booklet.
- When a student reads the book, they lay the transparent 5-page booklet over the relevant pages and read the combined text (Forward Pass $h = W_0 x + Delta W x$).
- If a different class needs a course on medieval metallurgy, you simply swap the 5-page overlay booklet for a metallurgy booklet in a fraction of a second, without touching the base encyclopedia.
- When you are ready to publish a permanent single edition, you can permanently print the transparent overlay annotations directly onto the base pages with zero future reading slowdown (Weight Merging).
Mathematically, LoRA achieves this extreme compression by factoring a massive weight change matrix $Delta W$ into the product of two ultra-narrow matrices ($B \times A$).
Historical background
Prior to 2021, parameter-efficient transfer learning relied on Adapter Layers (Houlsby et al., 2019) and Prefix Tuning (Li & Liang, 2021):
- Houlsby Adapters inserted small feed-forward neural layers sequentially between existing transformer self-attention and MLP blocks. While effective at reducing trainable parameter counts, sequential adapter layers introduced severe latency overhead during autoregressive generation because each token had to pass through extra sequential layer boundaries.
- Prefix Tuning prepended trainable continuous vector embeddings to the keys and values across all transformer attention layers. However, prefix tuning consumed precious context window space and proved difficult to optimize across diverse task distributions.
In 2021, Hu et al. published βLoRA: Low-Rank Adaptation of Large Language Modelsβ. Inspired by Aghajanyan et al.βs empirical discovery that over-parameterized neural networks possess a remarkably low intrinsic dimensionality, Hu et al. hypothesized that the weight updates $Delta W$ during task adaptation also reside on a low-dimensional manifold.
By decomposing $Delta W$ into parallel low-rank matrices $B * A$, LoRA solved the inference latency problem: because matrix multiplication is linear and associative, $B * A$ can be computed in parallel with base weights during training and mathematically merged into base weights prior to production serving, introducing zero added inference latency.
In 2023, Tim Dettmers and the University of Washington extended this breakthrough with QLoRA (Quantized LoRA), introducing 4-bit NormalFloat (NF4) base weight quantization and Double Quantization, enabling fine-tuning of 65B parameter models on a single 48GB GPU without performance degradation.
What it is β and what it is not
To avoid architectural mistakes, let us define what LoRA is and what it is not:
What LoRA Is:
- A parameter-efficient fine-tuning (PEFT) method that freezes base model weights $W_0$ and adds trainable low-rank decomposition matrices $A$ and $B$ to linear projections.
- A technique that reduces trainable parameters by 99% and optimizer memory by 75%.
- A modular architecture enabling multiple task-specific adapters (e.g.
medical-lora,legal-lora,sql-lora) to be served concurrently from a single loaded base model checkpoint. - Fully reversible and mergeable into base weights with zero inference latency overhead.
What LoRA Is Not:
- A replacement for pretraining (LoRA adapts existing capabilities; it cannot teach a model foundational language syntax from scratch).
- A real-time factual search database (LoRA adapts behavior, tone, and grammar; it should not be used to memorize real-time factual records).
- Restricted solely to attention layers (LoRA can be applied to attention Q, K, V, O projections as well as MLP gate, up, and down projections).
+-------------------------------------------------------------------------------+
| FULL FINE-TUNING VS LORA (7B MODEL) |
+------------------------------+-----------------------+------------------------+
| Metric | Full Fine-Tuning | LoRA Fine-Tuning (r=8) |
+------------------------------+-----------------------+------------------------+
| Base Model Parameters | 7,000,000,000 | 7,000,000,000 |
| Trainable Parameters | 7,000,000,000 (100%) | 19,922,944 (~0.28%) |
| VRAM Required (16-bit AdamW) | ~56 GB - 80 GB | ~16 GB - 24 GB |
| Checkpoint Storage Size | 14.0 GB per task | 40 MB per task |
| Multi-Tenant Adapter Swaps | Impossible | Sub-millisecond |
| Inference Latency Overhead | 0 ms | 0 ms (when merged) |
+------------------------------+-----------------------+------------------------+
Why it was created and what problems it solves
LoRA addresses three massive engineering bottlenecks in enterprise AI deployment:
1. The VRAM & Optimizer Memory Wall
When training with 16-bit mixed precision and the AdamW optimizer:
- Weights: 2 bytes per parameter
- Gradients: 2 bytes per parameter
- AdamW Momentum: 4 bytes per parameter
- AdamW Variance: 4 bytes per parameter
- Total state per parameter: 16 bytes minimum.
For a 70B model, optimizer states alone consume over 1.1 Terabytes of VRAM. By freezing 99.7% of parameters, LoRA restricts gradient and optimizer allocations exclusively to the tiny low-rank matrices $A$ and $B$, collapsing VRAM requirements into single-GPU territory.
2. The Multi-Task Storage & Checkpoint Explosion
If an enterprise builds 50 specialized models for distinct internal workflows (e.g. invoice extraction, code review, customer support routing, HR policy Q&A), full fine-tuning produces fifty 14GB checkpoints (700GB total storage) and requires 50 dedicated GPU clusters to serve them. With LoRA, the enterprise stores one shared 14GB base model and fifty 40MB adapter checkpoints, dynamically loading adapters into GPU memory on demand.
3. The Inference Latency Penalty of Prior Adapters
Sequential adapter architectures inserted extra layers between transformer blocks, adding 15β30% latency to every autoregressive generation step. LoRAβs parallel additive structure (W_0 x + Delta W x) allows offline matrix folding (W_(\text(merged)) = W_0 + Delta W), guaranteeing zero inference slowdown.
How it works
Let us derive the exact linear algebra and optimization dynamics of LoRA.
Input Vector x (d_in)
β
ββββ> [Frozen Linear Layer W_0 (d_out x d_in)] ββββββββββββββββββββββ> (W_0 * x)
β β
β [+] ββ> Output h (d_out)
β β
ββββ> [Trainable Down-Projection A (r x d_in)] ββ> [r] ββ> [Trainable Up-Projection B (d_out x r)] ββ> * (alpha / r) ββ> Delta W * x
1. Low-Rank Factorization Mathematics
Given a pretrained linear projection layer with weight matrix W_0 in R^(d_(\text(out)) \times d_(\text(in))), the forward computation for an input vector x in R^(d_(\text(in))) is:
h = W_0 x
LoRA modifies this forward pass by constraining the weight update $Delta W$ to a low-rank decomposition:
h = W_0 x + Delta W x = W_0 x + (alpha / r) * (B * A) x
Where:
A in R^(r \times d_(\text(in)))is the down-projection matrix.B in R^(d_(\text(out)) \times r)is the up-projection matrix.r much less than min(d_(\text(in)), d_(\text(out)))is the rank bottleneck dimension (typicallyr in \(4, 8, 16, 32\)).\alpha in R^+is a constant scaling hyperparameter.
2. Initialization Dynamics
At the beginning of training (step 0):
- Matrix $A$ is initialized from a zero-mean Gaussian distribution: $A \sim N(0, sigma^2)$.
- Matrix $B$ is initialized entirely to zero: $B = 0$.
Because $B = 0$, the product $Delta W = B * A = 0$ at initialization. Therefore:
h = W_0 x + (alpha / r) * (0 * A) x = W_0 x
This guarantees that the model begins training with its exact pretrained representations intact, preventing random gradient shocks at step 0.
3. The Scaling Factor $\alpha$ and Hyperparameter Selection
The scaling factor \frac(\alpha)(r) stabilizes training dynamics when tuning rank r:
- When rank $r$ is doubled from 8 to 16, the inner dimension doubles, which can scale the magnitude of activations. Setting $\alpha = 2r$ or keeping $\alpha = 16$ fixed ensures that gradient updates scale predictably without requiring retuning of the learning rate.
- Common rule of thumb: set $\alpha = 2r$ (e.g., $r=8, \alpha=16$ or $r=16, \alpha=32$).
4. Target Module Selection in Modern Transformers
In modern decoder-only architectures (Llama 3, Mistral, Qwen), linear projections exist in both self-attention and feed-forward MLP blocks:
- Self-Attention Projections:
- $W_q$ (Query projection)
- $W_k$ (Key projection)
- $W_v$ (Value projection)
- $W_o$ (Output projection)
- MLP / SwiGLU Projections:
W_(\text(gate))(Gate projection)W_(\text(up))(Up projection)W_(\text(down))(Down projection)
Targeting all linear projections (Attention + MLP) yields the highest empirical benchmark accuracy and catastrophic forgetting resistance.
5. Offline Zero-Overhead Weight Merging
Once training completes, the adapter matrices can be merged permanently into the base weights prior to export or quantization:
W_merged = W_0 + (alpha / r) * (B * A)
The merged tensor W_(\text(merged)) has the exact same shape (d_(\text(out)) \times d_(\text(in))) as W_0, eliminating the runtime branching and dual matrix multiplications entirely.
An everyday analogy
Think of a high-resolution 4K digital photograph ($3840 \times 2160$ pixels = 8.3 million pixels):
- Full Fine-Tuning is like replacing all 8.3 million pixel values individually. Storing each modification requires a massive raw uncompressed TIFF file.
- LoRA is like applying a low-rank Singular Value Decomposition (SVD) filter to the image. By capturing only the top 8 or 16 principal color gradients, you can store the entire visual adjustment in a tiny 50-kilobyte lookup table. When the image is displayed on screen, the display processor combines the base photograph with the lightweight color adjustment matrix seamlessly.
Examples in practice
Let us examine real-world configurations used to fine-tune open-weight models:
1. Fine-Tuning Llama 3 8B on Structured JSON Extraction
from peft import LoraConfig, get_peft_model
lora_config = LoraConfig(
r=16, # Rank bottleneck
lora_alpha=32, # Scaling factor (alpha = 2r)
target_modules=[ # Target all attention and MLP projections
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"
],
lora_dropout=0.05, # Regularization dropout on adapter inputs
bias="none", # Do not train layer biases
task_type="CAUSAL_LM"
)
2. Multi-Adapter Deployment on a Single Inference Pod
A financial SaaS platform hosts a single Llama 3 70B base model on an 8-GPU Kubernetes pod. They maintain three separate LoRA adapters:
adapter_sec_filings(32MB)adapter_customer_sentiment(32MB)adapter_sql_generator(32MB)
When an incoming user request arrives at the vLLM serving gateway, the router inspects the request header X-Adapter-ID: adapter_sql_generator. vLLM activates the corresponding LoRA weights in GPU cache for that specific request turn, serving all three distinct applications concurrently from a single foundation model cluster.
Implications: security, privacy, performance, scalability, and cost
1. Security & Adapter Provenance
LoRA adapter checkpoints are compact binary tensors (adapter_model.safetensors). When loading community adapters from public repositories (such as Hugging Face Hub), always verify that checkpoints use the safe safetensors format rather than legacy Python pickle files, which are vulnerable to arbitrary remote code execution during deserialization.
2. Privacy & Data Isolation
LoRA provides clean organizational multi-tenancy. An enterprise can train tenant-specific adapters on isolated customer datasets. Because base weights are frozen and adapter parameters are physically segregated into separate files, tenant data cannot bleed into the foundation model or cross-contaminate other customer adapters.
3. Training & Inference Performance
- Training Throughput: LoRA reduces backward pass gradient computation to $
<1%$ of model weights, increasing training throughput by 2.5x and cutting epoch training times drastically. - Inference Latency: In multi-tenant environments, dynamically switching unmerged LoRA adapters introduces a negligible 1β3ms GPU memory pointer swap. When merged offline ($W_0 + Delta W$), inference latency is 100% identical to the base model.
4. Financial Cost Architecture
Cost Category Full Fine-Tuning (70B) QLoRA / LoRA (70B)
---------------------------------------------------------------------------------
Hardware Required 8x NVIDIA H100 80GB GPUs 2x NVIDIA A100 80GB GPUs
Compute Cost per Run $1,200 - $3,500 $85 - $250
Storage per Checkpoint 140 GB 160 MB
Monthly Serving (5 Adapters) 5x Dedicated GPU Pods ($25k) 1x Shared GPU Pod ($5k)
Alternatives: free, open source, and commercial
| Framework / Tool | Approach | License / Tier | Best For | Tradeoffs |
|---|---|---|---|---|
| Hugging Face PEFT | LoRA, QLoRA, Prefix Tuning | Open Source (Apache 2.0) | Standard industry toolkit for PyTorch/Transformers | Standard PyTorch training speeds |
| Unsloth | Hand-written Triton GPU Kernels for LoRA | Open Source (Apache 2.0) / Commercial | 2x-5x faster LoRA training with 70% less VRAM on single GPUs | Limited to supported open-weight architectures |
| Torchtune (PyTorch) | Native modular PyTorch fine-tuning | Open Source (BSD-3) | Clean, hackable, zero-dependency PyTorch code | Less automated high-level abstraction |
| Axolotl | YAML-driven end-to-end trainer | Open Source (Apache 2.0) | Declarative configuration of LoRA/DPO pipelines | Steeper learning curve for custom layer hacks |
| OpenAI Fine-Tuning API | Hosted Managed LoRA | Commercial (Pay-per-token + training) | Serverless fine-tuning of GPT-4o-mini | Closed weights; cannot export model or run on-premise |
Comparison with related concepts
+-----------------------+-----------------------+-----------------------+-----------------------+
| Characteristic | Full Fine-Tuning | Standard LoRA (PEFT) | QLoRA |
+-----------------------+-----------------------+-----------------------+-----------------------+
| Base Weight Precision | FP16 / BF16 (16-bit) | FP16 / BF16 (16-bit) | NF4 (4-bit NormalFloat)|
| Adapter Precision | N/A (All Weights) | FP16 / BF16 | BF16 (16-bit) |
| Trainable Parameters | 100% | 0.1% - 1.0% | 0.1% - 1.0% |
| VRAM (7B Model) | ~56 GB | ~16 GB | ~6 GB |
| Hardware Requirement | Multi-GPU Cluster | Single Prosumer GPU | Consumer GPU (RTX 4090)|
| Final Model Accuracy | 100% Baseline | 99% - 100% of Full | 99% of Full |
+-----------------------+-----------------------+-----------------------+-----------------------+
When to use it β and when not to
Choose LoRA Fine-Tuning when:
- You need to teach a model a specialized syntax, schema, tone, or task procedure.
- You have between 500 and 20,000 verified demonstration examples.
- You want to train on a single workstation or cloud GPU without spending thousands of dollars.
- You need to deploy multiple specialized models for different tasks from a single base deployment.
Do NOT use LoRA Fine-Tuning when:
- You are trying to update dynamic facts (prices, real-time inventory, daily news) β use RAG instead.
- You have fewer than 100 examples β use few-shot prompting with prompt caching instead.
- You need to fundamentally alter the underlying tokenizer vocabulary or pretrain a language model on a completely new human language from scratch.
The AI connection
- Low-rank adaptation made fine-tuning accessible β a large model adapted on one device rather than a cluster.
- The insight is that adaptation is low-rank: the change a task needs is far smaller than the model it changes.
- One backbone with many adapters is a different serving architecture, not just a cheaper training run.
- Adapters can be merged for zero inference overhead, which removes the usual objection to adapter-based methods.
- An adapter is executable-adjacent β it changes model behaviour, so its provenance matters as much as a dependencyβs.
Knowledge check
- Why does
B=0initialization prevent gradient shocks at step zero? BecauseDelta W = \frac(\alpha)(r) B * A. WhenB=0,Delta Wis identically zero, ensuring the forward pass outputs exact base model activations prior to training updates. - What happens mathematically during weight merging? The adapter product
Delta W = \frac(\alpha)(r) B * Ais computed and added directly to the base weight tensorW_0. The resulting tensor has the exact same dimensions asW_0, requiring zero adapter code at runtime. - What is the difference between LoRA and QLoRA? LoRA stores base weights in 16-bit precision. QLoRA quantizes base weights into a specialized 4-bit NormalFloat (NF4) format with double quantization, while keeping the trainable LoRA adapter matrices in 16-bit Brain Floating Point (BF16), slashing base memory by 75%.
Hands-on exercise
In this lab, you will implement a Custom LoRA Linear Layer and Adapter Injection Module from scratch in pure Python and NumPy. You will build the low-rank factorization forward pass, implement parameter tracking, compute trainable parameter reduction ratios, and implement zero-overhead offline weight merging.
Expected output
============================= test session starts ==============================
collected 6 items
tests/test_lora_layer.py ...... [100%]
============================== 6 passed in 0.04s ===============================
[LORA INITIALIZATION] Base Dim: 4096 x 4096 | Rank: 8 | Alpha: 16
-> Base Parameters: 16,777,216 (FROZEN)
-> Trainable LoRA Parameters: 65,536 (0.39% of base)
-> Delta W at step 0: EXACT ZERO (Verified)
[WEIGHT MERGE] Base W0 + (alpha/r)*B*A
-> Merged Weight Shape: (4096, 4096)
-> Output Difference between LoRA Layer and Merged Layer: 0.000000e+00 (BIT-EXACT)
Validate your work
Run the pytest harness to verify low-rank forward pass arithmetic, initialization constraints, and bit-exact weight merging:
pytest tests/run_tests.sh
Troubleshooting
- Matrix multiplication dimension mismatch: Ensure down-projection
Ahas shape(r, d_(\text(in)))and up-projectionBhas shape(d_(\text(out)), r). - Non-zero initialization output: Verify that matrix $B$ is initialized with
np.zeros((d_out, r)).
Common mistakes
- Transposing matrices incorrectly during forward pass: computing $x A B$ instead of $(x A^T) B^T$.
- Forgetting the scaling factor
\frac(\alpha)(r)during weight merging.
Practice assignment
Extend the LoRALinear class to support LoRA Dropout during the forward pass and add support for tracking gradient updates during simulated backpropagation.
Extension challenge
Implement a multi-adapter dynamic switching container (MultiLoRAContainer) that holds three distinct low-rank adapter pairs ($A_i, B_i$) for a shared base linear layer $W_0$ and dynamically selects the active adapter based on a runtime adapter_id string without reallocating base tensors.
Quiz
Q1. In the LoRA formulation W = W_0 + (alpha / r) * B * A, how are matrices A and B initialized at the start of training, and why?
- Both matrices are initialized to random Gaussian values to maximize gradient diversity
- Matrix B is initialized with base weights W_0 and Matrix A is set to an identity matrix
- Both matrices are initialized to ones so scaling factor alpha has an immediate effect
- A is random, B is zero, so Delta W starts at zero
Show answer
Answer: D. A is random, B is zero, so Delta W starts at zero
Initializing B to zeros and A to Gaussian noise guarantees that Delta W = (alpha/r)*B*A equals exactly zero at the beginning of training, preserving the base model's pretrained representations at step 0.
Q2. What does the parameter 'r' represent in LoRA, and what is its typical empirical value for task adaptation?
- The low-rank bottleneck dimension; typically between 4 and 32
- The learning rate multiplier; typically 0.001
- The number of transformer layers skipped during training; typically 12
- The quantization bit-width; typically 4-bit NormalFloat
Show answer
Answer: A. The low-rank bottleneck dimension; typically between 4 and 32
The rank 'r' specifies the inner bottleneck dimension of matrices A (r x k) and B (d x r). Empirical studies show ranks between 4 and 16 capture most task-specific intrinsic dimensionality.
Q3. What is the purpose of the scaling factor (alpha / r) in LoRA?
- It normalizes GPU memory allocations to prevent Out-Of-Memory errors
- It scales the update, so rank can change without retuning
- It converts 32-bit floating point tensors into 4-bit integers
- It acts as a dropout mask during forward inference
Show answer
Answer: B. It scales the update, so rank can change without retuning
Scaling by alpha/r keeps the magnitude of the adapter gradient updates consistent when experimenting with different rank values, avoiding the need to retune hyperparameters.
Q4. How does LoRA achieve zero additional inference latency during deployment?
- By executing inference on specialized quantum hardware
- By skipping attention heads that have low activation values
- Merge the adapter into the base weights before serving
- By compiling prompts into C++ binaries
Show answer
Answer: C. Merge the adapter into the base weights before serving
Because matrix multiplication is associative, the trained adapter delta (alpha/r)*B*A can be added directly to the base weights W_0 offline, yielding a single standard weight matrix W_merged with zero inference latency overhead.
Q5. Which set of transformer projection weights yields the most parameter-efficient adaptation when targeted with LoRA?
- Embedding layer and final LayerNorm only
- The softmax classification head only
- Positional encodings only
- The query and value projections
Show answer
Answer: D. The query and value projections
Hu et al. demonstrated that targeting Query and Value projections yields the highest task adaptation efficiency, while targeting all linear projections (Q, K, V, O, MLP) maximizes performance on complex multi-task benchmarks.
Glossary
- LoRA (Low-Rank Adaptation)
- PEFT technique that freezes base weights and injects trainable rank-decomposition matrices into linear layers.
- Intrinsic Dimensionality
- The minimum number of independent coordinates required to approximate a model's task-specific parameter manifold.
- Rank (r)
- The inner bottleneck dimension of the low-rank adapter matrices B (d x r) and A (r x k).
- Alpha (alpha)
- A constant scaling factor in LoRA that balances the magnitude of adapter updates against base weights.
- Weight Merging
- The offline addition of adapter delta weights into base model weights (W_merged = W0 + Delta W) to eliminate inference latency.
- QLoRA
- Quantized LoRA that freezes base weights in 4-bit NormalFloat (NF4) while computing gradients in higher-precision 16-bit adapters.
- Target Modules
- The specific linear projection matrices in a transformer (e.g. q_proj, v_proj, mlp) where LoRA adapters are injected.
- PEFT
- Parameter-Efficient Fine-Tuning; an umbrella term for methods that train a tiny fraction (<1%) of model parameters.
Sources and further reading
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021) β arXiv / NeurIPS (accessed 2026-08-30)
- QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023) β NeurIPS (accessed 2026-08-30)
- Hugging Face PEFT Documentation: LoRA Implementation β Hugging Face (accessed 2026-08-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.