Deep Learning β€Ί Transformers β€Ί Day 231

Day 231: Fine-Tuning a Small Transformer

Day 231 of 365 β€” Fine-Tuning a Small Transformer

Master end-to-end transfer learning and fine-tuning on a lightweight transformer model (DistilBERT / MiniLM / RoBERTa-Small): understand catastrophic forgetting and parameter regularization, configure Layer-wise Learning Rate Decay (LLRD), implement learning rate warmup and cosine annealing, prevent small-dataset overfitting, and export to ONNX Runtime for sub-millisecond production inference.

Course
Deep Learning
Category
Transformers
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-29

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/deep-learning/day-231-fine-tuning-a-small-transformer

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/deep-learning/day-231-fine-tuning-a-small-transformer
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Over the past week, you built the entire theoretical and architectural foundation of Sequence Models and Transformers:

Now comes the ultimate capstone skill of Week 33: Fine-Tuning a Lightweight Transformer Model on Custom Domain Data and Deploying it for Sub-Millisecond Production Inference.

In real-world enterprise engineering, deploying a 70B parameter LLM for every single classification task is financially irresponsible (10,000s/month in GPU cloud bills) and introduces unacceptable user-facing latency (> 500$ ms per call).

A fine-tuned 66M parameter DistilBERT or 33M parameter MiniLM model achieves 99\% of GPT-4’s accuracy on domain-specific classification, runs in under 3 milliseconds on standard CPUs, costs 0.01\times to serve, and can be completely hosted on-premise for absolute data privacy.

Today, you will master the full fine-tuning engineering toolkit: avoiding catastrophic forgetting with Layer-wise Learning Rate Decay (LLRD), configuring AdamW and Warmup Schedules, and exporting models to ONNX Runtime.


The idea in plain language

Diagram: the five stages of fine-tuning a pretrained transformer

Think of hiring an experienced Oxford-educated translator to work at a specialized intellectual property patent law firm:


Historical background

Diagram: the historical progression from feature extraction to parameter-efficient fine-tuning

End-to-end transformer fine-tuning pipeline showing pretrained backbone layer-wise learning rates classifier head backpropagation and checkpointing

  1. 2018 (Howard & Ruder & ULMFiT): Introduced the foundational principles of transfer learning in NLP: Discriminative Fine-tuning (Layer-wise learning rates) and Slanted Triangular Learning Rates (Warmup + Decay).
  2. 2019 (Sanh et al. & DistilBERT): Proved that Knowledge Distillation during pretraining could shrink BERT by 40\% (66M params vs 110M) while retaining 97\% of language understanding and running 60\% faster.
  3. 2019 (Loshchilov & Hutter & AdamW): Identified a critical flaw in deep learning libraries where L2 regularization was incorrectly implemented in Adam, proving that true Decoupled Weight Decay (AdamW) dramatically improved generalization in Transformer fine-tuning.
  4. 2021 (Microsoft ONNX Runtime & Transformer Optimization): Released automated graph-level operator fusion and INT8 quantization for transformer backbones, slashing CPU latency below 2 ms.

What it is β€” and what it is not

Diagram: fine-tuning compared with training a model from random initialisation

What Transformer Fine-Tuning IS:

What it is NOT:


Why it was created and what problems it solves

Fine-tuning solved three monumental industrial barriers:

  1. The Small Data Problem: Deep neural networks typically require 1,000,000+ examples to train from scratch. With transfer learning, you can achieve state-of-the-art accuracy with as few as 200 labeled domain examples.
  2. Training Cost Reduction: Instead of spending 500,000 training a model from scratch on 1,000 GPUs, fine-tuning takes 5 minutes on a single GPU (costing < `0.10`).
  3. Serving Latency & Footprint: Compact distilled models fit into mobile app bundles and edge devices.

How it works

Let us dissect the mathematical and practical mechanics of production transformer fine-tuning.


1. Catastrophic Forgetting & Layer-wise Learning Rate Decay (LLRD)

Diagram: how a randomly initialised head damages pretrained layers and how decayed learning rates prevent it

Catastrophic forgetting versus layer-wise learning rate decay diagram comparing weight disruption

When fine-tuning, the newly initialized classification head W_{\text{clf}} produces large random gradients in the first few batches.

The LLRD Solution:

Assign exponentially decaying learning rates from top layers down to bottom layers:

\eta_l = \eta_{\text{top}} \cdot \xi^{(L - l)}

where:

Learning Rate Distribution Example (\xi = 0.85):


2. AdamW Decoupled Weight Decay (Loshchilov & Hutter, 2017)

In standard Adam with L2 penalty, the regularization gradient is added directly to the gradient vector g_t:

g_t = \nabla \mathcal{L}(\theta) + \lambda \theta_t

Because Adam rescales updates by the historical variance \sqrt{v_t}, weights with large historical gradients receive less weight decay, while weights with small historical gradients receive more weight decay.

AdamW Decouples Weight Decay:

\theta_{t+1} = \theta_t - \eta_t \lambda \theta_t - \eta_t \frac{m_t}{\sqrt{v_t} + \epsilon}

Weight decay is applied directly to the parameter vector independently of moving average moments, ensuring uniform, mathematically principled regularization.


3. Learning Rate Warmup & Cosine Annealing

Diagram: the three phases of a learning rate schedule for fine-tuning

To prevent early gradient instability:

  1. Linear Warmup: For the first 10\% of total training steps (t \le T_{\text{warmup}}):

    \eta(t) = \eta_{\text{max}} \cdot \frac{t}{T_{\text{warmup}}}
  2. Cosine Annealing Decay: For t > T_{\text{warmup}}:

    \eta(t) = \eta_{\text{min}} + \frac{1}{2}(\eta_{\text{max}} - \eta_{\text{min}})\left(1 + \cos\left( \frac{t - T_{\text{warmup}}}{T_{\text{total}} - T_{\text{warmup}}} \pi \right)\right)

This smooth trajectory lets the classification head calibrate its parameters before deep weights are gently adjusted.


4. Overfitting Countermeasures on Small Datasets

When fine-tuning on < 1,000 examples:

  1. Label Smoothing (\alpha = 0.1): Replaces hard 1-hot targets [0, 1] with soft distributions [0.05, 0.95]:

    y_{\text{smooth}} = (1 - \alpha) y_{\text{true}} + \frac{\alpha}{K}

    Prevents the model from becoming overconfident on noisy training labels.

  2. Early Stopping: Monitor validation loss after every epoch. If validation loss does not improve for 2 consecutive epochs (patience=2), halt training and reload the best checkpoint.

  3. Freezing Lower Backbone Layers: Freeze layers 1 \dots 4 and only train layers 5, 6 and the classification head.


5. Production Optimization with ONNX Runtime

After training, export the PyTorch model to ONNX:

import torch

torch.onnx.export(
    model,
    dummy_input,
    "model.onnx",
    input_names=["input_ids", "attention_mask"],
    output_names=["logits"],
    dynamic_axes={
        "input_ids": {0: "batch_size", 1: "seq_len"},
        "attention_mask": {0: "batch_size", 1: "seq_len"},
        "logits": {0: "batch_size"}
    },
    opset_version=17
)

6. Knowledge Distillation Loss Formulation (Hinton et al., 2015)

Diagram: how a small student model learns from a large teacher's full output distribution

When training a compact student model (DistilBERT) to inherit the reasoning capabilities of a massive teacher model (RoBERTa-Large):


7. INT8 Post-Training Dynamic Quantization in PyTorch

Diagram: how integer quantisation reduces model size and latency after training

To deploy the fine-tuned model to latency-sensitive edge hardware with zero GPU requirement:


8. Systematic Empirical Comparison: Tuning Paradigms

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚               TRANSFORMER ADAPTATION PARADIGMS ON SST-2                β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Tuning Strategy    β”‚ Accuracy     β”‚ VRAM Footprintβ”‚ Tuning Time (1 Epoch)β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Linear Probe (Head)β”‚ 82.4%        β”‚ 1.2 GB       β”‚ 22 seconds          β”‚
β”‚ Full Fine-Tuning   β”‚ 91.8%        β”‚ 3.8 GB       β”‚ 65 seconds          β”‚
β”‚ LLRD Fine-Tuning   β”‚ 93.4%        β”‚ 3.8 GB       β”‚ 65 seconds          β”‚
β”‚ LoRA (r=8, alpha=16β”‚ 92.9%        β”‚ 1.8 GB       β”‚ 34 seconds          β”‚
β”‚ QLoRA (4-bit NF4)  β”‚ 92.6%        β”‚ 1.1 GB       β”‚ 45 seconds          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

An everyday analogy

Think of a champion Formula 1 driver transitioning to rally car racing on dirt tracks:


Examples in practice

Let us build a Complete Layer-wise Learning Rate Decay Optimizer Builder in PyTorch:

import torch
import torch.nn as nn

def create_llrd_optimizer_grouped_parameters(
    model: nn.Module,
    base_lr: float = 5e-5,
    decay_rate: float = 0.85,
    weight_decay: float = 0.01
) -> list[dict]:
    # Creates parameter groups with Layer-wise Learning Rate Decay (LLRD)
    # and decoupled weight decay for no-decay parameters (biases and LayerNorm).
    no_decay = ["bias", "LayerNorm.weight", "layer_norm.weight"]
    optimizer_grouped_parameters = []

    # 1. Classification Head (Highest Learning Rate)
    head_params = [
        p for n, p in model.named_parameters()
        if "classifier" in n or "head" in n
    ]
    if head_params:
        optimizer_grouped_parameters.append({
            "params": [p for p in head_params if not any(nd in p for nd in no_decay)],
            "lr": base_lr * 2.0,
            "weight_decay": weight_decay
        })
        optimizer_grouped_parameters.append({
            "params": [p for p in head_params if any(nd in p for nd in no_decay)],
            "lr": base_lr * 2.0,
            "weight_decay": 0.0
        })

    # 2. Backbone Transformer Layers with Geometric LLRD
    num_layers = 6 # e.g. DistilBERT
    for layer_idx in range(num_layers - 1, -1, -1):
        layer_lr = base_lr * (decay_rate ** (num_layers - 1 - layer_idx))
        layer_params = [
            p for n, p in model.named_parameters()
            if f"layer.{layer_idx}." in n or f"transformer.layers.{layer_idx}." in n
        ]
        
        optimizer_grouped_parameters.append({
            "params": [p for p in layer_params if not any(nd in p for nd in no_decay)],
            "lr": layer_lr,
            "weight_decay": weight_decay
        })
        optimizer_grouped_parameters.append({
            "params": [p for p in layer_params if any(nd in p for nd in no_decay)],
            "lr": layer_lr,
            "weight_decay": 0.0
        })

    return optimizer_grouped_parameters

Implications: security, privacy, performance, scalability, and cost

Diagram: the cost, privacy, drift and licensing implications of fine-tuning

  1. Enterprise Cost & Latency Benchmark:
    • Calling GPT-4 API: $30.00 / 1M tokens, ~600 ms latency, requires sending corporate data over public internet.
    • Fine-Tuned DistilBERT: `0.15 / 1M tokens (on basic CPU instance), 2.5 ms latency, `100%$ air-gapped on-premise privacy compliance (HIPAA / GDPR).
  2. Quantization with INT8 Post-Training Quantization (PTQ):
    • Quantizing DistilBERT from FP32 (260 MB) to INT8 (65 MB) reduces memory footprint by 75\% while running natively on edge mobile devices (Apple Neural Engine / Android NPU).

Alternatives: free, open source, and commercial

ApproachModel SizeCPU LatencyMemoryCost / 1M Tokens
Fine-Tuned MiniLM33M params1.8 ms130 MB$0.05 (Self-Hosted)
Fine-Tuned DistilBERT66M params2.8 ms260 MB$0.10 (Self-Hosted)
Fine-Tuned RoBERTa125M params8.5 ms500 MB$0.25 (Self-Hosted)
Commercial LLM API>100B params450 msRemote API10.00 to 30.00

Diagram: full-parameter fine-tuning compared with adapter-based fine-tuning

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   FINE-TUNING PARADIGMS COMPARISON                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Strategy           β”‚ Trainable Paramsβ”‚ Tuning Cost      β”‚ Best Dataset β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Full Fine-Tuning   β”‚ 100%            β”‚ Low (Small Mod)  β”‚ 500 - 50k    β”‚
β”‚ LLRD Fine-Tuning   β”‚ 100%            β”‚ Low (Robust)     β”‚ 100 - 10k    β”‚
β”‚ LoRA / PEFT        β”‚ 0.1% - 1%       β”‚ Minimal          β”‚ Large Models β”‚
β”‚ Linear Probe Only  β”‚ Head Only (0.1%)β”‚ Instant          β”‚ < 100 samplesβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when fine-tuning is worthwhile compared with prompting a general model

When to USE a Fine-Tuned Small Transformer:

When NOT to use:


The AI connection

Knowledge check

  1. What is Catastrophic Forgetting and what mechanism causes it during naive fine-tuning?
  2. How does Layer-wise Learning Rate Decay (LLRD) assign learning rates across the depth of a Transformer?
  3. Why does AdamW decouple weight decay from moving average gradient moments?
  4. What is the benefit of exporting a trained model to ONNX Runtime format?
  5. When should you choose a fine-tuned DistilBERT model over a commercial LLM API?

Hands-on exercise

In this lab, you will build and test a complete modular fine-tuning management suite in PyTorch: implement the Layer-wise Learning Rate Decay (LLRD) parameter group generator, construct a lightweight classification model with custom head, configure decoupled AdamW optimization with cosine warmup schedule, and simulate an end-to-end fine-tuning step.

Expected output

[Small Transformer Fine-Tuning Verification Suite]
Testing LLRD Parameter Group Generator:
  Total Parameter Groups Created: 14 (Head + 6 Layers x 2 Decay Splits)
  Head Learning Rate: 0.0001
  Layer 6 Learning Rate: 0.00005
  Layer 1 Learning Rate: 0.0000222
  LayerNorm & Bias Groups Weight Decay: 0.0 [VERIFIED]
Testing Fine-Tuning Optimization Step:
  Initial Loss: 0.6931 -> Step 1 Loss: 0.6420 [GRADIENT DECENT APPLIED]
  Validation Early Stopping Monitor: Verified patience tracking
Test Suite: 3 passed in 0.42s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement an Early Stopping Callback class in pure Python with configurable patience and min_delta.
  2. Implement INT8 Dynamic Quantization on the fine-tuned model using torch.quantization.quantize_dynamic.

Extension challenge

Implement Knowledge Distillation Loss in PyTorch:

Quiz

Q1. What is Catastrophic Forgetting in the context of fine-tuning pretrained transformer models?

  1. When the computer runs out of disk space
  2. Updates overwrite what pretraining learned
  3. When the tokenizer deletes the vocabulary
  4. When the model switches from PyTorch to TensorFlow
Show answer

Answer: B. Updates overwrite what pretraining learned

Catastrophic forgetting occurs when downstream optimization wipes out pre-existing pretraining representations.

Q2. How does Layer-wise Learning Rate Decay (LLRD) protect early transformer layers from catastrophic forgetting?

  1. It doubles the learning rate after every epoch
  2. It sets the learning rate to zero for all layers
  3. Lower layers get smaller learning rates
  4. It trains only on even-numbered days
Show answer

Answer: C. Lower layers get smaller learning rates

LLRD assigns smaller learning rates to bottom layers, preserving generalized linguistic representations while specializing top layers.

Q3. Why is AdamW (Loshchilov & Hutter, 2017) preferred over standard Adam with L2 regularization when fine-tuning transformers?

  1. AdamW eliminates the need for backpropagation
  2. AdamW runs on CPUs only
  3. AdamW deletes negative gradients
  4. AdamW decouples weight decay from the gradient
Show answer

Answer: D. AdamW decouples weight decay from the gradient

Decoupled weight decay in AdamW ensures correct regularization behavior in adaptive gradient methods.

Q4. What is the primary benefit of exporting a fine-tuned transformer model to ONNX (Open Neural Network Exchange) Runtime format?

  1. Graph fusion and INT8 quantization
  2. ONNX makes the model train 10 times faster
  3. ONNX converts text into images
  4. ONNX requires no dependencies
Show answer

Answer: A. Graph fusion and INT8 quantization

ONNX Runtime performs graph level operator fusion and quantization, delivering low-latency production inference.

Q5. What is the recommended rule of thumb for fine-tuning learning rates on small transformer models (e.g. DistilBERT / BERT-Base)?

  1. Use 0.1 to train as fast as possible
  2. 2e-5 to 5e-5 with a short warmup
  3. Use 1e-9
  4. Keep learning rate constant at 1.0
Show answer

Answer: B. 2e-5 to 5e-5 with a short warmup

Pretrained transformers require gentle fine-tuning learning rates (2e-5 to 5e-5) with warmup to avoid destroying pretrained weights.

Glossary

Transfer Learning
Adapting a model pretrained on a massive general corpus to a specific downstream domain task using a small labeled dataset.
Catastrophic Forgetting
The destructive phenomenon where fine-tuning abruptly overwrites foundational pretraining feature representations.
Layer-wise Learning Rate Decay (LLRD)
A hyperparameter schedule applying exponentially decaying learning rates to lower layers of a transformer.
AdamW
An adaptive optimizer decoupling L2 weight decay from moving average gradient moment updates.
Learning Rate Warmup
Linearly scaling the learning rate from 0 to peak value over initial steps to protect early weights from violent updates.
ONNX Runtime
A high-performance inference engine for executing neural network computational graphs across heterogeneous hardware.
Early Stopping
A regularization technique halting training when validation loss fails to improve for a patience threshold.
Knowledge Distillation
A compression technique training a compact student model to mimic the output probability distributions of a large teacher.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.