Sandeep Bazar 365 Days of AI
Labs

Deep Learning › Training Deep Networks › Day 205

Hands-on lab — Day 205: Datasets and DataLoaders

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/datasets_and_dataloaders_lib.py

Test

./tests/run_tests.sh

File tree

examples/datasets_and_dataloaders_lib.py
examples/test_datasets_and_dataloaders_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/datasets_and_dataloaders_lib.py
starter/test_datasets_and_dataloaders_lib.py
tests/run_tests.sh
tests/test_datasets_and_dataloaders_lib.py
troubleshooting.md

Lab README

Lab: Day 205 -- Datasets and DataLoaders

Lesson

Day number: 205 of 365. Course: Course05-SS01 (Deep Learning - Neural Networks). Topic: Datasets and DataLoaders in PyTorch.

Purpose

Build and test high-throughput PyTorch data loading pipelines. Subclass torch.utils.data.Dataset, implement custom collate_fn functions for dynamic sequence padding, configure DataLoader parameters, and evaluate batching throughput.

Learning objectives

  • Subclass torch.utils.data.Dataset implementing __len__ and __getitem__.
  • Implement dynamic sequence padding and mask construction in a custom collate_fn.
  • Configure DataLoader options including batch size, shuffling, and dropped remnants.
  • Verify pipeline correctness with automated unit test assertions.

Prerequisites

  • Day 204 (PyTorch: autograd and nn.Module).
  • Python 3.11+ with PyTorch.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

PyTorch is free and open-source under the modified BSD license.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/datasets_and_dataloaders_lib.py: Student scaffold file.
  • examples/datasets_and_dataloaders_lib.py: Complete reference implementation.
  • tests/test_datasets_and_dataloaders_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/datasets_and_dataloaders_lib.py

What the commands do

  • Instantiates a ragged-sequence dataset.
  • Executes dynamic batch collation and padding.
  • Iterates across mini-batches verifying batch shapes.

Expected output

DataLoader Demo: Processed 96 samples across 6 batches. Batch 0 Shape: (16, 20)

Validation steps

  1. Verify that RaggedSequenceDataset returns valid tuples of (Tensor, int).
  2. Confirm that dynamic_padding_collate pads each batch to its local maximum length.
  3. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Shape Mismatch Error: Ensure collate_fn stacks inputs along dim=0.

Security notes

All data loading runs locally in memory without remote network transmissions.

Extension exercises

  1. Implement an IterableDataset with multi-worker partition sharding.
  2. Add random token masking inside __getitem__ for self-supervised pre-training.
  • Lesson title: Datasets and DataLoaders
  • Day number: 205 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-205-datasets-and-dataloaders
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-205-datasets-and-dataloaders when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 205

- `Processed samples`: Total count of samples delivered across mini-batches.
- `Total batches`: Number of batches yielded by DataLoader.
- `Batch 0 Shape`: Dimension tuple of the first collated batch.

examples-run.txt

DataLoader Demo: Processed 96 samples across 6 batches. Batch 0 Shape: (16, 20)

measured-values.txt

Processed samples: 96
Total batches: 6
Batch 0 Shape: (16, 20)

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_datasets_and_dataloaders_lib.py::test_ragged_dataset_len_and_getitem PASSED [ 25%]
tests/test_datasets_and_dataloaders_lib.py::test_dynamic_padding_collate_function PASSED [ 50%]
tests/test_datasets_and_dataloaders_lib.py::test_dataloader_iteration_and_batching PASSED [ 75%]
tests/test_datasets_and_dataloaders_lib.py::test_dataloader_drop_last PASSED [100%]

============================== 4 passed in 0.22s ===============================

Source files

examples/datasets_and_dataloaders_lib.py (2245 bytes) 
import torch
from torch.utils.data import Dataset, DataLoader
from typing import List, Tuple, Dict, Any

class RaggedSequenceDataset(Dataset):
    def __init__(self, num_samples: int = 200, min_len: int = 5, max_len: int = 25):
        super().__init__()
        self.num_samples = num_samples
        torch.manual_seed(42)
        self.samples = [
            (torch.randint(1, 100, (torch.randint(min_len, max_len + 1, (1,)).item(),)).float(),
             torch.randint(0, 2, (1,)).item())
            for _ in range(num_samples)
        ]

    def __len__(self) -> int:
        return self.num_samples

    def __getitem__(self, idx: int) -> Tuple[torch.Tensor, int]:
        return self.samples[idx]

def dynamic_padding_collate(batch: List[Tuple[torch.Tensor, int]]) -> Dict[str, torch.Tensor]:
    sequences = [item[0] for item in batch]
    labels = torch.tensor([item[1] for item in batch], dtype=torch.long)
    lengths = torch.tensor([len(s) for s in sequences], dtype=torch.long)

    max_len = int(lengths.max().item())
    batch_size = len(batch)

    padded_inputs = torch.zeros(batch_size, max_len, dtype=torch.float32)
    attention_mask = torch.zeros(batch_size, max_len, dtype=torch.float32)

    for i, seq in enumerate(sequences):
        seq_len = len(seq)
        padded_inputs[i, :seq_len] = seq
        attention_mask[i, :seq_len] = 1.0

    return {
        "inputs": padded_inputs,
        "mask": attention_mask,
        "lengths": lengths,
        "labels": labels
    }

def run_dataloader_demo():
    dataset = RaggedSequenceDataset(num_samples=100, min_len=5, max_len=20)
    loader = DataLoader(
        dataset=dataset,
        batch_size=16,
        shuffle=True,
        collate_fn=dynamic_padding_collate,
        drop_last=True
    )

    batch_count = 0
    sample_count = 0
    first_shape = None

    for batch in loader:
        if batch_count == 0:
            first_shape = tuple(batch["inputs"].shape)
        batch_count += 1
        sample_count += len(batch["labels"])

    print(f"DataLoader Demo: Processed {sample_count} samples across {batch_count} batches. Batch 0 Shape: {first_shape}")
    return batch_count, sample_count, first_shape

if __name__ == "__main__":
    run_dataloader_demo()
examples/test_datasets_and_dataloaders_lib.py (1630 bytes) 
import pytest
import torch
from examples.datasets_and_dataloaders_lib import RaggedSequenceDataset, dynamic_padding_collate, DataLoader

def test_ragged_dataset_len_and_getitem():
    ds = RaggedSequenceDataset(num_samples=50, min_len=4, max_len=15)
    assert len(ds) == 50
    seq, label = ds[0]
    assert isinstance(seq, torch.Tensor)
    assert isinstance(label, int)
    assert 4 <= len(seq) <= 15
    assert label in (0, 1)

def test_dynamic_padding_collate_function():
    batch = [
        (torch.tensor([1.0, 2.0, 3.0]), 0),
        (torch.tensor([4.0, 5.0]), 1),
        (torch.tensor([6.0, 7.0, 8.0, 9.0]), 0),
    ]
    collated = dynamic_padding_collate(batch)
    assert collated["inputs"].shape == (3, 4)
    assert collated["mask"].shape == (3, 4)
    assert collated["labels"].shape == (3,)
    # Verify padding positions are zero
    assert collated["inputs"][1, 2] == 0.0
    assert collated["mask"][1, 2] == 0.0
    assert collated["mask"][2, 3] == 1.0

def test_dataloader_iteration_and_batching():
    ds = RaggedSequenceDataset(num_samples=64, min_len=5, max_len=10)
    loader = DataLoader(ds, batch_size=16, shuffle=False, collate_fn=dynamic_padding_collate)
    batches = list(loader)
    assert len(batches) == 4
    for b in batches:
        assert b["inputs"].shape[0] == 16
        assert b["mask"].shape[0] == 16
        assert b["labels"].shape[0] == 16

def test_dataloader_drop_last():
    ds = RaggedSequenceDataset(num_samples=35, min_len=5, max_len=10)
    loader_drop = DataLoader(ds, batch_size=16, drop_last=True, collate_fn=dynamic_padding_collate)
    assert len(list(loader_drop)) == 2
metadata.yml (437 bytes) 
lesson_id: D205
day: 205
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/datasets_and_dataloaders_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (27 bytes) 
torch>=2.2.0
pytest>=8.0.0
starter/datasets_and_dataloaders_lib.py (727 bytes) 
import torch
from torch.utils.data import Dataset, DataLoader
from typing import List, Tuple, Dict, Any

class RaggedSequenceDataset(Dataset):
    def __init__(self, num_samples: int = 200, min_len: int = 5, max_len: int = 25):
        super().__init__()
        # TODO: Initialize dataset samples with variable lengths
        self.samples = []

    def __len__(self) -> int:
        # TODO: Return total samples
        pass

    def __getitem__(self, idx: int) -> Tuple[torch.Tensor, int]:
        # TODO: Return (sequence_tensor, label)
        pass

def dynamic_padding_collate(batch: List[Tuple[torch.Tensor, int]]) -> Dict[str, torch.Tensor]:
    # TODO: Implement dynamic padding and attention mask generation
    pass
starter/test_datasets_and_dataloaders_lib.py (1630 bytes) 
import pytest
import torch
from examples.datasets_and_dataloaders_lib import RaggedSequenceDataset, dynamic_padding_collate, DataLoader

def test_ragged_dataset_len_and_getitem():
    ds = RaggedSequenceDataset(num_samples=50, min_len=4, max_len=15)
    assert len(ds) == 50
    seq, label = ds[0]
    assert isinstance(seq, torch.Tensor)
    assert isinstance(label, int)
    assert 4 <= len(seq) <= 15
    assert label in (0, 1)

def test_dynamic_padding_collate_function():
    batch = [
        (torch.tensor([1.0, 2.0, 3.0]), 0),
        (torch.tensor([4.0, 5.0]), 1),
        (torch.tensor([6.0, 7.0, 8.0, 9.0]), 0),
    ]
    collated = dynamic_padding_collate(batch)
    assert collated["inputs"].shape == (3, 4)
    assert collated["mask"].shape == (3, 4)
    assert collated["labels"].shape == (3,)
    # Verify padding positions are zero
    assert collated["inputs"][1, 2] == 0.0
    assert collated["mask"][1, 2] == 0.0
    assert collated["mask"][2, 3] == 1.0

def test_dataloader_iteration_and_batching():
    ds = RaggedSequenceDataset(num_samples=64, min_len=5, max_len=10)
    loader = DataLoader(ds, batch_size=16, shuffle=False, collate_fn=dynamic_padding_collate)
    batches = list(loader)
    assert len(batches) == 4
    for b in batches:
        assert b["inputs"].shape[0] == 16
        assert b["mask"].shape[0] == 16
        assert b["labels"].shape[0] == 16

def test_dataloader_drop_last():
    ds = RaggedSequenceDataset(num_samples=35, min_len=5, max_len=10)
    loader_drop = DataLoader(ds, batch_size=16, drop_last=True, collate_fn=dynamic_padding_collate)
    assert len(list(loader_drop)) == 2
tests/run_tests.sh (227 bytes) 
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 205 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_datasets_and_dataloaders_lib.py (1630 bytes) 
import pytest
import torch
from examples.datasets_and_dataloaders_lib import RaggedSequenceDataset, dynamic_padding_collate, DataLoader

def test_ragged_dataset_len_and_getitem():
    ds = RaggedSequenceDataset(num_samples=50, min_len=4, max_len=15)
    assert len(ds) == 50
    seq, label = ds[0]
    assert isinstance(seq, torch.Tensor)
    assert isinstance(label, int)
    assert 4 <= len(seq) <= 15
    assert label in (0, 1)

def test_dynamic_padding_collate_function():
    batch = [
        (torch.tensor([1.0, 2.0, 3.0]), 0),
        (torch.tensor([4.0, 5.0]), 1),
        (torch.tensor([6.0, 7.0, 8.0, 9.0]), 0),
    ]
    collated = dynamic_padding_collate(batch)
    assert collated["inputs"].shape == (3, 4)
    assert collated["mask"].shape == (3, 4)
    assert collated["labels"].shape == (3,)
    # Verify padding positions are zero
    assert collated["inputs"][1, 2] == 0.0
    assert collated["mask"][1, 2] == 0.0
    assert collated["mask"][2, 3] == 1.0

def test_dataloader_iteration_and_batching():
    ds = RaggedSequenceDataset(num_samples=64, min_len=5, max_len=10)
    loader = DataLoader(ds, batch_size=16, shuffle=False, collate_fn=dynamic_padding_collate)
    batches = list(loader)
    assert len(batches) == 4
    for b in batches:
        assert b["inputs"].shape[0] == 16
        assert b["mask"].shape[0] == 16
        assert b["labels"].shape[0] == 16

def test_dataloader_drop_last():
    ds = RaggedSequenceDataset(num_samples=35, min_len=5, max_len=10)
    loader_drop = DataLoader(ds, batch_size=16, drop_last=True, collate_fn=dynamic_padding_collate)
    assert len(list(loader_drop)) == 2

Troubleshooting

Troubleshooting: Day 205 - Datasets and DataLoaders

Common Issues

  1. RuntimeError: stack expects each tensor to be equal size:
    • Cause: Using default collate on variable length sequences.
    • Fix: Pass a custom collate_fn that pads sequences dynamically.

Security notes

Security & Privacy: Day 205 - Datasets and DataLoaders

Security Guidance

  • Avoid using pickle.load inside __getitem__ when reading untrusted input files.