Computing Foundations › Inside the Machine › Day 3

Day 3: Memory Hierarchy: Registers, RAM, and Storage

Day 3 of 365 — Memory Hierarchy: Registers, RAM, and Storage

After this lesson you will be able to place every level of the memory hierarchy — registers, caches, RAM, SSD, HDD, and network storage — in order with approximate sizes and latencies, explain why the hierarchy exists, and use it to reason about swapping, caching, and why model weights must fit in fast memory.

Course
Computing Foundations
Category
Inside the Machine
Reading time
≈ 40 min
Practical time
≈ 30 min
Lesson duration
1h 10m
Last verified
2026-07-12

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-003-memory-hierarchy-registers-ram-and-storage

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/computing-foundations/day-003-memory-hierarchy-registers-ram-and-storage
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Diagram: memory latencies rescaled to human time

Every number a model touches lives somewhere, and where it lives decides how fast it can be reached.


The idea in plain language

No memory technology is fast, large and cheap at the same time, so machines layer several of them.

Diagram: the memory hierarchy pyramid with approximate sizes and latencies at every level


Historical background

Diagram: the widening gap between processor speed and memory latency

WhenWhat changedWhy it mattered
1946Burks, Goldstine and von Neumann propose a hierarchyThe idea predates the hardware by decades
1965Wilkes describes the cacheA small fast memory that fills itself
1970sDRAM becomes the standard for main memoryCheap capacity, at a latency cost
1980sProcessors outpace memoryThe “memory wall” opens and never closes
2000sMulti-level caches, L1 to L3One tier was no longer enough
2010sSSDs replace spinning disksA thousandfold latency improvement, one tier down
2016–High-bandwidth memory on acceleratorsStacked memory, terabytes per second

What it is — and what it is not

It is a set of tiers, each roughly an order of magnitude slower, larger and cheaper per byte than the one above.

It is not:


Why it was created and what problems it solves

Diagram: a machine built entirely of fast memory compared with one built entirely of slow memory

Fast memory needs more transistors per bit, more area, more power, and must sit physically close to the processor. All four are expensive.


How it works

Diagram: the sequence of a page fault when memory is oversubscribed

The levels, one by one

LevelTypical latencyTypical sizeWhat it holds
Registers~0.3 ns (1 cycle)~1 KBThe values in play right now
L1 cache~1 ns32–128 KBThe hottest recent data
L2 cache~4 ns0.5–2 MBSlightly cooler data
L3 cache~10–40 ns8–64 MBShared across cores
RAM~100 ns8–128 GBEverything currently running
SSD~100 µs256 GB–4 TBFiles, and anything swapped out
Network storage~1–100 msEffectively unlimitedDatasets, checkpoints, archives

At human scale, if a register access took one second: L1 is about three seconds, RAM is about five minutes, an SSD is about four days, and network storage is somewhere between a month and years.

Locality, and why loops are fast

Diagram: a loop reading data from cache versus reaching out to RAM

Access the same array in random order and every read is a miss. Same data, same loop, an order of magnitude slower.

Virtual memory and swapping


An everyday analogy

A chef, a counter, a pantry and a warehouse across town.

The kitchenThe machineThe cost of a trip
In your handsRegistersInstant
The cutting boardL1 cacheA turn of the head
The counterL2 and L3A step
The pantryRAMA walk to the next room
The warehouse across townSSDA drive
A supplier in another cityNetwork storageA delivery, tomorrow

Examples in practice

Diagram: sequential access compared with random access

Two measurements you will make in today’s lab:

The gap between them is the pyramid, measured on your own machine.

Where the money goes: per byte, each step down is roughly an order of magnitude cheaper. Registers are the most expensive memory ever built and there are about a kilobyte of them. Network storage is nearly free and is a thousand times further away than RAM.


Implications: security, privacy, performance, scalability, and cost

Diagram: temporal and spatial locality and why caching works


Alternatives: free, open source, and commercial

ResourceTypeWhat it gives you
The Day 3 labFreeYour own cache sizes and read timings
Colin Scott’s latency pageFree tool”Latency numbers every programmer should know”, animated by year
Crash Course Computer ScienceFree video seriesShort visual episodes on memory and storage
Drepper, What Every Programmer Should Know About MemoryFree paperThe deep treatment, still the reference
perf / InstrumentsFree toolsMeasure real cache misses on real programs
Hennessy & PattersonTextbookThe standard academic account

Diagram: latency compared with bandwidth

Often confusedThe actual difference
Memory vs storageVolatile and ~100 ns vs permanent and ~100 µs
Cache vs RAMAutomatic, small, on-chip vs addressed, large, off-chip
Latency vs bandwidthHow long one access takes vs bytes per second once flowing
Page cache vs cacheAn OS use of spare RAM vs hardware between CPU and RAM
Swap vs storageStorage borrowed to stand in for RAM vs storage used as itself

Latency and bandwidth are the pair to keep straight. A model’s size is bounded by capacity; its speed is usually bounded by bandwidth; and a single random access is bounded by latency.


When to use it — and when not to

Diagram: when to reason about the memory hierarchy

Reason at this level when something is slower than its arithmetic can explain, a job will not fit in memory, you are choosing hardware, or a program is fast on one machine and slow on another.

Ignore it when the data is small, the job is fast enough, or you have not measured yet. Optimising access patterns before profiling is how people spend a day to save a millisecond.

The tell: the processor is busy but the work is not progressing — or the same code is dramatically slower on a larger input than the size increase explains.


The AI connection

Diagram: how model size and memory bandwidth set a token-per-second ceiling


Knowledge check

Try these from memory before looking back:

  1. Draw the pyramid from registers to network storage and attach an approximate latency to each level. Which single gap is the largest, and roughly how large is it?
  2. Explain temporal and spatial locality with one everyday example each, and state which one cache lines exploit.
  3. A program sums a large table row by row in 2 seconds; summing column by column takes 11 seconds. Explain why, using the terms cache line and miss.
  4. Your laptop has 16 GB of RAM and becomes almost unusable when you open a 20 GB dataset, though it worked fine at 10 GB. Name the mechanism and describe what is happening at the level of pages.
  5. A 7-billion-parameter model is stored at 2 bytes per weight on a machine with roughly 400 GB/s of memory bandwidth. Estimate the ceiling on tokens generated per second, and explain what 4-bit quantization does to both the model’s footprint and that ceiling.

Hands-on exercise

Measure your own pyramid. Worked through fully in the Day 3 lab directory.

What you wantmacOSLinux
L1 and L2 sizessysctl hw.l1dcachesize hw.l2cachesizelscpu | grep -i cache
Performance-core cachessysctl hw.perflevel0.l1dcachesize—
Timed cold and warm readsbash examples/measure_read_speed.shsame

On Apple Silicon the plain names usually report the efficiency cores; perflevel0 is the performance cluster and is typically twice as large.

Expected output

=== Memory Hierarchy Measurements ===
L1 data cache: 131072 bytes (128 KiB)
L2 cache: 16777216 bytes (16 MiB)
RAM: 38654705664 bytes (36 GiB)
Cold read (first pass):  0.023 s (8695 MB/s)
Warm read (second pass): 0.022 s (9090 MB/s)
Warm read speed-up over cold: 1.0x

Validate your work

Troubleshooting

Common mistakes


Practice assignment

  1. Measure both reads. Time a warm read and a cold read of the same file on your machine, and record both numbers.

  2. Explain the gap in one sentence, using only the words cache, page cache and latency.

  3. Predict, then check. Before running it, write down what you expect a second cold read to cost, and why. Then run it.


Extension challenge

Make the cold read honestly cold, then reason about what you measured.

  1. Grow the file past RAM. Several times your memory, so the page cache cannot hold it. Mind free disk space, and delete it afterwards.

  2. Re-measure. Record the new cold and warm times and the ratio between them.

  3. Explain the difference from your first run, using only cache, page cache, locality, latency and bandwidth.

  4. Predict one more. What would a third read cost, and why? Write it down before running it.


Quiz

Q1. Why do computers use a hierarchy of memories instead of one single memory?

  1. Because programs must be stored separately from the data they use
  2. Because operating systems can only address one type of memory at a time
  3. Because older computers had many memory types and modern ones keep them for compatibility
  4. Because no memory technology is fast, large, and cheap at the same time
Show answer

Answer: D. Because no memory technology is fast, large, and cheap at the same time

Speed, size, and cost form a triangle: SRAM is fast but expensive and small, DRAM is bigger and slower, storage is bigger and slower again. Layering them — and relying on locality of reference — delivers nearly the speed of the fastest at nearly the price of the cheapest.

Q2. Which sequence orders the levels from fastest (lowest latency) to slowest?

  1. Registers, L1 cache, RAM, SSD, HDD
  2. L1 cache, registers, RAM, HDD, SSD
  3. Registers, RAM, L1 cache, SSD, HDD
  4. RAM, registers, L1 cache, SSD, HDD
Show answer

Answer: A. Registers, L1 cache, RAM, SSD, HDD

Registers respond in about a CPU cycle (~0.5 ns), L1 cache in ~1 ns, RAM in ~100 ns, an SSD random read in ~100 µs, and a hard disk seek in several milliseconds — each step roughly an order of magnitude or more.

Q3. A loop reads a large array element by element and runs at nearly cache speed even though the array lives in RAM. What best explains this?

  1. RAM automatically becomes as fast as cache when accessed by a loop
  2. A miss loads a whole 64-byte cache line, so the neighboring elements come free
  3. The compiler moves the entire array into the CPU's registers before the loop starts
  4. The operating system pauses all other programs so the loop gets the full memory bus
Show answer

Answer: B. A miss loads a whole 64-byte cache line, so the neighboring elements come free

Sequential access has strong spatial locality: a miss on a[0] brings a[1]..a[7] along in the same cache line, and hardware prefetching notices the regular pattern and stays ahead of the loop. Random access defeats both and can be many times slower.

Q4. What is the difference between latency and bandwidth for a memory or storage level?

  1. They are two names for the same measurement
  2. Latency applies only to disks, while bandwidth applies only to RAM
  3. Latency is how long one access takes; bandwidth is bytes per second once flowing
  4. Latency is measured in bytes and bandwidth in seconds
Show answer

Answer: C. Latency is how long one access takes; bandwidth is bytes per second once flowing

A level can pair high bandwidth with poor latency (network storage is the classic case). Latency punishes many small scattered accesses; bandwidth caps large sequential transfers — which is why batching small accesses into large ones is such a common optimization.

Q5. What actually happens when a machine starts swapping (thrashing)?

  1. The CPU overheats and reduces its clock speed until memory cools down
  2. The operating system compresses the CPU caches to make room for more programs
  3. The SSD takes over executing instructions while RAM is cleared
  4. RAM is full, so the OS pages memory out to disk and stalls on page faults
Show answer

Answer: D. RAM is full, so the OS pages memory out to disk and stalls on page faults

Virtual memory lets the OS back overflowing RAM with disk, but every touched-and-missing page costs a ~hundred-microsecond (or worse) trip. Sustained swapping effectively demotes RAM to SSD speed, which users feel as the machine becoming unresponsive.

Q6. Why is the second read of a large file often several times faster than the first?

  1. The OS kept the file in its page cache, so the second read never touched the disk
  2. The SSD physically speeds up once its circuits warm to operating temperature
  3. The file becomes smaller after it has been read once
  4. The first read defragments the file so later reads are sequential
Show answer

Answer: A. The OS kept the file in its page cache, so the second read never touched the disk

The OS uses otherwise-idle RAM as a cache of recently read disk contents. A warm read is served at RAM speed — the same caching trick as CPU caches, applied one level down the pyramid. Today's lab measures this directly.

Q7. A 7-billion-parameter model stored at 2 bytes per weight runs on hardware with roughly 400 GB/s of memory bandwidth. Why does 4-bit quantization speed up token generation?

  1. Quantization increases the memory bandwidth of the hardware
  2. Each token reads every weight, so a 4x smaller model moves a quarter of the bytes
  3. Quantization lets the model skip reading most of its weights for each token
  4. Quantized weights are stored in the CPU's registers permanently
Show answer

Answer: B. Each token reads every weight, so a 4x smaller model moves a quarter of the bytes

Token generation is usually memory-bandwidth-bound: the ceiling is approximately bandwidth divided by model size. Quantization shrinks the bytes that must stream through memory for every token — and also lets larger models fit in fast memory at all.

Q8. What makes unified memory (as on Apple Silicon Macs) interesting for running models locally?

  1. Unified memory is persistent, so models stay loaded after the machine powers off
  2. Unified memory is faster than the high-bandwidth memory used in datacenter GPUs
  3. CPU and GPU share one pool, so models too big for a discrete GPU still fit
  4. Unified memory eliminates the need for caches on the CPU
Show answer

Answer: C. CPU and GPU share one pool, so models too big for a discrete GPU still fit

Unified memory trades the raw bandwidth of dedicated GPU memory for one large shared pool — often far larger than a consumer GPU's onboard memory — so mid-sized models fit in fast memory on a laptop instead of spilling to a slower tier.

Glossary

memory hierarchy
The layered arrangement of all the places a computer keeps data — registers, caches, RAM, storage, network — with each level larger, slower, and cheaper per byte than the one above.
cache
A small, fast memory that holds copies of recently or soon-to-be-used data from a larger, slower level so most accesses are served at the fast level's speed.
L1 cache
The smallest, fastest cache, private to each CPU core and usually split into instruction and data halves; typically tens to a couple of hundred KB, reached in about a nanosecond.
L2 cache
The second cache level — larger and slightly slower than L1 (typically hundreds of KB to several MB), private to a core or shared by a cluster of cores.
L3 cache
The largest on-chip cache level, typically shared by all cores; some designs (such as Apple's) use a shared system-level cache in the same role under a different name.
cache line
The fixed-size chunk (commonly 64 bytes) that moves between RAM and cache: a miss on one byte loads the whole line, so neighboring data arrives for free.
latency
The time one access takes to complete, from request to first data — about 1 ns for L1 cache, ~100 ns for RAM, ~100 µs for an SSD random read (approximate orders of magnitude).
bandwidth
The rate at which a memory or storage level can move data once transfers are flowing, measured in bytes per second — distinct from latency, which measures a single access.
locality of reference
The tendency of programs to reuse recently touched data (temporal locality) and to touch data near recently touched data (spatial locality) — the property that makes caching work.
virtual memory
The illusion, maintained by the OS and the memory management unit, that each program has its own private address space, translated to real RAM in fixed-size pages.
page
The fixed-size unit (commonly 4 or 16 KB) in which virtual memory is mapped, protected, and, when necessary, moved between RAM and disk.
swap
Disk space the OS uses to hold memory pages evicted from full RAM; heavy, sustained swapping (thrashing) makes a machine crawl because disk is thousands of times slower than RAM.
page cache
The OS's use of otherwise-free RAM to keep copies of recently read disk contents, so repeated file reads are served at RAM speed instead of storage speed.
SSD
A solid-state drive: persistent storage built from flash memory with no moving parts, answering random reads in roughly a tenth of a millisecond.
HDD
A hard disk drive: persistent storage on spinning magnetic platters read by a moving head, cheap per byte but needing several milliseconds per seek.
unified memory
A design in which CPU and GPU share one physical memory pool, so a model or dataset can be large without being copied between separate CPU and GPU memories.
memory wall
The long-running growth gap between processor speed and memory speed, which makes data movement — not arithmetic — the bottleneck for much of modern computing.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.