Computing Foundations › Inside the Machine › Day 3
Day 3: Memory Hierarchy: Registers, RAM, and Storage
After this lesson you will be able to place every level of the memory hierarchy — registers, caches, RAM, SSD, HDD, and network storage — in order with approximate sizes and latencies, explain why the hierarchy exists, and use it to reason about swapping, caching, and why model weights must fit in fast memory.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-003-memory-hierarchy-registers-ram-and-storage
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
cd labs/sections/computing-foundations/day-003-memory-hierarchy-registers-ram-and-storage - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Order the levels of the memory hierarchy from registers to network storage and attach an approximate order-of-magnitude size and latency to each
- Explain the speed/size/cost trade-off that makes the hierarchy necessary, and why no single memory technology can be fast, large, and cheap at once
- Define temporal and spatial locality of reference and explain how cache lines and prefetching exploit them to make loops run at near-cache speed
- Describe caching as the recurring idea in computing and identify it at three different scales (CPU cache, OS page cache, content delivery network)
- Explain virtual memory, pages, swap, and thrashing in plain language, and diagnose the symptoms of a machine that has run out of RAM
- Measure your own machine's L1 and L2 cache sizes from the terminal and demonstrate the OS page-cache effect by timing a cold versus warm file read
- Connect the hierarchy to model deployment: compute whether a model fits in GPU or unified memory, estimate the token-rate ceiling set by memory bandwidth, and explain what quantization changes
Prerequisites
- Day 1 (the layered stack and your machine profile) and Day 2 (the CPU's fetch-decode-execute cycle)
- A terminal on macOS or Linux (Windows users: WSL), with your Day 1 worksheet at hand
Why this matters
Every number a model touches lives somewhere, and where it lives decides how fast it can be reached.
- The gap between the fastest and slowest tier is about a million to one.
- Almost every slow program is waiting for data, not computing on it.
- “Does it fit in GPU memory?” is the first question practitioners ask of any model, and this lesson is why.
The idea in plain language
No memory technology is fast, large and cheap at the same time, so machines layer several of them.
- Up the pyramid: faster, smaller, more expensive per byte.
- Down the pyramid: larger, slower, cheaper.
- It works because of locality. Programs reuse what they just touched and march through data in order, so hardware keeps hot data near the top by itself.
Historical background
| When | What changed | Why it mattered |
|---|---|---|
| 1946 | Burks, Goldstine and von Neumann propose a hierarchy | The idea predates the hardware by decades |
| 1965 | Wilkes describes the cache | A small fast memory that fills itself |
| 1970s | DRAM becomes the standard for main memory | Cheap capacity, at a latency cost |
| 1980s | Processors outpace memory | The “memory wall” opens and never closes |
| 2000s | Multi-level caches, L1 to L3 | One tier was no longer enough |
| 2010s | SSDs replace spinning disks | A thousandfold latency improvement, one tier down |
| 2016– | High-bandwidth memory on accelerators | Stacked memory, terabytes per second |
- The memory wall is the whole story. Processors got fast much quicker than memory got close, and every tier since exists to hide that gap.
What it is — and what it is not
It is a set of tiers, each roughly an order of magnitude slower, larger and cheaper per byte than the one above.
It is not:
-
Not something you configure. Hardware moves data between tiers automatically, using locality. You influence it by how you access data.
-
Not the same as memory versus storage. That is one boundary in the pyramid, not the whole of it.
-
Not a set of exact numbers. Carry the orders of magnitude, not the digits — they differ per machine and per year.
Why it was created and what problems it solves
Fast memory needs more transistors per bit, more area, more power, and must sit physically close to the processor. All four are expensive.
-
If all memory were register-speed, a laptop would cost more than a house and hold a few megabytes.
-
If all memory were disk-speed, a modern processor would spend essentially all its time waiting.
-
The pyramid buys nearly the speed of the fastest at nearly the price of the cheapest — and it only works because of locality.
How it works
The levels, one by one
| Level | Typical latency | Typical size | What it holds |
|---|---|---|---|
| Registers | ~0.3 ns (1 cycle) | ~1 KB | The values in play right now |
| L1 cache | ~1 ns | 32–128 KB | The hottest recent data |
| L2 cache | ~4 ns | 0.5–2 MB | Slightly cooler data |
| L3 cache | ~10–40 ns | 8–64 MB | Shared across cores |
| RAM | ~100 ns | 8–128 GB | Everything currently running |
| SSD | ~100 µs | 256 GB–4 TB | Files, and anything swapped out |
| Network storage | ~1–100 ms | Effectively unlimited | Datasets, checkpoints, archives |
At human scale, if a register access took one second: L1 is about three seconds, RAM is about five minutes, an SSD is about four days, and network storage is somewhere between a month and years.
Locality, and why loops are fast
- A miss loads a whole 64-byte cache line, not one value.
- So the neighbouring elements arrive free, and the next seven reads are hits.
- The prefetcher notices the pattern and fetches upcoming lines early.
- That is why sequential access runs at nearly cache speed even over data far larger than the cache.
Access the same array in random order and every read is a miss. Same data, same loop, an order of magnitude slower.
Virtual memory and swapping
-
Each process sees its own private address space, mapped to real memory by the hardware.
-
When RAM fills, the OS moves least-recently-used pages to disk.
-
Touching a swapped-out page causes a page fault, and the program stalls for an SSD access — roughly a thousand times a RAM access.
-
A machine that is swapping heavily feels broken rather than slow, and this is why.
An everyday analogy
A chef, a counter, a pantry and a warehouse across town.
| The kitchen | The machine | The cost of a trip |
|---|---|---|
| In your hands | Registers | Instant |
| The cutting board | L1 cache | A turn of the head |
| The counter | L2 and L3 | A step |
| The pantry | RAM | A walk to the next room |
| The warehouse across town | SSD | A drive |
| A supplier in another city | Network storage | A delivery, tomorrow |
- Good cooks stage ingredients before starting. That is locality.
- A recipe that sends you to the warehouse for each ingredient is a program with poor access patterns, and no amount of knife skill saves it.
Examples in practice
Two measurements you will make in today’s lab:
-
A warm read — the file is already in the page cache, so the number you get is RAM speed, not disk speed.
-
A cold read — the file is larger than RAM, so the cache cannot hold it and you see the SSD.
The gap between them is the pyramid, measured on your own machine.
Where the money goes: per byte, each step down is roughly an order of magnitude cheaper. Registers are the most expensive memory ever built and there are about a kilobyte of them. Network storage is nearly free and is a thousand times further away than RAM.
Implications: security, privacy, performance, scalability, and cost
-
Performance. Look at the hierarchy before optimising arithmetic. A cache miss costs more than hundreds of instructions.
-
Access order is a design decision. Row-major versus column-major traversal of the same array can differ by an order of magnitude.
-
Security. Cache timing leaks information across boundaries — Spectre and Meltdown both used it. Shared hardware means a shared cache.
-
Privacy. Data swapped to disk outlives the process, and is not erased when the program exits.
-
Cost. Renting a machine with more RAM is often cheaper than the engineering time spent making a job fit in less.
Alternatives: free, open source, and commercial
| Resource | Type | What it gives you |
|---|---|---|
| The Day 3 lab | Free | Your own cache sizes and read timings |
| Colin Scott’s latency page | Free tool | ”Latency numbers every programmer should know”, animated by year |
| Crash Course Computer Science | Free video series | Short visual episodes on memory and storage |
| Drepper, What Every Programmer Should Know About Memory | Free paper | The deep treatment, still the reference |
perf / Instruments | Free tools | Measure real cache misses on real programs |
| Hennessy & Patterson | Textbook | The standard academic account |
- Colin Scott’s page is the one to bookmark. Watching the numbers move by year makes the memory wall visible in a way prose does not.
Comparison with related concepts
| Often confused | The actual difference |
|---|---|
| Memory vs storage | Volatile and ~100 ns vs permanent and ~100 µs |
| Cache vs RAM | Automatic, small, on-chip vs addressed, large, off-chip |
| Latency vs bandwidth | How long one access takes vs bytes per second once flowing |
| Page cache vs cache | An OS use of spare RAM vs hardware between CPU and RAM |
| Swap vs storage | Storage borrowed to stand in for RAM vs storage used as itself |
Latency and bandwidth are the pair to keep straight. A model’s size is bounded by capacity; its speed is usually bounded by bandwidth; and a single random access is bounded by latency.
When to use it — and when not to
Reason at this level when something is slower than its arithmetic can explain, a job will not fit in memory, you are choosing hardware, or a program is fast on one machine and slow on another.
Ignore it when the data is small, the job is fast enough, or you have not measured yet. Optimising access patterns before profiling is how people spend a day to save a millisecond.
The tell: the processor is busy but the work is not progressing — or the same code is dramatically slower on a larger input than the size increase explains.
The AI connection
-
Bandwidth divided by model size is the token rate. That is the whole connection, and it is arithmetic rather than intuition.
-
Quantisation moves both terms in your favour — a smaller model fits where it did not, and moves fewer bytes per token.
-
Spilling out of fast memory is not a small penalty. It is a step down the pyramid, and each step is an order of magnitude.
Knowledge check
Try these from memory before looking back:
- Draw the pyramid from registers to network storage and attach an approximate latency to each level. Which single gap is the largest, and roughly how large is it?
- Explain temporal and spatial locality with one everyday example each, and state which one cache lines exploit.
- A program sums a large table row by row in 2 seconds; summing column by column takes 11 seconds. Explain why, using the terms cache line and miss.
- Your laptop has 16 GB of RAM and becomes almost unusable when you open a 20 GB dataset, though it worked fine at 10 GB. Name the mechanism and describe what is happening at the level of pages.
- A 7-billion-parameter model is stored at 2 bytes per weight on a machine with roughly 400 GB/s of memory bandwidth. Estimate the ceiling on tokens generated per second, and explain what 4-bit quantization does to both the model’s footprint and that ceiling.
Hands-on exercise
Measure your own pyramid. Worked through fully in the Day 3 lab directory.
| What you want | macOS | Linux |
|---|---|---|
| L1 and L2 sizes | sysctl hw.l1dcachesize hw.l2cachesize | lscpu | grep -i cache |
| Performance-core caches | sysctl hw.perflevel0.l1dcachesize | — |
| Timed cold and warm reads | bash examples/measure_read_speed.sh | same |
On Apple Silicon the plain names usually report the efficiency cores;
perflevel0 is the performance cluster and is typically twice as large.
Expected output
=== Memory Hierarchy Measurements ===
L1 data cache: 131072 bytes (128 KiB)
L2 cache: 16777216 bytes (16 MiB)
RAM: 38654705664 bytes (36 GiB)
Cold read (first pass): 0.023 s (8695 MB/s)
Warm read (second pass): 0.022 s (9090 MB/s)
Warm read speed-up over cold: 1.0x
-
Read the shape, not the digits. L1 in hundreds of KiB, L2 in MiB, RAM in GiB — three jumps of roughly a hundredfold.
-
A speed-up near 1.0x is itself evidence of the page cache. Both reads ran at RAM speed because the file was still cached from being written, so the “cold” read never touched the SSD.
-
To see a truly cold read, use a file much larger than RAM. The lab’s troubleshooting file shows how.
Validate your work
- L1 size stated and converted to KiB
- L2 (and L3, if present) stated and converted to MiB
- Both placed between registers and RAM, sizes ordered correctly
- Both a cold and a warm read time recorded
- You can say in one sentence which level served the warm read
-
bash tests/run_tests.shreports 0 failures
Troubleshooting
-
unknown oid 'hw.perflevel0...'— you are on an Intel Mac or older macOS. Use the plainhw.l1dcachesizenames. -
lscpushows totals, not per-core sizes. Some versions sum across cores (“L1d: 512 KiB (8 instances)”); divide by the instance count. -
Cold and warm reads take the same time. The file was still in the page cache. That is the page cache working, not the lab failing.
-
dd: operation not permitted— run from inside the lab directory, which the script requires precisely so it only ever writes there.
Common mistakes
-
Comparing your L1 to a friend’s L3. Levels compare like for like, and vendors split and label them differently.
-
Reporting the efficiency cores’ L1 as “the” L1. State which cores you measured.
-
Treating the measured MB/s as a property of the SSD. The cold read mixes SSD speed, page-cache effects and filesystem behaviour. It demonstrates the hierarchy; it is not a disk benchmark.
-
Forgetting the units. 131072 bytes is 128 KiB. Mixing KiB with KB muddles every comparison.
Practice assignment
-
Measure both reads. Time a warm read and a cold read of the same file on your machine, and record both numbers.
-
Explain the gap in one sentence, using only the words cache, page cache and latency.
-
Predict, then check. Before running it, write down what you expect a second cold read to cost, and why. Then run it.
Extension challenge
Make the cold read honestly cold, then reason about what you measured.
-
Grow the file past RAM. Several times your memory, so the page cache cannot hold it. Mind free disk space, and delete it afterwards.
-
Re-measure. Record the new cold and warm times and the ratio between them.
-
Explain the difference from your first run, using only cache, page cache, locality, latency and bandwidth.
-
Predict one more. What would a third read cost, and why? Write it down before running it.
Quiz
Q1. Why do computers use a hierarchy of memories instead of one single memory?
- Because programs must be stored separately from the data they use
- Because operating systems can only address one type of memory at a time
- Because older computers had many memory types and modern ones keep them for compatibility
- Because no memory technology is fast, large, and cheap at the same time
Show answer
Answer: D. Because no memory technology is fast, large, and cheap at the same time
Speed, size, and cost form a triangle: SRAM is fast but expensive and small, DRAM is bigger and slower, storage is bigger and slower again. Layering them — and relying on locality of reference — delivers nearly the speed of the fastest at nearly the price of the cheapest.
Q2. Which sequence orders the levels from fastest (lowest latency) to slowest?
- Registers, L1 cache, RAM, SSD, HDD
- L1 cache, registers, RAM, HDD, SSD
- Registers, RAM, L1 cache, SSD, HDD
- RAM, registers, L1 cache, SSD, HDD
Show answer
Answer: A. Registers, L1 cache, RAM, SSD, HDD
Registers respond in about a CPU cycle (~0.5 ns), L1 cache in ~1 ns, RAM in ~100 ns, an SSD random read in ~100 µs, and a hard disk seek in several milliseconds — each step roughly an order of magnitude or more.
Q3. A loop reads a large array element by element and runs at nearly cache speed even though the array lives in RAM. What best explains this?
- RAM automatically becomes as fast as cache when accessed by a loop
- A miss loads a whole 64-byte cache line, so the neighboring elements come free
- The compiler moves the entire array into the CPU's registers before the loop starts
- The operating system pauses all other programs so the loop gets the full memory bus
Show answer
Answer: B. A miss loads a whole 64-byte cache line, so the neighboring elements come free
Sequential access has strong spatial locality: a miss on a[0] brings a[1]..a[7] along in the same cache line, and hardware prefetching notices the regular pattern and stays ahead of the loop. Random access defeats both and can be many times slower.
Q4. What is the difference between latency and bandwidth for a memory or storage level?
- They are two names for the same measurement
- Latency applies only to disks, while bandwidth applies only to RAM
- Latency is how long one access takes; bandwidth is bytes per second once flowing
- Latency is measured in bytes and bandwidth in seconds
Show answer
Answer: C. Latency is how long one access takes; bandwidth is bytes per second once flowing
A level can pair high bandwidth with poor latency (network storage is the classic case). Latency punishes many small scattered accesses; bandwidth caps large sequential transfers — which is why batching small accesses into large ones is such a common optimization.
Q5. What actually happens when a machine starts swapping (thrashing)?
- The CPU overheats and reduces its clock speed until memory cools down
- The operating system compresses the CPU caches to make room for more programs
- The SSD takes over executing instructions while RAM is cleared
- RAM is full, so the OS pages memory out to disk and stalls on page faults
Show answer
Answer: D. RAM is full, so the OS pages memory out to disk and stalls on page faults
Virtual memory lets the OS back overflowing RAM with disk, but every touched-and-missing page costs a ~hundred-microsecond (or worse) trip. Sustained swapping effectively demotes RAM to SSD speed, which users feel as the machine becoming unresponsive.
Q6. Why is the second read of a large file often several times faster than the first?
- The OS kept the file in its page cache, so the second read never touched the disk
- The SSD physically speeds up once its circuits warm to operating temperature
- The file becomes smaller after it has been read once
- The first read defragments the file so later reads are sequential
Show answer
Answer: A. The OS kept the file in its page cache, so the second read never touched the disk
The OS uses otherwise-idle RAM as a cache of recently read disk contents. A warm read is served at RAM speed — the same caching trick as CPU caches, applied one level down the pyramid. Today's lab measures this directly.
Q7. A 7-billion-parameter model stored at 2 bytes per weight runs on hardware with roughly 400 GB/s of memory bandwidth. Why does 4-bit quantization speed up token generation?
- Quantization increases the memory bandwidth of the hardware
- Each token reads every weight, so a 4x smaller model moves a quarter of the bytes
- Quantization lets the model skip reading most of its weights for each token
- Quantized weights are stored in the CPU's registers permanently
Show answer
Answer: B. Each token reads every weight, so a 4x smaller model moves a quarter of the bytes
Token generation is usually memory-bandwidth-bound: the ceiling is approximately bandwidth divided by model size. Quantization shrinks the bytes that must stream through memory for every token — and also lets larger models fit in fast memory at all.
Q8. What makes unified memory (as on Apple Silicon Macs) interesting for running models locally?
- Unified memory is persistent, so models stay loaded after the machine powers off
- Unified memory is faster than the high-bandwidth memory used in datacenter GPUs
- CPU and GPU share one pool, so models too big for a discrete GPU still fit
- Unified memory eliminates the need for caches on the CPU
Show answer
Answer: C. CPU and GPU share one pool, so models too big for a discrete GPU still fit
Unified memory trades the raw bandwidth of dedicated GPU memory for one large shared pool — often far larger than a consumer GPU's onboard memory — so mid-sized models fit in fast memory on a laptop instead of spilling to a slower tier.
Glossary
- memory hierarchy
- The layered arrangement of all the places a computer keeps data — registers, caches, RAM, storage, network — with each level larger, slower, and cheaper per byte than the one above.
- cache
- A small, fast memory that holds copies of recently or soon-to-be-used data from a larger, slower level so most accesses are served at the fast level's speed.
- L1 cache
- The smallest, fastest cache, private to each CPU core and usually split into instruction and data halves; typically tens to a couple of hundred KB, reached in about a nanosecond.
- L2 cache
- The second cache level — larger and slightly slower than L1 (typically hundreds of KB to several MB), private to a core or shared by a cluster of cores.
- L3 cache
- The largest on-chip cache level, typically shared by all cores; some designs (such as Apple's) use a shared system-level cache in the same role under a different name.
- cache line
- The fixed-size chunk (commonly 64 bytes) that moves between RAM and cache: a miss on one byte loads the whole line, so neighboring data arrives for free.
- latency
- The time one access takes to complete, from request to first data — about 1 ns for L1 cache, ~100 ns for RAM, ~100 µs for an SSD random read (approximate orders of magnitude).
- bandwidth
- The rate at which a memory or storage level can move data once transfers are flowing, measured in bytes per second — distinct from latency, which measures a single access.
- locality of reference
- The tendency of programs to reuse recently touched data (temporal locality) and to touch data near recently touched data (spatial locality) — the property that makes caching work.
- virtual memory
- The illusion, maintained by the OS and the memory management unit, that each program has its own private address space, translated to real RAM in fixed-size pages.
- page
- The fixed-size unit (commonly 4 or 16 KB) in which virtual memory is mapped, protected, and, when necessary, moved between RAM and disk.
- swap
- Disk space the OS uses to hold memory pages evicted from full RAM; heavy, sustained swapping (thrashing) makes a machine crawl because disk is thousands of times slower than RAM.
- page cache
- The OS's use of otherwise-free RAM to keep copies of recently read disk contents, so repeated file reads are served at RAM speed instead of storage speed.
- SSD
- A solid-state drive: persistent storage built from flash memory with no moving parts, answering random reads in roughly a tenth of a millisecond.
- HDD
- A hard disk drive: persistent storage on spinning magnetic platters read by a moving head, cheap per byte but needing several milliseconds per seek.
- unified memory
- A design in which CPU and GPU share one physical memory pool, so a model or dataset can be large without being copied between separate CPU and GPU memories.
- memory wall
- The long-running growth gap between processor speed and memory speed, which makes data movement — not arithmetic — the bottleneck for much of modern computing.
Sources and further reading
- Memory hierarchy — Wikipedia (accessed 2026-07-12)
- CPU cache — Wikipedia (accessed 2026-07-12) — Well-cited reference for cache levels, cache lines, and real chip parameters.
- Interactive latency numbers — Colin Scott (accessed 2026-07-12) — The classic "latency numbers every programmer should know," animated year by year.
- Crash Course Computer Science — PBS Digital Studios (accessed 2026-07-12) — Free video series; the memory, storage, and files episodes complement this lesson.
- Operating Systems: Three Easy Pieces — Arpaci-Dusseau (free book) (accessed 2026-07-12) — Free textbook; its virtual-memory chapters are the recommended further reading for this day.
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.