Computing Foundations › Systems Foundations: Storage, Observability, and Tooling › Day 39

Day 39: Data Storage: Files, Databases, Object Storage, and Caches

Day 39 of 365 — Data Storage: Files, Databases, Object Storage, and Caches

After this lesson you will be able to name the four main ways software stores data — files, databases, object storage, and caches — explain the trade-offs that separate them, and choose the right store for a given workload on purpose instead of by habit.

Course
Computing Foundations
Category
Systems Foundations: Storage, Observability, and Tooling
Reading time
≈ 40 min
Practical time
≈ 30 min
Lesson duration
1h 10m
Last verified
2026-07-12

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-039-data-storage-files-databases-object-storage

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/computing-foundations/day-039-data-storage-files-databases-object-storage
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters


The idea in plain language

Diagram: how access pattern rather than data size determines the right store


Historical background


What it is — and what it is not

These are:

They are not:


Why it was created and what problems it solves


How it works

Diagram: the four storage shapes and the access pattern each suits

Files and filesystems

Diagram: the limits that force a move from a file to a database

Relational databases

Diagram: how a transaction keeps related changes consistent

SELECT customer, SUM(amount) AS total
FROM orders
WHERE state = 'OH'
GROUP BY customer;

Three storage shapes, often confused

Diagram: four storage types compared — a file, a relational database, an object store, and a cache, each with its shape and best use

NoSQL families

Diagram: the four NoSQL database families and what each suits

Object storage

Diagram: an object storage key compared with a filesystem path

Caches

Diagram: the cache hit and miss path


An everyday analogy

Diagram: replication compared with backup

Storing things in a building.

The buildingThe store
A labelled box on a shelfA file
A filing system with an index and rulesA relational database
A vast warehouse, everything by ticket numberObject storage
The tray on your deskA cache
The floor and the shelvingBlock storage

Examples in practice

Flowchart: a storage decision tree asking whether data is a big blob, whether it is structured and needs queries, and whether it needs extreme speed, leading to object storage, a relational database, a NoSQL store, or a cache


Implications: security, privacy, performance, scalability, and cost

Diagram: the relative cost of database storage and object storage


Alternatives: free, open source, and commercial

OptionTypeWhere it fits
SQLiteFree, open sourceA whole relational database in one file. Start here far more often than people do.
PostgreSQLFree, open sourceThe default relational choice; handles JSON and vectors too
RedisFree, open sourceCaching, sessions, queues
S3-compatible object storageCommercial, free tiersDatasets, media, checkpoints, backups
Parquet on object storageFree formatLarge analytical datasets, columnar and compressed
Vector databasesFree and commercialSimilarity search over embeddings

TermWhat it actually is
Block storageRaw fixed-size blocks; the foundation
File storageNamed files in a folder tree
Object storageA flat pool of blobs addressed by key
SchemaThe declared structure of a database
TransactionA group of changes that all apply or none do
IndexA structure making lookups fast
TTLHow long a cached entry stays valid

When to use it — and when not to

Diagram: the progression from a file to a database to object storage


The AI connection

Diagram: which store suits each kind of AI system data


Knowledge check

quiz:
  question: "A team stores a 200 GB training dataset as rows in a relational database. It is only ever read in full, sequentially, for training. What is the main problem?"
  options:
    - "They are paying database prices and overhead for an access pattern object storage serves better and far more cheaply"
    - "Relational databases cannot store datasets larger than a few gigabytes"
    - "Sequential reads are impossible in a relational database, so training will fail"
    - "The data must be normalised into multiple tables before it can be queried"
  answer_index: 0
  explanation: "The database's indexes, transactions and query planning all exist to serve selective access, and none of them helps a full sequential scan. The workload wants cheap bulk storage read whole, which is precisely what object storage is for — at roughly a tenth of the cost."

Hands-on exercise

Store the same data four ways and measure the difference. Worked through fully in the Day 39 lab directory.

What you are doingCommand
Query a filepython3 -c "import csv,sys; ..." over a CSV
Load it into SQLitesqlite3 data.db ".import data.csv orders"
Query with SQLsqlite3 data.db "SELECT customer, SUM(amount) ... GROUP BY customer"
Add an index and re-timeCREATE INDEX idx_state ON orders(state);
Store a blob by keyThe lab’s local object-store stand-in
Cache a resultStore it in a dictionary with a timestamp, and expire it
CompareTime all four for the same question

Expected output

scan whole CSV:        1.84s
sqlite, no index:      0.42s
sqlite, with index:    0.008s
cache hit:             0.00002s

Validate your work

Troubleshooting

Common mistakes


Practice assignment


Extension challenge


Quiz

Q1. Which storage shape is the right home for a plain configuration file that one program reads and that changes rarely?

  1. A relational database, for the ACID guarantees
  2. Object storage, because keys look like paths
  3. A file — the simplest thing that works for one reader
  4. A cache, because config is read often
Show answer

Answer: C. A file — the simplest thing that works for one reader

A file is the simplest correct choice when data is read far more than written and by one program at a time. You reach past files only when many concurrent writers or precise queries appear — none of which a rarely-changing config needs.

Q2. Two withdrawals hit a bank balance stored in a plain file at the same time; both read 100, subtract 30, and write 70, so one withdrawal is lost. Which property of relational databases prevents this?

  1. Indexes, because they sort the data
  2. Object immutability, because blobs cannot change
  3. A time-to-live, which expires stale data
  4. ACID transactions, making related changes all-or-nothing
Show answer

Answer: D. ACID transactions, making related changes all-or-nothing

ACID (Atomicity, Consistency, Isolation, Durability) wraps related changes in a transaction that fully happens or not at all and stops two clients from stepping on each other, so no update silently vanishes.

Q3. Why can a database answer "customers in Ohio" far faster than grepping a million-line file for the same thing?

  1. The database keeps an index, so it jumps instead of scanning
  2. The database stores the data in RAM and the file is always on disk
  3. grep is written in a slow programming language
  4. The database compresses the data so there is less to read
Show answer

Answer: A. The database keeps an index, so it jumps instead of scanning

An index is an extra sorted structure that lets the database find relevant rows directly, turning a full scan into a quick lookup. A plain file has no index, so answering the same question means reading and scanning everything.

Q4. In object storage, how is a stored object identified and retrieved?

  1. By its row number in a table
  2. By bucket and key, fetching the whole object back
  3. By opening a file handle and seeking to an offset
  4. By a SQL SELECT over the object contents
Show answer

Answer: B. By bucket and key, fetching the whole object back

Object storage puts each object in a bucket under a key and you GET the whole object back by that key. The namespace is flat — keys may look like paths but there are no real folders — and you cannot run SQL over the contents or edit an object in place.

Q5. Which statement about a cache is true?

  1. A cache is the durable source of truth for your data
  2. A cache must be a relational database
  3. A fast volatile copy in front of a slower store
  4. A cache permanently replaces the store behind it
Show answer

Answer: C. A fast volatile copy in front of a slower store

A cache holds copies of hot data in fast memory so most reads never touch the slow store. It is volatile and disposable — if you cannot afford to lose it, it is not a cache but a source of truth.

Q6. Which NoSQL family is the natural fit for "friends of friends" or shortest-path questions about connected entities?

  1. Key-value store
  2. Wide-column store
  3. Document store
  4. Graph database
Show answer

Answer: D. Graph database

A graph database stores nodes and the edges between them, making relationship traversals like "friends of friends" or shortest path natural — exactly the queries that tie a relational database in knots with repeated joins.

Q7. Where should a 5 GB training dataset live, and where should the record describing it (name, owner, date) live?

  1. The dataset in object storage; the record in the database
  2. Both in a cache, for speed
  3. Both in the relational database, in the same table
  4. The dataset in a relational database; the record in a plain file
Show answer

Answer: A. The dataset in object storage; the record in the database

Large blobs belong in cheap, durable object storage; the database stores only the key plus structured, queryable metadata. Putting the 5 GB blob in the database would bloat it, slow every backup, and cost far more.

Q8. What is "cache invalidation," and why is a time-to-live (TTL) a common way to handle it?

  1. It is deleting the underlying database; a TTL schedules that deletion
  2. Keeping stale copies fresh; a TTL expires each entry
  3. It is validating SQL syntax before a query runs
  4. It is encrypting cached data; a TTL sets the encryption key lifetime
Show answer

Answer: B. Keeping stale copies fresh; a TTL expires each entry

When the underlying data changes, cached copies can go stale, so they must be refreshed or evicted. A TTL expires each entry after a set number of seconds, trading a little possible staleness for simplicity instead of tracking every change precisely.

Glossary

file
A named blob of bytes stored on disk and organized by a filesystem into folders; the simplest storage shape, ideal when data is read far more than written and by one program at a time.
filesystem
The part of an operating system that organizes a disk into named files inside a tree of folders and tracks where each file's bytes physically live.
database
A running program whose whole job is to store data so it can be queried and changed safely, even when many clients access it at once.
relational database
A database that stores data in tables of rows and typed columns under a fixed schema, queried with SQL and protected by ACID transactions; examples include PostgreSQL, MySQL, and SQLite.
SQL
Structured Query Language, the standard language for asking questions of and changing data in a relational database, with clauses such as SELECT, WHERE, and GROUP BY.
schema
The declared structure of a relational database: which tables exist, which typed columns each has, and which rules (uniqueness, references) must hold.
NoSQL
A family of non-relational databases — key-value, document, wide-column, and graph — that trade some of the relational model's strict rules for scale, flexibility, or specialized query shapes.
ACID
The four guarantees a relational transaction provides — Atomicity (all-or-nothing), Consistency (no invalid state), Isolation (concurrent clients do not corrupt each other), and Durability (committed data survives a crash).
index
An extra sorted lookup structure a database maintains so it can jump straight to matching rows instead of scanning every row, turning a slow full scan into a fast lookup.
object storage
A flat store of large, immutable blobs addressed by a key inside a bucket; cheap, highly durable, and effectively unlimited, and the natural home for datasets, media, backups, and model files.
bucket
The top-level container in object storage that holds objects, each stored under a key; the S3-style equivalent of a namespace for your blobs.
key-value store
A NoSQL database that works like a giant dictionary — hand it a key, get a value back very fast — ideal for sessions, caches, and simple lookups; Redis and DynamoDB are examples.
cache
A fast, volatile copy of frequently used data placed in front of a slower store so most reads never reach the slow store; disposable, because the durable data lives elsewhere.
cache invalidation
The problem of keeping cached copies from going stale when the underlying data changes, often handled with a time-to-live (TTL) that expires each entry after a set time.
durability
The guarantee that once data is committed to a store it survives crashes and restarts; object storage and databases provide it, while a cache deliberately does not.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.