Computing Foundations › The Command Line › Day 10
Day 10: Working with Text: cat, grep, sed, and Pipes
After this lesson you will be able to compose small command-line tools into pipelines that search, filter, transform, and summarize text — and use them to inspect and clean the kind of real data you will feed to machine-learning workflows.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
cd labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Explain the Unix philosophy — small tools that each do one thing, composed with pipes — and why a universal text interface makes composition powerful
- Describe the three standard streams (stdin, stdout, stderr) and why keeping errors separate from results makes pipes safe
- Use the pipe `|` and the redirection operators `>`, `>>`, and `<` to connect commands and route output to and from files
- View text with cat, less, head, and tail, and search it with grep using the flags -i, -c, -n, -v, and -r plus a basic regular expression
- Transform and summarize text with sed substitution, cut, sort, uniq -c, and wc
- Build a top-N pipeline step by step and apply it to a real web log to find the busiest client, count errors, and list unique paths
- Connect these tools to AI practice: inspecting datasets, cleaning training data, and filtering generated text streams
Prerequisites
- Day 8 (Meet the Terminal) and Day 9 (Navigating the Filesystem) for basic command-line comfort
- A computer running macOS or Linux, or Windows with WSL, and permission to open a terminal
Why this matters
-
Training logs are text, and finding the one line that explains a failed run is a
grepaway. -
Datasets arrive as CSV and JSONL, and you will inspect them before you load them, every time.
-
A pipeline answers questions faster than writing a script, and often faster than opening the file.
-
These tools handle files larger than your RAM, because they stream rather than load.
-
Eight commands, combined, replace most of what you would otherwise write code for. Today is the highest ratio of leverage to effort in the whole week.
The idea in plain language
-
Text is the universal interface. Every tool reads it and writes it, so every tool connects to every other.
-
Each tool does one thing.
grepselects lines.sortorders them.wccounts them. None of them tries to do the others’ jobs. -
A pipe joins them. The
|sends one program’s output straight into the next program’s input. -
You compose the answer rather than looking for a program that already has it.
-
That is the whole idea, and it has outlasted every graphical tool built since.
Historical background
-
Pipes arrived in Unix in 1973, proposed by Doug McIlroy and implemented by Ken Thompson in a single evening.
-
They changed how the tools were written. Before pipes, programs wrote to files; after, they wrote to a stream someone else might be reading.
-
McIlroy’s summary became the Unix philosophy: write programs that do one thing well, and write them to work together, because text streams are a universal interface.
-
grepis named after an editor command,g/re/p— globally search a regular expression and print. -
sedandawkfollowed, andawk’s authors’ initials gave it its name. -
None of this has changed. The commands you type today would run unmodified on a machine from 1980.
What it is — and what it is not
-
Every program gets three channels when it starts: standard input (fd 0), standard output (fd 1), and standard error (fd 2).
-
By default input is your keyboard and both outputs are your screen, which is why everything appears mixed together.
-
The shell can rewire these before the program starts. That is the whole trick, and the program never knows.
It is not:
-
Not a programming language. There are no variables here yet; that is Day 12.
-
Not limited to small files. A pipeline streams, so it can process more data than you have memory or disk for.
-
Not a replacement for pandas or a database. It is what you use before you decide you need one.
Why it was created and what problems it solves
| Operator | What it does | Example |
|---|---|---|
| | stdout of the left into stdin of the right | cat log | grep 404 |
> | Redirect stdout to a file, overwriting | grep 404 log > errors.txt |
>> | Redirect stdout to a file, appending | grep 500 log >> errors.txt |
< | Take stdin from a file | wc -l < log |
2> | Redirect stderr to a file | sort huge 2> problems.txt |
2>&1 | Send stderr wherever stdout is going | ./run 2>&1 | less |
-
>overwrites without asking.>>is the one that appends, and confusing them destroys files. -
Both programs in a pipe run simultaneously, which is why
tail -f | grepshows matches as they arrive.
How it works
Viewing text
-
catdumps a file to standard output — good for short files and for feeding a pipe. -
headprints the first lines (10 by default;head -n 3for three). -
tailprints the last lines, andtail -ffollows a growing file live — invaluable for watching a training log. -
lessopens a scrollable pager: search with/, quit withq. -
The rule of thumb:
catto dump or pipe,head/tailto peek at the ends,lessto actually read.
Searching with grep
grep PATTERN fileprints every line containing the pattern.grep '^Error' logfinds lines that begin with Error.grep -c '404' access.logcounts how many lines mention 404.- Regular expressions are a deep topic — Day 38 covers them properly. Today you need only the anchors and the character classes.
Transforming
-
sed 's/old/new/'replaces the first occurrence per line; the trailinggins/old/new/greplaces every occurrence. -
sedprints a transformed stream and leaves your file untouched. -
cut -d',' -f1takes the first comma-separated field — made for CSV. -
sortorders lines;-nsorts numerically,-rreverses,-uremoves duplicates in one step. -
uniqcollapses adjacent identical lines;uniq -cprefixes each with a count. -
wc -lcounts lines,-wwords,-cbytes.
Building a pipeline
The question: which client addresses hit our server most often?
cat access.log
cat access.log | cut -d' ' -f1
cat access.log | cut -d' ' -f1 | sort
cat access.log | cut -d' ' -f1 | sort | uniq -c
cat access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head -n 5
- Each stage does exactly one job.
- You can inspect the stream after any stage by not typing the rest of the line.
- That habit is how professionals write these correctly the first time.
An everyday analogy
A factory conveyor belt.
| The factory | The pipeline |
|---|---|
| Raw material at one end | The input file |
| A station that sorts | grep |
| A station that trims | cut |
| A station that groups | sort |
| A station that tallies | uniq -c |
| The finished product | Your answer |
| The reject chute | stderr |
-
No station knows what the others do. Each takes what arrives, does its one job, and passes it on.
-
Adding a step means adding a station, not rebuilding the factory.
-
The reject chute is separate from the belt — which is exactly why error messages do not end up in your results.
Examples in practice
-
grep -c ERROR train.log— how many failures, in one second, on a file too big to open. -
tail -f train.log | grep loss— watch only the loss lines as training runs. -
cut -d',' -f3 data.csv | sort -u | wc -l— how many distinct values does column three have? -
grep -v '^#' config.yml— the config without its comments. -
wc -l *.jsonl— how many examples in each dataset file, and a total.
Implications: security, privacy, performance, scalability, and cost
-
Security:
>destroys silently. There is no prompt and no undo.>>appends; learn the difference before you need it. -
Privacy: grepping a log for a token puts the token in your shell history. So does a password on a command line.
-
Performance: pipelines stream, so memory use stays flat regardless of file size. A 200 GB log processes fine on a laptop.
-
sortis the exception — it must hold data to order it, and will spill to disk on large inputs. -
Cost: this is the cheapest analysis you will ever run. No cluster, no notebook, no data loaded twice.
Alternatives: free, open source, and commercial
| Tool | Type | When to reach for it |
|---|---|---|
grep, sed, awk | Free, built in | Everywhere. Always available. |
ripgrep (rg) | Free, open source | Much faster recursive search; respects .gitignore |
jq | Free, open source | JSON, where line-based tools break down |
csvkit, miller | Free, open source | CSV with quoting and headers done properly |
awk | Free, built in | When a pipeline needs arithmetic or per-line state |
| pandas | Free, open source | When the question needs joins, or you need the result in Python |
-
Reach for
jqthe moment your data is JSON.grepon JSON works until a field contains a newline, and then it silently does not. -
Reach for pandas when you need the answer inside a program, not on a screen.
Comparison with related concepts
| Term | What it actually is |
|---|---|
| Pipe | An in-memory channel joining two running programs |
| Redirect | A connection between a stream and a file |
| Filter | A program that reads stdin and writes stdout |
| Stream | Data that flows, rather than data that is loaded |
| Regular expression | A pattern language for describing text |
When to use it — and when not to
-
Use a pipeline when the question is about lines: how many, which ones, what is most common.
-
Use it when the file is too big to open, or when you are on a server with nothing else installed.
-
Use it to explore before committing to code. Five minutes of pipelines often removes the need for the script entirely.
-
Do not use
grepon JSON or XML. Structure that spans lines defeats line-based tools; usejqinstead. -
Do not build a twelve-stage pipeline you cannot read. At that point it is a script, and it should live in a file with a name.
-
Do not parse CSV with
cutif the fields are quoted. A comma inside quotes will break it, quietly.
The AI connection
-
Dataset inspection starts here. How many lines, how many distinct labels, are there duplicates — all answered before you write any Python.
-
Class imbalance is a one-liner:
cut -f2 labels.tsv | sort | uniq -c | sort -rn. -
Training logs are grepped, not read. Nobody scrolls 400,000 lines looking for the epoch where loss diverged.
-
Deduplicating training data is
sort -u, and it matters more than most people expect — duplicates inflate your metrics. -
Every remote GPU box has these tools and possibly nothing else. This is the toolkit that is always there.
Knowledge check
quiz:
question: "A pipeline counting distinct values reports far more entries than expected, each with a count of 1. What was most likely omitted?"
options:
- "The sort before uniq, since uniq only collapses lines that are already adjacent"
- "The -c flag on uniq, which is what produces the counts"
- "The -n flag on sort, which is needed for numeric ordering"
- "The cut stage, so whole lines are being compared rather than one field"
answer_index: 0
explanation: "uniq compares only neighbouring lines. Without sorting first, identical values scattered through the file never meet, so each is reported separately with a count of one — a wrong answer that looks entirely plausible."
Hands-on exercise
Answer a real question by growing a pipeline. Worked through fully in the Day 10 lab directory.
| What you are doing | Command |
|---|---|
| Look at the shape of the data | head -n 3 access.log |
| Count the lines | wc -l access.log |
| Find the failures | grep -c ' 404 ' access.log |
| Extract one column | cut -d' ' -f1 access.log | head |
| Tally and rank | cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -n 5 |
| Keep the result | ... > top-clients.txt |
Expected output
2000 access.log
143
412 10.0.0.14
198 10.0.0.7
95 192.168.1.10
- 143 lines mention 404, from 2,000 total.
- 10.0.0.14 made 412 requests, more than twice the runner-up.
Validate your work
-
You built the pipeline one stage at a time, checking output at each
| -
You can explain why
sortmust come beforeuniq -
You used
-vto find the lines that did not match -
You redirected a result to a file and confirmed it with
wc -l -
You caused an error mid-pipeline and saw it appear on screen rather than in the data
-
bash tests/run_tests.shreports 0 failures
Troubleshooting
-
uniqreports everything once. You forgot tosortfirst. -
cutreturns the whole line. The delimiter is wrong.-d' 'for spaces,-d','for commas, and tab needs$'\t'. -
grepfinds nothing you can see. Case. Add-i. -
Your file is empty after redirecting. You used
>where you meant>>, or redirected into the same file you were reading.
Common mistakes
-
cat file | grep xwheregrep x filewould do. Harmless, but the pipe is doing nothing. -
Reading
>as “append”. It overwrites. -
Parsing quoted CSV with
cut. It splits on every comma, including the ones inside quotes. -
Building the whole pipeline before testing any of it.
Practice assignment
-
Take any log file on your machine and answer three questions about it using only pipelines: how many lines, which value is most common, and how many are distinct.
-
Rewrite one pipeline three ways and time each with
time. Note which stage dominates. -
Deliberately omit the
sortbeforeuniq -cand record how wrong the answer is. This is the mistake worth making once, on purpose. -
Separate the streams: run a command that produces both output and errors, and send each to a different file.
Extension challenge
-
Process a file larger than your RAM. Generate one with
seq, then answer a question about it with a pipeline. Watch memory use stay flat. -
Find where
sortstops streaming. Increase the input size untilsortspills to disk, and explain why it must, whengrepnever does. -
Rewrite one pipeline in
awk, as a single program, and decide which version you would rather read in six months. -
Compare
grep -rwithrgon a large repository. The ratio is worth knowing before you next search a codebase.
Quiz
Q1. What does the pipe operator `|` do in a command like `cat log | grep 404`?
- It saves the output of `cat` to a temporary file that `grep` then opens
- It connects `cat`'s standard output to `grep`'s standard input
- It runs `cat` and `grep` on two separate files at the same time
- It tells the shell to ignore any errors from `cat`
Show answer
Answer: B. It connects `cat`'s standard output to `grep`'s standard input
A pipe streams one program's standard output straight into the next program's standard input, in memory, with no intermediate file — the core mechanism of composing small tools.
Q2. Which statement about the three standard streams is correct?
- Standard input, output, and error all go to the same place and cannot be separated
- Standard error is where a program reads its input from
- Standard output carries results; standard error carries diagnostics
- Standard output is only used when writing to a file
Show answer
Answer: C. Standard output carries results; standard error carries diagnostics
Results travel on stdout and diagnostics on stderr; keeping them separate is what lets a pipe carry clean data while error messages still reach your screen.
Q3. Why does `uniq` almost always need `sort` before it in a pipeline?
- `uniq` runs faster on sorted input but produces the same result either way
- `uniq` cannot read from a pipe unless `sort` opens the stream first
- `sort` is required to convert the text into a format `uniq` can read
- `uniq` only collapses adjacent duplicates, so sorting must come first
Show answer
Answer: D. `uniq` only collapses adjacent duplicates, so sorting must come first
`uniq` compares each line only with the one before it, so it collapses runs of adjacent duplicates; sorting first groups all identical lines together. `sort -u` does both steps at once.
Q4. What is the key difference between `>` and `|`?
- `>` writes output to a file; `|` streams it into another program
- They are identical; both send output to a file
- `>` appends to a file, while `|` overwrites it
- `|` writes to a file, while `>` sends output to another program
Show answer
Answer: A. `>` writes output to a file; `|` streams it into another program
A redirect (`>`) sends output to a file (overwriting it), while a pipe (`|`) sends output to another running program. `>>` is the append form of the redirect.
Q5. Which pipeline correctly prints the five most common values in the first column of a space-separated file?
- cut -d' ' -f1 file | uniq -c | sort -rn | head -n 5
- cut -d' ' -f1 file | sort | uniq -c | sort -rn | head -n 5
- sort file | cut -d' ' -f1 | uniq -c | head -n 5
- grep -c file | sort -rn | head -n 5
Show answer
Answer: B. cut -d' ' -f1 file | sort | uniq -c | sort -rn | head -n 5
Extract the column with cut, group duplicates with sort, count each run with uniq -c, rank by count with sort -rn, and keep the top five with head — the canonical top-N pattern.
Q6. What does `sed 's/cat/dog/g'` do to each line of its input?
- Deletes every line that contains the word cat
- Replaces only the first occurrence of cat with dog on each line
- Replaces every occurrence of cat with dog on each line
- Permanently rewrites the original file, replacing cat with dog
Show answer
Answer: C. Replaces every occurrence of cat with dog on each line
The trailing `g` (global) makes the substitution replace every match on a line, not just the first; by default sed prints the changed stream and leaves the source file untouched.
Q7. In the log line format `IP - - [time] "GET /path HTTP/1.1" status size`, which command counts requests that returned status 404 by matching the ninth field?
- grep -v 404 access.log | wc -l
- cut -d' ' -f1 access.log | wc -l
- sort access.log | uniq -c
- awk '$9 == 404' access.log | wc -l
Show answer
Answer: D. awk '$9 == 404' access.log | wc -l
awk splits each line into whitespace-separated fields, so `$9 == 404` selects only lines whose status field is 404; piping to `wc -l` counts them. Matching the exact field avoids false hits from a stray 404 elsewhere in the line.
Q8. Which best states the Unix philosophy behind these tools?
- Write small programs that each do one thing well and connect by text streams
- Always store intermediate results in files so each step can be inspected later
- Prefer graphical tools because they are easier for beginners to use
- Build one large program that can perform every text operation you might need
Show answer
Answer: A. Write small programs that each do one thing well and connect by text streams
The philosophy favors small, single-purpose programs composed through a universal text interface — which is exactly why pipes make the tools so powerful in combination.
Glossary
- Unix philosophy
- A design approach that favors small programs each doing one job well, built to work together, communicating through text streams as a universal interface.
- pipe
- The `|` operator, which connects one command's standard output directly to the next command's standard input so data flows between programs without any intermediate file.
- standard input
- The default stream (stdin, file descriptor 0) a program reads from — the keyboard by default, or a file or pipe when redirected.
- standard output
- The default stream (stdout, file descriptor 1) a program writes its results to — the terminal by default, or a file or the next command in a pipe when redirected.
- standard error
- A second output stream (stderr, file descriptor 2) reserved for error and diagnostic messages, kept separate from standard output so errors never contaminate the data flowing through a pipe.
- redirection
- Connecting a program's stream to a file: `>` overwrites a file with standard output, `>>` appends to it, `<` supplies standard input from a file, and `2>` sends standard error to a file.
- filter
- A program that reads a text stream, transforms it, and writes a text stream — such as grep, sed, sort, uniq, cut, or wc — the building block of a pipeline.
- cat
- A command that prints the contents of one or more files to standard output, commonly used to view a short file or feed a file into a pipeline.
- grep
- A command that searches its input line by line and prints the lines matching a pattern; common flags include -i (ignore case), -c (count), -n (line numbers), -v (invert), and -r (recursive).
- sed
- The stream editor: a command that applies editing operations — most often substitution with `s/old/new/` — to a flowing stream of text, printing the result without changing the source file by default.
- regular expression
- A compact notation for describing text patterns, where symbols such as ^ (start of line), $ (end of line), . (any character), and [0-9] (any digit) let a single expression match many strings.
- uniq
- A command that collapses adjacent identical lines into one, and with -c prefixes each with a count of how many times it repeated — usually paired with a preceding sort.
Sources and further reading
- The Missing Semester of Your CS Education — Data Wrangling — MIT (accessed 2026-07-12) — A hands-on lecture on chaining command-line tools to slice and transform text.
- The Linux Command Line — William Shotts (accessed 2026-07-12) — A free book-length introduction to the shell, pipes, redirection, and text tools.
- GNU Grep Manual — GNU Project (accessed 2026-07-12) — The authoritative reference for grep options and regular-expression syntax.
- GNU Sed Manual — GNU Project (accessed 2026-07-12) — The authoritative reference for sed commands, including substitution.
- Pipeline (Unix) — Wikipedia (accessed 2026-07-12) — Overview of Unix pipes, their history, and Douglas McIlroy's role in their creation.
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.