Computing Foundations › The Command Line › Day 10

Day 10: Working with Text: cat, grep, sed, and Pipes

Day 10 of 365 — Working with Text: cat, grep, sed, and Pipes

After this lesson you will be able to compose small command-line tools into pipelines that search, filter, transform, and summarize text — and use them to inspect and clean the kind of real data you will feed to machine-learning workflows.

Course
Computing Foundations
Category
The Command Line
Reading time
≈ 40 min
Practical time
≈ 30 min
Lesson duration
1h 10m
Last verified
2026-07-12

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters


The idea in plain language

Diagram: a monolithic tool compared with composable small tools


Historical background


What it is — and what it is not

Architecture diagram: a program's three standard streams — stdin, stdout, and stderr — and the pipes and redirects that connect them

It is not:


Why it was created and what problems it solves

Diagram: redirection to a file compared with a pipe between programs

OperatorWhat it doesExample
|stdout of the left into stdin of the rightcat log | grep 404
>Redirect stdout to a file, overwritinggrep 404 log > errors.txt
>>Redirect stdout to a file, appendinggrep 500 log >> errors.txt
<Take stdin from a filewc -l < log
2>Redirect stderr to a filesort huge 2> problems.txt
2>&1Send stderr wherever stdout is going./run 2>&1 | less

How it works

Viewing text

Diagram: text flowing through cat, grep, sort and uniq in a pipeline

Searching with grep

Diagram: the most useful grep flags and what each does

Transforming

Building a pipeline

The question: which client addresses hit our server most often?

cat access.log
cat access.log | cut -d' ' -f1
cat access.log | cut -d' ' -f1 | sort
cat access.log | cut -d' ' -f1 | sort | uniq -c
cat access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head -n 5

Flow diagram: text flowing through a cat, grep, sort, and uniq pipeline connected by pipes


An everyday analogy

Diagram: which text tool suits which data shape

A factory conveyor belt.

The factoryThe pipeline
Raw material at one endThe input file
A station that sortsgrep
A station that trimscut
A station that groupssort
A station that talliesuniq -c
The finished productYour answer
The reject chutestderr

Examples in practice


Implications: security, privacy, performance, scalability, and cost

Diagram: the sort, uniq and sort pipeline that tallies and ranks values


Alternatives: free, open source, and commercial

Diagram: streaming a file compared with loading it into memory

ToolTypeWhen to reach for it
grep, sed, awkFree, built inEverywhere. Always available.
ripgrep (rg)Free, open sourceMuch faster recursive search; respects .gitignore
jqFree, open sourceJSON, where line-based tools break down
csvkit, millerFree, open sourceCSV with quoting and headers done properly
awkFree, built inWhen a pipeline needs arithmetic or per-line state
pandasFree, open sourceWhen the question needs joins, or you need the result in Python

Diagram: standard output compared with standard error

TermWhat it actually is
PipeAn in-memory channel joining two running programs
RedirectA connection between a stream and a file
FilterA program that reads stdin and writes stdout
StreamData that flows, rather than data that is loaded
Regular expressionA pattern language for describing text

When to use it — and when not to


The AI connection

Diagram: growing a pipeline one stage at a time and checking the output


Knowledge check

quiz:
  question: "A pipeline counting distinct values reports far more entries than expected, each with a count of 1. What was most likely omitted?"
  options:
    - "The sort before uniq, since uniq only collapses lines that are already adjacent"
    - "The -c flag on uniq, which is what produces the counts"
    - "The -n flag on sort, which is needed for numeric ordering"
    - "The cut stage, so whole lines are being compared rather than one field"
  answer_index: 0
  explanation: "uniq compares only neighbouring lines. Without sorting first, identical values scattered through the file never meet, so each is reported separately with a count of one — a wrong answer that looks entirely plausible."

Hands-on exercise

Answer a real question by growing a pipeline. Worked through fully in the Day 10 lab directory.

What you are doingCommand
Look at the shape of the datahead -n 3 access.log
Count the lineswc -l access.log
Find the failuresgrep -c ' 404 ' access.log
Extract one columncut -d' ' -f1 access.log | head
Tally and rankcut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -n 5
Keep the result... > top-clients.txt

Expected output

    2000 access.log
     143
     412 10.0.0.14
     198 10.0.0.7
      95 192.168.1.10

Validate your work

Troubleshooting

Common mistakes


Practice assignment


Extension challenge


Quiz

Q1. What does the pipe operator `|` do in a command like `cat log | grep 404`?

  1. It saves the output of `cat` to a temporary file that `grep` then opens
  2. It connects `cat`'s standard output to `grep`'s standard input
  3. It runs `cat` and `grep` on two separate files at the same time
  4. It tells the shell to ignore any errors from `cat`
Show answer

Answer: B. It connects `cat`'s standard output to `grep`'s standard input

A pipe streams one program's standard output straight into the next program's standard input, in memory, with no intermediate file — the core mechanism of composing small tools.

Q2. Which statement about the three standard streams is correct?

  1. Standard input, output, and error all go to the same place and cannot be separated
  2. Standard error is where a program reads its input from
  3. Standard output carries results; standard error carries diagnostics
  4. Standard output is only used when writing to a file
Show answer

Answer: C. Standard output carries results; standard error carries diagnostics

Results travel on stdout and diagnostics on stderr; keeping them separate is what lets a pipe carry clean data while error messages still reach your screen.

Q3. Why does `uniq` almost always need `sort` before it in a pipeline?

  1. `uniq` runs faster on sorted input but produces the same result either way
  2. `uniq` cannot read from a pipe unless `sort` opens the stream first
  3. `sort` is required to convert the text into a format `uniq` can read
  4. `uniq` only collapses adjacent duplicates, so sorting must come first
Show answer

Answer: D. `uniq` only collapses adjacent duplicates, so sorting must come first

`uniq` compares each line only with the one before it, so it collapses runs of adjacent duplicates; sorting first groups all identical lines together. `sort -u` does both steps at once.

Q4. What is the key difference between `>` and `|`?

  1. `>` writes output to a file; `|` streams it into another program
  2. They are identical; both send output to a file
  3. `>` appends to a file, while `|` overwrites it
  4. `|` writes to a file, while `>` sends output to another program
Show answer

Answer: A. `>` writes output to a file; `|` streams it into another program

A redirect (`>`) sends output to a file (overwriting it), while a pipe (`|`) sends output to another running program. `>>` is the append form of the redirect.

Q5. Which pipeline correctly prints the five most common values in the first column of a space-separated file?

  1. cut -d' ' -f1 file | uniq -c | sort -rn | head -n 5
  2. cut -d' ' -f1 file | sort | uniq -c | sort -rn | head -n 5
  3. sort file | cut -d' ' -f1 | uniq -c | head -n 5
  4. grep -c file | sort -rn | head -n 5
Show answer

Answer: B. cut -d' ' -f1 file | sort | uniq -c | sort -rn | head -n 5

Extract the column with cut, group duplicates with sort, count each run with uniq -c, rank by count with sort -rn, and keep the top five with head — the canonical top-N pattern.

Q6. What does `sed 's/cat/dog/g'` do to each line of its input?

  1. Deletes every line that contains the word cat
  2. Replaces only the first occurrence of cat with dog on each line
  3. Replaces every occurrence of cat with dog on each line
  4. Permanently rewrites the original file, replacing cat with dog
Show answer

Answer: C. Replaces every occurrence of cat with dog on each line

The trailing `g` (global) makes the substitution replace every match on a line, not just the first; by default sed prints the changed stream and leaves the source file untouched.

Q7. In the log line format `IP - - [time] "GET /path HTTP/1.1" status size`, which command counts requests that returned status 404 by matching the ninth field?

  1. grep -v 404 access.log | wc -l
  2. cut -d' ' -f1 access.log | wc -l
  3. sort access.log | uniq -c
  4. awk '$9 == 404' access.log | wc -l
Show answer

Answer: D. awk '$9 == 404' access.log | wc -l

awk splits each line into whitespace-separated fields, so `$9 == 404` selects only lines whose status field is 404; piping to `wc -l` counts them. Matching the exact field avoids false hits from a stray 404 elsewhere in the line.

Q8. Which best states the Unix philosophy behind these tools?

  1. Write small programs that each do one thing well and connect by text streams
  2. Always store intermediate results in files so each step can be inspected later
  3. Prefer graphical tools because they are easier for beginners to use
  4. Build one large program that can perform every text operation you might need
Show answer

Answer: A. Write small programs that each do one thing well and connect by text streams

The philosophy favors small, single-purpose programs composed through a universal text interface — which is exactly why pipes make the tools so powerful in combination.

Glossary

Unix philosophy
A design approach that favors small programs each doing one job well, built to work together, communicating through text streams as a universal interface.
pipe
The `|` operator, which connects one command's standard output directly to the next command's standard input so data flows between programs without any intermediate file.
standard input
The default stream (stdin, file descriptor 0) a program reads from — the keyboard by default, or a file or pipe when redirected.
standard output
The default stream (stdout, file descriptor 1) a program writes its results to — the terminal by default, or a file or the next command in a pipe when redirected.
standard error
A second output stream (stderr, file descriptor 2) reserved for error and diagnostic messages, kept separate from standard output so errors never contaminate the data flowing through a pipe.
redirection
Connecting a program's stream to a file: `>` overwrites a file with standard output, `>>` appends to it, `<` supplies standard input from a file, and `2>` sends standard error to a file.
filter
A program that reads a text stream, transforms it, and writes a text stream — such as grep, sed, sort, uniq, cut, or wc — the building block of a pipeline.
cat
A command that prints the contents of one or more files to standard output, commonly used to view a short file or feed a file into a pipeline.
grep
A command that searches its input line by line and prints the lines matching a pattern; common flags include -i (ignore case), -c (count), -n (line numbers), -v (invert), and -r (recursive).
sed
The stream editor: a command that applies editing operations — most often substitution with `s/old/new/` — to a flowing stream of text, printing the result without changing the source file by default.
regular expression
A compact notation for describing text patterns, where symbols such as ^ (start of line), $ (end of line), . (any character), and [0-9] (any digit) let a single expression match many strings.
uniq
A command that collapses adjacent identical lines into one, and with -c prefixes each with a count of how many times it repeated — usually paired with a preceding sort.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.