Computing Foundations › The Command Line › Day 10

Hands-on lab — Day 10: Working with Text: cat, grep, sed, and Pipes

← Back to the Day 10 lesson
Open the hands-on files on GitHub — clone or download them from the public labs repository
Local path in your clone: labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed/

Commands

Setup

cd labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed

Run

bash examples/analyze_log.sh
bash starter/analyze_log.sh

Test

bash tests/run_tests.sh

File tree

examples/analyze_log.sh
examples/samples/access.log
expected-output/analyze_log.txt
expected-output/FIELDS.md
expected-output/run_tests.txt
metadata.yml
README.md
requirements/README.md
security.md
starter/analyze_log.sh
starter/text-pipelines-worksheet.md
tests/run_tests.sh
troubleshooting.md

Lab README

Day 010 lab — Pipelines on a Real Log

Lesson

Lesson title: Working with Text: cat, grep, sed, and Pipes
Day number: 10 of 365
Lesson article: https://ai-roadmap-365.github.io/day-010-working-with-text-cat-grep-sed
Lab files: everything you need is in this directory — follow “How to run” below.
Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-010-working-with-text-cat-grep-sed when the site is running.

Purpose

Day 10's lesson teaches the Unix philosophy: small tools, each doing one job, composed with pipes. This lab makes it concrete. You are handed a small web access log and answer real questions about it — how much traffic, from whom, how many errors, over how many pages — by building four pipelines that chain awk, grep, sort, uniq, and wc. This is exactly the shape of data-cleaning and log-analysis work you will do constantly.

Learning objectives

Read a pipeline left to right and name what each stage does.
Extract a single column from structured text with awk.
Count, sort, and de-duplicate lines with sort, uniq -c, and wc.
Build the classic "top-N" pipeline: sort | uniq -c | sort -rn | head.
Filter rows by a field value (all the 404 responses) and count them.
Run an automated test script and interpret its pass/fail output.

Prerequisites

The Day 10 lesson (read it first — it explains every tool this lab uses).
Day 8 (the terminal) and Day 9 (the filesystem) for basic command comfort.
A terminal: Terminal.app (macOS), any terminal (Linux), or WSL (Windows).

Supported operating systems

macOS — fully supported (tested on macOS with Apple Silicon).
Linux — fully supported (any distribution; uses only standard tools).
Windows — use WSL (wsl --install, then Ubuntu) and follow the Linux path. Native PowerShell is not covered — its text tools differ.

Hardware requirements

Any computer made in roughly the last 15 years. The lab reads one ~4 KB text file; it needs no minimum RAM, disk, or GPU.

Required software

bash (3.2 or newer — preinstalled on macOS and Linux).
Standard text tools: cat, grep, sed, awk, sort, uniq, wc, head, tr. All preinstalled on macOS and Linux.

Free and open-source options

Everything here is free and open source or ships with your OS. No account, API key, or purchase is needed. The lesson mentions ripgrep (rg) as a fast, free alternative to grep, but this lab never requires it.

Installation

None. Clone the repository (or copy this directory) and you are ready:

cd labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed

File structure

day-010-working-with-text-cat-grep-sed/
├── README.md                              ← you are here
├── metadata.yml                           ← machine-readable lab metadata
├── starter/
│   ├── analyze_log.sh                     ← YOUR working file (4 exercises)
│   └── text-pipelines-worksheet.md        ← record your findings here
├── examples/
│   ├── analyze_log.sh                     ← completed reference implementation
│   └── samples/
│       └── access.log                     ← the sample web log (40 lines, synthetic)
├── tests/
│   └── run_tests.sh                       ← automated checks (known-correct counts)
├── expected-output/
│   ├── analyze_log.txt                    ← real captured run of the reference script
│   ├── run_tests.txt                      ← real captured test run
│   └── FIELDS.md                          ← the known-correct answers + platform notes
├── requirements/
│   └── README.md                          ← dependency statement (none beyond the OS)
├── troubleshooting.md
└── security.md

How to run

From this directory:

## 1. Peek at the data (first few lines of the log)
head examples/samples/access.log

## 2. See the finished result first
bash examples/analyze_log.sh

## 3. Your task: complete the four pipelines in the starter, then run it
bash starter/analyze_log.sh

## 4. Check your work
bash tests/run_tests.sh

What the commands do

head examples/samples/access.log — prints the first 10 lines so you can see the log's shape: IP - - [timestamp] "METHOD path HTTP/1.1" status size.
bash examples/analyze_log.sh — runs the reference solution. It reports the total request count (wc -l), the top 5 client IPs (awk | sort | uniq -c | sort -rn | head), the number of 404 responses (awk '$9 == 404' | wc -l), and the number of unique paths (awk '{print $7}' | sort -u | wc -l).
bash starter/analyze_log.sh — the same report with four pipelines left for you to build. Each exercise comment names the exact tools to use. Replace every REPLACE_ME.
bash tests/run_tests.sh — recomputes the correct answers from the raw log, checks the reference script prints them, and — once your starter has no REPLACE_ME left — holds your script to the same known-good numbers.

Expected output

Running the reference script prints (see expected-output/analyze_log.txt):

=== Log Analysis Report ===
Log file: examples/samples/access.log
Total requests: 40
Top 5 IP addresses (count  IP):
  10 10.0.0.14
   7 10.0.0.7
   6 10.0.0.99
   5 192.168.1.23
   5 192.168.1.10
404 responses: 7
Unique paths: 13
...
=== End of report ===

The counts are fixed because the sample log is committed and unchanging. expected-output/FIELDS.md lists every known-correct answer and the one place output can differ between macOS and Linux (the order of two IPs that tie at 5 requests each).

Validation steps

Run bash starter/analyze_log.sh — it must exit without errors.
Confirm no line prints REPLACE_ME.
Confirm your numbers match: 40 requests, top IP 10.0.0.14, 7 404s, 13 unique paths.
Fill in starter/text-pipelines-worksheet.md.
Run the tests (next section) — all checks must pass.

Tests

bash tests/run_tests.sh

Expected final line: 16 checks, 0 failure(s). while the starter is still unfinished (the reference script is checked strictly; your starter is checked for structure only until you complete it). Once you have replaced every REPLACE_ME, the same command runs 20 checks, 0 failure(s). The command exits 0 on success and non-zero on any failure, so it can run in CI.

Cleanup

Nothing to clean up: the scripts only read the sample log and print to the console — no files are created or changed. To reset your work, restore the starter from git: git checkout -- starter/analyze_log.sh.

Troubleshooting

See troubleshooting.md for the full list (wrong field number, uniq without sort, 404 miscounts, tie-order differences, WSL notes).

Security notes

See security.md. Short version: the scripts run no network calls, need no elevated privileges, and read only synthetic data invented for this lab — the IPs are private-range addresses that never appear on the public internet, and there is no personal data.

Extension exercises

Status breakdown. Print how many responses had each status code: awk '{print $9}' examples/samples/access.log | sort | uniq -c | sort -rn. How many were successful (200) versus errors (4xx/5xx)?
Busiest path. Adapt the top-IP pipeline to field 7 to find the most requested path instead of the most active IP.
Total bytes served. The response size is field 10. Sum it with awk '{sum += $10} END {print sum}' — one line, no pipe needed.
Substitution with sed. Pipe the unique-paths list through sed 's#^/##' to strip the leading slash from each path, and notice how sed transforms a stream without touching the file.

Previous day: Day 9 — Navigating the Filesystem: Paths, Files, and Permissions (labs/sections/computing-foundations/day-009-navigating-the-filesystem-paths-files-and/).
Next day: Day 11 — Environment Variables and Shell Configuration (labs/sections/computing-foundations/day-011-environment-variables-and-shell-configuration/, to be written).

Expected output

FIELDS.md

# Expected output — Day 010 lab

The sample log (`examples/samples/access.log`) is fixed and committed, so the
numbers below are **exact and reproducible** on every platform.

## Captured runs in this directory

- `analyze_log.txt` — a real run of `bash examples/analyze_log.sh` (macOS,
  2026-07-12). The `Log file:` line is shown as a relative path for
  portability; on your machine it prints the absolute path to the sample.
- `run_tests.txt` — a real run of `bash tests/run_tests.sh` with the starter
  still unfinished (absolute lab path shortened to `<lab>`).

## The known-correct answers

| Question | Answer |
| --- | --- |
| Total requests | 40 |
| Top IP address | `10.0.0.14` |
| Top IP request count | 10 |
| 404 responses | 7 |
| Unique paths | 13 |
| Status 200 responses | 29 |

These are the values `tests/run_tests.sh` asserts. If your pipelines print
anything different, a stage is wrong — re-read the exercise comment.

## Platform notes (macOS vs Linux)

- `wc`, `sort`, `uniq`, `awk`, `grep`, and `sed` ship with both macOS (BSD
  versions) and Linux (GNU versions). Every command in this lab uses only
  options common to both, so the counts are identical.
- **Tie-breaking in the top-IP list is the one place output can differ.** Two
  IPs (`192.168.1.23` and `192.168.1.10`) each appear 5 times. When counts
  tie, BSD `sort` and GNU `sort` may order those two rows differently. This
  never affects the top IP (`10.0.0.14`, 10 hits) or any count — only the
  relative order of the two five-hit rows. The tests check counts and the top
  entry, not the order of tied rows, so both platforms pass.

analyze_log.txt

=== Log Analysis Report ===
Log file: examples/samples/access.log
Total requests: 40
Top 5 IP addresses (count  IP):
  10 10.0.0.14
   7 10.0.0.7
   6 10.0.0.99
   5 192.168.1.23
   5 192.168.1.10
404 responses: 7
Unique paths: 13
The unique paths were:
  - /about.html
  - /api/login
  - /api/orders
  - /api/users
  - /contact.html
  - /favicon.ico
  - /images/hero.jpg
  - /images/logo.png
  - /index.html
  - /missing-page
  - /nowhere
  - /old-home.html
  - /style.css
=== End of report ===

run_tests.txt

Checking the sample log ...
  ok: sample log exists
  ok: sample log has 40 lines
  ok: top IP is 10.0.0.14
  ok: top IP count is 10
  ok: 404 count is 7
  ok: unique paths is 13
Testing <lab>/examples/analyze_log.sh ...
  ok: script exits successfully
  ok: prints report header
  ok: prints report footer
  ok: reports Total requests: 40
  ok: shows top IP 10.0.0.14 with 10
  ok: reports 404 responses: 7
  ok: reports Unique paths: 13
Note: starter/analyze_log.sh still has unfinished exercises (REPLACE_ME) — testing structure only.
Testing <lab>/starter/analyze_log.sh ...
  ok: script exits successfully
  ok: prints report header
  ok: prints report footer

16 checks, 0 failure(s).

Source files

examples/analyze_log.sh (2303 bytes)

#!/usr/bin/env bash
# Day 010 lab — reference solution: analyze a web access log with pipelines.
#
# Every question below is answered by ONE pipeline of small tools connected
# with `|`. Read each pipeline left to right: the output of each command
# becomes the input (stdin) of the next. Nothing here modifies the log file;
# the pipelines only read it and print a report to standard output.
#
# Run from the lab directory:
#   bash examples/analyze_log.sh
set -euo pipefail

# Locate the sample log relative to THIS script, so the command works no
# matter which directory you launch it from.
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
log="${here}/samples/access.log"

echo "=== Log Analysis Report ==="
echo "Log file: ${log}"

# 1. How many requests are in the log? Each line is one request, so counting
#    lines counts requests. `wc -l` counts newlines; `< file` feeds the file
#    in on standard input so wc prints only the number (no filename).
total="$(wc -l < "${log}" | tr -d ' ')"
echo "Total requests: ${total}"

# 2. Which clients (IP addresses) made the most requests?
#    - awk '{print $1}'  -> print field 1 (the IP) of every line
#    - sort              -> group identical IPs next to each other
#    - uniq -c           -> collapse each run of duplicates to "count value"
#    - sort -rn          -> sort by that count, highest first (-r) numerically (-n)
#    - head -n 5         -> keep only the top 5 rows
echo "Top 5 IP addresses (count  IP):"
awk '{print $1}' "${log}" | sort | uniq -c | sort -rn | head -n 5

# 3. How many "404 Not Found" responses were served? In this log the status
#    code is field 9. `awk '$9==404'` prints only matching lines; piping to
#    `wc -l` counts them.
not_found="$(awk '$9 == 404' "${log}" | wc -l | tr -d ' ')"
echo "404 responses: ${not_found}"

# 4. How many DISTINCT paths were requested? Field 7 is the path.
#    `sort -u` sorts and removes duplicates in one step; `wc -l` counts what
#    remains — the number of unique paths.
unique_paths="$(awk '{print $7}' "${log}" | sort -u | wc -l | tr -d ' ')"
echo "Unique paths: ${unique_paths}"

# Bonus: list those unique paths, one per line, alphabetically.
echo "The unique paths were:"
awk '{print $7}' "${log}" | sort -u | sed 's/^/  - /'

echo "=== End of report ==="

examples/samples/access.log (3193 bytes)

10.0.0.14 - - [12/Jul/2026:08:01:12 +0000] "GET /index.html HTTP/1.1" 200 1043
10.0.0.7 - - [12/Jul/2026:08:01:47 +0000] "GET /style.css HTTP/1.1" 200 612
192.168.1.10 - - [12/Jul/2026:08:02:03 +0000] "GET /index.html HTTP/1.1" 200 1043
10.0.0.14 - - [12/Jul/2026:08:02:29 +0000] "GET /images/logo.png HTTP/1.1" 200 8894
192.168.1.23 - - [12/Jul/2026:08:03:11 +0000] "GET /favicon.ico HTTP/1.1" 404 209
10.0.0.99 - - [12/Jul/2026:08:03:52 +0000] "POST /api/login HTTP/1.1" 200 74
10.0.0.14 - - [12/Jul/2026:08:04:18 +0000] "GET /about.html HTTP/1.1" 200 2210
192.168.0.5 - - [12/Jul/2026:08:04:40 +0000] "GET /old-home.html HTTP/1.1" 404 209
10.0.0.7 - - [12/Jul/2026:08:05:07 +0000] "GET /api/users HTTP/1.1" 200 1580
10.0.0.14 - - [12/Jul/2026:08:05:33 +0000] "GET /index.html HTTP/1.1" 304 0
192.168.1.10 - - [12/Jul/2026:08:06:01 +0000] "GET /api/orders HTTP/1.1" 200 4021
10.0.0.42 - - [12/Jul/2026:08:06:44 +0000] "GET /missing-page HTTP/1.1" 404 209
10.0.0.14 - - [12/Jul/2026:08:07:19 +0000] "GET /style.css HTTP/1.1" 200 612
10.0.0.99 - - [12/Jul/2026:08:07:55 +0000] "POST /api/orders HTTP/1.1" 500 133
192.168.1.23 - - [12/Jul/2026:08:08:12 +0000] "GET /index.html HTTP/1.1" 200 1043
10.0.0.7 - - [12/Jul/2026:08:08:39 +0000] "GET /images/logo.png HTTP/1.1" 200 8894
10.0.0.14 - - [12/Jul/2026:08:09:04 +0000] "GET /api/users HTTP/1.1" 200 1580
192.168.0.5 - - [12/Jul/2026:08:09:41 +0000] "GET /about.html HTTP/1.1" 200 2210
10.0.0.7 - - [12/Jul/2026:08:10:08 +0000] "GET /contact.html HTTP/1.1" 301 0
192.168.1.10 - - [12/Jul/2026:08:10:35 +0000] "GET /favicon.ico HTTP/1.1" 404 209
10.0.0.14 - - [12/Jul/2026:08:11:02 +0000] "GET /index.html HTTP/1.1" 200 1043
10.0.0.99 - - [12/Jul/2026:08:11:29 +0000] "GET /api/users HTTP/1.1" 200 1580
192.168.1.23 - - [12/Jul/2026:08:12:00 +0000] "GET /images/hero.jpg HTTP/1.1" 200 40233
10.0.0.7 - - [12/Jul/2026:08:12:31 +0000] "GET /style.css HTTP/1.1" 304 0
10.0.0.42 - - [12/Jul/2026:08:13:03 +0000] "GET /old-home.html HTTP/1.1" 404 209
10.0.0.14 - - [12/Jul/2026:08:13:38 +0000] "GET /api/orders HTTP/1.1" 200 4021
192.168.1.10 - - [12/Jul/2026:08:14:09 +0000] "GET /about.html HTTP/1.1" 200 2210
10.0.0.99 - - [12/Jul/2026:08:14:44 +0000] "POST /api/login HTTP/1.1" 200 74
10.0.0.7 - - [12/Jul/2026:08:15:15 +0000] "GET /index.html HTTP/1.1" 200 1043
192.168.0.5 - - [12/Jul/2026:08:15:50 +0000] "GET /nowhere HTTP/1.1" 404 209
10.0.0.14 - - [12/Jul/2026:08:16:22 +0000] "GET /favicon.ico HTTP/1.1" 200 318
192.168.1.23 - - [12/Jul/2026:08:16:57 +0000] "GET /api/users HTTP/1.1" 200 1580
10.0.0.99 - - [12/Jul/2026:08:17:26 +0000] "GET /style.css HTTP/1.1" 200 612
192.168.1.10 - - [12/Jul/2026:08:17:58 +0000] "GET /index.html HTTP/1.1" 200 1043
10.0.0.14 - - [12/Jul/2026:08:18:30 +0000] "GET /contact.html HTTP/1.1" 200 1876
10.0.0.7 - - [12/Jul/2026:08:19:03 +0000] "GET /api/orders HTTP/1.1" 200 4021
192.168.0.5 - - [12/Jul/2026:08:19:35 +0000] "GET /about.html HTTP/1.1" 200 2210
10.0.0.42 - - [12/Jul/2026:08:20:11 +0000] "GET /images/logo.png HTTP/1.1" 200 8894
192.168.1.23 - - [12/Jul/2026:08:20:48 +0000] "GET /favicon.ico HTTP/1.1" 404 209
10.0.0.99 - - [12/Jul/2026:08:21:20 +0000] "GET /index.html HTTP/1.1" 200 1043

metadata.yml (564 bytes)

lesson_id: D010
day: 10
kind: data-processing
languages: [bash]
setup_commands:
  - cd labs/sections/computing-foundations/day-010-working-with-text-cat-grep-sed
run_commands:
  - bash examples/analyze_log.sh
  - bash starter/analyze_log.sh
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - 'git checkout -- starter/analyze_log.sh  # optional: reset your work'
requires_network: false
requires_api_key: false
estimated_minutes: 30
last_executed: '2026-07-12'
executed_on: 'macOS (Apple Silicon), bash tests/run_tests.sh → 16 checks, 0 failure(s)'

requirements/README.md (1177 bytes)

# Dependencies — Day 010 lab

**None beyond a POSIX shell and the standard text tools.** Everything this lab
uses is preinstalled on macOS and every mainstream Linux distribution:

- `bash` ≥ 3.2 — to run the scripts.
- `cat`, `head`, `tail`, `wc` — viewing and counting text.
- `grep` — searching lines by pattern.
- `sed` — stream editing and substitution.
- `awk` — field extraction (pulling out the IP, path, and status columns).
- `sort`, `uniq`, `cut`, `tr` — ordering, de-duplicating, and trimming.

All of these are part of the base system (GNU coreutils / util-linux on
Linux, the BSD equivalents on macOS). There is deliberately no
`requirements.txt` or `package.json`: this lab must run on a factory-fresh
machine with nothing installed.

## Optional (not required)

- `ripgrep` (`rg`) is a fast, free, open-source alternative to `grep`. The
  lesson mentions it, but this lab never depends on it — every exercise uses
  the always-present `grep`.

## Windows

Use **WSL** (Windows Subsystem for Linux): `wsl --install`, open Ubuntu, and
run the lab exactly as written. The native PowerShell shell has different text
tools and is not covered here.

starter/analyze_log.sh (2653 bytes)

#!/usr/bin/env bash
# Day 010 lab — YOUR working file. Build four pipelines that analyze the
# sample web access log, then fill in text-pipelines-worksheet.md with the
# answers your script prints.
#
# Each exercise below names the EXACT tools to use. Replace the placeholder
# after each `=` (or the `echo` line) with a single pipeline. When a value is
# captured into a variable, wrap the pipeline in "$( ... )".
#
# Run it as you go to see your progress:
#   bash starter/analyze_log.sh
# When every exercise is done, check your work:
#   bash tests/run_tests.sh
set -euo pipefail

# The sample log lives beside the reference script, in examples/samples/.
here="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
log="${here}/examples/samples/access.log"

echo "=== Log Analysis Report ==="
echo "Log file: ${log}"

# ---------------------------------------------------------------------------
# Exercise 1 — TOTAL REQUESTS.
# Each line of the log is one request. Count the lines.
# Tools: wc -l  (feed the file on stdin with `< "${log}"` so wc prints only
#        the number). Wrap the pipeline in "$( ... )" and trim spaces with
#        `| tr -d ' '`.
# Replace the word REPLACE_ME below.
total="REPLACE_ME"
echo "Total requests: ${total}"

# ---------------------------------------------------------------------------
# Exercise 2 — TOP 5 IP ADDRESSES.
# Field 1 of each line is the client IP. Build this pipeline:
#   awk '{print $1}' "${log}" | sort | uniq -c | sort -rn | head -n 5
# (awk prints the IP column, sort groups duplicates, uniq -c counts each run,
#  sort -rn ranks by count highest-first, head keeps the top 5.)
# Replace the echo line below with that pipeline.
echo "Top 5 IP addresses (count  IP):"
echo "REPLACE_ME: build the awk | sort | uniq -c | sort -rn | head pipeline"

# ---------------------------------------------------------------------------
# Exercise 3 — NUMBER OF 404s.
# The HTTP status code is field 9. Print only the lines whose status is 404,
# then count them.
# Tools: awk '$9 == 404' "${log}" | wc -l   (trim spaces with tr -d ' ').
# Replace the word REPLACE_ME below.
not_found="REPLACE_ME"
echo "404 responses: ${not_found}"

# ---------------------------------------------------------------------------
# Exercise 4 — COUNT OF UNIQUE PATHS.
# Field 7 is the requested path. Count how many DISTINCT paths appear.
# Tools: awk '{print $7}' "${log}" | sort -u | wc -l   (sort -u removes
#        duplicates; wc -l counts what is left). Trim spaces with tr -d ' '.
# Replace the word REPLACE_ME below.
unique_paths="REPLACE_ME"
echo "Unique paths: ${unique_paths}"

echo "=== End of report ==="

starter/text-pipelines-worksheet.md (1215 bytes)

# Text-pipelines worksheet — Day 010

Run your completed `starter/analyze_log.sh` against the sample log and record
the answers below. Each answer comes from one of the four pipelines you built.

## Your findings

| Question | Your answer | The pipeline you used |
| --- | --- | --- |
| 1. Total requests in the log | _____ | `wc -l < …` |
| 2. Top IP address (most requests) | _____ | `awk '{print $1}' … \| sort \| uniq -c \| sort -rn \| head` |
| 3. That top IP's request count | _____ | (same pipeline as row 2 — read the first column) |
| 4. Number of 404 responses | _____ | `awk '$9 == 404' … \| wc -l` |
| 5. Count of unique paths requested | _____ | `awk '{print $7}' … \| sort -u \| wc -l` |

## One thing you noticed

Write one or two sentences about a pattern in the log. For example: which
paths returned 404, whether one client dominated the traffic, or how many
responses were successful (status 200) versus errors.

_Your notes:_

## Check yourself

- Did every pipeline read the file without changing it? (Re-run
  `wc -l < examples/samples/access.log` — it should still report the same
  number of lines.)
- Could you rebuild each pipeline from memory, naming what every stage does?

tests/run_tests.sh (4423 bytes)

#!/usr/bin/env bash
# Tests for the Day 010 lab. Run from the lab directory:
#   bash tests/run_tests.sh
#
# Strategy: the counts in the committed sample log are FIXED and known, so we
# assert the reference script prints exactly those numbers. Then we run the
# learner's starter script; once they have replaced every REPLACE_ME, we hold
# it to the same known-good counts.
set -u

lab_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
log="${lab_dir}/examples/samples/access.log"
failures=0
checks=0

# Known-correct answers for the committed sample log (examples/samples/access.log).
EXPECT_TOTAL=40
EXPECT_TOP_IP="10.0.0.14"
EXPECT_TOP_COUNT=10
EXPECT_404=7
EXPECT_UNIQUE_PATHS=13

check() {
  local label="$1" ok="$2"
  checks=$((checks + 1))
  if [ "${ok}" = "yes" ]; then
    echo "  ok: ${label}"
  else
    echo "  FAIL: ${label}"
    failures=$((failures + 1))
  fi
}

# --- Sanity: the sample data is present and unmodified in shape. --------------
echo "Checking the sample log ..."
if [ -f "${log}" ]; then
  check "sample log exists" "yes"
else
  check "sample log exists" "no"
  echo "${checks} checks, ${failures} failure(s)."
  exit 1
fi
lines="$(wc -l < "${log}" | tr -d ' ')"
[ "${lines}" = "${EXPECT_TOTAL}" ] && check "sample log has ${EXPECT_TOTAL} lines" "yes" || check "sample log has ${EXPECT_TOTAL} lines (got ${lines})" "no"

# --- Independent recomputation of the ground truth from the raw log. ----------
# The tests do NOT trust the scripts; they recompute the answers here and
# compare against the fixed EXPECT_* values, then compare the scripts too.
top_ip="$(awk '{print $1}' "${log}" | sort | uniq -c | sort -rn | head -n 1 | awk '{print $2}')"
top_count="$(awk '{print $1}' "${log}" | sort | uniq -c | sort -rn | head -n 1 | awk '{print $1}')"
count_404="$(awk '$9 == 404' "${log}" | wc -l | tr -d ' ')"
unique_paths="$(awk '{print $7}' "${log}" | sort -u | wc -l | tr -d ' ')"

check "top IP is ${EXPECT_TOP_IP}" "$([ "${top_ip}" = "${EXPECT_TOP_IP}" ] && echo yes || echo no)"
check "top IP count is ${EXPECT_TOP_COUNT}" "$([ "${top_count}" = "${EXPECT_TOP_COUNT}" ] && echo yes || echo no)"
check "404 count is ${EXPECT_404}" "$([ "${count_404}" = "${EXPECT_404}" ] && echo yes || echo no)"
check "unique paths is ${EXPECT_UNIQUE_PATHS}" "$([ "${unique_paths}" = "${EXPECT_UNIQUE_PATHS}" ] && echo yes || echo no)"

# --- The reference script must report the same known-good numbers. -----------
run_report_checks() {
  local script="$1" strict="$2" output
  echo "Testing ${script} ..."
  if ! output="$(bash "${script}" 2>&1)"; then
    check "script exits successfully" "no"
    echo "${output}" | sed 's/^/    /'
    return
  fi
  check "script exits successfully" "yes"
  echo "${output}" | grep -q '^=== Log Analysis Report ===$' && check "prints report header" "yes" || check "prints report header" "no"
  echo "${output}" | grep -q '^=== End of report ===$' && check "prints report footer" "yes" || check "prints report footer" "no"

  if [ "${strict}" = "strict" ]; then
    echo "${output}" | grep -q "^Total requests: ${EXPECT_TOTAL}$" && check "reports Total requests: ${EXPECT_TOTAL}" "yes" || check "reports Total requests: ${EXPECT_TOTAL}" "no"
    # The top IP line is "  10 10.0.0.14" (leading spaces from uniq -c).
    echo "${output}" | grep -Eq "^[[:space:]]*${EXPECT_TOP_COUNT}[[:space:]]+${EXPECT_TOP_IP}$" && check "shows top IP ${EXPECT_TOP_IP} with ${EXPECT_TOP_COUNT}" "yes" || check "shows top IP ${EXPECT_TOP_IP} with ${EXPECT_TOP_COUNT}" "no"
    echo "${output}" | grep -q "^404 responses: ${EXPECT_404}$" && check "reports 404 responses: ${EXPECT_404}" "yes" || check "reports 404 responses: ${EXPECT_404}" "no"
    echo "${output}" | grep -q "^Unique paths: ${EXPECT_UNIQUE_PATHS}$" && check "reports Unique paths: ${EXPECT_UNIQUE_PATHS}" "yes" || check "reports Unique paths: ${EXPECT_UNIQUE_PATHS}" "no"
  fi
}

run_report_checks "${lab_dir}/examples/analyze_log.sh" strict

# --- The learner's starter: structure-only until they finish the exercises. --
if grep -q 'REPLACE_ME' "${lab_dir}/starter/analyze_log.sh"; then
  echo "Note: starter/analyze_log.sh still has unfinished exercises (REPLACE_ME) — testing structure only."
  run_report_checks "${lab_dir}/starter/analyze_log.sh" lenient
else
  run_report_checks "${lab_dir}/starter/analyze_log.sh" strict
fi

echo
echo "${checks} checks, ${failures} failure(s)."
[ "${failures}" -eq 0 ]

Troubleshooting

Troubleshooting — Day 010 lab

`No such file or directory` for the log

The scripts find the log relative to their own location, so run them from the lab directory as shown in the README (bash examples/analyze_log.sh). If you copied a pipeline out on its own, point it at the real path: examples/samples/access.log. Confirm the file is there with ls examples/samples/.

`awk: syntax error` or the wrong column comes out

awk '{print $1}' prints field 1; the fields are separated by spaces. In this log field 1 is the IP, field 7 is the path (e.g. /index.html), and field 9 is the status code (e.g. 200). If you get the wrong value, you probably used the wrong field number — recount from the left, starting at 1.

My 404 count is 0 (or far too high)

Two common causes:

You matched the wrong field. Use awk '$9 == 404' — the $9 == 404 compares the status field to the number 404.
You used grep 404 without spaces and matched a byte size or timestamp that happens to contain "404". Anchor to the status column instead, or use grep ' 404 ' (with surrounding spaces) as a quick approximation.

`uniq` didn't remove duplicates

uniq only collapses adjacent identical lines, so you must sort first. The correct order is always sort | uniq (or sort | uniq -c to count). sort -u does both in one step.

The top-IP rows come out in a different order than the sample

Two IPs tie at 5 requests each. sort on macOS (BSD) and Linux (GNU) may order tied rows differently. This is expected and harmless — the top IP and every count are still identical, and the tests only check those. See expected-output/FIELDS.md.

`Permission denied` when running a script

Run it through bash explicitly: bash tests/run_tests.sh. You do not need to chmod +x anything. If you prefer ./tests/run_tests.sh, first run chmod +x tests/run_tests.sh.

`bash: command not found` on Windows

Use WSL (wsl --install, then open Ubuntu). PowerShell does not have these Unix text tools.

Security notes

Security notes — Day 010 lab

What the scripts do: read one small text file (examples/samples/access.log) and print counts to the console. They make no network connections, write no files, and change no settings. Every pipeline is read-only.
The sample data is synthetic. The log is invented for this lab. Its IP addresses are drawn from the private ranges reserved by RFC 1918 (10.0.0.0/8 and 192.168.0.0/16) — addresses that never appear on the public internet — and it contains no names, emails, cookies, or credentials. Nothing in it identifies a real person or machine.
Privileges: everything runs as your normal user. No step needs sudo. If any tutorial ever tells you to sudo a script you have not read, stop and read it first — a habit this course reinforces.
Reading before running: both scripts are short and commented. Read them before executing. Running unread shell scripts is a common way developers get compromised; every lab script here is small enough to read in full first.
Handling real logs later: genuine web logs do contain personal data — IP addresses are often personal data under privacy law, and logs may hold session tokens or user IDs. When you apply these pipelines to real data, treat the log as sensitive: do not paste it into public forums, and strip or hash identifying fields before sharing.

Hands-on lab — Day 10: Working with Text: cat, grep, sed, and Pipes

Commands

Setup

Run

Test

File tree

Lab README

Day 010 lab — Pipelines on a Real Log

Lesson

Purpose

Learning objectives

Prerequisites

Supported operating systems

Hardware requirements

Required software

Free and open-source options

Installation

File structure

How to run

What the commands do

Expected output

Validation steps

Tests

Cleanup

Troubleshooting

Security notes

Extension exercises

Navigation

Expected output

FIELDS.md

analyze_log.txt

run_tests.txt

Source files

Troubleshooting

Troubleshooting — Day 010 lab

No such file or directory for the log

awk: syntax error or the wrong column comes out

My 404 count is 0 (or far too high)

uniq didn't remove duplicates

The top-IP rows come out in a different order than the sample

Permission denied when running a script

bash: command not found on Windows

Security notes

Security notes — Day 010 lab

`No such file or directory` for the log

`awk: syntax error` or the wrong column comes out

`uniq` didn't remove duplicates

`Permission denied` when running a script

`bash: command not found` on Windows