Computing Foundations › APIs and the Web › Day 27
Hands-on lab — Day 27: Rate Limits, Pagination, and Error Handling
- ← Back to the Day 27 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/computing-foundations/day-027-rate-limits-pagination-and-error-handling/
Commands
Setup
cd labs/sections/computing-foundations/day-027-rate-limits-pagination-and-error-handling Run
bash examples/resilient_client.sh
bash starter/resilient_client.sh Test
bash tests/run_tests.sh File tree
examples/resilient_client.sh expected-output/FIELDS.md expected-output/sample-run.txt metadata.yml README.md requirements/README.md security.md starter/resilience-worksheet.md starter/resilient_client.sh tests/run_tests.sh troubleshooting.md
Lab README
Day 027 lab — Handle Limits and Pages
Build and run a small resilient API client in shell against free public test
servers: back off from a 429, decide correctly which errors to retry, and
paginate across pages — collecting results without missing or repeating a row.
Lesson
- Lesson title: Rate Limits, Pagination, and Error Handling
- Day number: 27 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-027-rate-limits-pagination-and-error-handling
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-027-rate-limits-pagination-and-error-handlingwhen the site is running.
Purpose
Day 27's lesson explains the three habits of consuming a real API robustly. This lab makes them concrete and executable: you run a client that survives rate limits and server hiccups, classifies failures, and walks a paginated endpoint — then you complete a starter version yourself. Every request hits a live, free, key-less test server, so the numbers are real. This is Day 27 of the course.
Learning objectives
- Trigger a
429 Too Many Requestsresponse and read the status code back. - Watch an exponential-backoff-with-jitter loop retry and then give up gracefully — proving it terminates rather than looping forever.
- Classify a
500(retry) apart from a404(do not retry) and say why. - Paginate an endpoint with
_page/_limitand collect results across pages. - Run an automated test suite that verifies termination and the collected count.
Prerequisites
- The Day 27 lesson (read it first — it explains every concept this lab runs).
- Days 18-24: HTTP status codes and JSON.
- A terminal with
bash,curl, andpython3. Internet access for the live parts (offline is supported and degrades gracefully).
Supported operating systems
- macOS — fully supported (tested on macOS with curl 8.7.1).
- Linux — fully supported (any distribution with
bash,curl,python3). - Windows — use WSL and follow the Linux instructions.
Hardware requirements
Any computer from roughly the last 15 years. The lab only makes small HTTP requests; it needs no particular RAM, disk, or GPU.
Required software
bash3.2+,curl, andpython3(used only to count JSON items).- Standard utilities
awk,sed,grep,mktemp— all preinstalled. - See
requirements/README.mdfor one-step install commands per platform.
Free and open-source options
Everything here is free and needs no account or API key. curl and python3
are open source; the test servers (httpbin.org, jsonplaceholder.typicode.com)
are free public services. There is no paid component anywhere in this lab.
Installation
None beyond the tools above. From the repository root:
cd labs/sections/computing-foundations/day-027-rate-limits-pagination-and-error-handling
File structure
day-027-rate-limits-pagination-and-error-handling/
├── README.md ← you are here
├── metadata.yml ← machine-readable lab metadata
├── starter/
│ ├── resilient_client.sh ← YOUR working file (4 exercises)
│ └── resilience-worksheet.md ← record your real numbers here
├── examples/
│ └── resilient_client.sh ← completed reference implementation
├── tests/
│ └── run_tests.sh ← automated checks (with a watchdog)
├── expected-output/
│ ├── sample-run.txt ← a real captured run (online + offline)
│ └── FIELDS.md ← required lines on every platform
├── requirements/
│ └── README.md ← dependency statement
├── troubleshooting.md
└── security.md
How to run
From this directory:
## 1. See the finished client first (online: live demos; offline: self-test)
bash examples/resilient_client.sh
## 2. Your task: complete the four exercises in the starter, then run it
bash starter/resilient_client.sh
## 3. Check your work
bash tests/run_tests.sh
To move faster while experimenting, shrink the delays:
BASE_DELAY=0 DELAY_CAP=0 bash examples/resilient_client.sh
What the commands do
bash examples/resilient_client.sh— runs the reference client: a backoff-retry loop againsthttpbin.org/status/429that retries with growing, jittered delays up to a cap and then gives up gracefully (honoringRetry-Afterwhen present); adecidestep that reads a500and a404and states which to retry; and a paginator that walks two pages ofjsonplaceholder.typicode.com/postswith_page/_limitand sums the items. Offline, it runs a no-network self-test of the backoff logic and exits 0.bash starter/resilient_client.sh— the same skeleton with four spots blanked out (markedFILL_ME); each numbered exercise names the exact change to make.bash tests/run_tests.sh— structure checks, plus a watchdog-guarded run that proves the backoff terminates and the pagination collects the expected count.
Expected output
See expected-output/sample-run.txt for a real
captured run (online and offline). The key lines: the backoff prints several
attempt N: HTTP ... lines and exactly one giving up gracefully; the decide
step marks /status/500 as RETRY and /status/404 as do NOT retry; and the
paginator prints collected 10 posts across 2 pages. Your exact statuses, jitter
values, and retry counts will differ — that is the nature of a real API.
Validation steps
- Run
bash examples/resilient_client.sh— it must finish on its own (never hang) and end withDone.(online) or the offline self-test message. - Confirm the backoff prints exactly one
giving up gracefullyline. - Confirm
/status/500is classified as retry and/status/404as do-not-retry. - Confirm the paginator reports the total it collected across two pages.
- Complete the four exercises in the starter and record your numbers on the worksheet, then run the tests below.
Tests
bash tests/run_tests.sh
Expected final line: 17 checks, 0 failure(s), 0 skip(s). online, or with some
skip(s) offline. Each script run is wrapped in a watchdog that kills and
fails any run that does not terminate — so a passing suite is proof the retry
loop cannot spin forever. The command exits 0 on success, non-zero on any
failure, so it can run in CI.
Cleanup
Nothing to clean up: the scripts make read-only HTTP requests and write only a
short-lived temp file that they delete. To reset your work, restore the starter
from git: git checkout -- starter/resilient_client.sh.
Troubleshooting
See troubleshooting.md. Short version: transient 503/504
or timeouts from the shared test server are expected and are exactly what
your client is built to survive — re-run in a minute for a cleaner picture.
Security notes
See security.md. Short version: be a polite client — honor rate limits, add jitter, and cap your retries. Hammering an API can get your key or IP blocked, and an uncapped retry storm is indistinguishable from an attack.
Extension exercises
- Make the client honor
Retry-After: fetchhttps://httpbin.org/response-headers?Retry-After=3, read the header, and wait exactly that long instead of using the doubling delay. - Add a third page to the paginator and confirm the collected total rises accordingly; stop when a page returns zero items.
- Add an idempotency check: print a warning before retrying anything that is not
a
GET, since non-idempotent writes are unsafe to repeat blindly.
Navigation
- Previous day: Day 26 — Webhooks and Event-Driven APIs
(
labs/sections/computing-foundations/day-026-webhooks-and-event-driven-apis/). - Next day: Day 28 — Consuming a Public API from the Command Line
(
labs/sections/computing-foundations/day-028-consuming-a-public-api-from-the/).
Expected output
FIELDS.md
# Required fields in a run of `resilient_client.sh`
These lines must appear on every platform (macOS or Linux) and in both network
states. The exact status codes, jitter values, and retry counts vary run to
run — that variability is the point — but the *structure* is fixed.
## Online (network reachable)
| Section | Must contain |
| --- | --- |
| Header | `Day 027 — Resilient API client` |
| 1. Backoff | one or more `attempt N: HTTP ...` lines |
| 1. Backoff | exactly one `giving up gracefully` line (the loop MUST terminate) |
| 1. Backoff | a `made N retries before giving up` summary |
| 2. Decide | a `/status/500` line ending in `RETRY with backoff` |
| 2. Decide | a `/status/404` line ending in `do NOT retry` |
| 3. Paginate | two `page N: M items` lines |
| 3. Paginate | a `collected T posts across 2 pages` line (T is the sum) |
| Footer | `Done.` |
## Offline (network unreachable) or `--selftest`
| Must contain |
| --- |
| `OFFLINE:` (only in the auto-detected offline path) or `Self-test:` |
| a stubbed `attempt N: HTTP 503` sequence |
| exactly one `giving up gracefully` line — proof the loop terminates without a network |
| `loop terminated` |
## Platform notes
- The script needs only `bash`, `curl`, `awk`, `sed`, and (for the item count)
`python3`. All ship with macOS and are one package install away on Linux.
- The public test servers (`httpbin.org`, `jsonplaceholder.typicode.com`) are
shared and occasionally return transient 5xx or time out. The client is
built to survive exactly that, so a run during a hiccup is still a valid run.
sample-run.txt
Real captured run of `bash examples/resilient_client.sh` on macOS (curl 8.7.1),
online. Your exact numbers, jitter values, and the transient statuses from the
public test server will differ — that is the nature of a real API.
Note: during this capture, https://httpbin.org was under load and its gateway
returned 503/504 for some requests (including the /status/429 endpoint). The
client treats a 503 exactly like a 429 in the backoff loop — both are
retryable — so the demonstration still holds. When httpbin's gateway is
healthy, the attempts in section 1 read "HTTP 429" instead of "HTTP 503".
------------------------------------------------------------------------------
Day 027 — Resilient API client
Test servers: https://httpbin.org and https://jsonplaceholder.typicode.com
== 1. Backoff against an endpoint that ALWAYS returns 429 ==
[429-demo] up to 5 attempts, base 1s, cap 8s
attempt 1: HTTP 503 (server error — retry with backoff)
backoff: ~1s (jittered to 1.18s)
attempt 2: HTTP 503 (server error — retry with backoff)
backoff: ~2s (jittered to 2.89s)
attempt 3: HTTP 503 (server error — retry with backoff)
backoff: ~4s (jittered to 4.83s)
attempt 4: HTTP 503 (server error — retry with backoff)
backoff: ~8s (jittered to 8.48s)
attempt 5: HTTP 000 (no response — network failure or timeout)
-> reached the retry cap of 5; giving up gracefully
made 4 retries before giving up
== 2. Read a 500 and a 404 — decide whether to retry ==
https://httpbin.org/status/500: HTTP 500 — SERVER error -> RETRY with backoff
https://httpbin.org/status/404: HTTP 404 — CLIENT error -> do NOT retry
(a transient 502/503 in place of 500 is still a 5xx — same decision)
== 3. Paginate posts with _page and _limit ==
page 1: 5 items
page 2: 5 items
collected 10 posts across 2 pages
Done. Backoff terminated, errors were classified, and pages were collected.
------------------------------------------------------------------------------
Offline, the same command prints instead (no network required, still exits 0):
Day 027 — Resilient API client
Test servers: https://httpbin.org and https://jsonplaceholder.typicode.com
OFFLINE: cannot reach https://example.com. Running the offline backoff self-test instead.
Self-test: backoff against a stubbed always-503 server (no network).
[selftest] up to 3 attempts, base 0s, cap 0s
attempt 1: HTTP 503 (server error — retry with backoff)
backoff: ~0s (jittered to 0.92s)
attempt 2: HTTP 503 (server error — retry with backoff)
backoff: ~0s (jittered to 0.28s)
attempt 3: HTTP 503 (server error — retry with backoff)
-> reached the retry cap of 3; giving up gracefully
Self-test complete: the loop terminated after 2 retries.
Reconnect to the internet and re-run for the live 429, error, and pagination demos.
Source files
examples/resilient_client.sh (7597 bytes)
#!/usr/bin/env bash
# Day 027 lab — Resilient API client (completed reference implementation).
#
# Demonstrates the three habits of consuming a real API robustly, against two
# free public test servers (no account or API key needed):
# * https://httpbin.org — returns any status you ask for
# * https://jsonplaceholder.typicode.com — a paginated posts API
#
# 1. backoff_retry — retries a 429 / 5xx / network failure with an
# exponential, jittered delay, up to a hard cap, then GIVES UP GRACEFULLY
# (it never loops forever), honoring a Retry-After header when present.
# 2. decide — reads a 500 and a 404 and decides which to retry
# (retry 5xx server errors; do NOT retry 4xx client errors).
# 3. paginate — walks two pages of posts with _page/_limit and collects
# the results across pages.
#
# Run from the lab directory: bash examples/resilient_client.sh
# Offline it degrades to a self-test of the backoff logic and still exits 0.
#
# Tunable via environment (used by the tests to run fast):
# MAX_ATTEMPTS (default 5) BASE_DELAY (default 1) DELAY_CAP (default 8)
set -u
HTTPBIN="https://httpbin.org"
JSONPH="https://jsonplaceholder.typicode.com"
PROBE="https://example.com"
: "${MAX_ATTEMPTS:=5}"
: "${BASE_DELAY:=1}"
: "${DELAY_CAP:=8}"
: "${CURL_MAX_TIME:=15}" # per-request timeout (a hung request must not stall us)
: "${STABLE_TRIES:=6}" # how many times decide() rides through transient 5xx
# Set to non-empty by the offline self-test to simulate a failing server
# without any network call at all.
: "${STUB_STATUS:=}"
RETRIES_MADE=0 # set by backoff_retry so callers can report it
have_network() { curl -s -o /dev/null --max-time 12 "${PROBE}" 2>/dev/null; }
# Print the response class for a numeric status code.
classify_status() {
case "$1" in
2*) echo "success" ;;
429) echo "rate-limited — back off and retry" ;;
4*) echo "client error — fix the request, do not retry" ;;
5*) echo "server error — retry with backoff" ;;
*) echo "no response — network failure or timeout" ;;
esac
}
# Echo "CODE RETRY_AFTER" for a URL. If STUB_STATUS is set, return it without
# touching the network (used offline to prove the retry loop terminates).
fetch_status() {
local url="$1"
if [ -n "${STUB_STATUS}" ]; then
echo "${STUB_STATUS} ${STUB_RETRY_AFTER:-}"
return 0
fi
local tmp code ra
tmp="$(mktemp)"
code="$(curl -s -o /dev/null -D "${tmp}" -w '%{http_code}' --max-time "${CURL_MAX_TIME}" "${url}" 2>/dev/null)" || code="000"
ra="$(grep -i '^retry-after:' "${tmp}" 2>/dev/null | head -1 | sed 's/[^0-9]//g')"
rm -f "${tmp}"
echo "${code:-000} ${ra}"
}
# Retry 429 / 5xx / network failures with exponential backoff + jitter, capped
# at MAX_ATTEMPTS. ALWAYS terminates: attempt increments every pass and the cap
# check returns. Honors Retry-After when the server sends it.
backoff_retry() {
local url="$1" label="${2:-request}"
local attempt=1 delay="${BASE_DELAY}" code ra wait
RETRIES_MADE=0
echo " [${label}] up to ${MAX_ATTEMPTS} attempts, base ${BASE_DELAY}s, cap ${DELAY_CAP}s"
while : ; do
read -r code ra <<< "$(fetch_status "${url}")"
echo " attempt ${attempt}: HTTP ${code} ($(classify_status "${code}"))"
case "${code}" in
2*) echo " -> success on attempt ${attempt}"; return 0 ;;
429) : ;; # retryable, fall through
4*) echo " -> client error; NOT retrying (fix the request)"; return 0 ;;
5*|000) : ;; # retryable, fall through
esac
if [ "${attempt}" -ge "${MAX_ATTEMPTS}" ]; then
echo " -> reached the retry cap of ${MAX_ATTEMPTS}; giving up gracefully"
return 0
fi
if [ -n "${ra}" ]; then
wait="${ra}"
echo " obeying Retry-After: ${wait}s"
else
wait="$(awk -v d="${delay}" 'BEGIN { srand(); printf "%.2f", d + rand() }')"
echo " backoff: ~${delay}s (jittered to ${wait}s)"
fi
sleep "${wait}"
RETRIES_MADE=$((RETRIES_MADE + 1))
attempt=$((attempt + 1))
delay=$((delay * 2))
[ "${delay}" -gt "${DELAY_CAP}" ] && delay="${DELAY_CAP}"
done
}
# Read a status, retrying ONLY on transient infrastructure failures — a
# no-response (000) or a gateway error (502/503/504) — to ride through the free
# test server's hiccups and reach its real, deliberate status. It does NOT
# retry a genuine 500, 4xx, or 2xx — that is the answer we came to read. (A 500
# is the origin's deliberate error; 502/503/504 are proxy/gateway errors that
# are usually transient.)
fetch_status_stable() {
local url="$1" tries="${STABLE_TRIES}" i=1 code ra
while [ "${i}" -le "${tries}" ]; do
read -r code ra <<< "$(fetch_status "${url}")"
case "${code}" in
000|502|503|504) sleep 1 ;; # transient — try again
*) echo "${code} ${ra}"; return 0 ;; # a real answer — take it
esac
i=$((i + 1))
done
echo "${code} ${ra}"
}
# Read one status and state the retry decision without acting on it.
decide() {
local url="$1" code ra
read -r code ra <<< "$(fetch_status_stable "${url}")"
case "${code}" in
5*) echo " ${url}: HTTP ${code} — SERVER error -> RETRY with backoff" ;;
429) echo " ${url}: HTTP ${code} — rate-limited -> RETRY after waiting" ;;
4*) echo " ${url}: HTTP ${code} — CLIENT error -> do NOT retry" ;;
2*) echo " ${url}: HTTP ${code} — success -> nothing to retry" ;;
*) echo " ${url}: no response -> network failure (retry only if idempotent)" ;;
esac
}
# Walk two pages of posts with _page/_limit and collect the count across pages.
paginate() {
local total=0 page body count
for page in 1 2; do
body="$(curl -s --max-time "${CURL_MAX_TIME}" "${JSONPH}/posts?_page=${page}&_limit=5" 2>/dev/null)" || body=""
if [ -z "${body}" ]; then
echo " page ${page}: (could not fetch — transient error; try again)"
continue
fi
count="$(printf '%s' "${body}" | python3 -c 'import sys, json; print(len(json.load(sys.stdin)))' 2>/dev/null)" || count=0
echo " page ${page}: ${count} items"
total=$((total + count))
done
echo " collected ${total} posts across 2 pages"
}
run_offline_selftest() {
echo "Self-test: backoff against a stubbed always-503 server (no network)."
STUB_STATUS=503 MAX_ATTEMPTS=3 BASE_DELAY=0 DELAY_CAP=0 backoff_retry "stub:always-fails" "selftest"
echo "Self-test complete: the loop terminated after ${RETRIES_MADE} retries."
}
main() {
echo "Day 027 — Resilient API client"
echo "Test servers: ${HTTPBIN} and ${JSONPH}"
if [ "${1:-}" = "--selftest" ]; then
run_offline_selftest
exit 0
fi
if ! have_network; then
echo
echo "OFFLINE: cannot reach ${PROBE}. Running the offline backoff self-test instead."
run_offline_selftest
echo "Reconnect to the internet and re-run for the live 429, error, and pagination demos."
exit 0
fi
echo
echo "== 1. Backoff against an endpoint that ALWAYS returns 429 =="
backoff_retry "${HTTPBIN}/status/429" "429-demo"
echo " made ${RETRIES_MADE} retries before giving up"
echo
echo "== 2. Read a 500 and a 404 — decide whether to retry =="
decide "${HTTPBIN}/status/500"
decide "${HTTPBIN}/status/404"
echo " (a transient 502/503 in place of 500 is still a 5xx — same decision)"
echo
echo "== 3. Paginate posts with _page and _limit =="
paginate
echo
echo "Done. Backoff terminated, errors were classified, and pages were collected."
}
main "$@"
metadata.yml (610 bytes)
lesson_id: D027
day: 27
kind: api-example
languages: [bash]
setup_commands:
- cd labs/sections/computing-foundations/day-027-rate-limits-pagination-and-error-handling
run_commands:
- bash examples/resilient_client.sh
- bash starter/resilient_client.sh
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- 'git checkout -- starter/resilient_client.sh # optional: reset your work'
requires_network: true
requires_api_key: false
estimated_minutes: 30
last_executed: '2026-07-12'
executed_on: 'macOS (curl 8.7.1, python3), online, bash tests/run_tests.sh → 17 checks, 0 failure(s), 0 skip(s)'
requirements/README.md (1410 bytes)
# Requirements — Day 027 lab
This lab needs only tools that ship with, or install in one step on, macOS and
Linux. There is **no API key and no account** — the test servers are free and
public.
## Required
- **bash** 3.2 or newer — preinstalled on macOS and Linux.
- **curl** — the HTTP client used for every request. Preinstalled on macOS and
almost every Linux distribution. Check with `curl --version`.
- **python3** — used only to count the items in a JSON array (the pagination
step). Check with `python3 --version`.
- macOS: preinstalled, or `brew install python`.
- Debian/Ubuntu: `sudo apt install python3`.
- Fedora: `sudo dnf install python3`.
- Standard utilities `awk`, `sed`, `grep`, `mktemp` — all preinstalled.
## Network
- **Internet access** is required for the live parts (the 429/500/404 demos and
pagination). The two public test servers are:
- `https://httpbin.org` — returns any HTTP status you ask for.
- `https://jsonplaceholder.typicode.com` — a free paginated posts API.
- **Offline is supported**: with no network, the script and the tests degrade
to a self-test of the backoff logic (using a stubbed failing server, no
network calls) and still exit 0. You lose only the live demonstrations.
## Windows
Use **WSL** (Windows Subsystem for Linux) and follow the Linux instructions, or
any environment providing `bash`, `curl`, and `python3`.
starter/resilience-worksheet.md (1835 bytes)
# Resilience worksheet — Day 027
Fill this in from a real run of your completed `starter/resilient_client.sh`
(or the reference `examples/resilient_client.sh`). The point is to prove your
client is genuinely resilient — that it terminates, classifies, and paginates —
not merely optimistic.
## 1. Backoff terminates (it does NOT loop forever)
- Retry cap you set (`MAX_ATTEMPTS`): ______
- Base delay you set (`BASE_DELAY`, seconds): ______
- How many **retries** did the backoff function make before giving up? ______
(Read the `made N retries before giving up` line.)
- Status code the `/status/429` endpoint returned on your attempts: ______
(It may show `503`/`504` if the shared test server's gateway was busy — the
client treats those the same as `429`. Note whichever you saw.)
## 2. Classify: retry vs do not retry
- `/status/500` was classified as: ______________________ (retry? yes / no)
- `/status/404` was classified as: ______________________ (retry? yes / no)
- In one sentence, why is the 404 decision different from the 500 decision?
________________________________________________________________________
## 3. Pagination
- Items on page 1: ______
- Items on page 2: ______
- **Total posts collected across the 2 pages:** ______
## 4. Short reflection (4–6 sentences)
Explain, in your own words, why your backoff loop is **guaranteed to
terminate**, and what would go wrong if you removed the `MAX_ATTEMPTS` cap.
Mention what role jitter plays when many clients hit the same limit at once.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
starter/resilient_client.sh (7121 bytes)
#!/usr/bin/env bash
# Day 027 lab — Resilient API client (STARTER).
#
# Your job: complete the four numbered exercises below, then run this file:
# bash starter/resilient_client.sh
# Compare it against the finished reference in ../examples/resilient_client.sh
# only after you have tried each exercise yourself.
#
# The helper functions are provided. You fill in the four decisions that make a
# client resilient: the retry cap, the jitter, the retry decision, and the
# pagination URL. Each exercise ships with a safe placeholder so the file runs,
# marked with the sentinel FILL_ME so the tests can see what is unfinished.
set -u
HTTPBIN="https://httpbin.org"
JSONPH="https://jsonplaceholder.typicode.com"
PROBE="https://example.com"
# ---------------------------------------------------------------------------
# Exercise 1 — Give the retry loop a HARD CAP and a starting delay.
# A retry loop with no cap becomes an infinite loop the moment a failure is
# permanent. Set:
# MAX_ATTEMPTS to 5 (stop after 5 tries)
# BASE_DELAY to 1 (first backoff wait, in seconds)
# DELAY_CAP to 8 (never wait longer than this)
# Replace each 0 below.
# ---------------------------------------------------------------------------
MAX_ATTEMPTS=0 # FILL_ME (Exercise 1): set to 5
BASE_DELAY=0 # FILL_ME (Exercise 1): set to 1
DELAY_CAP=0 # FILL_ME (Exercise 1): set to 8
: "${STUB_STATUS:=}"
RETRIES_MADE=0
have_network() { curl -s -o /dev/null --max-time 12 "${PROBE}" 2>/dev/null; }
classify_status() {
case "$1" in
2*) echo "success" ;;
429) echo "rate-limited — back off and retry" ;;
4*) echo "client error — fix the request, do not retry" ;;
5*) echo "server error — retry with backoff" ;;
*) echo "no response — network failure or timeout" ;;
esac
}
fetch_status() {
local url="$1"
if [ -n "${STUB_STATUS}" ]; then echo "${STUB_STATUS} ${STUB_RETRY_AFTER:-}"; return 0; fi
local tmp code ra
tmp="$(mktemp)"
code="$(curl -s -o /dev/null -D "${tmp}" -w '%{http_code}' --max-time 15 "${url}" 2>/dev/null)" || code="000"
ra="$(grep -i '^retry-after:' "${tmp}" 2>/dev/null | head -1 | sed 's/[^0-9]//g')"
rm -f "${tmp}"
echo "${code:-000} ${ra}"
}
backoff_retry() {
local url="$1" label="${2:-request}"
local attempt=1 delay="${BASE_DELAY}" code ra wait
RETRIES_MADE=0
echo " [${label}] up to ${MAX_ATTEMPTS} attempts, base ${BASE_DELAY}s, cap ${DELAY_CAP}s"
while : ; do
read -r code ra <<< "$(fetch_status "${url}")"
echo " attempt ${attempt}: HTTP ${code} ($(classify_status "${code}"))"
case "${code}" in
2*) echo " -> success on attempt ${attempt}"; return 0 ;;
429) : ;;
4*) echo " -> client error; NOT retrying (fix the request)"; return 0 ;;
5*|000) : ;;
esac
if [ "${attempt}" -ge "${MAX_ATTEMPTS}" ]; then
echo " -> reached the retry cap of ${MAX_ATTEMPTS}; giving up gracefully"
return 0
fi
if [ -n "${ra}" ]; then
wait="${ra}"
echo " obeying Retry-After: ${wait}s"
else
# -----------------------------------------------------------------------
# Exercise 2 — Add JITTER to the backoff wait.
# A fixed wait makes a crowd of clients retry in lockstep and stampede
# the server again. Replace the line below so `wait` becomes the delay
# PLUS a small random fraction of a second, e.g.:
# wait="$(awk -v d="${delay}" 'BEGIN { srand(); printf "%.2f", d + rand() }')"
# -----------------------------------------------------------------------
wait="${delay}" # FILL_ME (Exercise 2): add jitter to this delay
echo " backoff: ~${delay}s (waiting ${wait}s)"
fi
sleep "${wait}"
RETRIES_MADE=$((RETRIES_MADE + 1))
attempt=$((attempt + 1))
delay=$((delay * 2))
[ "${delay}" -gt "${DELAY_CAP}" ] && delay="${DELAY_CAP}"
done
}
fetch_status_stable() {
local url="$1" tries=6 i=1 code ra
while [ "${i}" -le "${tries}" ]; do
read -r code ra <<< "$(fetch_status "${url}")"
case "${code}" in
000|502|503|504) sleep 1 ;;
*) echo "${code} ${ra}"; return 0 ;;
esac
i=$((i + 1))
done
echo "${code} ${ra}"
}
decide() {
local url="$1" code ra
read -r code ra <<< "$(fetch_status_stable "${url}")"
# ---------------------------------------------------------------------------
# Exercise 3 — Classify the status and state the retry DECISION.
# Fill the two branches marked FILL_ME so that:
# a 5xx -> "SERVER error -> RETRY with backoff"
# a 4xx -> "CLIENT error -> do NOT retry"
# ---------------------------------------------------------------------------
case "${code}" in
5*) echo " ${url}: HTTP ${code} — FILL_ME (Exercise 3): SERVER error -> RETRY with backoff" ;;
429) echo " ${url}: HTTP ${code} — rate-limited -> RETRY after waiting" ;;
4*) echo " ${url}: HTTP ${code} — FILL_ME (Exercise 3): CLIENT error -> do NOT retry" ;;
2*) echo " ${url}: HTTP ${code} — success -> nothing to retry" ;;
*) echo " ${url}: no response -> network failure (retry only if idempotent)" ;;
esac
}
paginate() {
local total=0 page body count
for page in 1 2; do
# -------------------------------------------------------------------------
# Exercise 4 — Build the PAGINATED URL and collect across pages.
# Replace the URL below so it requests page ${page} with 5 items:
# "${JSONPH}/posts?_page=${page}&_limit=5"
# (The starter fetches only page 1 with no limit, so it does not paginate.)
# -------------------------------------------------------------------------
body="$(curl -s --max-time 15 "${JSONPH}/posts?_limit=5" 2>/dev/null)" || body="" # FILL_ME (Exercise 4): add _page=${page}
if [ -z "${body}" ]; then echo " page ${page}: (could not fetch)"; continue; fi
count="$(printf '%s' "${body}" | python3 -c 'import sys, json; print(len(json.load(sys.stdin)))' 2>/dev/null)" || count=0
echo " page ${page}: ${count} items"
total=$((total + count))
done
echo " collected ${total} posts across 2 pages"
}
run_offline_selftest() {
echo "Self-test: backoff against a stubbed always-503 server (no network)."
STUB_STATUS=503 backoff_retry "stub:always-fails" "selftest"
echo "Self-test complete: the loop terminated after ${RETRIES_MADE} retries."
}
main() {
echo "Day 027 — Resilient API client (starter)"
if [ "${1:-}" = "--selftest" ]; then run_offline_selftest; exit 0; fi
if ! have_network; then
echo "OFFLINE: cannot reach ${PROBE}. Running the offline backoff self-test instead."
run_offline_selftest
exit 0
fi
echo; echo "== 1. Backoff against an endpoint that ALWAYS returns 429 =="
backoff_retry "${HTTPBIN}/status/429" "429-demo"
echo " made ${RETRIES_MADE} retries before giving up"
echo; echo "== 2. Read a 500 and a 404 — decide whether to retry =="
decide "${HTTPBIN}/status/500"
decide "${HTTPBIN}/status/404"
echo; echo "== 3. Paginate posts with _page and _limit =="
paginate
echo; echo "Done."
}
main "$@"
tests/run_tests.sh (5044 bytes)
#!/usr/bin/env bash
# Tests for the Day 027 lab. Run from the lab directory:
# bash tests/run_tests.sh
#
# Two kinds of checks:
# * Structure checks always run and must pass (files present, both scripts
# parse, the example defines the resilient pieces, the starter names its
# four exercises).
# * Behavior checks. OFFLINE, the network is SKIPPED, but the backoff logic
# is still verified to TERMINATE via a stubbed always-failing call (the
# --selftest path uses no network). ONLINE, the example is run fast and its
# backoff must give up (not loop forever) and its pagination must collect
# the expected count.
#
# Every script run is wrapped in a watchdog: if it does not finish within the
# time limit it is killed and the check FAILS — this is how we prove the retry
# loop can never spin forever. The script exits 0 when no check failed.
set -u
lab_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
example="${lab_dir}/examples/resilient_client.sh"
starter="${lab_dir}/starter/resilient_client.sh"
worksheet="${lab_dir}/starter/resilience-worksheet.md"
PROBE="https://example.com"
failures=0
checks=0
skips=0
pass() { checks=$((checks + 1)); echo " ok: $1"; }
fail() { checks=$((checks + 1)); failures=$((failures + 1)); echo " FAIL: $1"; }
skip() { skips=$((skips + 1)); echo " skip: $1"; }
# run_guarded <seconds> <outfile> <command...> -> returns command's exit code,
# or 137 if the watchdog had to kill it (i.e. it did not terminate in time).
run_guarded() {
local secs="$1" outf="$2"; shift 2
"$@" > "${outf}" 2>&1 &
local pid=$!
( sleep "${secs}"; kill -9 "${pid}" 2>/dev/null ) &
local watchdog=$!
wait "${pid}" 2>/dev/null; local rc=$?
kill "${watchdog}" 2>/dev/null; wait "${watchdog}" 2>/dev/null
return "${rc}"
}
echo "== Structure checks =="
[ -f "${example}" ] && pass "example script exists" || fail "example script missing"
[ -f "${starter}" ] && pass "starter script exists" || fail "starter script missing"
[ -f "${worksheet}" ] && pass "resilience worksheet exists" || fail "worksheet missing"
for needle in "backoff_retry" "MAX_ATTEMPTS" "Retry-After" "giving up gracefully" "_page=" "_limit="; do
if grep -q -- "${needle}" "${example}"; then pass "example uses '${needle}'"; else fail "example missing '${needle}'"; fi
done
ex_count="$(grep -cE 'Exercise [1-4]' "${starter}" 2>/dev/null || true)"
if [ "${ex_count:-0}" -ge 4 ]; then pass "starter names its four exercises"; else fail "starter should name 4 exercises (found ${ex_count:-0})"; fi
bash -n "${example}" 2>/dev/null && pass "example has valid bash syntax" || fail "example has a syntax error"
bash -n "${starter}" 2>/dev/null && pass "starter has valid bash syntax" || fail "starter has a syntax error"
echo
echo "== Backoff termination (always, no network needed) =="
# The offline self-test drives the retry loop with a stubbed always-503 server
# and MUST terminate on its own well within the watchdog window.
out="$(mktemp)"
run_guarded 30 "${out}" bash "${example}" --selftest
rc=$?
if [ "${rc}" -eq 137 ]; then
fail "backoff self-test did NOT terminate (watchdog had to kill it — infinite loop!)"
elif grep -q "giving up gracefully" "${out}" && grep -q "loop terminated" "${out}"; then
pass "backoff self-test terminates and gives up gracefully (no network)"
else
fail "backoff self-test did not report a graceful give-up"
sed 's/^/ /' "${out}"
fi
rm -f "${out}"
echo
echo "== Live behavior checks =="
if ! curl -s -o /dev/null --max-time 12 "${PROBE}" 2>/dev/null; then
skip "no network access — skipping live 429/pagination checks (expected offline)"
else
pass "network reachable (${PROBE} responded)"
out="$(mktemp)"
# Run fast: tiny delays so the capped backoff finishes quickly. The watchdog
# still guarantees termination even if the test server is slow.
# Short per-request timeout + few stable-retries keeps the run bounded even
# when the shared test server is slow, so the watchdog only ever fires on a
# genuine non-terminating loop (not on a slow network).
MAX_ATTEMPTS=3 BASE_DELAY=0 DELAY_CAP=0 CURL_MAX_TIME=4 STABLE_TRIES=2 \
run_guarded 90 "${out}" bash "${example}"
rc=$?
if [ "${rc}" -eq 137 ]; then
fail "example did NOT terminate within 90s (watchdog killed it)"
sed 's/^/ /' "${out}"
else
pass "example run terminated on its own"
grep -q "giving up gracefully" "${out}" \
&& pass "backoff gave up gracefully after its cap" \
|| fail "expected a 'giving up gracefully' line"
collected="$(sed -n 's/.*collected \([0-9]\{1,\}\) posts across 2 pages/\1/p' "${out}" | head -1)"
case "${collected}" in
10) pass "pagination collected 10 posts across 2 pages" ;;
'' ) skip "pagination count not found — test server (jsonplaceholder) may be transiently down" ;;
*) fail "pagination should collect 10 posts (got '${collected}')" ;;
esac
fi
rm -f "${out}"
fi
echo
echo "${checks} checks, ${failures} failure(s), ${skips} skip(s)."
[ "${failures}" -eq 0 ]
Troubleshooting
Troubleshooting — Day 027 lab
The test server returns 503 or 504, or times out, instead of the status I asked for
This is expected, and it is the whole point of the lesson. httpbin.org is
a free, shared service; when it is under load its gateway returns 502/503/
504 or simply times out, even for an endpoint like /status/429. A resilient
client must survive exactly this — and yours does: the backoff loop treats a
503 the same as a 429 (both retryable) and keeps its cap, and the decide
step rides through transient gateway 5xx/timeouts to read the endpoint's real,
deliberate status. If a run shows 503 where you expected 429, that is a real
teaching moment, not a bug. Re-run in a minute for a cleaner picture, or note
the 503 — the client handled it correctly either way.
command not found: python3
The pagination step uses python3 only to count items in a JSON array. Install
it (macOS: brew install python; Debian/Ubuntu: sudo apt install python3), or
adapt the counter. See requirements/README.md.
A run seems to hang
It should not — every request uses --max-time, and every retry loop has a hard
cap (MAX_ATTEMPTS). If your edited starter removed the --max-time flag or the
cap, put them back: a timeout turns a hung request into a handled failure, and a
cap turns a retry loop into one that always terminates. The tests wrap each run
in a watchdog that kills and fails any run that does not finish in time,
precisely to catch an accidental infinite loop.
curl: (6) Could not resolve host / (7) Failed to connect
You are offline or behind a restrictive network. That is a network failure, not a bug. The script detects it, runs the offline backoff self-test, and exits 0. Reconnect and re-run for the live demos.
The tests print skip: lines
Skips are not failures. Offline, the live network checks are skipped by design.
If jsonplaceholder.typicode.com is transiently down, the pagination-count
check skips rather than failing — an external outage is not your bug. The suite
still exits 0 as long as no structural or termination check failed.
The backoff waited far longer than I expected
Two causes. First, if the server sends a Retry-After header, the client obeys
it exactly (that can be seconds or more) instead of using its own delay — this
is correct behavior. Second, the default base delay doubles each attempt up to
the cap (1, 2, 4, 8, 8 s), so five attempts can wait ~23 s before giving up. Set
BASE_DELAY and DELAY_CAP lower to move faster while experimenting:
BASE_DELAY=0 DELAY_CAP=0 bash examples/resilient_client.sh.
Security notes
Security notes — Day 027 lab
This lab makes only outbound HTTPS GET requests to two well-known public test
servers, needs no API key, no account, and no elevated privileges, and writes
nothing outside its own console output (plus a short-lived temp file for
response headers, which it deletes). Still, the topic itself is about being a
good citizen of other people's servers, so the security lessons are the point.
Be a polite client
- Honor rate limits. When a server returns
429, back off — do not hammer it harder. When it sends aRetry-Afterheader, wait exactly that long; you cannot guess better than the server just told you. - Always add jitter. A fleet of clients that all retry at the same instant is a self-inflicted denial-of-service (the "thundering herd"). A small random component on every wait spreads the load and lets the server recover.
- Always cap your retries. An uncapped retry loop against a struggling
server looks exactly like an attack and can get your IP or key blocked. This
lab's loop has a hard
MAX_ATTEMPTSand a per-request--max-timefor exactly this reason. - Identify yourself honestly in real projects (a descriptive
User-Agent) and read each API's terms — some forbid scraping or set explicit request ceilings.
Handle errors and their bodies carefully
- Do not log secrets. Error responses sometimes include internal details, tokens, or user data. Log what you need to debug and redact the rest; never paste a raw error dump containing credentials into a public issue.
- Never retry a non-idempotent write blindly. Retrying a "create" or "charge" request after a timeout can duplicate the action, because the first attempt may have succeeded even though the response never arrived. Use an idempotency key or do not retry it.
What this lab does not do
- No credentials are sent or stored.
- No inbound connections are opened; nothing listens on a port.
- The only local write is a temporary header file created with
mktempand removed immediately after use.