Computing FoundationsAPIs and the Web › Day 27

Hands-on lab — Day 27: Rate Limits, Pagination, and Error Handling

Commands

Setup

cd labs/sections/computing-foundations/day-027-rate-limits-pagination-and-error-handling

Run

bash examples/resilient_client.sh
bash starter/resilient_client.sh

Test

bash tests/run_tests.sh

File tree

examples/resilient_client.sh
expected-output/FIELDS.md
expected-output/sample-run.txt
metadata.yml
README.md
requirements/README.md
security.md
starter/resilience-worksheet.md
starter/resilient_client.sh
tests/run_tests.sh
troubleshooting.md

Lab README

Day 027 lab — Handle Limits and Pages

Build and run a small resilient API client in shell against free public test servers: back off from a 429, decide correctly which errors to retry, and paginate across pages — collecting results without missing or repeating a row.

Lesson

Purpose

Day 27's lesson explains the three habits of consuming a real API robustly. This lab makes them concrete and executable: you run a client that survives rate limits and server hiccups, classifies failures, and walks a paginated endpoint — then you complete a starter version yourself. Every request hits a live, free, key-less test server, so the numbers are real. This is Day 27 of the course.

Learning objectives

  • Trigger a 429 Too Many Requests response and read the status code back.
  • Watch an exponential-backoff-with-jitter loop retry and then give up gracefully — proving it terminates rather than looping forever.
  • Classify a 500 (retry) apart from a 404 (do not retry) and say why.
  • Paginate an endpoint with _page/_limit and collect results across pages.
  • Run an automated test suite that verifies termination and the collected count.

Prerequisites

  • The Day 27 lesson (read it first — it explains every concept this lab runs).
  • Days 18-24: HTTP status codes and JSON.
  • A terminal with bash, curl, and python3. Internet access for the live parts (offline is supported and degrades gracefully).

Supported operating systems

  • macOS — fully supported (tested on macOS with curl 8.7.1).
  • Linux — fully supported (any distribution with bash, curl, python3).
  • Windows — use WSL and follow the Linux instructions.

Hardware requirements

Any computer from roughly the last 15 years. The lab only makes small HTTP requests; it needs no particular RAM, disk, or GPU.

Required software

  • bash 3.2+, curl, and python3 (used only to count JSON items).
  • Standard utilities awk, sed, grep, mktemp — all preinstalled.
  • See requirements/README.md for one-step install commands per platform.

Free and open-source options

Everything here is free and needs no account or API key. curl and python3 are open source; the test servers (httpbin.org, jsonplaceholder.typicode.com) are free public services. There is no paid component anywhere in this lab.

Installation

None beyond the tools above. From the repository root:

cd labs/sections/computing-foundations/day-027-rate-limits-pagination-and-error-handling

File structure

day-027-rate-limits-pagination-and-error-handling/
├── README.md                              ← you are here
├── metadata.yml                           ← machine-readable lab metadata
├── starter/
│   ├── resilient_client.sh                ← YOUR working file (4 exercises)
│   └── resilience-worksheet.md            ← record your real numbers here
├── examples/
│   └── resilient_client.sh                ← completed reference implementation
├── tests/
│   └── run_tests.sh                       ← automated checks (with a watchdog)
├── expected-output/
│   ├── sample-run.txt                     ← a real captured run (online + offline)
│   └── FIELDS.md                          ← required lines on every platform
├── requirements/
│   └── README.md                          ← dependency statement
├── troubleshooting.md
└── security.md

How to run

From this directory:

## 1. See the finished client first (online: live demos; offline: self-test)
bash examples/resilient_client.sh

## 2. Your task: complete the four exercises in the starter, then run it
bash starter/resilient_client.sh

## 3. Check your work
bash tests/run_tests.sh

To move faster while experimenting, shrink the delays:

BASE_DELAY=0 DELAY_CAP=0 bash examples/resilient_client.sh

What the commands do

  • bash examples/resilient_client.sh — runs the reference client: a backoff-retry loop against httpbin.org/status/429 that retries with growing, jittered delays up to a cap and then gives up gracefully (honoring Retry-After when present); a decide step that reads a 500 and a 404 and states which to retry; and a paginator that walks two pages of jsonplaceholder.typicode.com/posts with _page/_limit and sums the items. Offline, it runs a no-network self-test of the backoff logic and exits 0.
  • bash starter/resilient_client.sh — the same skeleton with four spots blanked out (marked FILL_ME); each numbered exercise names the exact change to make.
  • bash tests/run_tests.sh — structure checks, plus a watchdog-guarded run that proves the backoff terminates and the pagination collects the expected count.

Expected output

See expected-output/sample-run.txt for a real captured run (online and offline). The key lines: the backoff prints several attempt N: HTTP ... lines and exactly one giving up gracefully; the decide step marks /status/500 as RETRY and /status/404 as do NOT retry; and the paginator prints collected 10 posts across 2 pages. Your exact statuses, jitter values, and retry counts will differ — that is the nature of a real API.

Validation steps

  1. Run bash examples/resilient_client.sh — it must finish on its own (never hang) and end with Done. (online) or the offline self-test message.
  2. Confirm the backoff prints exactly one giving up gracefully line.
  3. Confirm /status/500 is classified as retry and /status/404 as do-not-retry.
  4. Confirm the paginator reports the total it collected across two pages.
  5. Complete the four exercises in the starter and record your numbers on the worksheet, then run the tests below.

Tests

bash tests/run_tests.sh

Expected final line: 17 checks, 0 failure(s), 0 skip(s). online, or with some skip(s) offline. Each script run is wrapped in a watchdog that kills and fails any run that does not terminate — so a passing suite is proof the retry loop cannot spin forever. The command exits 0 on success, non-zero on any failure, so it can run in CI.

Cleanup

Nothing to clean up: the scripts make read-only HTTP requests and write only a short-lived temp file that they delete. To reset your work, restore the starter from git: git checkout -- starter/resilient_client.sh.

Troubleshooting

See troubleshooting.md. Short version: transient 503/504 or timeouts from the shared test server are expected and are exactly what your client is built to survive — re-run in a minute for a cleaner picture.

Security notes

See security.md. Short version: be a polite client — honor rate limits, add jitter, and cap your retries. Hammering an API can get your key or IP blocked, and an uncapped retry storm is indistinguishable from an attack.

Extension exercises

  1. Make the client honor Retry-After: fetch https://httpbin.org/response-headers?Retry-After=3, read the header, and wait exactly that long instead of using the doubling delay.
  2. Add a third page to the paginator and confirm the collected total rises accordingly; stop when a page returns zero items.
  3. Add an idempotency check: print a warning before retrying anything that is not a GET, since non-idempotent writes are unsafe to repeat blindly.
  • Previous day: Day 26 — Webhooks and Event-Driven APIs (labs/sections/computing-foundations/day-026-webhooks-and-event-driven-apis/).
  • Next day: Day 28 — Consuming a Public API from the Command Line (labs/sections/computing-foundations/day-028-consuming-a-public-api-from-the/).

Expected output

FIELDS.md

# Required fields in a run of `resilient_client.sh`

These lines must appear on every platform (macOS or Linux) and in both network
states. The exact status codes, jitter values, and retry counts vary run to
run — that variability is the point — but the *structure* is fixed.

## Online (network reachable)

| Section | Must contain |
| --- | --- |
| Header | `Day 027 — Resilient API client` |
| 1. Backoff | one or more `attempt N: HTTP ...` lines |
| 1. Backoff | exactly one `giving up gracefully` line (the loop MUST terminate) |
| 1. Backoff | a `made N retries before giving up` summary |
| 2. Decide | a `/status/500` line ending in `RETRY with backoff` |
| 2. Decide | a `/status/404` line ending in `do NOT retry` |
| 3. Paginate | two `page N: M items` lines |
| 3. Paginate | a `collected T posts across 2 pages` line (T is the sum) |
| Footer | `Done.` |

## Offline (network unreachable) or `--selftest`

| Must contain |
| --- |
| `OFFLINE:` (only in the auto-detected offline path) or `Self-test:` |
| a stubbed `attempt N: HTTP 503` sequence |
| exactly one `giving up gracefully` line — proof the loop terminates without a network |
| `loop terminated` |

## Platform notes

- The script needs only `bash`, `curl`, `awk`, `sed`, and (for the item count)
  `python3`. All ship with macOS and are one package install away on Linux.
- The public test servers (`httpbin.org`, `jsonplaceholder.typicode.com`) are
  shared and occasionally return transient 5xx or time out. The client is
  built to survive exactly that, so a run during a hiccup is still a valid run.

sample-run.txt

Real captured run of `bash examples/resilient_client.sh` on macOS (curl 8.7.1),
online. Your exact numbers, jitter values, and the transient statuses from the
public test server will differ — that is the nature of a real API.

Note: during this capture, https://httpbin.org was under load and its gateway
returned 503/504 for some requests (including the /status/429 endpoint). The
client treats a 503 exactly like a 429 in the backoff loop — both are
retryable — so the demonstration still holds. When httpbin's gateway is
healthy, the attempts in section 1 read "HTTP 429" instead of "HTTP 503".

------------------------------------------------------------------------------
Day 027 — Resilient API client
Test servers: https://httpbin.org and https://jsonplaceholder.typicode.com

== 1. Backoff against an endpoint that ALWAYS returns 429 ==
  [429-demo] up to 5 attempts, base 1s, cap 8s
  attempt 1: HTTP 503 (server error — retry with backoff)
     backoff: ~1s (jittered to 1.18s)
  attempt 2: HTTP 503 (server error — retry with backoff)
     backoff: ~2s (jittered to 2.89s)
  attempt 3: HTTP 503 (server error — retry with backoff)
     backoff: ~4s (jittered to 4.83s)
  attempt 4: HTTP 503 (server error — retry with backoff)
     backoff: ~8s (jittered to 8.48s)
  attempt 5: HTTP 000 (no response — network failure or timeout)
  -> reached the retry cap of 5; giving up gracefully
   made 4 retries before giving up

== 2. Read a 500 and a 404 — decide whether to retry ==
  https://httpbin.org/status/500: HTTP 500 — SERVER error -> RETRY with backoff
  https://httpbin.org/status/404: HTTP 404 — CLIENT error -> do NOT retry
   (a transient 502/503 in place of 500 is still a 5xx — same decision)

== 3. Paginate posts with _page and _limit ==
  page 1: 5 items
  page 2: 5 items
  collected 10 posts across 2 pages

Done. Backoff terminated, errors were classified, and pages were collected.
------------------------------------------------------------------------------

Offline, the same command prints instead (no network required, still exits 0):

Day 027 — Resilient API client
Test servers: https://httpbin.org and https://jsonplaceholder.typicode.com

OFFLINE: cannot reach https://example.com. Running the offline backoff self-test instead.
Self-test: backoff against a stubbed always-503 server (no network).
  [selftest] up to 3 attempts, base 0s, cap 0s
  attempt 1: HTTP 503 (server error — retry with backoff)
     backoff: ~0s (jittered to 0.92s)
  attempt 2: HTTP 503 (server error — retry with backoff)
     backoff: ~0s (jittered to 0.28s)
  attempt 3: HTTP 503 (server error — retry with backoff)
  -> reached the retry cap of 3; giving up gracefully
Self-test complete: the loop terminated after 2 retries.
Reconnect to the internet and re-run for the live 429, error, and pagination demos.

Source files

examples/resilient_client.sh (7597 bytes)
#!/usr/bin/env bash
# Day 027 lab — Resilient API client (completed reference implementation).
#
# Demonstrates the three habits of consuming a real API robustly, against two
# free public test servers (no account or API key needed):
#   * https://httpbin.org           — returns any status you ask for
#   * https://jsonplaceholder.typicode.com — a paginated posts API
#
#   1. backoff_retry — retries a 429 / 5xx / network failure with an
#      exponential, jittered delay, up to a hard cap, then GIVES UP GRACEFULLY
#      (it never loops forever), honoring a Retry-After header when present.
#   2. decide        — reads a 500 and a 404 and decides which to retry
#      (retry 5xx server errors; do NOT retry 4xx client errors).
#   3. paginate      — walks two pages of posts with _page/_limit and collects
#      the results across pages.
#
# Run from the lab directory:  bash examples/resilient_client.sh
# Offline it degrades to a self-test of the backoff logic and still exits 0.
#
# Tunable via environment (used by the tests to run fast):
#   MAX_ATTEMPTS (default 5)  BASE_DELAY (default 1)  DELAY_CAP (default 8)
set -u

HTTPBIN="https://httpbin.org"
JSONPH="https://jsonplaceholder.typicode.com"
PROBE="https://example.com"

: "${MAX_ATTEMPTS:=5}"
: "${BASE_DELAY:=1}"
: "${DELAY_CAP:=8}"
: "${CURL_MAX_TIME:=15}"   # per-request timeout (a hung request must not stall us)
: "${STABLE_TRIES:=6}"     # how many times decide() rides through transient 5xx

# Set to non-empty by the offline self-test to simulate a failing server
# without any network call at all.
: "${STUB_STATUS:=}"

RETRIES_MADE=0   # set by backoff_retry so callers can report it

have_network() { curl -s -o /dev/null --max-time 12 "${PROBE}" 2>/dev/null; }

# Print the response class for a numeric status code.
classify_status() {
  case "$1" in
    2*)   echo "success" ;;
    429)  echo "rate-limited — back off and retry" ;;
    4*)   echo "client error — fix the request, do not retry" ;;
    5*)   echo "server error — retry with backoff" ;;
    *)    echo "no response — network failure or timeout" ;;
  esac
}

# Echo "CODE RETRY_AFTER" for a URL. If STUB_STATUS is set, return it without
# touching the network (used offline to prove the retry loop terminates).
fetch_status() {
  local url="$1"
  if [ -n "${STUB_STATUS}" ]; then
    echo "${STUB_STATUS} ${STUB_RETRY_AFTER:-}"
    return 0
  fi
  local tmp code ra
  tmp="$(mktemp)"
  code="$(curl -s -o /dev/null -D "${tmp}" -w '%{http_code}' --max-time "${CURL_MAX_TIME}" "${url}" 2>/dev/null)" || code="000"
  ra="$(grep -i '^retry-after:' "${tmp}" 2>/dev/null | head -1 | sed 's/[^0-9]//g')"
  rm -f "${tmp}"
  echo "${code:-000} ${ra}"
}

# Retry 429 / 5xx / network failures with exponential backoff + jitter, capped
# at MAX_ATTEMPTS. ALWAYS terminates: attempt increments every pass and the cap
# check returns. Honors Retry-After when the server sends it.
backoff_retry() {
  local url="$1" label="${2:-request}"
  local attempt=1 delay="${BASE_DELAY}" code ra wait
  RETRIES_MADE=0
  echo "  [${label}] up to ${MAX_ATTEMPTS} attempts, base ${BASE_DELAY}s, cap ${DELAY_CAP}s"
  while : ; do
    read -r code ra <<< "$(fetch_status "${url}")"
    echo "  attempt ${attempt}: HTTP ${code} ($(classify_status "${code}"))"

    case "${code}" in
      2*) echo "  -> success on attempt ${attempt}"; return 0 ;;
      429) : ;;                                   # retryable, fall through
      4*) echo "  -> client error; NOT retrying (fix the request)"; return 0 ;;
      5*|000) : ;;                                # retryable, fall through
    esac

    if [ "${attempt}" -ge "${MAX_ATTEMPTS}" ]; then
      echo "  -> reached the retry cap of ${MAX_ATTEMPTS}; giving up gracefully"
      return 0
    fi

    if [ -n "${ra}" ]; then
      wait="${ra}"
      echo "     obeying Retry-After: ${wait}s"
    else
      wait="$(awk -v d="${delay}" 'BEGIN { srand(); printf "%.2f", d + rand() }')"
      echo "     backoff: ~${delay}s (jittered to ${wait}s)"
    fi
    sleep "${wait}"

    RETRIES_MADE=$((RETRIES_MADE + 1))
    attempt=$((attempt + 1))
    delay=$((delay * 2))
    [ "${delay}" -gt "${DELAY_CAP}" ] && delay="${DELAY_CAP}"
  done
}

# Read a status, retrying ONLY on transient infrastructure failures — a
# no-response (000) or a gateway error (502/503/504) — to ride through the free
# test server's hiccups and reach its real, deliberate status. It does NOT
# retry a genuine 500, 4xx, or 2xx — that is the answer we came to read. (A 500
# is the origin's deliberate error; 502/503/504 are proxy/gateway errors that
# are usually transient.)
fetch_status_stable() {
  local url="$1" tries="${STABLE_TRIES}" i=1 code ra
  while [ "${i}" -le "${tries}" ]; do
    read -r code ra <<< "$(fetch_status "${url}")"
    case "${code}" in
      000|502|503|504) sleep 1 ;;               # transient — try again
      *) echo "${code} ${ra}"; return 0 ;;      # a real answer — take it
    esac
    i=$((i + 1))
  done
  echo "${code} ${ra}"
}

# Read one status and state the retry decision without acting on it.
decide() {
  local url="$1" code ra
  read -r code ra <<< "$(fetch_status_stable "${url}")"
  case "${code}" in
    5*) echo "  ${url}: HTTP ${code} — SERVER error -> RETRY with backoff" ;;
    429) echo "  ${url}: HTTP ${code} — rate-limited -> RETRY after waiting" ;;
    4*) echo "  ${url}: HTTP ${code} — CLIENT error -> do NOT retry" ;;
    2*) echo "  ${url}: HTTP ${code} — success -> nothing to retry" ;;
    *)  echo "  ${url}: no response -> network failure (retry only if idempotent)" ;;
  esac
}

# Walk two pages of posts with _page/_limit and collect the count across pages.
paginate() {
  local total=0 page body count
  for page in 1 2; do
    body="$(curl -s --max-time "${CURL_MAX_TIME}" "${JSONPH}/posts?_page=${page}&_limit=5" 2>/dev/null)" || body=""
    if [ -z "${body}" ]; then
      echo "  page ${page}: (could not fetch — transient error; try again)"
      continue
    fi
    count="$(printf '%s' "${body}" | python3 -c 'import sys, json; print(len(json.load(sys.stdin)))' 2>/dev/null)" || count=0
    echo "  page ${page}: ${count} items"
    total=$((total + count))
  done
  echo "  collected ${total} posts across 2 pages"
}

run_offline_selftest() {
  echo "Self-test: backoff against a stubbed always-503 server (no network)."
  STUB_STATUS=503 MAX_ATTEMPTS=3 BASE_DELAY=0 DELAY_CAP=0 backoff_retry "stub:always-fails" "selftest"
  echo "Self-test complete: the loop terminated after ${RETRIES_MADE} retries."
}

main() {
  echo "Day 027 — Resilient API client"
  echo "Test servers: ${HTTPBIN} and ${JSONPH}"

  if [ "${1:-}" = "--selftest" ]; then
    run_offline_selftest
    exit 0
  fi

  if ! have_network; then
    echo
    echo "OFFLINE: cannot reach ${PROBE}. Running the offline backoff self-test instead."
    run_offline_selftest
    echo "Reconnect to the internet and re-run for the live 429, error, and pagination demos."
    exit 0
  fi

  echo
  echo "== 1. Backoff against an endpoint that ALWAYS returns 429 =="
  backoff_retry "${HTTPBIN}/status/429" "429-demo"
  echo "   made ${RETRIES_MADE} retries before giving up"

  echo
  echo "== 2. Read a 500 and a 404 — decide whether to retry =="
  decide "${HTTPBIN}/status/500"
  decide "${HTTPBIN}/status/404"
  echo "   (a transient 502/503 in place of 500 is still a 5xx — same decision)"

  echo
  echo "== 3. Paginate posts with _page and _limit =="
  paginate

  echo
  echo "Done. Backoff terminated, errors were classified, and pages were collected."
}

main "$@"
metadata.yml (610 bytes)
lesson_id: D027
day: 27
kind: api-example
languages: [bash]
setup_commands:
  - cd labs/sections/computing-foundations/day-027-rate-limits-pagination-and-error-handling
run_commands:
  - bash examples/resilient_client.sh
  - bash starter/resilient_client.sh
test_commands:
  - bash tests/run_tests.sh
cleanup_commands:
  - 'git checkout -- starter/resilient_client.sh  # optional: reset your work'
requires_network: true
requires_api_key: false
estimated_minutes: 30
last_executed: '2026-07-12'
executed_on: 'macOS (curl 8.7.1, python3), online, bash tests/run_tests.sh → 17 checks, 0 failure(s), 0 skip(s)'
requirements/README.md (1410 bytes)
# Requirements — Day 027 lab

This lab needs only tools that ship with, or install in one step on, macOS and
Linux. There is **no API key and no account** — the test servers are free and
public.

## Required

- **bash** 3.2 or newer — preinstalled on macOS and Linux.
- **curl** — the HTTP client used for every request. Preinstalled on macOS and
  almost every Linux distribution. Check with `curl --version`.
- **python3** — used only to count the items in a JSON array (the pagination
  step). Check with `python3 --version`.
  - macOS: preinstalled, or `brew install python`.
  - Debian/Ubuntu: `sudo apt install python3`.
  - Fedora: `sudo dnf install python3`.
- Standard utilities `awk`, `sed`, `grep`, `mktemp` — all preinstalled.

## Network

- **Internet access** is required for the live parts (the 429/500/404 demos and
  pagination). The two public test servers are:
  - `https://httpbin.org` — returns any HTTP status you ask for.
  - `https://jsonplaceholder.typicode.com` — a free paginated posts API.
- **Offline is supported**: with no network, the script and the tests degrade
  to a self-test of the backoff logic (using a stubbed failing server, no
  network calls) and still exit 0. You lose only the live demonstrations.

## Windows

Use **WSL** (Windows Subsystem for Linux) and follow the Linux instructions, or
any environment providing `bash`, `curl`, and `python3`.
starter/resilience-worksheet.md (1835 bytes)
# Resilience worksheet — Day 027

Fill this in from a real run of your completed `starter/resilient_client.sh`
(or the reference `examples/resilient_client.sh`). The point is to prove your
client is genuinely resilient — that it terminates, classifies, and paginates —
not merely optimistic.

## 1. Backoff terminates (it does NOT loop forever)

- Retry cap you set (`MAX_ATTEMPTS`): ______
- Base delay you set (`BASE_DELAY`, seconds): ______
- How many **retries** did the backoff function make before giving up? ______
  (Read the `made N retries before giving up` line.)
- Status code the `/status/429` endpoint returned on your attempts: ______
  (It may show `503`/`504` if the shared test server's gateway was busy — the
  client treats those the same as `429`. Note whichever you saw.)

## 2. Classify: retry vs do not retry

- `/status/500` was classified as: ______________________  (retry? yes / no)
- `/status/404` was classified as: ______________________  (retry? yes / no)
- In one sentence, why is the 404 decision different from the 500 decision?

  ________________________________________________________________________

## 3. Pagination

- Items on page 1: ______
- Items on page 2: ______
- **Total posts collected across the 2 pages:** ______

## 4. Short reflection (4–6 sentences)

Explain, in your own words, why your backoff loop is **guaranteed to
terminate**, and what would go wrong if you removed the `MAX_ATTEMPTS` cap.
Mention what role jitter plays when many clients hit the same limit at once.

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________
starter/resilient_client.sh (7121 bytes)
#!/usr/bin/env bash
# Day 027 lab — Resilient API client (STARTER).
#
# Your job: complete the four numbered exercises below, then run this file:
#     bash starter/resilient_client.sh
# Compare it against the finished reference in ../examples/resilient_client.sh
# only after you have tried each exercise yourself.
#
# The helper functions are provided. You fill in the four decisions that make a
# client resilient: the retry cap, the jitter, the retry decision, and the
# pagination URL. Each exercise ships with a safe placeholder so the file runs,
# marked with the sentinel FILL_ME so the tests can see what is unfinished.
set -u

HTTPBIN="https://httpbin.org"
JSONPH="https://jsonplaceholder.typicode.com"
PROBE="https://example.com"

# ---------------------------------------------------------------------------
# Exercise 1 — Give the retry loop a HARD CAP and a starting delay.
#   A retry loop with no cap becomes an infinite loop the moment a failure is
#   permanent. Set:
#     MAX_ATTEMPTS to 5   (stop after 5 tries)
#     BASE_DELAY   to 1   (first backoff wait, in seconds)
#     DELAY_CAP    to 8   (never wait longer than this)
#   Replace each 0 below.
# ---------------------------------------------------------------------------
MAX_ATTEMPTS=0   # FILL_ME (Exercise 1): set to 5
BASE_DELAY=0     # FILL_ME (Exercise 1): set to 1
DELAY_CAP=0      # FILL_ME (Exercise 1): set to 8

: "${STUB_STATUS:=}"
RETRIES_MADE=0

have_network() { curl -s -o /dev/null --max-time 12 "${PROBE}" 2>/dev/null; }

classify_status() {
  case "$1" in
    2*)  echo "success" ;;
    429) echo "rate-limited — back off and retry" ;;
    4*)  echo "client error — fix the request, do not retry" ;;
    5*)  echo "server error — retry with backoff" ;;
    *)   echo "no response — network failure or timeout" ;;
  esac
}

fetch_status() {
  local url="$1"
  if [ -n "${STUB_STATUS}" ]; then echo "${STUB_STATUS} ${STUB_RETRY_AFTER:-}"; return 0; fi
  local tmp code ra
  tmp="$(mktemp)"
  code="$(curl -s -o /dev/null -D "${tmp}" -w '%{http_code}' --max-time 15 "${url}" 2>/dev/null)" || code="000"
  ra="$(grep -i '^retry-after:' "${tmp}" 2>/dev/null | head -1 | sed 's/[^0-9]//g')"
  rm -f "${tmp}"
  echo "${code:-000} ${ra}"
}

backoff_retry() {
  local url="$1" label="${2:-request}"
  local attempt=1 delay="${BASE_DELAY}" code ra wait
  RETRIES_MADE=0
  echo "  [${label}] up to ${MAX_ATTEMPTS} attempts, base ${BASE_DELAY}s, cap ${DELAY_CAP}s"
  while : ; do
    read -r code ra <<< "$(fetch_status "${url}")"
    echo "  attempt ${attempt}: HTTP ${code} ($(classify_status "${code}"))"

    case "${code}" in
      2*) echo "  -> success on attempt ${attempt}"; return 0 ;;
      429) : ;;
      4*) echo "  -> client error; NOT retrying (fix the request)"; return 0 ;;
      5*|000) : ;;
    esac

    if [ "${attempt}" -ge "${MAX_ATTEMPTS}" ]; then
      echo "  -> reached the retry cap of ${MAX_ATTEMPTS}; giving up gracefully"
      return 0
    fi

    if [ -n "${ra}" ]; then
      wait="${ra}"
      echo "     obeying Retry-After: ${wait}s"
    else
      # -----------------------------------------------------------------------
      # Exercise 2 — Add JITTER to the backoff wait.
      #   A fixed wait makes a crowd of clients retry in lockstep and stampede
      #   the server again. Replace the line below so `wait` becomes the delay
      #   PLUS a small random fraction of a second, e.g.:
      #     wait="$(awk -v d="${delay}" 'BEGIN { srand(); printf "%.2f", d + rand() }')"
      # -----------------------------------------------------------------------
      wait="${delay}"   # FILL_ME (Exercise 2): add jitter to this delay
      echo "     backoff: ~${delay}s (waiting ${wait}s)"
    fi
    sleep "${wait}"

    RETRIES_MADE=$((RETRIES_MADE + 1))
    attempt=$((attempt + 1))
    delay=$((delay * 2))
    [ "${delay}" -gt "${DELAY_CAP}" ] && delay="${DELAY_CAP}"
  done
}

fetch_status_stable() {
  local url="$1" tries=6 i=1 code ra
  while [ "${i}" -le "${tries}" ]; do
    read -r code ra <<< "$(fetch_status "${url}")"
    case "${code}" in
      000|502|503|504) sleep 1 ;;
      *) echo "${code} ${ra}"; return 0 ;;
    esac
    i=$((i + 1))
  done
  echo "${code} ${ra}"
}

decide() {
  local url="$1" code ra
  read -r code ra <<< "$(fetch_status_stable "${url}")"
  # ---------------------------------------------------------------------------
  # Exercise 3 — Classify the status and state the retry DECISION.
  #   Fill the two branches marked FILL_ME so that:
  #     a 5xx  -> "SERVER error -> RETRY with backoff"
  #     a 4xx  -> "CLIENT error -> do NOT retry"
  # ---------------------------------------------------------------------------
  case "${code}" in
    5*) echo "  ${url}: HTTP ${code} — FILL_ME (Exercise 3): SERVER error -> RETRY with backoff" ;;
    429) echo "  ${url}: HTTP ${code} — rate-limited -> RETRY after waiting" ;;
    4*) echo "  ${url}: HTTP ${code} — FILL_ME (Exercise 3): CLIENT error -> do NOT retry" ;;
    2*) echo "  ${url}: HTTP ${code} — success -> nothing to retry" ;;
    *)  echo "  ${url}: no response -> network failure (retry only if idempotent)" ;;
  esac
}

paginate() {
  local total=0 page body count
  for page in 1 2; do
    # -------------------------------------------------------------------------
    # Exercise 4 — Build the PAGINATED URL and collect across pages.
    #   Replace the URL below so it requests page ${page} with 5 items:
    #     "${JSONPH}/posts?_page=${page}&_limit=5"
    #   (The starter fetches only page 1 with no limit, so it does not paginate.)
    # -------------------------------------------------------------------------
    body="$(curl -s --max-time 15 "${JSONPH}/posts?_limit=5" 2>/dev/null)" || body=""   # FILL_ME (Exercise 4): add _page=${page}
    if [ -z "${body}" ]; then echo "  page ${page}: (could not fetch)"; continue; fi
    count="$(printf '%s' "${body}" | python3 -c 'import sys, json; print(len(json.load(sys.stdin)))' 2>/dev/null)" || count=0
    echo "  page ${page}: ${count} items"
    total=$((total + count))
  done
  echo "  collected ${total} posts across 2 pages"
}

run_offline_selftest() {
  echo "Self-test: backoff against a stubbed always-503 server (no network)."
  STUB_STATUS=503 backoff_retry "stub:always-fails" "selftest"
  echo "Self-test complete: the loop terminated after ${RETRIES_MADE} retries."
}

main() {
  echo "Day 027 — Resilient API client (starter)"
  if [ "${1:-}" = "--selftest" ]; then run_offline_selftest; exit 0; fi
  if ! have_network; then
    echo "OFFLINE: cannot reach ${PROBE}. Running the offline backoff self-test instead."
    run_offline_selftest
    exit 0
  fi
  echo; echo "== 1. Backoff against an endpoint that ALWAYS returns 429 =="
  backoff_retry "${HTTPBIN}/status/429" "429-demo"
  echo "   made ${RETRIES_MADE} retries before giving up"
  echo; echo "== 2. Read a 500 and a 404 — decide whether to retry =="
  decide "${HTTPBIN}/status/500"
  decide "${HTTPBIN}/status/404"
  echo; echo "== 3. Paginate posts with _page and _limit =="
  paginate
  echo; echo "Done."
}

main "$@"
tests/run_tests.sh (5044 bytes)
#!/usr/bin/env bash
# Tests for the Day 027 lab. Run from the lab directory:
#   bash tests/run_tests.sh
#
# Two kinds of checks:
#   * Structure checks always run and must pass (files present, both scripts
#     parse, the example defines the resilient pieces, the starter names its
#     four exercises).
#   * Behavior checks. OFFLINE, the network is SKIPPED, but the backoff logic
#     is still verified to TERMINATE via a stubbed always-failing call (the
#     --selftest path uses no network). ONLINE, the example is run fast and its
#     backoff must give up (not loop forever) and its pagination must collect
#     the expected count.
#
# Every script run is wrapped in a watchdog: if it does not finish within the
# time limit it is killed and the check FAILS — this is how we prove the retry
# loop can never spin forever. The script exits 0 when no check failed.
set -u

lab_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
example="${lab_dir}/examples/resilient_client.sh"
starter="${lab_dir}/starter/resilient_client.sh"
worksheet="${lab_dir}/starter/resilience-worksheet.md"
PROBE="https://example.com"

failures=0
checks=0
skips=0
pass() { checks=$((checks + 1)); echo "  ok:   $1"; }
fail() { checks=$((checks + 1)); failures=$((failures + 1)); echo "  FAIL: $1"; }
skip() { skips=$((skips + 1)); echo "  skip: $1"; }

# run_guarded <seconds> <outfile> <command...>  -> returns command's exit code,
# or 137 if the watchdog had to kill it (i.e. it did not terminate in time).
run_guarded() {
  local secs="$1" outf="$2"; shift 2
  "$@" > "${outf}" 2>&1 &
  local pid=$!
  ( sleep "${secs}"; kill -9 "${pid}" 2>/dev/null ) &
  local watchdog=$!
  wait "${pid}" 2>/dev/null; local rc=$?
  kill "${watchdog}" 2>/dev/null; wait "${watchdog}" 2>/dev/null
  return "${rc}"
}

echo "== Structure checks =="
[ -f "${example}" ] && pass "example script exists" || fail "example script missing"
[ -f "${starter}" ] && pass "starter script exists" || fail "starter script missing"
[ -f "${worksheet}" ] && pass "resilience worksheet exists" || fail "worksheet missing"

for needle in "backoff_retry" "MAX_ATTEMPTS" "Retry-After" "giving up gracefully" "_page=" "_limit="; do
  if grep -q -- "${needle}" "${example}"; then pass "example uses '${needle}'"; else fail "example missing '${needle}'"; fi
done

ex_count="$(grep -cE 'Exercise [1-4]' "${starter}" 2>/dev/null || true)"
if [ "${ex_count:-0}" -ge 4 ]; then pass "starter names its four exercises"; else fail "starter should name 4 exercises (found ${ex_count:-0})"; fi

bash -n "${example}" 2>/dev/null && pass "example has valid bash syntax" || fail "example has a syntax error"
bash -n "${starter}" 2>/dev/null && pass "starter has valid bash syntax" || fail "starter has a syntax error"

echo
echo "== Backoff termination (always, no network needed) =="
# The offline self-test drives the retry loop with a stubbed always-503 server
# and MUST terminate on its own well within the watchdog window.
out="$(mktemp)"
run_guarded 30 "${out}" bash "${example}" --selftest
rc=$?
if [ "${rc}" -eq 137 ]; then
  fail "backoff self-test did NOT terminate (watchdog had to kill it — infinite loop!)"
elif grep -q "giving up gracefully" "${out}" && grep -q "loop terminated" "${out}"; then
  pass "backoff self-test terminates and gives up gracefully (no network)"
else
  fail "backoff self-test did not report a graceful give-up"
  sed 's/^/    /' "${out}"
fi
rm -f "${out}"

echo
echo "== Live behavior checks =="
if ! curl -s -o /dev/null --max-time 12 "${PROBE}" 2>/dev/null; then
  skip "no network access — skipping live 429/pagination checks (expected offline)"
else
  pass "network reachable (${PROBE} responded)"
  out="$(mktemp)"
  # Run fast: tiny delays so the capped backoff finishes quickly. The watchdog
  # still guarantees termination even if the test server is slow.
  # Short per-request timeout + few stable-retries keeps the run bounded even
  # when the shared test server is slow, so the watchdog only ever fires on a
  # genuine non-terminating loop (not on a slow network).
  MAX_ATTEMPTS=3 BASE_DELAY=0 DELAY_CAP=0 CURL_MAX_TIME=4 STABLE_TRIES=2 \
    run_guarded 90 "${out}" bash "${example}"
  rc=$?
  if [ "${rc}" -eq 137 ]; then
    fail "example did NOT terminate within 90s (watchdog killed it)"
    sed 's/^/    /' "${out}"
  else
    pass "example run terminated on its own"
    grep -q "giving up gracefully" "${out}" \
      && pass "backoff gave up gracefully after its cap" \
      || fail "expected a 'giving up gracefully' line"
    collected="$(sed -n 's/.*collected \([0-9]\{1,\}\) posts across 2 pages/\1/p' "${out}" | head -1)"
    case "${collected}" in
      10) pass "pagination collected 10 posts across 2 pages" ;;
      '' ) skip "pagination count not found — test server (jsonplaceholder) may be transiently down" ;;
      *) fail "pagination should collect 10 posts (got '${collected}')" ;;
    esac
  fi
  rm -f "${out}"
fi

echo
echo "${checks} checks, ${failures} failure(s), ${skips} skip(s)."
[ "${failures}" -eq 0 ]

Troubleshooting

Troubleshooting — Day 027 lab

The test server returns 503 or 504, or times out, instead of the status I asked for

This is expected, and it is the whole point of the lesson. httpbin.org is a free, shared service; when it is under load its gateway returns 502/503/ 504 or simply times out, even for an endpoint like /status/429. A resilient client must survive exactly this — and yours does: the backoff loop treats a 503 the same as a 429 (both retryable) and keeps its cap, and the decide step rides through transient gateway 5xx/timeouts to read the endpoint's real, deliberate status. If a run shows 503 where you expected 429, that is a real teaching moment, not a bug. Re-run in a minute for a cleaner picture, or note the 503 — the client handled it correctly either way.

command not found: python3

The pagination step uses python3 only to count items in a JSON array. Install it (macOS: brew install python; Debian/Ubuntu: sudo apt install python3), or adapt the counter. See requirements/README.md.

A run seems to hang

It should not — every request uses --max-time, and every retry loop has a hard cap (MAX_ATTEMPTS). If your edited starter removed the --max-time flag or the cap, put them back: a timeout turns a hung request into a handled failure, and a cap turns a retry loop into one that always terminates. The tests wrap each run in a watchdog that kills and fails any run that does not finish in time, precisely to catch an accidental infinite loop.

curl: (6) Could not resolve host / (7) Failed to connect

You are offline or behind a restrictive network. That is a network failure, not a bug. The script detects it, runs the offline backoff self-test, and exits 0. Reconnect and re-run for the live demos.

The tests print skip: lines

Skips are not failures. Offline, the live network checks are skipped by design. If jsonplaceholder.typicode.com is transiently down, the pagination-count check skips rather than failing — an external outage is not your bug. The suite still exits 0 as long as no structural or termination check failed.

The backoff waited far longer than I expected

Two causes. First, if the server sends a Retry-After header, the client obeys it exactly (that can be seconds or more) instead of using its own delay — this is correct behavior. Second, the default base delay doubles each attempt up to the cap (1, 2, 4, 8, 8 s), so five attempts can wait ~23 s before giving up. Set BASE_DELAY and DELAY_CAP lower to move faster while experimenting: BASE_DELAY=0 DELAY_CAP=0 bash examples/resilient_client.sh.

Security notes

Security notes — Day 027 lab

This lab makes only outbound HTTPS GET requests to two well-known public test servers, needs no API key, no account, and no elevated privileges, and writes nothing outside its own console output (plus a short-lived temp file for response headers, which it deletes). Still, the topic itself is about being a good citizen of other people's servers, so the security lessons are the point.

Be a polite client

  • Honor rate limits. When a server returns 429, back off — do not hammer it harder. When it sends a Retry-After header, wait exactly that long; you cannot guess better than the server just told you.
  • Always add jitter. A fleet of clients that all retry at the same instant is a self-inflicted denial-of-service (the "thundering herd"). A small random component on every wait spreads the load and lets the server recover.
  • Always cap your retries. An uncapped retry loop against a struggling server looks exactly like an attack and can get your IP or key blocked. This lab's loop has a hard MAX_ATTEMPTS and a per-request --max-time for exactly this reason.
  • Identify yourself honestly in real projects (a descriptive User-Agent) and read each API's terms — some forbid scraping or set explicit request ceilings.

Handle errors and their bodies carefully

  • Do not log secrets. Error responses sometimes include internal details, tokens, or user data. Log what you need to debug and redact the rest; never paste a raw error dump containing credentials into a public issue.
  • Never retry a non-idempotent write blindly. Retrying a "create" or "charge" request after a timeout can duplicate the action, because the first attempt may have succeeded even though the response never arrived. Use an idempotency key or do not retry it.

What this lab does not do

  • No credentials are sent or stored.
  • No inbound connections are opened; nothing listens on a port.
  • The only local write is a temporary header file created with mktemp and removed immediately after use.