Computing Foundations › How the Internet Works › Day 21

Day 21: Inspecting Traffic with curl and Developer Tools

Day 21 of 365 — Inspecting Traffic with curl and Developer Tools

After this lesson you will be able to see any HTTP request and response on the wire — with curl and browser Developer Tools — and follow a systematic decision tree to diagnose exactly where a failed request went wrong.

Course
Computing Foundations
Category
How the Internet Works
Reading time
≈ 40 min
Practical time
≈ 30 min
Lesson duration
1h 10m
Last verified
2026-07-12

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-021-inspecting-traffic-with-curl-and-developer

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/computing-foundations/day-021-inspecting-traffic-with-curl-and-developer
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters


The idea in plain language

Diagram: curl compared with browser developer tools


Historical background


What it is — and what it is not

They are:

They are not:


Why it was created and what problems it solves


How it works

Diagram: narrowing an API failure from the application down to the network

The anatomy of a curl command

Diagram: what each prefix in curl verbose output means

Diagram: the anatomy of a curl command, each flag labelled with its job, and the request and response it produces

FlagWhat it doesWhy you reach for it
-vThe full conversationThe default debugging view
-IHeaders only (HEAD)Status and headers, no body download
-LFollow redirectsA 301 is not the answer; chase it
-HSet a request headerServers answer differently based on them
-XSet the methodExercise methods a browser will not send
-dSend a body; implies POSTReproduce your code’s exact payload
-wPrint chosen variables after transferJust the status, or a timing breakdown
-o / -sBody to a file / silence the meterKeep output clean
--resolveForce a hostname to an addressTest a new server before DNS points at it

Reading the network panel

Diagram: four recognisable patterns in a network waterfall

Open developer tools (F12, or Command-Option-I on a Mac) and reload with the Network tab open. Every request is a row.

ColumnWhat it tells youWhat to watch for
NameThe resource requestedFinding the one you care about
StatusThe HTTP status codeRed rows first, always
TypeDocument, script, fetch, image, fontFilter to Fetch/XHR for API calls only
InitiatorWhat caused the requestTracing a mystery request to its code
SizeBytes transferred, and cache hitsA surprisingly large or uncached download
TimeEnd to end durationThe slow request in a slow page
WaterfallA timeline bar per requestWhere the time actually went

A systematic method

Diagram: the five ordered checks for diagnosing a failed request

Flowchart: a troubleshooting decision tree for a failed request, from DNS to connection to TLS to status code to body

Ask these in order, and stop at the first “no”:

  1. Did the name resolve? If not, it is DNS. Check for typos; confirm with dig.

  2. Did the connection open? If not, the service is down, or a port or firewall is blocking you.

  3. Did TLS succeed? If not, the certificate is expired, untrusted, or for the wrong name.

  4. What status came back? 4xx means your request was wrong; 5xx means the server failed handling a valid one.

  5. Is the body the shape you expected? Only ask this once you have a 2xx.


An everyday analogy

Diagram: the curl flags that control output

A doctor working through a differential diagnosis.

The clinicThe request
”Where does it hurt?”Which stage failed
Taking a pulseping and dig
An X-raycurl -v
A full panel of testsThe network waterfall
Ruling things out in orderThe decision tree
Not prescribing before diagnosingNot editing code before knowing the stage

Examples in practice


Implications: security, privacy, performance, scalability, and cost

Diagram: what a copied curl command contains and how to share it safely


Alternatives: free, open source, and commercial

Diagram: curl's timing variables and the stage each measures

ToolTypeWhere it fits
curlFree, open sourceEverywhere. Present on every machine you will touch.
httpieFree, open sourceFriendlier syntax and colour; excellent for exploring
Browser devtoolsBundled, freeWhole-page view, timings, and the initiator chain
Postman / InsomniaFree tiersSaved collections, environments, team sharing
mitmproxyFree, open sourceWatching and rewriting traffic from apps you control
WiresharkFree, open sourceBelow HTTP, when the problem is TCP or TLS itself

TermWhat it actually is
Verbose outputcurl’s full record of the exchange and its own steps
WaterfallA timeline of every request a page made
InitiatorWhat caused a request to be made
Time to first byteWhen the response began, not when it finished
PreflightA browser’s OPTIONS check before a cross-origin request
Copy as cURLA devtools command that reproduces a request — with credentials

When to use it — and when not to

Diagram: what it means when curl succeeds and your code fails


The AI connection

Diagram: the three most common AI API status codes and their causes


Knowledge check

quiz:
  question: "A request succeeds from the command line with curl but fails from JavaScript in the browser, with the same URL, method and headers. What is the most likely cause?"
  options:
    - "The browser is enforcing cross-origin rules that curl does not implement"
    - "The browser is sending a different HTTP version than curl negotiated"
    - "curl is ignoring a certificate error the browser correctly rejects"
    - "The server is rate-limiting the browser but not the command line client"
  answer_index: 0
  explanation: "Cross-origin restrictions are enforced by browsers as a protection for the user, not by servers. curl has no origin and no same-origin policy, so it is unaffected — which is exactly why a request can work there and be blocked in a page."

Hands-on exercise

Reproduce every stage of failure deliberately, then read a real page’s waterfall. Worked through fully in the Day 21 lab directory.

What you are doingCommand
See the whole exchangecurl -v https://example.com
Headers onlycurl -I https://example.com
Follow a redirectcurl -IL http://example.com
Send JSONcurl -X POST -H 'Content-Type: application/json' -d '{"a":1}' URL
Time every stagecurl -w "@fmt.txt" -o /dev/null -s URL
Fail at DNScurl -v https://no-such-host.invalid
Fail at TLScurl -v https://expired.badssl.com

Expected output

*   Trying 93.184.216.34:443...
* Connected to example.com port 443
* SSL certificate verify ok.
> GET / HTTP/1.1
< HTTP/1.1 200 OK
time_namelookup: 0.024  time_connect: 0.061
time_appconnect: 0.128  time_starttransfer: 0.171

Validate your work

Troubleshooting

Common mistakes


Practice assignment


Extension challenge


Quiz

Q1. In the output of `curl -v`, what do lines beginning with `>` represent?

  1. The response headers the server returned
  2. The request headers curl sent to the server
  3. curl's own notes about resolving and connecting
  4. The body of the downloaded page
Show answer

Answer: B. The request headers curl sent to the server

In verbose output, `>` marks the request curl sent, `<` marks the response headers that came back, and `*` marks connection notes (DNS, TCP, TLS). Reading the three prefixes separately is the core skill of a verbose trace.

Q2. You request a URL and get back a `302`. What does adding `-L` do?

  1. It prints the response body instead of only the headers
  2. It retries the request until the server returns a 200
  3. It follows the redirect in the location header to the final destination
  4. It logs the request to a file for later inspection
Show answer

Answer: C. It follows the redirect in the location header to the final destination

A `301`/`302` is a forwarding address, not the final answer. `-L` tells curl to follow the `location` header to the real destination; without it you stop at the redirect and may wrongly conclude the endpoint returned nothing.

Q3. An API call from your code returns `429 Too Many Requests`. What does it mean?

  1. The server crashed while handling your valid request
  2. The URL does not exist on the server
  3. Your credentials are missing or invalid
  4. You are sending requests too fast and should back off
Show answer

Answer: D. You are sending requests too fast and should back off

`429` is a `4xx` client error meaning you have exceeded a rate limit. The fix is on your side: reduce request frequency or add a back-off delay, not rewrite the server or the endpoint.

Q4. Which flag makes curl print only the response headers using a HEAD request?

  1. `-I`
  2. `-d`
  3. `-X`
  4. `-o`
Show answer

Answer: A. `-I`

`-I` fetches headers only (a HEAD request) and prints the status line and response headers without downloading the body — useful for checking status and headers quickly.

Q5. In the failed-request decision tree, what does a `5xx` status code tell you compared with a `4xx`?

  1. `5xx` means DNS failed; `4xx` means TLS failed
  2. `5xx` is the server failing; `4xx` is your request being wrong
  3. `5xx` is always a temporary problem; `4xx` is always permanent
  4. They mean the same thing and can be treated identically
Show answer

Answer: B. `5xx` is the server failing; `4xx` is your request being wrong

A `4xx` means the client sent something wrong — bad URL, missing authentication, malformed body — so fix your request. A `5xx` means the server erred on a request it accepted. Blaming the wrong side wastes debugging time.

Q6. Which curl option lets you print a custom value such as the status code after a transfer?

  1. `-s`
  2. `-H`
  3. `-w`
  4. `-L`
Show answer

Answer: C. `-w`

`-w` (write-out) prints chosen variables after the transfer, such as `%{http_code}` for the status or timing variables like `%{time_total}`. Combined with `-o /dev/null` and `-s`, it extracts just the value you want.

Q7. In the Developer Tools Network tab, which filter shows only the API calls a page made?

  1. The `Img` filter
  2. The `CSS` filter
  3. The `Doc` filter
  4. The `Fetch/XHR` filter
Show answer

Answer: D. The `Fetch/XHR` filter

The `Fetch/XHR` type filter hides documents, scripts, styles, and images, leaving only the background data requests (the API calls) so you can focus on the request you are debugging.

Q8. When would you reach for Wireshark instead of curl or the Network tab?

  1. When the problem may be below HTTP, in packets or the connection
  2. When you want to save and share a collection of requests with a team
  3. When you want friendlier, coloured output for an ad-hoc request
  4. When you need to follow a redirect to its final destination
Show answer

Answer: A. When the problem may be below HTTP, in packets or the connection

Wireshark captures raw network packets, a full layer below the assembled HTTP messages that curl and the Network tab read. You reach for it in the rarer cases where the fault is beneath HTTP — in the TCP segments, the DNS queries, or the connection.

Glossary

curl
A command-line program that makes a single HTTP request you describe and prints the response, giving you precise control over and full visibility into one exchange.
verbose mode
curl's `-v` option, which prints the entire conversation: the request sent (lines starting with >), the response headers (lines starting with <), and connection notes (lines starting with *).
header
A named field of metadata attached to an HTTP request or response, such as Accept, Content-Type, or Authorization, that carries information alongside the body.
redirect
A response (status 301 or 302) that points the client to a different URL via a location header rather than returning the content directly; `curl -L` follows it to the final destination.
DevTools
A browser's built-in Developer Tools: a panel for inspecting a page's structure, console, performance, and network activity, opened with F12 or Command-Option-I.
Network tab
The Developer Tools panel that records every request a page makes, showing each one's status, type, size, timing, and full headers and body.
waterfall
A timeline view in the Network tab where each request is a horizontal bar showing when it started and how long each phase took, making slow or late requests easy to spot.
payload
The body of a request or response — the actual data being sent or returned — as distinct from the headers that describe it.
User-Agent
A request header identifying the client software making the request; curl sends a value like `curl/8.7.1`, while a browser sends a long string naming itself and its version.
httpie
A free, open-source command-line HTTP client, an alternative to curl designed for human readability with coloured, formatted output and a simpler syntax.
proxy
An intermediary that sits between a client and a server and relays their traffic; a debugging proxy such as mitmproxy records every request that passes through it, revealing what an application is sending.
status code
A three-digit number in an HTTP response signalling the outcome: 2xx success, 3xx redirection, 4xx a client (request) error, and 5xx a server error.
time to first byte
The elapsed time from sending a request to receiving the first byte of the response, reported by curl as `time_starttransfer` and by the Network tab in a request's Timing sub-tab.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.