Computing Foundations › How the Internet Works › Day 21
Day 21: Inspecting Traffic with curl and Developer Tools
After this lesson you will be able to see any HTTP request and response on the wire — with curl and browser Developer Tools — and follow a systematic decision tree to diagnose exactly where a failed request went wrong.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-021-inspecting-traffic-with-curl-and-developer
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
cd labs/sections/computing-foundations/day-021-inspecting-traffic-with-curl-and-developer - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Use curl to inspect a full request and response with -v, and read the request (>), response header (<), and connection (*) lines
- Apply the essential curl flags — -I, -L, -H, -X, -d, -w, -o, -s, --resolve — each to reveal or control one named part of an HTTP exchange
- Read the Developer Tools Network tab: interpret the Status, Type, Size, Time, and Waterfall columns and drill into a single request's headers, payload, and timing
- Follow a redirect chain with -L and read status codes deliberately, distinguishing 2xx success from 4xx client errors and 5xx server errors
- Walk a systematic failed-request decision tree — DNS, connection, TLS, status, body — naming the tool that answers each gate
- Choose the right inspection tool for a task, comparing curl, HTTPie, Postman, Wireshark, and mitmproxy on when each earns its place
- Reproduce a failing API call outside your own code with curl to isolate whether the fault is yours or the server's
Prerequisites
- Days 15-20: how a page loads, IP/DNS/routing, TCP/UDP/ports, HTTP methods, and HTTPS/TLS
- A terminal with curl (preinstalled on macOS and Linux) and any modern web browser
- Comfort running commands in a terminal (Days 8-14)
Why this matters
-
This is the week’s payoff. Days 15 to 20 explained the machinery; today you learn to look at it.
-
“The API call failed” is not a diagnosis. By the end of today it will be one of five specific, distinguishable failures.
-
curl -vshows every layer at once — DNS, TCP, TLS, request, response — in one screen. -
Reproducing a failure outside your code is the fastest way to find out whether your code is even involved.
-
You will use these two tools for the rest of your career, on every service you ever build or consume.
The idea in plain language
-
Inspecting traffic means composing a request from named parts and reading a response by its named parts.
-
curlis for one request, precisely controlled and endlessly repeatable. -
Developer tools are for a whole page — every request it made, in one timeline.
-
Neither guesses. They show you what was actually sent and what actually came back.
-
The skill is not the tools. It is asking the failure questions in the right order.
Historical background
-
curlwas released in 1998, written by Daniel Stenberg to fetch currency rates from a script. -
It now ships on essentially every operating system, including Windows, and runs in cars, phones and televisions — one of the most widely deployed programs ever written.
-
Its companion library, libcurl, is what most other tools use underneath, which is why their behaviour matches.
-
Browser developer tools arrived later. Firebug, a Firefox extension from 2006, established the shape; browsers built it in afterwards.
-
The network panel has barely changed since. The columns you will read today are the columns from 2010, because they were the right ones.
What it is — and what it is not
They are:
- Instruments. They report what happened, without opinion.
- Complementary.
curlisolates one request; developer tools show the whole page. - Exact.
curlsends precisely what you asked, which is the point.
They are not:
-
Not a substitute for reading the error. Most failures announce themselves clearly, and are read past.
-
Not identical to your program.
curlhas its own trust store, its own redirects policy, its own defaults — differences that are themselves informative. -
Not a load tester. Measuring one request tells you about latency, not capacity.
-
Not safe to paste publicly. A copied
curlcommand carries your live credentials.
Why it was created and what problems it solves
-
Isolating the variable. Was it your code, your network, the server, or the request itself? Nothing else answers this as fast.
-
Reproducibility. A
curlcommand is a bug report anyone can run. -
Automation. Anything you can inspect, you can script and schedule.
-
Seeing the invisible. A browser hides redirects, retries, caching and compression; these tools show them.
-
Measuring, not guessing.
-wturns “it feels slow” into five numbers, each pointing at a different stage.
How it works
The anatomy of a curl command
-
Bare
curl URLmakes a GET and prints the body. -
-vis the debugging view. Lines beginning>are what was sent,<are the response headers, and*are curl’s own notes about resolving, connecting and the TLS handshake. -
In one command you see every layer from this week, laid out in order.
| Flag | What it does | Why you reach for it |
|---|---|---|
-v | The full conversation | The default debugging view |
-I | Headers only (HEAD) | Status and headers, no body download |
-L | Follow redirects | A 301 is not the answer; chase it |
-H | Set a request header | Servers answer differently based on them |
-X | Set the method | Exercise methods a browser will not send |
-d | Send a body; implies POST | Reproduce your code’s exact payload |
-w | Print chosen variables after transfer | Just the status, or a timing breakdown |
-o / -s | Body to a file / silence the meter | Keep output clean |
--resolve | Force a hostname to an address | Test a new server before DNS points at it |
-
-wreads timing variables curl measures on every transfer:time_namelookup,time_connect,time_appconnect,time_starttransfer— time to first byte — andtime_total. -
That turns curl into a stopwatch that says which stage was slow.
-
-cand-bsave and send cookies, letting you reproduce a session a plain request would not have.
Reading the network panel
Open developer tools (F12, or Command-Option-I on a Mac) and reload with the Network tab open. Every request is a row.
| Column | What it tells you | What to watch for |
|---|---|---|
| Name | The resource requested | Finding the one you care about |
| Status | The HTTP status code | Red rows first, always |
| Type | Document, script, fetch, image, font | Filter to Fetch/XHR for API calls only |
| Initiator | What caused the request | Tracing a mystery request to its code |
| Size | Bytes transferred, and cache hits | A surprisingly large or uncached download |
| Time | End to end duration | The slow request in a slow page |
| Waterfall | A timeline bar per request | Where the time actually went |
-
Click a row for Headers, Payload, Response and Timing — the same phases
-wreports, per request. -
The waterfall assembles them into one picture, so a long bar or a late start jumps out.
-
Filter by type to hide everything but the API calls your application made.
A systematic method
Ask these in order, and stop at the first “no”:
-
Did the name resolve? If not, it is DNS. Check for typos; confirm with
dig. -
Did the connection open? If not, the service is down, or a port or firewall is blocking you.
-
Did TLS succeed? If not, the certificate is expired, untrusted, or for the wrong name.
-
What status came back?
4xxmeans your request was wrong;5xxmeans the server failed handling a valid one. -
Is the body the shape you expected? Only ask this once you have a
2xx.
- Walking the gates in order means you never rewrite request-sending code to fix
a DNS problem, and never blame a server for a
4xxyou caused.
An everyday analogy
A doctor working through a differential diagnosis.
| The clinic | The request |
|---|---|
| ”Where does it hurt?” | Which stage failed |
| Taking a pulse | ping and dig |
| An X-ray | curl -v |
| A full panel of tests | The network waterfall |
| Ruling things out in order | The decision tree |
| Not prescribing before diagnosing | Not editing code before knowing the stage |
-
A doctor does not start treatment before the diagnosis. Rewriting your client code before you know the failure stage is exactly that.
-
The cheapest tests come first.
digcosts a second; reading a stack trace costs twenty minutes.
Examples in practice
-
curl: (6) Could not resolve host— stage one. Nothing else has been attempted. -
curl: (7) Failed to connect— the name resolved and nothing accepted the connection. -
curl: (60) SSL certificate problem— stage three, and the message names which check failed. -
401with a correct-looking key — comparecurl -voutput against your code’s actual headers. It is usually a missingBearerprefix. -
A request that works in
curland fails in the browser — almost always CORS, whichcurldoes not enforce and browsers do.
Implications: security, privacy, performance, scalability, and cost
-
Security: “Copy as cURL” copies your credentials. That command contains live session cookies and tokens. Redact before sharing, every time.
-
A
curlcommand in your shell history is a credential in a file, readable by anything that can read your home directory. -
Never paste a verbose output into an issue tracker unredacted. The
Authorizationheader is right there in the>lines. -
Privacy: developer tools show third-party requests. Opening the network panel on a normal site is an education in who else is being told about your visit.
-
Performance: measure before optimising, and measure the right stage. The timing breakdown exists so you do not guess.
-
Cost: reproducing with
curlis free; guessing is not. The expensive debugging session is the one that changes code before identifying the stage.
Alternatives: free, open source, and commercial
| Tool | Type | Where it fits |
|---|---|---|
curl | Free, open source | Everywhere. Present on every machine you will touch. |
httpie | Free, open source | Friendlier syntax and colour; excellent for exploring |
| Browser devtools | Bundled, free | Whole-page view, timings, and the initiator chain |
| Postman / Insomnia | Free tiers | Saved collections, environments, team sharing |
mitmproxy | Free, open source | Watching and rewriting traffic from apps you control |
| Wireshark | Free, open source | Below HTTP, when the problem is TCP or TLS itself |
-
Learn
curlfirst even if you prefer a GUI. It is what documentation, error messages and colleagues will speak. -
Reach for
mitmproxywhen the client is not yours to instrument — a mobile app, a closed binary, a container you cannot modify.
Comparison with related concepts
| Term | What it actually is |
|---|---|
| Verbose output | curl’s full record of the exchange and its own steps |
| Waterfall | A timeline of every request a page made |
| Initiator | What caused a request to be made |
| Time to first byte | When the response began, not when it finished |
| Preflight | A browser’s OPTIONS check before a cross-origin request |
| Copy as cURL | A devtools command that reproduces a request — with credentials |
When to use it — and when not to
-
Reach for
curlwhen you can name the one request that is failing. -
Reach for developer tools when you cannot — when you do not yet know which request is the problem.
-
Reproduce in
curlbefore editing any code. Ifcurlsucceeds and your code fails, the difference is in your code, and you have halved the search. -
Use
--resolveto test a server before DNS points at it, rather than editing your hosts file and forgetting. -
Do not use
-kto make an error go away. It is stage three telling you something true. -
Do not paste a
curlcommand with real credentials anywhere, including into a model.
The AI connection
-
Every model provider’s documentation opens with a
curlexample. It is the reference implementation, and running it first tells you whether your key works before any code exists. -
401is your key;429is your rate;400is your payload. Three codes cover most of what goes wrong. -
Reproduce a failing generation with
curland you learn immediately whether the problem is your client library or the request itself. -
Streaming responses are visible in
curlwithout-s, arriving in chunks — which makes the mechanism concrete rather than magical. -
Timing a model call with
-wseparates network latency from generation time, and the two need entirely different fixes.
Knowledge check
quiz:
question: "A request succeeds from the command line with curl but fails from JavaScript in the browser, with the same URL, method and headers. What is the most likely cause?"
options:
- "The browser is enforcing cross-origin rules that curl does not implement"
- "The browser is sending a different HTTP version than curl negotiated"
- "curl is ignoring a certificate error the browser correctly rejects"
- "The server is rate-limiting the browser but not the command line client"
answer_index: 0
explanation: "Cross-origin restrictions are enforced by browsers as a protection for the user, not by servers. curl has no origin and no same-origin policy, so it is unaffected — which is exactly why a request can work there and be blocked in a page."
Hands-on exercise
Reproduce every stage of failure deliberately, then read a real page’s waterfall. Worked through fully in the Day 21 lab directory.
| What you are doing | Command |
|---|---|
| See the whole exchange | curl -v https://example.com |
| Headers only | curl -I https://example.com |
| Follow a redirect | curl -IL http://example.com |
| Send JSON | curl -X POST -H 'Content-Type: application/json' -d '{"a":1}' URL |
| Time every stage | curl -w "@fmt.txt" -o /dev/null -s URL |
| Fail at DNS | curl -v https://no-such-host.invalid |
| Fail at TLS | curl -v https://expired.badssl.com |
Expected output
* Trying 93.184.216.34:443...
* Connected to example.com port 443
* SSL certificate verify ok.
> GET / HTTP/1.1
< HTTP/1.1 200 OK
time_namelookup: 0.024 time_connect: 0.061
time_appconnect: 0.128 time_starttransfer: 0.171
*lines are curl’s own notes,>is what it sent,<is what came back.- Subtract neighbouring timings to get each stage’s cost, as on Day 15.
Validate your work
-
You produced a failure at DNS, at connection, and at TLS, and can tell them apart from the error alone
-
You reproduced a
POSTwith a JSON body from the command line -
You read a timing breakdown and named the slowest stage
-
You found a page’s slowest request in the waterfall
-
You traced one request back to the script that fired it, using Initiator
-
bash tests/run_tests.shreports 0 failures
Troubleshooting
-
curlprints nothing. The body may be empty. Add-ito see the status and headers. -
Your JSON is rejected. Shell quoting. Single-quote the whole payload, or use
--data-binary @file.json. -
-Ireturns405. Some servers reject HEAD. Use-s -o /dev/null -D -instead. -
The network panel is empty. It only records while open. Reload with it open.
-
Too many rows to read. Filter by
Fetch/XHR, and tick “Disable cache”.
Common mistakes
- Editing code before identifying the stage.
- Sharing a
curlcommand with live credentials in it. - Forgetting
-Land treating a301as the final answer. - Measuring one request and concluding something about capacity.
Practice assignment
-
Walk the decision tree on three deliberately broken requests and record which gate each one failed at, with the exact error text.
-
Convert one request from the network panel into a
curlcommand with “Copy as cURL”, then strip its credentials until it fails, to see which header mattered. -
Time the same endpoint from two networks and tabulate all five phases.
-
Find one third-party request on a site you use daily, and trace who it goes to.
Extension challenge
-
Write a health-check script that walks the whole decision tree automatically and reports which gate failed. Schedule it with Day 14’s cron.
-
Reproduce a CORS failure, read the preflight
OPTIONSrequest in the network panel, and explain exactly which response header was missing. -
Compare
curl’s trust store with your browser’s by finding a certificate one accepts and the other rejects. -
Capture the same request in
mitmproxyand compare what it shows againstcurl -v. The difference is the part your client was doing silently.
Quiz
Q1. In the output of `curl -v`, what do lines beginning with `>` represent?
- The response headers the server returned
- The request headers curl sent to the server
- curl's own notes about resolving and connecting
- The body of the downloaded page
Show answer
Answer: B. The request headers curl sent to the server
In verbose output, `>` marks the request curl sent, `<` marks the response headers that came back, and `*` marks connection notes (DNS, TCP, TLS). Reading the three prefixes separately is the core skill of a verbose trace.
Q2. You request a URL and get back a `302`. What does adding `-L` do?
- It prints the response body instead of only the headers
- It retries the request until the server returns a 200
- It follows the redirect in the location header to the final destination
- It logs the request to a file for later inspection
Show answer
Answer: C. It follows the redirect in the location header to the final destination
A `301`/`302` is a forwarding address, not the final answer. `-L` tells curl to follow the `location` header to the real destination; without it you stop at the redirect and may wrongly conclude the endpoint returned nothing.
Q3. An API call from your code returns `429 Too Many Requests`. What does it mean?
- The server crashed while handling your valid request
- The URL does not exist on the server
- Your credentials are missing or invalid
- You are sending requests too fast and should back off
Show answer
Answer: D. You are sending requests too fast and should back off
`429` is a `4xx` client error meaning you have exceeded a rate limit. The fix is on your side: reduce request frequency or add a back-off delay, not rewrite the server or the endpoint.
Q4. Which flag makes curl print only the response headers using a HEAD request?
- `-I`
- `-d`
- `-X`
- `-o`
Show answer
Answer: A. `-I`
`-I` fetches headers only (a HEAD request) and prints the status line and response headers without downloading the body — useful for checking status and headers quickly.
Q5. In the failed-request decision tree, what does a `5xx` status code tell you compared with a `4xx`?
- `5xx` means DNS failed; `4xx` means TLS failed
- `5xx` is the server failing; `4xx` is your request being wrong
- `5xx` is always a temporary problem; `4xx` is always permanent
- They mean the same thing and can be treated identically
Show answer
Answer: B. `5xx` is the server failing; `4xx` is your request being wrong
A `4xx` means the client sent something wrong — bad URL, missing authentication, malformed body — so fix your request. A `5xx` means the server erred on a request it accepted. Blaming the wrong side wastes debugging time.
Q6. Which curl option lets you print a custom value such as the status code after a transfer?
- `-s`
- `-H`
- `-w`
- `-L`
Show answer
Answer: C. `-w`
`-w` (write-out) prints chosen variables after the transfer, such as `%{http_code}` for the status or timing variables like `%{time_total}`. Combined with `-o /dev/null` and `-s`, it extracts just the value you want.
Q7. In the Developer Tools Network tab, which filter shows only the API calls a page made?
- The `Img` filter
- The `CSS` filter
- The `Doc` filter
- The `Fetch/XHR` filter
Show answer
Answer: D. The `Fetch/XHR` filter
The `Fetch/XHR` type filter hides documents, scripts, styles, and images, leaving only the background data requests (the API calls) so you can focus on the request you are debugging.
Q8. When would you reach for Wireshark instead of curl or the Network tab?
- When the problem may be below HTTP, in packets or the connection
- When you want to save and share a collection of requests with a team
- When you want friendlier, coloured output for an ad-hoc request
- When you need to follow a redirect to its final destination
Show answer
Answer: A. When the problem may be below HTTP, in packets or the connection
Wireshark captures raw network packets, a full layer below the assembled HTTP messages that curl and the Network tab read. You reach for it in the rarer cases where the fault is beneath HTTP — in the TCP segments, the DNS queries, or the connection.
Glossary
- curl
- A command-line program that makes a single HTTP request you describe and prints the response, giving you precise control over and full visibility into one exchange.
- verbose mode
- curl's `-v` option, which prints the entire conversation: the request sent (lines starting with >), the response headers (lines starting with <), and connection notes (lines starting with *).
- header
- A named field of metadata attached to an HTTP request or response, such as Accept, Content-Type, or Authorization, that carries information alongside the body.
- redirect
- A response (status 301 or 302) that points the client to a different URL via a location header rather than returning the content directly; `curl -L` follows it to the final destination.
- DevTools
- A browser's built-in Developer Tools: a panel for inspecting a page's structure, console, performance, and network activity, opened with F12 or Command-Option-I.
- Network tab
- The Developer Tools panel that records every request a page makes, showing each one's status, type, size, timing, and full headers and body.
- waterfall
- A timeline view in the Network tab where each request is a horizontal bar showing when it started and how long each phase took, making slow or late requests easy to spot.
- payload
- The body of a request or response — the actual data being sent or returned — as distinct from the headers that describe it.
- User-Agent
- A request header identifying the client software making the request; curl sends a value like `curl/8.7.1`, while a browser sends a long string naming itself and its version.
- httpie
- A free, open-source command-line HTTP client, an alternative to curl designed for human readability with coloured, formatted output and a simpler syntax.
- proxy
- An intermediary that sits between a client and a server and relays their traffic; a debugging proxy such as mitmproxy records every request that passes through it, revealing what an application is sending.
- status code
- A three-digit number in an HTTP response signalling the outcome: 2xx success, 3xx redirection, 4xx a client (request) error, and 5xx a server error.
- time to first byte
- The elapsed time from sending a request to receiving the first byte of the response, reported by curl as `time_starttransfer` and by the Network tab in a request's Timing sub-tab.
Sources and further reading
- curl Documentation — curl / Daniel Stenberg (accessed 2026-07-12) — The official documentation for curl, including the manual page for every flag used in this lesson.
- Everything curl — Daniel Stenberg (accessed 2026-07-12) — The book-length, freely available guide to curl and libcurl by its lead developer.
- Inspect network activity — Chrome DevTools — Google (accessed 2026-07-12) — Official reference for the Network tab: columns, filters, timing, and the waterfall.
- HTTP overview — MDN Web Docs (accessed 2026-07-12) — Background on HTTP messages, methods, headers, and status codes.
- httpbin — HTTP Request & Response Service — Kenneth Reitz (accessed 2026-07-12) — The public test service the lab uses to echo requests and return chosen status codes.
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.