Computing Foundations › Systems Foundations: Storage, Observability, and Tooling › Day 40

Day 40: Observability: Logs, Metrics, Traces, and Dashboards

Day 40 of 365 — Observability: Logs, Metrics, Traces, and Dashboards

After this lesson you will be able to explain how a running system reports what it is doing through logs, metrics, and traces, compute a real percentile from raw latencies, and reason about dashboards, alerting, sampling, and cost — the literacy you will reuse to instrument every service you build.

Course
Computing Foundations
Category
Systems Foundations: Storage, Observability, and Tooling
Reading time
≈ 40 min
Practical time
≈ 30 min
Lesson duration
1h 10m
Last verified
2026-07-12

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/computing-foundations/day-040-observability-logs-metrics-traces-and-dashboards

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/computing-foundations/day-040-observability-logs-metrics-traces-and-dashboards
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters


The idea in plain language

Diagram: the question each observability signal answers


Historical background


What it is — and what it is not

Diagram: how a service failure differs from a model failure

Observability is:

It is not:


Why it was created and what problems it solves


How it works

Logs

Diagram: plain text logs compared with structured logs

Plain:      2026-07-12 14:32:07 ERROR request for user 91 failed after 4200ms
Structured: {"ts":"2026-07-12T14:32:07Z","level":"ERROR","event":"request_failed","user_id":91,"latency_ms":4200}

Metrics

Diagram: an average latency compared with percentiles

TypeWhat it measuresDirectionExampleQuestion it answers
CounterA running totalUp onlyTotal requests served”How many per second?” — its rate
GaugeA value that rises and fallsEitherQueue length, memory in use”How full is it right now?”
HistogramA distribution across bucketsUp per bucketRequest latencies”What is the p95?”

Traces

Diagram: how nested spans reveal where a request spent its time

Diagram: the three pillars of observability — logs, metrics, and traces — feeding dashboards and alerts

Dashboards, alerting and service levels

Diagram: alerting on internal conditions compared with alerting on user experience


An everyday analogy

Diagram: how an SLI and an SLO produce a justified alert

A hospital.

The hospitalObservability
The patient’s chartLogs
The vital-signs monitorMetrics
Following one patient through every departmentA trace
The nurses’ station displayA dashboard
The alarm at the bedsideAn alert
”Temperature above 39”An SLI and its threshold

Examples in practice

Flowchart: one request drawn as a trace of nested spans across services, with the time spent in each step


Implications: security, privacy, performance, scalability, and cost

Diagram: low-cardinality metric labels compared with high-cardinality ones


Alternatives: free, open source, and commercial

Diagram: how trace sampling reduces volume while preserving signal

ToolTypeWhat it does
OpenTelemetryFree, open sourceVendor-neutral instrumentation for all three signals
Prometheus + GrafanaFree, open sourceMetrics collection and dashboards; the common default
Loki, ElasticsearchFree, open sourceLog aggregation and search
Jaeger, TempoFree, open sourceTrace storage and exploration
Datadog, Honeycomb, New RelicCommercialAll three, hosted, with a real bill
structlog, zapFree, open sourceStructured logging in your language

TermWhat it actually is
LogA discrete timestamped event
MetricA number sampled over time
TraceOne request’s path, as nested spans
SpanOne unit of work, with a duration
CardinalityHow many distinct values a label can take
SLIA measurement of user experience
SLOThe target you commit to for an SLI

When to use it — and when not to


The AI connection

Diagram: logs, metrics and traces compared with a quality signal


Knowledge check

quiz:
  question: "A service reports a 200 ms average latency and users complain it is slow. What should you look at first?"
  options:
    - "The latency percentiles, since an average can hide a slow tail affecting a minority of requests"
    - "The total request count, to confirm the service is receiving the expected traffic"
    - "The CPU and memory gauges, since resource exhaustion is the usual cause of slowness"
    - "The error rate, since users may be describing failures rather than slowness"
  answer_index: 0
  explanation: "An average is a single number over a distribution and says nothing about its shape. A p95 or p99 reveals whether a minority of requests are far slower — and those users are the ones who notice, complain and leave, which is exactly why percentiles rather than averages describe real experience."

Hands-on exercise

Emit all three signals from a small service, then find a bug with them. Worked through fully in the Day 40 lab directory.

What you are doingCommand or code
Log structurallyEmit JSON with ts, level, event and fields
Query the logsjq 'select(.level=="ERROR" and .latency_ms > 3000)' app.jsonl
Count by fieldjq -r .event app.jsonl | sort | uniq -c | sort -rn
Compute a rateCount over a window, divided by its length
Compute percentilesSort the latencies; take the value at 95%
Add a trace IDOne id per request, on every line it produces
Follow one requestjq 'select(.trace_id=="abc123")' app.jsonl

Expected output

count  event
  842  request_ok
   37  request_failed
mean latency: 204ms
p50: 180ms   p95: 790ms   p99: 4180ms
{"trace_id":"abc123","span":"db_query","ms":3980}

Validate your work

Troubleshooting

Common mistakes


Practice assignment


Extension challenge


Quiz

Q1. What are the three pillars of observability?

  1. Firewalls, backups, and load balancers
  2. Logs, metrics, and traces
  3. Testing, staging, and production
  4. CPU, memory, and disk
Show answer

Answer: B. Logs, metrics, and traces

The three pillars are logs (timestamped events), metrics (numbers over time), and traces (one request followed across services). Together they let you notice, locate, and explain a problem.

Q2. Why is structured (JSON) logging generally preferred over plain-text logging?

  1. It uses less disk space than plain text in every case
  2. It is the only format that operating systems can write
  3. Its fields are machine-readable, so you can query them
  4. It hides sensitive data automatically
Show answer

Answer: C. Its fields are machine-readable, so you can query them

Structured logs record events as machine-readable key-value data, so you can ask precise questions like "every ERROR with latency_ms over 3000" without brittle regular expressions over free-form text.

Q3. Which metric type is best suited to measuring a value that can rise and fall, such as the current length of a queue?

  1. A counter
  2. A trace
  3. A histogram
  4. A gauge
Show answer

Answer: D. A gauge

A gauge measures a value that can go up or down at any moment, such as current memory in use or queue length. A counter only ever increases, and a histogram records the distribution of many measured values.

Q4. A service has an average response time of 550 ms, but its p50 is 140 ms and its p95 is 4200 ms. What does this most likely indicate?

  1. A typical request is fast, but a slow tail pulls the average up
  2. Every request is consistently slow at around 550 ms
  3. The average is the most trustworthy number to report
  4. The percentiles were computed incorrectly
Show answer

Answer: A. A typical request is fast, but a slow tail pulls the average up

The median (p50) of 140 ms shows typical requests are fast, while the p95 of 4200 ms reveals a slow tail. The average is dragged upward by those few outliers, which is exactly why percentiles describe real user experience better than an average.

Q5. In distributed tracing, what is a span?

  1. The total number of servers a system runs on
  2. One unit of work, with a start, duration, and parent
  3. The gap in time between two separate requests
  4. A log line that contains no timestamp
Show answer

Answer: B. One unit of work, with a start, duration, and parent

A span records one unit of work — a database query, a service call, a computation — with its start, duration, and name. Spans nest under a parent span and all share a trace ID, so one request can be reassembled across many machines.

Q6. What is the key difference between monitoring and observability?

  1. Monitoring is free while observability always requires paid tools
  2. Observability only applies to hardware, monitoring only to software
  3. Monitoring asks questions set in advance; observability asks new
  4. They are two names for exactly the same thing
Show answer

Answer: C. Monitoring asks questions set in advance; observability asks new

Monitoring answers predefined questions ("is CPU over 90%?"). Observability is the richer property that, because you emit enough well-structured signal, you can investigate failures you never anticipated when you wrote the code.

Q7. What does the guidance "alert on symptoms, not causes" mean?

  1. Only send alerts about hardware failures
  2. Alert on internal conditions like high CPU rather than on user-facing behavior
  3. Never set up any alerts, because they are always noise
  4. Alert on what the user feels — errors or slowness
Show answer

Answer: D. Alert on what the user feels — errors or slowness

Good alerts fire on symptoms the user experiences, such as a high error rate or slow responses, not on internal causes like high CPU that may not be hurting anyone. Alerting on harmless causes trains teams to ignore their alerts.

Q8. Why do high-volume systems often sample their traces, keeping only a fraction of them?

  1. Storing every trace can cost more than the system itself
  2. Because traces are inaccurate unless sampled
  3. Because sampling makes the system run faster for users
  4. Because traces are illegal to store in full
Show answer

Answer: A. Storing every trace can cost more than the system itself

Telemetry has a real cost to ingest, index, and retain. Keeping, say, one trace in a hundred trades completeness for affordability while still preserving the statistical picture — one of the central cost trade-offs in observability.

Glossary

observability
The extent to which you can understand what is happening inside a running system purely from the signals it emits, including questions you did not think to ask in advance.
log
A timestamped record of a single event that happened in a system, such as a request completing or an error occurring — the diary of the system, one line per event.
structured logging
The practice of emitting each log entry as machine-readable key-value data (usually JSON) rather than free-form prose, so its fields can be queried precisely.
metric
A numeric measurement sampled over time, forming a time-series of (timestamp, value) pairs that can be charted and aggregated across many events.
counter
A metric that only ever increases (resetting to zero on restart), such as the total number of requests served; its rate of change per second is usually what you chart.
gauge
A metric that can rise and fall over time, such as current memory in use or the length of a queue right now.
histogram
A metric that records the distribution of many measured values into buckets, such as request latencies, from which percentiles like the p95 are computed.
percentile
A value below which a given fraction of measurements fall; the p95 latency is the value below which 95% of requests finished, describing the slow tail that averages hide.
trace
A record of a single request as it travels through a system, reassembled from many spans that share one trace identifier, showing where the time was spent.
span
One unit of work within a trace — a database query, a service call, a computation — with a start time, a duration, and a name, nested under a parent span.
dashboard
A screen of charts built from metrics that shows a system's vital signs at a glance, such as request rate, error rate, and latency percentiles, for a human who is actively watching.
alert
A rule that turns a metric crossing a threshold into a notification for a human who is not watching, best configured to fire on user-facing symptoms rather than internal causes.
SLO
A Service Level Objective: a target you commit to for a measurement of user experience, such as 99.9% of requests succeeding in under 300 ms over a 30-day window, giving alerting a principled basis.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.