LLMs and Generative AI › LLM APIs › Day 255
Hands-on lab — Day 255: Streaming Responses
- ← Back to the Day 255 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-255-streaming-responses/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/streaming_responses_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/streaming_responses_lib.py examples/test_streaming_responses_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/streaming_responses_lib.py starter/test_streaming_responses_lib.py tests/run_tests.sh tests/test_streaming_responses_lib.py troubleshooting.md
Lab README
Lab: Day 255 -- Streaming Responses
Lesson
Day number: 255 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: Streaming Responses, Server-Sent Events, and Latency Telemetry.
Purpose
Build and test a Streaming Token Aggregator and Latency Monitor in Python. Consume incremental token deltas, calculate Time-To-First-Token (TTFT) and Inter-Token Latency (ITL), and aggregate complete response payloads.
Learning objectives
- Process streaming token chunks from generators and async streams.
- Calculate Time-To-First-Token (TTFT) and Inter-Token Latency (ITL).
- Aggregate streaming chunks into full textual responses.
- Implement real-time terminal output rendering.
Prerequisites
- Day 254 (The OpenAI-Compatible Ecosystem).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard time/typing modules are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/streaming_responses_lib.py: Student scaffold file.examples/streaming_responses_lib.py: Complete reference implementation.tests/test_streaming_responses_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/streaming_responses_lib.py
What the commands do
- Initiates simulated token stream.
- Accumulates token deltas and calculates TTFT.
- Runs unit test assertions.
Expected output
Streaming Demo Executed. Text: Real-time streaming slashes perceived latency drastically.
Validation steps
- Verify TTFT is recorded upon receiving chunk 1.
- Confirm aggregated text matches all yielded deltas.
- Validate ITL computation across streamed chunks.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Zero token count: Ensure process_chunk is called for each emitted token.
Security notes
Streaming tokens are processed strictly in local memory.
Extension exercises
- Implement an async generator stream in FastAPI.
- Build a live words-per-second terminal speedometer.
Navigation
- Lesson title: Streaming Responses
- Day number: 255 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-255-streaming-responses
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-255-streaming-responseswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 255
- `Streaming Demo`: Status of streaming response aggregation.
examples-run.txt
Streaming Demo Executed. Text: Real-time streaming slashes perceived latency drastically.
measured-values.txt
Streaming Demo: Success
Tokens Streamed: 6
Text: Real-time streaming slashes perceived latency drastically.
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_streaming_responses_lib.py::test_streaming_aggregation PASSED [ 33%]
tests/test_streaming_responses_lib.py::test_empty_stream PASSED [ 66%]
tests/test_streaming_responses_lib.py::test_single_token_stream PASSED [100%]
============================== 3 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/streaming_responses_lib.py (2048 bytes)
import time
from typing import Generator, Dict, Any, List
# perf_counter, not time(): time() is a wall clock that an NTP correction can
# step backwards, which would report a negative latency. perf_counter is
# monotonic and is the right clock for measuring an interval.
class StreamingTokenAggregator:
def __init__(self):
self.accumulated_text = ""
self.token_count = 0
self.start_time = 0.0
self.ttft = 0.0
self.end_time = 0.0
def start(self):
self.start_time = time.perf_counter()
def process_chunk(self, delta_text: str):
if self.token_count == 0:
self.ttft = (time.perf_counter() - self.start_time) * 1000
self.accumulated_text += delta_text
self.token_count += 1
def finish(self) -> Dict[str, Any]:
self.end_time = time.perf_counter()
total_duration = (self.end_time - self.start_time) * 1000
# Inter-token latency is the gap BETWEEN tokens, so a stream of one
# token has none. Dividing by max(1, n-1) there would report the whole
# tail of the request as if it were a gap.
gaps = self.token_count - 1
itl = (total_duration - self.ttft) / gaps if gaps > 0 else 0.0
return {
"text": self.accumulated_text,
"token_count": self.token_count,
"ttft_ms": round(self.ttft, 2),
"itl_ms": round(itl, 2),
"total_duration_ms": round(total_duration, 2)
}
def simulate_mock_stream() -> Generator[str, None, None]:
words = ["Real-time", " streaming", " slashes", " perceived", " latency", " drastically."]
for word in words:
time.sleep(0.01)
yield word
def run_streaming_demo():
aggregator = StreamingTokenAggregator()
aggregator.start()
for chunk in simulate_mock_stream():
aggregator.process_chunk(chunk)
result = aggregator.finish()
print("Streaming Demo Executed. Text:", result["text"])
return result
if __name__ == "__main__":
run_streaming_demo()
examples/test_streaming_responses_lib.py (1397 bytes)
import pytest
from examples.streaming_responses_lib import (
StreamingTokenAggregator,
simulate_mock_stream
)
def test_streaming_aggregation():
aggregator = StreamingTokenAggregator()
aggregator.start()
for chunk in simulate_mock_stream():
aggregator.process_chunk(chunk)
result = aggregator.finish()
assert result["text"] == "Real-time streaming slashes perceived latency drastically."
assert result["token_count"] == 6
assert result["ttft_ms"] > 0
assert result["itl_ms"] > 0
def test_empty_stream():
aggregator = StreamingTokenAggregator()
aggregator.start()
result = aggregator.finish()
assert result["text"] == ""
assert result["token_count"] == 0
def test_single_token_stream():
aggregator = StreamingTokenAggregator()
aggregator.start()
aggregator.process_chunk("Hello")
result = aggregator.finish()
assert result["text"] == "Hello"
assert result["token_count"] == 1
# A chunk handed over with no delay legitimately has a time-to-first-token
# of ~0. Asserting > 0 here asserts that the clock is slow, not that the
# aggregator works. What must hold is that it is non-negative and inside
# the total, and that one token means no inter-token gap.
assert result["ttft_ms"] >= 0
assert result["ttft_ms"] <= result["total_duration_ms"]
assert result["itl_ms"] == 0.0
metadata.yml (432 bytes)
lesson_id: D255
day: 255
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/streaming_responses_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/streaming_responses_lib.py (773 bytes)
import time
from typing import Generator, Dict, Any, List
# perf_counter, not time(): time() is a wall clock that an NTP correction can
# step backwards, which would report a negative latency. perf_counter is
# monotonic and is the right clock for measuring an interval.
class StreamingTokenAggregator:
def __init__(self):
# TODO: Initialize tracking attributes
pass
def start(self):
# TODO: Record start time
pass
def process_chunk(self, delta_text: str):
# TODO: Record TTFT and accumulate text
pass
def finish(self) -> Dict[str, Any]:
# TODO: Calculate ITL and return summary
pass
def simulate_mock_stream() -> Generator[str, None, None]:
# TODO: Yield mock token chunks
pass
starter/test_streaming_responses_lib.py (1005 bytes)
import pytest
from examples.streaming_responses_lib import (
StreamingTokenAggregator,
simulate_mock_stream
)
def test_streaming_aggregation():
aggregator = StreamingTokenAggregator()
aggregator.start()
for chunk in simulate_mock_stream():
aggregator.process_chunk(chunk)
result = aggregator.finish()
assert result["text"] == "Real-time streaming slashes perceived latency drastically."
assert result["token_count"] == 6
assert result["ttft_ms"] > 0
assert result["itl_ms"] > 0
def test_empty_stream():
aggregator = StreamingTokenAggregator()
aggregator.start()
result = aggregator.finish()
assert result["text"] == ""
assert result["token_count"] == 0
def test_single_token_stream():
aggregator = StreamingTokenAggregator()
aggregator.start()
aggregator.process_chunk("Hello")
result = aggregator.finish()
assert result["text"] == "Hello"
assert result["token_count"] == 1
assert result["ttft_ms"] > 0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 255 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_streaming_responses_lib.py (1397 bytes)
import pytest
from examples.streaming_responses_lib import (
StreamingTokenAggregator,
simulate_mock_stream
)
def test_streaming_aggregation():
aggregator = StreamingTokenAggregator()
aggregator.start()
for chunk in simulate_mock_stream():
aggregator.process_chunk(chunk)
result = aggregator.finish()
assert result["text"] == "Real-time streaming slashes perceived latency drastically."
assert result["token_count"] == 6
assert result["ttft_ms"] > 0
assert result["itl_ms"] > 0
def test_empty_stream():
aggregator = StreamingTokenAggregator()
aggregator.start()
result = aggregator.finish()
assert result["text"] == ""
assert result["token_count"] == 0
def test_single_token_stream():
aggregator = StreamingTokenAggregator()
aggregator.start()
aggregator.process_chunk("Hello")
result = aggregator.finish()
assert result["text"] == "Hello"
assert result["token_count"] == 1
# A chunk handed over with no delay legitimately has a time-to-first-token
# of ~0. Asserting > 0 here asserts that the clock is slow, not that the
# aggregator works. What must hold is that it is non-negative and inside
# the total, and that one token means no inter-token gap.
assert result["ttft_ms"] >= 0
assert result["ttft_ms"] <= result["total_duration_ms"]
assert result["itl_ms"] == 0.0
Troubleshooting
Troubleshooting: Day 255 - Streaming Responses
Common Issues
- Stdout buffering delay:
- Cause: print without flush=True.
- Fix: Use sys.stdout.flush() or print(..., flush=True).
Security notes
Security & Privacy: Day 255 - Streaming Responses
Security Guidance
- Terminating canceled streams immediately protects against unnecessary token consumption.