LLMs and Generative AI › LLM APIs › Day 255

Hands-on lab — Day 255: Streaming Responses

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/streaming_responses_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/streaming_responses_lib.py
examples/test_streaming_responses_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/streaming_responses_lib.py
starter/test_streaming_responses_lib.py
tests/run_tests.sh
tests/test_streaming_responses_lib.py
troubleshooting.md

Lab README

Lab: Day 255 -- Streaming Responses

Lesson

Day number: 255 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: Streaming Responses, Server-Sent Events, and Latency Telemetry.

Purpose

Build and test a Streaming Token Aggregator and Latency Monitor in Python. Consume incremental token deltas, calculate Time-To-First-Token (TTFT) and Inter-Token Latency (ITL), and aggregate complete response payloads.

Learning objectives

  • Process streaming token chunks from generators and async streams.
  • Calculate Time-To-First-Token (TTFT) and Inter-Token Latency (ITL).
  • Aggregate streaming chunks into full textual responses.
  • Implement real-time terminal output rendering.

Prerequisites

  • Day 254 (The OpenAI-Compatible Ecosystem).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard time/typing modules are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/streaming_responses_lib.py: Student scaffold file.
  • examples/streaming_responses_lib.py: Complete reference implementation.
  • tests/test_streaming_responses_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/streaming_responses_lib.py

What the commands do

  • Initiates simulated token stream.
  • Accumulates token deltas and calculates TTFT.
  • Runs unit test assertions.

Expected output

Streaming Demo Executed. Text: Real-time streaming slashes perceived latency drastically.

Validation steps

  1. Verify TTFT is recorded upon receiving chunk 1.
  2. Confirm aggregated text matches all yielded deltas.
  3. Validate ITL computation across streamed chunks.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Zero token count: Ensure process_chunk is called for each emitted token.

Security notes

Streaming tokens are processed strictly in local memory.

Extension exercises

  1. Implement an async generator stream in FastAPI.
  2. Build a live words-per-second terminal speedometer.
  • Lesson title: Streaming Responses
  • Day number: 255 of 365
  • Lesson article: https://ai-roadmap-365.github.io/day-255-streaming-responses
  • Lab files: everything you need is in this directory — follow “How to run” below.
  • Browse the course locally: from the repository root, this lab also appears in the course website at /labs/day-255-streaming-responses when the site is running.

Expected output

FIELDS.md

# Expected Output Fields: Day 255

- `Streaming Demo`: Status of streaming response aggregation.

examples-run.txt

Streaming Demo Executed. Text: Real-time streaming slashes perceived latency drastically.

measured-values.txt

Streaming Demo: Success
Tokens Streamed: 6
Text: Real-time streaming slashes perceived latency drastically.

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_streaming_responses_lib.py::test_streaming_aggregation PASSED  [ 33%]
tests/test_streaming_responses_lib.py::test_empty_stream PASSED          [ 66%]
tests/test_streaming_responses_lib.py::test_single_token_stream PASSED   [100%]

============================== 3 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/streaming_responses_lib.py (2048 bytes)
import time
from typing import Generator, Dict, Any, List

# perf_counter, not time(): time() is a wall clock that an NTP correction can
# step backwards, which would report a negative latency. perf_counter is
# monotonic and is the right clock for measuring an interval.

class StreamingTokenAggregator:
    def __init__(self):
        self.accumulated_text = ""
        self.token_count = 0
        self.start_time = 0.0
        self.ttft = 0.0
        self.end_time = 0.0

    def start(self):
        self.start_time = time.perf_counter()

    def process_chunk(self, delta_text: str):
        if self.token_count == 0:
            self.ttft = (time.perf_counter() - self.start_time) * 1000
        self.accumulated_text += delta_text
        self.token_count += 1

    def finish(self) -> Dict[str, Any]:
        self.end_time = time.perf_counter()
        total_duration = (self.end_time - self.start_time) * 1000
        # Inter-token latency is the gap BETWEEN tokens, so a stream of one
        # token has none. Dividing by max(1, n-1) there would report the whole
        # tail of the request as if it were a gap.
        gaps = self.token_count - 1
        itl = (total_duration - self.ttft) / gaps if gaps > 0 else 0.0

        return {
            "text": self.accumulated_text,
            "token_count": self.token_count,
            "ttft_ms": round(self.ttft, 2),
            "itl_ms": round(itl, 2),
            "total_duration_ms": round(total_duration, 2)
        }

def simulate_mock_stream() -> Generator[str, None, None]:
    words = ["Real-time", " streaming", " slashes", " perceived", " latency", " drastically."]
    for word in words:
        time.sleep(0.01)
        yield word

def run_streaming_demo():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    for chunk in simulate_mock_stream():
        aggregator.process_chunk(chunk)
    result = aggregator.finish()
    print("Streaming Demo Executed. Text:", result["text"])
    return result

if __name__ == "__main__":
    run_streaming_demo()
examples/test_streaming_responses_lib.py (1397 bytes)
import pytest
from examples.streaming_responses_lib import (
    StreamingTokenAggregator,
    simulate_mock_stream
)

def test_streaming_aggregation():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    for chunk in simulate_mock_stream():
        aggregator.process_chunk(chunk)
    result = aggregator.finish()

    assert result["text"] == "Real-time streaming slashes perceived latency drastically."
    assert result["token_count"] == 6
    assert result["ttft_ms"] > 0
    assert result["itl_ms"] > 0

def test_empty_stream():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    result = aggregator.finish()

    assert result["text"] == ""
    assert result["token_count"] == 0

def test_single_token_stream():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    aggregator.process_chunk("Hello")
    result = aggregator.finish()

    assert result["text"] == "Hello"
    assert result["token_count"] == 1

    # A chunk handed over with no delay legitimately has a time-to-first-token
    # of ~0. Asserting > 0 here asserts that the clock is slow, not that the
    # aggregator works. What must hold is that it is non-negative and inside
    # the total, and that one token means no inter-token gap.
    assert result["ttft_ms"] >= 0
    assert result["ttft_ms"] <= result["total_duration_ms"]
    assert result["itl_ms"] == 0.0
metadata.yml (432 bytes)
lesson_id: D255
day: 255
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/streaming_responses_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/streaming_responses_lib.py (773 bytes)
import time
from typing import Generator, Dict, Any, List

# perf_counter, not time(): time() is a wall clock that an NTP correction can
# step backwards, which would report a negative latency. perf_counter is
# monotonic and is the right clock for measuring an interval.

class StreamingTokenAggregator:
    def __init__(self):
        # TODO: Initialize tracking attributes
        pass

    def start(self):
        # TODO: Record start time
        pass

    def process_chunk(self, delta_text: str):
        # TODO: Record TTFT and accumulate text
        pass

    def finish(self) -> Dict[str, Any]:
        # TODO: Calculate ITL and return summary
        pass

def simulate_mock_stream() -> Generator[str, None, None]:
    # TODO: Yield mock token chunks
    pass
starter/test_streaming_responses_lib.py (1005 bytes)
import pytest
from examples.streaming_responses_lib import (
    StreamingTokenAggregator,
    simulate_mock_stream
)

def test_streaming_aggregation():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    for chunk in simulate_mock_stream():
        aggregator.process_chunk(chunk)
    result = aggregator.finish()

    assert result["text"] == "Real-time streaming slashes perceived latency drastically."
    assert result["token_count"] == 6
    assert result["ttft_ms"] > 0
    assert result["itl_ms"] > 0

def test_empty_stream():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    result = aggregator.finish()

    assert result["text"] == ""
    assert result["token_count"] == 0

def test_single_token_stream():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    aggregator.process_chunk("Hello")
    result = aggregator.finish()

    assert result["text"] == "Hello"
    assert result["token_count"] == 1
    assert result["ttft_ms"] > 0
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 255 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_streaming_responses_lib.py (1397 bytes)
import pytest
from examples.streaming_responses_lib import (
    StreamingTokenAggregator,
    simulate_mock_stream
)

def test_streaming_aggregation():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    for chunk in simulate_mock_stream():
        aggregator.process_chunk(chunk)
    result = aggregator.finish()

    assert result["text"] == "Real-time streaming slashes perceived latency drastically."
    assert result["token_count"] == 6
    assert result["ttft_ms"] > 0
    assert result["itl_ms"] > 0

def test_empty_stream():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    result = aggregator.finish()

    assert result["text"] == ""
    assert result["token_count"] == 0

def test_single_token_stream():
    aggregator = StreamingTokenAggregator()
    aggregator.start()
    aggregator.process_chunk("Hello")
    result = aggregator.finish()

    assert result["text"] == "Hello"
    assert result["token_count"] == 1

    # A chunk handed over with no delay legitimately has a time-to-first-token
    # of ~0. Asserting > 0 here asserts that the clock is slow, not that the
    # aggregator works. What must hold is that it is non-negative and inside
    # the total, and that one token means no inter-token gap.
    assert result["ttft_ms"] >= 0
    assert result["ttft_ms"] <= result["total_duration_ms"]
    assert result["itl_ms"] == 0.0

Troubleshooting

Troubleshooting: Day 255 - Streaming Responses

Common Issues

  1. Stdout buffering delay:
    • Cause: print without flush=True.
    • Fix: Use sys.stdout.flush() or print(..., flush=True).

Security notes

Security & Privacy: Day 255 - Streaming Responses

Security Guidance

  • Terminating canceled streams immediately protects against unnecessary token consumption.