LLMs and Generative AI › LLM APIs › Day 254

Hands-on lab — Day 254: The OpenAI-Compatible Ecosystem

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/the_openai_compatible_ecosystem_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_the_openai_compatible_ecosystem_lib.py
examples/the_openai_compatible_ecosystem_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_the_openai_compatible_ecosystem_lib.py
starter/the_openai_compatible_ecosystem_lib.py
tests/run_tests.sh
tests/test_the_openai_compatible_ecosystem_lib.py
troubleshooting.md

Lab README

Lab: Day 254 -- The OpenAI-Compatible Ecosystem

Lesson

Day number: 254 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: The OpenAI-Compatible Ecosystem, /v1/chat/completions, and Multi-Provider Routing.

Purpose

Build and test an extensible Multi-Provider OpenAI-Compatible Router in Python. Configure tiered provider priority lists, execute automated failover on simulated network timeouts, and measure routing latencies.

Learning objectives

  • Structure API requests using the standard OpenAI /v1/chat/completions specification.
  • Redirect client requests using custom base_url endpoints.
  • Implement automated fallback failover across tiered model hosts.
  • Benchmark provider latency and track routing telemetry.

Prerequisites

  • Day 253 (First Calls to the Claude API).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard time/typing modules are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/the_openai_compatible_ecosystem_lib.py: Student scaffold file.
  • examples/the_openai_compatible_ecosystem_lib.py: Complete reference implementation.
  • tests/test_the_openai_compatible_ecosystem_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/the_openai_compatible_ecosystem_lib.py

What the commands do

  • Registers providers with priority ratings.
  • Dispatches chat completion request.
  • Runs unit test assertions.

Expected output

Router Demo Executed. Routed provider: vLLM_Primary

Validation steps

  1. Verify priority sorting routes to lowest priority integer first.
  2. Confirm failed primary routes failover to secondary provider.
  3. Validate empty provider list raises ValueError.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • All providers failing: Verify provider names do not contain 'fail' unless simulating errors.

Security notes

All routing logic executes locally in memory.

Extension exercises

  1. Implement a rolling latency load balancer.
  2. Build an automated circuit breaker with 60-second recovery timeouts.

Expected output

FIELDS.md

# Expected Output Fields: Day 254

- `Router Demo`: Status of multi-provider router completion dispatch.

examples-run.txt

Router Demo Executed. Routed provider: vLLM_Primary

measured-values.txt

Router Demo: Success
Routed Provider: vLLM_Primary
Total Tokens: 45

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 4 items

tests/test_the_openai_compatible_ecosystem_lib.py::test_router_priority_order PASSED [ 25%]
tests/test_the_openai_compatible_ecosystem_lib.py::test_automated_failover PASSED [ 50%]
tests/test_the_openai_compatible_ecosystem_lib.py::test_all_providers_failing_raises_runtime_error PASSED [ 75%]
tests/test_the_openai_compatible_ecosystem_lib.py::test_empty_router_raises_value_error PASSED [100%]

============================== 4 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_the_openai_compatible_ecosystem_lib.py (1753 bytes)
import pytest
from examples.the_openai_compatible_ecosystem_lib import (
    MultiProviderRouter,
    ProviderConfig
)

def test_router_priority_order():
    router = MultiProviderRouter()
    router.register_provider(ProviderConfig("Cloud_Tier2", "https://api.cloud.com/v1", "k2", "m2", priority=2))
    router.register_provider(ProviderConfig("Local_Tier1", "http://127.0.0.1:8000/v1", "k1", "m1", priority=1))

    resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert resp["routed_provider"] == "Local_Tier1"
    assert resp["choices"][0]["message"]["content"] == "Routed response success."

def test_automated_failover():
    router = MultiProviderRouter()
    # Primary fails, fallback succeeds
    router.register_provider(ProviderConfig("Failing_Primary", "http://bad:8000/v1", "k1", "m1", priority=1))
    router.register_provider(ProviderConfig("Healthy_Backup", "https://api.good.com/v1", "k2", "m2", priority=2))

    resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert resp["routed_provider"] == "Healthy_Backup"

def test_all_providers_failing_raises_runtime_error():
    router = MultiProviderRouter()
    router.register_provider(ProviderConfig("Failing_1", "http://bad1/v1", "k1", "m1", priority=1))
    router.register_provider(ProviderConfig("Failing_2", "http://bad2/v1", "k2", "m2", priority=2))

    with pytest.raises(RuntimeError) as exc_info:
        router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert "All 2 providers failed" in str(exc_info.value)

def test_empty_router_raises_value_error():
    router = MultiProviderRouter()
    with pytest.raises(ValueError):
        router.dispatch_completion([{"role": "user", "content": "Test"}])
examples/the_openai_compatible_ecosystem_lib.py (2588 bytes)
from typing import Dict, Any, List, Optional
import time

class ProviderConfig:
    def __init__(self, name: str, base_url: str, api_key: str, default_model: str, priority: int = 1):
        self.name = name
        self.base_url = base_url
        self.api_key = api_key
        self.default_model = default_model
        self.priority = priority

class MultiProviderRouter:
    def __init__(self):
        self.providers: List[ProviderConfig] = []

    def register_provider(self, provider: ProviderConfig) -> "MultiProviderRouter":
        self.providers.append(provider)
        self.providers.sort(key=lambda p: p.priority)
        return self

    def dispatch_completion(
        self,
        messages: List[Dict[str, str]],
        temperature: float = 0.2,
        max_tokens: int = 512
    ) -> Dict[str, Any]:
        if not self.providers:
            raise ValueError("No providers registered in router.")

        last_exception = None
        for provider in self.providers:
            try:
                start_time = time.time()
                response = self._execute_provider_call(provider, messages, temperature, max_tokens)
                latency_ms = (time.time() - start_time) * 1000
                response["routed_provider"] = provider.name
                response["latency_ms"] = latency_ms
                return response
            except Exception as err:
                last_exception = err
                continue

        raise RuntimeError(f"All {len(self.providers)} providers failed. Last error: {last_exception}")

    def _execute_provider_call(self, provider: ProviderConfig, messages, temperature, max_tokens):
        if "fail" in provider.name.lower():
            raise ConnectionError(f"Simulated network timeout on {provider.name}")
        return {
            "id": f"chatcmpl_{provider.name}",
            "choices": [{"message": {"role": "assistant", "content": "Routed response success."}}],
            "model": provider.default_model,
            "usage": {"total_tokens": 45}
        }

def run_router_demo():
    router = MultiProviderRouter()
    router.register_provider(ProviderConfig("vLLM_Primary", "http://127.0.0.1:8000/v1", "EMPTY", "llama-3.3-70b", priority=1))
    router.register_provider(ProviderConfig("OpenAI_Backup", "https://api.openai.com/v1", "sk-mock", "gpt-4o", priority=2))

    resp = router.dispatch_completion([{"role": "user", "content": "Hello router."}])
    print("Router Demo Executed. Routed provider:", resp["routed_provider"])
    return resp

if __name__ == "__main__":
    run_router_demo()
metadata.yml (444 bytes)
lesson_id: D254
day: 254
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/the_openai_compatible_ecosystem_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_the_openai_compatible_ecosystem_lib.py (1753 bytes)
import pytest
from examples.the_openai_compatible_ecosystem_lib import (
    MultiProviderRouter,
    ProviderConfig
)

def test_router_priority_order():
    router = MultiProviderRouter()
    router.register_provider(ProviderConfig("Cloud_Tier2", "https://api.cloud.com/v1", "k2", "m2", priority=2))
    router.register_provider(ProviderConfig("Local_Tier1", "http://127.0.0.1:8000/v1", "k1", "m1", priority=1))

    resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert resp["routed_provider"] == "Local_Tier1"
    assert resp["choices"][0]["message"]["content"] == "Routed response success."

def test_automated_failover():
    router = MultiProviderRouter()
    # Primary fails, fallback succeeds
    router.register_provider(ProviderConfig("Failing_Primary", "http://bad:8000/v1", "k1", "m1", priority=1))
    router.register_provider(ProviderConfig("Healthy_Backup", "https://api.good.com/v1", "k2", "m2", priority=2))

    resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert resp["routed_provider"] == "Healthy_Backup"

def test_all_providers_failing_raises_runtime_error():
    router = MultiProviderRouter()
    router.register_provider(ProviderConfig("Failing_1", "http://bad1/v1", "k1", "m1", priority=1))
    router.register_provider(ProviderConfig("Failing_2", "http://bad2/v1", "k2", "m2", priority=2))

    with pytest.raises(RuntimeError) as exc_info:
        router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert "All 2 providers failed" in str(exc_info.value)

def test_empty_router_raises_value_error():
    router = MultiProviderRouter()
    with pytest.raises(ValueError):
        router.dispatch_completion([{"role": "user", "content": "Test"}])
starter/the_openai_compatible_ecosystem_lib.py (821 bytes)
from typing import Dict, Any, List, Optional
import time

class ProviderConfig:
    def __init__(self, name: str, base_url: str, api_key: str, default_model: str, priority: int = 1):
        self.name = name
        self.base_url = base_url
        self.api_key = api_key
        self.default_model = default_model
        self.priority = priority

class MultiProviderRouter:
    def __init__(self):
        # TODO: Initialize provider list
        pass

    def register_provider(self, provider: ProviderConfig) -> "MultiProviderRouter":
        # TODO: Register and sort providers
        pass

    def dispatch_completion(
        self,
        messages: List[Dict[str, str]],
        temperature: float = 0.2,
        max_tokens: int = 512
    ) -> Dict[str, Any]:
        # TODO: Dispatch with failover
        pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 254 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_the_openai_compatible_ecosystem_lib.py (1753 bytes)
import pytest
from examples.the_openai_compatible_ecosystem_lib import (
    MultiProviderRouter,
    ProviderConfig
)

def test_router_priority_order():
    router = MultiProviderRouter()
    router.register_provider(ProviderConfig("Cloud_Tier2", "https://api.cloud.com/v1", "k2", "m2", priority=2))
    router.register_provider(ProviderConfig("Local_Tier1", "http://127.0.0.1:8000/v1", "k1", "m1", priority=1))

    resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert resp["routed_provider"] == "Local_Tier1"
    assert resp["choices"][0]["message"]["content"] == "Routed response success."

def test_automated_failover():
    router = MultiProviderRouter()
    # Primary fails, fallback succeeds
    router.register_provider(ProviderConfig("Failing_Primary", "http://bad:8000/v1", "k1", "m1", priority=1))
    router.register_provider(ProviderConfig("Healthy_Backup", "https://api.good.com/v1", "k2", "m2", priority=2))

    resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert resp["routed_provider"] == "Healthy_Backup"

def test_all_providers_failing_raises_runtime_error():
    router = MultiProviderRouter()
    router.register_provider(ProviderConfig("Failing_1", "http://bad1/v1", "k1", "m1", priority=1))
    router.register_provider(ProviderConfig("Failing_2", "http://bad2/v1", "k2", "m2", priority=2))

    with pytest.raises(RuntimeError) as exc_info:
        router.dispatch_completion([{"role": "user", "content": "Test"}])
    assert "All 2 providers failed" in str(exc_info.value)

def test_empty_router_raises_value_error():
    router = MultiProviderRouter()
    with pytest.raises(ValueError):
        router.dispatch_completion([{"role": "user", "content": "Test"}])

Troubleshooting

Troubleshooting: Day 254 - The OpenAI-Compatible Ecosystem

Common Issues

  1. Router exhaustion error:
    • Cause: All registered endpoints returned network errors.
    • Fix: Check provider health status and base_url configurations.

Security notes

Security & Privacy: Day 254 - The OpenAI-Compatible Ecosystem

Security Guidance