LLMs and Generative AI › LLM APIs › Day 254
Hands-on lab — Day 254: The OpenAI-Compatible Ecosystem
- ← Back to the Day 254 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-254-the-openai-compatible-ecosystem/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/the_openai_compatible_ecosystem_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/test_the_openai_compatible_ecosystem_lib.py examples/the_openai_compatible_ecosystem_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/test_the_openai_compatible_ecosystem_lib.py starter/the_openai_compatible_ecosystem_lib.py tests/run_tests.sh tests/test_the_openai_compatible_ecosystem_lib.py troubleshooting.md
Lab README
Lab: Day 254 -- The OpenAI-Compatible Ecosystem
Lesson
Day number: 254 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: The OpenAI-Compatible Ecosystem, /v1/chat/completions, and Multi-Provider Routing.
Purpose
Build and test an extensible Multi-Provider OpenAI-Compatible Router in Python. Configure tiered provider priority lists, execute automated failover on simulated network timeouts, and measure routing latencies.
Learning objectives
- Structure API requests using the standard OpenAI
/v1/chat/completionsspecification. - Redirect client requests using custom
base_urlendpoints. - Implement automated fallback failover across tiered model hosts.
- Benchmark provider latency and track routing telemetry.
Prerequisites
- Day 253 (First Calls to the Claude API).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard time/typing modules are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/the_openai_compatible_ecosystem_lib.py: Student scaffold file.examples/the_openai_compatible_ecosystem_lib.py: Complete reference implementation.tests/test_the_openai_compatible_ecosystem_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/the_openai_compatible_ecosystem_lib.py
What the commands do
- Registers providers with priority ratings.
- Dispatches chat completion request.
- Runs unit test assertions.
Expected output
Router Demo Executed. Routed provider: vLLM_Primary
Validation steps
- Verify priority sorting routes to lowest priority integer first.
- Confirm failed primary routes failover to secondary provider.
- Validate empty provider list raises ValueError.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- All providers failing: Verify provider names do not contain 'fail' unless simulating errors.
Security notes
All routing logic executes locally in memory.
Extension exercises
- Implement a rolling latency load balancer.
- Build an automated circuit breaker with 60-second recovery timeouts.
Navigation
- Lesson title: The OpenAI-Compatible Ecosystem
- Day number: 254 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-254-the-openai-compatible-ecosystem
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-254-the-openai-compatible-ecosystemwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 254
- `Router Demo`: Status of multi-provider router completion dispatch.
examples-run.txt
Router Demo Executed. Routed provider: vLLM_Primary
measured-values.txt
Router Demo: Success
Routed Provider: vLLM_Primary
Total Tokens: 45
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 4 items
tests/test_the_openai_compatible_ecosystem_lib.py::test_router_priority_order PASSED [ 25%]
tests/test_the_openai_compatible_ecosystem_lib.py::test_automated_failover PASSED [ 50%]
tests/test_the_openai_compatible_ecosystem_lib.py::test_all_providers_failing_raises_runtime_error PASSED [ 75%]
tests/test_the_openai_compatible_ecosystem_lib.py::test_empty_router_raises_value_error PASSED [100%]
============================== 4 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/test_the_openai_compatible_ecosystem_lib.py (1753 bytes)
import pytest
from examples.the_openai_compatible_ecosystem_lib import (
MultiProviderRouter,
ProviderConfig
)
def test_router_priority_order():
router = MultiProviderRouter()
router.register_provider(ProviderConfig("Cloud_Tier2", "https://api.cloud.com/v1", "k2", "m2", priority=2))
router.register_provider(ProviderConfig("Local_Tier1", "http://127.0.0.1:8000/v1", "k1", "m1", priority=1))
resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
assert resp["routed_provider"] == "Local_Tier1"
assert resp["choices"][0]["message"]["content"] == "Routed response success."
def test_automated_failover():
router = MultiProviderRouter()
# Primary fails, fallback succeeds
router.register_provider(ProviderConfig("Failing_Primary", "http://bad:8000/v1", "k1", "m1", priority=1))
router.register_provider(ProviderConfig("Healthy_Backup", "https://api.good.com/v1", "k2", "m2", priority=2))
resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
assert resp["routed_provider"] == "Healthy_Backup"
def test_all_providers_failing_raises_runtime_error():
router = MultiProviderRouter()
router.register_provider(ProviderConfig("Failing_1", "http://bad1/v1", "k1", "m1", priority=1))
router.register_provider(ProviderConfig("Failing_2", "http://bad2/v1", "k2", "m2", priority=2))
with pytest.raises(RuntimeError) as exc_info:
router.dispatch_completion([{"role": "user", "content": "Test"}])
assert "All 2 providers failed" in str(exc_info.value)
def test_empty_router_raises_value_error():
router = MultiProviderRouter()
with pytest.raises(ValueError):
router.dispatch_completion([{"role": "user", "content": "Test"}])
examples/the_openai_compatible_ecosystem_lib.py (2588 bytes)
from typing import Dict, Any, List, Optional
import time
class ProviderConfig:
def __init__(self, name: str, base_url: str, api_key: str, default_model: str, priority: int = 1):
self.name = name
self.base_url = base_url
self.api_key = api_key
self.default_model = default_model
self.priority = priority
class MultiProviderRouter:
def __init__(self):
self.providers: List[ProviderConfig] = []
def register_provider(self, provider: ProviderConfig) -> "MultiProviderRouter":
self.providers.append(provider)
self.providers.sort(key=lambda p: p.priority)
return self
def dispatch_completion(
self,
messages: List[Dict[str, str]],
temperature: float = 0.2,
max_tokens: int = 512
) -> Dict[str, Any]:
if not self.providers:
raise ValueError("No providers registered in router.")
last_exception = None
for provider in self.providers:
try:
start_time = time.time()
response = self._execute_provider_call(provider, messages, temperature, max_tokens)
latency_ms = (time.time() - start_time) * 1000
response["routed_provider"] = provider.name
response["latency_ms"] = latency_ms
return response
except Exception as err:
last_exception = err
continue
raise RuntimeError(f"All {len(self.providers)} providers failed. Last error: {last_exception}")
def _execute_provider_call(self, provider: ProviderConfig, messages, temperature, max_tokens):
if "fail" in provider.name.lower():
raise ConnectionError(f"Simulated network timeout on {provider.name}")
return {
"id": f"chatcmpl_{provider.name}",
"choices": [{"message": {"role": "assistant", "content": "Routed response success."}}],
"model": provider.default_model,
"usage": {"total_tokens": 45}
}
def run_router_demo():
router = MultiProviderRouter()
router.register_provider(ProviderConfig("vLLM_Primary", "http://127.0.0.1:8000/v1", "EMPTY", "llama-3.3-70b", priority=1))
router.register_provider(ProviderConfig("OpenAI_Backup", "https://api.openai.com/v1", "sk-mock", "gpt-4o", priority=2))
resp = router.dispatch_completion([{"role": "user", "content": "Hello router."}])
print("Router Demo Executed. Routed provider:", resp["routed_provider"])
return resp
if __name__ == "__main__":
run_router_demo()
metadata.yml (444 bytes)
lesson_id: D254
day: 254
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/the_openai_compatible_ecosystem_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_the_openai_compatible_ecosystem_lib.py (1753 bytes)
import pytest
from examples.the_openai_compatible_ecosystem_lib import (
MultiProviderRouter,
ProviderConfig
)
def test_router_priority_order():
router = MultiProviderRouter()
router.register_provider(ProviderConfig("Cloud_Tier2", "https://api.cloud.com/v1", "k2", "m2", priority=2))
router.register_provider(ProviderConfig("Local_Tier1", "http://127.0.0.1:8000/v1", "k1", "m1", priority=1))
resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
assert resp["routed_provider"] == "Local_Tier1"
assert resp["choices"][0]["message"]["content"] == "Routed response success."
def test_automated_failover():
router = MultiProviderRouter()
# Primary fails, fallback succeeds
router.register_provider(ProviderConfig("Failing_Primary", "http://bad:8000/v1", "k1", "m1", priority=1))
router.register_provider(ProviderConfig("Healthy_Backup", "https://api.good.com/v1", "k2", "m2", priority=2))
resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
assert resp["routed_provider"] == "Healthy_Backup"
def test_all_providers_failing_raises_runtime_error():
router = MultiProviderRouter()
router.register_provider(ProviderConfig("Failing_1", "http://bad1/v1", "k1", "m1", priority=1))
router.register_provider(ProviderConfig("Failing_2", "http://bad2/v1", "k2", "m2", priority=2))
with pytest.raises(RuntimeError) as exc_info:
router.dispatch_completion([{"role": "user", "content": "Test"}])
assert "All 2 providers failed" in str(exc_info.value)
def test_empty_router_raises_value_error():
router = MultiProviderRouter()
with pytest.raises(ValueError):
router.dispatch_completion([{"role": "user", "content": "Test"}])
starter/the_openai_compatible_ecosystem_lib.py (821 bytes)
from typing import Dict, Any, List, Optional
import time
class ProviderConfig:
def __init__(self, name: str, base_url: str, api_key: str, default_model: str, priority: int = 1):
self.name = name
self.base_url = base_url
self.api_key = api_key
self.default_model = default_model
self.priority = priority
class MultiProviderRouter:
def __init__(self):
# TODO: Initialize provider list
pass
def register_provider(self, provider: ProviderConfig) -> "MultiProviderRouter":
# TODO: Register and sort providers
pass
def dispatch_completion(
self,
messages: List[Dict[str, str]],
temperature: float = 0.2,
max_tokens: int = 512
) -> Dict[str, Any]:
# TODO: Dispatch with failover
pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 254 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_the_openai_compatible_ecosystem_lib.py (1753 bytes)
import pytest
from examples.the_openai_compatible_ecosystem_lib import (
MultiProviderRouter,
ProviderConfig
)
def test_router_priority_order():
router = MultiProviderRouter()
router.register_provider(ProviderConfig("Cloud_Tier2", "https://api.cloud.com/v1", "k2", "m2", priority=2))
router.register_provider(ProviderConfig("Local_Tier1", "http://127.0.0.1:8000/v1", "k1", "m1", priority=1))
resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
assert resp["routed_provider"] == "Local_Tier1"
assert resp["choices"][0]["message"]["content"] == "Routed response success."
def test_automated_failover():
router = MultiProviderRouter()
# Primary fails, fallback succeeds
router.register_provider(ProviderConfig("Failing_Primary", "http://bad:8000/v1", "k1", "m1", priority=1))
router.register_provider(ProviderConfig("Healthy_Backup", "https://api.good.com/v1", "k2", "m2", priority=2))
resp = router.dispatch_completion([{"role": "user", "content": "Test"}])
assert resp["routed_provider"] == "Healthy_Backup"
def test_all_providers_failing_raises_runtime_error():
router = MultiProviderRouter()
router.register_provider(ProviderConfig("Failing_1", "http://bad1/v1", "k1", "m1", priority=1))
router.register_provider(ProviderConfig("Failing_2", "http://bad2/v1", "k2", "m2", priority=2))
with pytest.raises(RuntimeError) as exc_info:
router.dispatch_completion([{"role": "user", "content": "Test"}])
assert "All 2 providers failed" in str(exc_info.value)
def test_empty_router_raises_value_error():
router = MultiProviderRouter()
with pytest.raises(ValueError):
router.dispatch_completion([{"role": "user", "content": "Test"}])
Troubleshooting
Troubleshooting: Day 254 - The OpenAI-Compatible Ecosystem
Common Issues
- Router exhaustion error:
- Cause: All registered endpoints returned network errors.
- Fix: Check provider health status and base_url configurations.
Security notes
Security & Privacy: Day 254 - The OpenAI-Compatible Ecosystem
Security Guidance
- Redirecting base_url to local servers (http://127.0.0.1:8000/v1) ensures complete data privacy on premises.