LLMs and Generative AI › Prompt Engineering › Day 249
Hands-on lab — Day 249: Structured Output: Getting Reliable JSON
- ← Back to the Day 249 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-249-structured-output-getting-reliable-json/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/structured_output_getting_reliable_json_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/structured_output_getting_reliable_json_lib.py examples/test_structured_output_getting_reliable_json_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/structured_output_getting_reliable_json_lib.py starter/test_structured_output_getting_reliable_json_lib.py tests/run_tests.sh tests/test_structured_output_getting_reliable_json_lib.py troubleshooting.md
Lab README
Lab: Day 249 -- Structured Output: Getting Reliable JSON
Lesson
Day number: 249 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Structured Output, JSON Generation, and Constrained CFG Decoding.
Purpose
Build and test a type-safe JSON extraction and auto-repair engine in Python. Sanitize markdown code fences and conversational preambles, repair trailing commas and unclosed brackets, and validate schemas.
Learning objectives
- Implement regex and AST sanitization for raw LLM JSON responses.
- Auto-repair trailing commas and truncated closing brackets.
- Validate required schema keys against strict contracts.
- Understand the mechanics of Constrained Grammar FSM logit masking.
Prerequisites
- Day 248 (Few-Shot Examples and Chain of Thought).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python standard json and regex modules are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/structured_output_getting_reliable_json_lib.py: Student scaffold file.examples/structured_output_getting_reliable_json_lib.py: Complete reference implementation.tests/test_structured_output_getting_reliable_json_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/structured_output_getting_reliable_json_lib.py
What the commands do
- Cleans conversational preambles and markdown fences.
- Auto-repairs truncated JSON strings.
- Runs unit test assertions.
Expected output
JSON Demo Extracted Successfully: INC-101
Validation steps
- Verify
clean_json_stringremoves markdown code fences. - Confirm trailing comma regex fixes trailing commas.
- Validate missing key detection raises
KeyError. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- No JSON delimiter: Ensure the raw input contains at least one object or array opening delimiter.
Security notes
Never use eval() to parse untrusted model outputs.
Extension exercises
- Implement a nested Pydantic V2 validator with custom regex validators.
- Build an FSM-based logit masking simulator for token vocabularies.
Navigation
- Lesson title: Structured Output: Getting Reliable JSON
- Day number: 249 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-249-structured-output-getting-reliable-json
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-249-structured-output-getting-reliable-jsonwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 249
- `JSON Demo`: Status of JSON extraction and auto-repair.
examples-run.txt
JSON Demo Extracted Successfully: INC-101
measured-values.txt
JSON Demo: Success
Extracted ID: INC-101
Severity: HIGH
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_structured_output_getting_reliable_json_lib.py::test_json_markdown_cleaning PASSED [ 33%]
tests/test_structured_output_getting_reliable_json_lib.py::test_auto_closing_truncated_payload PASSED [ 66%]
tests/test_structured_output_getting_reliable_json_lib.py::test_missing_required_keys PASSED [100%]
============================== 3 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/structured_output_getting_reliable_json_lib.py (2008 bytes)
import json
import re
from typing import Dict, Any, Optional
class JSONExtractorEngine:
@staticmethod
def clean_json_string(raw_text: str) -> str:
cleaned = re.sub(r"^```(?:json)?\s*", "", raw_text.strip(), flags=re.IGNORECASE)
cleaned = re.sub(r"\s*```$", "", cleaned)
first_brace = cleaned.find("{")
first_bracket = cleaned.find("[")
if first_brace == -1 and first_bracket == -1:
raise ValueError("No JSON object or array delimiter found in response.")
if first_brace != -1 and (first_bracket == -1 or first_brace < first_bracket):
start_idx = first_brace
end_idx = cleaned.rfind("}")
if end_idx != -1:
cleaned = cleaned[start_idx:end_idx + 1]
else:
cleaned = cleaned[start_idx:] + "}"
else:
start_idx = first_bracket
end_idx = cleaned.rfind("]")
if end_idx != -1:
cleaned = cleaned[start_idx:end_idx + 1]
else:
cleaned = cleaned[start_idx:] + "]"
cleaned = re.sub(r",\s*([\}\]])", r"\1", cleaned)
return cleaned.strip()
@classmethod
def parse_and_validate(cls, raw_text: str, required_keys: Optional[list] = None) -> Dict[str, Any]:
cleaned = cls.clean_json_string(raw_text)
data = json.loads(cleaned)
if required_keys and isinstance(data, dict):
missing = [k for k in required_keys if k not in data]
if missing:
raise KeyError(f"Missing required schema keys: {missing}")
return data
def run_json_demo():
raw_payload = "Sure! Here is the data: ```json\n{\"incident_id\": \"INC-101\", \"severity\": \"HIGH\",}\n```"
result = JSONExtractorEngine.parse_and_validate(raw_payload, required_keys=["incident_id", "severity"])
print("JSON Demo Extracted Successfully:", result["incident_id"])
return result
if __name__ == "__main__":
run_json_demo()
examples/test_structured_output_getting_reliable_json_lib.py (748 bytes)
import pytest
from examples.structured_output_getting_reliable_json_lib import JSONExtractorEngine
def test_json_markdown_cleaning():
raw = "Sure! Here is the output:\n```json\n{\"status\": \"OK\", \"code\": 200,}\n```"
data = JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "code"])
assert data["status"] == "OK"
assert data["code"] == 200
def test_auto_closing_truncated_payload():
raw = "{\"status\": \"PENDING\""
cleaned = JSONExtractorEngine.clean_json_string(raw)
assert cleaned.endswith("}")
def test_missing_required_keys():
raw = "{\"status\": \"OK\"}"
with pytest.raises(KeyError):
JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "missing_field"])
metadata.yml (452 bytes)
lesson_id: D249
day: 249
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/structured_output_getting_reliable_json_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/structured_output_getting_reliable_json_lib.py (396 bytes)
import json
import re
from typing import Dict, Any, Optional
class JSONExtractorEngine:
@staticmethod
def clean_json_string(raw_text: str) -> str:
# TODO: Strip markdown and preamble
pass
@classmethod
def parse_and_validate(cls, raw_text: str, required_keys: Optional[list] = None) -> Dict[str, Any]:
# TODO: Clean, parse, and validate keys
pass
starter/test_structured_output_getting_reliable_json_lib.py (748 bytes)
import pytest
from examples.structured_output_getting_reliable_json_lib import JSONExtractorEngine
def test_json_markdown_cleaning():
raw = "Sure! Here is the output:\n```json\n{\"status\": \"OK\", \"code\": 200,}\n```"
data = JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "code"])
assert data["status"] == "OK"
assert data["code"] == 200
def test_auto_closing_truncated_payload():
raw = "{\"status\": \"PENDING\""
cleaned = JSONExtractorEngine.clean_json_string(raw)
assert cleaned.endswith("}")
def test_missing_required_keys():
raw = "{\"status\": \"OK\"}"
with pytest.raises(KeyError):
JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "missing_field"])
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 249 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_structured_output_getting_reliable_json_lib.py (748 bytes)
import pytest
from examples.structured_output_getting_reliable_json_lib import JSONExtractorEngine
def test_json_markdown_cleaning():
raw = "Sure! Here is the output:\n```json\n{\"status\": \"OK\", \"code\": 200,}\n```"
data = JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "code"])
assert data["status"] == "OK"
assert data["code"] == 200
def test_auto_closing_truncated_payload():
raw = "{\"status\": \"PENDING\""
cleaned = JSONExtractorEngine.clean_json_string(raw)
assert cleaned.endswith("}")
def test_missing_required_keys():
raw = "{\"status\": \"OK\"}"
with pytest.raises(KeyError):
JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "missing_field"])
Troubleshooting
Troubleshooting: Day 249 - Structured Output: Getting Reliable JSON
Common Issues
- Unescaped quotes inside strings:
- Cause: Text quotes inside json fields.
- Fix: Use structured grammar decoding or regex pre-sanitization.
Security notes
Security & Privacy: Day 249 - Structured Output: Getting Reliable JSON
Security Guidance
- Strongly typed Pydantic models prevent SQL injection and downstream payload execution.