LLMs and Generative AI › Prompt Engineering › Day 249

Hands-on lab — Day 249: Structured Output: Getting Reliable JSON

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/structured_output_getting_reliable_json_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/structured_output_getting_reliable_json_lib.py
examples/test_structured_output_getting_reliable_json_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/structured_output_getting_reliable_json_lib.py
starter/test_structured_output_getting_reliable_json_lib.py
tests/run_tests.sh
tests/test_structured_output_getting_reliable_json_lib.py
troubleshooting.md

Lab README

Lab: Day 249 -- Structured Output: Getting Reliable JSON

Lesson

Day number: 249 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Structured Output, JSON Generation, and Constrained CFG Decoding.

Purpose

Build and test a type-safe JSON extraction and auto-repair engine in Python. Sanitize markdown code fences and conversational preambles, repair trailing commas and unclosed brackets, and validate schemas.

Learning objectives

  • Implement regex and AST sanitization for raw LLM JSON responses.
  • Auto-repair trailing commas and truncated closing brackets.
  • Validate required schema keys against strict contracts.
  • Understand the mechanics of Constrained Grammar FSM logit masking.

Prerequisites

  • Day 248 (Few-Shot Examples and Chain of Thought).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python standard json and regex modules are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/structured_output_getting_reliable_json_lib.py: Student scaffold file.
  • examples/structured_output_getting_reliable_json_lib.py: Complete reference implementation.
  • tests/test_structured_output_getting_reliable_json_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/structured_output_getting_reliable_json_lib.py

What the commands do

  • Cleans conversational preambles and markdown fences.
  • Auto-repairs truncated JSON strings.
  • Runs unit test assertions.

Expected output

JSON Demo Extracted Successfully: INC-101

Validation steps

  1. Verify clean_json_string removes markdown code fences.
  2. Confirm trailing comma regex fixes trailing commas.
  3. Validate missing key detection raises KeyError.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • No JSON delimiter: Ensure the raw input contains at least one object or array opening delimiter.

Security notes

Never use eval() to parse untrusted model outputs.

Extension exercises

  1. Implement a nested Pydantic V2 validator with custom regex validators.
  2. Build an FSM-based logit masking simulator for token vocabularies.

Expected output

FIELDS.md

# Expected Output Fields: Day 249

- `JSON Demo`: Status of JSON extraction and auto-repair.

examples-run.txt

JSON Demo Extracted Successfully: INC-101

measured-values.txt

JSON Demo: Success
Extracted ID: INC-101
Severity: HIGH

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_structured_output_getting_reliable_json_lib.py::test_json_markdown_cleaning PASSED [ 33%]
tests/test_structured_output_getting_reliable_json_lib.py::test_auto_closing_truncated_payload PASSED [ 66%]
tests/test_structured_output_getting_reliable_json_lib.py::test_missing_required_keys PASSED [100%]

============================== 3 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/structured_output_getting_reliable_json_lib.py (2008 bytes)
import json
import re
from typing import Dict, Any, Optional

class JSONExtractorEngine:
    @staticmethod
    def clean_json_string(raw_text: str) -> str:
        cleaned = re.sub(r"^```(?:json)?\s*", "", raw_text.strip(), flags=re.IGNORECASE)
        cleaned = re.sub(r"\s*```$", "", cleaned)

        first_brace = cleaned.find("{")
        first_bracket = cleaned.find("[")

        if first_brace == -1 and first_bracket == -1:
            raise ValueError("No JSON object or array delimiter found in response.")

        if first_brace != -1 and (first_bracket == -1 or first_brace < first_bracket):
            start_idx = first_brace
            end_idx = cleaned.rfind("}")
            if end_idx != -1:
                cleaned = cleaned[start_idx:end_idx + 1]
            else:
                cleaned = cleaned[start_idx:] + "}"
        else:
            start_idx = first_bracket
            end_idx = cleaned.rfind("]")
            if end_idx != -1:
                cleaned = cleaned[start_idx:end_idx + 1]
            else:
                cleaned = cleaned[start_idx:] + "]"

        cleaned = re.sub(r",\s*([\}\]])", r"\1", cleaned)
        return cleaned.strip()

    @classmethod
    def parse_and_validate(cls, raw_text: str, required_keys: Optional[list] = None) -> Dict[str, Any]:
        cleaned = cls.clean_json_string(raw_text)
        data = json.loads(cleaned)

        if required_keys and isinstance(data, dict):
            missing = [k for k in required_keys if k not in data]
            if missing:
                raise KeyError(f"Missing required schema keys: {missing}")

        return data

def run_json_demo():
    raw_payload = "Sure! Here is the data: ```json\n{\"incident_id\": \"INC-101\", \"severity\": \"HIGH\",}\n```"
    result = JSONExtractorEngine.parse_and_validate(raw_payload, required_keys=["incident_id", "severity"])

    print("JSON Demo Extracted Successfully:", result["incident_id"])
    return result

if __name__ == "__main__":
    run_json_demo()
examples/test_structured_output_getting_reliable_json_lib.py (748 bytes)
import pytest
from examples.structured_output_getting_reliable_json_lib import JSONExtractorEngine

def test_json_markdown_cleaning():
    raw = "Sure! Here is the output:\n```json\n{\"status\": \"OK\", \"code\": 200,}\n```"
    data = JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "code"])

    assert data["status"] == "OK"
    assert data["code"] == 200

def test_auto_closing_truncated_payload():
    raw = "{\"status\": \"PENDING\""
    cleaned = JSONExtractorEngine.clean_json_string(raw)
    assert cleaned.endswith("}")

def test_missing_required_keys():
    raw = "{\"status\": \"OK\"}"
    with pytest.raises(KeyError):
        JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "missing_field"])
metadata.yml (452 bytes)
lesson_id: D249
day: 249
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/structured_output_getting_reliable_json_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/structured_output_getting_reliable_json_lib.py (396 bytes)
import json
import re
from typing import Dict, Any, Optional

class JSONExtractorEngine:
    @staticmethod
    def clean_json_string(raw_text: str) -> str:
        # TODO: Strip markdown and preamble
        pass

    @classmethod
    def parse_and_validate(cls, raw_text: str, required_keys: Optional[list] = None) -> Dict[str, Any]:
        # TODO: Clean, parse, and validate keys
        pass
starter/test_structured_output_getting_reliable_json_lib.py (748 bytes)
import pytest
from examples.structured_output_getting_reliable_json_lib import JSONExtractorEngine

def test_json_markdown_cleaning():
    raw = "Sure! Here is the output:\n```json\n{\"status\": \"OK\", \"code\": 200,}\n```"
    data = JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "code"])

    assert data["status"] == "OK"
    assert data["code"] == 200

def test_auto_closing_truncated_payload():
    raw = "{\"status\": \"PENDING\""
    cleaned = JSONExtractorEngine.clean_json_string(raw)
    assert cleaned.endswith("}")

def test_missing_required_keys():
    raw = "{\"status\": \"OK\"}"
    with pytest.raises(KeyError):
        JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "missing_field"])
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 249 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_structured_output_getting_reliable_json_lib.py (748 bytes)
import pytest
from examples.structured_output_getting_reliable_json_lib import JSONExtractorEngine

def test_json_markdown_cleaning():
    raw = "Sure! Here is the output:\n```json\n{\"status\": \"OK\", \"code\": 200,}\n```"
    data = JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "code"])

    assert data["status"] == "OK"
    assert data["code"] == 200

def test_auto_closing_truncated_payload():
    raw = "{\"status\": \"PENDING\""
    cleaned = JSONExtractorEngine.clean_json_string(raw)
    assert cleaned.endswith("}")

def test_missing_required_keys():
    raw = "{\"status\": \"OK\"}"
    with pytest.raises(KeyError):
        JSONExtractorEngine.parse_and_validate(raw, required_keys=["status", "missing_field"])

Troubleshooting

Troubleshooting: Day 249 - Structured Output: Getting Reliable JSON

Common Issues

  1. Unescaped quotes inside strings:
    • Cause: Text quotes inside json fields.
    • Fix: Use structured grammar decoding or regex pre-sanitization.

Security notes

Security & Privacy: Day 249 - Structured Output: Getting Reliable JSON

Security Guidance

  • Strongly typed Pydantic models prevent SQL injection and downstream payload execution.