LLMs and Generative AI › LLM APIs › Day 257

Hands-on lab — Day 257: Working with Images and Documents

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/working_with_images_and_documents_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/test_working_with_images_and_documents_lib.py
examples/working_with_images_and_documents_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/test_working_with_images_and_documents_lib.py
starter/working_with_images_and_documents_lib.py
tests/run_tests.sh
tests/test_working_with_images_and_documents_lib.py
troubleshooting.md

Lab README

Lab: Day 257 -- Working with Images and Documents

Lesson

Day number: 257 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: Working with Images and Documents, Base64 Payloads, and Vision Token Economics.

Purpose

Build and test a Multimodal Document Analysis Engine in Python. Formulate Base64-encoded image and PDF payloads, validate media types, and compute vision tile token costs.

Learning objectives

  • Encode image files into Base64 ASCII strings.
  • Structure multimodal message arrays matching standard schemas.
  • Calculate vision token costs across resolution detail modes.
  • Implement media validation guardrails.

Prerequisites

  • Day 256 (Tool Use and Function Calling).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard base64/math modules are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/working_with_images_and_documents_lib.py: Student scaffold file.
  • examples/working_with_images_and_documents_lib.py: Complete reference implementation.
  • tests/test_working_with_images_and_documents_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/working_with_images_and_documents_lib.py

What the commands do

  • Constructs Base64 image payload.
  • Calculates vision token costs.
  • Runs unit test assertions.

Expected output

Multimodal Demo Executed. Tokens: 765

Validation steps

  1. Verify payload construction includes image and prompt blocks.
  2. Confirm unsupported media types raise ValueError.
  3. Validate low-detail returns fixed 85 tokens.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Invalid media type error: Use standard MIME types (e.g. image/jpeg).

Security notes

Base64 payload processing executes locally in memory.

Extension exercises

  1. Build an automated receipt expense parser with Pydantic.
  2. Implement automated client-side image downscaling.

Expected output

FIELDS.md

# Expected Output Fields: Day 257

- `Multimodal Demo`: Status of multimodal payload generation.

examples-run.txt

Multimodal Demo Executed. Tokens: 765

measured-values.txt

Multimodal Demo: Success
Vision Tokens (1024x1024): 765
Payload Media Type: image/png

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_working_with_images_and_documents_lib.py::test_payload_construction PASSED   [ 33%]
tests/test_working_with_images_and_documents_lib.py::test_unsupported_media_type PASSED [ 66%]
tests/test_working_with_images_and_documents_lib.py::test_vision_token_calculations PASSED [100%]

============================== 3 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/test_working_with_images_and_documents_lib.py (1194 bytes)
import pytest
from examples.working_with_images_and_documents_lib import (
    MultimodalDocumentAnalyzer
)

def test_payload_construction():
    analyzer = MultimodalDocumentAnalyzer()
    payload = analyzer.build_image_payload("b64mockdata", "image/png", "Extract text")

    assert payload["role"] == "user"
    assert len(payload["content"]) == 2
    assert payload["content"][0]["type"] == "image"
    assert payload["content"][0]["source"]["media_type"] == "image/png"
    assert payload["content"][1]["text"] == "Extract text"

def test_unsupported_media_type():
    analyzer = MultimodalDocumentAnalyzer()
    with pytest.raises(ValueError):
        analyzer.build_image_payload("b64mockdata", "image/bmp", "Extract text")

def test_vision_token_calculations():
    analyzer = MultimodalDocumentAnalyzer()
    # Low detail
    assert analyzer.calculate_vision_tokens(2000, 2000, "low") == 85
    # High detail 1024x1024 = 2x2 = 4 tiles -> 4 * 170 + 85 = 765
    assert analyzer.calculate_vision_tokens(1024, 1024, "high") == 765
    # High detail 1920x1080 -> tiles_x=4, tiles_y=3 -> 12 tiles -> 12 * 170 + 85 = 2125
    assert analyzer.calculate_vision_tokens(512, 512, "high") == 255
examples/working_with_images_and_documents_lib.py (1673 bytes)
import base64
import math
from typing import Dict, Any, List

class MultimodalDocumentAnalyzer:
    def __init__(self):
        self.supported_image_types = ["image/jpeg", "image/png", "image/webp"]
        self.supported_doc_types = ["application/pdf"]

    def build_image_payload(self, base64_data: str, media_type: str, prompt: str) -> Dict[str, Any]:
        if media_type not in self.supported_image_types:
            raise ValueError(f"Unsupported image media type: {media_type}")
        return {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": media_type,
                        "data": base64_data
                    }
                },
                {"type": "text", "text": prompt}
            ]
        }

    def calculate_vision_tokens(self, width: int, height: int, detail: str = "high") -> int:
        if detail == "low":
            return 85
        tiles_x = math.ceil(min(width, 2048) / 512)
        tiles_y = math.ceil(min(height, 2048) / 512)
        total_tiles = tiles_x * tiles_y
        return (total_tiles * 170) + 85

def run_multimodal_demo():
    analyzer = MultimodalDocumentAnalyzer()
    payload = analyzer.build_image_payload("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "image/png", "Extract invoice.")
    tokens = analyzer.calculate_vision_tokens(1024, 1024, "high")
    print(f"Multimodal Demo Executed. Tokens: {tokens}")
    return payload, tokens

if __name__ == "__main__":
    run_multimodal_demo()
metadata.yml (446 bytes)
lesson_id: D257
day: 257
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/working_with_images_and_documents_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_working_with_images_and_documents_lib.py (1194 bytes)
import pytest
from examples.working_with_images_and_documents_lib import (
    MultimodalDocumentAnalyzer
)

def test_payload_construction():
    analyzer = MultimodalDocumentAnalyzer()
    payload = analyzer.build_image_payload("b64mockdata", "image/png", "Extract text")

    assert payload["role"] == "user"
    assert len(payload["content"]) == 2
    assert payload["content"][0]["type"] == "image"
    assert payload["content"][0]["source"]["media_type"] == "image/png"
    assert payload["content"][1]["text"] == "Extract text"

def test_unsupported_media_type():
    analyzer = MultimodalDocumentAnalyzer()
    with pytest.raises(ValueError):
        analyzer.build_image_payload("b64mockdata", "image/bmp", "Extract text")

def test_vision_token_calculations():
    analyzer = MultimodalDocumentAnalyzer()
    # Low detail
    assert analyzer.calculate_vision_tokens(2000, 2000, "low") == 85
    # High detail 1024x1024 = 2x2 = 4 tiles -> 4 * 170 + 85 = 765
    assert analyzer.calculate_vision_tokens(1024, 1024, "high") == 765
    # High detail 1920x1080 -> tiles_x=4, tiles_y=3 -> 12 tiles -> 12 * 170 + 85 = 2125
    assert analyzer.calculate_vision_tokens(512, 512, "high") == 255
starter/working_with_images_and_documents_lib.py (495 bytes)
import base64
import math
from typing import Dict, Any, List

class MultimodalDocumentAnalyzer:
    def __init__(self):
        # TODO: Initialize supported media types
        pass

    def build_image_payload(self, base64_data: str, media_type: str, prompt: str) -> Dict[str, Any]:
        # TODO: Construct multimodal content payload
        pass

    def calculate_vision_tokens(self, width: int, height: int, detail: str = "high") -> int:
        # TODO: Calculate tile tokens
        pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 257 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_working_with_images_and_documents_lib.py (1194 bytes)
import pytest
from examples.working_with_images_and_documents_lib import (
    MultimodalDocumentAnalyzer
)

def test_payload_construction():
    analyzer = MultimodalDocumentAnalyzer()
    payload = analyzer.build_image_payload("b64mockdata", "image/png", "Extract text")

    assert payload["role"] == "user"
    assert len(payload["content"]) == 2
    assert payload["content"][0]["type"] == "image"
    assert payload["content"][0]["source"]["media_type"] == "image/png"
    assert payload["content"][1]["text"] == "Extract text"

def test_unsupported_media_type():
    analyzer = MultimodalDocumentAnalyzer()
    with pytest.raises(ValueError):
        analyzer.build_image_payload("b64mockdata", "image/bmp", "Extract text")

def test_vision_token_calculations():
    analyzer = MultimodalDocumentAnalyzer()
    # Low detail
    assert analyzer.calculate_vision_tokens(2000, 2000, "low") == 85
    # High detail 1024x1024 = 2x2 = 4 tiles -> 4 * 170 + 85 = 765
    assert analyzer.calculate_vision_tokens(1024, 1024, "high") == 765
    # High detail 1920x1080 -> tiles_x=4, tiles_y=3 -> 12 tiles -> 12 * 170 + 85 = 2125
    assert analyzer.calculate_vision_tokens(512, 512, "high") == 255

Troubleshooting

Troubleshooting: Day 257 - Working with Images and Documents

Common Issues

  1. Unsupported media type error:
    • Cause: Passing image/jpg instead of image/jpeg.
    • Fix: Normalize image extensions to standard MIME strings.

Security notes

Security & Privacy: Day 257 - Working with Images and Documents

Security Guidance

  • Redact PII (e.g. SSNs or credit cards) visually before sending document scans to cloud APIs.