LLMs and Generative AI › LLM APIs › Day 257
Hands-on lab — Day 257: Working with Images and Documents
- ← Back to the Day 257 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-257-working-with-images-and-documents/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/working_with_images_and_documents_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/test_working_with_images_and_documents_lib.py examples/working_with_images_and_documents_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/test_working_with_images_and_documents_lib.py starter/working_with_images_and_documents_lib.py tests/run_tests.sh tests/test_working_with_images_and_documents_lib.py troubleshooting.md
Lab README
Lab: Day 257 -- Working with Images and Documents
Lesson
Day number: 257 of 365. Course: Course06-SS02 (LLMs and Generative AI - LLM APIs). Topic: Working with Images and Documents, Base64 Payloads, and Vision Token Economics.
Purpose
Build and test a Multimodal Document Analysis Engine in Python. Formulate Base64-encoded image and PDF payloads, validate media types, and compute vision tile token costs.
Learning objectives
- Encode image files into Base64 ASCII strings.
- Structure multimodal message arrays matching standard schemas.
- Calculate vision token costs across resolution detail modes.
- Implement media validation guardrails.
Prerequisites
- Day 256 (Tool Use and Function Calling).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard base64/math modules are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/working_with_images_and_documents_lib.py: Student scaffold file.examples/working_with_images_and_documents_lib.py: Complete reference implementation.tests/test_working_with_images_and_documents_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/working_with_images_and_documents_lib.py
What the commands do
- Constructs Base64 image payload.
- Calculates vision token costs.
- Runs unit test assertions.
Expected output
Multimodal Demo Executed. Tokens: 765
Validation steps
- Verify payload construction includes image and prompt blocks.
- Confirm unsupported media types raise ValueError.
- Validate low-detail returns fixed 85 tokens.
- Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Invalid media type error: Use standard MIME types (e.g.
image/jpeg).
Security notes
Base64 payload processing executes locally in memory.
Extension exercises
- Build an automated receipt expense parser with Pydantic.
- Implement automated client-side image downscaling.
Navigation
- Lesson title: Working with Images and Documents
- Day number: 257 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-257-working-with-images-and-documents
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-257-working-with-images-and-documentswhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 257
- `Multimodal Demo`: Status of multimodal payload generation.
examples-run.txt
Multimodal Demo Executed. Tokens: 765
measured-values.txt
Multimodal Demo: Success
Vision Tokens (1024x1024): 765
Payload Media Type: image/png
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_working_with_images_and_documents_lib.py::test_payload_construction PASSED [ 33%]
tests/test_working_with_images_and_documents_lib.py::test_unsupported_media_type PASSED [ 66%]
tests/test_working_with_images_and_documents_lib.py::test_vision_token_calculations PASSED [100%]
============================== 3 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/test_working_with_images_and_documents_lib.py (1194 bytes)
import pytest
from examples.working_with_images_and_documents_lib import (
MultimodalDocumentAnalyzer
)
def test_payload_construction():
analyzer = MultimodalDocumentAnalyzer()
payload = analyzer.build_image_payload("b64mockdata", "image/png", "Extract text")
assert payload["role"] == "user"
assert len(payload["content"]) == 2
assert payload["content"][0]["type"] == "image"
assert payload["content"][0]["source"]["media_type"] == "image/png"
assert payload["content"][1]["text"] == "Extract text"
def test_unsupported_media_type():
analyzer = MultimodalDocumentAnalyzer()
with pytest.raises(ValueError):
analyzer.build_image_payload("b64mockdata", "image/bmp", "Extract text")
def test_vision_token_calculations():
analyzer = MultimodalDocumentAnalyzer()
# Low detail
assert analyzer.calculate_vision_tokens(2000, 2000, "low") == 85
# High detail 1024x1024 = 2x2 = 4 tiles -> 4 * 170 + 85 = 765
assert analyzer.calculate_vision_tokens(1024, 1024, "high") == 765
# High detail 1920x1080 -> tiles_x=4, tiles_y=3 -> 12 tiles -> 12 * 170 + 85 = 2125
assert analyzer.calculate_vision_tokens(512, 512, "high") == 255
examples/working_with_images_and_documents_lib.py (1673 bytes)
import base64
import math
from typing import Dict, Any, List
class MultimodalDocumentAnalyzer:
def __init__(self):
self.supported_image_types = ["image/jpeg", "image/png", "image/webp"]
self.supported_doc_types = ["application/pdf"]
def build_image_payload(self, base64_data: str, media_type: str, prompt: str) -> Dict[str, Any]:
if media_type not in self.supported_image_types:
raise ValueError(f"Unsupported image media type: {media_type}")
return {
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": media_type,
"data": base64_data
}
},
{"type": "text", "text": prompt}
]
}
def calculate_vision_tokens(self, width: int, height: int, detail: str = "high") -> int:
if detail == "low":
return 85
tiles_x = math.ceil(min(width, 2048) / 512)
tiles_y = math.ceil(min(height, 2048) / 512)
total_tiles = tiles_x * tiles_y
return (total_tiles * 170) + 85
def run_multimodal_demo():
analyzer = MultimodalDocumentAnalyzer()
payload = analyzer.build_image_payload("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "image/png", "Extract invoice.")
tokens = analyzer.calculate_vision_tokens(1024, 1024, "high")
print(f"Multimodal Demo Executed. Tokens: {tokens}")
return payload, tokens
if __name__ == "__main__":
run_multimodal_demo()
metadata.yml (446 bytes)
lesson_id: D257
day: 257
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/working_with_images_and_documents_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/test_working_with_images_and_documents_lib.py (1194 bytes)
import pytest
from examples.working_with_images_and_documents_lib import (
MultimodalDocumentAnalyzer
)
def test_payload_construction():
analyzer = MultimodalDocumentAnalyzer()
payload = analyzer.build_image_payload("b64mockdata", "image/png", "Extract text")
assert payload["role"] == "user"
assert len(payload["content"]) == 2
assert payload["content"][0]["type"] == "image"
assert payload["content"][0]["source"]["media_type"] == "image/png"
assert payload["content"][1]["text"] == "Extract text"
def test_unsupported_media_type():
analyzer = MultimodalDocumentAnalyzer()
with pytest.raises(ValueError):
analyzer.build_image_payload("b64mockdata", "image/bmp", "Extract text")
def test_vision_token_calculations():
analyzer = MultimodalDocumentAnalyzer()
# Low detail
assert analyzer.calculate_vision_tokens(2000, 2000, "low") == 85
# High detail 1024x1024 = 2x2 = 4 tiles -> 4 * 170 + 85 = 765
assert analyzer.calculate_vision_tokens(1024, 1024, "high") == 765
# High detail 1920x1080 -> tiles_x=4, tiles_y=3 -> 12 tiles -> 12 * 170 + 85 = 2125
assert analyzer.calculate_vision_tokens(512, 512, "high") == 255
starter/working_with_images_and_documents_lib.py (495 bytes)
import base64
import math
from typing import Dict, Any, List
class MultimodalDocumentAnalyzer:
def __init__(self):
# TODO: Initialize supported media types
pass
def build_image_payload(self, base64_data: str, media_type: str, prompt: str) -> Dict[str, Any]:
# TODO: Construct multimodal content payload
pass
def calculate_vision_tokens(self, width: int, height: int, detail: str = "high") -> int:
# TODO: Calculate tile tokens
pass
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 257 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_working_with_images_and_documents_lib.py (1194 bytes)
import pytest
from examples.working_with_images_and_documents_lib import (
MultimodalDocumentAnalyzer
)
def test_payload_construction():
analyzer = MultimodalDocumentAnalyzer()
payload = analyzer.build_image_payload("b64mockdata", "image/png", "Extract text")
assert payload["role"] == "user"
assert len(payload["content"]) == 2
assert payload["content"][0]["type"] == "image"
assert payload["content"][0]["source"]["media_type"] == "image/png"
assert payload["content"][1]["text"] == "Extract text"
def test_unsupported_media_type():
analyzer = MultimodalDocumentAnalyzer()
with pytest.raises(ValueError):
analyzer.build_image_payload("b64mockdata", "image/bmp", "Extract text")
def test_vision_token_calculations():
analyzer = MultimodalDocumentAnalyzer()
# Low detail
assert analyzer.calculate_vision_tokens(2000, 2000, "low") == 85
# High detail 1024x1024 = 2x2 = 4 tiles -> 4 * 170 + 85 = 765
assert analyzer.calculate_vision_tokens(1024, 1024, "high") == 765
# High detail 1920x1080 -> tiles_x=4, tiles_y=3 -> 12 tiles -> 12 * 170 + 85 = 2125
assert analyzer.calculate_vision_tokens(512, 512, "high") == 255
Troubleshooting
Troubleshooting: Day 257 - Working with Images and Documents
Common Issues
- Unsupported media type error:
- Cause: Passing image/jpg instead of image/jpeg.
- Fix: Normalize image extensions to standard MIME strings.
Security notes
Security & Privacy: Day 257 - Working with Images and Documents
Security Guidance
- Redact PII (e.g. SSNs or credit cards) visually before sending document scans to cloud APIs.