LLMs and Generative AI βΊ LLM APIs βΊ Day 257
Day 257: Working with Images and Documents
Master multimodal API processing: construct Base64 and hosted URL multimodal message payloads for images (JPEG, PNG, WebP) and documents (PDF, text, CSV) across Anthropic (document / image source blocks) and OpenAI (image_url content parts) APIs, understand vision tokenization and image patch resolution grids (High vs. Low detail modes), extract structured JSON data from complex financial tables and architectural blueprints, and build a multimodal document analysis engine in Python.
Hands-on lab for this lesson
Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-257-working-with-images-and-documents
- Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git cd ai-roadmap-365.github.io - Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β section / subsection / week / day:
cd labs/sections/llms-and-generative-ai/day-257-working-with-images-and-documents - Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β read it before running anything.
- Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
bash tests/run_tests.sh # or the test command named in the lab README
You can also open the lab as a local page (works offline, shows the file tree and expected output).
Learning objectives
By the end of this lesson you will be able to:
- Encode local images and PDF documents into Base64 data URIs for API payloads.
- Construct multimodal message arrays using Anthropic and OpenAI image content formats.
- Calculate vision token costs across low-detail and tiled high-detail image patch modes.
- Extract structured JSON entities from image scans, receipts, and multi-page PDFs.
- Build an automated multimodal image and document analyzer in Python.
Prerequisites
Why this matters
The real world is not purely text. The most valuable information in enterprise organizations lives trapped inside unstructured visual formats: scanned invoices, medical x-rays, multi-page financial PDF statements, architectural blueprints, UI wireframes, and handwritten whiteboards.
Traditional OCR engines (such as Tesseract) struggle immensely when layouts become complex, failing on nested tables, multi-column articles, and rotated text.
Modern Multimodal Language Models (MLLMs) β such as Claude 3.5 Sonnet, GPT-4o, and Gemini 1.5/2.0 β natively perceive visual pixels alongside linguistic tokens:
- They inspect visual spatial layout, table cell boundaries, font weights, and hierarchical indentation.
- They cross-reference charts and graphs with accompanying textual footnotes.
- They extract clean, validated Pydantic JSON structures directly from raw imagery with near-human reading comprehension.
Today, you will master Base64 media encoding, formulate multimodal payload arrays across Anthropic and OpenAI APIs, calculate vision patch token economics, and build an automated multimodal document analysis engine in Python.
The idea in plain language
Think of a doctor examining a patient chart:
- Text-Only Approach (Traditional OCR): A clerk transcribes the patient file by reading characters left-to-right without understanding formatting. If the file has a 2-column table with lab values, the text becomes garbled, mixing heart rate numbers with cholesterol levels (Layout Destruction & Medical Errors).
- Multimodal Visual Intelligence (MLLM):
- The doctor holds the physical x-ray and lab report in their hands (Image & Document Payload).
- Their visual cortex processes spatial geometry: identifying bone fractures in the scan while reading the exact blood pressure numbers in the top-right box (Vision Patch Attention).
- The doctor writes a structured diagnosis synthesizing visual evidence with medical reasoning (Structured JSON Output).
Historical background
- 2020 (Vision Transformer ViT): Google Research proved that treating image patches like text tokens allowed transformer architectures to excel at computer vision tasks.
- 2023 (GPT-4V & Multimodal API Emergence): OpenAI launched GPT-4 Vision, allowing developers to pass image URLs and base64 strings directly in
/v1/chat/completions. - 2024 (Claude 3.5 & Native PDF Understanding): Anthropic introduced high-precision OCR and native PDF document block ingestion, allowing direct multi-page document parsing without third-party converters.
- 2024 (Omni Architectures): Native multimodal tokenizers unified audio, vision, video, and text into a single cohesive autoregressive transformer backbone.
What it is β and what it is not
What Multimodal API Processing IS:
- Spatial Visual Reasoning: Sending raw images or PDFs to a foundation model to extract structured data, diagnose anomalies, or analyze visual geometry.
What it is NOT:
- Not Pixel Generation / Diffusion: MLLMs analyze and interpret existing imagery; they do not generate new photorealistic diffusion artwork (like Midjourney or FLUX).
Why it was created and what problems it solves
Multimodal API processing eliminates 4 legacy enterprise hurdles:
- Broken OCR Pipelines: Replaces fragile multi-step OCR + regex parsing stacks with a single end-to-end model call.
- Complex Layout Understanding: Seamlessly parses nested tables, rotated receipts, handwritten forms, and multi-column magazine layouts.
- Chart & Diagram Synthesis: Interprets line charts, scatter plots, and network flowcharts, outputting the underlying numerical data points.
- UI Wireframe-to-Code: Converts screenshots of Figma wireframes directly into functioning React/Tailwind frontend code.
How it works
Let us examine Base64 encoding, API message schemas, and vision token cost mathematics.
1. Encoding Local Images into Base64 in Python
import base64
from pathlib import Path
def encode_image_to_base64(image_path: str) -> str:
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
- The resulting Base64 string represents raw image bytes encoded in 64 ASCII characters, allowing binary image data to be serialized safely inside standard JSON request bodies without binary transmission corruption.
- In addition, validating file size boundaries before Base64 conversion ensures that memory allocations remain bounded when processing high-resolution media.
2. OpenAI Vision API Message Schema
from openai import OpenAI
client = OpenAI()
b64_image = encode_image_to_base64("invoice.png")
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract invoice number, date, and total amount as JSON."},
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{b64_image}",
"detail": "high" # "low", "high", or "auto"
}
}
]
}
],
max_tokens=500
)
print(response.choices[0].message.content)
3. Anthropic Vision & PDF Document Message Schema
import anthropic
client = anthropic.Anthropic()
b64_pdf = encode_image_to_base64("quarterly_report.pdf")
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[
{
"role": "user",
"content": [
{
"type": "document",
"source": {
"type": "base64",
"media_type": "application/pdf",
"data": b64_pdf
}
},
{"type": "text", "text": "Summarize key financial risks mentioned in this PDF."}
]
}
]
)
print(response.content[0].text)
4. Vision Token Economics: Low-Detail vs. High-Detail Mode
How do vision APIs calculate prompt token costs?
- Low-Detail Mode (
detail: "low"):- The image is downsampled to a fixed
512 * 512square. - Cost is flat: exactly 85 tokens, regardless of the original image dimensions.
- Ideal for broad scene tagging and thumbnail indexing.
- The image is downsampled to a fixed
- High-Detail Mode (
detail: "high"):-
The image is scaled to fit within a
2048 * 2048box. -
The shortest side is scaled to
768px. -
The canvas is partitioned into
512 * 512pixel tiles. -
Each tile consumes 170 tokens.
-
An additional 85 base tokens are added for the global overview:
Total Tokens = (N_{tiles} * 170) + 85 -
Example: A
1024 * 1024image creates 4 tiles:(4 * 170) + 85 = 765 tokens.
-
5. Multi-Image Comparative Reasoning & Cross-Modal Attention
You can pass multiple image blocks inside a single content array:
- Medical Diagnostics: Compare before and after CT scans across 6-month intervals to detect tumor shrinkage or bone remodeling.
- Visual Diffing & Quality Assurance: Submit expected design mockups alongside live frontend screenshots to automatically identify CSS layout regressions and misaligned margins.
- Multi-Angle Object Inspection: Provide 4 orthographic views (top, side, front, isometric) of an industrial component to evaluate mechanical tolerances and surface defects.
- Cross-Modal Attention Mechanics: The underlying transformer cross-attends between image patch embeddings from different images simultaneously, allowing the model to perform holistic spatial comparison without requiring multi-turn conversational back-and-forth.
6. Extracting Structured Pydantic Schemas from Complex Image Scans
Combining vision inputs with strict JSON schemas transforms messy physical paperwork into strongly typed database records:
from pydantic import BaseModel, Field
from typing import List
class LineItem(BaseModel):
description: str = Field(description="Item name or service description")
quantity: int = Field(description="Units purchased")
unit_price: float = Field(description="Price per unit in USD")
total_price: float = Field(description="Total line item cost")
class InvoiceExtraction(BaseModel):
invoice_number: str = Field(description="Unique invoice identifier")
vendor_name: str = Field(description="Company issuing the invoice")
invoice_date: str = Field(description="Date formatted as YYYY-MM-DD")
subtotal: float = Field(description="Pre-tax amount")
tax_amount: float = Field(description="Calculated tax")
total_amount: float = Field(description="Final total charge")
items: List[LineItem] = Field(description="List of purchased line items")
- When instructed with schema definitions, the multimodal model cross-references table column headers with individual numerical values, correctly resolving misaligned lines, currency symbols, and multi-line item descriptions that break conventional regex and OCR parsers.
7. Client-Side Image Pre-Processing & Dimension Optimization
Sending raw 48-megapixel photos directly to cloud vision APIs causes massive network latency and unnecessary token billing:
-
Optimal Scaling Dimensions: Vision models partition images using
512 * 512pixel patch grids. Scaling images such that the shortest side is768pxand the longest side does not exceed2048pxretains 100% of the visual fidelity needed for small text reading while minimizing tile counts. -
Client-Side Python Compression Pipeline:
from PIL import Image import io def optimize_image_for_vision(image_path: str, max_dimension: int = 2048) -> bytes: with Image.open(image_path) as img: # Convert RGBA to RGB for JPEG serialization if img.mode in ("RGBA", "P"): img = img.convert("RGB") img.thumbnail((max_dimension, max_dimension), Image.Resampling.LANCZOS) buffer = io.BytesIO() img.save(buffer, format="JPEG", quality=85, optimize=True) return buffer.getvalue() -
This automated pre-processing step slashes Base64 payload sizes by up to 80% and accelerates HTTP upload times significantly.
8. Multi-Page PDF Document Ingestion & Sliding Window Budgeting
When analyzing enterprise PDF documents (such as 100-page SEC 10-K filings or clinical trials reports):
- Native Document Blocks vs. Page Rendering: Anthropic Claude 3.5 Sonnet natively accepts PDF document blocks up to 32MB. Internally, each PDF page is rasterized into high-resolution vision tiles consuming approximately 1,500 prompt tokens per page.
- Token Budget Protection: Submitting a 50-page PDF in a single turn consumes ~75,000 input tokens.
- Selective Page Slicing Strategy:
- Use lightweight libraries (
pypdforpdfplumber) to extract the document table of contents or search for targeted keywords. - Slice only the relevant 3 to 5 page subset into an in-memory PDF buffer.
- Submit the focused PDF slice to the multimodal API, preserving prompt token limits and slashing inference costs.
- Use lightweight libraries (
9. Spatial Grounding & Normalized Bounding Box Coordinates
Modern frontier multimodal models can locate specific UI elements and physical objects using Normalized Bounding Box Coordinates:
- The model expresses object locations as a 4-element array:
[ymin, xmin, ymax, xmax]normalized on a scale from0to1000(or0.0to1.0). - Visual Grounding Applications:
- GUI Navigation Agents: Identifying the exact click coordinates
[ymin, xmin]for a βSubmit Applicationβ button in web browser automation. - Defect Detection in Manufacturing: Highlighting micro-cracks on circuit board scans with bounding boxes.
- Chart Data Verification: Returning bounding coordinates of legend labels and bar chart peaks to provide verifiable visual citations.
- GUI Navigation Agents: Identifying the exact click coordinates
10. Automated Redaction & PII Masking in Multimodal Pipelines
Processing sensitive documents (e.g. government IDs, tax returns, medical records) requires strict privacy controls:
- Pre-Ingestion Redaction: Detect high-entropy text regions (Social Security Numbers, Credit Card numbers) using local OCR or facial recognition cascades and apply Gaussian blur or solid black rectangles prior to Base64 serialization.
- Zero Data Retention Compliance: Ensure enterprise API agreements mandate that uploaded image and document payloads are never cached, logged, or utilized for model training.
- In addition, combining visual OCR with automated schema validation guarantees that extracted entities conform to strict enterprise data contracts before downstream ingestion into analytical data warehouses.
- Furthermore, modern multimodal pipelines maintain end-to-end data provenance by recording source image coordinates alongside extracted numerical fields for automated compliance auditing.
- In summary, mastering multimodal image and document engineering empowers developers to bridge physical paperwork with automated digital intelligence.
An everyday analogy
Think of a magnifying glass with adjustable zoom:
- Low-Detail Mode: You hold the magnifying glass 3 feet above a map to see whether it is a map of France or Germany (85 Tokens / Broad Classification).
- High-Detail Mode: You press the magnifying glass directly against the paper, moving it square by square across each town and street name (765 Tokens / Precise High-Resolution Reading).
Examples in practice
Let us build an Automated Multimodal Document Analyzer in Python:
import base64
from typing import Dict, Any, List
class MultimodalDocumentAnalyzer:
def __init__(self):
self.supported_image_types = ["image/jpeg", "image/png", "image/webp"]
self.supported_doc_types = ["application/pdf"]
def build_image_payload(self, base64_data: str, media_type: str, prompt: str) -> Dict[str, Any]:
if media_type not in self.supported_image_types:
raise ValueError(f"Unsupported image media type: {media_type}")
return {
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": media_type,
"data": base64_data
}
},
{"type": "text", "text": prompt}
]
}
def calculate_vision_tokens(self, width: int, height: int, detail: str = "high") -> int:
if detail == "low":
return 85
# Calculate 512x512 tiles for high-detail mode
import math
tiles_x = math.ceil(min(width, 2048) / 512)
tiles_y = math.ceil(min(height, 2048) / 512)
total_tiles = tiles_x * tiles_y
return (total_tiles * 170) + 85
Implications: security, privacy, performance, scalability, and cost
- Client-Side Compression:
- Resizing high-resolution images prior to Base64 encoding reduces HTTP upload latency by 5x.
- Deterministic Token Billing:
- Pre-computing tile token counts prevents budget overruns before submitting large batches of image scans.
Alternatives: free, open source, and commercial
| Vision Solution | Type | OCR Precision | Best Use Case |
|---|---|---|---|
| Claude 3.5 Sonnet | Commercial MLLM | State-of-the-Art | Complex PDF documents & multi-column tables |
| GPT-4o Vision | Commercial MLLM | State-of-the-Art | General visual QA & chart understanding |
| Llama 3.2 Vision (OSS) | Open-Weights | Strong | Self-hosted on-premise vision inference |
| Tesseract OCR (OSS) | Rule/CNN | Basic | Plain single-column text extraction |
Comparison with related concepts
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β VISION PROCESSING COMPARISON β
ββββββββββββββββββββββ¬ββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββ€
β Feature β Low-Detail Mode β High-Detail Mode β Native PDF β
ββββββββββββββββββββββΌββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββ€
β Token Cost β Fixed 85 Tokens β Tiled (765+ Tok) β ~1500 T/Page β
β OCR Capability β Minimal β High-Precision β High-Precisionβ
β Processing Latency β Fast (< 500ms) β Moderate (1-2s) β Moderate β
β Best For β Scene Tagging β Tables/Diagrams β Legal Reportsβ
ββββββββββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββ
When to use it β and when not to
When to Use Vision APIs:
- Document OCR, receipt and invoice extraction, visual chart analysis, wireframe-to-code, medical imaging research, and visual defect inspection.
When NOT to use:
- Documents that already exist as clean digital text or CSV files (sending raw text directly is 10x cheaper and faster).
The AI connection
- Multimodal input removed the need for a separate extraction pipeline β a scanned form and a chart are now ordinary inputs.
- Images are billed as tokens, so resolution and page count are direct cost decisions rather than technical details.
- Document understanding is the highest-value early use in most organisations, because it replaces manual data entry.
- Vision inherits every text failure mode and adds its own, including confident misreading of low-quality scans.
- An image can carry a prompt injection, which makes untrusted documents an attack surface as much as untrusted text.
Knowledge check
- What encoding standard is used to embed binary images into JSON API payloads?
- How is token consumption calculated in OpenAI βdetail: highβ vision mode?
- How does Claude 3.5 Sonnet process multi-page PDF documents natively?
- When should an application select βdetail: lowβ instead of βdetail: highβ?
- Why should high-resolution photographs be resized client-side before Base64 encoding?
Hands-on exercise
In this lab, you will build and test a Multimodal Document Analysis Engine in Python: formulate Base64-encoded image and PDF message blocks, calculate exact tile token economics for various image resolutions, and verify output assertions against automated test suites.
Expected output
[Multimodal Vision & Document Processing Verification Suite]
Testing Base64 Payload Formulation:
Image Block Media Type: 'image/png' [VALIDATED]
Content Array Structure: 2 blocks (image + prompt) [PASS]
Testing Vision Token Cost Calculator:
Low-Detail Mode: 85 tokens [PASS]
High-Detail Mode (1024x1024): 765 tokens (4 tiles) [PASS]
High-Detail Mode (1920x1080): 1445 tokens (8 tiles) [PASS]
Test Suite: 3 passed in 0.38s
Validate your work
Run the automated test runner:
./tests/run_tests.sh
Troubleshooting
- Ensure media_type strings use valid IANA MIME types (e.g.
image/jpeg, notimage/jpg).
Common mistakes
- Omitting the data URI prefix in OpenAI format: OpenAI requires
data:image/png;base64,{data}, whereas Anthropic accepts the raw Base64 string insource.data.
Practice assignment
- Implement an automated Receipt Expense Parser in Python that extracts Merchant, Date, Tax, and Total from receipt images into a typed Pydantic dataclass.
- Build a PDF Multi-Page Token Estimator that inspects PDF page count and calculates projected API costs before dispatching requests.
Extension challenge
Implement a Smart Visual Redaction Gateway:
- Build a Python pre-processor that detects high-entropy text blocks (e.g. credit card numbers or passport signatures) in image scans and applies Gaussian blur masks to those coordinates before Base64 serialization.
Quiz
Q1. What is the standard encoding used to transmit local image files directly inside API JSON payloads?
- Zipping the images into an archive
- Raw binary hexadecimal strings without headers
- Uploading files via FTP
- Base64 with a declared media type
Show answer
Answer: D. Base64 with a declared media type
Base64 encoding converts binary media bytes into standard ASCII text compatible with JSON payloads.
Q2. In OpenAI Vision models (e.g. GPT-4o), how does "detail: high" calculate token consumption?
- It tiles the image into 512x512 blocks and counts them
- It charges a flat 10,000 tokens per image
- It charges 1 token per pixel
- It is completely free
Show answer
Answer: A. It tiles the image into 512x512 blocks and counts them
High-detail mode slices large images into 512x512 tiles at 170 tokens per tile plus 85 base tokens.
Q3. How does Anthropic Claude 3.5 Sonnet natively process multi-page PDF documents in the Messages API?
- By requiring the user to convert PDF text to Word .docx
- With a document block of base64 PDF data
- By downloading the PDF via BitTorrent
- PDFs cannot be processed by Claude
Show answer
Answer: B. With a document block of base64 PDF data
Claude natively accepts base64-encoded PDF document content blocks directly in the messages array.
Q4. When is "detail: low" the recommended setting for image processing?
- When reading architectural blueprints
- When extracting tiny handwritten numbers from financial checks
- When only coarse recognition is needed, not OCR
- When reading medicine bottle labels
Show answer
Answer: C. When only coarse recognition is needed, not OCR
Low-detail mode is ultra-fast and consumes only 85 tokens, perfect for high-level scene categorization.
Q5. What is Visual Question Answering (VQA) in multimodal engineering?
- Compressing MP4 video files
- Playing video games with a gamepad
- Taking screenshots of code
- An image plus a question, answered as text or JSON
Show answer
Answer: D. An image plus a question, answered as text or JSON
VQA combines visual feature embeddings with linguistic reasoning to answer questions about imagery.
Glossary
- Multimodal LLM (MLLM)
- A foundation model capable of simultaneously processing and reasoning across multiple data modalities including text, images, audio, and documents.
- Base64 Encoding
- A binary-to-text encoding scheme translating raw media bytes into standard ASCII string representations for JSON transmission.
- Vision Token
- A vector embedding representation of an image patch processed by the vision transformer encoder.
- Image Patch Tiling
- The process of partitioning a high-resolution image into discrete sub-grid tiles (e.g. 512x512 pixels) for detailed inspection.
- High-Detail Mode
- An API vision setting that decomposes images into high-resolution patch grids to enable fine-grained OCR and diagram reading.
- Low-Detail Mode
- An API vision setting that resizes images to a low fixed resolution (85 tokens) for rapid broad scene classification.
- Visual Document Extraction
- The automated parsing of unstructured visual documents (PDFs, invoices, receipts) into validated structured JSON schemas.
- MIME Type
- A standardized label (e.g. image/png, application/pdf) identifying the media format of an encoded payload.
Sources and further reading
- Vision Processing with OpenAI Chat Completions β OpenAI Developer Documentation (accessed 2026-08-30)
- Visual Capabilities and PDF Processing in Claude β Anthropic Developer Documentation (accessed 2026-08-30)
- Patch-Level Vision Transformer Tokenization β Google Research / ICLR (accessed 2026-08-30)
Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.