LLMs and Generative AI β€Ί LLM APIs β€Ί Day 257

Day 257: Working with Images and Documents

Day 257 of 365 β€” Working with Images and Documents

Master multimodal API processing: construct Base64 and hosted URL multimodal message payloads for images (JPEG, PNG, WebP) and documents (PDF, text, CSV) across Anthropic (document / image source blocks) and OpenAI (image_url content parts) APIs, understand vision tokenization and image patch resolution grids (High vs. Low detail modes), extract structured JSON data from complex financial tables and architectural blueprints, and build a multimodal document analysis engine in Python.

Course
LLMs and Generative AI
Category
LLM APIs
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-257-working-with-images-and-documents

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-257-working-with-images-and-documents
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

The real world is not purely text. The most valuable information in enterprise organizations lives trapped inside unstructured visual formats: scanned invoices, medical x-rays, multi-page financial PDF statements, architectural blueprints, UI wireframes, and handwritten whiteboards.

Traditional OCR engines (such as Tesseract) struggle immensely when layouts become complex, failing on nested tables, multi-column articles, and rotated text.

Modern Multimodal Language Models (MLLMs) β€” such as Claude 3.5 Sonnet, GPT-4o, and Gemini 1.5/2.0 β€” natively perceive visual pixels alongside linguistic tokens:

Today, you will master Base64 media encoding, formulate multimodal payload arrays across Anthropic and OpenAI APIs, calculate vision patch token economics, and build an automated multimodal document analysis engine in Python.


The idea in plain language

Diagram: how an image is processed into tokens a language model can attend to

Think of a doctor examining a patient chart:


Historical background

Diagram: the historical progression of automated document understanding

Multimodal vision and document payload pipeline diagram

  1. 2020 (Vision Transformer ViT): Google Research proved that treating image patches like text tokens allowed transformer architectures to excel at computer vision tasks.
  2. 2023 (GPT-4V & Multimodal API Emergence): OpenAI launched GPT-4 Vision, allowing developers to pass image URLs and base64 strings directly in /v1/chat/completions.
  3. 2024 (Claude 3.5 & Native PDF Understanding): Anthropic introduced high-precision OCR and native PDF document block ingestion, allowing direct multi-page document parsing without third-party converters.
  4. 2024 (Omni Architectures): Native multimodal tokenizers unified audio, vision, video, and text into a single cohesive autoregressive transformer backbone.

What it is β€” and what it is not

What Multimodal API Processing IS:

What it is NOT:


Why it was created and what problems it solves

Multimodal API processing eliminates 4 legacy enterprise hurdles:

  1. Broken OCR Pipelines: Replaces fragile multi-step OCR + regex parsing stacks with a single end-to-end model call.
  2. Complex Layout Understanding: Seamlessly parses nested tables, rotated receipts, handwritten forms, and multi-column magazine layouts.
  3. Chart & Diagram Synthesis: Interprets line charts, scatter plots, and network flowcharts, outputting the underlying numerical data points.
  4. UI Wireframe-to-Code: Converts screenshots of Figma wireframes directly into functioning React/Tailwind frontend code.

How it works

Let us examine Base64 encoding, API message schemas, and vision token cost mathematics.


1. Encoding Local Images into Base64 in Python

import base64
from pathlib import Path

def encode_image_to_base64(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

2. OpenAI Vision API Message Schema

from openai import OpenAI

client = OpenAI()
b64_image = encode_image_to_base64("invoice.png")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract invoice number, date, and total amount as JSON."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{b64_image}",
                        "detail": "high" # "low", "high", or "auto"
                    }
                }
            ]
        }
    ],
    max_tokens=500
)
print(response.choices[0].message.content)

3. Anthropic Vision & PDF Document Message Schema

import anthropic

client = anthropic.Anthropic()
b64_pdf = encode_image_to_base64("quarterly_report.pdf")

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document",
                    "source": {
                        "type": "base64",
                        "media_type": "application/pdf",
                        "data": b64_pdf
                    }
                },
                {"type": "text", "text": "Summarize key financial risks mentioned in this PDF."}
            ]
        }
    ]
)
print(response.content[0].text)

4. Vision Token Economics: Low-Detail vs. High-Detail Mode

Diagram: low-detail image processing compared with high-detail processing

Vision patch tokenization and resolution detail cost grid diagram

How do vision APIs calculate prompt token costs?

  1. Low-Detail Mode (detail: "low"):
    • The image is downsampled to a fixed 512 * 512 square.
    • Cost is flat: exactly 85 tokens, regardless of the original image dimensions.
    • Ideal for broad scene tagging and thumbnail indexing.
  2. High-Detail Mode (detail: "high"):
    • The image is scaled to fit within a 2048 * 2048 box.

    • The shortest side is scaled to 768px.

    • The canvas is partitioned into 512 * 512 pixel tiles.

    • Each tile consumes 170 tokens.

    • An additional 85 base tokens are added for the global overview: Total Tokens = (N_{tiles} * 170) + 85

    • Example: A 1024 * 1024 image creates 4 tiles: (4 * 170) + 85 = 765 tokens.


5. Multi-Image Comparative Reasoning & Cross-Modal Attention

You can pass multiple image blocks inside a single content array:


6. Extracting Structured Pydantic Schemas from Complex Image Scans

Diagram: the four steps of extracting structured data from a scanned document

Combining vision inputs with strict JSON schemas transforms messy physical paperwork into strongly typed database records:

from pydantic import BaseModel, Field
from typing import List

class LineItem(BaseModel):
    description: str = Field(description="Item name or service description")
    quantity: int = Field(description="Units purchased")
    unit_price: float = Field(description="Price per unit in USD")
    total_price: float = Field(description="Total line item cost")

class InvoiceExtraction(BaseModel):
    invoice_number: str = Field(description="Unique invoice identifier")
    vendor_name: str = Field(description="Company issuing the invoice")
    invoice_date: str = Field(description="Date formatted as YYYY-MM-DD")
    subtotal: float = Field(description="Pre-tax amount")
    tax_amount: float = Field(description="Calculated tax")
    total_amount: float = Field(description="Final total charge")
    items: List[LineItem] = Field(description="List of purchased line items")

7. Client-Side Image Pre-Processing & Dimension Optimization

Diagram: the four preprocessing steps that reduce image token cost

Sending raw 48-megapixel photos directly to cloud vision APIs causes massive network latency and unnecessary token billing:


8. Multi-Page PDF Document Ingestion & Sliding Window Budgeting

When analyzing enterprise PDF documents (such as 100-page SEC 10-K filings or clinical trials reports):


9. Spatial Grounding & Normalized Bounding Box Coordinates

Modern frontier multimodal models can locate specific UI elements and physical objects using Normalized Bounding Box Coordinates:


10. Automated Redaction & PII Masking in Multimodal Pipelines

Diagram: the four security considerations for processing untrusted images

Processing sensitive documents (e.g. government IDs, tax returns, medical records) requires strict privacy controls:


An everyday analogy

Diagram: a described image compared with a directly examined one

Think of a magnifying glass with adjustable zoom:


Examples in practice

Let us build an Automated Multimodal Document Analyzer in Python:

import base64
from typing import Dict, Any, List

class MultimodalDocumentAnalyzer:
    def __init__(self):
        self.supported_image_types = ["image/jpeg", "image/png", "image/webp"]
        self.supported_doc_types = ["application/pdf"]

    def build_image_payload(self, base64_data: str, media_type: str, prompt: str) -> Dict[str, Any]:
        if media_type not in self.supported_image_types:
            raise ValueError(f"Unsupported image media type: {media_type}")
        return {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": media_type,
                        "data": base64_data
                    }
                },
                {"type": "text", "text": prompt}
            ]
        }

    def calculate_vision_tokens(self, width: int, height: int, detail: str = "high") -> int:
        if detail == "low":
            return 85
        # Calculate 512x512 tiles for high-detail mode
        import math
        tiles_x = math.ceil(min(width, 2048) / 512)
        tiles_y = math.ceil(min(height, 2048) / 512)
        total_tiles = tiles_x * tiles_y
        return (total_tiles * 170) + 85

Implications: security, privacy, performance, scalability, and cost

Diagram: the capability, cost, privacy and accuracy implications of vision input

  1. Client-Side Compression:
    • Resizing high-resolution images prior to Base64 encoding reduces HTTP upload latency by 5x.
  2. Deterministic Token Billing:
    • Pre-computing tile token counts prevents budget overruns before submitting large batches of image scans.

Alternatives: free, open source, and commercial

Vision SolutionTypeOCR PrecisionBest Use Case
Claude 3.5 SonnetCommercial MLLMState-of-the-ArtComplex PDF documents & multi-column tables
GPT-4o VisionCommercial MLLMState-of-the-ArtGeneral visual QA & chart understanding
Llama 3.2 Vision (OSS)Open-WeightsStrongSelf-hosted on-premise vision inference
Tesseract OCR (OSS)Rule/CNNBasicPlain single-column text extraction

Diagram: optical character recognition compared with multimodal understanding

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   VISION PROCESSING COMPARISON                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Feature            β”‚ Low-Detail Mode β”‚ High-Detail Mode β”‚ Native PDF   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Token Cost         β”‚ Fixed 85 Tokens β”‚ Tiled (765+ Tok) β”‚ ~1500 T/Page β”‚
β”‚ OCR Capability     β”‚ Minimal         β”‚ High-Precision   β”‚ High-Precisionβ”‚
β”‚ Processing Latency β”‚ Fast (< 500ms)  β”‚ Moderate (1-2s)  β”‚ Moderate     β”‚
β”‚ Best For           β”‚ Scene Tagging   β”‚ Tables/Diagrams  β”‚ Legal Reportsβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when multimodal input is the right approach

When to Use Vision APIs:

When NOT to use:


The AI connection

Knowledge check

  1. What encoding standard is used to embed binary images into JSON API payloads?
  2. How is token consumption calculated in OpenAI β€œdetail: high” vision mode?
  3. How does Claude 3.5 Sonnet process multi-page PDF documents natively?
  4. When should an application select β€œdetail: low” instead of β€œdetail: high”?
  5. Why should high-resolution photographs be resized client-side before Base64 encoding?

Hands-on exercise

In this lab, you will build and test a Multimodal Document Analysis Engine in Python: formulate Base64-encoded image and PDF message blocks, calculate exact tile token economics for various image resolutions, and verify output assertions against automated test suites.

Expected output

[Multimodal Vision & Document Processing Verification Suite]
Testing Base64 Payload Formulation:
  Image Block Media Type: 'image/png' [VALIDATED]
  Content Array Structure: 2 blocks (image + prompt) [PASS]
Testing Vision Token Cost Calculator:
  Low-Detail Mode: 85 tokens [PASS]
  High-Detail Mode (1024x1024): 765 tokens (4 tiles) [PASS]
  High-Detail Mode (1920x1080): 1445 tokens (8 tiles) [PASS]
Test Suite: 3 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement an automated Receipt Expense Parser in Python that extracts Merchant, Date, Tax, and Total from receipt images into a typed Pydantic dataclass.
  2. Build a PDF Multi-Page Token Estimator that inspects PDF page count and calculates projected API costs before dispatching requests.

Extension challenge

Implement a Smart Visual Redaction Gateway:

Quiz

Q1. What is the standard encoding used to transmit local image files directly inside API JSON payloads?

  1. Zipping the images into an archive
  2. Raw binary hexadecimal strings without headers
  3. Uploading files via FTP
  4. Base64 with a declared media type
Show answer

Answer: D. Base64 with a declared media type

Base64 encoding converts binary media bytes into standard ASCII text compatible with JSON payloads.

Q2. In OpenAI Vision models (e.g. GPT-4o), how does "detail: high" calculate token consumption?

  1. It tiles the image into 512x512 blocks and counts them
  2. It charges a flat 10,000 tokens per image
  3. It charges 1 token per pixel
  4. It is completely free
Show answer

Answer: A. It tiles the image into 512x512 blocks and counts them

High-detail mode slices large images into 512x512 tiles at 170 tokens per tile plus 85 base tokens.

Q3. How does Anthropic Claude 3.5 Sonnet natively process multi-page PDF documents in the Messages API?

  1. By requiring the user to convert PDF text to Word .docx
  2. With a document block of base64 PDF data
  3. By downloading the PDF via BitTorrent
  4. PDFs cannot be processed by Claude
Show answer

Answer: B. With a document block of base64 PDF data

Claude natively accepts base64-encoded PDF document content blocks directly in the messages array.

Q4. When is "detail: low" the recommended setting for image processing?

  1. When reading architectural blueprints
  2. When extracting tiny handwritten numbers from financial checks
  3. When only coarse recognition is needed, not OCR
  4. When reading medicine bottle labels
Show answer

Answer: C. When only coarse recognition is needed, not OCR

Low-detail mode is ultra-fast and consumes only 85 tokens, perfect for high-level scene categorization.

Q5. What is Visual Question Answering (VQA) in multimodal engineering?

  1. Compressing MP4 video files
  2. Playing video games with a gamepad
  3. Taking screenshots of code
  4. An image plus a question, answered as text or JSON
Show answer

Answer: D. An image plus a question, answered as text or JSON

VQA combines visual feature embeddings with linguistic reasoning to answer questions about imagery.

Glossary

Multimodal LLM (MLLM)
A foundation model capable of simultaneously processing and reasoning across multiple data modalities including text, images, audio, and documents.
Base64 Encoding
A binary-to-text encoding scheme translating raw media bytes into standard ASCII string representations for JSON transmission.
Vision Token
A vector embedding representation of an image patch processed by the vision transformer encoder.
Image Patch Tiling
The process of partitioning a high-resolution image into discrete sub-grid tiles (e.g. 512x512 pixels) for detailed inspection.
High-Detail Mode
An API vision setting that decomposes images into high-resolution patch grids to enable fine-grained OCR and diagram reading.
Low-Detail Mode
An API vision setting that resizes images to a low fixed resolution (85 tokens) for rapid broad scene classification.
Visual Document Extraction
The automated parsing of unstructured visual documents (PDFs, invoices, receipts) into validated structured JSON schemas.
MIME Type
A standardized label (e.g. image/png, application/pdf) identifying the media format of an encoded payload.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.