LLMs and Generative AI › Customizing and Running Models › Day 277

Day 277: Running Local Models with Ollama

Day 277 of 365 — Running Local Models with Ollama

Master local LLM serving with Ollama: understand the Go orchestrator and llama.cpp execution backend, write declarative Modelfiles with custom parameters and system prompts, optimize Unified Memory allocations on Apple Silicon, and interact with the REST API for streaming generation.

Course
LLMs and Generative AI
Category
Customizing and Running Models
Reading time
≈ 35 min
Practical time
≈ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-277-running-local-models-with-ollama

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path — section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-277-running-local-models-with-ollama
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work — read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

For the first several years of the modern generative AI revolution, accessing state-of-the-art language models required sending proprietary company data across the public internet to third-party closed APIs (such as OpenAI or Anthropic). For organizations handling confidential customer health records (HIPAA), privileged legal documents (attorney-client privilege), sensitive financial transactions, or classified government infrastructure, relying on cloud APIs is legally or architecturally impossible.

Furthermore, cloud API pricing models introduce variable per-token operational costs that scale linearly with user volume. When building high-frequency background agent loops or enterprise code auto-complete tools processing millions of requests daily, cloud API bills rapidly escalate into hundreds of thousands of dollars.

Local model execution completely transforms this paradigm. Thanks to quantization breakthroughs and modern high-bandwidth hardware (such as Apple Silicon Unified Memory and consumer NVIDIA GPUs), software engineers can now run 8-billion to 70-billion parameter models directly on local developer laptops and private on-premises servers.

Ollama emerged as the de facto industry standard runtime for local LLMs. By wrapping the raw C++ tensor engine of llama.cpp in an intuitive Docker-like CLI, declarative Modelfile specification, and standardized HTTP REST API, Ollama allows developers to pull, customize, and serve state-of-the-art open-weight models (Llama 3, Mistral, Qwen, DeepSeek) with a single command.

Ollama Runtime Architecture Diagram showing REST API server, Modelfile parser, and llama.cpp execution engine

The idea in plain language

Diagram: the four steps to running a language model on local hardware

Think of running language models like playing video games:

With Ollama running locally, your AI applications execute with zero internet connectivity, zero monthly API costs, zero data exfiltration risk, and single-digit millisecond network latencies.

Historical background

Diagram: the historical progression of running language models locally

The journey toward local LLM execution was catalyzed by two monumental open-source milestones:

In March 2023, Meta’s original LLaMA weights leaked to the public. Within hours, Bulgarian open-source developer Georgi Gerganov created llama.cpp—a pure C/C++ implementation of transformer autoregressive inference with zero external dependencies, designed specifically to run LLaMA models on Apple Silicon MacBooks via Apple’s Metal Performance Shaders. Gerganov’s breakthrough proved that 4-bit quantized models could achieve 30+ tokens per second on consumer laptops without specialized NVIDIA server cards.

However, using early llama.cpp required complex CLI commands, manual GGML/GGUF compilation, custom prompt template formatting, and raw memory flag management.

In mid-2023, Jeffrey Morgan and Michael Chi founded Ollama. Inspired by Docker’s containerization ergonomics, they built an orchestration daemon in Go that packaged GGUF weights, system prompts, tokenizer templates, and sampling parameters into unified, portable “Model Images”. Ollama introduced the Modelfile format, created a public model registry (ollama.com/library), and exposed standard HTTP endpoints compatible with modern web frameworks.

Today, Ollama is downloaded millions of times monthly, serving as the foundational local inference backend for developer tools (Cursor, Continue.dev, Claude Code local fallbacks, LangChain, and private enterprise agent pipelines).

What it is — and what it is not

Diagram: local model serving compared with a hosted model service

Let us establish clear operational boundaries for Ollama:

What Ollama Is:

What Ollama Is Not:

+-------------------------------------------------------------------------------+
|                         LOCAL RUNTIME STACK COMPARISON                        |
+----------------------+--------------------+-----------------------------------+
| Feature              | Ollama             | Raw llama.cpp     | vLLM          |
+----------------------+--------------------+-----------------------------------+
| Primary Language     | Go + C++           | Pure C/C++        | Python + C++  |
| Primary Target       | Workstations & Edge| Edge / Embedded   | Server GPU Pod|
| Setup Complexity     | 1-Click Install    | Build from Source | Docker/CUDA   |
| Modelfile Packaging  | Native             | Manual CLI flags  | Config args   |
| Continuous Batching  | Basic              | Basic             | State-of-the-Art|
| Apple Silicon Metal  | Native Auto-Detect | Native Auto-Detect| CPU only (No Metal)|
+----------------------+--------------------+-------------------+---------------+

Why it was created and what problems it solves

Ollama eliminates the major friction points of self-hosted local AI:

1. The Prompt Template Mismatch Nightmare

Every open-weight model family uses a different prompt formatting template:

Diagram: how a mismatched prompt template silently degrades local model output

Passing the wrong template degrades model reasoning by 40% and causes infinite generation loops. Ollama bundles the exact Go text template inside the model manifest, automatically formatting multi-turn chat messages into the model’s native delimiter syntax.

2. Automatic Hardware Acceleration & Memory Layer Offloading

Running raw models requires calculating how many transformer layers can fit in GPU VRAM versus CPU RAM (-ngl flag in llama.cpp). Ollama automatically queries available VRAM at startup, splits layers dynamically between GPU and CPU, and enables Apple Metal or NVIDIA CUDA kernels with zero user configuration.

Diagram: unified memory architecture compared with discrete accelerator memory

3. Unified REST API for Application Integration

Instead of writing custom Python subprocess wrappers to pipe text into CLI binaries, Ollama provides a standard HTTP REST API, allowing Node.js, Python, Go, and Rust applications to interact with local models seamlessly.

How it works

Let us inspect the internal architectural layers of an Ollama inference session:

[HTTP Client / Application]
            │  POST http://127.0.0.1:11434/api/chat
            ▼
[Ollama Daemon (Go HTTP Server)]
            │  1. Parse JSON payload & resolve Model Manifest
            │  2. Apply Modelfile System Prompt & Template
            ▼
[llama.cpp Engine (C++ Backend)]
            │  3. Allocate KV Cache & Load GGUF Tensors
            │  4. Execute Metal / CUDA Kernels (Prefill + Autoregressive Decode)
            ▼
[Streaming HTTP Response] ──> Emits chunked NDJSON event stream to Client

1. The Modelfile Specification

A Modelfile is a declarative configuration file that defines how a model is compiled and served:

Diagram: the four components of a local model package definition

# 1. Base Model Checkpoint (Local GGUF or Registry Tag)
FROM ./custom_llama3_lora_q4.gguf

# 2. Hyperparameter Configuration
PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 4096

# 3. System Prompt Customization
SYSTEM """
You are an expert Linux Systems Administrator.
Provide concise, secure bash commands. Wrap all code in markdown blocks.
"""

# 4. Prompt Template Delimiters (ChatML)
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""

Building the custom model is a single command:

ollama create my-sysadmin-model -f Modelfile

2. Hardware Memory Dynamics (Apple Silicon UMA vs Discrete GPUs)

On modern Apple Silicon (M1/M2/M3/M4 Max and Ultra), memory is unified:

On discrete NVIDIA GPUs (e.g. RTX 4090 with 24GB VRAM):

3. REST API Interaction Lifecycle

Ollama exposes a clean REST API:

Diagram: the four stages of a request to a local model server

POST /api/generate (Single-turn raw prompt)

{
  "model": "llama3:8b",
  "prompt": "Explain PagedAttention in one sentence.",
  "stream": false,
  "options": {
    "temperature": 0.1
  }
}

POST /api/chat (Multi-turn structured messages)

{
  "model": "my-sysadmin-model",
  "messages": [
    {"role": "user", "content": "How do I check open ports on Ubuntu?"}
  ],
  "stream": true
}

When stream: true, the server returns a stream of newline-delimited JSON chunks:

{"model":"my-sysadmin-model","message":{"role":"assistant","content":"Use"},"done":false}
{"model":"my-sysadmin-model","message":{"role":"assistant","content":" the"},"done":false}
{"model":"my-sysadmin-model","message":{"role":"assistant","content":" `ss`"},"done":false}
...
{"model":"my-sysadmin-model","done":true,"total_duration":450123890,"eval_count":32,"eval_duration":380000000}

Ollama Modelfile to Serving Flow illustrating GGUF packaging, parameter setting, and HTTP streaming endpoint

An everyday analogy

Diagram: hosted inference compared with local inference by data movement

Think of Ollama like Docker for Machine Learning:

Just as Docker standardized software distribution across disparate Linux kernels, Ollama standardizes AI model execution across heterogeneous CPU, GPU, and Metal architectures.

Examples in practice

Let us examine how developers integrate Ollama into modern Python applications:

1. Python Client with Streaming Token Consumer

import json
import urllib.request

def stream_ollama_chat(model: str, user_prompt: str):
    url = "http://127.0.0.1:11434/api/chat"
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": user_prompt}],
        "stream": True
    }
    
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"}
    )
    
    with urllib.request.urlopen(req) as response:
        for line in response:
            if line:
                chunk = json.loads(line.decode("utf-8"))
                if "message" in chunk and "content" in chunk["message"]:
                    print(chunk["message"]["content"], end="", flush=True)
                if chunk.get("done", False):
                    print("
[Done. Generation complete.]")

2. Custom Modelfile for Air-Gapped Medical Triage

FROM llama3:8b-instruct-q8_0
PARAMETER temperature 0.0
PARAMETER top_p 0.1
PARAMETER num_ctx 8192

SYSTEM """
You are a clinical decision support assistant.
Strictly categorize triage symptoms into: Level 1 (Emergency), Level 2 (Urgent), Level 3 (Routine).
Always state that output is for clinician review only.
"""

Implications: security, privacy, performance, scalability, and cost

Diagram: the privacy, cost, capability and hardware implications of local model serving

1. Security & Local Sandboxing

Because Ollama binds to 127.0.0.1 by default, model inference is accessible only to local processes on the host. In multi-user server environments, ensure that OLLAMA_HOST is not exposed to the public internet without an authenticating reverse proxy (such as Nginx with mTLS or API token verification).

Diagram: the four security considerations for locally served models

2. Complete Data Privacy

Zero bytes of prompt text, system data, or generated completions ever leave the physical machine. This satisfies strict air-gapped compliance mandates (ITAR, HIPAA, GDPR on-premises requirements).

3. Inference Throughput & Latency

4. Financial Economics

Cost Category                Cloud API (Claude 3.5 Sonnet)   Local Ollama (Llama 3 8B)
--------------------------------------------------------------------------------------
1,000,000 Requests/Month     ~$7,500.00 / month              $0.00 / month (Hardware Owned)
Electricity Cost (250W GPU)  $0.00                           ~$15.00 / month
Network Egress / Ingress     Metered Cloud Bandwidth         Zero Network Egress
Capital Expenditure (CapEx)  $0 (Zero Upfront)               $1,500 - $3,500 (Workstation)

Alternatives: free, open source, and commercial

Tool / EngineEcosystemPrimary FocusBest ForTradeoffs
OllamaGo / llama.cppDeveloper ergonomics, Modelfiles, REST APIsLocal developer workstations, internal toolsBasic concurrent batching
llama.cppPure C/C++Maximum hardware control, zero dependenciesEmbedded devices, minimal CLI wrappersComplex configuration flags
LM StudioDesktop GUI (Proprietary)Visual chat interface, local model discoveryNon-programmers, visual desktop chatClosed-source GUI application
vLLMPython / PagedAttentionHigh-throughput multi-user production servingCloud GPU clusters with high concurrencyHeavy setup, no Apple Metal support
LocalAIGo / ModularDrop-in OpenAI API replacement with audio/visionMulti-modal local drop-in microservicesLarger resource footprint
+-----------------------+-----------------------+-----------------------+-----------------------+
| Attribute             | Ollama                | llama.cpp Server      | vLLM Serving          |
+-----------------------+-----------------------+-----------------------+-----------------------+
| Primary Deployment    | Local Laptop / Server | Embedded / CLI        | Production Cloud Pod  |
| Model Packaging       | Modelfile Registry    | Raw GGUF Files        | HF Safetensors / AWQ  |
| Memory Backend        | Metal / CUDA / CPU    | Metal / CUDA / CPU    | NVIDIA CUDA / ROCm    |
| API Support           | Native + OpenAI v1    | OpenAI Compatible     | OpenAI Compatible     |
| Continuous Batching   | Simple Queue          | Simple Queue          | Advanced PagedAttention|
| Target Concurrency    | 1 - 5 Users           | 1 - 5 Users           | 100 - 1,000+ Users    |
+-----------------------+-----------------------+-----------------------+-----------------------+

When to use it — and when not to

Diagram: when local model serving is appropriate

Choose Ollama when:

Do NOT use Ollama when:

The AI connection

Knowledge check

  1. What is the function of the PARAMETER num_ctx setting in a Modelfile? It defines the allocated context window size (in tokens) for memory allocation and positional attention calculations in llama.cpp.
  2. How does Unified Memory Architecture benefit local LLM execution? It allows the GPU to access the entire system RAM (up to 128GB on Mac M-series) without being restricted by discrete PCIe bus transfer bottlenecks.
  3. What is the format of the streaming response from Ollama’s /api/generate endpoint? Newline-Delimited JSON (NDJSON), where each line contains a JSON object with token content and a done boolean field.

Hands-on exercise

In this lab, you will build an end-to-end Ollama Client, Modelfile Generator, and Local Mock REST Server in Python. Your system will:

  1. Parse and generate valid declarative Modelfile definitions with parameters (temperature, top_p, num_ctx) and system prompts.
  2. Implement a local HTTP REST server simulating Ollama’s /api/generate, /api/chat, and /api/tags endpoints.
  3. Implement an asynchronous/streaming Python client that consumes chunked newline-delimited JSON (NDJSON) tokens and tracks generation metrics (eval count, total duration).

Expected output

============================= test session starts ==============================
collected 6 items

tests/test_ollama_client.py ......                                       [100%]

============================== 6 passed in 0.04s ===============================
[MODELFILE GENERATION] Compiled Modelfile for 'custom-sql-expert':
  FROM ./llama3-8b-q4.gguf
  PARAMETER temperature 0.2
  PARAMETER num_ctx 4096
  SYSTEM "You are an expert SQL engineer."

[LOCAL SERVER RUNTIME] Bound to http://127.0.0.1:11434 (Mock Server)
[STREAMING INFERENCE] Prompt: "Write SQL to find top 5 customers"
  -> Emitted 18 tokens | Total Duration: 120ms | eval_rate: 150.0 tok/s

Validate your work

Run the pytest suite to verify Modelfile parsing, streaming token generation, and REST endpoint contracts:

pytest tests/run_tests.sh

Troubleshooting

  1. Port binding collision: The mock server defaults to an ephemeral port during unit testing to avoid colliding with a live Ollama daemon on 11434.
  2. Streaming chunk parsing error: Ensure the client splits responses on and skips empty lines before calling json.loads().

Common mistakes

Practice assignment

Extend the OllamaClient class to support the OpenAI-compatible /v1/chat/completions endpoint and verify compatibility with existing OpenAI SDK request payloads.

Extension challenge

Implement a Local Model Benchmark Runner in Python that executes a standardized 10-prompt suite against a local Ollama instance, measures Time-To-First-Token (TTFT) and tokens-per-second throughput across various context lengths (512, 1024, 2048, 4096), and exports a Markdown performance report.

Quiz

Q1. What is the internal relationship between Ollama and llama.cpp?

  1. Ollama is a completely separate neural network architecture written in Rust
  2. Ollama is a Go daemon wrapping llama.cpp
  3. llama.cpp is a cloud hosting provider that hosts Ollama models
  4. Ollama translates Python scripts into JavaScript
Show answer

Answer: B. Ollama is a Go daemon wrapping llama.cpp

Ollama provides high-level model packaging, Docker-like Modelfile management, and HTTP REST APIs in Go, delegating low-level hardware tensor operations directly to llama.cpp.

Q2. Which Modelfile command specifies the base GGUF model checkpoint or parent model tag?

  1. IMPORT
  2. BASE
  3. FROM
  4. MODEL
Show answer

Answer: C. FROM

The 'FROM' command defines the base model image or local GGUF path (e.g. 'FROM ./my_model.gguf' or 'FROM llama3:8b').

Q3. Why is Apple Silicon (M-series chips) uniquely effective for running local 8B–70B parameter models with Ollama?

  1. Because Apple Silicon disables floating-point quantization
  2. Because it executes models on the SSD instead of RAM
  3. Because macOS runs models inside cloud containers by default
  4. Unified memory lets CPU and GPU share one pool
Show answer

Answer: D. Unified memory lets CPU and GPU share one pool

Apple Silicon's Unified Memory Architecture provides up to 400–800 GB/s bandwidth shared directly between CPU and Metal GPU, allowing large models (e.g. 70B in Q4) to fit entirely in unified memory without PCIe transfer bottlenecks.

Q4. What is the standard local port used by the Ollama REST API daemon?

  1. 11434
  2. 8080
  3. 5000
  4. 3000
Show answer

Answer: A. 11434

Ollama runs its local HTTP REST server on port 11434 (e.g. http://127.0.0.1:11434).

Q5. How does Ollama stream response tokens over HTTP in its /api/generate endpoint?

  1. Via a binary WebSocket connection
  2. Newline-delimited JSON, one object per token
  3. By returning a single massive JSON blob after 30 seconds
  4. By sending emails to the system administrator
Show answer

Answer: B. Newline-delimited JSON, one object per token

Ollama streams responses by emitting a stream of newline-delimited JSON (NDJSON) chunks, each containing the latest generated token string and a 'done' boolean indicator.

Glossary

Ollama
Open-source local LLM runner and management tool that packages models and serves REST APIs over llama.cpp.
Modelfile
A declarative configuration file defining a model's base weights, system prompts, sampling parameters, and prompt templates.
GGUF
The standard binary file format used by llama.cpp and Ollama for storing quantized model weights and metadata.
Unified Memory Architecture (UMA)
Memory architecture where CPU and GPU share a single high-bandwidth physical memory pool.
Metal
Apple's low-level, low-overhead hardware-accelerated 3D graphic and compute API used on macOS.
Layer Offloading
Assigning a specific number of transformer layers to execute on the GPU while retaining remainder in CPU RAM.
NDJSON (Newline-Delimited JSON)
A streaming format where each JSON payload is serialized on a single line separated by a newline character.
Context Window (num_ctx)
Ollama parameter specifying the maximum token capacity allocated in memory for prompt and generation.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.