LLMs and Generative AI β€Ί LLM APIs β€Ί Day 259

Day 259: Building a CLI Assistant

Day 259 of 365 β€” Building a CLI Assistant

Synthesize the full LLM API engineering stack into an end-to-end interactive Command-Line Assistant: build an asynchronous REPL (Read-Eval-Print Loop) with prompt_toolkit and Rich terminal styling, integrate real-time token streaming with live markdown rendering, implement tool calling with local system execution sandboxes, maintain multi-turn conversation memory with automated context sliding windows and prompt caching, enforce token bucket rate limiting and expenditure tracking, and build a modular terminal AI assistant in Python.

Course
LLMs and Generative AI
Category
LLM APIs
Reading time
β‰ˆ 35 min
Practical time
β‰ˆ 50 min
Lesson duration
1h 25m
Last verified
2026-08-30

Hands-on lab for this lesson

Lab files on GitHub: https://github.com/ai-roadmap-365/ai-roadmap-365.github.io/tree/main/labs/sections/llms-and-generative-ai/day-259-building-a-cli-assistant

  1. Get the hands-on files. Clone the labs repository once (you can reuse this clone for every lesson). This works on macOS, Linux, and Windows (PowerShell or WSL):
    git clone https://github.com/ai-roadmap-365/ai-roadmap-365.github.io.git
    cd ai-roadmap-365.github.io
  2. Open this lesson's lab. Move into the directory for this specific day. Every lab lives at the same predictable path β€” section / subsection / week / day:
    cd labs/sections/llms-and-generative-ai/day-259-building-a-cli-assistant
  3. Read the lab guide. Open `README.md` in that directory. It lists the exact commands, what each does, the expected output, and how to check your work β€” read it before running anything.
  4. Run it and check your work. Follow the README's "How to run" section: run the example first to see the finished result, then complete the numbered exercises in `starter/`, then run the tests. The tests pass (exit 0) only when your work is correct.
    bash tests/run_tests.sh   # or the test command named in the lab README

You can also open the lab as a local page (works offline, shows the file tree and expected output).

Learning objectives

By the end of this lesson you will be able to:

Prerequisites

Why this matters

Over the past six days, you have mastered the individual components of the modern LLM API ecosystem:

  1. Day 253: First calls to Claude API, system prompts, temperature, and JSON mode.
  2. Day 254: The OpenAI-compatible gateway standard, multi-provider routing, and local vLLM/Ollama execution.
  3. Day 255: Server-Sent Events (SSE), async token generators, and sub-500ms TTFT streaming.
  4. Day 256: JSON Schema tool use, parallel dispatch, and recursive multi-turn execution loops.
  5. Day 257: Multimodal Base64 payloads, vision patch tokenization, and PDF document extraction.
  6. Day 258: Prompt caching KV breakpoints, Token Bucket rate limiting, and full jitter retries.

Now, it is time to bring all these foundational pieces together into a unified, production-grade application: an Interactive Command-Line AI Assistant.

Command-line assistants (such as Claude Code, GitHub Copilot CLI, and Aider) are among the most productive developer tools in the AI era. Building your own terminal assistant gives you full control over tool integration, privacy boundaries, token cost telemetry, and model routing.

Today, you will build a complete, modular CLI Assistant in Python featuring an interactive REPL, real-time Markdown streaming, local file/calculator tools, sliding window memory, and live token cost tracking.


The idea in plain language

Diagram: the four components of a working AI assistant

Think of building a Formula 1 racing car:


Historical background

CLI AI assistant terminal REPL architecture diagram

  1. 1970s (Classic Unix REPLs): The Read-Eval-Print Loop became the gold standard for interactive developer tooling (sh, Python interactive shell, Lisp).
  2. 2023 (First Gen AI Terminals): Early CLI tools simply piped user strings into curl commands, printing raw unformatted JSON responses without streaming or memory.
  3. 2024 (Rich UI & Modern Agentic CLIs): Tools like Aider and Claude Code revolutionized developer workflows by embedding full syntax highlighting, in-place markdown repainting, tool execution sandboxes, and prompt caching into clean terminal interfaces.
  4. 2025 (Local-First Hybrid Assistants): Modern CLIs seamlessly switch between local private models (via Ollama/vLLM) and cloud reasoning frontier models depending on query complexity and cost budgets.

What it is β€” and what it is not

Diagram: a supervised assistant compared with an autonomous agent

What a Production CLI Assistant IS:

What it is NOT:


Why it was created and what problems it solves

A unified CLI assistant solves 4 primary developer workflow problems:

  1. Context Switching Elimination: Programmers get immediate AI assistance directly in their native terminal without switching to browser tabs.
  2. Local Project Awareness: The assistant can read local codebase files, inspect git diffs, and run test suites directly via local tool calls.
  3. Zero Token Waste: Automated sliding window memory and prompt caching prevent token blowups during long coding sessions.
  4. Cost & Rate Limit Transparency: Live session cost ledgers keep developers fully aware of their exact spending on every query.

How it works

Let us examine the 5 core subsystems of a production CLI assistant.


1. The Interactive Asynchronous REPL Loop

Diagram: the five stages of an interactive assistant loop

from prompt_toolkit import PromptSession
from prompt_toolkit.history import InMemoryHistory
from rich.console import Console

console = Console()
session = PromptSession(history=InMemoryHistory())

async def run_cli_loop(assistant):
    console.print("[bold green]AI Assistant CLI Initialized. Type /help for commands, /exit to quit.[/bold green]")
    while True:
        try:
            user_input = await session.prompt_async("\n❯ ")
            if not user_input.strip():
                continue
            if user_input.startswith("/"):
                assistant.handle_slash_command(user_input)
                continue
            await assistant.process_query(user_input)
        except (KeyboardInterrupt, EOFError):
            console.print("\n[yellow]Session terminated by user.[/yellow]")
            break

2. Context Sliding Window & Memory Management

Diagram: how conversation history is bounded as an assistant session grows

Context sliding window and memory truncation architecture diagram

As conversations progress, context arrays grow indefinitely. The Sliding Window Memory Manager:

  1. Pins the System Prompt: The system instructions and tool definitions are permanently pinned at index 0 with prompt caching breakpoints.
  2. Slides Recent Turns: When history exceeds max_turns = 10, the manager prunes the oldest non-system user/assistant message pairs.
  3. Enforces Token Caps: If total estimated tokens exceed the budget (e.g. 8,000 tokens), it summarizes older conversation turns into a compact 200-word memory digest.

3. Real-Time Streaming with Rich Live Markdown

from rich.live import Live
from rich.markdown import Markdown

async def stream_response_to_terminal(stream_generator):
    accumulated_text = ""
    with Live(Markdown(""), refresh_per_second=15, console=console) as live:
        async for token in stream_generator:
            accumulated_text += token
            live.update(Markdown(accumulated_text))
    return accumulated_text

4. Sandboxed Local Tool Execution Suite

Register safe developer tools with strict path validation:


5. Slash Commands & Telemetry Dashboards

Provide built-in terminal shortcuts that intercept user commands before sending network queries:


6. Asynchronous Non-Blocking Architecture with asyncio

A command-line assistant must remain responsive even during long network calls:


7. Graceful Signal Handling: Aborting Streams without Crashing

Diagram: how an assistant cancels a running generation without losing state

Nothing frustrates a developer more than a CLI crashing when they press Ctrl+C:


Diagram: the four controls that contain a locally executing tool

Granting an AI assistant access to the local filesystem requires defensive security engineering:


9. Dynamic Model Routing & Local-Cloud Hybrid Failover

Diagram: how an assistant routes between a local model and a hosted one

A production CLI assistant should adapt intelligently to network connectivity and budget constraints:


10. Observability, Session Export, & Telemetry Dashboards

To facilitate continuous developer productivity analysis:


An everyday analogy

Diagram: individual capabilities compared with an assembled working system

Think of a personal flight navigation assistant for a pilot:


Examples in practice

Let us build a Complete Modular CLI Assistant Core in Python:

import time
from typing import Dict, Any, List, Optional
from collections import deque

class CLIAssistantCore:
    def __init__(self, system_prompt: str, max_history_turns: int = 6):
        self.system_prompt = system_prompt
        self.max_history_turns = max_history_turns
        self.history: deque = deque(maxlen=max_history_turns * 2)
        self.tools: Dict[str, Any] = {}
        self.session_cost_usd = 0.0
        self.total_tokens_consumed = 0

    def register_tool(self, name: str, func: Any):
        self.tools[name] = func

    def build_message_context(self, new_user_message: str) -> List[Dict[str, Any]]:
        # Constructs the prompt array with pinned system prompt and sliding history
        messages = [{"role": "system", "content": self.system_prompt}]
        for turn in self.history:
            messages.append(turn)
        messages.append({"role": "user", "content": new_user_message})
        return messages

    def record_turn(self, user_text: str, assistant_text: str, tokens: int = 150, cost: float = 0.0005):
        self.history.append({"role": "user", "content": user_text})
        self.history.append({"role": "assistant", "content": assistant_text})
        self.total_tokens_consumed += tokens
        self.session_cost_usd += cost

    def handle_slash_command(self, cmd: str) -> str:
        cmd = cmd.strip().lower()
        if cmd == "/help":
            return "Available commands: /help, /cost, /clear, /tools, /exit"
        elif cmd == "/cost":
            return f"Session Tokens: {self.total_tokens_consumed} | Total Cost: ${self.session_cost_usd:.4f}"
        elif cmd == "/clear":
            self.history.clear()
            return "Conversation history cleared. System prompt preserved."
        elif cmd == "/tools":
            return f"Registered Tools: {list(self.tools.keys())}"
        return f"Unknown command: {cmd}"

Implications: security, privacy, performance, scalability, and cost

Diagram: the access, cost, privacy and reliability implications of a local AI assistant

  1. Path Sandboxing:
    • Always validate that file paths requested by tools remain strictly within the current working directory.
  2. Context Window Cost Bounding:
    • Setting max_history_turns = 6 guarantees that long 2-hour coding sessions never degrade into massive 50,000-token prompt billing spikes.

Alternatives: free, open source, and commercial

CLI AssistantTypeTool IntegrationModel Support
Custom Python CLIDIY Open Source100% CustomizableMulti-Provider
AiderOpen SourceGit & File RefactoringClaude / OpenAI / Local
Claude CodeCommercial CLIAgentic Shell & SearchClaude 3.7 Sonnet
GitHub Copilot CLICommercialTerminal Shell helperOpenAI Models

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   CLI ASSISTANT ARCHITECTURAL MODES                    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Feature            β”‚ Basic Script    β”‚ Streaming REPL   β”‚ Agentic CLI  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Input Mode         β”‚ Single prompt   β”‚ Interactive loop β”‚ Multi-turn   β”‚
β”‚ Output Display     β”‚ Blocking print  β”‚ Rich Live stream β”‚ Markdown + UIβ”‚
β”‚ Tool Capability    β”‚ None            β”‚ Read-only math   β”‚ Shell + Code β”‚
β”‚ Memory Management  β”‚ Stateless       β”‚ Sliding Window   β”‚ Summarizationβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

When to use it β€” and when not to

Diagram: when a command-line assistant is the right form

When to Use a CLI Assistant:

When NOT to use:


The AI connection

Knowledge check

  1. What is the primary role of an interactive REPL loop in a terminal assistant?
  2. How does a sliding window memory manager prevent prompt context blowups?
  3. Why does Rich Live rendering provide a superior user experience during token streaming?
  4. What security guardrail must be applied to local file reading tools?
  5. What metrics should be recorded in a session expenditure ledger?

Hands-on exercise

In this lab, you will build and test a Modular CLI Assistant Core in Python: implement the prompt context builder with sliding history, register local calculation and lookup tools, parse slash commands (/help, /cost, /clear, /tools), track cumulative session token and financial expenditures, and verify output assertions against automated test suites.

Expected output

[Building a CLI Assistant Verification Suite]
Testing Context Builder & Sliding Memory:
  Initial Messages: 3 blocks (System + 1 Turn) [PASS]
  Sliding Window Eviction: Oldest turn pruned when maxlen exceeded [PASS]
Testing Slash Command Handler:
  /help -> Returned command list [PASS]
  /cost -> Formatted tokens and USD cost string [PASS]
  /clear -> Cleared history buffer [PASS]
Testing Tool Registration & Execution:
  Tool 'calc' registered and executed successfully [PASS]
Test Suite: 4 passed in 0.38s

Validate your work

Run the automated test runner:

./tests/run_tests.sh

Troubleshooting

Common mistakes


Practice assignment

  1. Implement a Git Diff Inspection Tool in Python that extracts git diff --staged and generates structured conventional commit messages.
  2. Build an automated Session Export Command (/export) that writes the complete formatted conversation transcript to a local Markdown file.

Extension challenge

Implement a Self-Correcting Test Runner Agent in the CLI:

Quiz

Q1. What is the primary role of an asynchronous REPL (Read-Eval-Print Loop) in a CLI assistant?

  1. To compile the C++ code into raw machine language
  2. To reboot the computer on every prompt
  3. To mine cryptocurrency
  4. To read input, hold state, and stream the reply
Show answer

Answer: D. To read input, hold state, and stream the reply

The REPL loop orchestrates user input, memory buffering, API dispatch, and live terminal rendering.

Q2. Why is a sliding window algorithm used for multi-turn conversation history management?

  1. To keep the prompt and recent turns in budget
  2. To delete user files after 10 minutes
  3. To invert text upside down
  4. To increase network bandwidth
Show answer

Answer: A. To keep the prompt and recent turns in budget

Sliding windows keep context within safe token and cost bounds while maintaining recent conversational state.

Q3. How does Rich Live rendering improve the terminal user experience during token streaming?

  1. It changes terminal fonts to Comic Sans
  2. It repaints markdown in place as chunks arrive
  3. It converts the terminal to an audio podcast
  4. It hides all error messages
Show answer

Answer: B. It repaints markdown in place as chunks arrive

Rich Live updates in-place, rendering formatted markdown smoothly as tokens arrive.

Q4. Why is it critical to sandbox local file and shell tools executed by a CLI assistant?

  1. To disable the internet connection
  2. Because sandboxing makes Python run a hundred times faster
  3. To stop the model running destructive shell commands
  4. Because sandboxing is required by Apple
Show answer

Answer: C. To stop the model running destructive shell commands

Sandboxing and path validation ensure the CLI assistant only accesses permitted project directories.

Q5. What telemetry data should a production CLI assistant display upon session exit?

  1. Nothing at all
  2. The user's computer serial number and model
  3. A random number
  4. Turns, tokens, cost, and time to first token
Show answer

Answer: D. Turns, tokens, cost, and time to first token

Displaying comprehensive token and financial telemetry gives developers complete visibility into API resource consumption.

Glossary

CLI Assistant
A terminal-based conversational AI agent providing interactive code analysis, system automation, and Q&A.
REPL
Read-Eval-Print Loop; an interactive command-line environment that takes user inputs, executes logic, and returns results.
Sliding Context Window
A memory eviction strategy that retains system instructions and the most recent N turns while pruning older exchanges.
Rich Live Display
A terminal UI component that updates text and Markdown in-place dynamically as streaming tokens arrive.
Tool Sandbox
A secure execution boundary restricting tool file access to designated project working directories.
Session Ledger
An in-memory telemetry tracker recording cumulative token expenditures and financial costs for a CLI session.
Slash Command
A special terminal instruction (e.g. /help, /cost, /clear) intercepted by the REPL before dispatching to the LLM.
Prompt History Buffer
A persistent storage mechanism recording previous terminal inputs for arrow-key recall and replay.

Sources and further reading


Kept in this browser, no account needed. Your progress page turns the whole record into one link you can bookmark or open on another device.