LLMs and Generative AI › Prompt Engineering › Day 248

Hands-on lab — Day 248: Few-Shot Examples and Chain of Thought

Commands

Setup

pip install -r requirements/requirements.txt

Run

python3 examples/few_shot_examples_and_chain_of_lib.py

Test

./tests/run_tests.sh

File tree

conftest.py
examples/few_shot_examples_and_chain_of_lib.py
examples/test_few_shot_examples_and_chain_of_lib.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/starter-run.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/few_shot_examples_and_chain_of_lib.py
starter/test_few_shot_examples_and_chain_of_lib.py
tests/run_tests.sh
tests/test_few_shot_examples_and_chain_of_lib.py
troubleshooting.md

Lab README

Lab: Day 248 -- Few-Shot Examples and Chain of Thought

Lesson

Day number: 248 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Few-Shot In-Context Learning, Chain of Thought, and Self-Consistency.

Purpose

Build and test a modular Few-Shot Chain-of-Thought compilation engine and Self-Consistency voting aggregator in Python. Construct exemplar pools with XML tags, compile multi-turn CoT scratchpads, and evaluate majority consensus across stochastic rollout distributions.

Learning objectives

  • Formulate XML-delimited Few-Shot exemplar pools.
  • Construct Chain-of-Thought scratchpads to expand autoregressive compute.
  • Implement Self-Consistency majority voting over stochastic sample paths.
  • Prevent exemplar biases (majority label bias, recency bias).

Prerequisites

  • Day 247 (System Prompts and Role Design).
  • Python 3.11+ with Pytest.

Supported operating systems

  • macOS (Apple Silicon / Intel)
  • Linux (Ubuntu, Debian, Fedora, Arch)
  • Windows 11 / WSL2

Hardware requirements

  • 1+ CPU cores.
  • 1 GB RAM.
  • 100 MB disk space.

Required software

  • Python 3.11 or newer.
  • pip package manager.
  • virtualenv or venv module.

Free and open-source options

Python and standard math libraries are open source.

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/few_shot_examples_and_chain_of_lib.py: Student scaffold file.
  • examples/few_shot_examples_and_chain_of_lib.py: Complete reference implementation.
  • tests/test_few_shot_examples_and_chain_of_lib.py: Pytest automated validation suite.
  • expected-output/: Verified output logs and baseline values.

How to run

Execute the reference demonstration script:

python3 examples/few_shot_examples_and_chain_of_lib.py

What the commands do

  • Compiles Few-Shot CoT XML prompts.
  • Aggregates stochastic sampled rollouts.
  • Runs unit test assertions.

Expected output

CoT Demo Compiled Successfully. Consensus: 224

Validation steps

  1. Verify compile_prompt generates <examples> and <scratchpad> tags.
  2. Confirm aggregate_self_consistency computes consensus mode and confidence.
  3. Validate empty list handling raises ValueError.
  4. Ensure all unit test assertions pass.

Tests

Run the test runner script:

./tests/run_tests.sh

Cleanup

find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +

Troubleshooting

  • Empty sample list: Pass at least 1 sample to aggregate_self_consistency.

Security notes

All prompt generation executes locally in memory.

Extension exercises

  1. Implement a dynamic cosine-similarity exemplar retriever.
  2. Build a Tree-of-Thoughts breadth-first search navigator.

Expected output

FIELDS.md

# Expected Output Fields: Day 248

- `CoT Demo`: Status of Few-Shot CoT compilation and Self-Consistency aggregation.

examples-run.txt

CoT Demo Compiled Successfully. Consensus: 224

measured-values.txt

CoT Demo: Success
Consensus Winner: 224
Confidence: 80%

starter-run.txt

Starter scaffold executed. Ready for student implementation.

test-run.txt

============================= test session starts ==============================
collected 3 items

tests/test_few_shot_examples_and_chain_of_lib.py::test_few_shot_cot_compilation PASSED [ 33%]
tests/test_few_shot_examples_and_chain_of_lib.py::test_self_consistency_majority_voting PASSED [ 66%]
tests/test_few_shot_examples_and_chain_of_lib.py::test_self_consistency_empty_validation PASSED [100%]

============================== 3 passed in 0.38s ===============================

Source files

conftest.py (585 bytes)
"""Make this lab importable when the suite runs.

pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""

import os
import sys

LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
    if os.path.isdir(candidate) and candidate not in sys.path:
        sys.path.insert(0, candidate)
examples/few_shot_examples_and_chain_of_lib.py (2383 bytes)
from typing import List, Dict, Any
from collections import Counter

class FewShotCoTEngine:
    def __init__(self, task_description: str):
        self.task_description = task_description.strip()
        self.exemplars: List[Dict[str, str]] = []

    def add_exemplar(self, question: str, reasoning: str, answer: str) -> "FewShotCoTEngine":
        self.exemplars.append({
            "question": question.strip(),
            "reasoning": reasoning.strip(),
            "answer": answer.strip()
        })
        return self

    def compile_prompt(self, query: str) -> str:
        parts = [f"<instructions>\n{self.task_description}\nThink step by step before providing the final answer.\n</instructions>"]

        if self.exemplars:
            parts.append("<examples>")
            for i, ex in enumerate(self.exemplars):
                parts.append(
                    f'<example id="{i+1}">\n'
                    f'<input>{ex["question"]}</input>\n'
                    f'<reasoning>{ex["reasoning"]}</reasoning>\n'
                    f'<answer>{ex["answer"]}</answer>\n'
                    f'</example>'
                )
            parts.append("</examples>")

        parts.append(f"<query>\n{query.strip()}\n</query>\n\n<scratchpad>\nLet us reason step by step:")
        return "\n\n".join(parts)

    @staticmethod
    def aggregate_self_consistency(sampled_answers: List[str]) -> Dict[str, Any]:
        if not sampled_answers:
            raise ValueError("Sampled answers list cannot be empty.")

        counts = Counter(sampled_answers)
        majority_answer, vote_count = counts.most_common(1)[0]
        confidence = vote_count / len(sampled_answers)

        return {
            "consensus_answer": majority_answer,
            "confidence": confidence,
            "votes": dict(counts),
            "total_samples": len(sampled_answers)
        }

def run_cot_demo():
    engine = FewShotCoTEngine("Solve multi-step math problems")
    engine.add_exemplar("What is 12 * 15?", "12 * 10 = 120, 12 * 5 = 60, 120 + 60 = 180", "180")
    prompt = engine.compile_prompt("What is 14 * 16?")

    samples = ["224", "224", "214", "224", "224"]
    result = engine.aggregate_self_consistency(samples)

    print("CoT Demo Compiled Successfully. Consensus:", result["consensus_answer"])
    return prompt, result

if __name__ == "__main__":
    run_cot_demo()
examples/test_few_shot_examples_and_chain_of_lib.py (991 bytes)
import pytest
from examples.few_shot_examples_and_chain_of_lib import FewShotCoTEngine

def test_few_shot_cot_compilation():
    engine = FewShotCoTEngine("Math Solver")
    engine.add_exemplar("1+1", "1 plus 1 is 2", "2")
    prompt = engine.compile_prompt("2+2")

    assert "<instructions>" in prompt
    assert "<examples>" in prompt
    assert '<example id="1">' in prompt
    assert "<reasoning>1 plus 1 is 2</reasoning>" in prompt
    assert "<query>\n2+2\n</query>" in prompt
    assert "<scratchpad>" in prompt

def test_self_consistency_majority_voting():
    samples = ["A", "B", "A", "A", "C"]
    result = FewShotCoTEngine.aggregate_self_consistency(samples)

    assert result["consensus_answer"] == "A"
    assert result["confidence"] == 0.6
    assert result["votes"] == {"A": 3, "B": 1, "C": 1}
    assert result["total_samples"] == 5

def test_self_consistency_empty_validation():
    with pytest.raises(ValueError):
        FewShotCoTEngine.aggregate_self_consistency([])
metadata.yml (443 bytes)
lesson_id: D248
day: 248
kind: lab
languages:
  - python
setup_commands:
  - 'pip install -r requirements/requirements.txt'
run_commands:
  - 'python3 examples/few_shot_examples_and_chain_of_lib.py'
test_commands:
  - './tests/run_tests.sh'
cleanup_commands:
  - 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/few_shot_examples_and_chain_of_lib.py (631 bytes)
from typing import List, Dict, Any
from collections import Counter

class FewShotCoTEngine:
    def __init__(self, task_description: str):
        self.task_description = task_description
        self.exemplars: List[Dict[str, str]] = []

    def add_exemplar(self, question: str, reasoning: str, answer: str) -> "FewShotCoTEngine":
        # TODO: Add exemplar
        pass

    def compile_prompt(self, query: str) -> str:
        # TODO: Compile XML CoT prompt
        pass

    @staticmethod
    def aggregate_self_consistency(sampled_answers: List[str]) -> Dict[str, Any]:
        # TODO: Perform majority voting
        pass
starter/test_few_shot_examples_and_chain_of_lib.py (991 bytes)
import pytest
from examples.few_shot_examples_and_chain_of_lib import FewShotCoTEngine

def test_few_shot_cot_compilation():
    engine = FewShotCoTEngine("Math Solver")
    engine.add_exemplar("1+1", "1 plus 1 is 2", "2")
    prompt = engine.compile_prompt("2+2")

    assert "<instructions>" in prompt
    assert "<examples>" in prompt
    assert '<example id="1">' in prompt
    assert "<reasoning>1 plus 1 is 2</reasoning>" in prompt
    assert "<query>\n2+2\n</query>" in prompt
    assert "<scratchpad>" in prompt

def test_self_consistency_majority_voting():
    samples = ["A", "B", "A", "A", "C"]
    result = FewShotCoTEngine.aggregate_self_consistency(samples)

    assert result["consensus_answer"] == "A"
    assert result["confidence"] == 0.6
    assert result["votes"] == {"A": 3, "B": 1, "C": 1}
    assert result["total_samples"] == 5

def test_self_consistency_empty_validation():
    with pytest.raises(ValueError):
        FewShotCoTEngine.aggregate_self_consistency([])
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 248 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_few_shot_examples_and_chain_of_lib.py (991 bytes)
import pytest
from examples.few_shot_examples_and_chain_of_lib import FewShotCoTEngine

def test_few_shot_cot_compilation():
    engine = FewShotCoTEngine("Math Solver")
    engine.add_exemplar("1+1", "1 plus 1 is 2", "2")
    prompt = engine.compile_prompt("2+2")

    assert "<instructions>" in prompt
    assert "<examples>" in prompt
    assert '<example id="1">' in prompt
    assert "<reasoning>1 plus 1 is 2</reasoning>" in prompt
    assert "<query>\n2+2\n</query>" in prompt
    assert "<scratchpad>" in prompt

def test_self_consistency_majority_voting():
    samples = ["A", "B", "A", "A", "C"]
    result = FewShotCoTEngine.aggregate_self_consistency(samples)

    assert result["consensus_answer"] == "A"
    assert result["confidence"] == 0.6
    assert result["votes"] == {"A": 3, "B": 1, "C": 1}
    assert result["total_samples"] == 5

def test_self_consistency_empty_validation():
    with pytest.raises(ValueError):
        FewShotCoTEngine.aggregate_self_consistency([])

Troubleshooting

Troubleshooting: Day 248 - Few-Shot Examples and Chain of Thought

Common Issues

  1. Low reasoning diversity:
    • Cause: Sampling with Temperature = 0 in Self-Consistency.
    • Fix: Use Temperature = 0.7 when sampling multiple CoT paths.

Security notes

Security & Privacy: Day 248 - Few-Shot Examples and Chain of Thought

Security Guidance

  • Encapsulate few-shot examples inside tags to prevent prompt injection.