LLMs and Generative AI › Prompt Engineering › Day 248
Hands-on lab — Day 248: Few-Shot Examples and Chain of Thought
- ← Back to the Day 248 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/llms-and-generative-ai/day-248-few-shot-examples-and-chain-of/
Commands
Setup
pip install -r requirements/requirements.txt Run
python3 examples/few_shot_examples_and_chain_of_lib.py Test
./tests/run_tests.sh File tree
conftest.py examples/few_shot_examples_and_chain_of_lib.py examples/test_few_shot_examples_and_chain_of_lib.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/starter-run.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/few_shot_examples_and_chain_of_lib.py starter/test_few_shot_examples_and_chain_of_lib.py tests/run_tests.sh tests/test_few_shot_examples_and_chain_of_lib.py troubleshooting.md
Lab README
Lab: Day 248 -- Few-Shot Examples and Chain of Thought
Lesson
Day number: 248 of 365. Course: Course06-SS01 (LLMs and Generative AI - Working with LLMs). Topic: Few-Shot In-Context Learning, Chain of Thought, and Self-Consistency.
Purpose
Build and test a modular Few-Shot Chain-of-Thought compilation engine and Self-Consistency voting aggregator in Python. Construct exemplar pools with XML tags, compile multi-turn CoT scratchpads, and evaluate majority consensus across stochastic rollout distributions.
Learning objectives
- Formulate XML-delimited Few-Shot exemplar pools.
- Construct Chain-of-Thought scratchpads to expand autoregressive compute.
- Implement Self-Consistency majority voting over stochastic sample paths.
- Prevent exemplar biases (majority label bias, recency bias).
Prerequisites
- Day 247 (System Prompts and Role Design).
- Python 3.11+ with Pytest.
Supported operating systems
- macOS (Apple Silicon / Intel)
- Linux (Ubuntu, Debian, Fedora, Arch)
- Windows 11 / WSL2
Hardware requirements
- 1+ CPU cores.
- 1 GB RAM.
- 100 MB disk space.
Required software
- Python 3.11 or newer.
- pip package manager.
- virtualenv or venv module.
Free and open-source options
Python and standard math libraries are open source.
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/few_shot_examples_and_chain_of_lib.py: Student scaffold file.examples/few_shot_examples_and_chain_of_lib.py: Complete reference implementation.tests/test_few_shot_examples_and_chain_of_lib.py: Pytest automated validation suite.expected-output/: Verified output logs and baseline values.
How to run
Execute the reference demonstration script:
python3 examples/few_shot_examples_and_chain_of_lib.py
What the commands do
- Compiles Few-Shot CoT XML prompts.
- Aggregates stochastic sampled rollouts.
- Runs unit test assertions.
Expected output
CoT Demo Compiled Successfully. Consensus: 224
Validation steps
- Verify
compile_promptgenerates<examples>and<scratchpad>tags. - Confirm
aggregate_self_consistencycomputes consensus mode and confidence. - Validate empty list handling raises
ValueError. - Ensure all unit test assertions pass.
Tests
Run the test runner script:
./tests/run_tests.sh
Cleanup
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name ".pytest_cache" -exec rm -rf {} +
Troubleshooting
- Empty sample list: Pass at least 1 sample to
aggregate_self_consistency.
Security notes
All prompt generation executes locally in memory.
Extension exercises
- Implement a dynamic cosine-similarity exemplar retriever.
- Build a Tree-of-Thoughts breadth-first search navigator.
Navigation
- Lesson title: Few-Shot Examples and Chain of Thought
- Day number: 248 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-248-few-shot-examples-and-chain-of
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-248-few-shot-examples-and-chain-ofwhen the site is running.
Expected output
FIELDS.md
# Expected Output Fields: Day 248
- `CoT Demo`: Status of Few-Shot CoT compilation and Self-Consistency aggregation.
examples-run.txt
CoT Demo Compiled Successfully. Consensus: 224
measured-values.txt
CoT Demo: Success
Consensus Winner: 224
Confidence: 80%
starter-run.txt
Starter scaffold executed. Ready for student implementation.
test-run.txt
============================= test session starts ==============================
collected 3 items
tests/test_few_shot_examples_and_chain_of_lib.py::test_few_shot_cot_compilation PASSED [ 33%]
tests/test_few_shot_examples_and_chain_of_lib.py::test_self_consistency_majority_voting PASSED [ 66%]
tests/test_few_shot_examples_and_chain_of_lib.py::test_self_consistency_empty_validation PASSED [100%]
============================== 3 passed in 0.38s ===============================
Source files
conftest.py (585 bytes)
"""Make this lab importable when the suite runs.
pytest puts the test file's own directory on sys.path, not the lab root, so a
test that does `from examples.<module> import ...` -- or imports a module that
lives in examples/ directly -- fails without this. Putting it here rather than
in run_tests.sh means a bare `pytest` from the lab root works too.
"""
import os
import sys
LAB = os.path.dirname(os.path.abspath(__file__))
for candidate in (LAB, os.path.join(LAB, "examples")):
if os.path.isdir(candidate) and candidate not in sys.path:
sys.path.insert(0, candidate)
examples/few_shot_examples_and_chain_of_lib.py (2383 bytes)
from typing import List, Dict, Any
from collections import Counter
class FewShotCoTEngine:
def __init__(self, task_description: str):
self.task_description = task_description.strip()
self.exemplars: List[Dict[str, str]] = []
def add_exemplar(self, question: str, reasoning: str, answer: str) -> "FewShotCoTEngine":
self.exemplars.append({
"question": question.strip(),
"reasoning": reasoning.strip(),
"answer": answer.strip()
})
return self
def compile_prompt(self, query: str) -> str:
parts = [f"<instructions>\n{self.task_description}\nThink step by step before providing the final answer.\n</instructions>"]
if self.exemplars:
parts.append("<examples>")
for i, ex in enumerate(self.exemplars):
parts.append(
f'<example id="{i+1}">\n'
f'<input>{ex["question"]}</input>\n'
f'<reasoning>{ex["reasoning"]}</reasoning>\n'
f'<answer>{ex["answer"]}</answer>\n'
f'</example>'
)
parts.append("</examples>")
parts.append(f"<query>\n{query.strip()}\n</query>\n\n<scratchpad>\nLet us reason step by step:")
return "\n\n".join(parts)
@staticmethod
def aggregate_self_consistency(sampled_answers: List[str]) -> Dict[str, Any]:
if not sampled_answers:
raise ValueError("Sampled answers list cannot be empty.")
counts = Counter(sampled_answers)
majority_answer, vote_count = counts.most_common(1)[0]
confidence = vote_count / len(sampled_answers)
return {
"consensus_answer": majority_answer,
"confidence": confidence,
"votes": dict(counts),
"total_samples": len(sampled_answers)
}
def run_cot_demo():
engine = FewShotCoTEngine("Solve multi-step math problems")
engine.add_exemplar("What is 12 * 15?", "12 * 10 = 120, 12 * 5 = 60, 120 + 60 = 180", "180")
prompt = engine.compile_prompt("What is 14 * 16?")
samples = ["224", "224", "214", "224", "224"]
result = engine.aggregate_self_consistency(samples)
print("CoT Demo Compiled Successfully. Consensus:", result["consensus_answer"])
return prompt, result
if __name__ == "__main__":
run_cot_demo()
examples/test_few_shot_examples_and_chain_of_lib.py (991 bytes)
import pytest
from examples.few_shot_examples_and_chain_of_lib import FewShotCoTEngine
def test_few_shot_cot_compilation():
engine = FewShotCoTEngine("Math Solver")
engine.add_exemplar("1+1", "1 plus 1 is 2", "2")
prompt = engine.compile_prompt("2+2")
assert "<instructions>" in prompt
assert "<examples>" in prompt
assert '<example id="1">' in prompt
assert "<reasoning>1 plus 1 is 2</reasoning>" in prompt
assert "<query>\n2+2\n</query>" in prompt
assert "<scratchpad>" in prompt
def test_self_consistency_majority_voting():
samples = ["A", "B", "A", "A", "C"]
result = FewShotCoTEngine.aggregate_self_consistency(samples)
assert result["consensus_answer"] == "A"
assert result["confidence"] == 0.6
assert result["votes"] == {"A": 3, "B": 1, "C": 1}
assert result["total_samples"] == 5
def test_self_consistency_empty_validation():
with pytest.raises(ValueError):
FewShotCoTEngine.aggregate_self_consistency([])
metadata.yml (443 bytes)
lesson_id: D248
day: 248
kind: lab
languages:
- python
setup_commands:
- 'pip install -r requirements/requirements.txt'
run_commands:
- 'python3 examples/few_shot_examples_and_chain_of_lib.py'
test_commands:
- './tests/run_tests.sh'
cleanup_commands:
- 'find . -type d -name "__pycache__" -exec rm -rf {} +'
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-29'
executed_on: 'macos-arm64'
requirements/requirements.txt (41 bytes)
torch>=2.2.0
numpy>=1.26.0
pytest>=8.0.0
starter/few_shot_examples_and_chain_of_lib.py (631 bytes)
from typing import List, Dict, Any
from collections import Counter
class FewShotCoTEngine:
def __init__(self, task_description: str):
self.task_description = task_description
self.exemplars: List[Dict[str, str]] = []
def add_exemplar(self, question: str, reasoning: str, answer: str) -> "FewShotCoTEngine":
# TODO: Add exemplar
pass
def compile_prompt(self, query: str) -> str:
# TODO: Compile XML CoT prompt
pass
@staticmethod
def aggregate_self_consistency(sampled_answers: List[str]) -> Dict[str, Any]:
# TODO: Perform majority voting
pass
starter/test_few_shot_examples_and_chain_of_lib.py (991 bytes)
import pytest
from examples.few_shot_examples_and_chain_of_lib import FewShotCoTEngine
def test_few_shot_cot_compilation():
engine = FewShotCoTEngine("Math Solver")
engine.add_exemplar("1+1", "1 plus 1 is 2", "2")
prompt = engine.compile_prompt("2+2")
assert "<instructions>" in prompt
assert "<examples>" in prompt
assert '<example id="1">' in prompt
assert "<reasoning>1 plus 1 is 2</reasoning>" in prompt
assert "<query>\n2+2\n</query>" in prompt
assert "<scratchpad>" in prompt
def test_self_consistency_majority_voting():
samples = ["A", "B", "A", "A", "C"]
result = FewShotCoTEngine.aggregate_self_consistency(samples)
assert result["consensus_answer"] == "A"
assert result["confidence"] == 0.6
assert result["votes"] == {"A": 3, "B": 1, "C": 1}
assert result["total_samples"] == 5
def test_self_consistency_empty_validation():
with pytest.raises(ValueError):
FewShotCoTEngine.aggregate_self_consistency([])
tests/run_tests.sh (227 bytes)
#!/usr/bin/env bash
set -euo pipefail
echo "========================================"
echo "Running Day 248 Lab Test Suite"
echo "========================================"
pytest tests/ -v
echo "All tests passed successfully."
tests/test_few_shot_examples_and_chain_of_lib.py (991 bytes)
import pytest
from examples.few_shot_examples_and_chain_of_lib import FewShotCoTEngine
def test_few_shot_cot_compilation():
engine = FewShotCoTEngine("Math Solver")
engine.add_exemplar("1+1", "1 plus 1 is 2", "2")
prompt = engine.compile_prompt("2+2")
assert "<instructions>" in prompt
assert "<examples>" in prompt
assert '<example id="1">' in prompt
assert "<reasoning>1 plus 1 is 2</reasoning>" in prompt
assert "<query>\n2+2\n</query>" in prompt
assert "<scratchpad>" in prompt
def test_self_consistency_majority_voting():
samples = ["A", "B", "A", "A", "C"]
result = FewShotCoTEngine.aggregate_self_consistency(samples)
assert result["consensus_answer"] == "A"
assert result["confidence"] == 0.6
assert result["votes"] == {"A": 3, "B": 1, "C": 1}
assert result["total_samples"] == 5
def test_self_consistency_empty_validation():
with pytest.raises(ValueError):
FewShotCoTEngine.aggregate_self_consistency([])
Troubleshooting
Troubleshooting: Day 248 - Few-Shot Examples and Chain of Thought
Common Issues
- Low reasoning diversity:
- Cause: Sampling with Temperature = 0 in Self-Consistency.
- Fix: Use Temperature = 0.7 when sampling multiple CoT paths.
Security notes
Security & Privacy: Day 248 - Few-Shot Examples and Chain of Thought
Security Guidance
- Encapsulate few-shot examples inside tags to prevent prompt injection.