AI Engineering: Agents and Applications › AI Coding Agents › Day 304

Hands-on lab — Day 304: Effective Agentic Coding Workflows

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/self_healing_runner.py

Test

bash tests/run_tests.sh

File tree

examples/self_healing_runner.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/self_healing_runner.py
tests/run_tests.sh
tests/test_self_healing.py
troubleshooting.md

Lab README

Day 304 Lab: Effective Agentic Coding Workflows

Lesson

Purpose

Build an Automated Self-Healing Test Runner and Diagnostic Parser in Python that captures test execution results, extracts stack trace diagnostics, and compiles repair prompts.

Learning objectives

  • Execute shell test commands and capture multi-stream outputs.
  • Parse Python tracebacks to extract failing files, lines, and error types.
  • Format structured repair prompts for coding agents.
  • Enforce max iteration safety limits.

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library, Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/self_healing_runner.py: Starter implementation skeleton
  • examples/self_healing_runner.py: Verified reference implementation
  • tests/test_self_healing.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/self_healing_runner.py

What the commands do

  • Executes test command, parses failing traceback, and formats diagnostic repair prompt.

Expected output

Execution output: Tests OK

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Passing command execution and exit code 0
  • Failing command execution and exit code != 0
  • Traceback parser extracting file, line, and exception message
  • Structured repair prompt compilation

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Verify regex matches Python standard traceback format.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add support for pytest summary parsing (FAILED tests/test_x.py::test_y).

Day number: 304 of 365

Expected output

FIELDS.md

# Measured Values
- `tracebacks_parsed`: 1
- `repair_prompts_generated`: 1

examples-run.txt

Execution output: Tests OK

measured-values.txt

tracebacks_parsed=1
repair_prompts_generated=1

run-output.txt

Execution output: Tests OK

starter-run.txt

An untouched starter fails the suite. Captured from a real run:


self = <examples.self_healing_runner.SelfHealingTestRunner object at 0x10abd1e00>
diagnostics = {'failing_file': 'auth.py', 'line_number': 15, 'error_type': 'ValueError', 'error_message': 'Invalid token'}
raw_output = 'Raw log output'

    def format_repair_prompt(self, diagnostics: Dict[str, Any], raw_output: str) -> str:
>       raise NotImplementedError('TASK 3: implement format_repair_prompt.')
E       NotImplementedError: TASK 3: implement format_repair_prompt.

examples/self_healing_runner.py:25: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_self_healing.py::test_execute_passing_command - NotImplemen...
FAILED tests/test_self_healing.py::test_execute_failing_command - NotImplemen...
FAILED tests/test_self_healing.py::test_extract_traceback_diagnostics - NotIm...
FAILED tests/test_self_healing.py::test_format_repair_prompt - NotImplemented...
============================== 4 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-304-effective-agentic-coding-workflows
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 4 items

tests/test_self_healing.py::test_execute_passing_command PASSED          [ 25%]
tests/test_self_healing.py::test_execute_failing_command PASSED          [ 50%]
tests/test_self_healing.py::test_extract_traceback_diagnostics PASSED    [ 75%]
tests/test_self_healing.py::test_format_repair_prompt PASSED             [100%]

============================== 4 passed in 0.09s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-304-effective-agentic-coding-workflows
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 4 items

tests/test_self_healing.py::test_execute_passing_command PASSED          [ 25%]
tests/test_self_healing.py::test_execute_failing_command PASSED          [ 50%]
tests/test_self_healing.py::test_extract_traceback_diagnostics PASSED    [ 75%]
tests/test_self_healing.py::test_format_repair_prompt PASSED             [100%]

============================== 4 passed in 0.09s ===============================

Source files

examples/self_healing_runner.py (2430 bytes)
import subprocess
import re
from typing import Dict, Any

class SelfHealingTestRunner:
    def __init__(self, workspace_root: str, max_iterations: int = 4):
        self.workspace_root = workspace_root
        self.max_iterations = max_iterations
        
    def execute_test_command(self, test_cmd: str) -> Dict[str, Any]:
        result = subprocess.run(
            test_cmd,
            shell=True,
            cwd=self.workspace_root,
            capture_output=True,
            text=True
        )
        return {
            "exit_code": result.returncode,
            "passed": result.returncode == 0,
            "stdout": result.stdout,
            "stderr": result.stderr,
            "combined_output": result.stdout + "\n" + result.stderr
        }
        
    def extract_traceback_diagnostics(self, raw_output: str) -> Dict[str, Any]:
        diagnostics = {
            "failing_file": None,
            "line_number": None,
            "error_type": "UnknownError",
            "error_message": ""
        }
        
        file_match = re.search(r'File "([^"]+\.py)", line (\d+)', raw_output)
        if file_match:
            diagnostics["failing_file"] = file_match.group(1)
            diagnostics["line_number"] = int(file_match.group(2))
            
        error_match = re.search(r'([A-Za-z]+Error|Exception): (.+)', raw_output)
        if error_match:
            diagnostics["error_type"] = error_match.group(1)
            diagnostics["error_message"] = error_match.group(2).strip()
            
        return diagnostics
        
    def format_repair_prompt(self, diagnostics: Dict[str, Any], raw_output: str) -> str:
        return (
            f"# TEST FAILURE DETECTED\n"
            f"Failing File: {diagnostics.get('failing_file')}\n"
            f"Line Number: {diagnostics.get('line_number')}\n"
            f"Error: {diagnostics.get('error_type')}: {diagnostics.get('error_message')}\n\n"
            f"## Full Terminal Output:\n```text\n{raw_output.strip()}\n```\n\n"
            f"## Instructions:\n"
            f"1. Analyze the root cause of this failure.\n"
            f"2. Apply a targeted search-and-replace fix.\n"
            f"3. Do not modify unrelated functions."
        )

if __name__ == "__main__":
    runner = SelfHealingTestRunner(".")
    res = runner.execute_test_command("python3 -c 'print(\"Tests OK\")'")
    print("Execution output:", res["stdout"].strip())
metadata.yml (406 bytes)
lesson_id: D304
day: 304
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/self_healing_runner.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/self_healing_runner.py (1199 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import subprocess
import re
from typing import Dict, Any

class SelfHealingTestRunner:

    def __init__(self, workspace_root: str, max_iterations: int=4):
        self.workspace_root = workspace_root
        self.max_iterations = max_iterations

    def execute_test_command(self, test_cmd: str) -> Dict[str, Any]:
        raise NotImplementedError('TASK 1: implement execute_test_command.')

    def extract_traceback_diagnostics(self, raw_output: str) -> Dict[str, Any]:
        raise NotImplementedError('TASK 2: implement extract_traceback_diagnostics.')

    def format_repair_prompt(self, diagnostics: Dict[str, Any], raw_output: str) -> str:
        raise NotImplementedError('TASK 3: implement format_repair_prompt.')
if __name__ == '__main__':
    runner = SelfHealingTestRunner('.')
    res = runner.execute_test_command('python3 -c \'print("Tests OK")\'')
    print('Execution output:', res['stdout'].strip())
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_self_healing.py (1632 bytes)
import pytest
from examples.self_healing_runner import SelfHealingTestRunner

def test_execute_passing_command():
    runner = SelfHealingTestRunner(".")
    res = runner.execute_test_command("python3 -c 'print(\"Passed\")'")
    assert res["exit_code"] == 0
    assert res["passed"] is True
    assert "Passed" in res["stdout"]

def test_execute_failing_command():
    runner = SelfHealingTestRunner(".")
    res = runner.execute_test_command("python3 -c 'assert False, \"Failed Assertion\"'")
    assert res["exit_code"] != 0
    assert res["passed"] is False
    assert "AssertionError" in res["combined_output"]

def test_extract_traceback_diagnostics():
    sample_trace = """
Traceback (most recent call last):
  File "calculator.py", line 42, in divide
    return a / b
ZeroDivisionError: division by zero
"""
    runner = SelfHealingTestRunner(".")
    diag = runner.extract_traceback_diagnostics(sample_trace)
    assert diag["failing_file"] == "calculator.py"
    assert diag["line_number"] == 42
    assert diag["error_type"] == "ZeroDivisionError"
    assert diag["error_message"] == "division by zero"

def test_format_repair_prompt():
    runner = SelfHealingTestRunner(".")
    diag = {
        "failing_file": "auth.py",
        "line_number": 15,
        "error_type": "ValueError",
        "error_message": "Invalid token"
    }
    prompt = runner.format_repair_prompt(diag, "Raw log output")
    assert "# TEST FAILURE DETECTED" in prompt
    assert "Failing File: auth.py" in prompt
    assert "Line Number: 15" in prompt
    assert "ValueError: Invalid token" in prompt
    assert "## Instructions:" in prompt

Troubleshooting

Troubleshooting - Day 304

Verify test command exit codes are properly evaluated.

Security notes

Security - Day 304

Ensure subshell commands run in safe workspace environments.