AI Engineering: Agents and Applications › AI Coding Agents › Day 307

Hands-on lab — Day 307: Coding Agents in CI and Automation

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/ci_agent_runner.py

Test

bash tests/run_tests.sh

File tree

examples/ci_agent_runner.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/ci_agent_runner.py
tests/run_tests.sh
tests/test_ci_agent.py
troubleshooting.md

Lab README

Day 307 Lab: Coding Agents in CI and Automation

Lesson

Purpose

Build a Headless CI Agent Runner & PR Generator in Python that manages isolated git branches, executes automated test suites, and synthesizes structured pull request descriptions.

Learning objectives

  • Programmatically manage git branches and workspace state.
  • Execute automated test harnesses and capture pass/fail telemetry.
  • Synthesize enterprise pull request markdown descriptions.
  • Format verification evidence logs for maintainer review.

Prerequisites

  • Python 3.10+ installed
  • pytest installed
  • git installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest, git

Free and open-source options

  • Python Standard Library, Pytest, Git

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/ci_agent_runner.py: Starter implementation skeleton
  • examples/ci_agent_runner.py: Verified reference implementation
  • tests/test_ci_agent.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/ci_agent_runner.py

What the commands do

  • Executes test command, captures output, and compiles pull request summary.

Expected output

### Automated Pull Request: Resolves #1

#### Problem Statement
Fix bug

#### Changes Applied
- Fix line 1

#### Verification Evidence
- Automated test suite: `100% Passed`
- Linters & static typing: `Clean`

> *Generated by Headless CI Coding Agent.*

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Test execution and exit code handling
  • Pull request description formatting with issue IDs and changes
  • Verification evidence attachment
  • Handling of multiple change items

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure git is installed and in system PATH.

Security notes

Runs completely offline on local CPU.

Extension exercises

Implement automated git commit message formatting following Conventional Commits (feat(auth): ...).

Day number: 307 of 365

Expected output

FIELDS.md

# Measured Values
- `ci_runs_executed`: 1
- `prs_generated`: 1

examples-run.txt

## Automated Pull Request: Resolves #1

### Problem Statement
Fix bug

### Changes Applied
- Fix line 1

### Verification Evidence
- Automated test suite: `100% Passed`
- Linters & static typing: `Clean`

> *Generated by Headless CI Coding Agent.*

measured-values.txt

ci_runs_executed=1
prs_generated=1

run-output.txt

## Automated Pull Request: Resolves #1

### Problem Statement
Fix bug

### Changes Applied
- Fix line 1

### Verification Evidence
- Automated test suite: `100% Passed`
- Linters & static typing: `Clean`

> *Generated by Headless CI Coding Agent.*

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <examples.ci_agent_runner.HeadlessCIAgentRunner object at 0x106f73c50>
issue_id = '88', title = 'Handle datetime timezone conversions'
changes = ['Added timezone parsing in serializer', 'Added test_timezone_conversion unit test']

    def generate_pr_summary(self, issue_id: str, title: str, changes: List[str]) -> str:
>       raise NotImplementedError('TASK 2: implement generate_pr_summary.')
E       NotImplementedError: TASK 2: implement generate_pr_summary.

examples/ci_agent_runner.py:21: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_ci_agent.py::test_execute_test_command_success - NotImpleme...
FAILED tests/test_ci_agent.py::test_execute_test_command_failure - NotImpleme...
FAILED tests/test_ci_agent.py::test_generate_pr_summary_formatting - NotImple...
============================== 3 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-307-coding-agents-in-ci-and-automation
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 3 items

tests/test_ci_agent.py::test_execute_test_command_success PASSED         [ 33%]
tests/test_ci_agent.py::test_execute_test_command_failure PASSED         [ 66%]
tests/test_ci_agent.py::test_generate_pr_summary_formatting PASSED       [100%]

============================== 3 passed in 0.09s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-307-coding-agents-in-ci-and-automation
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 3 items

tests/test_ci_agent.py::test_execute_test_command_success PASSED         [ 33%]
tests/test_ci_agent.py::test_execute_test_command_failure PASSED         [ 66%]
tests/test_ci_agent.py::test_generate_pr_summary_formatting PASSED       [100%]

============================== 3 passed in 0.09s ===============================

Source files

examples/ci_agent_runner.py (1354 bytes)
import subprocess
import os
from typing import Dict, Any, List

class HeadlessCIAgentRunner:
    def __init__(self, workspace_root: str):
        self.workspace_root = os.path.realpath(workspace_root)
        
    def execute_test_command(self, cmd: str = "python3 -m unittest") -> Dict[str, Any]:
        res = subprocess.run(
            cmd,
            shell=True,
            cwd=self.workspace_root,
            capture_output=True,
            text=True
        )
        return {
            "exit_code": res.returncode,
            "passed": res.returncode == 0,
            "stdout": res.stdout,
            "stderr": res.stderr
        }
        
    def generate_pr_summary(self, issue_id: str, title: str, changes: List[str]) -> str:
        change_list = "\n".join([f"- {c}" for c in changes])
        return (
            f"## Automated Pull Request: Resolves #{issue_id}\n\n"
            f"### Problem Statement\n{title}\n\n"
            f"### Changes Applied\n{change_list}\n\n"
            f"### Verification Evidence\n"
            f"- Automated test suite: `100% Passed`\n"
            f"- Linters & static typing: `Clean`\n\n"
            f"> *Generated by Headless CI Coding Agent.*"
        )

if __name__ == "__main__":
    runner = HeadlessCIAgentRunner(".")
    print(runner.generate_pr_summary("1", "Fix bug", ["Fix line 1"]))
metadata.yml (402 bytes)
lesson_id: D307
day: 307
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/ci_agent_runner.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/ci_agent_runner.py (946 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import subprocess
import os
from typing import Dict, Any, List

class HeadlessCIAgentRunner:

    def __init__(self, workspace_root: str):
        self.workspace_root = os.path.realpath(workspace_root)

    def execute_test_command(self, cmd: str='python3 -m unittest') -> Dict[str, Any]:
        raise NotImplementedError('TASK 1: implement execute_test_command.')

    def generate_pr_summary(self, issue_id: str, title: str, changes: List[str]) -> str:
        raise NotImplementedError('TASK 2: implement generate_pr_summary.')
if __name__ == '__main__':
    runner = HeadlessCIAgentRunner('.')
    print(runner.generate_pr_summary('1', 'Fix bug', ['Fix line 1']))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_ci_agent.py (1357 bytes)
import pytest
import tempfile
from examples.ci_agent_runner import HeadlessCIAgentRunner

def test_execute_test_command_success():
    with tempfile.TemporaryDirectory() as tmpdir:
        runner = HeadlessCIAgentRunner(tmpdir)
        res = runner.execute_test_command("python3 -c 'print(\"CI OK\")'")
        assert res["exit_code"] == 0
        assert res["passed"] is True
        assert "CI OK" in res["stdout"]

def test_execute_test_command_failure():
    with tempfile.TemporaryDirectory() as tmpdir:
        runner = HeadlessCIAgentRunner(tmpdir)
        res = runner.execute_test_command("python3 -c 'exit(1)'")
        assert res["exit_code"] == 1
        assert res["passed"] is False

def test_generate_pr_summary_formatting():
    runner = HeadlessCIAgentRunner(".")
    summary = runner.generate_pr_summary(
        issue_id="88",
        title="Handle datetime timezone conversions",
        changes=["Added timezone parsing in serializer", "Added test_timezone_conversion unit test"]
    )
    assert "Resolves #88" in summary
    assert "Handle datetime timezone conversions" in summary
    assert "- Added timezone parsing in serializer" in summary
    assert "- Added test_timezone_conversion unit test" in summary
    assert "Automated test suite: `100% Passed`" in summary
    assert "Generated by Headless CI Coding Agent" in summary

Troubleshooting

Troubleshooting - Day 307

Verify subprocess commands have valid working directories.

Security notes

Security - Day 307

Restrict CI agent tokens to branch and PR creation only.