AI Engineering: Agents and Applications › AI Coding Agents › Day 307
Hands-on lab — Day 307: Coding Agents in CI and Automation
- ← Back to the Day 307 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-307-coding-agents-in-ci-and-automation/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/ci_agent_runner.py Test
bash tests/run_tests.sh File tree
examples/ci_agent_runner.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/ci_agent_runner.py tests/run_tests.sh tests/test_ci_agent.py troubleshooting.md
Lab README
Day 307 Lab: Coding Agents in CI and Automation
Lesson
- Lesson title: Coding Agents in CI and Automation
- Day number: 307 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-307-coding-agents-in-ci-and-automation
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-307-coding-agents-in-ci-and-automationwhen the site is running.
Purpose
Build a Headless CI Agent Runner & PR Generator in Python that manages isolated git branches, executes automated test suites, and synthesizes structured pull request descriptions.
Learning objectives
- Programmatically manage git branches and workspace state.
- Execute automated test harnesses and capture pass/fail telemetry.
- Synthesize enterprise pull request markdown descriptions.
- Format verification evidence logs for maintainer review.
Prerequisites
- Python 3.10+ installed
- pytest installed
- git installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest, git
Free and open-source options
- Python Standard Library, Pytest, Git
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/ci_agent_runner.py: Starter implementation skeletonexamples/ci_agent_runner.py: Verified reference implementationtests/test_ci_agent.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/ci_agent_runner.py
What the commands do
- Executes test command, captures output, and compiles pull request summary.
Expected output
### Automated Pull Request: Resolves #1
#### Problem Statement
Fix bug
#### Changes Applied
- Fix line 1
#### Verification Evidence
- Automated test suite: `100% Passed`
- Linters & static typing: `Clean`
> *Generated by Headless CI Coding Agent.*
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Test execution and exit code handling
- Pull request description formatting with issue IDs and changes
- Verification evidence attachment
- Handling of multiple change items
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure git is installed and in system PATH.
Security notes
Runs completely offline on local CPU.
Extension exercises
Implement automated git commit message formatting following Conventional Commits (feat(auth): ...).
Navigation
Day number: 307 of 365
Expected output
FIELDS.md
# Measured Values
- `ci_runs_executed`: 1
- `prs_generated`: 1
examples-run.txt
## Automated Pull Request: Resolves #1
### Problem Statement
Fix bug
### Changes Applied
- Fix line 1
### Verification Evidence
- Automated test suite: `100% Passed`
- Linters & static typing: `Clean`
> *Generated by Headless CI Coding Agent.*
measured-values.txt
ci_runs_executed=1
prs_generated=1
run-output.txt
## Automated Pull Request: Resolves #1
### Problem Statement
Fix bug
### Changes Applied
- Fix line 1
### Verification Evidence
- Automated test suite: `100% Passed`
- Linters & static typing: `Clean`
> *Generated by Headless CI Coding Agent.*
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <examples.ci_agent_runner.HeadlessCIAgentRunner object at 0x106f73c50>
issue_id = '88', title = 'Handle datetime timezone conversions'
changes = ['Added timezone parsing in serializer', 'Added test_timezone_conversion unit test']
def generate_pr_summary(self, issue_id: str, title: str, changes: List[str]) -> str:
> raise NotImplementedError('TASK 2: implement generate_pr_summary.')
E NotImplementedError: TASK 2: implement generate_pr_summary.
examples/ci_agent_runner.py:21: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_ci_agent.py::test_execute_test_command_success - NotImpleme...
FAILED tests/test_ci_agent.py::test_execute_test_command_failure - NotImpleme...
FAILED tests/test_ci_agent.py::test_generate_pr_summary_formatting - NotImple...
============================== 3 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-307-coding-agents-in-ci-and-automation
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 3 items
tests/test_ci_agent.py::test_execute_test_command_success PASSED [ 33%]
tests/test_ci_agent.py::test_execute_test_command_failure PASSED [ 66%]
tests/test_ci_agent.py::test_generate_pr_summary_formatting PASSED [100%]
============================== 3 passed in 0.09s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-307-coding-agents-in-ci-and-automation
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 3 items
tests/test_ci_agent.py::test_execute_test_command_success PASSED [ 33%]
tests/test_ci_agent.py::test_execute_test_command_failure PASSED [ 66%]
tests/test_ci_agent.py::test_generate_pr_summary_formatting PASSED [100%]
============================== 3 passed in 0.09s ===============================
Source files
examples/ci_agent_runner.py (1354 bytes)
import subprocess
import os
from typing import Dict, Any, List
class HeadlessCIAgentRunner:
def __init__(self, workspace_root: str):
self.workspace_root = os.path.realpath(workspace_root)
def execute_test_command(self, cmd: str = "python3 -m unittest") -> Dict[str, Any]:
res = subprocess.run(
cmd,
shell=True,
cwd=self.workspace_root,
capture_output=True,
text=True
)
return {
"exit_code": res.returncode,
"passed": res.returncode == 0,
"stdout": res.stdout,
"stderr": res.stderr
}
def generate_pr_summary(self, issue_id: str, title: str, changes: List[str]) -> str:
change_list = "\n".join([f"- {c}" for c in changes])
return (
f"## Automated Pull Request: Resolves #{issue_id}\n\n"
f"### Problem Statement\n{title}\n\n"
f"### Changes Applied\n{change_list}\n\n"
f"### Verification Evidence\n"
f"- Automated test suite: `100% Passed`\n"
f"- Linters & static typing: `Clean`\n\n"
f"> *Generated by Headless CI Coding Agent.*"
)
if __name__ == "__main__":
runner = HeadlessCIAgentRunner(".")
print(runner.generate_pr_summary("1", "Fix bug", ["Fix line 1"]))
metadata.yml (402 bytes)
lesson_id: D307
day: 307
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/ci_agent_runner.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/ci_agent_runner.py (946 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import subprocess
import os
from typing import Dict, Any, List
class HeadlessCIAgentRunner:
def __init__(self, workspace_root: str):
self.workspace_root = os.path.realpath(workspace_root)
def execute_test_command(self, cmd: str='python3 -m unittest') -> Dict[str, Any]:
raise NotImplementedError('TASK 1: implement execute_test_command.')
def generate_pr_summary(self, issue_id: str, title: str, changes: List[str]) -> str:
raise NotImplementedError('TASK 2: implement generate_pr_summary.')
if __name__ == '__main__':
runner = HeadlessCIAgentRunner('.')
print(runner.generate_pr_summary('1', 'Fix bug', ['Fix line 1']))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_ci_agent.py (1357 bytes)
import pytest
import tempfile
from examples.ci_agent_runner import HeadlessCIAgentRunner
def test_execute_test_command_success():
with tempfile.TemporaryDirectory() as tmpdir:
runner = HeadlessCIAgentRunner(tmpdir)
res = runner.execute_test_command("python3 -c 'print(\"CI OK\")'")
assert res["exit_code"] == 0
assert res["passed"] is True
assert "CI OK" in res["stdout"]
def test_execute_test_command_failure():
with tempfile.TemporaryDirectory() as tmpdir:
runner = HeadlessCIAgentRunner(tmpdir)
res = runner.execute_test_command("python3 -c 'exit(1)'")
assert res["exit_code"] == 1
assert res["passed"] is False
def test_generate_pr_summary_formatting():
runner = HeadlessCIAgentRunner(".")
summary = runner.generate_pr_summary(
issue_id="88",
title="Handle datetime timezone conversions",
changes=["Added timezone parsing in serializer", "Added test_timezone_conversion unit test"]
)
assert "Resolves #88" in summary
assert "Handle datetime timezone conversions" in summary
assert "- Added timezone parsing in serializer" in summary
assert "- Added test_timezone_conversion unit test" in summary
assert "Automated test suite: `100% Passed`" in summary
assert "Generated by Headless CI Coding Agent" in summary
Troubleshooting
Troubleshooting - Day 307
Verify subprocess commands have valid working directories.
Security notes
Security - Day 307
Restrict CI agent tokens to branch and PR creation only.