AI Engineering: Agents and Applications › AI Coding Agents › Day 306
Hands-on lab — Day 306: Reviewing and Trusting AI-Written Code
- ← Back to the Day 306 lesson
- Open the hands-on files on GitHub — clone or download them from the public labs repository
- Local path in your clone:
labs/sections/ai-engineering/day-306-reviewing-and-trusting-ai-written-code/
Commands
Setup
pip install -r requirements/requirements.txt Run
python examples/code_review_scanner.py Test
bash tests/run_tests.sh File tree
examples/code_review_scanner.py expected-output/examples-run.txt expected-output/FIELDS.md expected-output/measured-values.txt expected-output/run-output.txt expected-output/starter-run.txt expected-output/test-output.txt expected-output/test-run.txt metadata.yml README.md requirements/requirements.txt security.md starter/code_review_scanner.py tests/run_tests.sh tests/test_code_review.py troubleshooting.md
Lab README
Day 306 Lab: Reviewing and Trusting AI-Written Code
Lesson
- Lesson title: Reviewing and Trusting AI-Written Code
- Day number: 306 of 365
- Lesson article: https://ai-roadmap-365.github.io/day-306-reviewing-and-trusting-ai-written-code
- Lab files: everything you need is in this directory — follow “How to run” below.
- Browse the course locally: from the repository root, this lab also appears in the course website at
/labs/day-306-reviewing-and-trusting-ai-written-codewhen the site is running.
Purpose
Build an Automated AI Code Review & Security Scanner in Python that analyzes Python AST nodes to detect unapproved hallucinated dependencies (slopsquatting), insecure shell executions, and weak test assertions.
Learning objectives
- Traverse Python Abstract Syntax Trees using the
astmodule. - Detect insecure shell calls (
os.system,os.popen). - Audit package imports against an approved package whitelist.
- Identify weak tautological assertions (
assert True).
Prerequisites
- Python 3.10+ installed
- pytest installed
Supported operating systems
- macOS, Linux, Windows WSL2
Hardware requirements
- Standard CPU, 512MB RAM
Required software
- Python 3.10+, pytest
Free and open-source options
- Python Standard Library (ast), Pytest
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt
File structure
starter/code_review_scanner.py: Starter implementation skeletonexamples/code_review_scanner.py: Verified reference implementationtests/test_code_review.py: Test suiteexpected-output/: Captured execution logs
How to run
python3 examples/code_review_scanner.py
What the commands do
- Parses Python source files into ASTs and identifies security/quality rule violations.
Expected output
{'file': '<string>', 'passed': False, 'issue_count': 1, 'issues': [{'line': 2, 'rule': 'INSECURE_SHELL_EXECUTION', 'severity': 'CRITICAL', 'message': 'Avoid using os.system; use safe subprocess APIs.'}]}
Validation steps
Run test runner:
bash tests/run_tests.sh
Tests
The test suite validates:
- Passing clean code with approved imports
- Detection of unapproved external imports (slopsquatting risk)
- Detection of
os.systemandos.popenshell calls - Detection of weak
assert Truestatements
Cleanup
rm -rf .pytest_cache __pycache__ .venv
Troubleshooting
Ensure Python code to scan contains valid syntax without SyntaxErrors.
Security notes
Runs completely offline on local CPU.
Extension exercises
Add detection of hardcoded AWS and OpenAI API keys.
Navigation
Day number: 306 of 365
Expected output
FIELDS.md
# Measured Values
- `scans_executed`: 1
- `rules_verified`: 3
examples-run.txt
{'file': '<string>', 'passed': False, 'issue_count': 1, 'issues': [{'line': 2, 'rule': 'INSECURE_SHELL_EXECUTION', 'severity': 'CRITICAL', 'message': 'Avoid using os.system; use safe subprocess APIs.'}]}
measured-values.txt
scans_executed=1
rules_verified=3
run-output.txt
{'file': '<string>', 'passed': False, 'issue_count': 1, 'issues': [{'line': 2, 'rule': 'INSECURE_SHELL_EXECUTION', 'severity': 'CRITICAL', 'message': 'Avoid using os.system; use safe subprocess APIs.'}]}
starter-run.txt
An untouched starter fails the suite. Captured from a real run:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <examples.code_review_scanner.AICodeReviewScanner object at 0x108b0a190>
source_code = 'def test_login():\n assert True', filename = '<string>'
def scan_code(self, source_code: str, filename: str='<string>') -> Dict[str, Any]:
> raise NotImplementedError('TASK 1: implement scan_code.')
E NotImplementedError: TASK 1: implement scan_code.
examples/code_review_scanner.py:17: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_code_review.py::test_clean_code_passes - NotImplementedErro...
FAILED tests/test_code_review.py::test_unapproved_dependency_detected - NotIm...
FAILED tests/test_code_review.py::test_insecure_shell_detected - NotImplement...
FAILED tests/test_code_review.py::test_weak_assertion_detected - NotImplement...
============================== 4 failed in 0.03s ===============================
Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.
test-output.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-306-reviewing-and-trusting-ai-written-code
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 4 items
tests/test_code_review.py::test_clean_code_passes PASSED [ 25%]
tests/test_code_review.py::test_unapproved_dependency_detected PASSED [ 50%]
tests/test_code_review.py::test_insecure_shell_detected PASSED [ 75%]
tests/test_code_review.py::test_weak_assertion_detected PASSED [100%]
============================== 4 passed in 0.01s ===============================
test-run.txt
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-306-reviewing-and-trusting-ai-written-code
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 4 items
tests/test_code_review.py::test_clean_code_passes PASSED [ 25%]
tests/test_code_review.py::test_unapproved_dependency_detected PASSED [ 50%]
tests/test_code_review.py::test_insecure_shell_detected PASSED [ 75%]
tests/test_code_review.py::test_weak_assertion_detected PASSED [100%]
============================== 4 passed in 0.01s ===============================
Source files
examples/code_review_scanner.py (3105 bytes)
import ast
from typing import Dict, Any, List
class AICodeReviewScanner:
def __init__(self, approved_packages: List[str]):
self.approved_packages = set(approved_packages)
def scan_code(self, source_code: str, filename: str = "<string>") -> Dict[str, Any]:
issues = []
try:
tree = ast.parse(source_code, filename=filename)
except SyntaxError as e:
return {
"file": filename,
"passed": False,
"syntax_error": str(e),
"issues": []
}
for node in ast.walk(tree):
# Check 1: Insecure Shell Execution
if isinstance(node, ast.Call):
if isinstance(node.func, ast.Attribute) and node.func.attr in ("system", "popen"):
issues.append({
"line": node.lineno,
"rule": "INSECURE_SHELL_EXECUTION",
"severity": "CRITICAL",
"message": f"Avoid using os.{node.func.attr}; use safe subprocess APIs."
})
# Check 2: Unapproved External Imports
if isinstance(node, ast.Import):
for alias in node.names:
pkg_root = alias.name.split('.')[0]
if pkg_root not in self.approved_packages:
issues.append({
"line": node.lineno,
"rule": "UNAPPROVED_DEPENDENCY",
"severity": "HIGH",
"message": f"Import '{pkg_root}' is not in approved package whitelist."
})
elif isinstance(node, ast.ImportFrom):
if node.module:
pkg_root = node.module.split('.')[0]
if pkg_root not in self.approved_packages:
issues.append({
"line": node.lineno,
"rule": "UNAPPROVED_DEPENDENCY",
"severity": "HIGH",
"message": f"ImportFrom '{pkg_root}' is not in approved whitelist."
})
# Check 3: Weak/Tautological Assertions
if isinstance(node, ast.Assert):
if isinstance(node.test, ast.Constant) and node.test.value is True:
issues.append({
"line": node.lineno,
"rule": "WEAK_ASSERTION",
"severity": "MEDIUM",
"message": "Found tautological 'assert True' in test code."
})
return {
"file": filename,
"passed": len(issues) == 0,
"issue_count": len(issues),
"issues": issues
}
if __name__ == "__main__":
scanner = AICodeReviewScanner(["os", "sys", "math", "pytest"])
sample = "import os\nos.system('ls')"
print(scanner.scan_code(sample))
metadata.yml (406 bytes)
lesson_id: D306
day: 306
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/code_review_scanner.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/code_review_scanner.py (789 bytes)
"""STARTER -- implement each TASK below, then run the tests.
Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""
import ast
from typing import Dict, Any, List
class AICodeReviewScanner:
def __init__(self, approved_packages: List[str]):
self.approved_packages = set(approved_packages)
def scan_code(self, source_code: str, filename: str='<string>') -> Dict[str, Any]:
raise NotImplementedError('TASK 1: implement scan_code.')
if __name__ == '__main__':
scanner = AICodeReviewScanner(['os', 'sys', 'math', 'pytest'])
sample = "import os\nos.system('ls')"
print(scanner.scan_code(sample))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
$PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
pytest tests/test_*.py -v
else
python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_code_review.py (1267 bytes)
import pytest
from examples.code_review_scanner import AICodeReviewScanner
def test_clean_code_passes():
scanner = AICodeReviewScanner(["math", "typing", "pytest"])
code = """
import math
def circle_area(r: float) -> float:
return math.pi * r * r
"""
res = scanner.scan_code(code)
assert res["passed"] is True
assert res["issue_count"] == 0
def test_unapproved_dependency_detected():
scanner = AICodeReviewScanner(["math"])
code = "import phantom_jwt_lib\nfrom secret_crypto import decrypt"
res = scanner.scan_code(code)
assert res["passed"] is False
assert res["issue_count"] == 2
rules = [i["rule"] for i in res["issues"]]
assert "UNAPPROVED_DEPENDENCY" in rules
def test_insecure_shell_detected():
scanner = AICodeReviewScanner(["os"])
code = "import os\nos.system('rm -rf /tmp/data')"
res = scanner.scan_code(code)
assert res["passed"] is False
assert any(i["rule"] == "INSECURE_SHELL_EXECUTION" for i in res["issues"])
def test_weak_assertion_detected():
scanner = AICodeReviewScanner(["pytest"])
code = "def test_login():\n assert True"
res = scanner.scan_code(code)
assert res["passed"] is False
assert any(i["rule"] == "WEAK_ASSERTION" for i in res["issues"])
Troubleshooting
Troubleshooting - Day 306
Verify target Python syntax is valid.
Security notes
Security - Day 306
AST parsing evaluates code statically without executing foreign binaries.