AI Engineering: Agents and Applications › AI Coding Agents › Day 306

Hands-on lab — Day 306: Reviewing and Trusting AI-Written Code

Commands

Setup

pip install -r requirements/requirements.txt

Run

python examples/code_review_scanner.py

Test

bash tests/run_tests.sh

File tree

examples/code_review_scanner.py
expected-output/examples-run.txt
expected-output/FIELDS.md
expected-output/measured-values.txt
expected-output/run-output.txt
expected-output/starter-run.txt
expected-output/test-output.txt
expected-output/test-run.txt
metadata.yml
README.md
requirements/requirements.txt
security.md
starter/code_review_scanner.py
tests/run_tests.sh
tests/test_code_review.py
troubleshooting.md

Lab README

Day 306 Lab: Reviewing and Trusting AI-Written Code

Lesson

Purpose

Build an Automated AI Code Review & Security Scanner in Python that analyzes Python AST nodes to detect unapproved hallucinated dependencies (slopsquatting), insecure shell executions, and weak test assertions.

Learning objectives

  • Traverse Python Abstract Syntax Trees using the ast module.
  • Detect insecure shell calls (os.system, os.popen).
  • Audit package imports against an approved package whitelist.
  • Identify weak tautological assertions (assert True).

Prerequisites

  • Python 3.10+ installed
  • pytest installed

Supported operating systems

  • macOS, Linux, Windows WSL2

Hardware requirements

  • Standard CPU, 512MB RAM

Required software

  • Python 3.10+, pytest

Free and open-source options

  • Python Standard Library (ast), Pytest

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements/requirements.txt

File structure

  • starter/code_review_scanner.py: Starter implementation skeleton
  • examples/code_review_scanner.py: Verified reference implementation
  • tests/test_code_review.py: Test suite
  • expected-output/: Captured execution logs

How to run

python3 examples/code_review_scanner.py

What the commands do

  • Parses Python source files into ASTs and identifies security/quality rule violations.

Expected output

{'file': '<string>', 'passed': False, 'issue_count': 1, 'issues': [{'line': 2, 'rule': 'INSECURE_SHELL_EXECUTION', 'severity': 'CRITICAL', 'message': 'Avoid using os.system; use safe subprocess APIs.'}]}

Validation steps

Run test runner:

bash tests/run_tests.sh

Tests

The test suite validates:

  • Passing clean code with approved imports
  • Detection of unapproved external imports (slopsquatting risk)
  • Detection of os.system and os.popen shell calls
  • Detection of weak assert True statements

Cleanup

rm -rf .pytest_cache __pycache__ .venv

Troubleshooting

Ensure Python code to scan contains valid syntax without SyntaxErrors.

Security notes

Runs completely offline on local CPU.

Extension exercises

Add detection of hardcoded AWS and OpenAI API keys.

Day number: 306 of 365

Expected output

FIELDS.md

# Measured Values
- `scans_executed`: 1
- `rules_verified`: 3

examples-run.txt

{'file': '<string>', 'passed': False, 'issue_count': 1, 'issues': [{'line': 2, 'rule': 'INSECURE_SHELL_EXECUTION', 'severity': 'CRITICAL', 'message': 'Avoid using os.system; use safe subprocess APIs.'}]}

measured-values.txt

scans_executed=1
rules_verified=3

run-output.txt

{'file': '<string>', 'passed': False, 'issue_count': 1, 'issues': [{'line': 2, 'rule': 'INSECURE_SHELL_EXECUTION', 'severity': 'CRITICAL', 'message': 'Avoid using os.system; use safe subprocess APIs.'}]}

starter-run.txt

An untouched starter fails the suite. Captured from a real run:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <examples.code_review_scanner.AICodeReviewScanner object at 0x108b0a190>
source_code = 'def test_login():\n    assert True', filename = '<string>'

    def scan_code(self, source_code: str, filename: str='<string>') -> Dict[str, Any]:
>       raise NotImplementedError('TASK 1: implement scan_code.')
E       NotImplementedError: TASK 1: implement scan_code.

examples/code_review_scanner.py:17: NotImplementedError
=========================== short test summary info ============================
FAILED tests/test_code_review.py::test_clean_code_passes - NotImplementedErro...
FAILED tests/test_code_review.py::test_unapproved_dependency_detected - NotIm...
FAILED tests/test_code_review.py::test_insecure_shell_detected - NotImplement...
FAILED tests/test_code_review.py::test_weak_assertion_detected - NotImplement...
============================== 4 failed in 0.03s ===============================

Implement each TASK in starter/, copy it over examples/, and the same
suite goes green.

test-output.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-306-reviewing-and-trusting-ai-written-code
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 4 items

tests/test_code_review.py::test_clean_code_passes PASSED                 [ 25%]
tests/test_code_review.py::test_unapproved_dependency_detected PASSED    [ 50%]
tests/test_code_review.py::test_insecure_shell_detected PASSED           [ 75%]
tests/test_code_review.py::test_weak_assertion_detected PASSED           [100%]

============================== 4 passed in 0.01s ===============================

test-run.txt

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv-tools/bin/python3.14
cachedir: .pytest_cache
rootdir: <repo>/labs/sections/ai-engineering/day-306-reviewing-and-trusting-ai-written-code
plugins: cov-7.1.0, anyio-4.14.2
collecting ... collected 4 items

tests/test_code_review.py::test_clean_code_passes PASSED                 [ 25%]
tests/test_code_review.py::test_unapproved_dependency_detected PASSED    [ 50%]
tests/test_code_review.py::test_insecure_shell_detected PASSED           [ 75%]
tests/test_code_review.py::test_weak_assertion_detected PASSED           [100%]

============================== 4 passed in 0.01s ===============================

Source files

examples/code_review_scanner.py (3105 bytes)
import ast
from typing import Dict, Any, List

class AICodeReviewScanner:
    def __init__(self, approved_packages: List[str]):
        self.approved_packages = set(approved_packages)
        
    def scan_code(self, source_code: str, filename: str = "<string>") -> Dict[str, Any]:
        issues = []
        try:
            tree = ast.parse(source_code, filename=filename)
        except SyntaxError as e:
            return {
                "file": filename,
                "passed": False,
                "syntax_error": str(e),
                "issues": []
            }
            
        for node in ast.walk(tree):
            # Check 1: Insecure Shell Execution
            if isinstance(node, ast.Call):
                if isinstance(node.func, ast.Attribute) and node.func.attr in ("system", "popen"):
                    issues.append({
                        "line": node.lineno,
                        "rule": "INSECURE_SHELL_EXECUTION",
                        "severity": "CRITICAL",
                        "message": f"Avoid using os.{node.func.attr}; use safe subprocess APIs."
                    })
                    
            # Check 2: Unapproved External Imports
            if isinstance(node, ast.Import):
                for alias in node.names:
                    pkg_root = alias.name.split('.')[0]
                    if pkg_root not in self.approved_packages:
                        issues.append({
                            "line": node.lineno,
                            "rule": "UNAPPROVED_DEPENDENCY",
                            "severity": "HIGH",
                            "message": f"Import '{pkg_root}' is not in approved package whitelist."
                        })
            elif isinstance(node, ast.ImportFrom):
                if node.module:
                    pkg_root = node.module.split('.')[0]
                    if pkg_root not in self.approved_packages:
                        issues.append({
                            "line": node.lineno,
                            "rule": "UNAPPROVED_DEPENDENCY",
                            "severity": "HIGH",
                            "message": f"ImportFrom '{pkg_root}' is not in approved whitelist."
                        })
                        
            # Check 3: Weak/Tautological Assertions
            if isinstance(node, ast.Assert):
                if isinstance(node.test, ast.Constant) and node.test.value is True:
                    issues.append({
                        "line": node.lineno,
                        "rule": "WEAK_ASSERTION",
                        "severity": "MEDIUM",
                        "message": "Found tautological 'assert True' in test code."
                    })
                    
        return {
            "file": filename,
            "passed": len(issues) == 0,
            "issue_count": len(issues),
            "issues": issues
        }

if __name__ == "__main__":
    scanner = AICodeReviewScanner(["os", "sys", "math", "pytest"])
    sample = "import os\nos.system('ls')"
    print(scanner.scan_code(sample))
metadata.yml (406 bytes)
lesson_id: D306
day: 306
kind: python
languages:
- python
setup_commands:
- pip install -r requirements/requirements.txt
run_commands:
- python examples/code_review_scanner.py
test_commands:
- bash tests/run_tests.sh
cleanup_commands:
- rm -rf __pycache__ .pytest_cache
requires_network: false
requires_api_key: false
estimated_minutes: 45
last_executed: '2026-08-30'
executed_on: macOS 15.0 Apple Silicon
requirements/requirements.txt (14 bytes)
pytest>=7.0.0
starter/code_review_scanner.py (789 bytes)
"""STARTER -- implement each TASK below, then run the tests.

Every function that raises NotImplementedError is yours to write. The
imports, constants and data structures are given; the logic is not.
Run `bash tests/run_tests.sh` to see which tasks are still outstanding.
"""

import ast
from typing import Dict, Any, List

class AICodeReviewScanner:

    def __init__(self, approved_packages: List[str]):
        self.approved_packages = set(approved_packages)

    def scan_code(self, source_code: str, filename: str='<string>') -> Dict[str, Any]:
        raise NotImplementedError('TASK 1: implement scan_code.')
if __name__ == '__main__':
    scanner = AICodeReviewScanner(['os', 'sys', 'math', 'pytest'])
    sample = "import os\nos.system('ls')"
    print(scanner.scan_code(sample))
tests/run_tests.sh (253 bytes)
#!/usr/bin/env bash
set -e
export PYTHONPATH=examples:.
if [ -n "$PYTEST" ]; then
    $PYTEST tests/test_*.py -v
elif command -v pytest &> /dev/null; then
    pytest tests/test_*.py -v
else
    python3 -m unittest discover -s tests -p 'test_*.py' -v
fi
tests/test_code_review.py (1267 bytes)
import pytest
from examples.code_review_scanner import AICodeReviewScanner

def test_clean_code_passes():
    scanner = AICodeReviewScanner(["math", "typing", "pytest"])
    code = """
import math
def circle_area(r: float) -> float:
    return math.pi * r * r
"""
    res = scanner.scan_code(code)
    assert res["passed"] is True
    assert res["issue_count"] == 0

def test_unapproved_dependency_detected():
    scanner = AICodeReviewScanner(["math"])
    code = "import phantom_jwt_lib\nfrom secret_crypto import decrypt"
    res = scanner.scan_code(code)
    assert res["passed"] is False
    assert res["issue_count"] == 2
    rules = [i["rule"] for i in res["issues"]]
    assert "UNAPPROVED_DEPENDENCY" in rules

def test_insecure_shell_detected():
    scanner = AICodeReviewScanner(["os"])
    code = "import os\nos.system('rm -rf /tmp/data')"
    res = scanner.scan_code(code)
    assert res["passed"] is False
    assert any(i["rule"] == "INSECURE_SHELL_EXECUTION" for i in res["issues"])

def test_weak_assertion_detected():
    scanner = AICodeReviewScanner(["pytest"])
    code = "def test_login():\n    assert True"
    res = scanner.scan_code(code)
    assert res["passed"] is False
    assert any(i["rule"] == "WEAK_ASSERTION" for i in res["issues"])

Troubleshooting

Troubleshooting - Day 306

Verify target Python syntax is valid.

Security notes

Security - Day 306

AST parsing evaluates code statically without executing foreign binaries.