Code review automation

Review pipeline (PR → CI → bot → human)

  [ Open / update PR ]
        │
        ▼
  ┌─────────────┐     Gates: lint / test / types / security scan
  │  CI workflow │──── Artifacts: logs, coverage, SBOM (as needed)
  └─────────────┘
        │
        ▼
  ┌─────────────┐     Summarize: one thread per check; link docs & local commands
  │   PR bot    │──── Reproducible failures: config paths, suppression syntax, issue template
  └─────────────┘
        │
        ▼
  ┌─────────────┐     Semantics: design, edge cases, naming, evolution, whether exceptions are OK
  │ Human/agent │──── Do not repeat formatting issues the bot already flagged
  └─────────────┘

Enforce vs advisory

Prefer report-only (or a separate workflow) for new checks to collect a baseline before flipping to enforce; do not oscillate the same rule between “comment only” and “block merge” without data.

Complete GitHub Actions workflow for AI-assisted code review (paste and replace secret names):

name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize, reopened]
    paths-ignore:
      - "**/*.md"
      - "dist/**"
      - "**/package-lock.json"

jobs:
  ai-review:
    runs-on: ubuntu-latest
    if: >
      !github.event.pull_request.draft &&
      !contains(github.event.pull_request.title, '[skip-review]')
    permissions:
      pull-requests: write
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get changed files
        id: changed
        run: |
          git diff --name-only origin/${{ github.base_ref }}...HEAD \
            > changed_files.txt
          cat changed_files.txt

      - name: Run AI review
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
        run: |
          # Output review comments as structured JSON
          node .github/scripts/ai-review.js \
            --changed-files changed_files.txt \
            --output review-output.json

      - name: Post review comments
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          node .github/scripts/post-comments.js \
            --review review-output.json \
            --pr ${{ github.event.pull_request.number }}

Structured JSON output format for review comments (review-output.json):

{
  "summary": "Found 2 P0 issues, 3 P1 issues, 1 P2 issue",
  "comments": [
    {
      "file": "src/api/handlers.ts",
      "line": 42,
      "severity": "P0",
      "message": "User ID concatenated directly into SQL string: SQL injection risk",
      "suggestion": "Use a parameterized query: db.query('SELECT * FROM users WHERE id = $1', [userId])",
      "rule": "security/no-sql-injection"
    },
    {
      "file": "src/services/payment.ts",
      "line": 87,
      "severity": "P1",
      "message": "catch block swallows StripeError; caller cannot distinguish error types",
      "suggestion": "Map StripeError to a business error code and re-throw",
      "rule": "error-handling/no-swallow"
    }
  ]
}

Differentiation strategy: apply different rule sets based on file path:

// Differentiation strategy in .github/scripts/ai-review.js
function getRulesForFile(filePath) {
  if (filePath.endsWith('.env') || filePath.includes('config/secrets')) {
    return ['security/no-plaintext-secrets', 'security/no-credentials'];
  }
  if (filePath.endsWith('.sql') || filePath.includes('migrations/')) {
    return ['db/reversible-migration', 'db/no-drop-without-backup'];
  }
  if (filePath.endsWith('.tsx') || filePath.endsWith('.jsx')) {
    return ['react/no-direct-mutation', 'a11y/interactive-aria-roles', 'security/no-dangerouslySetInnerHTML'];
  }
  return ['general/no-hardcoded-secrets', 'general/error-handling'];
}

reviewdog integration configuration (.reviewdog.yml):

runner:
  ai-review:
    cmd: node .github/scripts/ai-review.js --format rdjson
    format: rdjson
    reporter: github-pr-review
    level: warning
    fail_on_error: true
    filter_mode: added

False-positive marking format (inline comment in source; the AI review script recognizes and skips the line):

// ai-review-ignore: hardcoded here is a test-env mock, not a production secret
const MOCK_API_KEY = "test-key-not-real";

// ai-review-ignore: legacy code, migration plan tracked in JIRA-1234
const result = db.query(`SELECT * FROM users WHERE id = ${userId}`);

PR bot comment etiquette

• One thread per check: fold or summarize multiple alerts from the same tool to avoid spam.
• Actionable: link docs, one local command to reproduce, official fix or suppression syntax (with file paths).
• False positives: explain how to file an issue or add an allowlist — not “please ignore” in a comment thread.
• Respect drafts: default Draft PRs to lightweight checks or advisory only; run full enforce before merge (match conditions in the workflow lab).

Noise and metrics

Drive noise down with data: track bot comments per PR, share of comments marked invalid or false positive, repeat triggers per rule, and median time from PR open to all green.

• Post-merge review: if a rule is routinely ignored or heavily suppressed, move it to advisory or narrow scope.
• Use paths-ignore or a separate job for generated dirs, lockfiles, and large assets to avoid pointless runs.
• Include “comment flood” and “merge delay” in team retros, alongside the enforce list.

Human review vs automation

Automation: rules you can enumerate in syntax and policy, consistent style across repos, security and license baselines, reproducible builds and test results.

Humans / agents: whether requirements are implemented correctly, whether abstractions fit, whether error handling and observability match the scenario, whether hidden coupling or over-engineering crept in.

Workflow snippet lab

Toggle options below to generate a pasteable GitHub Actions sketch: a paths-ignore list and jobs.<id>.if conditions (tune expressions for your trigger events).