agent-audit statically analyzes AI agent codebases for prompt injection vectors, tainted tool inputs, and unsafe MCP configurations, giving authorized teams a CI-enforceable security gate.
| Tool | HeadyZhang/agent-audit — static security scanner for LLM agent code, covering prompt injection, MCP config auditing, and taint analysis |
| Category | Static analysis / SAST for AI agent workloads |
| Primary Use | Scanning LangChain, CrewAI, AutoGen, and MCP-based projects before merge or deploy, with SARIF output and --fail-on CI gating |
| Safe Use | Defensive code review on codebases you own or are contractually authorized to assess; entirely read-only analysis, suitable for CI pipelines and security labs |
| Telemetry Note | Pure static local analysis; the inspect stdio mode is documented as read-only and never calls tools, so footprint on target directories is minimal and observable only through normal process/file-access logs |
agent-audit positions itself as security linting for AI agents, and the framing is apt: instead of generic Python SAST, it targets the specific failure modes that emerge when an LLM is wired to tools, subprocesses, and external systems. The premise in the README is blunt — agents execute code, call tools, and touch real systems, so a single unsafe input path becomes a production incident. The tool answers that with three analysis pillars: tool-boundary taint tracking, MCP configuration auditing, and semantic secret detection, all packaged behind a pip install agent-audit distribution on PyPI and an agent-audit scan ./project CLI.
The detection surface is organized as 66 rules mapped to the OWASP Agentic Top 10 (2026), with rule identifiers like AGENT-001 for command injection via unsanitized input reaching subprocess, AGENT-010 for user input concatenated into system prompts, and AGENT-041 for SQL injection through f-string interpolation. What distinguishes it from running Bandit or Semgrep on the same tree is the agent-specific coverage: MCP tool poisoning via hidden instructions in tool descriptions (AGENT-056/AGENT-057), tool shadowing where multiple servers register identical tool names (AGENT-055), rug-pull drift where server tools change after an audit (AGENT-054), sub-agent privilege escalation (AGENT-112), and HITL bypass through delegation (AGENT-118).
The README's benchmark table is unusually candid about this gap. On the labeled ground-truth corpus (81 samples, 238 labels, GT v2.2), agent-audit v0.20.0 reports raw precision of 68.86% and recall of 84.32% (F1 0.7581), while Bandit 1.8 and Semgrep 1.x sit at roughly 27-30% recall on the expressible injection/RCE subset — and 0% on MCP configuration findings, because neither tool parses MCP config files at all. That zero on Set B is the strongest argument for a dedicated scanner: configuration-borne risks like exposed env keys in mcp_config.json (AGENT-031) simply do not exist as a category for general-purpose linters.
The precision number deserves the scrutiny the project itself gives it. The validation notes explain that precision fell from v0.19.0 not because of detection regression but because two file-discovery fixes surfaced previously unreachable findings that the ground-truth corpus does not yet label — at least 21 of the 90 false positives are correct by their rules' own definitions, per docs/decisions/METRICS_RETRACTION.md. A post-hoc adjusted F1 of 0.84 is computable but deliberately not used as a headline because it is not directly reproducible from precision_recall.py. This level of metrics hygiene is a good sign for a young tool, and the whole benchmark is reproducible from a clean clone.
Architecturally, the README describes a multi-scanner pipeline feeding a central RuleEngine. A PythonScanner performs AST analysis with a TaintTracker resolving source-to-sink reachability and a DangerousOperationAnalyzer detecting tool boundaries — this is the mechanism that connects a user-controlled string to an eventual shell=True sink. A SecretScanner generates regex candidates refined by a three-stage SemanticAnalyzer (known formats, entropy/placeholder filtering, context), which addresses the classic false-positive plague of naive key-matching. An MCPConfigScanner evaluates server provenance, path permissions, and auth, and a PrivilegeScanner looks at daemon, sudoers, sandbox, and credential-store configuration.
Output handling is where the tool earns its CI-gating role. Beyond the human-readable terminal report with a 0-10 risk score and tiered findings (BLOCK at confidence ≥ 90%, then WARN and INFO), it emits --format json, sarif, or markdown. The --severity flag filters what is reported while --fail-on high (or critical) controls the exit code, making it trivially composable as a merge gate. Baseline scanning rounds this out: --save-baseline baseline.json followed by --baseline baseline.json --fail-on-new lets a team adopt the tool on a legacy codebase without drowning in pre-existing findings, tracking only new regressions per commit.
The MCP-specific capabilities extend to live inspection. agent-audit inspect stdio -- npx -y @modelcontextprotocol/server-filesystem /tmp attaches to a running MCP server, and the README is explicit that this mode is read-only and never calls tools — an important property when auditing third-party servers you do not fully trust. Config-hook poisoning rules (AGENT-120) check for malicious hooks in .claude/settings.json, .cursor/, and .mcp.json, referencing CVE-2025-59536, which places the tool squarely in the emerging agentic supply-chain defense niche.
Framework awareness is another differentiator: the scanner has specific detection logic for LangChain, CrewAI, AutoGen, and AgentScope, meaning it understands @tool-decorated functions accepting raw strings without validation (AGENT-034) rather than treating them as ordinary Python. There is also an OpenClaw integration published on ClawHub via npx clawhub@latest install agent-audit-scanner, adding seven skill-specific rules (AGENT-058 through AGENT-064) for auditing installed agent skills — a sensible extension point given that skill marketplaces are becoming a distribution vector for malicious prompts.
For teams wanting pipeline integration without hand-rolling YAML, the project ships a GitHub Action at HeadyZhang/agent-audit@v1 with inputs for path, format, severity, fail-on, baseline, and upload-sarif, the last uploading results directly to the GitHub Security tab. The defaults are sensible — sarif output, fail on high — so a two-line workflow addition gives pull-request-level regression detection. Notably, the test badge reports 1924 passing tests with one deliberately red, another small signal of engineering discipline rather than a weekend wrapper around regex.
Caveats worth flagging for an operator: precision at ~69% raw means triage burden is real, especially on codebases with heavy dynamic tool construction; the ground-truth refresh for the 18 newest rules (AGENT-053 and above) is only planned for mid-2026, so newest-rule findings deserve extra human review; and the risk score is a summary heuristic, not a verdict. Used within its stated role — a pre-deploy gate and structured review aid for agent codebases you own or are authorized to assess — agent-audit fills a genuine gap that classic SAST tools structurally cannot, and its reproducible metrics make it possible to hold it to its own claims.
HeadyZhang/agent-audit.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.