AgentStalker is an end-to-end audit framework that models LLM agents as static taint graphs, synthesizes contextualized attacks, and replays them in a Docker sandbox for authorized security assessments.
| Tool | Gach0ng/AgentStalker — four-stage static-plus-dynamic audit kernel for LLM agents |
| Category | LLM agent security auditing / taint analysis / sandbox verification |
| Primary Use | Auditing agent codebases you own: quick mode as a CI gate, standard pre-release audits, deep red-team preparation with sandbox replay |
| Safe Use | Educational and documentary analysis for authorized professionals running assessments against their own agent code in isolated lab sandboxes with mock services |
| Telemetry Note | All dynamic verification runs inside an instrumented Docker sandbox with an nginx front, a LiteLLM proxy for the LLM backend, mock services, and a Tracee eBPF monitor — evidence is JSON artifacts under evidence/, leaving the repo self-contained rather than touching external systems |
AgentStalker positions itself with a clear philosophical stance: treat an LLM agent as a system to be audited, not a model to be aligned. The name is a compression of its architecture — Agent Static + Attack-graph + Live-replay Kernel — and the repo, written in Python 3.11+, delivers exactly that: a four-stage pipeline labeled MODEL → ATTACK → VERIFY → REPORT, connected end to end by a typed taint graph. What makes the design notable is that every stage emits a JSON artifact (agent_model.json, attack_graph.json, evidence/*.json, audit_report.md) that the next stage consumes, which means stages are independently swappable and the whole audit is reviewable after the fact.
A critical architectural decision, stated explicitly in the README, is that the framework does not attempt to replace an orchestrating LLM such as Claude Code. Instead it provides deterministic tools, YAML facts, and Jinja2 templates that an LLM orchestrator composes into an audit run. This keeps the underlying machinery auditable while leaving the orchestrator's decisions reviewable — a sensible separation of concerns for a security tool, since the last thing you want in an audit framework is a nondeterministic component you cannot inspect at the core of the pipeline.
Stage 1 (MODEL) is where static analysis happens. The core/ module contains ast_extractor.py and ast_extractor_rust.py alongside taint_tracker.py and taint_tracker_rust.py, supporting dual-stack Python and Rust agent codebases with auto-detection from pyproject.toml or Cargo.toml. The analyzer extracts a typed taint graph with tagged sources (USER_INPUT, RAG_CONTEXT, MCP_RESPONSE, MEMORY_READ, TOOL_RESULT, WEB_FETCH, FILE_CONTENT, SYSTEM_PROMPT) and tagged sinks (TOOL_CALL, SQL_QUERY, SHELL_CMD, HTTP_OUT, PROMPT, FILE_WRITE), with propagation rules covering concatenation, decoding across base64/URL/HTML/Unicode, and structured-field extraction.
The pattern library in core/agent_patterns.yaml encodes high-confidence detectors with explicit confidence scores: R-CMDI-001 for shell-command concat sinks at 0.95, R-SQLI-001 for SQL string-concat sinks at 0.85, R-RAG-POISON-001 for RAG-to-prompt pollution at 0.80, and a trio of MCP-specific rules — tool-name shadowing (R-MCP-SQUAT-001), description poisoning (R-MCP-DESC-001), and token passthrough (R-MCP-TOKEN-001). Rust-specific rules include instruction-file loading from files like AGENTS.md and CLAUDE.md plus instruction/memory markdown patterns, and approval-mode bypass via serde_yaml::from_str deserialization sinks — a genuinely specific and modern detection surface.
Stage 2 (ATTACK) is where the tool departs from a classic SAST engine. The payloads/ directory contains fourteen payload categories plus ten multi-turn attack chains, and the key design idea is that payloads are not bare PoCs. Each entry in payloads/*.yaml carries three metadata blocks: first_pass (how to statically filter candidates before replay), detection (runtime success markers such as response codes, log keywords, and timing), and sandbox_monitoring (which monitoring layers to enable). This metadata is what contextualizes the same injection payload differently when it targets a LangChain query_db tool versus an AutoGen db_exec call — the payload adapts to the tool surface it is aimed at.
Stage 3 (VERIFY) runs the attack graph against a live LLM backend inside an instrumented seven-container Docker sandbox: agent-under-test, an LLM proxy via LiteLLM, mock-db, mock-mail, mock-api, an eBPF monitor through Tracee, and nginx, with an OPA policy layer enforcing tool whitelists, SSRF-to-metadata blocking, and sensitive-path access controls. The use of mock services is important for authorized use — dynamic verification never needs to touch real infrastructure, and every confirmed finding carries a replayable evidence file. The sandbox/ module also includes ten framework adapters, executors for API/CLI/MCP/Web/Replay modes, and eight monitoring layers including one dedicated to MCP.
Stage 4 (REPORT) is a deterministic verdict engine with LLM fallback: eleven rules (R001–R011) cover roughly 95 percent of high-confidence cases, and unmatched evidence falls through to an LLM-as-judge. This is a pragmatic hybrid — deterministic rules where they suffice, probabilistic judgment only at the margin — and it mirrors the tool's approach to taint itself. The semantic_taint.py engine models LLM hops as probabilistic propagators rather than boolean ones, with each flow carrying a cumulative confidence. The stated goal is reducing the everything-through-an-LLM-alarms false-positive pattern while preserving the boolean is_exploitable contract for downstream consumers.
The dedicated MCP audit module deserves its own paragraph. It performs static extraction of tool names, descriptions, and transport (stdio/sse/http), runs three detectors — tool squatting, description poisoning, and token passthrough — and verifies at runtime via MCPMonitor with VerdictEngine rules R009–R011. The README claims this closes the OWASP ASI04 gap most agent-audit tools leave open, and the badges map coverage against OWASP ASI01–ASI10. The seven-layer attack-surface model (user input, context/memory, tool call, MCP/plugin, identity/permission, multi-agent, observability) gives the whole audit a coherent taxonomy to organize findings against.
The project backs its claims with a documented end-to-end run against a real target: CodeWhale v0.8.52, a Rust agent CLI of 16 crates and 332 .rs files, audited with a live DeepSeek API backend and Docker isolation. The reported results — 10 test cases covering ASI01/02/04/05/06/09, 7 confirmed exploitable findings for a 70 percent attack success rate, 4 critical and 2 high vulnerabilities, 12 MCP static findings, roughly 8 minutes of dynamic runtime with no mocks — are backed by replayable evidence files. The vulnerability classes read as a taxonomy of contemporary agent failure modes: persistent privilege escalation via config rewrite, human-in-the-loop bypass, direct prompt injection, SSH private-key exfiltration, memory poisoning, multi-turn social engineering, and overly coarse MCP allowlists.
What earns the project credibility is the unusually honest capability-boundaries section. The Rust AST extractor resolves tool definitions less precisely than the Python one. The fast taint tracker (source body plus up to five callers) trades coverage for speed, while the full BFS callgraph does not complete in reasonable time on 300+ files — the README calls this the core precision/performance tension being worked on. The semantic taint engine ships as a skeleton; the eBPF breakpoint and static-dynamic bridge are roadmap items requiring Linux, Tracee, and a real agent runtime to validate. Runtime MCP verification only fires when the agent actually registers an MCP server, OAuth scope and session hijack detectors are documented but not implemented, sandbox container hardening is out of scope, and a standardized 100-agent benchmark does not yet exist.
Operationally, the tool offers three modes matched to different authorized workflows. quick mode is static-only at 5-10 minutes, intended as a CI gate; standard adds the attack graph for pre-release audits at 30-60 minutes; deep runs the full pipeline with sandbox replay for red-team preparation, taking hours and requiring Docker Engine 24+, rootless eBPF with Tracee 0.8+, and an LLM API key. One operational caveat stands out: mode cannot be downgraded mid-run, and if the runtime budget is insufficient the orchestrator extends the attack plan rather than reducing coverage. The test suite — 76 tests, runnable with pytest tests/ -q — needs no Docker, eBPF, or API key, which lowers the barrier for anyone evaluating the static core before committing to sandbox infrastructure.
For professionals building or assessing LLM agents, AgentStalker is best understood as a documentary and research instrument: it formalizes the agent attack surface into typed taint flows, contextualized payloads, and deterministic verdicts, then proves exploitability in an isolated sandbox with mock services rather than against live systems. Its honest boundary statements, replayable evidence model, and dual Python/Rust support make it a serious contribution to the emerging agent-security tooling space — one whose v2 roadmap, particularly the standardized benchmark and the static-dynamic bridge, will be worth watching.
Gach0ng/AgentStalker.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.