AgentPoison is a research framework for poisoning the long-term memory of retrieval-augmented LLM agents, letting authorized red teams measure how memory corruption steers agent behavior in controlled evaluations.
| Tool | AI-secure/AgentPoison — official NeurIPS 2024 implementation for red-teaming LLM agents via poisoning of memory and RAG corpora |
| Category | AI/ML security research framework (Python) |
| Primary Use | Controlled evaluation of how poisoned RAG memory backdoors steer LLM agent planning and tool selection in authorized research settings |
| Safe Use | Academic red-teaming and defensive research on self-owned agent stacks; evaluating robustness of retrieval memory before deployment in production assistants |
| Telemetry Note | As a research tool it operates entirely within the operator's own evaluation pipeline; defenders benefit from its outputs — detection benchmarks for anomalous retrieval hits and poisoned corpus entries in RAG stores |
AgentPoison is the official implementation accompanying the NeurIPS 2024 paper "AgentPoison: Red-teaming LLM Agents via Memory," published by the AI-secure organization. The repository positions itself squarely in the emerging subfield of LLM agent security: rather than attacking the model itself or the prompt directly, it studies how an adversary who can influence the agent's long-term memory — typically the vector store backing a retrieval-augmented generation pipeline — can implant triggers that steer the agent's downstream planning and tool-selection behavior. With roughly 250 stars, a Python codebase, and an MIT license, it reads as a serious academic artifact rather than a throwaway proof of concept.
The core insight the project builds on is architectural. Modern agents — the ReAct-style planners, autonomous web agents, and RAG-backed assistants that increasingly sit in front of enterprise data — do not rely on the raw weights of the underlying LLM alone. They consult retrieved context: memory banks of past trajectories, document corpora chunked and embedded into a vector database, and few-shot exemplars pulled at inference time. AgentPoison treats that retrieval layer as the attack surface, which is a materially different threat model from jailbreaking or direct prompt injection, and one that defenders have historically instrumented poorly.
What distinguishes the approach, per the paper's framing reflected in the repo, is the use of a poisoning optimizer to craft the injected memory entries. Naive poisoning of a corpus often fails for a subtle reason: a malicious entry that is only retrieved when a trigger phrase appears may not rank highly enough for general queries, while an entry that ranks highly always will degrade benign behavior and expose the implant. The method optimizes trigger-embedded poison samples so that they are preferentially retrieved when the trigger is present and rarely otherwise — a dual objective that keeps the backdoor both effective and quiet. That optimization is the technical heart of the framework.
For an authorized operator, the practical value is as an evaluation harness. The repository ships with Python code intended to reproduce the paper's experiments across multiple agent benchmarks — the README in the upstream project was not populated in the metadata available at press time, but the repo topics (llm-agent, red-team, retrieval-augmented-generation) and the paper's methodology make the scope clear. An operator clones the code, points it at a supported agent evaluation task, trains the poison samples against the target pipeline's embedder and retriever, and then measures success and side effects — how often the agent is diverted to the attacker-chosen outcome, and how much benign task performance degrades.
The threat model deserves emphasis because it is more realistic than it first sounds. Anyone who can contribute documents to a knowledge base — a wiki editor, a support-forum poster whose content gets scraped into a corpus, a third-party data feed, or an insider with write access to the vector store — satisfies the attacker precondition without ever touching the model, the API keys, or the serving infrastructure. Poisoned memory also survives prompt-level mitigations: system-prompt hardening, guardrail filters, and model swaps do not remove a backdoor living in retrieved context. That persistence is exactly why red teams need tooling that models it.
From a defensive research perspective, AgentPoison is best understood as a stress test for retrieval integrity. Its outputs give defenders concrete failure cases: retrieval logs showing an implanted entry surfacing only under trigger conditions, planning traces where the agent's tool calls pivot to the attacker's objective, and measurable deltas in benign accuracy that quantify the stealth/effectiveness trade-off. Those artifacts translate directly into detection engineering — anomaly scoring on corpus contributions, monitoring for documents whose retrieval distribution is sharply conditional on specific token sequences, and regression suites that replay agent trajectories after any corpus update.
Operationally, the framework is a standard Python research repo: MIT-licensed, living on the master branch, with the usual expectations of configuration-driven experiments rather than a packaged CLI product. An operator would typically git clone https://github.com/AI-secure/AgentPoison and work through the experiment entry points and configuration files as documented in the paper's reproduction instructions. Because the poisoned samples are optimized for a specific embedding model and retrieval configuration, results transfer only when the evaluation target matches the harness assumptions — a point anyone using it for a security assessment of their own agent stack should verify up front.
Where this fits in the broader landscape is worth noting. Prompt-injection tooling has proliferated over the past two years, but memory poisoning remains comparatively underexplored, and AgentPoison's NeurIPS pedigree makes it a reference implementation for the subfield. Adjacent work covers poisoning of RAG corpora for open-domain QA, but the focus here on agentic behavior — steering which tools an agent calls and which plans it executes — is the differentiator, and the one with the sharpest enterprise implications as agents gain write access to real systems.
The limitations are the flip side of the contribution. The optimizer assumes visibility into the target pipeline's embedding and retrieval mechanics, so fully black-box application against a production agent is out of scope of the published method. Effects are also benchmark- and model-dependent; a successful diversion on one agent framework does not certify the same on another. Responsible use therefore means running it against pipelines you own or are contractually authorized to assess, which is precisely the deployment pattern the paper's authors intend.
For teams building or deploying RAG-backed agents, the actionable takeaway is procedural: treat the corpus as part of the trusted computing base. Contributed-document review, provenance tracking on ingested content, periodic canary queries that check whether retrieval distributions shift unexpectedly, and post-update agent regression tests are the mitigations this line of research motivates. AgentPoison supplies the adversarial ground truth that makes those controls testable rather than aspirational — which is, in the end, what good red-team tooling is for.
AI-secure/AgentPoison.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.