api-relay-audit is a Python security scanner that audits locally deployed AI API relays and LLM proxies for prompt injection, model substitution and tool-call rewriting risks during authorized assessments.
| Tool | toby-bridges/api-relay-audit — local security audit tool for AI API relays and LLM proxies |
| Category | Python security scanner for LLM infrastructure |
| Primary Use | Auditing self-owned AI gateways and openai-api/anthropic-compatible relays for prompt-injection, model substitution and tool-call-rewriting weaknesses |
| Safe Use | For authorized assessments of relays you own or operate, lab evaluations and defensive research into LLM proxy hardening |
| Telemetry Note | As a local passive auditor it generates no attacker-side traffic; defenders benefit from its findings as hardening checklists for AI gateway configurations |
The rapid adoption of LLM infrastructure has produced a new class of middleware that most security teams still treat as a black box: the API relay. toby-bridges/api-relay-audit is a Python tool, currently sitting at 862 stars and licensed under AGPL-3.0, that aims to give that middleware layer a proper audit function. Its self-description — a local security audit for AI API relays and LLM proxies that detects prompt injection, model substitution and related weaknesses — positions it squarely in the emerging llm-security niche, where the proxy between your application and the model provider becomes both a convenience and an attack surface.
The tool's GitHub topic list is unusually specific and tells you exactly which threats it models. Alongside the expected prompt-injection, llm-audit and ai-security tags, it flags model-substitution and tool-call-rewriting — two failure modes that are easy to overlook. In a relay architecture, a compromised or misconfigured intermediary could silently swap the model you think you are calling for a cheaper or malicious one, or rewrite the tool calls your agent issues before they reach their destination. The fact that the tool names these explicitly suggests its checks were designed around real relay attack taxonomy rather than generic web scanning.
Model substitution deserves a moment of explanation because it is the subtlest of the threats here. When an application talks to an openai-api or anthropic-compatible endpoint through a relay, it typically trusts the model identifier it sends and the responses it receives back. A hostile relay can accept a request for one model, forward it to a different backend, and return plausible-looking output. api-relay-audit appears to probe for exactly this class of integrity violation, which places it closer to supply-chain auditing than to traditional DAST — a reading reinforced by the supply-chain-security topic on the repository.
Prompt injection detection is the other headline capability. For a relay or proxy in the path, the interesting question is not only whether user input can hijack the model's instructions, but whether content arriving through the relay — system prompt fragments, cached responses, injected headers — can alter downstream behavior. The prompt-injection and llm-proxy topics together suggest the tool examines the relay's handling of these channels rather than only testing raw prompts against a model, which would be the natural design for something that audits infrastructure rather than models.
The presence of both ai-agents and tool-call-rewriting topics points at agentic workflows as a first-class concern. Agents that call tools through a relay — fetching data, executing code, or in the web3 case suggested by web3-wallet and web3-security tags, signing transactions — are catastrophically sensitive to any middleware that can modify the tool call in flight. A relay that rewrites a tool name, mutates arguments, or redirects a destination turns a trusted agent into an attacker's hands. Auditing for that mutation capability is a genuinely useful specialization, and it is the kind of check that generic API gateways scanners simply do not have.
That web3 angle is worth dwelling on because it sharpens the threat model. If an LLM agent is authorized to interact with a wallet — even a hot wallet with limited funds — then the integrity of every tool call in that chain matters in a financial sense. The tool's topics imply it can evaluate whether a relay configuration exposes wallet-touching agent flows to rewriting or substitution risk, making it relevant to teams building trading agents, DeFi assistants or anything that combines ai-agents with on-chain operations. This is documentary analysis, of course: the tool is designed for auditing your own infrastructure, and its findings are most valuable before such an agent ever handles real assets.
From an architecture standpoint, the dsh-plugin topic is the most cryptic signal in the metadata. It suggests api-relay-audit ships as, or integrates with, a plugin for some plugin-capable host framework rather than only as a standalone CLI. For operators, that matters: plugin-form scanners can be dropped into existing security hubs and scheduled alongside other tooling, whereas standalone auditors tend to live in run-once scripts. Either way, the python language tag and security-scanner classification indicate a conventional installable Python package workflow for anyone evaluating it in a lab.
Where does this fit in an authorized workflow? The obvious scenario is pre-deployment review: you stand up a relay — perhaps an openai-api-compatible gateway fronting multiple providers, or an anthropic-keyed proxy with caching — and run the audit against your own configuration before production traffic flows through it. The second scenario is periodic reassessment after relay software updates, since relay projects in the LLM ecosystem change quickly and a patch can silently alter how prompts or tool calls are forwarded. Both uses are defensive by nature; the tool examines infrastructure you control and reports weaknesses in it.
For defenders and blue teams, the telemetry perspective is favorable. A local audit tool generates its findings from configuration and behavioral probes against infrastructure you own, leaving no footprint on third-party systems. Its reports — presumably covering injection exposure, substitution integrity and rewriting risk — double as hardening checklists: fixing each finding closes a gap that an actual attacker would have to exploit through the same relay. That makes api-relay-audit as much a defensive instrument as an offensive assessment aid.
A note on sourcing and honesty: the repository's README was not available in the metadata at the time of writing, so this analysis is derived from the project's own description, topic taxonomy, language, license and star count. The threat categories — prompt-injection, model-substitution, tool-call-rewriting — are all declared by the project itself; the precise mechanisms it uses to detect them, its dependency footprint, and its configuration surface should be verified directly from the repository before reliance. The AGPL-3.0 license is also operationally relevant: anyone embedding it into a commercial SaaS audit pipeline should review copyleft obligations with counsel.
Caveats aside, the tool addresses a real and under-served gap. Most LLM security effort concentrates on the model and the prompt, while the relay layer — the component that sees every request and response and can modify both — receives far less scrutiny. By making substitution and rewriting first-class audit targets alongside injection, api-relay-audit reflects a maturing understanding of where LLM infrastructure actually fails. For teams operating AI gateways in production, adding it to an authorized pre-deployment checklist is a reasonable, low-cost control.
toby-bridges/api-relay-audit.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.