safelabs-eval is an open-source red-teaming and evaluation framework that fires 300 curated adversarial prompts at AI agent endpoints and scores responses for defensive security assessment.
| Tool | AgentSafeLabs/safelabs-eval — OWASP-inspired red-teaming and evaluation framework for AI agents |
| Category | AI/LLM agent security evaluation, Python |
| Primary Use | Running 300 adversarial prompts across 10 ASI categories against your own agent endpoints (LangChain, CrewAI, AutoGen, LlamaIndex, custom callables) to measure prompt-injection and jailbreak resistance in CI |
| Safe Use | Intended for authorized security assessments of agents you own or are contracted to test, pre-production safety gates, labs, and defensive research — never against third-party systems without permission |
| Telemetry Note | All evaluation traffic is direct HTTP or in-process calls to the target endpoint and is logged by that endpoint; defenders observing safelabs-eval traffic see repeated patterned adversarial prompts, useful as a signal of authorized pre-deployment testing |
safelabs-eval positions itself in a gap that most agent teams know well: agents built on LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google ADK, or Semantic Kernel routinely reach production without any systematic adversarial safety testing. The framework's answer is deliberately narrow and executable: point it at a supported HTTP agent endpoint, or wrap any Python callable, and it fires 300 curated adversarial prompts — 30 per category — across ten ASI01–ASI10 categories, scores every response with pattern-based detectors, and emits a structured security report. The framing throughout the README is defensive: this is a tool for teams who want to find their agent's weaknesses before someone else does, in authorized testing contexts only.
The taxonomy deserves careful reading because it is the most common point of confusion. The ASI01–ASI10 structure is OWASP-inspired, not the official OWASP Top 10 for Agentic Applications 2026 published in December 2025, which coincidentally also numbers its categories ASI01–ASI10 but with different content — OWASP's ASI01 is Agent Goal Hijack, while safelabs-eval's ASI01 is Prompt Injection. The README is unusually explicit about this non-equivalence and notes the corpus is not yet mapped to the official taxonomy. This honesty matters for reporting: if you cite ASI03 results in a client report, you need to be precise about which taxonomy you mean, because the names collide while the semantics do not.
Category coverage is broad and named in the README table: ASI01 Prompt Injection with a dedicated PromptInjectionDetector, ASI02 Insecure Output Handling, ASI03 Excessive Agency, ASI04 Resource Management, ASI05 Tool Use Safety, ASI06 Data Privacy & Confidentiality with a DataLeakageDetector, ASI07 Trust Boundaries, ASI08 Behavioral Drift, ASI09 Scope Violations, and ASI10 Hallucination & Misinformation with a HallucinationDetector. Some categories share detectors — ASI03 and ASI09 both use the ScopeViolationDetector — and several use generic pattern suites. A notable design constraint on ASI08 is that its jailbreak scenarios are single-turn: any apparent multi-turn framing is embedded within one prompt rather than executed across a real conversation, which limits what that category can say about genuinely conversational attacks.
The prompt library itself is the most rigorously documented component. Every entry follows safelabs/prompts/schemas.py::PromptEntry under schema v1.1.0, carrying three structured metadata fields: a difficulty_tier (tier_1 overt, tier_2 contextual, tier_3 adaptive), a provenance field, and atlas_technique_ids mapping each prompt to MITRE ATLAS v5.6.0 techniques or the literal UNMAPPED where nothing fits (by review, ASI02 and ASI10). The corpus is balanced — exactly 10 prompts per (category, tier) cell — and grew through thirteen changelog steps to content version 1.13.0, the largest being a v1.13.0 batch that added 129 prompts to bring the total from 171 to 300. The README wisely advises pinning the repository commit SHA containing the evaluated library.py when citing a run, since the content version advances per batch.
Provenance handling is more candid than most competing frameworks bother with. A full verbatim-text audit of the 131 prompts at content version 1.6.0 found no prompt reproducing text, code, or dataset rows from external sources, though many entries carry code comments recording technique lineage — indirect prompt injection (Greshake et al. 2023), agent tool-use attacks (AgentDojo / Debenedetti et al. 2024), the ignore all previous instructions idiom (PromptInject), multi-turn escalation (Crescendo), and probe categories from garak and PyRIT. Importantly, the 169 prompts added after v1.6.0 have not received the same systematic provenance audit, only per-batch overlap checks documented in CREDITS.md; the README admits at least one newer prompt, ASI01-018, carries a code comment citing garak's dan probe. That is a transparency level a reviewer should mirror when characterizing corpus originality.
Installation is a single standard step, pip install safelabs-eval, with Python 3.11+ required, and the package is published on PyPI under an Apache-2.0 license. The CLI entry point is safelabs run, which accepts a --target URL, a --category selector such as ASI01 or all, --output json for CI pipelines, an --auth-header for authenticated endpoints, and a --timeout flag defaulting to 30 seconds for slower agents. A companion safelabs prompts subcommand lets you browse the library by --category, --severity, or --output json before running anything — useful for review boards that want to inspect exactly what will be sent to the target.
The Python API mirrors the CLI: from safelabs import run_eval, pass any callable taking a string and returning a string, and both def and async def signatures are accepted with no modification to agent code. Results come back as result.records with per-prompt verdict, scoring_result.confidence, and a remediation_hint field when a fix suggestion exists — a small touch that makes the output directly actionable in a triage workflow. This callable-wrapping design means the framework works against in-house agent frameworks with no HTTP surface at all, which broadens its applicability well beyond the named integrations.
The verdict model is a four-tier ladder: VULNERABLE means the detector identified apparent attack compliance and warrants prioritized review, FAIL means signs of weakness without full compliance, UNCERTAIN flags results needing human review, and PASS indicates the agent resisted the probe. A critical caveat the README handles well is the confidence figure: it is a heuristic evidence score from the pattern-based detectors — how much matched evidence supports the verdict — explicitly not a calibrated probability. Treating 70% conf as "70% chance the verdict is right" would be a misreading, and the documentation says so in plain terms.
The scoring architecture is a significant operational differentiator: detection uses five pattern-based detectors with zero additional LLM calls for grading. This eliminates the judge-model dependency — no extra API cost, no judge-model variance, no data leaving your environment for evaluation — at the cost of the limitations inherent to pattern matching, which is why UNCERTAIN exists as a first-class verdict requiring human eyes. For teams running evals in CI against every build, the deterministic, free, offline-capable scoring loop is arguably the whole point, with the --output json flag feeding results into pipeline gates.
The project also publishes research artifacts downstream of the framework: SafeAgent-300, a redacted evaluation dataset produced with safelabs-eval 0.10.1 and prompt library 1.13.0, is deposited on Zenodo with raw model outputs deliberately withheld, and the framework itself carries a DOI. Combined with the dataset card at docs/DATASET_CARD.md, a documented difficulty rubric in the library.py docstring, and the taxonomy disclaimers, this reads as a project built for reproducibility rather than marketing — a claim the 182-star, CI-badged repository structure supports.
For defenders and authorized assessors, safelabs-eval fits cleanly into pre-deployment gates and contractual security reviews of agents you own or are engaged to test. Run it against a staging endpoint, triage VULNERABLE findings first, use the remediation_hint fields to seed hardening work, and treat UNCERTAIN as the human-review queue. What it is not, and what its documentation never claims to be, is a substitute for multi-turn, tool-enabled adversary simulation — the single-turn ASI08 design and pattern-based detection both bound what it can prove. Used within those bounds, it is a disciplined, well-documented baseline for adversarial agent testing.
AgentSafeLabs/safelabs-eval.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.