Krawl is a cloud-native deception server that lures scanners and malicious crawlers into fake web applications, then turns their activity into dashboards, banlists, and alerts for authorized defenders.
| Tool | BlessedRebuS/Krawl — customizable cloud-native web deception server and anti-crawler tarpit with a real-time analytics dashboard |
| Category | Web deception / honeypot / blue-team monitoring |
| Primary Use | Deploying decoy web apps (admin panels, fake logins, spider traps) alongside production services to detect, delay, and analyze malicious scanners in authorized environments |
| Safe Use | Intended for defenders running authorized deception programs on infrastructure they own — production shielding via reverse proxy, homelab research, and threat-intelligence gathering |
| Telemetry Note | Generates rich defensive telemetry: captured credentials, attack-type classification (SQLi, XSS, path traversal), TLSH campaign clustering, GeoIP/ASN per IP, weekly published banlist.txt, and webhook forwarding to firewalls |
Krawl, hosted at BlessedRebuS/Krawl and written in Python under an MIT license, is a deception server built for defenders who want visibility into who is actually probing their web estate. Rather than merely blocking scanners, it presents realistic fake attack surface — admin panels, exposed configuration files, fake credentials, and paths advertised in robots.txt — so that automated tooling and curious attackers reveal themselves by interacting with assets that no legitimate user would ever touch. The project has accumulated roughly 739 stars, and its topic list (honeypot, deception, blue-team, kubernetes, grafana, ai-security) signals a defensive-first design philosophy rather than an offensive toolkit.
The core deception primitives are what make Krawl interesting from an architecture standpoint. Spider trap pages generate infinite random links — an idea credited to the spidertrap project — to burn crawler budget and keep hostile spiders circling. Fake login pages mimic recognizable high-value targets like WordPress and phpMyAdmin admin panels. Decoy credentials and API keys look plausible enough to be picked up by credential harvesters, and randomized server headers inject uncertainty about the underlying stack, degrading fingerprint-driven tooling. Random error injection mimics real server flakiness, which is a subtle but smart touch: perfectly clean decoys are a detection signal in themselves.
The most distinctive feature is AI-generated deception pages. Instead of a static set of lures, Krawl can let attacker behavior influence the fake attack surface that gets generated — effectively turning reconnaissance against the reconnoiterer. Generated pages can be managed, exported, and imported from the dashboard's Deception tab, which opens the door to community sharing of lure templates. For a blue-team operator, this shifts honeypot maintenance from a hand-crafted chore toward something that adapts to the scanning campaigns your infrastructure actually receives.
Observability is where Krawl earns its production ambitions. The dashboard is served at a random secret path generated at startup, or a custom one via KRAWL_DASHBOARD_SECRET_PATH, keeping the management plane hidden from anyone enumerating the honeypot itself. Six tabs cover the workflow: Overview for GeoIP mapping and top IPs, User-Agents, and paths; Attacks for credential captures and classification of SQLi, XSS, and path traversal behavior; Threats, which groups payloads into campaigns via TLSH fuzzy hashing so a webshell and its edited variants cluster as one operation; and IP Insight, a forensic deep-dive per address combining ISP/ASN data, behavioral timelines, referer history, and full access logs. Post-authentication tabs add watchlists, banlist management with IPTables export, tarpit review, deception management, and webhook forwarding to firewalls such as CloudFlare.
Deployment architecture is cleanly split into two modes controlled by KRAWL_MODE or the mode key in config.yaml. Standalone mode runs a single container with SQLite in WAL mode and an in-memory Python dictionary cache — zero external dependencies, suited to homelabs and workloads under roughly 500k requests. Scalable mode swaps in PostgreSQL for storage and Redis with multi-tier TTL caching for hot-path performance, enabling horizontal scaling of multiple Krawl replicas behind a load balancer for high-traffic production honeypots. The Helm chart defaults to scalable mode with bundled PostgreSQL and Redis, and images ship via ghcr.io/blessedrebus/krawl.
The simplest authorized deployment is a single docker run command: pull ghcr.io/blessedrebus/krawl:latest, publish port 5000, set KRAWL_DASHBOARD_SECRET_PATH and KRAWL_DASHBOARD_PASSWORD, and mount the krawl-data volume for persistence. For cluster operators, helm install krawl oci://ghcr.io/blessedrebus/krawl-chart --version 2.4.0 -n krawl-system --create-namespace with overrides for the database, Redis, and dashboard passwords brings up the full scalable stack. Both paths are documented in the README, and production-ready compose files live in the docker/ directory — note the README's own caution that example compose files use default passwords that must be changed before production.
A community-intelligence layer sets Krawl apart from single-tenant honeypots. It maintains a banlist.txt of IPs that triggered its traps, published weekly and downloadable even by people who don't run the tool. More interesting is federation: each instance can expose its banlist on an unauthenticated path via banlist.export_path and pull plain-text lists from other instances through banlist.sources, merged into local ban decisions on a refresh_interval (defaulting to 3600 seconds in the example). This turns independent honeypots into a loosely coupled blocklist mesh — though as with any community blocklist, operators should review merge behavior and false-positive risk before automating enforcement against it.
Operationally, Krawl is designed to sit beside real services rather than replace them. The README describes exposing it through a reverse proxy so legitimate applications are shielded while crawlers and malicious users get tarpitted and profiled; a docs/reverse-proxy.md file covers CDN and proxy scenarios where client-IP handling matters. The tarpit concept — holding hostile connections open to waste their resources — combined with spider traps means the tool doesn't just observe, it actively imposes cost, helping distinguish patient malicious automation from benign crawlers like search engine bots.
For defenders considering adoption, the value proposition is high-fidelity, low-noise detection: anything that submits credentials to a fake phpMyAdmin page or crawls honeypot paths from robots.txt is presumptively hostile, cutting through the alert fatigue of signature-based web monitoring. Watch items include the usual honeypot hygiene concerns — isolation from production networks, careful handling of captured malicious uploads indexed in the Threats tab, and disciplined dashboard secret management. Within those bounds, Krawl reads as a mature, well-documented deception platform that fits naturally into an authorized blue-team program, whether as a standalone homelab experiment or a scaled Kubernetes deployment feeding firewall blocklists and webhook alerting.
BlessedRebuS/Krawl.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.