TorBot crawls .onion and clearnet sites over Tor, collecting page titles, live-link status and link trees as JSON for authorized OSINT investigations.
| Tool | DedSecInside/TorBot — an OWASP-affiliated open source intelligence crawler for the dark web |
| Category | OSINT / Tor crawling and link-graph analysis (Python CLI) |
| Primary Use | Crawling .onion sites over a local SOCKS5 Tor proxy, gathering titles and host names, and exporting JSON link trees for investigations |
| Safe Use | For authorized investigations, law-enforcement OSINT, research labs, and defensive analysts mapping dark web infrastructure within legal and organizational boundaries |
| Telemetry Note | Crawling generates direct TCP connections through the Tor network to target hidden services; operators leave crawler fingerprints in server-side logs and their own Tor exit/entry traffic is observable by network monitors |
TorBot is a Python-based open source intelligence crawler built specifically for traversing .onion services on the Tor network, and it carries the credibility badge of an OWASP project. Hosted at DedSecInside/TorBot with roughly 5,100 stars and an active dev branch, it positions itself as a CLI-first utility that gathers and analyzes data from Tor sites, returning page titles, host names, live-link status, and the structural relationships between links. What distinguishes it from a generic web spider is the Tor-specific plumbing: it routes requests through a local SOCKS5 proxy, defaults to 127.0.0.1:9050, and understands the quirks of hidden-service resolution that break ordinary crawlers. For an authorized investigator mapping dark web infrastructure, that specialization matters, because generic tooling silently fails on .onion addresses.
Architecturally, TorBot is a breadth-first crawler. The curated feature list explicitly marks Implement BFS Search for webcrawler as complete, which tells you the traversal model: it discovers links on a seed page, enqueues them, and explores outward to a bounded depth controlled by --depth (defaulting to 1). Recent improvements noted in the README show active maintenance of the crawling core — relative links are now resolved against the current page, and malformed or unsupported href values are skipped gracefully rather than crashing the run. That is the kind of unglamorous engineering that separates a research toy from a tool an analyst can leave running against a large onion graph without losing half the dataset to parser exceptions.
Connectivity is deliberately configurable. If a local Tor daemon is running, the default SOCKS5 behavior works out of the box; if not, the operator passes --disable-socks5 and TorBot executes plain HTTP requests, which also makes it usable against clearnet or custom domains — the feature list includes Crawl custom domains as a supported capability. The --host and --port flags let you point at a proxy other than the default 9050, which is useful when Tor runs in a container or on a dedicated relay box within a lab environment. The README links out to a dedicated Tor.md document covering torrc configuration, signaling that the maintainers expect operators to manage their own Tor setup rather than bundling one.
Output is where TorBot shows its investigative orientation. The --save flag writes results to a file in tree or JSON format, and --visualize renders the gathered link relationships as a tree, JSON, or table — the visualization module revamp is checked off in the curated roadmap. The -i/--info mode gathers basic site information, returning the page title or, when no title exists, the host name plus a short description of the site. The liveness check (Check if the link is live) is a small but operationally significant feature, because dark web mirrors churn constantly and a graph full of dead nodes is worse than useless for attribution work.
The most interesting recent addition is what the README calls evidence-first AI analysis. A two-stage workflow first persists a versioned crawl with --save result --result-file crawl-result.json, then a separate torbot analyze subcommand processes that artifact with an LLM provider, producing a cited investigation bundle in a specified output directory. The privacy design is thoughtful: the analyst component defaults to a local Ollama endpoint (the README example uses the qwen3 model), and connecting to any remote OpenAI-compatible provider requires an explicit --allow-remote flag. Every finding must cite captured evidence, and a provider outage never blocks the deterministic evidence bundle from being written — a sensible separation of verifiable facts from model-generated narrative.
Operationally, the CLI surface is compact and coherent. Beyond --url, --depth, --save, and --visualize, there is -v for debug-level logging, -q/--quiet to suppress the header and IP display, --version, and a built-in --update mechanism that pulls the latest stable release. The recent changelog notes that --version and --update now run without requiring a URL, a small usability fix that indicates real usage feedback is shaping development. Setup follows standard Python practice: create a venv, install requirements.txt, then python -m pip install -e ., after which the tool runs as python main.py or via the installed torbot entry point.
There is also an optional desktop application, deliberately kept out of the core. The TorBotApp is a separate Electron project, so Node dependencies never pollute the CLI installation; it is launched with torbot app when checked out as a sibling directory, or located explicitly via TORBOT_APP_DIR or --app-dir. The README points to a docs/DESKTOP_APP.md architecture document describing how the CLI, UI, and backend services interact. This separation reflects a clear design philosophy: the Python CLI is the product, the GUI is an accessory, and neither forces the other's dependency tree on you.
Project health looks solid. GitHub Actions workflows run pytest and flake8 badges, contributions target the dev branch with a pytest gate before pull requests, and the project has participated in Hacktoberfest. The roadmap still lists unfinished items — database persistence of results is marked Not done, and randomized Tor identity, keyword/phrase search, social media integration, screenshot capture, and increased anonymity remain open checkboxes. An operator evaluating TorBot today should know that result persistence beyond file output and automatic circuit rotation are not yet there, which affects both long-running investigations and operational fingerprinting concerns.
From a defensive and tradecraft perspective, several observations are warranted. Crawling hidden services generates direct, patterned requests from your Tor circuits to the targets; onion service operators can log crawler-like behavior, and the roadmap's Randomize Tor Connection item being unchecked means request patterns are more identifiable than a mature scraper. Conversely, defenders running their own decoy .onion services can observe exactly this kind of traversal — bounded-depth BFS with rapid sequential fetches is a recognizable signature in hidden-service logs. The extensive academic reference list in the README, spanning OSINT theory, dark web structure analysis, and illicit-content classification papers, reinforces that this is a research-grade tool intended for documented, lawful intelligence work.
Where TorBot fits in an authorized workflow is narrow but valuable: it is a mapping tool, not an exploitation tool. An analyst handed a set of seed .onion addresses — from a case file, a threat report, or a research dataset — uses it to build the surrounding link graph, verify which services are live, and export a structured JSON artifact that downstream tooling or the local-only analyst component can enrich. Combined with the OWASP affiliation, permissive GNU public licensing, and a maintained Python 3.10+ codebase, it is a reasonable choice for teams that need reproducible, evidence-cited dark web enumeration inside sanctioned investigations rather than ad hoc manual browsing.
DedSecInside/TorBot.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.