TorCrawl.py is a Python script for anonymous web scraping and crawling through the TOR network, aimed at OSINT and authorized research workflows.
| Tool | MikeMeliz/TorCrawl.py — a Python crawler and page extractor that routes requests through the TOR network via a local SOCKS5 proxy |
| Category | Web crawling and content extraction over anonymity networks |
| Primary Use | Collecting and archiving webpage content and link structures, including .onion services, during authorized OSINT investigations and research |
| Safe Use | Educational and documentary analysis for authorized professionals: use only against systems and sites you are permitted to crawl, in labs, sanctioned investigations, or with respect for site terms and robots.txt |
| Telemetry Note | From a defender's perspective, its traffic appears as ordinary TOR exit-node requests; the -rua user-agent rotation and -p pacing exist to vary request fingerprints, while on the operator side it can leave links.txt, JSON/XML exports, a SQLite database, and a -l log of visited URLs and response codes |
TorCrawl.py, hosted at MikeMeliz/TorCrawl.py with around 544 stars and a GPL-3.0 license, is a Python script built around a single idea: anonymous retrieval of web content should not require an elaborate toolchain. The script routes its HTTP requests through a local TOR circuit — by default a SOCKS5 proxy at 127.0.0.1:9050, the standard port of a stock tor daemon — and wraps both one-shot page extraction and recursive link crawling behind a compact argument interface. It is written in Python 3, is packaged on PyPi as torcrawl, and targets an audience the README describes bluntly: terminal operators who want clean output that can be piped into other tools.
The architecture, as far as the README reveals, is deliberately thin. There is no browser automation and no JavaScript rendering engine; TorCrawl.py fetches raw markup and works with it directly. That choice matters for operators because it makes behavior predictable — what you get is the served HTML, not a post-execution DOM — and it keeps the dependency surface small. The two core modes are extraction (-e) and crawling (-c), and they compose: a crawl at depth -d 2 followed by -e will walk the discovered links and dump each page into a structured output folder, all in one invocation.
Extraction mode is where the Unix philosophy shows through. A single command such as torcrawl -u http://www.example.com/ | grep 'some-string' fetches a page through TOR and streams the markup to stdout, letting grep, jq, or anything else do the downstream work. Output can be redirected to a file with -o, and a batch of URLs can be fed in from a line-separated list with -i. The README is careful to note that URLs supplied as .onion addresses work exactly like regular ones, which is the whole point of routing through the network in the first place.
Crawling mode discovers internal links and writes them to links.txt by default. Depth is controlled with -d (default 1), and -p inserts a configurable pause in seconds between requests — the changelog notes the previous one-second default pause was removed, so pacing is now opt-in, which is worth knowing for anyone concerned about hammering a small hidden service. Verbose mode -v produces step-by-step counts and echoes the exit IP as seen through the circuit, a useful sanity check that routing is actually in effect.
Two features added in the v1.33 and v1.35 releases round out the operational picture. First, -rua enables random User-Agent rotation, and -rpr enables random proxy rotation from res/proxies.txt — though notably the README states the proxy option requires -w, meaning direct, non-TOR operation. Second, output formatting grew beyond plain text: -j exports findings as JSON, -x as XML, and -DB writes crawl results plus the link graph into a SQLite database, from which -vis generates an interactive HTML visualization using PyVis. That database-plus-graph pipeline quietly turns a scraper into a site-mapping instrument.
The keyword search capability is where TorCrawl.py leans closest to analyst tooling. The -y flag performs a yara-based keyword search over retrieved content, with two modes: -y h (or -y 0) matches against the entire raw HTML object, while -y t (or -y 1) restricts matching to visible text. The rules live in res/keyword.yar, and the README explicitly instructs users to edit that file for their own terms. Repurposing yara — a pattern engine most defenders associate with malware triage — as a page-content filter is an unusual design choice and arguably the tool's most distinctive internal trick.
Installation paths are straightforward: pip install torcrawl from PyPi, or a manual git clone https://github.com/MikeMeliz/TorCrawl.py.git followed by pip install -r requirements.txt. The harder prerequisite is the TOR service itself, and the README walks through it per platform: apt-get install tor and service tor start on Debian/Ubuntu, tor.exe --service install on Windows, brew install tor with brew services start tor on macOS. The -w flag bypasses TOR entirely for direct fetching, and -px/-pr let you point at any alternative SOCKS5 endpoint instead of the defaults.
Operationally, the tool sits in a narrow but real niche. Scrapy or wget give you industrial crawling but no anonymity layer; raw curl --socks5-hostname 127.0.0.1:9050 gives you anonymity but no link discovery, output structuring, or keyword matching. TorCrawl.py is essentially that curl incantation plus a minimal crawler, a formatter, and a yara filter, which is precisely why it appeals to OSINT practitioners cataloguing .onion content or verifying what a hidden service serves — activities that are documentational rather than intrusive.
The README deserves credit for an explicit ethics note. It reminds users that crawling is not illegal but violating copyright is, that sites signal their preferences via robots.txt, and that although the crawler can technically ignore it, the author recommends respecting it. That framing aligns the tool with responsible research practice rather than collection-at-any-cost, and reviewers of the project consistently highlight this section as unusually candid for the category.
From a defensive standpoint, there is little here that should alarm a blue team. The tool fetches pages like any other client; there is no exploit logic, no auth bypass, and no payload generation. What it does produce is evidence of its own activity on the operator's machine: links.txt, per-site extraction folders, optional JSON/XML exports, a SQLite artifact, and a -l log recording visited URLs and their response codes. The -vis HTML graph is likewise a local artifact. In authorized contexts — sanctioned investigations, lab mirroring of one's own hidden services, archival research — those artifacts double as documentation of scope and methodology, which is exactly what an audit trail should look like.
Caveats worth flagging: the project's own shields show it is a small maintainer effort, quality is tracked through SonarCloud, and the changelog history (v1.2 through v1.35) shows steady but unhurried maintenance. Anyone relying on it for serious collection should pin a release, review requirements.txt, and remember that anonymity here is only as strong as the underlying tor configuration — a stock daemon without hardened circuits is a starting point, not a guarantee. Used within its lane, TorCrawl.py remains one of the simplest defensible answers to the question of how to look at .onion content responsibly.
MikeMeliz/TorCrawl.py.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.