spiderfoot for automating OSINT reconnaissance and attack surface mapping

spiderfoot automates open source intelligence collection across 200+ data source modules, correlating results for use in authorized penetration tests, red team reconnaissance, and defensive attack surface monitoring.

Tool	smicallef/spiderfoot — OSINT automation platform integrating 200+ modules for threat intelligence gathering and attack surface mapping
Category	OSINT automation / reconnaissance framework
Primary Use	Automated enumeration and correlation of publicly available data about domains, IPs, e-mails, usernames and organizations during authorized assessments or self-audits
Safe Use	Intended for authorized penetration tests, red team engagements with defined scope, and defensive review of your own organization's external exposure
Telemetry Note	Many modules issue outbound queries to third-party APIs (SHODAN, Censys, crt.sh) and some perform active scanning; targets and their log owners will see these lookups, and API providers log every request against your keys

spiderfoot occupies a unique position in the reconnaissance ecosystem: rather than being yet another scanner, it is an orchestration layer that fuses together more than two hundred data sources — from crt.sh certificate transparency logs to SHODAN, HaveIBeenPwned, GreyNoise, BinaryEdge and Censys — and then applies a correlation engine on top of the results. Written in Python 3.7+ and MIT-licensed, it has been actively developed since 2012, which the README itself calls out as a notable longevity milestone. With roughly 22,000 GitHub stars and a stable v4.0 release, it is one of the most battle-tested open source intelligence frameworks available to authorized assessors and defensive teams alike. The project frames its own dual use clearly: offensive reconnaissance during scoped red team or penetration test work, and defensive discovery of what you or your organization have inadvertently exposed on the internet.

Architecture-wise, the interesting design decision is the publisher/subscriber model that connects the modules. Each module's output — a discovered sub-domain, an e-mail address, a Bitcoin wallet — becomes an event that other modules can subscribe to and act upon, recursively expanding the dataset until the graph of entities saturates. This is what turns spiderfoot from a scripted list of API calls into genuine intelligence automation: a domain scan can cascade into sub-domain enumeration, certificate transparency mining, web scraping, e-mail and human name extraction, and then pivot on those e-mails into breach database lookups. The README explicitly frames this as ensuring maximum data extraction, and in practice it is the reason scans grow non-linearly in both runtime and value.

Version 4.0 introduced what is arguably the project's most significant feature: a YAML-configurable correlation engine with 37 pre-defined rules shipped in the correlations/ directory. The documentation points to a template.yaml walkthrough, and the existing rules are described as readable enough to serve as starting points for custom logic. This matters because raw OSINT output is usually a firehose of loosely related facts; correlation rules convert that into signal, for example linking an e-mail found in a breach to a domain registration to a hosting IP. For an analyst doing attack surface mapping, this is the difference between a spreadsheet and an intelligence product. The rule format being YAML means teams can encode organizational knowledge — what counts as an interesting relationship for their environment — without writing Python.

The target model is broad. A scan can be launched against an IP address, a domain or sub-domain, a hostname, a CIDR network subnet, an ASN, an e-mail address, a phone number, a username, a person's name, or even a Bitcoin address. That entity-agnostic design is unusual and is what enables pivoting workflows: start with a single corporate domain, and the tool will branch into infrastructure, identities, and cryptocurrency artifacts associated with the target. For defenders, running it against your own organization's domains is a sobering exercise in seeing how identity and infrastructure data chains together from entirely public sources.

The module catalog is the heart of the tool and the README documents it extensively, including whether each integration needs an API key. Notably, most modules do not require keys at all, and many that do have free tiers — a deliberate accessibility choice. The internal modules that need no external service at all include an Account Finder that checks for associated accounts across over 500 social and other websites, a Base64 Decoder, a Binary String Extractor, and extractors for cookies, company names, country names, credit card patterns, and cryptocurrency addresses. Cloud storage exposure is covered by dedicated finders for Amazon S3 buckets, Azure blobs, and DigitalOcean storage, each of which searches for candidate containers associated with the target and attempts to list their contents — a common source of accidental disclosure that defenders should proactively check on their own estates.

Reputation and threat intelligence modules form another major cluster: abuse.ch, AbuseIPDB, AlienVault IP Reputation, AlienVault OTX, blocklist.de, BitcoinAbuse, and CINS Army among them. These are equally useful from the defensive seat, letting a blue team check whether their own netblocks or hosting appear on public blacklists — sometimes the first indication of a compromise. Historical research is covered through Archive.org Wayback Machine lookups, which identify historic versions of interesting files and pages, and CommonCrawl URL discovery. Certificate transparency via crt.sh and CertSpotter provides sub-domain discovery that is passive from the target's perspective, since it queries public CT logs rather than the target's infrastructure.

Operationally, spiderfoot ships with an embedded web server providing a browser-based UI, but can be driven entirely from the command line. Installation follows the standard pattern: grab a packaged release or git clone the repository, install dependencies with pip3 install -r requirements.txt, and launch with python3 ./sf.py -l 127.0.0.1:5001. The README sensibly recommends packaged releases over master, since the development branch carries bleeding-edge modules that are not fully tested. A Dockerfile is included for containerized deployments, and TOR integration is built in for dark web searching — with the Ahmia search engine module covering hidden-service mentions of a target. Note the deliberate binding to 127.0.0.1 in the documented invocation: the web UI has no authentication in the open source edition, so leaving it on a routable interface would expose your scan data and API keys to anyone on the network.

The tool can also act as a meta-orchestrator, calling out to established external utilities such as DNSTwist, Whatweb, Nmap and CMSeeK. This is worth pausing on from both a capability and a footprint perspective: some of those child tools are active, meaning scans are not purely passive. Passive collection dominates the module list, but port scanning, banner grabbing, DNS zone transfer attempts, and web probing modules exist and will touch the target's infrastructure directly. Anyone running spiderfoot against systems must ensure that active modules are within their authorization scope, and conversely defenders should understand that a spiderfoot scan may manifest as straightforward service probes indistinguishable from other reconnaissance traffic.

Data persistence is handled through a SQLite back-end, which enables custom querying of results after a scan completes, and export is supported in CSV, JSON and GEXF formats — the latter being the graph format used by tools like Gephi, which pairs naturally with the entity-relationship data spiderfoot produces. Visualizations are built in, and API keys can be exported and imported, a small but genuinely practical feature when rebuilding a testing VM or syncing configuration across a team. The scan database being local SQLite also means results stay on your machine, an important consideration when handling client-adjacent data during engagements.

The README is transparent about the commercial sibling, SpiderFoot HX, a cloud-managed version adding attack surface monitoring with change notifications via e-mail, REST and Slack, multi-target scans, multi-user collaboration with authentication and 2FA, screenshotting, and integrations that feed scan data into Splunk and ElasticSearch. The open source core remains fully functional and MIT-licensed, but the split is worth knowing: continuous monitoring — arguably the most valuable defensive use case, where you re-scan your estate on a schedule and diff the results — is where the commercial product positions itself. Open source users can approximate this by scheduling scans and diffing SQLite exports themselves.

From a defensive tradecraft standpoint, spiderfoot is best understood as a mirror: everything it finds was already public, held by third parties, or observable by anyone. Running it against your own organization before an adversary does is the canonical blue-team use, with the cloud bucket finders, sub-domain hijacking susceptibility checks, and breach data lookups being the highest-yield starting points. The sub-domain hijacking module specifically checks whether dangling DNS records point at services that could be re-registered — a class of vulnerability that is trivially found with OSINT and catastrophically effective when exploited.

The community footprint reinforces the project's health: a Discord server for support and feature requests, a Twitter presence, CI tests with visible Codecov coverage, and documented ascinema recordings walking through individual capabilities like sub-domain enumeration, e-mail extraction, and threat intelligence queries. For professionals who want a single orchestration point over the fragmented OSINT landscape — and who operate strictly within authorization — spiderfoot remains the reference implementation, and the 4.0 correlation engine makes its output genuinely analyst-grade rather than a raw data dump.

Official project repository for smicallef/spiderfoot.

Download Tool

Educational analysis for authorized security professionals. Use only in controlled, authorized environments.