Scavenger is a Python OSINT crawler that continuously monitors paste sites like pastebin.com, pastes.io, GitHub Gist and dpaste.com for leaked credentials, serving defenders monitoring their organization's exposure.
| Tool | rndinfosecguy/Scavenger — multi-source OSINT bot that scrapes paste sites for leaked credentials and sensitive strings |
| Category | OSINT / credential-leak monitoring crawler |
| Primary Use | Continuously crawling pastebin.com, pastes.io, GitHub Gist and dpaste.com archives to detect email:password pairs and configurable sensitive strings for breach-exposure research |
| Safe Use | Intended for authorized defensive monitoring, OSINT research on your own organization's leaked credentials, and lab-based analysis of public paste content |
| Telemetry Note | The bot generates identifiable scraping traffic against paste-site archives and public APIs; site operators can observe rapid sequential paste fetches, and findings accumulate locally under data/ folders |
Scavenger, at rndinfosecguy/Scavenger, is a Python-based OSINT crawler whose entire purpose is to watch public paste sites for signs of leaked secrets. Written and maintained as an MIT-licensed personal bot (currently version 3.0, with 829 stars), it targets pastebin.com, pastes.io, GitHub Gist and dpaste.com, sweeping their public content for email:password combinations plus a configurable list of sensitive substrings. For defenders and authorized security teams, this is fundamentally a breach-exposure monitoring instrument: the value is in knowing when credentials belonging to your domain, or secrets matching your infrastructure patterns, surface in public dumps before an adversary triages them.
The detection logic is refreshingly simple and worth understanding because it dictates both the strengths and blind spots of the tool. Scavenger always collects email:password pairs regardless of configuration, and beyond that it matches pastes against the contents of configs/searchterms.txt combined with a set of regexes living in classes/utility.py. The default search-term list reads like a highlight reel of accidental secret disclosure: BEGIN RSA PRIVATE KEY, github_pat_, AIza (the Google API key prefix), sk_live_ and rk_live_ (Stripe and Recharge live secret keys), client_secret, and a family of database connection URIs such as mongodb://, postgresql://, mysql://, jdbc:, mongodb+srv:// and redis://.
That default list tells you the tool's threat model is about developer mistake rather than targeted attack. Strings like mysqli_connect(, insert into, Return-Path: and -----BEGIN catch the residues of copy-pasted code, debug output and misconfigured backups. The .onion term flags pastes referencing hidden services, useful as a rough signal for potentially illicit content worth human review. Because matching is plain substring-based, false positives are expected and the workflow assumes an analyst will triage what lands in data/otherSensitivePastes — this is a collection tool, not an analytic engine.
Each source gets a dedicated module selected via numbered command-line switches. For pastebin.com there are two distinct modes: -0 (--pbincom) scrapes the public archive, while -1 (--pbincomTrack) tracks specific users who habitually publish leaks, with targets listed line-by-line in configs/users.txt. For pastes.io, module -2 takes a different approach entirely: rather than walking an archive, it guesses random paste IDs and inspects whatever it hits, which requires supplying an API key in configs/pastesio.json along with an indication of whether you are on the free or pro tier.
The remaining modules cover -3 (--githubgist) for scraping the latest GitHub gists, and -4 (--dpaste) for regularly scraping the dpaste.com public archive. Two more switches are workflow conveniences: -5 opens the search-terms file for editing and -6 edits the tracked-users list, both funneling through the single scavenger.py control script so you rarely need to touch config files by hand. There is also -7, which launches an integrated web dashboard showing statistics about the bot's activity, and the README notes you may want to customize the IP and port it binds.
Setup is straightforward: run setup.sh to install the non-standard Python dependencies, then delete the placeholder README.md files in each subfolder. The README then walks through tuning configs/searchterms.txt — additions are made line by line, and the author explicitly invites contributions of missing terms. That customization path is where a defender gets the most leverage: an organization monitoring for its own exposure can add internal domain fragments, project codenames, or internal hostnames, effectively turning Scavenger into a lightweight, self-hosted leak-alerting service keyed to its own environment.
The storage architecture is documented in useful detail. Every crawled paste lands in data/raw_pastes regardless of whether anything was detected. Pastes containing email:password combinations are additionally copied to data/files_with_passwords, while those matching other sensitive terms go to data/otherSensitivePastes. This three-bucket separation makes downstream triage and automation easy — you can point a separate script at data/files_with_passwords and, in an authorized context, match the email side of each pair against your own user directory to force password resets.
Capacity management is handled automatically: data/raw_pastes accumulates until it holds 48000 files, after which the contents are zipped and moved to an archive folder. This design matters for anyone running the bot long-term, since paste volumes on popular archives are high and unbounded raw storage would quickly become a problem. The zip-and-archive rotation keeps the hot directory performant while preserving historical data for retrospective searches — valuable when investigating when a particular credential first appeared publicly.
Operationally, Scavenger leans on tmux for persistence. When you start the pastebin.com modules with python3 scavenger.py -0 or -1, the control script spawns background tmux sessions (pastebincomArchive, pastebincomTrack) that keep the crawlers running. You interact with them via tmux a -t <session> and detach with Ctrl+b d. Each module can also be invoked directly — pbincomArchiveScrape.py and pbincomTrackUser.py — if you prefer managing processes with your own supervisor, and flags combine, e.g. python3 scavenger.py -0 -1.
From a detection-and-telemetry standpoint, it is worth being clear about what this tool looks like from the other side. Scavenger produces high-volume, sequential scraping traffic against paste-site archives and public APIs; a pastes.io integration is even gated behind an API key, meaning that usage is attributable to your account. Site operators can trivially profile this pattern. Anyone running the bot should respect rate expectations implied by the API tiers, and defenders should note that finding your organization's data via this tool means it is equally available to every other party operating a similar crawler — a sobering but useful framing for incident response prioritization.
The project has drawn community attention: the README links writeups on jakecreps.com covering OSINT collection tools for pastebin and Scavenger specifically, plus a video mention, and the bot publishes findings publicly via a Twitter account (leak_scavenger). That public-feed dimension is interesting for researchers studying what kinds of secrets actually leak — a documentary window into the ecosystem — though anyone replicating it should think carefully about responsible disclosure of what they find rather than amplifying exposure.
In sum, Scavenger occupies a pragmatic niche: it is not a commercial dark-web monitoring service, but a compact, hackable, self-hosted crawler that any authorized security team can point at the paste-site ecosystem within an afternoon. The substring-matching design trades precision for simplicity and total customizability, the tmux-based operation and archive rotation make it viable as an always-on collector, and the modular flag interface keeps each source independently controllable. For credential-exposure monitoring on a budget, or as an educational window into how OSINT leak bots are actually built, it remains a solid reference implementation.
rndinfosecguy/Scavenger.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.