urx is a Rust CLI that pulls historical URLs for a domain from the Wayback Machine, Common Crawl and other archives, then filters and formats them for authorized reconnaissance and bug-bounty workflows.
| Tool | hahwul/urx — extracts and enriches URLs for a domain from OSINT archives such as the Wayback Machine and Common Crawl |
| Category | OSINT / attack-surface reconnaissance CLI |
| Primary Use | Building endpoint inventories and wordlist outputs from archived web history during --scope-file-constrained bug-bounty and internal asset audits |
| Safe Use | Educational and documentary analysis for professionals performing authorized assessments, bug-bounty programs with defined scopes, and defensive asset-inventory research on domains they own or are contracted to test |
| Telemetry Note | Because urx queries public archives rather than the target itself, defenders see little direct signal; archived history it mines (robots.txt, old endpoints, forgotten backup files) is equally visible to attackers, so defenders should monitor their own archived footprint and use --check-status style live-testing with care since those requests do hit the target with the configured --user-agent |
urx, written in Rust by hahwul, addresses a deceptively simple problem that every web-focused operator hits early: given a domain, what URLs have ever existed on it? Rather than brute-forcing paths, it treats OSINT archives as the source of truth, querying the wayback, cc (Common Crawl) and otx providers by default, in parallel, and then applying a deep layer of filtering, normalization and formatting on top. The default provider set is keyless, meaning an operator can get value from a bare urx example.com with zero setup, which is a deliberate design choice that lowers the barrier for legitimate recon work.
The provider architecture is the first thing worth studying. Keyless sources include Wayback Machine, Common Crawl, OTX, Arquivo.pt and anonymous URLScan, while keyed providers — VirusTotal, URLScan (authenticated), ZoomEye, GitHub Code Search and BeVigil (which surfaces URLs extracted from unpacked Android apps) — require API keys passed via flags like --vt-api-key or environment variables such as URX_GITHUB_API_KEY. Notably, keys support rotation: you can repeat the flag or supply comma-separated values, which is a pragmatic answer to rate limits during large authorized engagements. --all-providers enables everything, with keyed providers activating only when a key is present.
A more unusual architectural touch is the --cdx-endpoint option, which lets you point urx at any CDX index server — any pywb, OutbackCDX, or classic Internet-Archive-style CDX API, such as a national web archive. Each endpoint becomes a first-class provider with an id like cdx:<host>, honoring --subs, date ranges and archive filters. The --cdx-dialect flag handles protocol drift by probing whether a server speaks pywb or classic dialect. This turns urx from a fixed multi-source tool into a general CDX client, which matters for regional archives that often hold captures the big players miss.
Filtering is where the README shows its maturity. There are two distinct filter layers: archive-side filters (--from, --to, --archive-status, --archive-mime) that run inside the CDX query itself — the README is explicit that --archive-status "costs no extra requests" because the index applies it — and client-side metadata filters (--meta-*) applied to every provider's capture records. The --archive-mime filter is a smart inclusion: it catches JSON endpoints with no file extension, which extension-based filters structurally cannot. Presets like only-js, only-secrets, only-backup, only-config and only-api encode common operator intents, alongside --match-regex, --filter-regex and substring matching.
Deduplication and parameter analysis get their own tooling. --normalize-url sorts query parameters and strips trailing slashes, --merge-endpoint collapses endpoints sharing a path, and --dedup-similar goes further by collapsing URLs that differ only in variable data such as numeric IDs, UUIDs, hashes and dates. For fuzzing pipelines, --fuzz-placeholder FUZZ rewrites every parameter value into a placeholder and emits one URL per parameter signature — output the README says feeds straight into ffuf or dalfox. --params and --params-by-endpoint produce parameter inventories, with id-looking path segments collapsing to {id}, which is exactly the shape of data a tester needs when reasoning about parameter coverage in scope.
Output flexibility is thorough: plain, json, jsonl, csv and wordlist formats (the latter extracting deduplicated, sorted path segments and parameter names), plus --stream to emit URLs as providers report them rather than buffering the full set. --output-dir writes one file per URL host, which scales well to multi-domain runs. Per-URL capture metadata — first_seen, last_seen, mime, archive_status, digest — gives temporal context that pure URL lists lack, letting an analyst distinguish a decade-old endpoint from one archived last month.
Beyond passive collection, urx optionally goes live: --check-status and --check-title probe discovered URLs, --extract-links and --extract-js-endpoints mine further surfaces, and --archive-body mines links from archived page bodies that no longer exist on the live site, issuing one request per distinct body with --archive-body-dir to retain them. --expand-specs parses OpenAPI 3.x, Swagger 2.0 and GraphQL introspection documents into endpoints, and --archived-discovery reads every distinct archived version of robots.txt and sitemap.xml the Wayback Machine holds — a technique for recovering paths deliberately removed from current files. Importantly, authentication flags like -H, --cookie and --user-agent are documented as applied only to target requests, never to archives, a clean separation of trust boundaries.
Caching is a differentiator at this level of tooling. urx supports a SQLite cache out of the box and Redis with a --features redis-cache build, enabling --incremental scans that avoid re-querying unchanged data across sessions. The urx cache subcommand exposes stats, list, prune, drop <DOMAIN> and clear for maintenance, and --notify webhooks can signal pipeline completion. For teams running recurring surface monitoring on assets they own, this makes urx closer to a scheduled component than a one-shot script.
Installation is well covered across platforms: cargo install urx from crates.io, brew install urx, yay -S urx on AUR, choco install urx on Windows, plus Docker via ghcr.io/hahwul/urx (the image has no entrypoint, so append ./urx), and prebuilt release binaries each accompanied by a .sha256 checksum. The project generates its own shell completions (--completions for bash, zsh, fish, powershell, elvish) and man page via --manpage, small signals of a maintainer who cares about operational polish; hahwul is also the author behind well-known tooling in this space, which lends the project credibility.
From a defensive standpoint, urx is equally useful in the other direction: security teams can run it against their own domains to see exactly what historical surface attackers can enumerate — forgotten backup files, stale config endpoints, deprecated API paths — and prioritize cleanup. Configuration lives in ~/.config/urx/provider-config.toml (or %APPDATA%\urx\ on Windows) with a clear precedence of CLI/env over provider config over main config, and --scope-file plus path-scoped targets like urx example.com/shop keep runs inside contracted boundaries. The tool queries archives, not victims' servers, by default; the only target-touching modes are the opt-in --check-* and --extract-* flags, which an authorized operator should gate on scope agreements.
Within the recon-tooling landscape, urx competes with the classic waybackurls/gau lineage but operates at a noticeably higher abstraction level: more providers, key rotation, archive-side filtering, API-spec expansion, archived-body mining, caching and structured output. The 191-star count understates its utility; the README's density — codecov integration, contribution guidelines, checksummed releases across three OSes — reads as an actively engineered project rather than a weekend script. For professionals assembling an authorized recon pipeline, it is a strong single-binary backbone for the URL-enumeration stage.
hahwul/urx.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.