Yelp's threat_intel bundles Python wrappers for the Umbrella Investigate, VirusTotal v2.0, and ShadowServer APIs, letting defenders enrich domains, IPs, URLs, and file hashes from one library.
| Tool | Yelp/threat_intel — Python wrappers for Umbrella Investigate, VirusTotal API v2.0, and ShadowServer APIs |
| Category | Threat intelligence / API client library |
| Primary Use | Batch enrichment of domains, IPs, URLs, and file hashes against commercial and community threat feeds during incident triage and authorized assessments |
| Safe Use | Designed for defensive analysts, SOC teams, and authorized security assessments doing IOC enrichment and research on artifacts from their own environments |
| Telemetry Note | Purely a client library — it sends queries to third-party APIs with your key, so every artifact you submit is visible to the feed provider; local JSON caching via cache_file_name keeps results on disk, which is itself a data-handling consideration |
threat_intel is Yelp's in-house answer to a problem every incident responder knows intimately: enrichment sprawl. When you are triaging a cluster of suspicious domains, a handful of IPs, and a stack of file hashes, you end up juggling half a dozen vendor APIs, each with its own auth scheme, rate limits, and response shapes. Rather than reinventing that plumbing per investigation, threat_intel packages three of the most commonly consulted feeds — Cisco Umbrella Investigate, VirusTotal API v2.0, and ShadowServer — behind clean, batch-friendly Python classes. It is worth stating up front that this is a defensive library through and through; it consumes threat intelligence, it does not generate attacks.
The Umbrella side is exposed through the InvestigateApi class, importable from threat_intel.opendns. Initialization is exactly what you would expect: pass your API key as the constructor argument. The genuinely thoughtful bit is the optional cache_file_name parameter, which transparently persists API responses to a local JSON file so repeated lookups on the same domains or IPs during a long investigation do not burn your query quota or your bandwidth. For a SOC that iterates over the same indicator list across multiple analysts and scripts, that cache alone justifies the dependency.
The breadth of the Investigate wrapper is the library's strongest selling point. Beyond the obvious categorization(domains) call — which returns Umbrella content and security categories plus a -1/0/1 status score where -1 flags malicious — you get security(domains) for DGA and RIP scores, cooccurrences(domains) for domains accessed by the same users in a tight time window, and related_domains(domains) for infrastructure frequently requested within a 60-second window. Each of these maps one-to-one to a documented Investigate endpoint such as domains/categorization/?showLabels, security/name/, and links/name/, which makes the wrapper easy to reason about if you already know the upstream API.
Pivot capability is where the design shows its operator pedigree. The rr_history(ips) method hits the dnsdb/ip/a/ endpoint to enumerate domains resolving to a given IP, while dns_rr(domains) walks DNS resource record history from the domain side, returning geo statistics, TTL means, and flags like is_subdomain. There is a full WHOIS suite too: whois_domains(domains), whois_emails(emails), whois_nameservers(nameservers), and whois_domains_history(domains) for registrant pivoting across time. Anyone who has traced a phishing kit's registration footprint by registrar email will recognize how much friction this removes. Note that the email and nameserver endpoints cap out at 500 results per query, which the README is candid about.
The VirusTotal wrapper, VirusTotalApi from threat_intel.virustotal, covers report retrieval for domains via get_domain_reports(domains), URLs via get_url_reports(urls), and files via hash lookups against file/report, with md5, sha1, and sha256 all accepted. It also supports URL scanning submissions. The interesting engineering detail is the resources_per_req parameter: the public VirusTotal API caps a single call at 4 resources, while the private tier allows 25. The wrapper handles chunking internally — pass it an arbitrarily long list and it splits the request into as many API calls as your quota allows, defaulting to the private-tier assumption of 25.
That batching behavior reflects a deliberate design philosophy across the whole library: every method accepts a list or any Python enumerable, and every response comes back as a dict keyed by the queried artifact. This makes threat_intel composable in ways ad-hoc requests scripts never are. You can feed the output of one enrichment step — say, the related domains from cooccurrences — directly into categorization or into get_domain_reports, building a graph-expansion pipeline in a dozen lines. That shape-keyed convention also means results slot cleanly into pandas workflows or graph databases without intermediate parsing.
The ShadowServer wrapper rounds out the trio, completing coverage of a typical open-source and community intelligence stack. The README's detail on that module is thinner than the other two, which is worth flagging honestly: the Umbrella and VirusTotal sections are extensively documented with worked examples and sample responses, while ShadowServer gets a listing only. If your workflow leans heavily on ShadowServer's abuse reports, expect to read the source for that module rather than the docs.
Installation follows the standard packaging path — the README links to PyPI where the package ships as threat_intel, so pip install threat_intel gets you running. There is a Travis CI badge pinned to the master branch indicating continuous integration, and the MIT license means embedding it into internal tooling carries no compliance friction. With 286 stars it is a modestly popular project, which tells you it found a niche as quiet infrastructure rather than a headline tool — entirely appropriate for what it is.
From a defender's telemetry perspective, understand what using this library implies: every indicator you submit is disclosed to the upstream provider under your API key, and the optional JSON cache files written to paths like /tmp/cache.virustotal.json will contain the full enrichment results on disk. During sensitive incident response, especially around suspected insider activity or pre-disclosure vulnerabilities, that egress and local persistence is a data-handling decision your IR plan should account for. The tool itself is passive — it only reads intelligence — but the artifacts you enrich with it deserve the same care as any evidence bag.
In an authorized workflow, threat_intel slots in at the triage and scoping phases: pull indicators from logs or alerts, batch-enrich them, pivot on the relationships the feeds expose, and prioritize what actually matters. It is not a detection engine, a scanner, or an EDR — it is the connective tissue between your indicator lists and the intelligence sources that give them context. If your team still enriches by pasting domains into vendor web consoles one at a time, this library is a straightforward productivity upgrade, and its clean, cache-aware, batch-oriented design makes it a good template even if you eventually need to wrap feeds it does not cover.
Yelp/threat_intel.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.