ThreatIngestor is a Python framework that polls RSS, Twitter, git, and web sources for IOCs and YARA rules, routing them to MISP or ThreatKB for authorized threat intelligence teams.
| Tool | pedramamini/ThreatIngestor — extensible Python tool that extracts and aggregates IOCs from threat feeds |
| Category | threat intelligence aggregation / SOAR automation, Python 3.6+, GPL-2.0 |
| Primary Use | Polling sources like RSS and Twitter for malicious IPs, domains, and YARA signatures, then pushing them to MISP, ThreatKB, SQLite, or SQS |
| Safe Use | Defensive intelligence collection on owned or lab infrastructure during authorized threat-intel operations, feed curation, and CTI pipeline research |
| Telemetry Note | Purely collection-side: it makes outbound polling requests to configured sources and writes to your own outputs; defenders observe it as benign scheduled feed readers, not as an attack tool |
ThreatIngestor, maintained under the InQuest umbrella (repo pedramamini/ThreatIngestor), solves one of the least glamorous problems in threat intelligence: the plumbing. Rather than hand-curating indicators from a dozen Twitter accounts and blog feeds, it sits as a long-running daemon that polls configured sources, extracts meaningful artifacts — malicious IPs, domains, and YARA signatures — and forwards them to whatever system you use for analysis. With roughly 929 stars, a GPL-2.0 license, and production use powering the InQuest Labs IOC-DB at labs.inquest.net/iocdb, it is a battle-tested option rather than a weekend experiment.
The architecture is a clean plugin split between source (input) and operator (output) modules. Sources currently include RSS, Twitter, git repositories, GitHub repository search, gists by username, sitemap XML feeds, generic web pages, Amazon SQS queues, and — unusually — an image extraction source. Operators cover MISP, ThreatKB, CSV files, MySQL and SQLite tables, SQS, and even a Twitter output for rebroadcasting curated indicators. This input/output symmetry means the tool composes into nearly any existing CTI workflow without forcing a particular platform choice.
Operationally, you write a config.yml (layout documented in config.example.yml), wire up sources and operators, and invoke threatingestor config.yml. The README notes that by default it runs forever, polling each configured source every 15 minutes. That cadence is a sensible default for Twitter-based IOC accounts and blog feeds, where freshness matters but hammering endpoints is counterproductive and rude. Everything runs locally against your own credentials and storage, which keeps the whole pipeline inside infrastructure you control.
Installation is straightforward from PyPI: pip install threatingestor, with optional plugin dependencies pulled in via pip install threatingestor[all]. The README specifies Python 3.6+ with development headers. If you want the image extraction capability — OCR-style scraping of indicators embedded in screenshots posted by researchers — you need Python 3.7 or later and the extra stack pip install opencv-python pytesseract numpy. That dependency cluster explains the version bump: opencv-python and pytesseract are heavyweight and move the tool from pure text parsing into computer-vision territory.
The image extraction source deserves a closer look, because it reflects real analyst workflow. A large fraction of the threat-intel community shares IOC evidence as screenshots of terminal output, sandbox reports, or PCAP details, which is invisible to conventional scrapers. By running tesseract OCR over images pulled from configured sources, ThreatIngestor recovers indicators that would otherwise be lost to automation. The Dockerfile.dev notes in the README even call out the tesseract-ocr and opencv-python lines as common build pain points, which is honest documentation of a fragile dependency chain.
For output, the headline integrations are MISP and ThreatKB. MISP is the de facto open-source standard for threat event sharing, and ThreatKB is InQuest's own knowledge base for managing IOCs and YARA/Sigma rules. The fact that the tool ships first-class operators for both means a small team can bootstrap a functioning threat-intelligence platform without writing glue code. The SQLite and CSV operators lower the barrier further for analysts who just want a queryable local corpus before committing to a full platform deployment.
The SQS operator and source pair point at the tool's SOAR ambitions. Feeding an AWS SQS queue turns ThreatIngestor into the collection stage of a distributed pipeline, where downstream Lambdas or workers can enrich, deduplicate, and score indicators at scale. The topics list on the repo — soar, threat-intelligence-platform, ioc, yara, dfir, threat-hunting — accurately reflects this positioning as infrastructure glue rather than an analysis product.
Getting started intel sources are thoughtfully provided. The README points to an InQuest Twitter List of accounts posting C2 domains and IPs, though it honestly warns that a Twitter developer account application is required to use that source — a real operational hurdle given API access restrictions. For quicker setup, rss.example.yml ships pre-configured with several security blogs, which is the lowest-friction path to a working pipeline and a good first exercise for anyone evaluating the tool in a lab.
Deployment is well covered. A production Dockerfile builds a runnable container, and Dockerfile.dev supports development from a locally built .whl via python3 -m build. The inclusion of extra helper scripts under scripts/ with their own documentation, plus PEP8 contribution guidelines and published ReadTheDocs documentation, signals a project that takes maintainability seriously. CI runs through a workflow.yml GitHub Actions badge, and the package is versioned on PyPI, so updates arrive through normal package management.
From a defensive standpoint, ThreatIngestor is exactly the kind of automation that separates a mature CTI program from manual Notepad curation. Aggregated, timestamped, source-attributed indicators can feed blocklists, YARA scanning campaigns, and hunting hypotheses — all defensive postures. Because it operates purely as a collector against public or subscribed sources and writes only to systems you configure, there is no offensive capability here at all; it is a research and hardening asset.
Caveats worth flagging before production adoption: the Twitter source is subject to API availability and developer-account approval, which has become the most fragile link in many IOC pipelines. The 15-minute default poll interval should be tuned per source to avoid aggressive scraping of blogs. And as with any aggregator, indicator quality is only as good as the curated source list — garbage feeds produce garbage IOCs, so the MISP taxonomy and sighting workflow you push into becomes your quality gate. Treat the tool as the transport layer, and keep human review in the loop before anything hits enforcement.
For authorized security teams, researchers building IOC corpora, and organizations standing up their first threat-intel capability, ThreatIngestor offers a rare combination: sane plugin architecture, production pedigree, and near-zero barrier to a first working pipeline. The quick walkthrough in the official documentation gets you from pip install to running collection in minutes, and the plugin development guide means the integration ceiling is effectively whatever you are willing to write.
pedramamini/ThreatIngestor.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.