estorides is an open-source OSINT aggregation and correlation engine that fans out across 99+ free public sources, resolves entities into a knowledge graph, and exports STIX 2.1, MISP, or GraphML for authorized analysts.
| Tool | grisuno/estorides — OSINT aggregator and correlation engine with knowledge graph, LLM analyst, and STIX/MISP export |
| Category | OSINT / intelligence analysis platform |
| Primary Use | Aggregating 99+ free OSINT sources in parallel for a single target, resolving entities into a NetworkX/Kùzu knowledge graph, and exporting STIX 2.1, MISP, or GraphML deliverables |
| Safe Use | Passive collection for authorized security assessments, sanctioned investigations, CTI research, and lab environments; the tool performs no active scanning and ships an SSRF egress guard |
| Telemetry Note | Purely passive against public APIs, so target organizations see nothing directly; the tool itself maintains an append-only JSONL audit log with IP, query, and latency, plus a default 30 req/min per-IP rate limit |
estorides is an ambitious open-source reimagining of an earlier toolchain the author calls fucklantir / osint_palantir, and the README is refreshingly explicit about its intent: no payloads, no active scanning, just 99+ free public OSINT sources fanned out in parallel and fused into a single intelligence picture. The stated inspirations are Palantir, Bellingcat, Maltego, and Citizen Lab workflows, which tells you the target audience immediately — analysts who need entity correlation and structured reporting rather than offensive operators looking for exploitation paths. Coming from the creators of the LazyOwn Redteam Framework, the project leans hard into the analytical side of reconnaissance, and the acronym itself spells out the mission: Entities, Signals, Targeted, Extraction, Reconnaissance, Open-source, Intelligence, Data, Engine, Scraper.
The core execution model is an async orchestrator. A single query such as example.com triggers a parallel fanout across the full source catalogue, with retries, backoff, and a circuit breaker per source — a significant operational improvement over the sequential HTTP model of the original tool, which the comparison table cites as one of ~20 sources versus estorides at 99. Raw HTTP responses are translated into structured dicts by 50+ dedicated parsers registered via @register_parser("name") in estorides_core/parsers.py. This parser-per-source design is what separates a serious aggregator from a scraping script: each source gets typed field extraction, deduplication, and consistent entity typing rather than regex soup.
Once parsed, observations flow into entity resolution and a knowledge graph. v1.1 hardened this layer considerably: the cross-feed intel_resolver consults Wikidata SPARQL, the OFAC SDN list, IP-API, and NVD to canonicalize identities, while fuzzy clustering via difflib SequenceMatcher at a 0.85 threshold surfaces aliases that exact-match dedup would miss. The graph persists in Kùzu, an embedded database supporting Cypher queries with 99 node labels and 9 relationship types, exposed through a read-only endpoint at /api/intel/graph?q=... with a write-keyword guard to prevent query mutation. A SQLite case store with foreign keys between runs, observations, and entities allows analysts to replay past investigations via GET /api/cases/<id>?full=1.
The v1.4 sprint is where the project gets genuinely interesting from a tradecraft perspective, because it tackles the weakest link in most OSINT tooling: confidence. Every observation now carries a grounded reliability score computed through a NATO Admiralty model — reliability grades A–F crossed with credibility 1–6 — multiplied by corroboration and freshness decay, replacing an ad-hoc +0.1 heuristic. A source_hierarchy module classifies 42 curated sources as PRIMARY, SECONDARY, or TERTIARY, and a probabilistic_fusion module replaces the old MAX() confidence merge in FusionStore with Bayesian, source-weighted merging that specifically prevents tertiary-source inflation. This is the kind of epistemic hygiene you normally only see in government-grade intelligence platforms, and its presence in a free tool is notable.
Two companion modules round out the information layer. A hypothesis_engine generates typed, testable inferences — domain-belongs-to-actor, email-alias-to-person, ip-shared-infra, asn-shared-infra — with deterministic SHA1 identifiers and full evidence trails, so every analytical claim is auditable back to source observations. A change_detection module diffs two snapshots of the same target across eight typed change kinds, including new, disappeared, property_changed, edge_added, edge_removed, and confidence_shifted, with reliability-weighted scoring and bounded output via max_changes. For monitoring-style authorized investigations, this temporal diffing capability effectively turns estorides into a watchlist engine.
The LLM analyst layer is pluggable through @register("name") in estorides_llm/manager.py, supporting four backends — Ollama, OpenAI, Anthropic, plus a stub — and producing BLUF, tactical, and system-level outputs. From a data-governance standpoint, local inference via Ollama matters: sensitive investigation details never need to leave the analyst's machine. The multi-backend design also means the summarization layer consumes the same grounded confidence scores as every other consumer, so hallucination risk is at least constrained by an auditable evidentiary substrate rather than raw scraped text.
Export and interoperability are strong. The engine emits STIX 2.1 bundles, MISP event JSON, GraphML for Gephi, and JSONL for training pipelines, with optional age encryption opt-in via ?key=age1… for at-rest protection of sensitive case files. Additional keyless endpoints added in v1.1 include BGP, MAC, phone, GitHub, leaks, CISA KEV, and malware C2 lookups — the KEV feed integration in particular makes this useful as a defensive exposure-monitoring tool for your own organization's infrastructure. An ontology engine cross-checks results against OFAC sanctions data, and a MITRE ATT&CK mapper auto-tags roughly 40 techniques for context.
Operationally, the defensive engineering is better than most tools in this space. An SSRF guard blocks RFC1918 ranges and cloud IMDS endpoints at egress, which prevents the orchestrator from being abused as an internal-network pivot. Every request lands in an append-only JSONL audit log with IP, query, and latency, and a sliding-window per-IP rate limit defaults to 30 requests per minute, tunable by environment variable. API keys are handled per-source via environment variables rather than baked into configs, and paid sources are strictly flag-based opt-ins so the default posture stays free and keyless.
The development doctrine described in CLAUDE.md — spec-driven development, TDD, BDD, fuzzing, and visual review — is visible in the module table: the observation_models module closed with 100% mutation coverage across 79 mutants, backed by strict Pydantic v2 contracts using strict=True and extra="forbid" with bounds centralized in an env-tunable SchemaConfig. That level of contract discipline in the data layer reduces the odds of malformed upstream data silently corrupting a case, a real risk when you are merging 99 heterogeneous public APIs. The source_health_monitoring module adds a dashboard computing health from success rate, latency, and freshness weights, which addresses the practical problem of OSINT endpoints rotting over time.
Installation is minimal friction: pip install estorides followed by estorides serve --port 5050 brings up the web UI, which includes a Leaflet 2D map for geolocated results, a D3.js force-directed graph view, a timeline, bundle inspection, and the Cases tab with a full-entity inspector. Real-time feed layers for earthquakes, fires, and news add situational-context overlays that investigators in the Bellingcat tradition will appreciate. Licensed under AGPL-3.0 and sitting at 112 stars with HTML as the dominant language, this is a young but unusually well-engineered entry in the OSINT platform category, best suited to authorized threat-intelligence work, sanctioned investigations, and monitoring your own attack surface — not anything involving unauthorized collection against third parties.
grisuno/estorides.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.