Sunday, November 15, 2026

DarkNyx for self-hosted dark web search and threat intelligence

DarkNyx for self-hosted dark web search and threat intelligence

DarkNyx is an open-source, self-hostable Tor-crawling search engine and threat-intelligence platform for authorized OSINT research, defensive monitoring, and safety-conscious dark web investigation.

ToolNexvisionLab/Darkweb-search-engine — self-hosted dark-web search engine, crawler, indexer, and OSINT safety tool suite (DarkNyx Community)
CategoryOSINT / dark web threat intelligence platform
Primary UseRunning your own .onion crawler and search index for full-text, semantic, and entity search during authorized investigations and defensive monitoring
Safe UseThreat-intelligence gathering, brand/watchlist monitoring, breach-exposure checks, and defensive research on infrastructure you own, in labs, or under explicit authorization
Telemetry NoteThe crawler fetches .onion pages exclusively through its own Tor + Privoxy containers, leaving Tor circuit footprints at crawled hosts; the breach checker hashes emails before lookup and never stores or logs them

NexvisionLab/Darkweb-search-engine, branded DarkNyx Community, is a genuinely complete self-hosted dark-web search engine rather than a demo funneling users toward a paid tier. The README is unusually explicit about this: the Community codebase is a full product, and the closed-source Pro tier (bulk AI summarization, an investigation copilot, risk-score narratives) lives in an add-on gateway the open code never imports. With 159 stars, a Python codebase, an MIT license, and topics spanning osint, threat-intelligence, tor, and crawler, it presents as a serious piece of infrastructure for analysts who want their own dark-web index instead of relying on commercial feeds.

Functionally, the platform covers four layers: a Tor-crawling search core, a public read-only API, a set of free OSINT safety tools, and a single-page web frontend. Search operates over crawled .onion pages in both full-text and semantic (embedding) modes, classifying content into 14 categories including marketplace, forum, leak-dump, ransomware, and carding. Entity search is the standout capability for investigators: the system extracts and indexes structured entities — Bitcoin, Ethereum, and Monero addresses, IPs, CVEs, API keys, IBANs, PGP keys, .onion links, and 15 other types — so you can pivot directly from an indicator to every crawled page that mentions it.

The architecture diagram in the README shows a clean separation of concerns. A Scrapy-based crawler/ fetches pages through a tor-privoxy-0 container — the crawler never touches the open internet directly for .onion fetches — and writes structured data to Postgres while indexing content into OpenSearch as it goes. An api/ FastAPI application reads both stores, with Valkey handling per-IP rate limiting. The frontend in web/ is a static single-page app talking to /api/* over HTTPS. Notably, the API contains no crawling code, which means it can be deployed and scaled independently of the crawler — a sensible design for anyone exposing a search interface to a wider analyst team.

The six-stage ingestion pipeline is documented in unusual detail. Discovery candidates from outbound links are recorded by DiscoveryPipeline against their source host and immediately dropped until actually crawled. Enrichment then runs each page through classification.py for the 14 categories, pii.py which flags personal data without exposing it, entity_extract.py, email_extract.py, price_extract.py for marketplace listings, and clone_detect.py for near-duplicate detection against known-legitimate mirrors — the mechanism that distinguishes a real onion service from a phishing clone of one. For semantic search, page content is embedded with all-MiniLM-L6-v2 (384 dimensions), deliberately reusing the same model classification.py already loads rather than paying for a second model in memory.

The safety-rating logic in safety_rating.py is where the project's editorial stance shows. Domains roll up to Verified, Unrated, Suspicious, or Dangerous, with most categories severity-ranked (extremism-violence and leak-dump sitting highest). Two signals override the ranking outright: a page linking to malware, or a near-duplicate of a known-legitimate page hosted on a different domain. Most notably, CSAM is checked before every other signal — a single CSAM-classified page makes the domain CSAM-confirmed unconditionally, and api/labels.py drops it from search, entity search, and domain lookups entirely. The URL checker refuses to even live-fetch or explain why it refuses such a host. This is a thoughtful, defensive design choice that analysts running their own instances should understand before wondering why a host returns nothing.

For threat-intelligence workflows, the ransomware tracking features are the practical draw. A live victim feed sourced from RansomLook powers per-group pages with known-mirror status, safety-reviewed screenshots, and per-victim detail. Watchlist alerts fire a webhook the moment a keyword or regex matches a new ransomware victim, plus an immediate retro-hunt against the existing corpus — this is exactly the mechanism a defensive team monitoring for its own organization's name, brand strings, or partner domains would use. Marketplace price trend tracking adds a monitoring angle for researchers watching stolen-data economics, and clone clusters expose the largest groups of pages with identical opening text across many onion sites — hosting placeholders, copied scam kits, mirrored markets — searchable via clone:<id>, with host addresses deliberately never shown in the cluster listing.

The seven free OSINT safety tools under /tools/* are described as thin API wrappers around the same engines the platform runs internally, and several have notable privacy properties. The breach and credential exposure checker hashes the email before lookup and never stores or logs it; a match tells you the email exists in flagged leak-dump PII and how many times it was seen, never which domain or page it came from. The image metadata scrubber re-encodes uploads rather than stripping in place. The URL safety checker consults the crawled index first so checking a known host never requires a live connection, only falling back to live fetch for uncrawled hosts. The QR code checker decodes uploaded codes and safety-checks the destination before you open it — built against quishing — while the Tor Browser configuration checker runs entirely client-side, reporting on plugin exposure, language leakage, and hardware-concurrency fingerprinting signals.

The optional AI page summarizer is a separate llama.cpp server running Qwen3.5-4B, called by the API over HTTP via SUMMARIZER_URL, and deliberately excluded from docker-compose.yml. Zero external API calls and zero per-query cost — a meaningful property for sensitive investigations where sending crawled dark-web content to a third-party LLM provider would be operationally unacceptable. The phishing email analyzer does have an honest gap documented: no SPF/DKIM/DMARC verification yet, since that requires live DNS lookups the author classifies as a distinct, unbuilt capability.

Deployment is container-first: four services (Postgres, OpenSearch, Valkey, Tor + Privoxy) start with one docker compose command, while the crawler, API, and web frontend are ordinary Python programs you start yourself. Requirements are modest — Python 3.11+, Docker with Compose, Git, and roughly 8 GB of RAM. Production guidance exists in docs/ARCHITECTURE.md, docs/API.md, and docs/DEPLOYMENT.md, the latter covering systemd units, nginx, and TLS via DNS-01 — a level of operational documentation most hobby OSINT repos never reach. A typical local bring-up looks like:

git clone https://github.com/NexvisionLab/Darkweb-search-engine followed by docker compose up -d to bring the backing services online, then the crawler and API are launched from their respective directories. Once running, the frontend serves at http://127.0.0.1:8080. Keep in mind there is nothing to search until the crawler has saved pages — a fresh instance starts with an empty index, so building a useful corpus takes time and careful seeding.

Operationally, anyone deploying this should think through the trust and legal boundary carefully. Crawling .onion services from your own infrastructure creates observable Tor circuit activity at crawled hosts, and the content categories involved — leak dumps, carding, ransomware victim data — sit in legally sensitive territory in most jurisdictions. The tool's own design leans protective: PII is flagged without exposure, CSAM domains are hard-dropped, breach checks are hash-based, and host addresses are withheld from clone listings. Used within authorized threat-intelligence engagements, internal brand-monitoring programs, or research labs, DarkNyx offers something genuinely rare: a fully self-hosted, privacy-conscious alternative to commercial dark-web monitoring with the architecture documented well enough to actually operate it.

Official project repository for NexvisionLab/Darkweb-search-engine.
Download Tool

Educational analysis for authorized security professionals. Use only in controlled, authorized environments.

Sponsored
Share articleFacebookXLinkedIn

Continue exploring

Browse all articles →

0 comentários:

Post a Comment

Note: Only a member of this blog may post a comment.