ArkhamMirror (marketed internally as SHATTERED) is a self-hosted, privacy-first analysis platform that turns document corpora into searchable, structured intelligence using local LLMs and classic structured analytic techniques.
| Tool | mantisfury/ArkhamMirror — modular, local-first document analysis and investigative research platform written in Python and TypeScript |
| Category | Document intelligence / OSINT analysis platform |
| Primary Use | Ingesting, extracting entities from, and analyzing large document sets with OCR, RAG, knowledge graphs, and ACH matrices for authorized investigations |
| Safe Use | Intended for authorized investigative journalism, analyst research, lab environments, and defensive intelligence work on data you own or are licensed to process |
| Telemetry Note | The project states no telemetry and no cloud dependencies; all processing is local unless an operator explicitly configures a cloud LLM endpoint, so activity is largely invisible to external observers |
ArkhamMirror is an ambitious open-source platform, internally branded SHATTERED in the README, that aims to give investigators a self-hosted alternative to commercial intelligence tooling in the Palantir mold. What makes it interesting from a technical standpoint is not any single feature but the architecture: a Python backend of 17 core services, a React 18 / TypeScript shell, and 26 plug-and-play "shards" that provide everything from OCR to structured analytic techniques. With Python 3.10+, an MIT license, and just under 500 stars, it sits in that narrow category of security-adjacent tooling that treats analyst tradecraft as a first-class engineering concern. The framing throughout is investigative journalism, but the underlying machinery is equally applicable to authorized DFIR triage and threat-intelligence corroboration.
The design philosophy is stated plainly: build domain-agnostic infrastructure, keep the data local, and avoid telemetry entirely. The README's "meta-pattern" codifies every investigative workflow as INGEST --> EXTRACT --> ORGANIZE --> ANALYZE --> ACT, with core shards handling ingestion and extraction, domain shards handling organization and analysis, and output shards handling action. This is a useful mental model for anyone evaluating the platform, because it explains why the feature list is so broad: each stage of the pipeline is independently replaceable, and the platform is explicitly designed so non-coders can contribute configuration-driven shards. Local-first is not marketing here — the default posture assumes your data never leaves the machine unless you deliberately point the LLMService at a cloud endpoint.
Architecturally the project follows what it calls a Voltron pattern: an immutable frame (ArkhamFrame, 17 services), a UI renderer (arkham-shell), and shards that communicate exclusively through an EventBus rather than importing each other. Each shard gets its own PostgreSQL schema, which enforces isolation at the database layer — a genuinely defensive design choice that limits blast radius when a domain shard misbehaves. The frame also advertises graceful degradation: the platform works with or without AI or GPU capability, falling back from Vision-LLM OCR to PaddleOCR, for example. For operators, that means the same deployment can run on a beefy GPU workstation or a modest air-gapped laptop, a flexibility reinforced by the repo's air-gapped and offline-ai topics.
The document pipeline is where most of the engineering effort is visible. Ingestion supports PDF, DOCX, images, HTML, and TXT with batch processing and duplicate detection. Parsing offers 8 chunking strategies plus metadata, relations, and table extraction. Embedding supports multiple models with incremental updates, stored via pgvector in the VectorService. Entity extraction runs spaCy-powered NER across standard labels (PERSON, ORG, GPE, DATE) with relationship detection and duplicate merging, and there is a separate claim-extraction stage that tracks source attribution and verification status — a detail that signals the author understands evidentiary rigor, not just keyword extraction.
Search is hybrid by design: vector similarity through pgvector, PostgreSQL full-text search with BM25 ranking, and configurable weighted combinations of the two, plus similarity search against a reference document and faceted filtering by project, type, date, and entities. The README also documents built-in regex presets for PII, financial data, and technical indicators within the search UI. For an analyst processing a leaked corpus or a disclosure packet, that hybrid approach matters — pure semantic search misses exact identifiers, pure keyword search misses paraphrase, and the tool explicitly supports both.
The structured analytic techniques module is the most distinctive surface. Full ACH (Analysis of Competing Hypotheses) support includes matrix analysis, evidence scoring, premortem analysis, cone-of-plausibility reasoning, scenario planning, and a devil's advocate mode that generates AI counter-arguments. Alongside it sit automated contradiction detection across documents with severity scoring, pattern recognition with statistical significance testing, anomaly detection (statistical, contextual, collective), credibility assessment using the MOM, POP, MOSES, and EVE deception-detection checklists, and provenance tracking with evidence chains and audit trails. These are recognized tradecraft frameworks from the intelligence-analysis canon, and their presence suggests the author is building for analysts rather than repackaging a chatbot wrapper.
Visualization is unusually deep for an open-source project. Graph analysis offers more than ten modes — force-directed with physics simulation, hierarchical, circular, Sankey, adjacency matrix, geographic via Leaflet, causal, argumentation tied to ACH, i2 Analyst Notebook-style link analysis, and temporal graph evolution. Graph analytics include centrality measures (degree, betweenness, closeness, eigenvector, PageRank), community detection, multiple path-finding variants, cycle detection, and component analysis. A separate timeline shard handles temporal event extraction, date normalization across formats, overlap conflict detection, phase management, gap analysis, and clustering. Together these cover the bulk of what investigators currently do across several separate commercial tools.
The AI layer is deliberately provider-agnostic. The LLMService speaks the OpenAI-compatible API, so the same code path drives LM Studio, Ollama, vLLM, or cloud providers such as OpenAI and Groq, with one-click switching from the dashboard. An AIJuniorAnalystService provides cross-shard LLM analysis — anomaly flagging, contradiction surfacing, credibility assessment — and LLM summarization ships in brief, standard, detailed, executive, and key-point formats. The crucial operational property is that AI is additive: the platform's degradation model means an analyst in an air-gapped environment loses assistance, not capability.
Media forensics rounds out the feature set with image authenticity tooling: EXIF extraction, Error Level Analysis (ELA), perceptual hashing, and reverse image search integration. Export is handled through a Jinja2-based template system with placeholder validation, producing JSON, CSV, PDF, and DOCX outputs, analytical reports, templated FOIA requests and legal correspondence, and versioned "packets" that bundle a complete investigation for sharing. The frame's remaining services — WorkerService with PostgreSQL SKIP LOCKED job queues, APScheduler scheduling, email/webhook notifications, and YAML-plus-environment configuration — are the standard substrate you'd expect under a system of this scope.
From an authorized-use perspective, this is analysis tooling, not attack tooling: nothing in the README facilitates intrusion, and the privacy-first posture actively resists data leakage. The realistic deployment scenarios are a newsroom processing source documents, a research team corroborating claims across a disclosure corpus, or a defensive analyst organizing threat-intelligence material — all contexts where the operator already holds the data lawfully. Deployment is Docker-oriented per the repo topics, and the typical invocation path is cloning the repository and following the Quick Start (git clone https://github.com/mantisfury/ArkhamMirror), then configuring the LLM backend of choice.
Caveats worth noting: the README is long on capability tables and short on operational hardening guidance, and the authentication/security section is referenced in the navigation but truncated in the available material, so operators deploying it for sensitive material should independently verify access controls before exposing the UI beyond localhost. Per-shard PostgreSQL schemas and the immutable-frame rule are good structural hygiene, but a platform this feature-dense deserves a careful audit of its ingest path and template rendering before it touches real source material. Those reservations aside, ArkhamMirror is one of the more serious attempts to industrialize structured analytic tradecraft in open source, and its local-first architecture makes it a natural fit for privacy-constrained, authorized investigative work.
mantisfury/ArkhamMirror.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.