socid_extractor turns any public profile URL into a flat, machine-readable record of usernames, bios, links and stable internal IDs, powering authorized OSINT investigations.
| Tool | soxoj/socid-extractor — Python engine that parses 250+ site schemes into structured account records with stable cross-service identifiers |
| Category | OSINT / SOCMINT data extraction library and CLI |
| Primary Use | Enriching public profiles into normalized dictionaries via a single extract() call, and correlating accounts across renames using stable IDs like GAIA or Facebook UID |
| Safe Use | Educational and documentary analysis for authorized investigations, law-enforcement work, incident response, and research on public data only |
| Telemetry Note | Emits ordinary HTTP requests to the target platform from the analyst's IP — no headless browser or API keys — so site-side rate limits and access logs are the primary defensive observable |
socid_extractor is a Python library and CLI from soxoj that answers a deceptively narrow question: given the HTML or API response text of a public profile page, what can be structurally extracted from it? The answer, per the README, is a flat dictionary covering username, fullname, bios, avatars, locations, created_at dates, follower counts, external links, and — the project's real differentiator — the stable internal identifiers platforms embed in their pages, such as Google's GAIA ID, Facebook UID, Yandex Public ID, and Instagram pk. The tool covers 250+ parsing schemes spanning roughly 2,000 of the sites checked by its sibling project Maigret, all without API keys or a headless browser. This is documentation-grade parsing of publicly served data, positioned squarely for authorized OSINT work.
The architecture, as far as the README reveals it, is scheme-driven: each supported site has a parser definition, and the full inventory lives in METHODS.md, with the parsing logic centralized in socid_extractor/schemes.py. The output side is governed by a normalized field ontology documented in FIELDS.md — field names like username, created_at, and is_verified are consistent across platforms, which means a downstream pipeline ingests one schema instead of maintaining per-site mappings for 250 services. The README claims this saves consumers from writing "130 mappings," and it is a fair point: the value proposition is uniformity, not raw scraping capability. Both the CLI entry point (socid_extractor) and the library entry point (socid_extractor.extract()) share this single code path.
Installation is deliberately boring: pip install socid-extractor on Python 3.10+, or pipx install socid-extractor for an isolated CLI install. The dev version is available via pip install -U git+https://github.com/soxoj/socid-extractor.git. A practical batch tip from the README deserves attention: passing --skip-fetch-if-no-url-hint skips the HTTP fetch entirely when the URL doesn't match any known site hint, which speeds up large runs at the cost of missing generic engines like forum templates. For an operator running thousands of URLs through a checker like Maigret, that flag is the difference between a fast pre-filter and unnecessary network noise.
What makes the tool analytically interesting is its obsession with identifiers that outlive the account's visible surface. Usernames change, platforms redesign, accounts get deleted — but a GAIA ID or Facebook UID embedded in a page's data payloads tends to persist. The README's stated use cases are correlation-oriented: re-identifying the same person after renames and deletions, and pivoting a stable UID into leaked databases, breach corpora, or search-engine and archive.org indices where the UID appears in captured URLs. All of this is passive analysis of already-public material; nothing in the tool authenticates, bypasses access controls, or touches private data. It reads what the platform itself serves to an anonymous visitor.
The documented CLI example illustrates the output shape well: pointing socid_extractor at a DeviantArt profile returns country, created_at, gender, username, website, a list of outbound links to other platforms, and a tagline. The library example against a Patreon page yields patreon_id, patreon_username, fullname, and parsed links — note that Facebook numeric IDs show up directly inside those link URLs, which is exactly the cross-service pivot the tool is built around. Both examples in the README reference public creator accounts; in practice, an authorized investigator would run the same extraction against subject profiles within the scope of a case, then join the resulting UIDs against their own datasets.
Site coverage is broad and deliberately heterogeneous. The README enumerates major networks (Facebook including group pages, Instagram, VK.com, OK.ru, Reddit, TikTok, Bluesky, Tumblr), the Google ecosystem (Docs and Maps contributions with cookies required, Google Play, YouTube), Mail.ru properties, and a long tail of dev and writing platforms (GitHub, Stack Overflow via both HTML and API, Medium, Substack, Hashnode, WordPress.org). Notably, it also ships universal detectors for platform software rather than specific domains — Discourse forums, MediaWiki/Fandom wikis, and Mastodon instances — meaning any self-hosted instance of those is parseable, not just flagship deployments. Niche coverage runs from Chess.com to Roblox to the TON/Telegram Fragment service.
Engineering hygiene is a strong signal here. The README mandates that every new scheme ships with an end-to-end test in tests/test_e2e.py hitting a real URL, complemented by inline-fixture unit tests in tests/test_socid_improvements.py; CI runs the suite with pytest-xdist parallelism and markers like not rate_limited to keep flakes manageable. This matters because the project's known weakness is intrinsic to its design: parsers rot as platforms change markup and API shapes. The README is candid about this, explicitly citing maintenance burden as the reason a commercial offering exists — a hosted extraction API and paid parser maintenance, contactable via the listed email — with the MIT-licensed core remaining free for unrestricted commercial use.
The ecosystem around the tool tells you where it sits operationally. It is the extraction engine behind Maigret, the widely used namechecker that reports account presence across 3,000+ sites. Other consumers listed include TheScrapper for site-scraping emails and social accounts, InfoHunter for general OSINT collection, YaSeeker for Yandex account enumeration by login, and Marple for search-engine result scraping. The README also cross-references ShareTrace, a companion project that works in the opposite direction: instead of reading profile pages, it decodes share links (TikTok, Instagram, ChatGPT, Discord, Telegram) to recover who created or forwarded them, partially offline. The project even maps itself to SOWEL technique taxonomy entries (SOTL-1.4 Analyze Internal Identifiers, SOTL-11.1 Check Outdated And Unused Functionality), which is a thoughtful touch for anyone building formal OSINT training or methodology documentation.
From a defensive perspective, socid_extractor is worth understanding even if you never run it. Its core insight — that stable internal identifiers leak through public pages and enable long-horizon correlation of accounts — is a lesson for platform engineers designing what their HTML and API responses expose. Defenders monitoring for investigative attention against their own infrastructure should note the tool's footprint: plain HTTP requests with no browser fingerprint, no API keys, and rate-limit sensitivity acknowledged in the test suite's markers. Site-side anomaly detection on scraping-like access patterns is the natural countermeasure, and the tool's own reliance on URL hints (exploitable via --skip-fetch-if-no-url-hint) means its coverage is only as good as its scheme registry.
For the authorized practitioner — an investigator working a sanctioned case, an analyst enriching indicators from public sources, a researcher studying cross-platform identity linkage — socid_extractor is a well-maintained, honestly documented component rather than a flashy one. Its strengths are the uniform extract() interface, the normalized field ontology, the breadth of schemes, and a testing culture that catches parser decay early. Its limits are inherited from the problem domain: sites not in METHODS.md return nothing, cookie-gated surfaces like Google contributions need session material the tool doesn't manage, and output quality tracks whatever the platform currently exposes. Used within scope and against public data only, it earns its place as the quiet parsing layer under a good deal of the modern OSINT toolchain.
soxoj/socid-extractor.Educational analysis for authorized security professionals. Use only in controlled, authorized environments.
Related coverage
0 comentários:
Post a Comment
Note: Only a member of this blog may post a comment.