Tuesday, November 17, 2026

Telemetry engineering: which logs actually answer intrusion questions

Telemetry engineering: which logs actually answer intrusion questions

A field guide to designing telemetry pipelines that answer real intrusion questions, moving defenders from log hoarding toward hypothesis-driven collection and analysis.

TopicTelemetry engineering for intrusion investigation — selecting and structuring logs that answer investigative questions
CategoryDefensive engineering / detection strategy
Primary UseDesigning auditd, Sysmon, EDR, proxy, and auth log coverage so authorized defenders can reconstruct adversary activity
Safe UseEducational analysis for defensive professionals operating telemetry on systems they are authorized to administer and monitor
Telemetry NoteTelemetry itself is the subject: well-engineered collection is observable to attackers who probe for agent presence and logging blind spots

Most security programs do not have a logging problem; they have a telemetry engineering problem. Logs accumulate by default — web servers emit access records, authentication systems emit login events, endpoints emit process creations — but accumulation is not coverage. The defining question of telemetry engineering is brutally simple: when an investigator forms a hypothesis about an intrusion, can the existing telemetry answer it, and how quickly? Every design decision, from sensor placement to retention policy to schema normalization, should be evaluated against that question rather than against a compliance checkbox or a vendor dashboard.

The starting point is question-driven collection. Instead of asking "what logs should we ingest," a telemetry engineer asks "what would I need to see to confirm or deny persistence, lateral movement, credential theft, or exfiltration?" Each of those adversary behaviors decomposes into observable primitives. Persistence on a Linux host surfaces in auditd rules watching cron, systemd unit files, ~/.ssh/authorized_keys, and shell profile modifications. On Windows, Sysmon event IDs 11, 13, and 12 covering file creation, registry value set, and registry add/delete answer the same question. If a hypothesis cannot be mapped to at least one event source that is actually collected, normalized, and retained, the telemetry is decorative for that class of investigation.

Process telemetry deserves its reputation as the highest-value endpoint signal, but only when it is complete and contextually rich. A bare process create event with a binary name answers little; the same event enriched with command line, parent process, hash, user context, and integrity level becomes the backbone of behavioral reconstruction. This is precisely why Sysmon configuration quality matters more than Sysmon deployment: a default configuration that omits command-line logging or hashes turns the richest sensor on the platform into a name registry. Similarly, on Linux, auditd configurations that log execve without arguments leave analysts guessing whether curl fetched a page or a second-stage payload. Engineering means tuning the sensor until the event answers the next question an analyst will inevitably ask.

Authentication telemetry is where intrusion questions become multi-system. A single failed logon is noise; the sequence of a failed Kerberos pre-authentication storm followed by a successful logon from an unusual host is a hypothesis. The engineering work here is less about collection — domain controllers, identity providers, and VPN concentrators all log authentication — and more about enrichment and correlation. Defenders need the fields that make events comparable across sources: source address normalized to a consistent format, account identifiers canonicalized across SAM, UPN, and cloud formats, and timestamps synchronized tightly enough that ordering survives analysis. NTP drift of even a few seconds across sources can invert cause and effect in a reconstructed timeline, which is why time synchronization is a telemetry engineering concern, not merely an infrastructure hygiene item.

Network telemetry is often over-collected and under-structured. Full packet capture is expensive to store and slow to query, while flow records (NetFlow, IPFIX) answer volume and topology questions but strip away payload context. DNS logs occupy a uniquely valuable middle ground: query names reveal tooling fingerprints, domain generation patterns, and tunneled protocols in a compact, high-signal format. Proxy and egress logs close the loop by connecting those resolutions to internal actors. The mature pattern is layered network telemetry — DNS answers "what was asked," flow answers "who talked to whom and how much," proxy answers "what was actually retrieved" — with each layer retained at different depths according to how quickly it loses investigative value.

Retention is the quiet killer of investigations. Telemetry that answers questions but evaporates in seven days helps only fast incidents, and intrusions routinely dwell far longer. Retention tiers should follow investigative half-life: raw high-volume events can age out quickly, but normalized, queryable summaries — authentication anomalies, process lineage for privileged execution, DNS histories — deserve months of availability. Storage economics, not strategy, usually set these numbers, which is why an explicit retention model tied to incident-response requirements is a mark of engineering discipline. The question "could we reconstruct an intrusion that started ninety days ago" should have a documented, tested answer rather than a hopeful one.

Normalization and schema discipline determine whether telemetry can be queried at the speed of thought. Heterogeneous sources describing the same concept — user, host, address, action — in incompatible formats force analysts to become data plumbers mid-incident. Common schemas such as ECS, OTel semantic conventions, or OSSEM provide the vocabulary to map sources into a shared model, so a query for "all process executions by this account" works identically across Sysmon, EDR, and auditd. The engineering investment is the mapping layer and its maintenance as sources update; the payoff is that intrusion questions become single queries instead of bespoke scripting exercises repeated under pressure.

Coverage validation closes the loop between engineering and efficacy. The standard technique is adversary emulation: execute documented, authorized test behaviors in a controlled lab — file creation in a startup folder, a scheduled task, a suspicious DNS resolution — and verify each produces the expected event in the pipeline with the expected fields, within the expected latency. Gaps discovered this way are prioritized by threat relevance, not by vendor defaults. This is also where telemetry engineering meets detection engineering: an analytic that fires on Sysmon event 1 with a given parent-child pattern is only as trustworthy as the sensor configuration and schema mapping beneath it, and silent coverage regressions after an agent upgrade are among the most common causes of detection failure.

Finally, telemetry itself has a threat model. Logging agents, forwarders, and the pipeline's control plane are high-value targets: an adversary with administrative access can disable Sysmon, clear auditd rules, tamper with forwarder configurations, or delete indices. Mature designs assume this and add integrity signals — agent heartbeat monitoring, alerting on logging-service state changes, immutable or append-only storage tiers, and detection for log deletion events such as Windows event log clear (event 1102) or auditd rule removal. The uncomfortable truth is that the absence of telemetry is itself a signal, and a pipeline that cannot notice its own silence will be exploited precisely where it is blind. Engineering for that silence is what separates a defensible telemetry program from an impressive but brittle one.

Educational analysis for authorized security professionals. Use only in controlled, authorized environments.

Sponsored
Share articleFacebookXLinkedIn

Continue exploring

Browse all articles →

0 comentários:

Post a Comment

Note: Only a member of this blog may post a comment.