Inside antidbg: syscall-only anti-debugging that hardens Windows x64 software against reverse engineering

antidbg is an x64 user-mode anti-debugging library for Windows that blocks debugger attachment and analysis in software you own, using direct syscalls and forty-plus detection heuristics.

Tool	NotRequiem/antidbg — x64 userland anti-debugging library for Windows written largely in Assembly, MIT-licensed
Category	Anti-analysis / software protection library (C/C++/Assembly)
Primary Use	Embedding debugger-detection into your own Windows x64 applications via StartDebugProtection() guard mode or one-shot isProgramBeingDebugged() checks
Safe Use	For developers hardening their own licensed software, and for authorized red-team/blue-team research into anti-analysis techniques in controlled lab environments
Telemetry Note	In debug builds it logs detection attempts via core/debug.c; release builds silently crash the process with INT 29h / STATUS_SXS_EARLY_DEACTIVATION and may queue an APC to terminate — defenders should note the abrupt, handler-bypassing crash signature as an anti-analysis indicator

NotRequiem/antidbg is an x64 user-mode anti-debugging library for Windows, written predominantly in Assembly, that turns a single function call into a continuously running protection layer against debuggers. The README positions it honestly: consider it a base for your anti-debugging protection, not your only defense. With roughly 322 stars, an MIT license, zero external dependencies, and a claimed footprint of around 1% CPU and under 2MB of memory, it is engineered to be dropped into a product build without meaningfully changing its performance profile.

The architectural story is what makes this project interesting. Everything sensitive is done through direct syscalls implemented with inline assembly, deliberately avoiding Win32/NTDLL exported functions that a debugger or instrumentation framework could hook. When a check can only be performed through a non-syscallable export, the library manually reverse-engineers that function and reconstructs it inside the module address space of the protection thread. User-mode memory structures such as the PEB are walked by direct memory introspection rather than through APIs, which removes the entire class of user-mode hooking from the trust surface.

The threat model is explicit: an attacker may attempt to intercept the protected process from any privilege level, and detections degrade in effectiveness as the attacker moves up to higher CPL. Against trivial user-mode interception, the library hardens itself by enforcing virtual memory protection and injection mitigation policies on its own process, protecting critical stubs as non-writable memory, and monitoring them afterwards with hardware-accelerated hashing running on a separate thread. It also detects or overwrites non-legitimate instrumentation callbacks that would otherwise let a hypervisor or edr-style tool spoof syscall return values in RAX.

A particularly crafty detail is the on-disk versus in-memory comparison of monitored .text sections. Any inline patch — a software breakpoint's 0xCC byte, a detour, a hotpatch — shows up as a divergence between the image as it exists in the file on disk and the image as it exists in memory. Complementing that, the library plants deliberate memory honeypots: execution paths intentionally left unguarded against user-mode hooks, used for state comparison and to confuse whoever is doing the analysis. For reverse engineers, this means some of what you observe is bait by design.

The enumeration of detection techniques is the meat of the README, catalogued in abdg.c and supplementary concepts in the antidbg/archived folder. It spans the classics — reading PEB.BeingDebugged, both via the kernel32 export and via a raw __readgsqword(0x60) read at offset 2, NtQueryInformationProcess with ProcessDebugObjectHandle and ProcessDebugPort, NtQuerySystemInformation with SystemKernelDebuggerInformation, plus a direct read of KdDebuggerEnabled in the KUSER_SHARED_DATA page. Hardware breakpoints are caught by inspecting Dr0 through Dr7, including the subtler case where a debugger clears LBR/BTF bits in DR7 to perform its own single-stepping.

Beyond the classics, the list gets into genuinely obscure territory. Interrupt-based checks include INT 2D (watching whether the following byte is skipped), INT 3D, the ICE breakpoint 0xF1, and a prefix edge case using 0xF3 0x64 where a REP prefix forces an icebp skip. Heap forensics appear twice: once via the NT global flag mask (FLG_HEAP_ENABLE_TAIL_CHECK, FLG_HEAP_ENABLE_FREE_CHECK, FLG_HEAP_VALIDATE_PARAMETERS) and once by walking the heap directly looking for 0xABABABAB and 0xFEEEFEEE fill magic. There are checks for DBG_CONTROL_C and DBG_RIPEXCEPTION delivery, OutputDebugString side effects, CTRL_C_EVENT console routing, external suspension via NtResumeProcess, copy-on-write page touches, unimplemented syscalls common in emulators, and even a LoadLibrary file-handle behavior test on whether the on-disk image's first byte is 0xCC.

Parent-process ancestry inspection looks for launchers like vsjitdebugger or x64dbg, and window enumeration looks for debugger UI artifacts. Timing analysis detects the overhead of single-stepping, breakpoints, and dynamic binary instrumentation or JIT recompilation. The library also counts EXCEPTION_SINGLE_STEP deliveries through a VEH after arming four hardware execution breakpoints in DR0–DR3 on consecutive NOPs — a clean way to verify the debug-register plumbing is behaving. Guard-page and page-exception scenarios verify STATUS_GUARD_PAGE_VIOLATION delivery, while handle-duplication and HANDLE_FLAG_PROTECT_FROM_CLOSE tests probe whether a debugger is touching, stripping, or filtering handles.

Integration is deliberately minimal. Guard mode is one call — StartDebugProtection() — which spawns a monitoring thread that continuously checks for attached debuggers and, on detection, logs the attempt (in debug builds) and forcefully exits while preventing anything else from stopping the crash. Single-run mode exposes isProgramBeingDebugged() for a point-in-time check you can call whenever it suits your application's logic. The detection routines run pseudo-randomly, with entropy sourced from hardware-based ASLR, stack behavior, and arithmetic — explicitly without calling the kernel, using user-mode APIs, or issuing instructions that hypervisors conditionally intercept, which itself is an anti-analysis measure.

When a violation is detected, the kill path is aggressive by design: the process crashes itself via INT 29h with STATUS_SXS_EARLY_DEACTIVATION, a status chosen to bypass all exception handlers so an analyst cannot simply swallow the fault and continue. In some cases it also queues an APC to the kernel to terminate the process outright. Additional hardening includes trapping debugger entrypoints like DbgBreakPoint and DbgUiRemoteBreakin, hiding all threads from debugger events and process freeze while preserving thread priority state, and guarding the process entrypoint with TLS callbacks that perform thread start address inspection to catch attachment at launch.

Building it is standard CMake fare. The test runner comes from -DBUILD_EXAMPLE=ON, which compiles example/main.c into antidebug_runner without polluting the core library with an entry point. By default CMake produces a static library (antidebug.lib / libantidebug.a); -DBUILD_SHARED_LIBS=ON switches to a DLL plus import library. Both MSVC and clang-cl toolchains are supported, and one detail worth remembering: Debug builds enable diagnostic logging through core/debug.c, while Release builds strip logging entirely — so if you want visibility into what your own protection is firing on, build Debug in the lab first.

The repo's topic tags — anti-attach, anti-dump, anti-disassembly, malware-evasion — situate antidbg in an awkward but well-documented dual-use space. The techniques are identical to those used by malicious packers to resist analysis, which is precisely why the library is valuable to defensive researchers and detection engineers: understanding how syscall-only anti-debugging, honeypot pages, and instrumentation-callback checks work is a prerequisite for building sandboxes and analysts' tooling that can survive them. Used as intended — protecting software you own and license, or studying anti-analysis in an isolated lab — it is a compact, well-documented reference implementation of the entire Windows anti-debugging canon in one codebase.

For the authorized professional, antidbg is best read two ways at once. As a product developer, it is a ready-made, permissive, dependency-free hardening layer you can graft into an x64 Windows release with one call. As a reverse engineer or detection author, it is effectively a curriculum: forty-three catalogued detection heuristics, each individually explained in source order in abdg.c, each mapping to a specific analyst technique it defeats. Either way, the README's own framing is the correct one — this is a base layer, not a boundary, and anything serious should compose it with additional protections rather than lean on it alone.

Official project repository for NotRequiem/antidbg.

Download Tool

Educational analysis for authorized security professionals. Use only in controlled, authorized environments.