Introduction
Ransomware has been a persistent threat, costing organizations millions in damages due to compromised credentials, unpatched software, and misconfigurations. To highlight how serious the issue is, the Cybersecurity and Infrastructure Security Agency(CISA) has created a dedicated page to stop Ransomware. There isn’t a single reason why ransomware continues to be so successful; instead, the causes range from poorly educated employees whose credentials were stolen, to unpatched software, to various misconfigurations. That being said the real magic of ransomware occurs in that “last mile,” when the actual file encryption takes place. Evading antimalware usually boils down to a general conclusion , and that is writing custom code with new techniques that have not been identified as malicious. Some of the techniques that are being used in the wild include:
Advanced Obfuscation and Polymorphism
Fileless and “Living off the Land” Attacks
Disabling or Tampering with Security Tools
Overwhelming the Defenses via Quantity and Variant Proliferation
Traditional antimalware solutions attempt to intercept or monitor encryption related APIs, but what if those APIs aren’t called in the usual way? In this post, we walk through a proof-of-concept ransomware-like tool that avoids detection by using inline assembly for everything important such as filesystem operations, socket interactions, and AES encryption, thus bypassing many conventional detection technique.
Why Inline Assembly
This technique is referred to as Native API by MITRE. In short, API hooking is one of the last lines of defense for many EDR (Endpoint Detection and Response) and antivirus tools. They rely on intercepting calls to known functions, like open(), write(), or EVP_EncryptInit_ex(). By implementing these in pure assembly, we effectively hide our footprints. Standard hooking solutions see something that resembles benign syscalls (or doesn’t show up at all the way they expect), which can let the code slip by.
The snippets shown in this blog have been truncated for brevity, you can find the complete code here.
Please note that this post does not create fully functional ransomware. Instead, it implements a limited set of features solely to demonstrate the proof of concept.
Minimal Syscall Wrappers
Why these matter: We’re calling Linux syscalls directly using inline assembly:
xyq_sc3: Places the syscall number in rax and up to three arguments (rdi, rsi, rdx) before the syscall instruction.
xyq_sc2, xyq_sc1: Same idea but for fewer arguments.
This approach avoids the standard C library (libc) and its well known function calls. The AV hooking logic that expects calls to fopen, read, or write never sees them (although these can still be intercepted at the kernel level)
We define a few more convenient wrappers:
They correspond to open(), read(), write(), close(), exit(), and getdents64(). They rely on those assembly stubs above, basically passing the correct syscall number (e.g. XYQ_OPN == 2 for Linux’s sys_open).
AES-256 Key Expansion and Single-Block AES-NI
We also need encryption but we don’t want to rely on calls like EVP_EncryptUpdate(). Instead, we do AES-NI instructions directly:
We hold the standard AES S-box and round constants in arrays to do the AES-256 key schedule ourselves.
Q3(...): Performs the full AES-256 key expansion. It takes your 32-byte key and produces 15 round keys (since AES-256 has 14 rounds + initial round key).
Q4(...): Encrypts a single 16-byte block using AES-NI inline assembly. It loads the block into xmm0, XORs with the round key in xmm1, and then calls aesenc repeatedly for each round.
This means no more suspicious calls to OpenSSL’s AES_encrypt or EVP_Encrypt* the encryption is effectively “hidden” in the final binary’s inline assembly. Hooking solutions that monitor known encryption APIs won’t trigger.
Minimal GCM Implementation
For even more stealth, we do AES-GCM in pure code:
Q5(...): XORs two 16-byte blocks.
Q6(...): Galois field multiplication for GHASH. This is the heart of GCM’s authentication.
Q7(...): Another GHASH helper, combining blocks for integrity.
Q8(...): Our chunk-based encryption in CTR mode. It uses Q4 (AES single-block) to generate the keystream, XORs that keystream with plaintext, and updates the GHASH each time.
Q9(...): Final GCM steps, including the tag calculation for authenticity.
Each chunk of the file is encrypted in Q8, and it all happens inlined, with no calls to standard library encryption
Reading & Encrypting Files in Place
After the encryption routines, we have code to iterate over directory entries with getdents64, then open, read, and write files:
struct XYQ_dirent64: Our own minimal struct to interpret directory entries.
Q12 & Q13: Simple string helpers (length, path concatenation).
We store found filenames, and if they’re regular files, we do two steps:
Upload the file via raw TCP (function Q14(...)). This also uses direct syscalls for socket(), connect(), etc.
Encrypt in place using Q10(...).
Here’s a snippet of the encryption function:
We end up with something that, from a hooking perspective, looks like repeated calls to a custom “assembly-based read” and “assembly-based write,” plus some memory manipulations. No fancy library calls to ring any alarm bells.
Socket-based Upload
The upload function Q14(...):
Opens the source file.
Creates a socket (via xyq_sc3(41, ...)—the raw Linux socket() call).
Connects to aa.bb.cc.dd:port.
Reads the file in 4096-byte chunks, then sends them with xyq_w (again, raw syscall).
No standard functions like send(), fopen(), or fread() are used
TLS Key Download
We do use OpenSSL’s TLS client code here, but we skip the certificate checks fhe purpose of the demo:
The function Q16_tls(...):
We only rely on a minimal set of OpenSSL calls for the TLS handshake. Then we retrieve exactly 32 bytes from the server, presumably the AES-256 key. Once that’s done, we never call the usual encryption routines from OpenSSL; we do it ourselves in assembly.
Bringing It All Together (main())
This main function is straightforward: it ensures we have an input directory, downloads the key, scans the directory, uploads each file, then overwrites it with an encrypted version.
Detection Results
When compiled into a single binary and uploaded to VirusTotal, it came back with 0 detections., the hash is
7827c68031d2ca2a754ed742136872f256041b709e8a346f46d6e71a81ea086c
We also copy it to a windows machine that has defender enabled and it runs with no flags
Meanwhile, the Deep Application Profiler detects it instantly and gave it a higher maliciousness score as seen in the video below
We also create a windows version and run it on a windows machine with defender enabled, see minimal hooking triggers because the code never calls the typical WinAPI crypto functions and the code runs successfully
API Hooking Blind Spots
Hooking solutions often watch for calls like:
But our approach:
Replaces the standard library with direct syscalls.
Implements AES with inline aesenc instructions.
Conclusion & Next Steps
While this is just a proof-of-concept, it underscores how trivial it is to bypass many antimalware solutions by:
- Avoiding standard library calls to encryption functions.
- Inlining assembly for syscalls and crypto operations.
- Masking or skipping typical import table entries that hooking engines watch.
Most antimalware solutions rely heavily on rigid rules, lacking the deeper context and understanding needed to accurately identify advanced threats. With DAP, we took a different approach. As demonstrated, DAP was the only solution capable of detecting this malware because it leverages neural networks to gain a deep, contextual understanding of executables, without depending solely on API calls, import tables, or basic heuristics.