EDR solutions

A first, but not last, line of defense

Jarett Tice
| 12/12/2022
EDR solutions: A first, but not last, line of defense

EDR solutions can help mitigate the risk of malware attacks, but they’re fallible.

Cybersecurity is a constant battle of trying to stay ahead of bad actors seeking to infiltrate networks and wreak havoc on systems, and cybercriminals constantly devise innovative ways to bypass the defense mechanisms organizations put in place. 

Of these defenses, even the antivirus and endpoint detection and response (EDR) solutions meant to protect systems from malware can be bypassed with the right techniques. Understanding what EDR software can do and how malicious actors can circumvent it is critical for organizations that want to strengthen their security postures. 

Sign up to receive the latest cybersecurity insights on identifying threats, managing risk, and strengthening your organization’s security posture.

EDR solutions: A first line of defense

EDR solutions support the first line of defense for users at the local machine level. EDR solutions differ from antivirus solutions because they provide antivirus functionality as well as other high-level tools, including endpoint monitoring, forensic logging, and centrally managed incident responses. 

The added functionality over normal antivirus solutions make EDR solutions an ideal choice for enterprises because they help reduce the overall risk of malware infection. However, cybercriminals (and the pen testers who combat them) continue to discover new ways to bypass the latest and greatest protection mechanisms. 

Therefore, it’s important for organizations to understand the limitations and fallibility of EDR software so that they can have more realistic expectations of the protection EDR solutions offer. Specifically, attacker techniques such as shellcode injection and bypassing userland hooking demonstrate how EDR software cannot be considered a one-and-done solution.

Attacker technique: Shellcode injection

EDR software easily detects malware made in the form of a simple executable file. When the executable file is launched, the instructions within are scanned for malicious intent and killed before any code is executed. To avoid detection, bad actors inject small pieces of malware called shellcode inside an already running process. 

This common method of hiding malware inside a running process is referred to as “shellcode injection.” When the malware is packed inside a safe process, its instructions can blend in with the process’s normal functions. This technique is especially effective when the host process uses the same sorts of instructions or accesses the same locations in memory.

For shellcode injection to be successful, a few factors need to be in place:

Shellcode encoding. Before shellcode can be injected into a host process, the shellcode must first find its way onto the machine without being detected. To detect shellcode on the host, antivirus and EDR software scans the shellcode’s signature then compares it to known malware signatures. This initial scan often results in the shellcode being flagged before any injection can occur. To bypass initial EDR scans, malicious actors encode the shellcode to obfuscate its true intentions and make the shellcode essentially unpredictable. Encoding can be done many ways. Tools such as pwntools or Mortar can make a script that encodes shellcode and then decodes it at runtime. However, many encoding and decoding solutions are custom-made scripts that can avoid detection and don’t match common malware signatures. 

Entropy reduction. Malicious actors encode the shellcode to hide the malicious instructions in the shellcode file, but doing so raises some red flags. If a file’s binaries are mostly unreadable – that is, if they have a high level of entropy – EDR software might flag it under the assumption that parts of the file have been intentionally obfuscated. Cybercriminals want to avoid such flags, so their solution is to the reduce the level of entropy by adding more readable binary code around the encoded shellcode. The extra packing can come from anywhere as long as it’s seen as benign by EDR software. Often, strings of words are pulled from the dictionary or literature to pad the shellcode with unremarkable content, thus packing a bunch of safe content around the sketchy content. With the safe content around the encoded shellcode, EDR software might not flag the code as malicious. This method makes it easier for malware to fly under the radar and eventually get injected into a process.

EDR functionality: Userland hooking

Any program, malicious or not, that interacts with the Microsoft Windows™ operating system (OS) uses Windows application programming interfaces (APIs). Essentially, every program that runs on a Windows machine makes calls to the Windows API. For example, a program accessing user input from the keyboard uses one or more of the Windows APIs to retrieve the data. In addition to the Windows API, there is another API built into the OS called the Native API. Generally, the Native API works at a lower level in the OS than the Windows API and has different purposes. However, the Native API is another tool set that software (or malware) developers can use to interact with the Windows OS.

Userland hooking is a common mechanism built into many EDR solutions that makes it possible to monitor a process’s code execution in real time. EDR solutions achieve this by inserting their own code into a process loaded in memory. Usually, code injected into the process is in the form of assembler JMP statements that simply redirect to a dynamic link library (DLL) file loaded by the EDR solution. These JMP statements are placed before specific Windows or Native API functions, making them into “hooked” API calls. EDR software “hooks” these APIs and their respective functions to make sure the programs using the APIs don’t have bad intentions.

Choosing which API functions are hooked is up to the EDR solution. Once a process’s execution reaches a hook, code execution is redirected to the EDR solution’s DLL that will scan the process’s instructions to look for malicious activity. If nothing is detected, execution of the process’s instructions will continue. If the EDR solution deems it malicious, the process will be killed before any more code is executed.

Attacker technique: Bypassing userland hooking

The first method bad actors might try to circumvent EDR software hooks is to simply avoid them. Using several different tools, these hooks can be automatically detected or found by a user manually examining the code execution. After determining which API function calls are being hooked, malware with different Windows API functions can be injected into the process that won’t be hooked by the EDR solution. Avoiding the hooked APIs might bypass an EDR solution’s detection mechanisms, but it also limits the types of attacks cybercriminals can deploy because they are constrained to using only the APIs not hooked by the EDR software. For instance, executing a keylogger might be difficult if the Windows keyboard API is hooked.

Another method of getting around EDR software hooks is to unhook them by removing the injected code or hooks added by the EDR software. Different techniques of unhooking a process include:

Patching. The first unhooking technique is through patching the code injected by the EDR solution. Using a dissembler tool like Ghidra or IDA Pro, the raw contents in memory are shown as assembly code instructions. Once the API hook is found in those instructions, an attacker can manually remove the hook or replace the hooked instructions with the original process instructions using a debugger tool such as WinDbg. After the EDR software hook is patched, the process’s execution is never redirected and will not be killed by the EDR solution. Patching applies to legitimate processes and processes infected with malware.

Full DLL unhooking. The second way to unhook a process is by fully reloading the clean process. Like patching, the process is patched in memory. However, instead of patching only a portion of the code, the entire clean process is loaded from disk storage and overwrites the hooked process in memory. Malicious actors can use this technique to completely unhook the process and eliminate the tedious act of finding the hooks manually.

Going beyond EDR solutions 

As antivirus and EDR solutions get smarter by including more robust detection mechanisms, individuals who try to bypass them seem to grow more clever, too. Even though EDR solutions are constantly updated with mechanisms to prevent new attacks and detect threats, they are not infallible. 

While EDR software is an excellent tool, it is not a replacement for following cybersecurity best practices. It is crucial for organizations to understand the latest methods threat actors deploy so that they can better detect and respond when their endpoints are under attack. 

Additionally, a comprehensive cybersecurity strategy should be in place that doesn’t solely rely on EDR software and its detective and preventive abilities. As always, the human component is a major factor when considering cybersecurity threats, so taking steps to educate employees about cybersecurity awareness also should be a top priority. 


Microsoft and Windows are trademarks of the Microsoft group of companies.