|
|
Since honeypots have become a popular research tool and deployed over an increasing number of networks, more and more people are interesting in defeating them. As mentioned earlier, a fake release of the well-known Phrack magazine [12,13] contained several articles on honeypot detection. In these articles, the author introduced several ways to fingerprint honeypots, either locally or remotely. Recently, more academic researchers have focused their attention on honeypot detection, too. In the following, we describe several techniques aimed at detecting high-interaction honeypots or the tools usually associated with them.
In Section 2.2.5, we introduced Sebek, a tool for collecting forensic data from compromised high-interaction honeypots. Since Sebek is one of the most popular data capture tools employed by honeynet researchers, it has received special attention from the underground community. In the following, we want to present some possibilities for detecting and disabling Sebek. This highlights some ways an attacker could detect the presence of this typical honeypot tool. Some of the outlined techniques do not work for the current version of Sebek, but we nevertheless present them to give an overview of the arms race between attackers and defenders that started a few years ago and that will presumably continue for several years.
In September 2003, the fake Phrack publication Nr. 62 contained an article describing how to disable Sebek by simply overwriting Sebek's read() hook in the system call table, with the original value and thus completely disabling it [12]. The article also presented other techniques for detecting the presence of Sebek. For example, it is possible to detect the outgoing connection limit used on most honeynets and also the existence of snort_inline [84], an intrusion prevention system that corrupts harmful looking packets.
Because Sebek is implemented as a kernel module, it leaves several traces of its existence in memory. Let's discuss how to detect the presence of Sebek by tracking down these traces. One of our first experiments was originally aimed at making Sebek's logging mechanism unreliable. Instead, we found an easy way to detect Sebek that is available even to unprivileged local users:
user@honey:~ dd if=/dev/zero of=/dev/null bs=1
Running this simple command caused the packet and byte counters of the network interface to run backward. By checking the output of ifconfig, any user was able to detect Sebek. After pointing out this flaw to Sebek's maintainers, it was fixed in version 2.1.7.
To understand why using dd might result in unreliable logging by Sebek, we have to take a closer look at the logging process. Sebek replaces the normal sys_read() call in the system call table with its own version that allows it to record all data accessed via read(). The logged data is then transferred covertly via UDP over the network [99]. By reading a single byte with read(1), Sebek has to transfer nearly 100 bytes of data, including all the network headers, over the network to the logging host. By executing read(1) a few thousands times per second, we expect the logging mechanism to cause network congestion and eventually lead to dropped packets. The expected network congestion can also be verified with the help of the ping command.
This insight leads to another way of detecting Sebek, which we called the dd-attack, since it can be accomplished with a simple command like dd, as just seen. To detect Sebek on a host, we first ping a local IP-address — for example, the gateway — to get an idea of current network performance. We then start dd in the background and run the ping command. If Sebek is installed on a host, this leads to an enormous increase in the average round-trip time. In our experiments, this attack increased the average round-trip time from 0.7 milliseconds to over 4800 milliseconds. This approach is very similar to detecting correlation in low-interaction honeypots, described in Section 9.1. In both cases, we were observing an unexpected performance correlation that revealed the presence of honeypot tools. The dd-attack is fixed for the latest Sebek version, since it is now possible to filter what should be logged and what should not.
Ironically, when the flaw was fixed in Sebek 2.1.7, the changes led to yet another way of detecting it. In an attempt to prevent backward counting transmit statistics, Sebek left the counters untouched and instead modified the output routines for /proc/net/dev by deducting the number of Sebek packets from the total number of transmitted packets. Unfortunately, this enabled an adversary to compare dev->get stats()->tx bytes with statistics kept in /proc/net/dev, which contain information on every Ethernet device supported by the system. Any differences can only be explained by the presence of Sebek.
Another way to detect Sebek, also mentioned in fake Phrack 63 [13], relies on that fact that Sebek is a kernel module and needs to use some kernel memory for storing information like its magic number, source, or destination port. Although Sebek 2.1.7 uses randomization and other stealth techniques to prevent anyone from finding these values, it still leaves some traces in memory:
By default, Sebek chooses a random number below 1000000000 as its module name. As a result, Sebek's module can be detected easily just by searching for a numeric name in the lists of modules generated by the methods outlined by madsys [53]. A minor problem when doing this is that some legitimate modules also use purely numeric names — for example, 8390, a driver for a generic Ethernet chipset.
Another giveaway is that Sebek's symbols all follow the same naming convention: a letter followed by a number of up to three digits.
In addition to changing /proc/net/dev output, Sebek also leaves a conversion table for translating IP addresses into integers in memory and keeps more than 200 bytes of strings around, which can be located without great effort.
To hide its presence, Sebek takes the same route that Adore, a popular rootkit, does: It uses cleaner.c, a simple loadable kernel module that unlinks the last module in the kernel's module list. This unlinking prevents programs like lsmod from displaying the presence of Sebek on a host. However, Sebek's complete module header structure is still retained in memory. This structure can be found with very little effort [53] and used to locate Sebek's code, such as its cleanup function and variable storage.
Although the location of variables in Sebek's variable storage block is different every time Sebek is compiled, we can still locate the variable storage via the module header that allows us to successfully guess most variables like magic number, source, and destination port in just a few tries.
We can find all these traces by traversing memory and looking for the appropriate patterns. As a result, we can detect hidden kernel modules, including the Sebek kernel module, and extract secrets from them. Implementing this approach leads to yet another way to detect Sebek on a host.
For Solaris, fake Phrack 62 [12] proposed a way to detect Sebek's changes to the system call table. When Sebek installs its read() monitoring function, it replaces the function pointer in the system call table at NR_read with a pointer to its own function, declared in the module. While on a system without Sebek, the system call pointers are close to one another in memory, their relation changes if Sebek is loaded. The address of Sebek's read function isn't anywhere near the address of sys_write, the successor of sys_read in the system call table. If the addresses of sys_read and sys_write lie far apart — more than 4096 bytes — we conclude that the system call table was modified by some rootkit. Furthermore, there is another similar way to detect Sebek. After Sebek has replaced the function pointer in the system call table, sys_read points inside the memory range of a module. This is a clear sign that the system calls were changed and a rootkit is present.
Figure 9.2 shows the location of two system calls in memory for Linux as operating system before and after Sebek is loaded. One can easily see that the system call table was modified, an indicator that something suspicious is going on.
before: _ _NR_read = 0xc0132ecc
_ _NR_write = 0xc0132fc8
after: _ _NR_read = 0xc884e748
_ _NR_write = 0xc0132fc8 |
Now that we know how to detect Sebek, let us see how we can completely disable it. One brute-force solution to disabling Sebek is simply rebooting the honeypot host. Currently, Sebek does not support automatic loading on system startup, but this can easily be achieved by implementing a custom loading mechanism via shell scripts upon system startup. However, integrating the installation of Sebek into the boot process would result in additional evidence of Sebek's existence and in a vector that would allow us to disable Sebek by removing the boot scripts and rebooting.
Another possible approach for disabling Sebek is more complicated: reconstruction of the original system call table. Tan Chew Keong showed a way to do exactly that for Sebek on Windows [47]. By writing directly to \device\physical-memory, a malicious program can restore the running kernel's SDT service table. The only difficulty is finding the original values of the overwritten system calls. However, on Windows XP, a complete copy of the SDT service table exists within the kernel file ntoskrnl.exe and can be used to restore the in-memory SDT service table to its original state.
Probably the most elegant solution to disable Sebek is to ask it to remove itself. If we can obtain the address of Sebek's cleanup_module() function, we can call this function to let Sebek restore the original state of the system call table and thus disable further logging. This is possible because we can retrieve information about the module structures, and then it is fairly straightforward to implement a kernel module that jumps to the memory location of Sebek's cleanup_module(), executes this function, and, as a result, removes Sebek from the host. This works because Sebek reconstructs the pointer to the original read() system call, ord in the code snippet following:
if (sct && ord) {
sct[__NR_read] = (unsigned long *)ord;
} |
After calling cleanup_module(), the system call table has been reverted to its original state, and no further logging takes place. This flaw of Sebek is also fixed by now. The cleanup function does not reconstruct the original state again. Similar techniques for disabling Sebek are also applied to other operating systems that have Sebek support like OpenBSD.
Before we discuss evading honeypot technologies in general, here is another example of an easily detected countermeasure. Most honeynet operators rightfully fear liability if intruders use their honeynet as a stepping stone to attack third-party systems. Therefore, they employ two common techniques to minimize the risk of a successful attack on a third party's system: First, outgoing connections are heavily restricted to numbers as low as only 15 outgoing TCP connections per day. Unfortunately, such extreme measures to control outgoing traffic are trivial to detect. For this particular containment policy, we only have to initiate a large number of outgoing connections (SYN packets to other hosts) and see when connections are being blocked by the honeywall. (We saw other containment policies in Chapter 7). The most sophisticated containment policy allowed an unlimited number of connections but only up to n different destination hosts. However, such a policy can also be easily detected by making outgoing connections to a number of different hosts. The outcome of such an experiment might not always be clear because, these days, some ISPs deploy mechanisms to contain worm spread that limit connections in a similar fashion.
Deploying snort_inline is another way to prevent outbound attacks from causing damage. snort_inline examines outgoing traffic for known exploit payloads and corrupts them in such a fashion that they stop working. An example used by honeynet operators all the time is to replace the string /bin/sh in shellcode with another similar but nonfunctional string /ben/sh. Figure 9.3 shows an example of a replacement policy for known shell code that renders the corresponding attack useless.
alert ip $HONEYNET any -> $EXTERNAL_NET any (msg:"SHELLCODE x86 stealth NOOP"; sid:651; content:"|EB 02 EB 02 EB 02|"; replace:"|24 00 99 DE 6C 3E|";) |
As an adversary, it is easy to detect the presence of a Honeywall by constructing a traffic stream containing strings that match snort_inline's rewriting database and check if all packets are received unmodified. This can easily be confirmed by running the experiment yourself.
So far, we have shown how easy it is to detect the presence of monitoring tools and attack prevention mechanisms. Now we will demonstrate how to circumvent logging, even if the logging mechanism is passive or hidden on the network. There are several ways to circumvent logging on a honeynet. An attacker can make logged data unintelligible, make logging itself unreliable, or use methods that are not to be logged at all.
In the following, we again take the view of an attacker and show how we could evade a honeypot. Some parts of this are an overview, and not all techniques work anymore, but you should be aware of the fact that an attacker could try to circumvent the logging taking place.
There is little that can be done to avoid logging of network communications by a network component like the Honeywall. We could try to compromise the logging host and wipe data from the machine, but this is not often feasible, and as such, we cannot rely on it. Instead, we have to find another way to avoid logging. If we were to encrypt all of our network communication, a passive or external system would not be able to make any sense out of it. The main problem in doing so is that the initial exploit of the target is usually impossible to encrypt. Notable exceptions are exploits delivered via HTTPS. If the target offers an HTTPS web service, we have a wide range of opportunities to exploit the web server, vulnerable CGI programs, or other programs accessible via the web interface. This allows us to communicate in encrypted form and therefore unintelligible for network traffic logging.
To circumvent logging by host-based systems such as Sebek, we need to take a more detailed look at how Sebek works. Sebek obtains access to all data processed via the kernel's sys_read function. When a process calls read(), Sebek records its PID and UID, the file descriptor, the first 12 characters of the program's name, and the data read. Nearly as interesting as what is being logged is what is not being logged. For example, neither the path of the accessed file nor the path of the calling program is recorded. This means that forensic analysis requires guessing at who is reading what. This allows us to introduce arbitrary noise into Sebek logs. For example, we could create a file looking like a legitimate shadow-password file in /tmp and read it via cat. A honeynet operator analyzing the logs will probably assume that we were able to read the real shadow password file. We could also randomly rename the binaries we plan on using to make determining which programs were executed much harder. While we have no direct control over our PID, we can add executions fork() frequently while terminating the parent process to keep the PID a moving target. Inserting chaff commands, commands that do not serve any particular purpose but complicate analysis, is another way to reduce the value of collected logs. A well-instrumented logging system is capable of automatically following our indirections. However, if we manage to get a human to look at our chaff commands, the efficiency of forensic analysis has been significantly reduced.
Another attack against logging systems is to make them operate unreliably. As shown in Section 9.2.1, Sebek's logging becomes unreliable when under dd-attack. The attack increases the rate of network communication to store logs so dramatically that not all log packets reach the storage system, either due to network congestion or system overload. As the number of dropped packets varies greatly, this circumvention method is not very reliable. However, we expect that an adversary will not be able to fully reconstruct a session while a dd-attack is taking place.
The ultimate way to circumvent logging is to simply avoid it altogether. To prevent Sebek from recording activities on a compromised honeynet, we can just stop using read() system calls, since that's where Sebek gets all its data from. To read the contents of a file without going via read() system calls, we can use mmap() to map files directly into memory and therefore bypass logging. With mmap(), we can read all regular files on a system without Sebek noticing anything at all. Unfortunately, device files and files on special filesystems like /proc cannot be read in this manner. Also keep in mind that there is no reason why Sebek should not be hooking other system calls or why other host-based monitoring systems might not be fooled this way. However, for the time being, we will continue exploring how to avoid Sebek by not calling read().
It is notable that executing programs via exec() does not involve read() either. This basically means that we can, in theory, execute arbitrary programs without logging by Sebek taking place. It turns out that this is only true to a limited extent. At program startup, the dynamic linker loads various dynamic libraries into memory. This is done mostly via mmap() but also sometimes via read(). So if we execute an application, there is a chance that Sebek is going to log a read()-call of a library.
To experiment with some of the limitations in Sebek, Maximillian Dornseif, Christian Klein, and one of the authors constructed a proof-of-concept toolkit called Kebes. Kebes is written in Python, an easy to understand and use high-level language, and is designed to enable a wide variety of actions on the target system without using read() at all. Without the help of Max and Christian, this complete work would not have been possible.
For simplicity, Kebes uses a traditional client/server setup. The server resides on the target, listens on a TCP socket, and forks children for each connecting client. Kebes's communication channel is called the crypt layer. It is message-based and uses AES-encryption, random padding, and compression via zlib to obfuscate the message length. Ephemeral encryption keys are established using a Diffie-Hellman key exchange and does not require preshared secrets between the Kebes server and client. Without authenticating the key exchange, the crypt layer is vulnerable to man-in-the-middle attacks. However, Kebes's design is not meant to thwart an active adversary because it is assumed that the adversary already has complete control over the target host. Instead, Kebes's design was meant to make a posteriori analysis of network and filesystem harder, if not impossible.
Kebes does not come with a complicated command structure. The Kebes server initially understands just a single command: ADDCOMMAND. All further commands desired by the client are dynamically loaded via the network in the server process. This basically means that the server is only a communication stub that is dynamically extended via an encrypted network channel. This does not only makes maintenance much easier, since updates can be pushed by the client into the server without restarting the server, but also makes forensic analysis of Kebes's inner workings much harder. As long as no paging or core dump of the Kebes process occurs, there is no permanent record of the more advanced functionality added to the server by pushing code for that functionality to the client. Both this simplicity and extensibility are made possible because Python allows objects, including any code, to be serialized over the network.
Currently supported commands are listing directories, getting file information, creating files, and retrieving system information. Other functionality, like deleting files, is implemented in a more secure fashion. For example, to delete a file, Kebes renames the file to a random name, overwrites its content with a random pattern, syncs the file to the disk, and repeats the whole procedure several times before finally unlinking the file.
Reading a file is implemented by using mmap(). This instructs the virtual memory system to map the file contents into Kebes's address space, from which the file content is then readily available. A copy of this memory area is sent over the network to the Kebes clients. As a result, reading files with Kebes cannot be logged by Sebek.
In addition to executing binaries on the target system, Kebes also has a mode in which binaries can be received from the client over the network. The Kebes server saves the binary under an arbitrary name, executes it, and finally deletes it securely. While there is only limited control over which of the executed programs uses read(), the analysis of Sebek's data is much harder due to the randomized process name.
For highly sensitive code that under no circumstances can fall into the hands of an adversary, Kebes supports a way to execute code without ever writing it to disk. To support this, the server uses the shellcode-executer extension to Python. The extension takes a Python string — which can contain ASCII NULL characters — and changes the control flow so that it continues with a specified instruction within that string allowing the execution of arbitrary machine code by Kebes. This idea can be extended further to support in-memory execution of complete ELF binaries [33].
Whenever cryptography is used to communicate sensitive information, a good source of randomness is needed. This is also the case with the Diffie-Hellman key exchange used by Kebes to establish a secure communication channel. Usually, we could just read from /dev/random, but on a honeypot the randomness would be logged and an adversary might be able to reconstruct any of the keys we created with it. Unfortunately, /dev/random is a device that cannot be mapped via mmap(). Instead, Kebes tries to accumulate entropy from clock jitter, thread races, and the contents of some files. This process looks as follows. Kebes starts two threads that execute exactly the same algorithm. For every file on the host found in /var and /tmp, Kebes keeps track of the following information:
A high-resolution timestamp of when processing of the file started.
The ID of the thread to capture subtle scheduling differences. It is hard to predict which thread is going to start processing first.
SHA-1 of the file in question, if the file is readable and smaller than one megabyte; if the file is readable but bigger, then only the last megabyte of the file. Reading is done via mmap() to avoid logging by Sebek.
A high-resolution timestamp of when processing stopped.
The captured information is then stirred into a 1960-bit entropy pool that can be used to derive random keys. While this method is unsuitable for strong cryptography, it is good enough to make forensic examination of network traffic much harder.
Although, Kebes is unlikely to stop more sophisticated host-monitoring solutions, it demonstrates that if we know which monitoring tools are being deployed, we can try to work around them. Both Unix and Windows operating systems have a large and complicated set of system calls that can be employed in unexpected ways to make the life of honeypot operators much harder.
We use VMware as an example of a hardware virtualization layer that is very popular for deploying high-interaction honeypots. VMware's hardware virtualization creates the appearance of an x86 hardware platform that allows installation of almost any operating system — for example, Linux, Windows, or Solaris 10. Virtualization provides many benefits for running honeypots. Instead of requiring one physical server per honeypot, we can use virtualization to run multiple operating systems simultaneously on the same physical hardware. In addition to better resource management, the main advantage for honeypot operators is the isolation capability provided by the virtual machine monitor (VMM). Unfortunately, the traditional x86 architecture lacks support for hardware virtualization. One way to provide fault isolation is to run the guest operating system at lower privileges. Whenever the guest executes a privileged instruction, the VMM receives a trap and can emulate the desired behavior. However, other privileged states, such as page table entries, may reside in the guest's memory and can be overwritten by the guest without causing a trap. This problem can be overcome by making this memory write-protected and creating a page fault when the guest tries to write to it. The page fault is intercepted by the VMM, which then emulates the appropriate behavior. Such a page fault is called a hidden page fault. However, there are other problems that cannot be overcome that easily. Some privileged instructions like popf cannot be properly virtualized, as they do not generate a trap when run with less privileges. When executing popf in unprivileged mode, it changes ALU flags. However, when executing popf in privileged mode, it changes ALU and system flags! Because the VMM does not get control, the proper behavior cannot be emulated. VMware solves this problem by employing binary translation of the privileged guest code, usually the guest's operating system kernel. For performance improvements, the code is translated on demand as it is being executed. Using binary translation, VMware is able to guarantee proper isolation of the guest OS.
To virtualize network or other I/O devices, VMware provides its own set of virtual devices to provide proper abstraction. As a result, the guest OS requires installation of special device drivers before the virtual devices can be used. Now that we have a better understanding of the technical details behind hardware virtualization, we take a step back and look at what a virtual machine monitor needs to provide to be functional. These requirements were established by Popek and Goldberg in 1974 [65]:
Fidelity allows a VMM to run any software.
Performance allows fast execution of guest instructions — that is, mostly without intervention of the VMM.
Safety requires that the VMM provides proper isolation of the underlying hardware resources.
As you might notice, fulfilling these requirements allows us to run a guest operating system under a virtual machine monitor. To sum it up in one word, these requirements provide compatibility. However, none of them imply stealth or transparency against a dedicated adversary. The way that virtual machines are constructed, even with the fairly new x86 hardware virtualization, makes them easily detectable to anyone who cares. Before we present some very sophisticated means of detecting virtual machines, let's discuss a list of easy giveaways.
In addition to virtualizing different guest operating systems, VMware also needs to virtualize the hardware layer such as disk, network, and other I/O devices. To do so, the VMM provides a virtual device layer that the guest operating system needs to interface with via special drivers. So one easy way to detect VMware is to look at the hardware layer it provides. Prior to VMware version 4.5, some virtual devices were not configurable:
The video card: VMware Inc [VMware SVGA II] PCI Display Adapter
The network card: Advanced Micro Devices [AMD] 79c970 [PCnet 32 LANCE] (rev 10)
The name of IDE and SCSI devices: VMware Virtual IDE Hard Drive, NECVMWar VMware IDE CDR10, VMware SCSI Controller
For example, to get information about the Video BIOS without using any specialized tools, we can run the following command:
sudo dd if=/dev/mem bs=64k skip=12 count=1 | strings -n10
On a real machine, we might get output similar to the following:
IBM VGA Compatible NVIDIA P119 GeForce4 MX 4000 VBIOS Version 4.18.20.39.00 Copyright (C) 1996-2003 NVIDIA Corp. NV18 Board - 119s2937 Chip Rev A4 WinFast A180B VGA BIOS V01.12.2004 Copyright (C) 2000-2005 Leadtek Research Inc. Date:01/12/2004(V8.1) ... |
However, when running under a virtual machine, the Video BIOS is almost assuredly going to look quite different. As an example, we take a look at Parallels, another VMM available for Mac OS X (http://www.parallels.com/): with the same command as the preceding, we find an open source VGABios that even includes some CVS revision numbers. In VMware, we find the following strings:
IBM VGA Compatible PhoenixView(tm) VGA-Compatible BIOS Version Copyright (C) 1984-1992 Phoenix Technologies Ltd. All Rights Reserved ... |
There are many more ways to easily detect virtual machines. It is also possible to identify a running VMware in default mode by looking at the MAC address of the network interfaces [36]. The following ranges of MAC addresses are assigned to VMWare, Inc., by IEEE [39]:
00-05-69-xx-xx-xx 00-0C-29-xx-xx-xx 00-50-56-xx-xx-xx
The MAC address of a network interface can be retrieved by looking at the data related to the interface (Unix systems: ifconfig; Windows systems: ipconfig/all) or by looking at cached MAC addresses via arp -a. If a MAC address corresponds to a VMware prefix, it's a clear sign of not running on real hardware. If an adversary has just compromised a host and finds out that it's running under a virtual machine, suspecting a honeypot is not far fetched.
Other virtual machines, like Parallels, do not provide easily recognizable MAC addresses. Instead, we can try to conclude the reverse. For example, running Linux under Parallels on MAC OS X, we get the following kernel output:
eth0: RealTek RTL-8029 found at 0x1080, IRQ 10, 00:A1:9B:XX:XX:XX
We see that the kernel detected an Ethernet card manufactured by RealTek using a MAC address with 00-A1-9B as prefix. According to the IEEE registry, RealTek has been assigned 00-E0-4C, and the prefix 00-A1-9B is not registered. The discrepancy in this information allows us to conclude that we are not running on real hardware. Information on registered MAC addresses can be obtained via the search interface at http://standards.ieee.org/regauth/oui/index.shtml.
VMware is also known for its I/O backdoor that can be used to configure VMware during runtime. An analysis of Agobot, an IRC-controlled bot with network spreading capabilities, revealed that it detects VMware's presence by probing for its I/O backdoor. The following sequence is used by Agobot to detect if it is running under VMware:
mov eax, VMWARE_MAGIC ; 0x564D5868 'VMXh' mov ebx, 0x00 ; command parameter mov cx, 0x0A ; Get Version (command number) mov dx, VMWARE_PORT ; 0x5658 'VX' in eax, dx cmp ebx, VMWARE_MAGIC ; 0x564D5868 'VMXh' je Detected_VMware ; jump to code that deals with running under VMware |
Although the in instruction is usually affected only by the DX register, when running under VMware, the registers EAX, EBX, and CX are used for passing parameters to VMware and back. To talk with VMware, register EAX is initialized with a magic number to "authenticate" the backdoor commands. Register EBX stores parameters for the commands and the command itself is loaded in register CX. Table 9.1 gives an overview over some possible commands [46].
| Command number | Description |
|---|---|
| 05h | Set current mouse cursor position |
| 07h | Read data from host's clipboard |
| 09h | Send data to host's clipboard |
| 0Ah | Get VMware version |
| 0Bh | Get device information |
Many of these VMware specific artifacts can be hidden using a tool developed by Kostya Kortchinsky [50]. The tool is available from http://honeynet.rstack.org/tools/vmpatch.c and in addition to changing the device identifiers we mentioned earlier can also completely disable the backdoor. Unfortunately, it is only available for older versions of VMware. The tool works by directly patching the VMware binary. It is not recommended to run the patched VMware binary on already installed virtual machines as it might cause unintended side affects. So if you plan on taking this route, patch the binary first and install fresh virtual machines afterward. The tool provides the following functionality:
Disabling the I/O backdoor: The tool locates the code sequence that compares EAX against the magic number 0x564D5868 and replaces the conditional jump with an NOP instruction. As a result, the code implementing the backdoor functionality can no longer be reached.
Changing the MAC adddress: As mentioned earlier, VMware uses three different MAC prefixes for its virtual network interfaces. As a result of running Kortchinsky's tool, you can change the prefix to any number you like. Choose wisely. We mentioned earlier that the network interface is supposed to be an AMD PCnet 32 Lance!
Changing the Video BIOS: Another way of identifying VMware is noticing that the video adapter runs a VMware bios with VMware PCI identifiers. The tool allows you to replace the Video BIOS by uploading a completely new one. To facilitate this task, the tool contains a mode that allows saving the Video BIOS from a different machine to a file. Not all the BIOSs you might try are going to work in VMware, so some experimentation is required.
As we have seen, some of these identifiers can be changed, and artifacts like the I/O backdoor can be hidden. It seems that it might be possible to create a virtual machine that is indistinguishable from regular hardware. Unfortunately, this is not the case, and we alluded to some of the reasons earlier. Tal Garfinkel and other researchers have recently written a comprehensive position paper arguing the following thesis:
We conclude that building a transparent VMM is fundamentally infeasible, as well as impractical from a performance and engineering standpoint.
Their paper provides a detailed overview of why virtual machines appear different from native hardware. They organize the different ways virtual machines can be detected into three different categories:
Logical Discrepancies: According to Garfinkel et al., logical discrepancies are differences in the interfaces of real and virtual hardware. For example, differences in the execution of some nonvirtualizable instructions,[1] such as SIDT, SGDT, and SLDT, allow inspection of the privileged state from the user-level. Faithful emulation of these instructions is not required for compatibility, so VMMs do not go out of their way to be transparent here. Another example is the chipset emulated by a VMM. VMware always emulates an Intel i440bx chipset, which can lead to strange hardware configurations such as two AMD Operton CPUs running on an Intel motherboard from a time period where SMP motherboards were not available.
[1] We call an instruction nonvirtualizable if no trap is generated when it is executed by a guest application. Without a trap, the VMM has no opportunity to change the instruction's behavior. This is the case for almost all instructions but becomes problematic when the privileged state can be inspected because that reveals the presenced of a VMM.
Resource Discrepancies: Because the VMM needs to share physical resources across all guest operating systems, resource availability is not always guaranteed or equally available. An easy example is the translation look-aside buffer (TLB). Virtual address mappings in the VMM and the guest OS both compete for a constrained resource. By cleverly changing page table entries, it is possible to measure the size of the TLB. An adversary could detect the presence of a VMM by first executing the sizing algorithm without causing any VMM invocations and then measuring the size again while interleaving VMM invocations. When running under a VMM, the second time the sizing algorithm is run, the TLB will appear smaller.
Timing Discrepancies: We already mentioned the difficulty of evading timing-based detection earlier in the chapter. In particular, evading detection becomes almost impossible when remote timing sources are available to an adversary. Timing differences are not only due to virtualization overhead; these could potentially be fixed by running on faster hardware or software. When running under a VMM, the latency of any two operations may change over time. Garfinkel et al. provide device virtualization as a compelling example of timing anomalies: A PCI device register may take a hundred cycles to read on physical hardware but might be available with only a single cycle when the virtual hardware register is in the processor cache. Due to caching behavior, the variance of this operation on a VMM can be much higher than one would expect from hardware.
The logical discrepancies when executing privileged instructions such as SIDT, SGDT, and SLDT on Intel processors have been described in detail by Robin and Irvine in 2000 [73]. These instructions allow the kernel to discover the location of the interrupt, global, and local descriptor tables, respectively. As these tables are used very frequently, VMware does not use binary translation to gate access to them but instead provides different tables at different locations to the guest OS. As a result, it is possible to detect the presence of a virtual machine just by checking if the locations of these tables correspond to a host or guest OS. For example, the SIDT instruction stores the location and size of the interrupt descriptor table in a 6-byte memory array. Based on Robin and Irvine's paper, Tobias Klein created a VMware Fingerprint Suite in July 2003, called Scoopy Doo, to detect different versions of VMware [48]. Klein's scoopy code uses the SIDT instructions among other techniques to determine the presence of a VMM. His code can even tell which Windows or VMware version is running. In November 2004, more than a year later, Joanna Rutkowska also made use of SIDT to create the now infamous red pill VMM detection code [74]:
/* VMM detector, based on SIDT trick * written by joanna at invisiblethings.org * should compile and run on any Intel based OS * http://invisiblethings.org */ #include <stdio.h> int main () { unsigned char m[2+4], rpill[] = "\x0f\x01\x0d\x00\x00\x00\x00\xc3"; *((unsigned*)&rpill[3]) = (unsigned)m; ((void(*)())&rpill)(); printf ("idt base: %#x\n", *((unsigned*)&m[2])); if (m[5]>0xd0) printf ("Inside Matrix!\n", m[5]); else printf ("Not in Matrix.\n"); return 0; } |
There are also some ways how to prevent the detection of VMware's presence. The simple ways of detecting virtual machines can be prevented when hardware virtualization such as Pacifica (SVM) or Vanderpool (VT) is available. However, as mentioned earlier, even with hardware assistance, timing discrepancies are still going to give away the VMM to a determined adversary. Although we have just argued that high-interaction honeypots based on virtual machines are inherently detectable, that should not be discouraging. Many interesting insights can still be gained from them. Adversaries in general do not make any special effort to detect virtual machines, and virtual machines are becoming more popular for general purpose use because they are so much easier to manage than physical machines.
As it turns out, VMware also has some undocumented "features" to hide its presence. These were published by Liston and Skoudis in a presentation of how to thwart virtual machine detection [52]. If you add the following lines to the .vmx file of your virtual machine, the tricks used by RedPill, Scoopy, and some other tools are prohibited:
isolation.tools.getPtrLocation.disable = "TRUE" isolation.tools.setPtrLocation.disable = "TRUE" isolation.tools.setVersion.disable = "TRUE" isolation.tools.getVersion.disable = "TRUE" monitor_control.disable_directexec = "TRUE" monitor_control.disable_chksimd = "TRUE" monitor_control.disable_ntreloc = "TRUE" monitor_control.disable_selfmod = "TRUE" monitor_control.disable_reloc = "TRUE" monitor_control.disable_btinout = "TRUE" monitor_control.disable_btmemspace = "TRUE" monitor_control.disable_btpriv = "TRUE" monitor_control.disable_btseg = "TRUE" |
Earlier in Chapter 2, we already modified the .vmx file, which is the configuration file used by VMware to customize virtual machines. With the help of the instructions in the listing, several options of VMware are disabled. You have to make these changes after the virtual machine is powered off. As a result, detecting the presence of VMware becomes harder, but it also has the side effect that the VMware tools are then broken, and also the normal communication channel between host and guest system, which could, for example, be used to copy and paste text between both, cannot be used anymore. Since these are undocumented features, beware of other side effects, and use this technique at your own risk! More information about these changes is available in the presentation by Liston and Skoudis [52].
We described QEMU in Section 2.2.4 as a virtual machine emulator. Emulation instead of virtualization has the benefit that the behavior of nonvirtualizable instructions such as SIDT can be emulated faithfully. As a result, many of the techniques for detecting VMware do not apply to QEMU. Unfortunately, this assumes that QEMU provides correct emulation for all CPU instructions and that it is much more difficult than it seems. Peter Ferrie from Symantec published a very comprehensive overview on how to attack virual machine emulators [26]. He described the following techniques for detecting the presence of QEMU:
When running under an AMD CPU, the CPUID instruction does not return the correct value. Ferrie's example code first checks if QEMU is running on an AMD system. If it is, EAX = 80000000h is used to return the highest extended function supported. This test is meant to ensure that the CPUID instruction accepts EAX = 80000002h for retrieving the processor brand string. Under QEMU, it is QEMU Virtual CPU version ... rather than AMD [processor name] Processor.
A similar problem with CPUID also exists when checking for an Easter egg. Using CPUID with EAX = 0x8FFFFFFF on an AMD K8 and K8 processor returns IT'S HAMMER TIME in EBX, EC X, and EDX, whereas QEMU returns nothing.
Ferrier mentions several other ways to detect QEMU in his paper. In addition to them, timing attacks are also possible against QEMU. They operate in a similar fashion as timing attacks against VMware or other virtual machine monitors.
User-Mode Linux (UML) is a very specialized virtual machine. It supports running a Linux kernel from within Linux itself. To make the following discussion a little bit easier, we call the initial Linux kernel the host kernel (or host OS), while the one started by the command linux is called the guest OS. The guest OS runs "above" the host kernel, all in user-land. This is somewhat similar to the VMware architecture described in Figure 9.4. UML is essentially a tweaked kernel, able to run in user-land, and it requires that you provide a filesystem containing your preferred Linux distribution.
There has been some interest to use UML as a honeypot [98], and although we are going to discuss UML, we strongly encourage you to find a different solution for your honeypots. By default, UML executes in Tracing Thread (TT) mode. One main thread ptrace()s each new process started in the guest OS. On the host OS, you can see this tracing with the help of ps:
host$ ps a [...] 1039 pts/6 S 0:00 linux [(tracing thread)] 1044 pts/6 S 0:00 linux [(kernel thread)] 1049 pts/6 S 0:00 linux [(kernel thread)] [...] 1066 pts/6 S 0:00 linux [(kernel thread)] 1068 pts/6 S 0:00 linux [/sbin/init] 1268 pts/6 S 0:00 linux [ile] 1272 pts/6 S 0:00 linux [/bin/sh] 1348 pts/6 S 0:00 linux [dd] [...] |
You can see the main thread (PID 1039) and several threads that are ptrace()d: some kernel threads (PID 1044 – 1066), init (PID 1068), ile (PID 1268), a shell (PID 1272), and dd (PID 1348). You can get access to similar information from the guest OS if hostfs, a module to mount a host OS directory into the UML filesystem, is available:
uml# mount -t hostfs /dev/hda1 /mnt uml# find /mnt/proc -name exe | xargs ls -l
When used with default values, UML is not meant to be hidden as the output of dmesg shows:
Code View: uml$ dmesg
Linux version 2.6.10-rc2
...
Kernel command line: ubd0=[...]
...
Checking that ptrace can change system call
numbers...OK
Checking syscall emulation patch for ptrace...
missing
Checking that host ptys support output SIGIO...Yes
Checking that host ptys support SIGIO on close...
No, enabling workaround
Checking for /dev/anon on the host...Not
available (open failed with errno 2)
NET: Registered protocol family 16
mconsole (version 2) initialized on [...]mconsole
UML Audio Relay (host dsp = /dev/sound/dsp,
host mixer = /dev/sound/mixer)
Netdevice 0 : TUN/TAP backend -
divert: allocating divert_blk for eth0
...
Initializing software serial port version 1
/dev/ubd/disc0: unknown partition table
...
|
The preceding output clearly identifies UML and is available to any adversary gaining access to the system. The usage of the TUN/TAP backend as a network device is another indicator of UML. Yet another giveaway is the fake IDE device, called /dev/ubd*, that UML uses instead of a real hard disk. Simply checking the contents of /etc/fstab or executing the command mount allows an adversary to notice the presence of UML. To hide that information, it is possible to start UML with the options fake_ide and fakehd. However, this still leaves some information detectable, like the major number of the disk device as 98(0x62), which is different from the one for IDE or SCSI drives.
UML can also be easily identified by taking a look at /proc. Most of the entries in this directory show signs of UML as the following two examples show: In the first example, the file /proc/cpuinfo, which contains CPU and system specific information, allows us to determine this is a UML system in TT-mode. In the second example, the contents of /proc/ksyms give away UML, too.
$ cat /proc/cpuinfo processor : 0 vendor_id : User Mode Linux model name : UML mode : tt [...] $ egrep "uml|honey" /proc/ksysms a02eb408 uml_physmem a02ed688 honeypot |
In addition, the files iomen, filesystems, interrupts, and many others look suspicious and allow fingerprinting of UML. To make identifying UML a little bit harder, it is possible to use hppfs (Honeypot procfs, [37]) and customize the entries in the /proc hierarchy. However, this is a time-consuming and error-prone task.
Another place to investigate is a process's address space. The file /proc/-self/maps contains the currently mapped memory regions and access permissions of the current process. On the host OS, the address space looks as follows:
The first column shows which regions have been mapped into the process's address space. The second column is a set of permissions (r = read, w = write, x = execute, and p = private), and the third column in this listing is the pathname. In comparison, the address space inside the guest OS looks a little bit different:
uml:~# cat /proc/self/maps
08048000-0804c000 r-xp [...] /bin/cat
0804c000-0804d000 rw-p [...] /bin/cat
0804d000-0806e000 rw-p [...]
40000000-40016000 r-xp [...] /lib/ld-2.3.2.so
40016000-40017000 rw-p [...] /lib/ld-2.3.2.so
40017000-40018000 rw-p [...]
4001b000-4014b000 r-xp [...]
/lib/tls/libc-2.3.2.so
4014b000-40154000 rw-p [...]
/lib/tls/libc-2.3.2.so
40154000-40156000 rw-p [...]
9ffff000-a0000000 rw-p [...]
beffe000-befff000 ---p [...] |
The highest address, which indicates the end of the stack, is slightly different in the guest OS. Depending on the amount of available memory, the end of the stack is usually at 0xc0000000. However, in the guest OS, it is 0xbefff000. The reason for this is that UML maps the Linux kernel into the address space of every process and the kernel takes up the space between 0xbefff000 and 0xc0000000. This means that each process can access, change, or do whatever it wants with the UML kernel. Once an adversary gains root in the guest OS, she can break out of UML and start compromising the host OS.
To fix most of these problems, it is possible to start UML either with the argument honeypot or in skas mode (Separate Kernel Address Space [82]). However, using skas requires manual patching of UML, and in the past, the resulting host kernel was quite unstable. Placing the kernel into a separate address space makes it more difficult for an adversary to corrupt UML — for example, it is no longer possible to directly modify kernel memory.
|
|