Lisa Lorenzin, principal solutions architect, Juniper Networks
Lisa Lorenzin, principal solutions architect, Juniper Networks

Network access control (NAC) prevents the spread of malware by restricting network access for infected or vulnerable endpoints. One major challenge to the security of NAC is the “lying endpoint problem,” which arises when an infected endpoint lies about its health. This reduces the value of NAC, allowing infected machines to gain access to the network.

Several countermeasures exist to catch these “lying” endpoints, falling into three primary approaches: protection, detection, and prevention. Each approach has advantages and caveats; understanding them helps a security administrator determine which one is appropriate for his or her network environment.

Malware: The root of the problem
The first step in fighting malware is to understand it. Though there are thousands of varieties of malware, a few main kinds stand out. A keystroke logger captures everything typed on an endpoint (including passwords) and sends it to malicious parties. Bot software (like Storm and Conficker) adds an infected computer (or “bot”) into a “botnet,” an army of computers controlled by a master criminal. The master criminal can use the botnet to mount distributed attacks on websites and obtain protection money. The botnet can also be rented to spammers seeking stealthy ways to send bulk email. The worst attacks are rootkits, which burrow into a computer, modifying the operating system itself to hide their files and even running processes from standard security controls. Rootkits are increasingly popular with attackers because they provide complete control of an endpoint with minimal chance of detection.

Defending against malware
Before NAC, malware defenses focused either on the network or on the endpoint. Network firewalls, intrusion prevention systems (IPS), and gateway anti-virus deployed in the network control the flow of traffic and inspect it for unauthorized or malicious content. Sensitive traffic is encrypted to protect it from session hijacking or sniffing attacks launched by infected machines elsewhere in the network. Endpoint security software, such as host anti-virus, host IPS, and patch management, attempts to protect the endpoint from infection. Each of these technologies is a point solution, addressing a particular aspect of the overall security problem; greater security is achieved when they are brought together by NAC.

As described above, NAC in its simplest form restricts access for infected or vulnerable endpoints. This integrates network and endpoint malware defenses, which is good. However, a potential weakness of NAC is that it relies on integrity reported by the endpoint. If an infection causes the endpoint to lie about its integrity, then the infected machine may gain access to the network and may, additionally, infect other machines. Fortunately, compromising the endpoint's integrity reporting is generally beyond the capabilities of conventional malware. As a result, the lying endpoint problem primarily occurs when an endpoint is compromised by a rootkit

An obvious approach to addressing the problem of lying endpoints is to protect the endpoint from infection in the first place. As previously mentioned, several endpoint security software options exist, including anti-virus, anti-spyware, personal firewall, and host-based IPS. An automated patch management system helps keep operating system and software applications current, protecting against known vulnerabilities.

In the real world, however, these protections are limited in their effectiveness; they can be circumvented intentionally, by disabling the protective software, or inadvertently, such as when a device is left offline for long enough that its signature database becomes outdated before it is reconnected. A compromised patch management system can even become a distribution mechanism for malware! Endpoint protection can reduce the likelihood of infection, but not eliminate it.

Another option is to focus on detection, either on the endpoint itself or at the network level. An endpoint scan can be the most effective way of flushing out rootkits and other threats. However, for the scan to be successful, the endpoint must be booted from external media that loads a known-good OS and rootkit detection software. The clean boot enables scanning of the endpoint hard disk for traces of the installed rootkit; since the rootkit doesn't get a chance to run, it can't cover its tracks.

The downside of endpoint scanning is that a clean boot and disk scan is inconvenient and time-consuming; as a result, it's cost-prohibitive to perform regularly on all machines. Also, a rootkit that has managed to penetrate to the firmware will not be detected.

Network monitoring enables an administrator to leverage existing network security infrastructure in pursuit of infected endpoints. By observing endpoints after connection to the network, the monitoring technology can detect inappropriate activity that indicates a compromised machine. Additionally, network monitoring systems can communicate with other NAC components to dynamically change access privileges.

In a fully integrated NAC environment, network monitoring provides the ability to detect and take action against infected endpoints. However, if the attacker has successfully gained access to the network, the infected endpoint may have compromised other systems before detection. Also, network monitoring -- by definition -- only identifies network traffic; an intelligent attacker may avoid detection by limiting his activities to the local machine. Network monitoring can increase the likelihood of detecting infected machines, but still can't eliminate the problem of lying endpoints.

The third approach is to prevent the endpoint from undetectably lying about its integrity state. Prevention, in this case, doesn't eliminate infected endpoints, but does eliminate lying endpoints, by providing reliable information on whether an endpoint can accurately report its own integrity. Hardware security can overcome the limitations of software-based endpoint security approaches by providing hardware support for integrity measurements and enabling reliable remote attestation, thus ensuring the highest possible level of trust in the endpoint.

A NAC environment can leverage the integrity measurement and reporting functions of the Trusted Platform Module (TPM) to validate an endpoint's configuration. A TPM-equipped PC with a trusted BIOS can boot up, take a series of measurements, and store them in the TPM. Measurements could include information about the BIOS, the boot block, and the booting process itself, as well as parts of the OS at startup. Throughout the startup process, and also later, the security and trust status of the system can therefore be interrogated via the TPM.

As a result, TPM integrity reporting can be used to unmask rootkits, thus identifying potential lying endpoints. During the trusted boot sequence described above, platform components are checked before they are run, and the details are stored in the TPM. When the endpoint connects to the network, the TPM can securely send the platform security measurements to a Policy Decision Point (PDP) along with the platform's other self-reported integrity measurements; the PDP would compare the data against a set of known good configurations. If the information matches, the endpoint configuration is OK; if not, then the endpoint is known to be non-compliant, and it has the potential to be lying about its integrity.
Unfortunately, there's still no silver bullet for network security. As with the other two approaches, there are caveats to hardware-based prevention of lying endpoints. One factor is hardware attacks; an attacker who gains access to the physical connection between the TPM and the motherboard can attempt to circumvent the TPM. Another is operational considerations; broad deployments of TPM-enabled NAC require an infrastructure to support key management if the TPM is used to report user or platform identification, and to create and maintain the registry of known good configurations.

Another issue is the complexity of current operating systems and applications, which provide a challenging number of dynamically linked libraries (DLLs), drivers, and services to be hashed and verified. Finally, privacy concerns are also a widely discussed aspect of TPM deployment. The technology is designed to provide a spectrum ranging from most privacy protection to most visibility and accountability, and each enterprise must determine where their policies place them along that spectrum.

Lying endpoints are a critical challenge to network security. No single solution is perfect, and each approach has its advantages and caveats; however, combined solutions can offer an effective response. Current protection mechanisms such as endpoint security software are necessary, but prove inadequate in the real world. Detection helps, but has serious drawbacks: regular, global clean boot/disk scans are unrealistic, and network monitoring can only alert you to the problem you already have. Hardware security, integrated with NAC, offers a new approach to preventing the lying endpoint problem.