Endpoint Security Controls

Another form of intrusion prevention is the host IPS (HIPS). Often referred to as endpoint security controls, a HIPS consists of operating system security controls or security agent software installed on hosts that can include desktops PCs, laptops, or servers. Host IPSs in most cases extend the native security controls protecting an operating system or its applications. Endpoint security controls can monitor local operating system processes and protect critical systems resources. HIPSs fundamentally have two essential elements: a software package installed on the endpoint or agent to protect it and a management system to manage the endpoints or agents.

In most cases, operating systems today split the runtime functions of the operating systems into two concurrently running modes known as Kernel mode and User mode. Kernel mode is the software that has complete access to the operating system hardware; thus, all the software running in Kernel mode can act without restrictions. Generally, the software running in Kernel mode includes the hardware drivers, operating system scheduler, and the application programming interfaces (API). User mode is the software that requires kernel services to execute applications in the form of processes but don’t have direct access to the hardware components of the operating system. There is required protection in the system hardware that separates the two modes so that the User mode applications cannot tamper with the Kernel mode software.

Access control enforcement for an operating system can be done using local system resources (native operating system access control) or remote system resources (RADIUS, TACACS, and so on). The local system of user or process privileges and permissions on the discretion of the logical owner/administrator is known as Discretionary Access Control (DAC). Another local system access control that extends the functionality by using the user’s role in the organization is known as Role-Based Access Control (RBAC) capability. Access control lists (ACL) are often used to define which systems or networks have access and in which direction. Audit trails (system logs) can aid in the detection of system misuse and attacks to protected objects. The same access control mechanism that decides whether to permit or deny access usually provides this audit trail, showing successful and unsuccessful access attempts. Buffer and heap overflow protection is critical for local applications that contain input-validation vulnerabilities. Protection against buffer and heap overflow attacks is often embedded into hardware and operating systems that provide specialized protection against this specific class of threats. Table 1-5 summarizes the features and limitations of endpoint security.

Table 1-5. Features and Limitations of Endpoint Security

Features	Limitations
Identity association, meaning that the endpoint security control can provide the information about the attacker.	Platform flexibility (some operating systems might not support endpoint security controls).
System-specific or customized to protect the system it is protecting and resides on.	Inability to correlate whether a single endpoint or agent is deployed.
Ability to see malicious network data; consequences of network attacks even if encrypted.	Every host requires an agent. Thus, the cost of endpoint security controls can become quite large in some environments and also be quite challenging to manage with only a single or a few administrators to manage the hosts.
Detection of the success of an attack and can take action after the system is stable.	If an attack is successful in accessing the host prior to the endpoint security reacting, the host is compromised.

Host-Based Firewalls

Endpoint security isn’t complete without a form of host-based firewall. There are two basic implementations, which include packet filtering and socket filtering (also known as API call filtering):

• Packet filtering: Host firewalls use stateful and stateless packet filtering, and typically support dynamic applications such as HTTPS, FTP, and so on. Filtering is based on Open Systems Interconnection (OSI) Layer 3 and 4 information, so it can control connections based on host addresses, protocols, and port numbers. Similar in behavior to a network firewall.

• Socket filtering (API call filtering): Controlling application requests to either create an outgoing or accept an incoming connection by filtering network-related API calls. API call filtering is applications aware, so there is no need to require intelligence to support dynamic sessions.

API and System Call Interception

Secondary Security Reference Monitor (SSRM) is an operating system security extension that provides a “second opinion” or layered approach of security by extending and duplicating the functionality of the native operating security model. SSRMs are often third-party extensions for the operating system kernel. They use API interception to insert themselves into the access control path. API interception has a low performance impact while consuming less than 5 percent of additional CPU resources; therefore, most of today’s HIPS products implement SSRM functionality. API interception (also called API hooking) is when an API call is intercepted and the SSRM registers itself as the replacement handler code for the API call it considers important enough to intercept. This allows the SSRM to enforce its own security policy. The SSRM can act as the host firewall, now controlling all applications’ access to the network.

Cisco Security Agent

The Cisco HIPS is Cisco Security Agent (CSA), which complements the Cisco NIPS, protecting the integrity of applications and operating systems. Malicious activity is blocked before damage is done by using behavior-based technology that monitors application behaviors. CSA protects against known and new/unknown attacks. Residing between the kernel and applications, CSA enables maximum application visibility with little impact to the performance and stability of the underlying operating system. A few of the numerous network security benefits CSA offers are as follows:

• Zero-update protection reduces emergency patching in response to vulnerability announcements, minimizing patch-related downtime and IT expenses.

• Visibility and control of sensitive data protect against loss from both user actions and targeted malware.

• Predefined compliance and acceptable use policies allow efficient management, reporting, and auditing of activities.

• System is protected at all times, even when users are not connected to the corporate network or lack the latest patches. This is often referred to as “always vigilant” security.

As stated in the previous paragraph, host IPSs and network IPSs are complementary. Table 1-6 illustrates this point.

Table 1-6. Host IPS (HIPS) and Network IPS (NIPS)

Host IPS	Network IPS
CSA can inspect the behavior of applications (encrypted or nonencrypted).	Requires constant updates for new vulnerabilities.
CSA is a behavior-based HIPS.	Can prevent known attacks.
CSA does not need constant updates.	Can protect complete network.
CSA can protect the host (server, desktop, and so on) efficiently, communicate with IPSs, and stop known and unknown (Day Zero) attacks.	—
CSA cannot “name” the attack or protect unsupported platforms.	—

Antimalware Agents

Antivirus and antispyware are primarily designed to find file-based malware threats and scan the content to identify known patterns of malware. This tends to be a permissive security approach. File and memory content can both contain traces of known malware, and fortunately antimalware scanners can examine both. Some antimalware scanners can perform scanning using the following methods or approaches:

• Using on-demand scanning when the user initiates a thorough system scan.

• Using real-time scanning, which in some cases isn’t as thorough as offline/on-demand, especially if executable code is populated in memory and the files being scanned are busy writing or reading from the file system.

• Using scanning in a scheduled manner in which all files are scanned thoroughly on the endpoint.

Viruses, spyware, adware, Trojan horses, worms that use file-based infections, rootkit software, and general attack tools can all be detected using file-based antimalware software, as long as that type of malware is known (through the malware database) and can be located using the file and memory scanning.

Typically, the antimalware scans files and memory for known patterns of virus code. This is compared to a database of known malware signatures. In some instances for accuracy, a lot of antivirus scanners today require content matching through multiple, independent detectors for the same virus. Scanners that analyze content for suspicious coding tricks, runtime attributes, structure, and behavior associated with malicious code use heuristic antimalware. Heuristics are not that reliable for new viruses and often will use various techniques that weight malicious features to determine whether the code should be classified as malicious. A common antimalware scanning technique is known as code emulation. In code emulation, the antimalware software executes suspicious code in a simple virtual machine that is isolated or sandboxed from the rest of the system. The antimalware scanner can (or attempts to) determine the behavior and actions that the suspicious code performs. The learned behavior is then stored in a database of executable signatures that can detect known patterns of execution to detect the virus in the future.

Data Loss Prevention Agents

Another form of endpoint security is known as Data Loss Prevention (DLP) extensions. DLP controls mobile data distributed on users’ systems to prevent users from accidentally or deliberately transferring sensitive data to uncontrolled systems. Examples of uncontrolled systems would be paper (using printers), open network systems (file sharing), and mobile storage (USB keys, portable hard disks, and so on). There are different forms of implementation when it comes to DLPs, but two common examples would be using content scanning to identify sensitive content (assuming that the content is labeled appropriately with a standardized labeling systems identifying sensitive material) and controlling transfer of data off the system using interception of users’ and applications’ actions.

Cryptographic Data Protection

One of the most discussed and well-known approaches to endpoint security today is file integrity checking to detect unauthorized changes to sensitive files or the system itself. Integrity-checking software calculates a secure fingerprint (HMAC [Hash Message Authentication Code]) for every important file on the system with a secret key. These fingerprints are created when the file(s) are known to be trusted and not modified from their original states. There are periodic rescans of the files and file fingerprints compared to a database of known good fingerprints, which identify whether they have been tampered with.

Integrity checkers rescan files in a specified interval or time, so they can only provide detection of attacks rather than provide real-time detection. It’s important to note that integrity checkers can be compromised with the system, given that they are usually a user-mode application.

Encryption is also an important method to prevent data from being stolen or compromised physically from a system, disk drive, third-party add-on, or file system. The user holds the decryption keys with Windows EFS (Encrypting File System) that are transparently linked to user credentials and provide access to encrypted information. Lost cryptographic keys can lead to sensitive data loss, which is why many security policies require the creation of a backup decryption key. Key generation might be left to the user, which substantially weakens cryptography protection of data if operated poorly. If stolen, an attacker must attempt to decrypt protected information; however, this is very difficult to do if cryptographic implementation and key management are done properly.