New security features in Mac OS X v10.5 Leopard

Introduction

Although the version 10.5 of Mac OS X, also known as Leopard was released nearly one year ago, we think it’s important to talk about it today mainly for two reasons:

• This last release represents a major evolution for the Mac OS X system in terms of security.
• Some of the new introduced security features are even completely new regarding the whole commercial and open-source operating systems.

This article will therefore describe some key features introduced in Mac OS X Leopard.  

“Library randomization” and “NX bit”: how to protect systems against buffer overflow attacks

Even though this feature is completely transparent to the end-user, it is probably one of the most significant improvements brought by Mac OS X v10.5. This new feature, called “Library Randomization” adds some forms of randomization (usually called entropy) in the address space used to load dynamically linked libraries into memory.

In fact, one of the most common vulnerability when talking about application security is the so-called buffer overflow. In this scenario, an attacker provides overly long arguments or parameters to a vulnerable application, thus overrunning the size of the memory allocated to store them. This may cause for example the return memory address of a function to be overwritten and as a consequence the program execution flaw can be modified. The shell code, which is the code the attacker wants to execute on the target system without the user’s consent, may contain calls to system functions hard-coded with memory addresses. On many operating systems, these addresses can be predicted which makes easier for an attacker to remotely execute arbitrary code on such systems. On a Mac OS X Leopard system, these addresses are now randomly chosen at system start-up, which makes almost impossible for someone to exploit these security flaws because in this case the attacker has to guess the correct function memory address among several millions of possibilities.

Note: this feature is also known as ASLR (Address Space Layout Randomization) and has for example been implemented on Microsoft Windows Vista and Microsoft Windows Server 2008. For more information on this topic, refer to this article from Wikipedia.

Still regarding code execution prevention, Mac OS X v10.5 now offers the ability to disable code execution (feature called “NoExecute”, “ExecuteDisable” or “NX bit”) on specific memory regions, ant this for all supported architectures. This allows the system to prevent the processor from executing code in memory originally allocated by an application to store data (stack, heap and so on …).

Quarantine applications and application tagging

Applications that download files from the Internet or receive files from external sources (such as mail attachments) can use the Quarantine feature to provide a first line of defense against malicious software such as Trojan horses. When an application receives an unknown file, it automatically adds metadata (quarantine attributes) to the file. Thus, files downloaded using Safari, Mail, or iChat are tagged with metadata indicating that they are downloaded files and referring to the URL, date, and time of the download. An interesting point here is that this metadata is propagated from the downloaded archive files (such as ZIP or DMG files). Any file extracted from an archive is therefore also tagged with the same information.

This metadata is used by the download inspector to prevent dangerous file types from being opened unexpectedly. The first time a user tries to run an application that has been downloaded, Download Inspector inspects the file, prompts the user with a warning asking whether he wants to run the application, and displays the date, time, and location of the download. The user can then continue to open the application or cancel his attempt if he doesn’t recognize or trust it. After an application has been opened, this message does not appear again for that application and the quarantine attributes are withdrawn.

This new feature allows reducing the number of warnings displayed by the system while attempting to open a potentially dangerous application. Moreover, the file information displayed to the user is now more useful and more relevant.

Signed applications

Mac OS X v10.5 offers the ability for all software developers to sign their applications. The system uses a signature to both verify the application identity (that it’s not a fake program) and its integrity (check that the application has not been modified). All applications shipped with the Mac OS X system are now signed by Apple. Application signing does not provide intrinsic protection, but it integrates with several other features to enhance the whole system’s security.

For example, features such as parental controls, Keychain (the password management applet) and the firewall use application signing to verify that the applications they are working with are the correct and unmodified ones provided by the editor.

Regarding the application firewall, the signature is used to authenticate and check the integrity of application for which the right to access the network has been granted. If the application has not been signed by the editor, the system automatically adds this signature to allow the firewall to identify this program and to check that it remains unmodified.

Application signing can also be used to ensure that one program replacing another is truly an "update", and carry any special security privileges from the initial version to its new version.

This feature brings a better user experience (fewer systematic confirmation requests) and additionally, an efficient protection against executable file alteration attempts (which is a common worms or viruses behaviour).

Mandatory Access Control

Mac OS X v10.5 now implements a support for the mandatory access controls security model, often shortened to MAC. By definition, MAC security policies are security constraints that cannot be defined or overridden by the end-user. These security constraints are for example created by the software developer. To be more precise, we can talk about Mandatory Access Controls when the security policy applied in an information system imposes that protection decisions regarding objects (files or user’s processes) must not be made by the owner of the so-called objet, and when these decisions have to be made and set by that information system.

This kind of security policies contrasts with the traditional and well-known Discretionary Access Controls policies (DAC). Contrary to MAC policies, DAC considers that the owner of an object is in charge of its protection and can therefore modify its access rights according to his preferences (the typical example of DAC policies is access rights management on common UNIX systems).

Therefore, on Mac OS X, security restrictions are defined or can be defined at the operating system kernel level. As a consequence it can not be overridden. Most of mandatory access controls in Mac OS X v10.5 are not visible to users, but they are the underlying technology for several important new features, including application sandboxing, parental controls and a safety net feature for the system backup solution Time Machine. Time Machine illustrates the difference between mandatory access controls and the user privilege model. It allows files within Time Machine backups to be modified or deleted only by programs related to Time Machine. From the command line, no user, not even one logged in as root, can modify or delete files in a Time Machine backup. This mechanism allows the system to prevent unexpected alteration of a system backup, and all forms of malware by modifying or destroying it.

Mandatory access controls are also integrated with the exec system service to prevent the execution of unauthorized applications. This is the basis for the parental control mechanism in Mac OS X v10.5. In the case of the new sandboxing facility, mandatory access controls restrict access to system resources as determined by a special sandboxing profile that is provided for each sandboxed application. This means that even processes running as root can have extremely limited access to system resources.

Application sandboxing

The sandboxing feature helps ensure that applications do only what they are supposed to by placing controls on applications to set security restrictions defining:

• What files they can access,
• Whether they can talk to the network,
• Whether they can be used to launch other applications.

Here we can consider this feature as an implementation of a Role-Based Access Control (RBAC) policy. Each application gets a well-defined role, and the permissions granted to it have to exactly match this role. To achieve this, the sandboxing feature relies on mandatory access controls described in the previous paragraph, that is, implemented policies  at the kernel level. Specific profiles can thus be built for each application that has to be executed in a restricted environment. These profiles basically describe a set of system resources that can or can not be accessed by an application.

By default on Mac OS X v10.5, sandboxing profiles are already defined for services or applications used to ensure the network connectivity. For example, services such as MDNSResponder (the service that enable the support for DNS multicast, needed by the system to be compatible with the "Bonjour" protocol specification) or the Kerberos KDC (Key Distribution Center) are sandboxed by a very restrictive profile. It allows to guard them from abuse by attackers trying to access the system through a vulnerability in those services.

In addition, services such as “quick look” (allowing to display a file preview in one click) or the “SpotLight” daemon (file indexing service) that routinely take untrusted input are also sandboxed. It prevents the system from being compromised by potentially crafted malicious files.

Application-Based firewall

A new application-based firewall introduced in Mac OS X 10.5 makes it easier for inexperienced users to get the benefits of an efficient firewall protection. The new firewall allows or blocks incoming connections on a per-application basis rather than on a per-port (TCP/UDP) basis as firewalls usually do in UNIX operating systems.

Users can restrict firewall access to essential network services (such as those needed for DHCP, BOOTP, IPSec VPNs, and Bonjour connectivity). They can also allow or block access to selected applications on an individual basis. As we already said it above, the application firewall uses digital signatures to verify the identity of applications. If an unsigned application is selected in the firewall (if a rule referring to this application is added), the system automatically signs that application to uniquely identify it.

Expert users can still use the IPFW firewall on the system. Because IPFW handles packets at the protocol-layer of the networking stack (layer 3 and 4 in the OSI model) and that the application firewall is an application layer filter (layer 7), IPFW rules take precedence over application rules.

Secure guest account

The guest account feature allows a user to connect and use a Mac OS X system without the need to enter a password. This feature can of course be enabled or disabled according to the system’s administrator preferences in order to easily grant someone a temporary access to the system. The characteristic of such an account compared to an ordinary unprivileged user account is that all the data associated with this account are erased when the guest closes the session. These data can include among other things desktop preferences, e-mails, web browsing history and cookies, and so on…

This approach has several valuable qualities:

• The guest account exactly appears like an ordinary Mac OS X user account. In other words, all the previously mentioned security constraints such as parental control or sandboxing can also be applied to it. So, this account doesn’t allow a potentially malicious guest user to compromise the host system. The parental control can especially be used to forbid the guest to execute specific applications.
• As the account’s home directory is completely erased when the guest user disconnects, no potentially malicious file can remain on the system. Erasing the home directory is also a security feature preventing the guest user from inadvertently disclosing its potentially confidential information.
• Finally, as the guest account always has the same unique ID, only one guest can use this account at the same time.

However, it appears, if we refer to an article entitled “A Roundup Of Leopard Security Features” published on the Matasano.com blog, that a malicious guest user may perform special actions allowing him to remain resident on the system even after disconnecting. The list of these actions includes the ability to create cron jobs (scheduled tasks) or the ability to mount remote file systems. Obviously, the idea to implement a secure guest account is really interesting and this feature could really be a good Mac OS X specific evolution. However, needless to say, the secure guest account doesn’t bring security protection one might expect.

Conclusion

Mac OS X v10.5 additionally introduces a large number of new security features such as the handling of multiple certificates by a single user, parental control, the system backup solution (Time Machine), the ability to use stronger cryptographic keys to encrypt the contents of a hard disk and finally a new universal VPN client application. That offers a better compatibility with existing VPN solutions on the market. Of course, it was not possible in the context of this article to deal with all the improvements brought by Mac OS X Leopard. However, considering the improvements brought by this last release, it is probably the most significant evolution in terms of security in the history of Mac OS X systems. Among others, the integration of a powerful backup feature to examine the system, is completely new and specific to the Apple system.

We could however deplore that certain features have not been fully or properly implemented as previously mentioned in Thomas Ptacek's article from Matasano.com. For example, the sandboxing feature seems not to be set around the most critical applications such as the Safari web browser or the Mac OS X mail client. As for the Library Randomization feature, it seems that it has been only partially implemented - not all the dynamic libraries benefit from it. Finally, in its documentation, Apple claims that all the applications shipped with the operating system are digitally signed, but it seems not to be the case.

We can however hope that these various issues be progressively fixed with the upcoming system security updates.

For more information

• Mac OS X Leopard - 300+ New Features (Apple)
• Leopard security configuration (Apple)
• Address space layout randomization (Wikipedia)
• A Roundup Of Leopard Security Features (Matasano.com)