Thesis Committee

Stephen Taylor, Ph.D. (Chair)

George Cybenko, Ph.D.

Eric Hansen, Ph.D.

Jeff Boleng, Ph.D.

Abstract

Historically, full memory encryption (FME) has been propounded as a mechanism to mitigate vulnerabilities associated with code and data stored in the clear (unencrypted) in random access memory. Unfortunately, until recently the CPU-memory bottleneck has represented a roadblock to using this concept to design usable operating systems with acceptable overheads.  Recently however, a variety of commodity processors, including the Intel i7, AMD bulldozer, and multiple ARM variants, have emerged that include security hardware -- in particular, encryption engines -- tightly integrated on-chip. By taking advantage of this new hardware and integrating it at the core of an operating system it will be possible and practical to protect data via encrypting code and data in RAM.   This thesis explores this idea and introduces a collection of novel operating system technologies that provide automated, transparent confidentiality and integrity protection via memory encryption.  These techniques raise the difficulty for attackers, making it significantly more challenging to determine the vulnerabilities present on a system, apply the same attack vector against multiple hosts, steal sensitive information, reverse engineer code, modify data at rest or in flight, and inject code onto a platform.

To examine the underlying hypothesis that vulnerabilities can be eliminated, with reasonable performance impact, using security-enhanced commodity processors to encypt memory, a secure microkernel was ported to the ARM Cortex A8 architecture.  This microkernel was extended with novel operating system techniques to provide memory encryption at process segment granularity.  These techniques were developed and tested both with and without caching mechanisms to represent several classes of processors including low-power microcontrollers to more powerful application processors.  System performance was validated through the use of three different benchmark applications comparing the unprotected performance against that of the protected (encrypted RAM) system.  Additionally, the confidentiality and integrity protections afforded by ME were validated.  For the average mobile workload the overhead is approximately 1.3% and the results support the hypothesis.