Safety Features Built Into the Java Virtual Machine

Once the Java virtual machine has loaded a class and performed passes one through three of class file verification, the bytecodes are ready to be executed. Besides the verification of symbolic references (pass four of class file verification), the Java virtual machine has several other built-in security mechanisms operating as bytecodes are executed. These mechanisms, most of which are elements of Java's type safety, are listed in Chapter 1 as features of the Java programming language that make Java programs robust. They are, not surprisingly, also features of the Java virtual machine:

• type-safe reference casting
• structured memory access (no pointer arithmetic)
• automatic garbage collection (can't explicitly free allocated memory)
• array bounds checking
• checking references for null

By granting a Java program only type-safe, structured ways to access memory, the Java virtual machine makes Java programs more robust, but it also makes their execution more secure. A program that corrupts memory, crashes, and possibly causes other programs to crash represents one kind of security breach. If you are running a mission critical server process, for example, it is critical that the process doesn't crash. This level of robustness is also important in embedded systems, such as a cell phone, which people don't usually expect to have to reboot. Another reason unrestrained memory access would be a security risk is because a wily cracker could potentially use it to subvert the security system. If, for example, a cracker could learn where in memory a class loader is stored, it could assign a pointer to that memory and manipulate the class loader's data. By enforcing structured access to memory, the Java virtual machine yields programs that are robust, but also frustrates crackers who dream of harnessing the internal memory of the Java virtual machine for their own devious plots.

Another safety feature built into the Java virtual machine--one that serves as a backup to structured memory access--is the unspecified manner in which the runtime data areas are laid out inside the Java virtual machine. The runtime data areas are the memory areas in which the Java virtual machine stores the data it needs to execute a Java application: Java stacks (one for each thread), a method area, where bytecodes are stored, and a garbage-collected heap, where the objects created by the running program are stored. If you peer into a class file, you won't find any memory addresses. When the Java virtual machine loads a class file, it decides where in its internal memory to put the bytecodes and other data it parses from the class file. When the Java virtual machine starts a thread, it decides where to put the Java stack it creates for the thread. When it creates a new object, it decides where in memory to put the object. Thus, a cracker cannot predict by looking at a class file where in memory the data representing that class, or objects instantiated from that class, will be kept. What's worse (for the cracker) is the cracker can't tell anything about memory layout by reading the Java virtual machine specification either. The manner in which a Java virtual machine lays out its internal data is not part of the specification. The designers of each Java virtual machine implementation decide which data structures their implementation will use to represent the runtime data areas, and where in memory their implementation will place them. As a result, even if a cracker were somehow able to break through the Java virtual machine's memory access restrictions, they would next be faced with the difficult task of finding something to subvert by looking around.

The prohibition on unstructured memory access is not something the Java virtual machine must actively enforce on a running program; rather, it is intrinsic to the bytecode instruction set itself. Just as there is no way to express an unstructured memory access in the Java programming language, there is also no way to express it in bytecodes--even if you write the bytecodes by hand. Thus, the prohibition on unstructured memory access is a firm barrier against the malicious manipulation of memory.

There is, however, a way to penetrate the security barriers erected by the mechanisms that support type safety in a Java virtual machine. Although the bytecode instruction set doesn't give you an unsafe, unstructured way to access memory, there is a way you can go around bytecodes: native methods. Basically, when you call a native method, Java's security sandbox becomes dust in the wind. First of all, the robustness guarantees don't hold for native methods. Although you can't corrupt memory from a Java method, you can from a native method. But most importantly, native methods don't go through the Java API (they are how you go around the Java API) so the security manager isn't checked before a native method attempts to do something that could be potentially damaging. (This is, of course, often how the Java API itself gets anything done. But the native methods used by the Java API are "trusted.") Thus, once a thread gets into a native method, no matter what security policy was established inside the Java virtual machine, it doesn't apply anymore to that thread, so long as that thread continues to execute the native method. This is why the security manager includes a method that establishes whether or not a program can load dynamic libraries, which are necessary for invoking native methods. Untrusted applets, for example, aren't allowed to load a new dynamic library, therefore they can't install their own new native methods. They can, however, call methods in the Java API, methods which may be native, but which are always trusted. When a thread invokes a native method, that thread leaps outside the sandbox. The security model for native methods is, therefore, the same security model described earlier as the traditional approach to computer security: you have to trust a native method before you call it.

One final mechanism built into the Java virtual machine that contributes to security is structured error handling with exceptions. Because of its support for exceptions, the Java virtual machine has something structured to do when a security violation occurs. Instead of crashing, the Java virtual machine can throw an exception or an error, which may result in the death of the offending thread, but shouldn't crash the system. Throwing an error (as opposed to throwing an exception) almost always results in the death of the thread in which the error was thrown. This is usually a major inconvenience to a running Java program, but won't necessarily result in termination of the entire program. If the program has other threads doing useful things, those threads may be able to carry on without their recently departed colleague. Throwing an exception, on the other hand, may result in the death of the thread, but is often just used as a way to transfer control from the point in the program where the exception condition arose to the point in the program where the exception condition is handled.