Designing with Dynamic Extension

How Dynamic Extension Works in Java and How to Use it in Your Designs

by Bill Venners
December 15, 1998

First published in JavaWorld, December 1998
Summary
In this installment of Design Techniques, I take a look at a less commonly understood aspect of Java's architecture: dynamic extension. I discuss the two kinds of dynamic extension, forName() and class loaders, and offer guidelines on how to use these tools to make your programs more customizable.

At the Second Annual JavaOne Developer Conference in 1997, I met a programmer from Mexico named Gerardo Horvilleur. Gerardo had created a product that he was publicizing from a booth on the convention floor. His product interested me because in those days I was working on my Java virtual machine book, and his product was one of the few I'd seen that seized new design opportunities made possible by Java's architecture.

Gerardo's product, a Java library called Montage, enabled Java applets or applications to display a special kind of image. The image was special because it was defined not just by data, but also by code.

To display one of these images, the Montage library would first show a background image (stored as plain old image data), then it would dynamically load and execute some Java code (called painters) that massaged the background image. To massage the background image, a painter could process the image to produce a special effect or animate the image -- anything that could be done to an image with Java code a painter could do. The result was the Montage image.

This product stoked my inner nerd sufficiently enough that I wanted to know the details of how he did it. I arranged a meeting with Gerardo in the Hacker's Lounge, a social gathering spot at JavaOne.

There he described the inner workings of the class loader he used to load painters. Our conversation drifted to class loaders, in general, and to Java's overall architecture. Gerardo was very enthusiastic about Java and excited in particular about the new opportunities made possible by its architecture.

Eventually, Gerardo and I left the Hacker's Lounge and headed for a JavaOne session. As we made our way through the crowd, he continued his animated discussion of the benefits of Java.

One thing he said that stuck in my mind was: "When you look at C++'s linking model, it is really just C's linking model with mangled names tacked on. If you look at C's linking model, it is really just Fortran's linking model. But when you look at Java's linking model, it doesn't look like any of those. That's the significant difference about Java!", he exclaimed. "It's the linking model!"

In this article, I'll discuss how to take advantage of one of the more interesting design opportunities made possible by Java's linking model: dynamic extension. As a background to the design guidelines, I'll give an overview of Java's linking model, touching on topics such as name spaces, forName(), and class loaders.

For the full details on these topics, I'll refer you to Chapter 7 of my book Inside the Java Virtual Machine, which, partly in honor of my conversation with Gerardo, I named "The Linking Model." In this chapter, I explain in painstaking detail the process of dynamic linking in the JVM, and show how to write and use class loaders.

But you needn't rush to the bookstore to read this chapter; I have excerpted it in its entirety on my Web site. (See the Resources section for a link to this material.)

Dynamic extension
Java's architecture enables you to write programs that dynamically extend themselves at runtime. Java programs can dynamically extend themselves by choosing at runtime classes and interfaces to load and use.

Looked at another way, dynamic extension means that at compile time, you don't necessarily need to know about all the classes and interfaces your program will use at runtime. In fact, some of those classes and interfaces may not even exist when you do your compile.

Java has two ways to do dynamic extension: forName(), and class loaders. forName(), a static method in java.lang.Class, is the simple, straightforward way to do dynamic extension. Class loaders, subclasses of java.lang.ClassLoader, are the more complicated (and more powerful) way to do dynamic extension.

In a Java program that uses dynamic extension, irrespective of whether it uses forName() or class loaders or both, names of types (classes and interfaces) will be passed around in the program as Strings. To request a certain class be loaded, the program will pass as a String the fully qualified name of the desired type to forName() or the class loader. Because the type name is handed to forname() or the class loader as a String at runtime, the program can be written such that the actual contents of the Strings (the names of the types your program will load at runtime via dynamic extension) are not known at compile time.

Class loaders
Whether you knew it at the time, if you've run a Java program, you have already been using class loaders. Every type (class or interface) used by a Java program must be loaded into the Java virtual machine (JVM), and the JVM loads types through class loaders.

Java has two kinds of class loaders -- the primordial class loader and class loader objects. The primordial class loader is sometimes called the system class loader or the default class loader. Class loader objects are sometimes called custom class loaders.

The difference between these two kinds of class loaders is important to understand. Whereas the primordial class loader is part of the JVM implementation, class loader objects are part of the running Java application.

For example, when I downloaded the JDK for Windows 95, I got a JVM. This JVM, to the best of my knowledge, is written primarily in C. In my case, the primordial class loader is a part of the C program that defines my JVM.

Armed with this installation of Java on my computer, I can then write and run a Java application that uses class loader objects. In this case, I would define class loaders in Java, which my Java application would instantiate and use.

So the difference is this: The primordial class loader (there is only one of these per JVM implementation) is designed and written by the creators of each JVM. Class loader objects are designed and written by Java programmers. Class loader objects are defined as classes (subclasses of java.lang.ClassLoader) and instantiated into regular Java objects on the heap.

Whenever the JVM loads a class or interface, it will use either the primordial class loader or a class loader object. Because it is part of the JVM implementation, one (and only one) primordial class loader is always available to a running application. Class loader objects, by contrast, behave like objects. At any one time during an application's lifetime, the application may have zero to many class loader objects in existence and in use.

Name spaces
When you start a Java application, you give the name of a class that has a main() method to a JVM. The JVM starts executing Java at the indicated main() method, and the application is off and running.

Each application lives inside its very own "instance" of the JVM, with memory areas that are private just to that application. Although different threads of the same application can all access a common heap, for example, threads in one Java application can't directly access the heap in another Java application. Each Java application gets its own Java virtual machine.

Inside a JVM, each class loader (be it primordial or object) gets its own name space. A name space is simply the set of fully qualified names of all the classes and interfaces that a class loader has loaded so far. A fully qualified name of a class is the name of its package, plus a dot, plus its simple name. For example, the fully qualified name of the Hashtable class from the java.util package is java.util.Hashtable.

Within a single name space, all fully qualified names are unique. Two different name spaces, however, can contain identical fully qualified names. This architecture enables a single Java application to load and use two (or more) different classes that have the same fully qualified name. So long as those like-named classes are loaded by different class loaders, which will cause the classes to be placed into different name spaces, the Java application will be able to load and use them.

As an example, Figure 1 shows a snapshot of two name spaces in a single Java application. Two fully qualified names are common to both name spaces: java.lang.Object and Mouse. The upper name space contains two other fully qualified names (Animal and Cat) that don't appear in the lower name space. Likewise, the lower name space contains two fully qualified names (Device and Keyboard) that don't appear in the upper name space. Within each name space, however, all names are unique.


Figure 1. Two name spaces

All of this is fine and good, but begs the question: How are the names in a name space related to the actual types being named? When a JVM loads a type (class or interface), it parses and extracts information about the type from a class file. It places this information (in some data structure devised by the people who implement the JVM) into a logical area of the JVM's memory called the method area. So when the JVM loads a class, a chunk of type data is added to the method area and a fully qualified name is inserted into a name space. Every fully qualified name in every name space is associated with a chunk of data in the method area that defines that named type.

For example, Figure 2 shows the relationship between names (Mouse and java.lang.Object) in the name spaces of the previous example and type data in the method area. One fully qualified name that both name spaces contain is java.lang.Object. In the case of this name, both entries (one in each name space) are associated with the same chunk of type data in the method area.

By contrast, although both name spaces also contain the fully qualified name Mouse, each entry (one in each name space) is associated with a different chunk of type data in the method area. In the upper name space, the name Mouse is associated with a chunk of type data that describes a small, furry animal with a pink nose. In the lower name space, the name Mouse is associated with a chunk of type data that describes a device you attach to your computer.


Figure 2. Names and definitions

Dynamic linking and name spaces
When you compile a Java program, you get one class file for each type (class or interface) you define in source code. The class file is an intermediate compiled binary format for the type. Class files, which haven't been linked, contain symbolic references to other types in a list called the constant pool. At runtime, the JVM dynamically links the Java application by resolving the symbolic references contained in the constant pools of class files. This process is called constant pool resolution.

Name spaces in the JVM arise from a simple rule that all JVMs must follow when they resolve the symbolic references contained inside class files. Sometimes, an entry in a constant pool may refer symbolically to a type that hasn't yet been loaded. When such entries are resolved, the JVM must load the referred-to type. Because all types must be loaded by a class loader, the JVM must at that point decide which class loader to ask to load the type.

To choose a class loader, the JVM uses this simple rule:

The Resolution Rule - The JVM loads referenced types via the same class loader that loaded the referencing type.

An example with no dynamic extension
This one fundamental "Resolution Rule" of Java's linking model is what generates name spaces inside a JVM. If a class named Cat, for example, contains a symbolic reference to a class named Mouse, the JVM will load Mouse via the same class loader that loaded Cat. Because Mouse is loaded via the same class loader that loaded Cat, Mouse will end up in the same name space as Cat.

Thus, given these classes...

// In file dynaext/ex1/Cat.java
public class Cat {
    public static void main(String[] args) {
        Rodent myToy = new Mouse();
        myToy.scurry();
    }
}

// In file dynaext/ex1/Rodent.java
public class Rodent {
    public void scurry() {
        System.out.println("Rodent scurrying");
    }
}

// In file dynaext/ex1/Mouse.java
public class Mouse extends Rodent {
    public void scurry() {
        System.out.println("Mouse scurrying");
    }
}

the name spaces of the Cat application (just after scurry() is invoked on the Rodentreference) will look as shown in Figure 3. Because no class loader object is instantiated in this application, only one name space (that of the primordial class loader) exists. Because the primordial loader loaded Cat, the JVM will use the primordial loader to load Mouse.


Figure 3. The rule in action

Dynamic extension with forName()
Class java.lang.Class contains a static method, forName(), that enables you to dynamically load types into your program. For example, the following application, named Cat, dynamically loads classes that descend from Rodent, makes an instance, and invokes scurry() on the resulting object.

// In file dynaext/ex2/Cat.java
public class Cat {

    public static void main(String[] args)
        throws ClassNotFoundException, IllegalAccessException,
               InstantiationException {

        Class c = Class.forName(args[0]);
        Rodent myToy = (Rodent) c.newInstance();
        myToy.scurry();
    }
}

// In file dynaext/ex2/Rodent.java
public class Rodent {
    public void scurry() {
        System.out.println("Rodent scurrying");
    }
}

// In file dynaext/ex2/Mouse.java
public class Mouse extends Rodent {
    public void scurry() {
        System.out.println("Mouse scurrying");
    }
}

In this version of class Cat, the first argument to the application is passed to forName(). (Note that this is an example of a type name being passed around as a String, a telltale sign of an application that uses dynamic extension.) Assume that the application is invoked with the argument "Mouse". This String will be passed to forName(), which will attempt to load the type.

Figure 4 shows what the name spaces look like after scurry() is invoked on the Mouse object. Once again, only one name space, that of the primordial loader, exists. This looks the same as that of the previous application that didn't use dynamic extension.

This is because forName() follows the same rule the JVM follows when it resolves a symbolic reference in the constant pool of a class file. forName() will use the same class loader to load the requested class that loaded the class that contained the code that invoked forName(). Because Cat, the class that invoked forName(), was loaded by the primordial loader, forName() will use the primordial loader to load Mouse.


Figure 4. forName() follows the rule

Class java.lang.Class
To understand dynamic extension, you need to know a bit about class java.lang.Class. For every type that a JVM loads, it creates an instance of class java.lang.Class to "represent" the type to the rest of the application. Given a reference to a type's Class instance, you can access the information that the JVM extracted from the class file that defined the type.

For example, you can find out whether the type is a class or an interface. If it is a class, you can find out if it is abstract and get a reference to the Class instance for its superclass. You can get an array of Class instances for it directly implements, if any. If the SecurityManager allows it, you can even find out what fields and methods the class contains and invoke the methods.

The most important aspect to dynamic extension is that if you have a reference to a Class instance for a non-abstract class, you can instantiate an instance of the class by invoking newInstance() on the Class object. If forName() is successful at loading the type (or if the type was already loaded into the current name space), forName() will return a reference to the Class instance that represents the type. By invoking newInstance() on the Class object returned by forName(), the Cat application creates an instance of class Mouse.

Interacting with a dynamically loaded class
When Cat invokes scurry() on the Rodent reference myToy, the JVM does a dynamic bind and executes Mouse's implementation of scurry(). The application prints out "Mouse scurrying.". This is true even though Cat didn't know about Mouse at compile time. At compile time, Cat knew only about Rodent. This is the most common way to interact with a dynamically loaded class: to invoke methods defined in a supertype (superclass or superinterface) of the dynamically loaded class on an instance of that class.

Another way to interact with a dynamically loaded type is to use reflection to find interesting methods in the object, and to invoke these methods. This approach is taken by bean builder tools.

When you drop a bean into a bean builder, the bean builder dynamically extends itself with the classes of the bean. As the bean builder has no idea what kind of bean is coming, it uses reflection to find methods that adhere to JavaBeans style rules, and uses those methods to manipulate the bean.

Jumping out of the current name space
So, how does an application ever get out of the primordial name space? It does this by creating a class loader object and asking that class loader object to load the type.

To ask a class loader to load a class, simply invoke loadClass() on the class loader object, thereby passing in the fully qualified name of the type to load. The loadClass() method basically performs the same task as forName().

But loadClass() is rebellious -- it doesn't follow the Resolution Rule. Instead of using the same class loader to load the requested type as it loaded the class containing the code that invoked loadClass(), it uses itself to load the class. (After all, it is a class loader.) Thus, it loads the requested type into its own name space.

Here's an example of a Cat application that uses a class loader object to load a Rodent:

// In file dynaext/ex3/Cat.java
public class Cat {
    public static void main(String[] args)
        throws ClassNotFoundException, IllegalAccessException,
               InstantiationException {
        RodentClassLoader rcl = new RodentClassLoader();
        Class c = rcl.loadClass(args[0]);
        Rodent myToy = (Rodent) c.newInstance();
        myToy.scurry();
    }
}
// In file dynaext/ex2/Rodent.java
public class Rodent {
    public void scurry() {
        System.out.println("Rodent scurrying");
    }
}
// In file dynaext/ex2/Mouse.java
public class Mouse extends Rodent {
    public void scurry() {
        System.out.println("Mouse scurrying");
    }
}

This Cat application first instantiates RodentClassLoader (shown below). The resulting object is a class loader object because class RodentClassLoader extends class java.lang.ClassLoader. Cat invokes loadClass() on this class loader object, passing it the first argument to the application. Assuming the first argument is "Mouse", loadClass() will attempt to load Mouse into its name space. If it is successful, loadClass() will return a reference to the Class instance that represents the newly loaded type. After that, this Cat application looks the same as the previous Cat that used forName(). But the name spaces look different.

Figure 5 shows what the name spaces of this Cat application look like after the execution of the scurry() method. The application has two name spaces because it has two class loaders -- the primordial class loader and the class loader object it instantiated, RodentClassLoader. Although every other class was loaded by the primordial loader and is, therefore, in its name space, class Mouse was loaded by the class loader object, so it sits in the class loader object's name space.


Figure 5. Two name spaces

Writing a class loader
Now, the reason you would usually write a class loader in the first place is to customize the loading process. The purpose of any class loader is (given a fully qualified name) to produce a binary stream of data (usually in the Java class file format) that represents that type and to import that type into the JVM. So the designers of each JVM implementation get to decide how their primordial class loader is going to produce the binary stream of data given the fully qualified name.

The JVM that I received in the JDK for Windows 95 looks down the directories and zip files listed in an environment variable called the CLASSPATH. The primordial loaders of other JVMs can do it differently. Often, the reason you write a class loader is because you want to load classes in some other way than the default way provided by the primordial loader.

You may, for example, want to download class files across a network. This is one reason why the class files for applets are loaded through class loader objects. You may want to decipher encrypted class files as they are read in. You may want to extract binary class definitions out of some proprietary database. You may want to load class files that end with the extension .purple instead of .class Or, you may even want to generate class definitions on the fly, so that your Java application can continually make itself smarter.

Because you define class loaders in Java code, you are free to do anything to produce a binary stream of data in the Java class file format given a fully qualified name. (Well, at least you are free to write and compile the class loader. But in some runtime environments, the SecurityManager may prevent class loaders from being instantiated. Untrusted applets, for example, can't create class loader objects.)

Class RodentClassLoader, shown below, loads classes in a special subdirectory of the current directory named hole. This class loader is a simple example of another potential reason for writing a class loader: to load class files not located in the class path.

// In file RodentClassLoader.java
import java.io.*;
import java.util.Hashtable;
public class RodentClassLoader extends ClassLoader {
    public synchronized Class loadClass(String typeName,
        boolean resolveIt) throws ClassNotFoundException {

        // See if type as already been loaded by
        // this class loader
        Class result = findLoadedClass(typeName);
        if (result != null) {
            // Return an already-loaded class
            return result;
        }

        // Check with the primordial class loader
        try {
            result = super.findSystemClass(typeName);
            // Return a system class
            return result;
        }
        catch (ClassNotFoundException e) {
        }

        // Don't attempt to load a system file except
        // through the primordial class loader
        if (typeName.startsWith("java.")) {
            throw new ClassNotFoundException();
        }

        // Try to load it from subdirectory hole.
        byte typeData[] = getTypeFromHole(typeName);
        if (typeData == null) {
            throw new ClassNotFoundException();
        }

        // Parse it
        result = defineClass(typeName, typeData, 0,
            typeData.length);
        if (result == null) {
            throw new ClassFormatError();
        }

        if (resolveIt) {
            resolveClass(result);
        }

        // Return class from hole
        return result;
    }

    private byte[] getTypeFromHole(String typeName) {

        FileInputStream fis;
        String fileName = "hole" + File.separatorChar +
            typeName.replace('.', File.separatorChar)
            + ".class";

        try {
            fis = new FileInputStream(fileName);
        }
        catch (FileNotFoundException e) {
            return null;
        }

        BufferedInputStream bis =
            new BufferedInputStream(fis);
        ByteArrayOutputStream out =
            new ByteArrayOutputStream();

        try {
            int c = bis.read();
            while (c != -1) {
                out.write(c);
                c = bis.read();
            }
        }
        catch (IOException e) {
            return null;
        }
        return out.toByteArray();
    }
}

I'm not going to delve into the nitty gritty details of how to write a class loader here. For that you can refer to Chapter 7 of Inside the Java Virtual Machine, which, as I mentioned above, is excerpted on my Web site. (See Resources for a link to the chapter.) Be sure to read the section entitled "Example: The Dynamic Extension of the Greet Application."

Design guidelines
Now that you've seen how Java's linking model supports dynamic extension, you may be wondering how you should put this cool architecture to use in your designs. The main design insight that I have for dynamic extension (which, while not that profound, is quite useful) is that dynamic extension facilitates customization.

You can think of the chunks of Java code that your application loads dynamically as "plug-ins" or "personality modules." Perhaps you would like users to be able to plug their own algorithms into your product, to customize it to their needs and preferences. You can also design your product such that users can define a class that encapsulates the custom algorithm and then load it into your product dynamically. (This is the strategy pattern from the "Gang of Four" book, by the way, with dynamic extension added.) Or you can customize your own product for individual customers in the same way.

Applets, for example, are a dynamic extension of a Java application fired off by a Web browser. They enable applet programmers to customize the browser. Servlets are dynamic extensions of a Web server that enable servlet programmers to customize the Web server. Resources, which are dynamically loaded based on the current locale, are customizations of internationalized Java programs that enable translators to localize the Java programs.

Interacting with dynamically loaded types
In general, you should interact with dynamically loaded types by instantiating them and invoking methods declared in a superclass or superinterface about which you know. This, for example, is how browsers interact with applets.

Every applet you write must descend from a superclass that the browser knows about, java.applet.Applet. The browser dynamically loads and instantiates your applet and casts the resulting reference to type java.applet.Applet. It then invokes init(), start(), and stop() on your applet, methods which are declared in java.applet.Applet.

The other way to interact with dynamically loaded types (by discovering interesting methods via reflection) is reasonable only if you have absolutely no knowledge of what's coming. If you are writing a bean builder, you'll need to do this. Otherwise, you'll usually write your program such that it expects, as the dynamic extension, some subclass of a known class or some class that implements a known interface.

forName() versus class loader objects
Once you have decided to use dynamic extension, how do you choose between forName() and class loader objects? In general, you should try to use forName() if possible, because forName() is simpler to use and understand.

You should use class loader objects if:

  1. you need name spaces;
  2. you need to customize the load process; or
  3. you need to dynamically replace existing customizations in a live Java application.

Using name spaces
Why would you need name spaces? There are two reasons. Firstly, if your application may encounter name conflicts among the types it loads dynamically, you can use class loader objects to load types with potentially conflicting names into different name spaces. Browsers, for example, usually load applets from different sources into different name spaces. This enables a browser user to visit two different pages containing two applets written by two different people, both of whom may have given the same name to one of the classes that make up their applet.

The second reason you may need name spaces is security. Java's linking model lets you shield types loaded into different name spaces from each other. You can design a system of dynamic extension in which types loaded into one name space can't even see types loaded into other name spaces. This capability is one aspect of Java's built-in security model.

Security is the other reason (aside from avoiding potential name conflicts) that browsers usually load applets from different sources into different name spaces. It keeps untrusted applets from fooling around with trusted applets loaded from another source. However, this kind of security is optional, as the programmer who creates an application that dynamically extends itself via class loaders can decide to shield code in different name spaces from each other. It is also possible (at the programmer's option) to enable types loaded into one name space to get at and interact with types loaded into other name spaces.

  • Customizing the load process -- Another reason you might use class loaders is if you really need to customize the loading process. If you want to download class files across a network, for example, and the primordial loader doesn't know how, you'll need to write a class loader that can do it. This is another reason browsers use class loaders to load applets -- the primordial loader of the JVM used by the browser doesn't know how to download class files across the network.
  • Live replacement -- The third reason you might need a class loader is if you want to swap out a customization as an application is running. In other words, class loaders will enable you to dynamically extend your application with a class named Mouse, and later, to throw away that Mouse and dynamically extend the application with another class also named Mouse. You can't do this with forName() because once a type is loaded into a name space, it can't be removed unless the whole name space is removed.

    With a class loader, you can throw away a particular class loader object by dropping out all references to it, to any instances of any classes it loaded, and to any Class instances for the types it loaded. When you drop all these references, the class loader object and all the classes it loaded become unreferenced and available for garbage collection by the JVM. You can then create a new class loader object with a fresh (and empty) name space, and load the revised versions of the classes you just threw out with the old name space. This is how Web servers (running 24 hours a day, 7 days a week) enable servlet programmers to replace a servlet with a new version of the same servlet.

Next month
In next month's Design Techniques article, I'll talk about designing with type information.

A request for reader participation
I encourage your comments, criticisms, suggestions, flames (all kinds of feedback) about the material presented in this column. If you disagree with something, or have something to add, please let me know.

You can either participate in a discussion forum devoted to this material or e-mail me directly at bv@artima.com.

Resources

This article was first published under the name Designing with Dynamic Extension in JavaWorld, a division of Web Publishing, Inc., December 1998.

Talk back!

Have an opinion? Be the first to post a comment about this article.

About the author

Bill Venners has been writing software professionally for 12 years. Based in Silicon Valley, he provides software consulting and training services under the name Artima Software Company. Over the years he has developed software for the consumer electronics, education, semiconductor, and life insurance industries. He has programmed in many languages on many platforms: assembly language on various microprocessors, C on Unix, C++ on Windows, Java on the Web. He is author of the book: Inside the Java Virtual Machine, published by McGraw-Hill.

Copyright © 1996-2024 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use
Copyright © 1996-2024 Artima, Inc. All Rights Reserved.