Introducing JackPot

Jackpot is a tool for source code examination and transformation that lets you automate the process of inspecting, reporting on, and modifying source files. It was developed by James Gosling and Tom Ball, with the help of Neal Gafter, who made the javac compiler reentrant so Jackpot could use it.

To define the changes you want to make, you create a pattern-based series of production rules and save them in a file. (I think of that file as a filter.) The rules have the form: source-pattern => result-pattern, where => is the production operator. When Jackpot runs, code that matches the source pattern on the left is transformed into the result pattern on the right.

Note: You can also write your own transformation class, instead of a series of rules. It requires knowledge of the Abstract Syntax Tree produced by javac, and an understanding of the Visitor pattern--the most useful pattern there is for operating on tree structures.

Jackpot is based on javac, so anything the compiler can process, Jackpot can process. And in addition to using javac to read the source file coming in, Jackpot uses javac to verify that resulting source code, after all changes have been applied, will still compile. That helps rules-authors to avoid mistakes.

Why You Want It

There is tremendous power in the concept of a rule set. In essence, it captures the expertise of the author. When you run that rule set, you have the expert at your elbow, suggesting changes. The transformations might produce cleaner code, better performing code, more localizable code, or adhere to any other principles that might guide an expert when modifying a class.

In effect, Jackpot puts a variety of experts at your beck and call, so you can ask them:

"If you wanted to make this code (cleaner, faster, localizable, take your pick), how would you modify it?"

In addition to making changes, Jackpot can be useful for examining code, as well. In all, Tom Ball noted several general categories of use for Jackpot:

• Source auditing: Examining code for adherence to conventions (like lint).
• Source archeology: Finding patterns in existing code and figuring out how it works.
• Refactoring: Making the code better.
• Reengineering: Converting to new APIs or new technologies.
• Pretty-printing: Making code more readable and maintainable.

Some of the rule sets that Tom Ball demonstrated were designed to:

• Clean up code (40-some rules that eliminate extraneous code)
• Make code that compiles faster
• Convert code that uses StreamTokenizer to use the new Scanner class
• Simplify loops so they run faster

Some other possible applications for transformation rules include:

• Making code localizable
• Optimizing code for performance
• Removing debug logic before compiling the production version

There is a lot to know in many of those areas. It's difficult to acquire the knowledge necessary to make code that is maintainable and localizable and high-performance and well-documented and makes use of the latest APIs--and oh, by the way, it has to run properly, too. Jackpot, and the library of transformation rules that will surely grow up around it, will make it possible to get the expert advice you need to achieve all of those goals.

Writing Production Rules

When you write a production rules, you include meta-variables that define place holders in the current syntax tree. At the moment, meta-variables are defined as syymbols that have a trailing underscore (although that may change if a better suggestion comes in).

For example, in the expression, if (v == true), v is a literal that only matches the symbol "v" in the source code. But in the expression, if (v_ == true), v_ is a meta-variable that matches any expression in that position in the syntax tree.

Having defined v_, you can now use it the right-hand side of the production. For example: if (v_ == true) => if (v_)

That rule says to change an expression like if (succeeded==true) to if (succeeded) --an obvious simplification.

You can also qualify an expression with a "where" clause like you used to see in mathematics texts. In this case, you introduce the where clause with a double-colon (::) For example, you could rewrite the previous expression like this: if (v_ == true) => if (v_) :: v_ instanceof Boolean

(You can also check to see if v_ extends some class, or is a superclass of it, among other things.)

Fast Operation, with Simple, Safe Undo

Under the Hood, Jackpot stores, examines, and/or operates on three trees provided by javac:

• Abstract Syntax Tree (AST) (syntax hierarchy--code nesting)
• Type System (class hierarchy--inheritance)
• Symbol Table (instantiation hierarchy--ownership)

Using those three hierarchies allows precise operation. That operation occurs at high speed, for a variety of reasons:

First, the javac compiler isn't generating byte codes and it's not doing optimizations. It only parses the class files and produces an AST (abstract syntax tree). So that part of the process takes seconds, even for a large number of files.

Second, Jackpot is optimized for the AST. Patterns are found as early in the tree in possible, before the exponential explosion drains performance.

Third, although Jackpot creates a new AST for the result, it shares copies of existing nodes. So a minimum number of changes made, and most of those involve swapping a few pointers around.

Note: Node-sharing posible, because no node knows what its parent is.

When Jackpot modifies a node, that node and its parents are copied to to the new tree. When copying parents, Jackpot follows a linear chain up to the root, without the exponential explosion that occurs when you follow the branches down towards the leaves, so it operates in essentially linear time.

Undo is accomplished by removing the node, along with its parents--again, a linear operation that is both fast and safe, because the original nodes remain untouched.

In short, Jackpot runs very quickly, in essentialy linear time (Order N, where N is the product of the number of rules to process and the amount of code you're operating on).

Format Control

You can use a rule set to reformat an entire file, although in many cases that's undesirable. You can also use them to format the results of source code modifications.

Reformatting an entire file is useful when you're the sole author, especially when you've inherited a project. But for a large project with multiple authors, where the files are under source control, massive reformatting generally isn't a good idea--unless everyone knows it's coming and all files are checked in first. Otherwise, format changes will swamp the real differences, making them virtually impossible to locate.

When making a small change to a source file, therefore, it's generally not a good idea for Jackpot to reformat the entire file. On the other hand, it's impractical for Jackpot to deduce the style that happens to exist in that file. So when Jackpot makes changes, it uses your specified style settings (of which there are many) to format the new code.

As a result, the new code may or may not adhere to the style exhibited in that file. But it will definitely conform to your coding standard--and you can always run a Jackpot pretty-print filter on the whole file, if you have that luxury.

What Jackpot Doesn't Do

Jackpot doesn't parse code comments the way that javadoc does, for example, so it won't help very much in that area. For that, you still need to use a tool like DocCheck and make changes manually. (DocCheck is available at http://java.sun.com/j2se/javadoc/doccheck/)

Note: As the author of DocCheck, I can tell you that it does a thorough job. Another tool I wrote that may be published by Sun's java tools group in the near future is CommentMerge--a heavily-tested program that merges API comments into existing source code. At some point, I would dearly love to see the javadoc syntax tree grafted onto the AST, and use Jackpot rules to implement a merge of DocCheck and CommentMerge for comment transformations!

The only other thing that Jackpot doesn't do, besides slicing your bread, is checking out files from the source control system. If you're making changes to a large number of files, that's obviously an issue. The solutions are to:

1. Use Jackpot as part of an IDE that will check out the files automatically.
2. Use Jackpot to generate a list of files that need to be changed, and funnel that list to a script that checks them out. (That would be an excellent job for Groovy, in fact. See JavaOne, Day 1: It's a Groovy Day!, at http://www.artima.com/weblogs/viewpost.jsp?thread=116723)

Availability

Although the Jackpot engine is working, it isn't currently available because the NetBeans integration isn't totally working, at the moment. (The IDE will provide a GUI, so you can examine suggested changes and choose whether or not to apply them.)

To find out when Jackpot is released as a NetBeans module, register at http://www.netbeans.org/.

On the other hand, Tom Ball also left open the possiblity of creating a command-line version that could be built into an ANT task and eventually integrated into other IDEs. The best way to check progress on that front is most likely to monitor his blog at http://weblogs.java.net/blog/tball/.

Resources

• NetBeans (get on the mailing list for announcements): http://www.netbeans.org/
• Tom Ball's weblog: http://weblogs.java.net/blog/tball/
• His post on Jackpot: http://weblogs.java.net/blog/tball/archive/2005/06/project_jackpot.html
• DocCheck: http://java.sun.com/j2se/javadoc/doccheck/
• Other comment-processing doclets intended for use with javadoc: http://www.doclet.com/
• JavaOne, Day 1: It's a Groovy Day! http://www.artima.com/weblogs/viewpost.jsp?thread=116723