How to Log and Analyze a Space-Based Distributed System
In this two-part series of articles, we are going to look at how to build a system that lets you invisibly record activities of JavaSpace clients by producing space events and logging them for immediate or future analysis. We will specifically concentrate on the collection of events associated with the JavaSpaces API calls (rather than the way we view and analyze the events). We will develop a system architecture over several iterations and show the various architectural models that you can use to implement a space-based logging system. In addition, we will give you a taste of several implementations of the logging system. By characterizing each solution, we can improve the system design and understand the situations in which you might use the various solutions. We will assume some basic knowledge of Jini and JavaSpaces. See Resources for links to Jini and JavaSpaces tutorials.
Understanding your System
In many cases, we have found building distributed systems with JavaSpaces lets us reach solutions quicker than with traditional techniques. With the right approach, JavaSpaces can help alleviate problems common to many distributed systems, such as partial failure and tight coupling between clients and services.
Although using JavaSpaces provides us with a means (among other things) to build loosely coupled systems, using JavaSpaces can lead to a lower level of predictability and confidence in the system's overall behavior. Or, to put it another way, by using JavaSpaces you can give up some control and visibility over the system's activities as a whole.
When you deploy space-based systems, you need to have some confidence that they will work and continue to work as you scale them up. At design and build time it is easy for developers make errors based on false assumptions about the way a space-based system will operate. Therefore, we suggest you always build a small system subset that you intend to deploy to test your assumptions of the system's behavior against what you can observe.
Hint: Build and test in the small first to test your assumptions.
Even with small systems, finding out what occurs 'in the space' can be difficult. Shifting reads, writes, and takes in your code or inserting debug statements in important places may give you clues. But these techniques are effective only in the simplest systems, where you can easily imagine the flow of objects around the system. Systems of anything other than the most trivial complexity require not only stretching your imagination, but also using tools to help you imagine space activity. Tools that let systems developers or designers view the activities of the system's various clients can help restore confidence in understanding the system's behavior as a whole.
In general, software debuggers apply to one executable and let you observe the behavior of the software you have written. So you not only use debuggers to debug systems, but also to understand the way they work. Debuggers also provide clues for code refactoring and perhaps optimizing the local system's behavior.
When you apply this model to spaces, you need a tool that lets you see a distributed system's behavior as a whole. In the same way local debuggers let you see a local program's behavior and individual thread activity, you need to see what each client of the space does in order to see how your whole system behaves.
One of the most useful tools we have developed while designing and building space-based systems has been a logging system that lets you record the operations performed on the space's API. Later in this article, we'll analyze the log to see which operations were performed and in what order.
A Logging JavaSpace Proxy
One main benefit of a logging system is that it lets us see how clients use the space without having to insert debug/logging statements into the client code. To automate this process without touching our client code, we will use a simple yet powerful idea.
We know that all space-based systems load a proxy at runtime to support the remote calls on the JavaSpaces API. Using this knowledge, we can easily build new proxies that hide the logging operations from the client, but still look like the standard JavaSpace proxy.
So in our system, the Jini proxy for our logging system will implement the JavaSpaces interface. However, instead of calling the backend of the JavaSpace directly, the proxy will collect the information we need and then forward the JavaSpaces API calls onto another proxy that uses the backend JavaSpaces server.
To avoid confusion, throughout this article we will call the proxy that produces logging information the logging proxy. The standard proxy that calls into the space backend is called the delegate proxy, and the combination of delegate proxy and the backend space will naturally be called the delegate space.
Figure 1 shows the general structure for building the logging proxies and using the delegate space.
Figure 1. A Logging JavaSpace Proxy Architecture.
The interface to the logging proxy, LoggingSpace, extends JavaSpace:
import net.jini.space.JavaSpace
public interface LoggingSpace extends JavaSpace {
// no additional methods to those in the JavaSpace interface
}
LoggingSpace has exactly the same methods as the JavaSpace interface. By declaring a LoggingSpace as a subinterface of JavaSpace, it can be used wherever a JavaSpace is called for. But because it is a subinterface, we can use a type-based lookup in the Jini lookup service to find a JavaSpace proxy that also does logging.
Now that we have established a general structure for the logging proxy, let's consider how we will record and structure the logging information.
A Logging Information Model
We have two candidate object models for our logging information. The first is a class-based model illustrated in Figure 2. A quick view of the JavaSpace interface can be mapped to an inheritance graph representing the various operations that we might record in the logging system.
Figure 2. A Class Hierarchy for Logging JavaSpace Operations.
As you can see from the class hierarchy, there is an operation class for every method call in the JavaSpace interface and others to deal with the more general or abstract information.
This model has the general advantage that only the information that applies to a particular operation type will be stored and recorded in the object representing that operation. It also has the standard property of extensibility and flexibility normally associated with type- based inheritance systems. However, the model does add complexity and raise the class count dramatically.
The other alternative holds all the method information and parameters in one class using null information for fields not used for a particular operation. In this case, we need to add a field to represent the operation being called instead of using the Java type system to encode the operation. Here is the Java code for a simple compound SpaceOperation class.
import java.io.Serializable;
import java.net.InetAddress;
import net.jini.core.entry.Entry;
import net.jini.core.lease.Lease;
import net.jini.core.transaction.Transaction;
public class SpaceOperation
implements Serializable {
private long _timeStamp;
private String _host;
private String _thread;
private String _method;
private String _entry;
private String _txn;
private long _timeVal;
private static String _hostCache;
SpaceOperation(String method, Entry entry,
Transaction txn, long timeVal) {
_timeStamp = System.currentTimeMillis();
_host = getHostDetails();
_thread = Thread.currentThread().getName();
_method = method;
if (entry != null) {
_entry = entry.getClass().getName();
}
if (txn != null) {
_txn = txn.toString();
}
_timeVal = timeVal;
}
private String getHostDetails() {
if (_hostCache == null) {
try {
InetAddress host =
InetAddress.getLocalHost();
_hostCache = host.toString();
}
catch (java.net.UnknownHostException exp) {
_hostCache = "No Host Details";
}
}
return _hostCache;
}
private static final char comma = ',';
private static final int BUF_SIZE = 128;
public String toString() {
StringBuffer buf = new StringBuffer(BUF_SIZE);
buf.append(new java.util.Date(_timeStamp));
buf.append(comma);
buf.append(_host);
buf.append(comma);
buf.append(_thread);
buf.append(comma);
buf.append(_method);
buf.append(comma);
buf.append(_entry);
buf.append(comma);
buf.append(_txn);
buf.append(comma);
buf.append(_timeVal);
return buf.toString();
}
}
Every time a call is made on the JavaSpaces API in a logging proxy, we can create a new instance of the SpaceOperation class to log the operation type and its parameters. The class also overrides the toString() method of the Object class to encode the information in a comma-separated format, so that we can easily parse it with other software tools.
This single-class model seems adequate for our purposes while still being simple and lightweight. Now that we can record and format our logging information, we can look at our first logging proxy implementation.
A Local Logging Proxy Implementation
Using the general architecture shown in Figure 1, one simple implementation would be to write the logging information on the local console or a local disk. In that way, the information is recorded locally to each client making calls on the space proxy.
When a call is made to the logging proxy's API, the logging proxy constructs a SpaceOperation object for the associated JavaSpaces API call and then forwards the call to DelegateProxy.
For simplicity's sake, the logOperation() method simply uses the system's output stream to write the formatted SpaceOperation object to the local console, but we could implement it to write to a local disk file.
Here are the highlights of the LocalLoggingProxy implementation:
package jsip.space;
public class LocalLoggingProxy
implements LoggingSpace, Externalizable {
private JavaSpace _delegateSpace;
public void readExternal(ObjectInput in)
throws IOException, ClassNotFoundException {
//Nothing to read
Debug.message("readExternal() finding space");
try {
_delegateSpace = SpaceLocator.getSpace();
}
catch (Exception exp) {
throw new IOException(
"Can't find JavaSpace " + exp.toString());
}
}
public void writeExternal(ObjectOutput out)
throws IOException {
//Nothing to write
}
public Lease write(Entry entry,
Transaction txn, long lease)
throws TransactionException,
RemoteException {
logOperation(
new SpaceOperation("write", entry, txn, lease));
return _delegateSpace.write(entry, txn, lease);
}
// all of the other JavaSpaces API methods
public Entry snapshot(Entry entry)
throws RemoteException {
logOperation(
new SpaceOperation("snapshot", entry, null, 0));
return _delegateSpace.snapshot(entry);
}
private void logOperation(SpaceOperation op) {
System.out.println(op.toString());
}
}
In the previous code example, we have skipped most of the methods that exist in the JavaSpaces API, because they essentially follow the same pattern as the write method. In the snapshot case, we have included default information for some of the parameters.
The most interesting thing to note in the LocalLoggingProxy class is how it dynamically embeds the delegate space proxy. Because we know that Java's serialization mechanism invokes the readExternal() method when a client deserializes the proxy, we can insert the code to find the delegate space and assign its proxy to the field _delegateSpace. The LocalLoggingProxy uses that field to invoke the matching methods on the JavaSpace.
We can use the readExternal() technique safely to locate the space because we know that the proxy gets serialized to the Jini Lookup Service as a MarshalledObject. This means that the proxy itself isn't deserialized in the lookup service. If this wasn't the case, using this technique would be a bad idea because the readExternal() method would also get invoked in the lookup service.
Why do we defer the discovery of the JavaSpace until the LocalLoggingProxy is deserialized in the client, rather than discovering the space and embedding it in the proxy when it is registered with a Jini Lookup Service? Because that way we can stop and restart the space without having to stop and re-register the LocalLoggingProxy.
This type of adapter or smart Jini proxy has many interesting uses, for example, we could write a fault tolerant JavaSpace proxy by adding exception handling around the delegated method calls to detect if the space became unavailable. If the space became unavailable, the smart proxy could then reinitiate discovery and connect to another space or attempt to reconnect to the failed space.
There are more examples and variations of smart Jini proxies, like the LocalLoggingProxy presented in our book JavaSpaces In Practice (see Resources).
The LoggingSpace Service
To use the LocalLoggingSpace, you need a server that registers the proxy with the Jini lookup service. In the code to register the proxy, it's important to note that since the proxy is a simple serializable object and not an RMI (remote method invocation) object, such as a UnicastRemoteObject, we must have a loop at the end of main() to stop the VM from exiting. If the VM did exit, the JoinManager wouldn't be able to renew the proxy's lease in the lookup service, resulting in the proxy being removed (from the lookup service) when the lease expired.
import jsip.debug.Debug;
public class RegisterProxy {
private static JoinManager _joinManager;
public static void main(String[] args) {
try {
System.setSecurityManager(
new java.rmi.RMISecurityManager());
//Create a proxy to put into Reggie
LocalLoggingProxy proxy = new LocalLoggingProxy();
//Join stuff, ignoring persistent service id for now.
ServiceIDListener listener = new ServiceIDListener() {
public void serviceIDNotify(ServiceID serviceID) {
//Not storing for now
System.out.println("serviceIDNotify()");
}
};
Entry[] attribs = new Entry[]{
new Name("Local Logging Space")
};
_joinManager =
new JoinManager(proxy, attribs, listener, null, null);
//Keep the VM alive, as we're not using
//a UnicastRemoteObject.
//If the VM exits, then the proxy's
//lease with the LUS would expire.
while (true) {
Thread.sleep(10000);
}
}
catch (Exception ex) {
ex.printStackTrace();
}
}
}
A second point to note in the RegisterProxy example is that we don't store and reuse the ServiceID the lookup service provides. However, this is to keep the example simple; you should always store and reuse ServiceIDs. The simplest way to do this is to serialize ServiceIDs to the local file system; that way, each time you want to register a proxy the code can check for the existence of the file and use the stored ServiceID if one exists. Failure to reuse ServiceIDs results in dead proxies residing in the lookup service until their leases expire. If you reuse a ServiceID, by contrast, the lookup service replaces the matching proxy with the new one.
The LoggingSpace Client
Now that we have our first logging model complete let's take a quick look at how this affects the client code.
To use the LocalLoggingProxy you simply change the name of the class used for discovery from net.jini.space.JavaSpace.class to jsip.space.LocalLoggingProxy.class
We would suggest using something like a system property that your client code can pick up on to allow logging to be enabled/disabled at start up. A simple way of doing this is to set a system property to the appropriate class name as follows:
java -DspaceClass=net.jini.space.JavaSpace MyClient
Then in your client code do something like this
//get the name of the class, either
// LoggingProxy or JavaSpace
String cname=System.getProperty("spaceClass");
//Load the class object
Class spaceClass=Class.forName(cname);
//Create the ServiceTemplate
Class [] types=new Class[]{spaceClass};
ServiceTemplate tmpl=new ServiceTemplate(null,types,null);
Review the design
The LocalLoggingProxy may work well if you are simply testing one client on a single machine; for example, if you use the space as a simplified persistence mechanism for a Java application. However, a local logging proxy for a distributed system doesn't really make too much sense, so we need to consider a couple of potential problems with this implementation thus far.
First, if we run clients on various machines, we must have some way to collect the information to get an overall view of the system's behavior. This means that we need access to all the local files in order to merge them to get a coherent picture, which is an arduous task.
Second, the way in which Jini distributes proxies means that we can't predict where the spaces proxies will run. Also, if we intend to send our logging information to a disk, for example, there is no guarantee that a disk will be present on the device running the client and proxy, or that the device will be able to store more than trivial amounts of debugging information.
However, we do know that the system must have some kind of network connection or the proxy could not have arrived in the first place, unless of course the local machine runs a lookup service and the space runs locally. Therefore, we could build the proxy to move the information back over the network onto a machine that we know has adequate resources.
In Part 2 of this article, coming next week, we will present a second model that retains the LoggingProxy interface but reimplements the proxy to provide its information to a Jini logging service.
Resources
JavaSpaces in Practice, a new book by by Philip Bishop and Nigel Warren, is at Amazon.com at:
http://www.amazon.com/exec/obidos/ASIN/0321112318/
The website for JavaSpaces in Practice is here:
http://www.jsip.info/
JavaSpaces: Principles, Patterns, and Practice by Eric Freeman, Susanne Hupfer, and Ken Arnold, an introduction to JavaSpaces, is at Amazon.com at:
http://www.amazon.com/exec/obidos/ASIN/0201309556/
The source code for the examples appearing in this article can be downloaded here:
http://www.djip.co.uk/downloads.html#artima
The Jini Community, the central site for signers of the Jini Sun Community Source License to interact:
http://www.jini.org
Download JavaSpaces from:
http://java.sun.com/products/javaspaces/
Make Room for JavaSpaces, Part I - An introduction to JavaSpaces, a simple and powerful distributed programming tool:
http://www.artima.com/jini/jiniology/js1.html
Make Room for JavaSpaces, Part II - Build a compute server with JavaSpaces, Jini's coordination service:
http://www.artima.com/jini/jiniology/js2.html
Make Room for JavaSpaces, Part III - Coordinate your Jini applications with JavaSpaces:
http://www.artima.com/jini/jiniology/js3.html
Make Room for JavaSpaces, Part IV - Explore Jini transactions with JavaSpaces:
http://www.artima.com/jini/jiniology/js4.html
Make Room for JavaSpaces, Part V - Make your compute server robust and scalable with Jini and JavaSpaces:
http://www.artima.com/jini/jiniology/js5.html
Make Room for JavaSpaces, Part VI - Build and use distributed data structures in your JavaSpaces programs:
http://www.artima.com/jini/jiniology/js6.html
Talk back!
Have an opinion? Readers have already posted 4 comments about this article. Why not add yours?
About the author
Dr. Susanne Hupfer is director of product development for Mirror Worlds Technologies, a Java- and Jini-based software applications company, and a research affiliate in the Department of Computer Science at Yale University, where she completed her PhD in space-based systems and distributed computing. Previously, she taught Java network programming as an assistant professor of computer science at Trinity College. Susanne coauthored the Sun Microsystems book JavaSpaces Principles, Patterns, and Practice.
Artima provides consulting and training services to help you make the most of Scala, reactive
and functional programming, enterprise systems, big data, and testing.
2070 N Broadway Unit 305
Walnut Creek CA 94597
USA
(925) 918-1769 (Phone)