While memory profilers are the most common way to gauge an application's memory use, using a profiler exclusively for that purpose has a few drawbacks, according to Petr Nejedly, creator of the Insane heap analysis tool:

Profiling structures using profiler needs complicated instrumentation and is hard to perform automatically from memory regression tests... Long-term running the application in the instrumented environment significantly reduces its performance and you need to prepare instrumentation in advance.

To solve these problems, Nejedly created what he calls a "post-mortem memory leak analysis tool," Insane. Nejedly released Insane 1.1 yesterday.

According to Nejedly, Insane can be used either to measure the size of a data structure in memory, or to analyze the entire heap, and both modes are based on the same memory scanning engine:

The engine performs a BFS scan of the object graph from [a] given collection of roots, counting visited instances of each type and their size. The size computation is based on the Sun JDK1.3/1.4 storage format and for these JDKs is exact. The engine puts each newly found object into an identity map, assigns it unique ID and adds it to the end of processing queue. Already known objects are ignored. Processing ends when the queue is empty—the algorithm has traversed full transitive closure of [the] given roots.

The engine is able to skip some objects, specifically java.lang.Class instances, instances of its own classes and instances from the user filter. The output of the engine is the size distribution collection, containing instance count and total size of each found class.

When measuring the size of an object in memory, Insane allows you to specify a root collection that references the object to measure.

In the heap analysis mode:

The tool does a best effort to find all the real JVM GC roots using introspection on all known classloaders and evaluating the static references. This is also the limitation of the tool - it can't find stack-local and JNI references, but it is only a marginal problem in the event-driven application domain.

In this mode, the tool uses a visitor pattern to notify all found objects and all the references between objects. The default implementation of the visitor logs all the information to a simple format XML file.

The resulting XML output can be large, depending, of course, on the size of the heap. However, the XML format is self-describing, and is amenable to easy input into other tools:

The format of the file ... is self-describing, there are only two types of tags, object and ref. The object tags carry ID, name, size of the instance and, in case the type is [C[character array], the textual representation of the character array. The ref tags are either static references, carrying the fully qualified field name and the ID of the referenced object, or instance references, carrying also the ID of the referring object.

What do you think of the idea of introspecting the heap from within a running application? And what are some of the limitations you encounter with a more traditional memory profiler?