And, if anyone wondered what the part of the engine is that the system from Azul is improving in the Linux kernel, here's a more detailed description:
http://www.managedruntime.org/sites/www.managedruntime.org/files/downloads/AzulVmemMetricsMRI.pdf

In short, they introduced a way to collect and batch commit changes to the virtual memory mappings, so that the VM needs less safe-points at which the commits may occur.

Nono. Consider the case of a simple stop-and-copy collector. At any given time, all of the objects are in a single arena. When you hit the high water mark, you stop consing and allocate a new, bigger arena. You then iterate across all the root pointers in your system, copying every object reached by the root pointers into the new arena. Whenever you copy an object, you leave a fixup object in its place; this object has the address of the moved object. When you are scanning the roots, whenever you come to a pointer to a fixup object, you fix it and stop scanning, knowing that every object reached through the fixed-up object has already been reached. Once you have explored all the roots, all reachable objects will have been copied into the new arena, and you can just discard the old arena, knowing that any objects not reached are garbage.

So for example suppose you have an object A, which references object B. And you have another object, C, which references B as well. And suppose both A and C are reached by different roots. When you go to collect, you will reach A, and A will point to B in the old arena. So you'll move A into the new arena, and then move B into the new arena, and then adjust A's pointer to B. And you will update the version of B in the old arena to be a fixup object. Later on in the collection pass when you reach C, C's pointer to B will be pointing to the fixup object in the old arena. So you will copy C into the new arena, and update C's pointer to B using the information in the fixup object.

Now in this read-barrier model, suppose you move A and B, and fix up A's reference to B. And now suppose that your program accesses C after you've moved A and B. C's pointer to B will be pointing into the old arena, which has a read barrier, so you'll get a trap at that point, and you'll have the old address of B in the trap exception data. But what you won't have in the exception data is the address of C. And you need the address of C in order to update C's pointer to B.

So my original question was not how you got the address of B, but how you got the address of C. The only way I can think of to do it is to use your knowledge of the JVM--perhaps you know that when B is referenced, the object that referenced B must be in some register. If that's how they're doing it, then this isn't a general mechanism--it will only work with a specific JVM.

Hm, so are you saying that you take the fault when you reference C, which is in the old arena, not when you reference B? In that case, the fault address would be the address of C, and you'd be able to move C and update C's pointers at that time. If that's the case, then I've been missing what's novel about this approach, because using a read barrier in this way has been done before.

Achilleas, to address/clarify a couple of your points:

>...
> The pages that have objects that were moved need to stay
> in the 'moved' state until all references to the objects
> in that page have been successfully modified to point to
> the new objects. This causes greater memory consumption.
>

The "secret" behind quick-release is that the above statement is true for the *virtual* addresses of the pages (that have objects that were moved) - as the addresses themselves cannot be safely recycled until all references to them have been hunted down and fixed (which will happen by the end of our next Mark-Remap phase). However, the actual physical memory resources backing those virtual page addresses is not needed once the object contents have been moved, and can be recycled immediately at new virtual addresses without waiting for the remaps to complete.

We can therefor compact the entire heap using a single empty page, by doing hand over hand compaction - feeding each page compaction with physical pages release from the previous page's compaction. In reality we use a bit more than that (one empty page per compacting GC thread), but we can always compact the heap, and never need to worry about not having enough empty memory to perform the compaction...

> The above are my assumptions anyway. Another way to solve
> this issue would be to use a unique key for each object,
> and then search a map for a reference to the object, based
> on the key. But that would be highly inefficient.

Your assumption is correct for most relocating collectors, but it is also what keeps them from being able to recycle the memory quickly. In the Azul VM, we keep this forwarding information metadata outside of the pages being compacted from, and store them in pretty compact and efficient ways Keeping it outside of the physical pages and original object space allows quick release to be performed (otherwise, as you noted, we would not be able to release the page until all references to them were identified and remapped).