Analyzing large Java heap dumps when Eclipse Memory Analyzer (MAT) UI fails

If you find yourself trying to analyze a big heap dump (20-30GB) downloaded from your production server to your staging/test machines.. only to find out that X-over-SSH is too slow then this article is for you.

As of Nov 2013, we have 2 options - Eclipse MAT and a hidden gem called Bheapsampler.

Option 1:
Eclipse Memory Analyzer is obviously the best tool for this job. However, trying to get the UI to run remotely is very painful. Launching Eclipse and updating the UI is an extra load on the JVM that is already busy analyzing a 30G heap dump. Fortunately, there is a script that comes with MAT to parse the the heap dump and generate HTML reports without ever having to launch Eclipse! It's just that the command line option is not well advertised.

Command line heap analysis using Eclipse MAT:

Assuming Eclipse MAT is installed and we are inside the mat/ directory, modify MemoryAnalyzer.ini heap settings to use a large heap to handle large dumps:

    -startup
    plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar
    --launcher.library
    plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.100.v20110505
    -vmargs
    -Xms24g
    -Xmx24g

Run MAT against the heap dump:

    ./ParseHeapDump.sh ../today_heap_dump/jvm.hprof

This takes a while to execute and generates indices and other files to make repeated analysis faster. Then use the indices created in the previous step and run a "Leak suspects" report on the heap dump.

    ./ParseHeapDump.sh ../today_heap_dump/jvm.hprof org.eclipse.mat.api:suspects

The output is a small and easy to download jvm_Leak_Suspects.zip. This has HTML files just like the MAT Eclipse UI. It can be easily SCP'ed/emailed around.

Other report types possible.

    org.eclipse.mat.api:suspects
    org.eclipse.mat.api:overview
    org.eclipse.mat.api:top_components
   
More details - http://wiki.eclipse.org/index.php/MemoryAnalyzer/FAQ.

Option 2:
http://dr-brenschede.de/bheapsampler is something I chanced upon. It is a sampling heap dump reader and so it works for very large heap dumps where MAT sometimes fails. Being a sampling reader, the output is also a little imprecise but helps a great deal when you have nothing else. The tool seems to be closed source and is very sensitive to heap dump corruptions.

As an aside, here's something that might be useful for the initial heap dump quickly - https://blogs.atlassian.com/2013/03/so-you-want-your-jvms-heap/.

Book review: Getting Started with Hazelcast

A few weeks ago Packt Publishing sent me a free copy of their new publication - Getting Started with Hazelcast by Mat Johns to read and write about. I have used distributed caches and compute grids quite a bit at work. So, I was happy to do a quick review of this book. I've used Oracle Coherence quite a lot and Hazelcast for some experiments.

The book is a gentle guide to building distributed compute and data grids. It assumes nothing about the reader and hence does a good job of doing what it says in the book's title - "getting started". I'd advice this book for anyone who is completely new to this area which is not to be confused with Hadoop, Storm, Cassandra or the other more "popular/hyped" cousins. I would say that for medium sized data, logic heavy, transactional/near real time applications, compute grids are the way to scale out.

Obviously this book is about using Hazelcast, which is a nice Apache software licensed, Java, distributed grid/cache. It is surprisingly feature rich and in terms of usability, features and elegance it comes very close to its more expensive, older, rock solid cousin which is Oracle Coherence.

The book explores the essential aspects of using such frameworks effectively. Such as - distributed maps, replication, network partitions, fault tolerance, data affinity, moving code closer to where data is etc. It does this without being too overwhelming for first timers.

For a full and more thorough treatment I would obviously recommend the Hazelcast documentation. And if you are curious to know about other frameworks check out my old write up - Scalable compute & storage frameworks - A Refcard.

Ashwin.