Tag Archives: tools

Java server application troubleshooting using JDK tools

by Mikhail Vorontsov

1. Introduction
2. Troubleshooting scenarios
    2.1. Getting a list of running JVMs
    2.2. Making a heap dump
    2.3. Analyzing a class histogram
    2.4. Making a thread dump
    2.5. Running Java Flight Recorder

1. Introduction

In the Java world most of us are used to GUI tools for all stages of Java application development: writing your code, debugging and profiling it. We often prefer to set up the server environment on our dev boxes and try to reproduce the problems locally using familiar tools. Unfortunately, it is often impossible to reproduce some issues locally for various reasons. For example, you may not be authorised to access the real client data which is processed by your server application.

In situations like this one you need to troubleshoot the application remotely on the server box. You should keep in mind that you can not properly troubleshoot an application with bare JRE in your hands: it contains all the troubleshooting functionality, but there is literally no way to access it. As a result, you need either a JDK or some 3rd party tools on the same box. This article will describe JDK tools, because you are likely to be allowed to use it on production boxes compared to any 3rd party tools which require security audit in many organizations.

In general case, it is sufficient just to unpack the JDK distribution to your box – you don’t need to install it properly for troubleshooting purposes (actually, proper installation may be undesirable in a lot of cases). For JMX based functionality you can install literally any Java 7/8 JDK, but some tools can not recognize the future JDK, so I advice you to install either the latest Java 7/8 JDK or the build exactly matching to server JRE – it allows you to dump the app heap for applications with no safepoints being accessed at the moment (some applications in the idle mode are the easy example of “no safepoints” applications).

2. Troubleshooting scenarios

2.1. Getting a list of running JVMs

In order to start working you nearly always need to get a list of running JVMs, their process IDs and command line arguments. Sometimes it may enough: you may find a second instance of the same application doing the same job concurrently (and damaging the output files / reopening sockets / doing some other stupid things).

Just run jcmd without any arguments. It will print you a list of running JVMs:

3824 org.jetbrains.idea.maven.server.RemoteMavenServer
780 sun.tools.jcmd.JCmd

Now you can see what diagnostic commands are available for a given JVM by running jcmd <PID> help command. Here is a sample output for VisualVM:

>jcmd 3036 help

The following commands are available:

Type jcmd <PID> <COMMAND_NAME> to either run a diagnostic command or get an error message asking for command arguments:

>jcmd 3036 GC.heap_dump

java.lang.IllegalArgumentException: The argument 'filename' is mandatory.

You can get more information about a diganostic command arguments by using the following command: jcmd <PID> help <COMMAND_NAME>. For example, here is the output for GC.heap_dump command:

>jcmd 3036 help GC.heap_dump
Generate a HPROF format dump of the Java heap.

Impact: High: Depends on Java heap size and content. Request a full GC unless the '-all' option is specified.

Permission: java.lang.management.ManagementPermission(monitor)

Syntax : GC.heap_dump [options] <filename>

filename :  Name of the dump file (STRING, no default value)

Options: (options must be specified using the <key> or <key>=<value> syntax)
-all : [optional] Dump all objects, including unreachable objects (BOOLEAN, false)

Continue reading

Memory introspection using sun.misc.Unsafe and reflection

by Mikhail Vorontsov

It is useful for a serious Java developer to realize how much memory is occupied by one or another Java object. You may have heard that we live in a world there memory is not an issue anymore. This may be true for your text editor ( though, try to open a large document with tons of embedded images and charts and see how much memory will be consumed by your favourite editor ). This may be true for a dedicated server software (at least, until your business would grow to a bigger market or you will run another piece of software on the same server). This is may even be true for a cloud-based software, if you are rich enough in order to pay a premium for a top-class server hardware.

Still, in the real world your software will once reach a point where it makes sense to spend money in its optimization rather than trying to obtain an even better hardware (currently the most you can get on a commodity class server is 64G RAM). At this point you will have to analyze your application and find which data structures consume most of application memory. The best tool for such task is a good profiler, but you can start with a cheaper approach of analyzing your objects inside your code. This article describes an Oracle JDK based ClassIntrospector class, which will allow you to analyze your application memory consumption.

I have already mentioned the Java object memory layout in the String packing part 1: converting characters to bytes article. For example, I have written that a 28 character long String should occupy 104 bytes before Java 1.7.0_06. To be honest, at the time of writing of that article I used my profiler to get a proof for my calculations. Now it is about time to implement a Java object introspector using pure Java and Oracle JDK specific sun.misc.Unsafe class.

We will use the following sun.misc.Unsafe methods:

Continue reading