|2. Troubleshooting scenarios|
In the Java world most of us are used to GUI tools for all stages of Java application development: writing your code, debugging and profiling it. We often prefer to set up the server environment on our dev boxes and try to reproduce the problems locally using familiar tools. Unfortunately, it is often impossible to reproduce some issues locally for various reasons. For example, you may not be authorised to access the real client data which is processed by your server application.
In situations like this one you need to troubleshoot the application remotely on the server box. You should keep in mind that you can not properly troubleshoot an application with bare JRE in your hands: it contains all the troubleshooting functionality, but there is literally no way to access it. As a result, you need either a JDK or some 3rd party tools on the same box. This article will describe JDK tools, because you are likely to be allowed to use it on production boxes compared to any 3rd party tools which require security audit in many organizations.
In general case, it is sufficient just to unpack the JDK distribution to your box – you don’t need to install it properly for troubleshooting purposes (actually, proper installation may be undesirable in a lot of cases). For JMX based functionality you can install literally any Java 7/8 JDK, but some tools can not recognize the future JDK, so I advice you to install either the latest Java 7/8 JDK or the build exactly matching to server JRE – it allows you to dump the app heap for applications with no safepoints being accessed at the moment (some applications in the idle mode are the easy example of “no safepoints” applications).
2. Troubleshooting scenarios
2.1. Getting a list of running JVMs
In order to start working you nearly always need to get a list of running JVMs, their process IDs and command line arguments. Sometimes it may enough: you may find a second instance of the same application doing the same job concurrently (and damaging the output files / reopening sockets / doing some other stupid things).
jcmd without any arguments. It will print you a list of running JVMs:
3824 org.jetbrains.idea.maven.server.RemoteMavenServer 2196 780 sun.tools.jcmd.JCmd
Now you can see what diagnostic commands are available for a given JVM by running
jcmd <PID> help command. Here is a sample output for VisualVM:
>jcmd 3036 help 3036: The following commands are available: JFR.stop JFR.start JFR.dump JFR.check VM.native_memory VM.check_commercial_features VM.unlock_commercial_features ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start GC.rotate_log Thread.print GC.class_stats GC.class_histogram GC.heap_dump GC.run_finalization GC.run VM.uptime VM.flags VM.system_properties VM.command_line VM.version help
jcmd <PID> <COMMAND_NAME> to either run a diagnostic command or get an error message asking for command arguments:
>jcmd 3036 GC.heap_dump 3036: java.lang.IllegalArgumentException: The argument 'filename' is mandatory.
You can get more information about a diganostic command arguments by using the following command:
jcmd <PID> help <COMMAND_NAME>. For example, here is the output for
>jcmd 3036 help GC.heap_dump 3036: GC.heap_dump Generate a HPROF format dump of the Java heap. Impact: High: Depends on Java heap size and content. Request a full GC unless the '-all' option is specified. Permission: java.lang.management.ManagementPermission(monitor) Syntax : GC.heap_dump [options] <filename> Arguments: filename : Name of the dump file (STRING, no default value) Options: (options must be specified using the <key> or <key>=<value> syntax) -all : [optional] Dump all objects, including unreachable objects (BOOLEAN, false)