JVM Profiling From Afar

Carl Chesser contributed to this write-up.

When it comes to profiling Java applications, the VisualVM application [1] is a great tool to get a standard set of information about how your code is performing.  The challenge is when you want to profile a remote application with this tool, you can only get the high level monitoring capabilities (thread/heap usage); as the Profile tab is not enabled for remote processes (see Feature matrix [2]).  To get this functionality, you can use the profiler within the Netbeans IDE (which is what VisualVM is built from).  This allows you to profile remote applications just as you would with a local JVM instance, which greatly improves your ability to analyze performance in how the code is actually being used.


Step 1: Download Netbeans and install the IDE.  I have the Jave EE bundle, but the Java SE bundle would be sufficient. http://netbeans.org/downloads/

Step 2: Open Netbeans, and go to the "Profile" menu option, then "Attach Profiler…"



Step 3: In the Attach Profiler window, you will want to select the "CPU" option then select "Advanced (instrumented)".  For the Filter, you can start with just excluding the Java core classes (but this can be set with any filtering you desire to keep a stronger focus on what you are profiling).

Step 4: If this is your first time opening the "Attach Profiler" window, the "Attach Mode:" line at the bottom will ask you to define what you are attaching (otherwise it will list what you have set before, and ask you if you want to change it).  Click on the link to either "change..." or "define...".  Set the following:

• Target Type: Application
• Attach method: Remote
• Attach invocation: Direct (only option)

Click Next

Step 5: Specify the hostname of the remote JVM to which you are planning to connect.  You will also need to specify the host OS / JVM.  In my example, I'm going to connect to ipgoconceptsvc01, and it is a Linux host (64-bit version of the JVM).  If you are not sure, just run "java -version" on the remote host.  The output will look similar to this:

22:39:42 # java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Step 6: In the next window, it will have you review your attach settings (click Next).

Step 7: Here, you will want to select the version of your JVM that the remote applicaiton will run on.  You will then want to click the "Generate Remote Pack..." which will create a zip file that has some bash scripts to calibrate your profiler.

Step 8: Specify a folder where you’d like to create the zip file and click Save.

Step 9: Now click Finish in the Attach Wizard window, and it will bring you back to the "Attach Profiler" window.  You now need to copy the remote pack (zip file) to the remote system and get your JVM running.

scp ~/remote-pack/profiler-server-linuxamd64.zip root@ipgoconceptsvc01:/root

At this point we will need to take different steps depending on whether we are profiling a standalone JVM or one from a tomcat instance.


Regular JVM


Step 1: SSH onto the remote host and unzip the remote pack and execute the calibrate-16.sh shell script: 

ssh root@ipgoconceptsvc01
unzip profiler-server-linuxamd64.zip -d remote
./remote/bin/calibrate-16.sh 

Step 2: The profiler is now calibrated.  I now want to stop my current Java process and run it with the profiler.  For this example, my Java app is test-server:

23:17:11 # ps -ef | grep test-server
tester    4430     1  0 18:18 ?        00:00:18 java -Xmx512m -Xms128m -Dcom.sun.management.jmxremote.port=9903 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -cp /opt/tester/test-server/conf:/opt/tester/conf:/opt/tester/test-server/test-server-2.1-SNAPSHOT.jar com.dbdevs.test.server.TestServer
root     10613  9891  0 23:19 pts/0    00:00:00 grep test-server

Step 3: I will stop this Java process (currently running as a service), and then run it with profiler:

service test-server stop
./remote/bin/profile-16.sh -Xmx512m -Xms128m -Dcom.sun.management.jmxremote.port=9903 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -cp /opt/tester/test-server/conf:/opt/tester/conf:/opt/tester/test-server/test-server-2.1-SNAPSHOT.jar com.dbdevs.test.server.TestServer

It will now start up the JVM and wait for the connection with the profiler:

Profiler Agent: Initializing...
Profiler Agent: Options: >/root/remote/bin/../lib/,5140<
Profiler Agent: Initialized successfully
Profiler Agent: Waiting for connection on port 5140 (Protocol version: 12)

Tomcat JVM

I wasn’t able to find anything about profiling remotely for apps running on tomcat, and this has the potential of being very useful for profiling systems closer to their actual production state. Carl helped me out with this implementation and I thought I’d share it with anyone else who hadn’t found any write-ups on the issue. This will be a different server name, but don’t let that distract you from the main points.


Step 1: SSH onto the remote host and copy the remote pack to '/opt/profiler' and unzip it. Then change owner to tomcat, or whichever user tomcat runs under:

ssh root@ipgoconceptsvc01
mkdir /opt/profiler
cd /opt/profiler
cp /root/profiler-server-linuxamd64.zip profiler-server-linuxamd64.zip
unzip profiler-server-linuxamd64.zip
chown tomcat:tomcat * -R

Step 2: Switch into the tomcat user, or whichever user tomcat runs under, and execute the calibrate-16.sh script:

su - tomcat
./profiler/bin/calibrate-16.sh

Step 3: Exit tomcat user. Navigate to your tomcat apps /bin folder. We’ll be updating the setenv.sh script with the agent information:

exit # to exit the tomcat user
cd /opt/tomcat/tester/bin
nano setenv.sh

Step 4: Add this line to the CATALINA_OPTS environment variable (Note: The only part that needs to be added is the part in quotes, and it should just be added to what is already there.):

export CATALINA_OPTS="-agentpath:/opt/profiler/lib/deployed/jdk16/linux-amd64/libprofilerinterface.so=/opt/profiler/lib/,5140"

Hint: The last number is the port number to listen on.

Step 5: Now restart tomcat. It may not look like anything is happening, but the agent is waiting for Netbeans to connect. Once the startup is done, go ahead and attach the profiler in Netbeans.

service tomcat_tester start

Note: Additional filtering should be used as it will be quite slow if everything is coming through. 

(We now resume the tutorial for both paths)

Step 10: Go back into Netbeans and click the "Attach" button (lower right of the window).  This will connect you to the JVM and open the Profiler view:

Step 11: Now click "Live Results" to start seeing your profiled code!

You can then start running tests and taking snapshots to see the costs of your profiled methods.

[1] http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/
[2] http://visualvm.java.net/features.html