JVM Profiling From Afar

Carl Chesser contributed to this write-up.

When it comes to profiling Java applications, the VisualVM application [1] is a great tool to get a standard set of information about how your code is performing.  The challenge is when you want to profile a remote application with this tool, you can only get the high level monitoring capabilities (thread/heap usage); as the Profile tab is not enabled for remote processes (see Feature matrix [2]).  To get this functionality, you can use the profiler within the Netbeans IDE (which is what VisualVM is built from).  This allows you to profile remote applications just as you would with a local JVM instance, which greatly improves your ability to analyze performance in how the code is actually being used.


Step 1: Download Netbeans and install the IDE.  I have the Jave EE bundle, but the Java SE bundle would be sufficient. http://netbeans.org/downloads/

Step 2: Open Netbeans, and go to the "Profile" menu option, then "Attach Profiler…"



Step 3: In the Attach Profiler window, you will want to select the "CPU" option then select "Advanced (instrumented)".  For the Filter, you can start with just excluding the Java core classes (but this can be set with any filtering you desire to keep a stronger focus on what you are profiling).

Step 4: If this is your first time opening the "Attach Profiler" window, the "Attach Mode:" line at the bottom will ask you to define what you are attaching (otherwise it will list what you have set before, and ask you if you want to change it).  Click on the link to either "change..." or "define...".  Set the following:

• Target Type: Application
• Attach method: Remote
• Attach invocation: Direct (only option)

Click Next

Step 5: Specify the hostname of the remote JVM to which you are planning to connect.  You will also need to specify the host OS / JVM.  In my example, I'm going to connect to ipgoconceptsvc01, and it is a Linux host (64-bit version of the JVM).  If you are not sure, just run "java -version" on the remote host.  The output will look similar to this:

22:39:42 # java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Step 6: In the next window, it will have you review your attach settings (click Next).

Step 7: Here, you will want to select the version of your JVM that the remote applicaiton will run on.  You will then want to click the "Generate Remote Pack..." which will create a zip file that has some bash scripts to calibrate your profiler.

Step 8: Specify a folder where you’d like to create the zip file and click Save.

Step 9: Now click Finish in the Attach Wizard window, and it will bring you back to the "Attach Profiler" window.  You now need to copy the remote pack (zip file) to the remote system and get your JVM running.

scp ~/remote-pack/profiler-server-linuxamd64.zip root@ipgoconceptsvc01:/root

At this point we will need to take different steps depending on whether we are profiling a standalone JVM or one from a tomcat instance.


Regular JVM


Step 1: SSH onto the remote host and unzip the remote pack and execute the calibrate-16.sh shell script: 

ssh root@ipgoconceptsvc01
unzip profiler-server-linuxamd64.zip -d remote
./remote/bin/calibrate-16.sh 

Step 2: The profiler is now calibrated.  I now want to stop my current Java process and run it with the profiler.  For this example, my Java app is test-server:

23:17:11 # ps -ef | grep test-server
tester    4430     1  0 18:18 ?        00:00:18 java -Xmx512m -Xms128m -Dcom.sun.management.jmxremote.port=9903 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -cp /opt/tester/test-server/conf:/opt/tester/conf:/opt/tester/test-server/test-server-2.1-SNAPSHOT.jar com.dbdevs.test.server.TestServer
root     10613  9891  0 23:19 pts/0    00:00:00 grep test-server

Step 3: I will stop this Java process (currently running as a service), and then run it with profiler:

service test-server stop
./remote/bin/profile-16.sh -Xmx512m -Xms128m -Dcom.sun.management.jmxremote.port=9903 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -cp /opt/tester/test-server/conf:/opt/tester/conf:/opt/tester/test-server/test-server-2.1-SNAPSHOT.jar com.dbdevs.test.server.TestServer

It will now start up the JVM and wait for the connection with the profiler:

Profiler Agent: Initializing...
Profiler Agent: Options: >/root/remote/bin/../lib/,5140<
Profiler Agent: Initialized successfully
Profiler Agent: Waiting for connection on port 5140 (Protocol version: 12)

Tomcat JVM

I wasn’t able to find anything about profiling remotely for apps running on tomcat, and this has the potential of being very useful for profiling systems closer to their actual production state. Carl helped me out with this implementation and I thought I’d share it with anyone else who hadn’t found any write-ups on the issue. This will be a different server name, but don’t let that distract you from the main points.


Step 1: SSH onto the remote host and copy the remote pack to '/opt/profiler' and unzip it. Then change owner to tomcat, or whichever user tomcat runs under:

ssh root@ipgoconceptsvc01
mkdir /opt/profiler
cd /opt/profiler
cp /root/profiler-server-linuxamd64.zip profiler-server-linuxamd64.zip
unzip profiler-server-linuxamd64.zip
chown tomcat:tomcat * -R

Step 2: Switch into the tomcat user, or whichever user tomcat runs under, and execute the calibrate-16.sh script:

su - tomcat
./profiler/bin/calibrate-16.sh

Step 3: Exit tomcat user. Navigate to your tomcat apps /bin folder. We’ll be updating the setenv.sh script with the agent information:

exit # to exit the tomcat user
cd /opt/tomcat/tester/bin
nano setenv.sh

Step 4: Add this line to the CATALINA_OPTS environment variable (Note: The only part that needs to be added is the part in quotes, and it should just be added to what is already there.):

export CATALINA_OPTS="-agentpath:/opt/profiler/lib/deployed/jdk16/linux-amd64/libprofilerinterface.so=/opt/profiler/lib/,5140"

Hint: The last number is the port number to listen on.

Step 5: Now restart tomcat. It may not look like anything is happening, but the agent is waiting for Netbeans to connect. Once the startup is done, go ahead and attach the profiler in Netbeans.

service tomcat_tester start

Note: Additional filtering should be used as it will be quite slow if everything is coming through. 

(We now resume the tutorial for both paths)

Step 10: Go back into Netbeans and click the "Attach" button (lower right of the window).  This will connect you to the JVM and open the Profiler view:

Step 11: Now click "Live Results" to start seeing your profiled code!

You can then start running tests and taking snapshots to see the costs of your profiled methods.

[1] http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/
[2] http://visualvm.java.net/features.html

Mavericks Multi-Monitor Magnificence

Yes, I like alliterations, but that isn't the topic of this short post. I have just installed Mavericks on my Mac, and I have tested some of the new functionality. My first problem arose after waking my Mac up. The gestures to move between spaces and to the command center, so all gestures requiring more than two fingers, wouldn't work at all. As I sought a solution to this problem, I found that this seems to be fairly common in Mavericks. I hope it isn't an inherent problem in their new multi-monitor support.

That new functionality is exactly what I thought multi-monitor support should be. I should state that I am no Mac fanboy. In fact, I have never owned a Mac and have vowed to never own a Mac unless I really need one for development purposes for iOS. However, Apple has finally done something good since Steve Jobs passed. I did have to change some of my habits for this new functionality as all monitors do not slide at the same time. Now, each monitor has its own set of desktops.

Apple has basically set each monitor as its own separate entity. Each monitor now has its own menu bar at the top. This was a serious flaw when I first got my Mac, and something that is very annoying for someone who comes from a system where the menu bar is on the top of each window. This also makes it much more clear which window the menu bar currently represents. I can't tell you how many times I quickly jumped up to the menu bar and tried some function only to find another window on a different monitor responding. Apple also adds transparency to the menu bars in the monitors that aren't in focus. This also helps to know where your inputs will appear.

I think one problem with this setup is where the desktops go when you disconnect a monitor. This may have been what was causing problems with my gesture inputs. I usually have a total of three monitors, but last night I only had two. I had windows that I simply couldn't find, but I knew they were somewhere. This may be a bug that will need to get fixed later. I just tested it, and the windows go to another desktop, but that isn't very helpful. I'd rather have the whole desktop move.

I have seen several complaints about the monitors not swiping between spaces at the same time, but I am guessing that is a trade-off for other features. I think I can find ways around this limitation, but it would be nice to have the option. This was probably done to allow for full-screen apps on one monitor without blacking out the others. This is probably the greatest benefit of the Mavericks upgrade. Essentially, that entire feature of full-screen apps was useless for multi-monitor setups. However, each monitor still seems to require a desktop instance. This isn't really a nuisance, but it is a waste of resources.

Overall, I think this is a pretty good upgrade and much more user-friendly. I hope they are able to fix some of the issues with minor updates, but I think I can live with them for now. But I won't be too happy if I have to restart my computer every morning because my gestures no longer work.

Ray Kurzweil Blows Your Mind!

Everyone should watch this video. This man speaks the thoughts I think, but then he actually does them. A lot of people have called him crazy and said he's wrong, but Google seems to believe in him. He should scare every one of us, but we should also be inspired by his vision and abilities. Please take the time to watch this video and then seek out more of his movies and writings.

Make a hashtag fueled slideshow! (ruby)

I made this for my wedding so our guests could post pictures of the wedding and post them to social media and they would appear on screen in realtime. This ended up not working due to the connection abilities of our venue, but by the end of the night a laptop was setup at the exit showing the slideshow. It was still pretty cool, and now I have a way to grab photo's with a particular hashtag from the big three social media outlets, Twitter, Facebook, and Instagram. I haven't hooked this up with Google+ yet.

Each of these platforms has a ruby gem that makes the interaction with the API much simpler and intuitive. To note, I am no expert in ruby as I knew almost nothing about it a year ago and I only work on some ruby code here at Cerner. Instagram [1] and Twitter [2] have self-named gems, but the best gem I found for Facebook is called Koala [3]. To get the correct authorization tokens you need to follow the associated walk-throughs for each platform. For Twitter, there will be a reference on the gem README to create an app [4] and get the auth token from there. This documentation is pretty explicit, and was surprisingly simple to set up everything.

Twitter is slightly harder, but still pretty straight forward. You will need to login on the developer site [5], and then click on "Register Your Application." You can then create an application with the appropriate information. This will yield you a client id and client secret, but you'll still need an access token. To generate the access token, I followed Dave Olsen's advice [6] and it worked well. There may be better sources out there, but this worked for my short-term goals.

You might think Facebook being such a big company and fostering a developer culture would have better documentation and a better system for interacting with their content, but they were the hardest for me to setup. Once I finally found out how to get an access token without using redirects and a user login, everything went smoothly, but this information isn't easy to find. You'll need to create a new app [7], and then you'll be given an App ID and an App Secret. Use the App ID as the client_id and App Secret as the client_secret in this url:https://graph.facebook.com/oauth/access_token?client_id=XXXXXXXXXXXXX&client_secret=XXXXXXXXXXXXXXXX&grant_type=client_credentials. This will produce the access token you'll need for the API.

Now you are ready to plug this information into my code [8], and give it a test. I have made modifications to this code, but the basic function is the same. I have added a loop and some error handling to allow it to run continuously.

The last thing you need is a slideshow that can update in realtime. For the Mac, this was harder to find than I had imagined. I actually only found one [9] viable solution, but I have to imagine there are more. Please post in the comments if you find a better solution. There was talk of using Automator or an AppleScript, but I didn't have enough time to investigate those fully. This could also be integrated into a website, which was my goal originally, but again time was not my friend.

[1] https://github.com/Instagram/instagram-ruby-gem

[2] https://github.com/sferik/twitter

[3] https://github.com/arsduo/koala

[4] https://apps.twitter.com/

[5] http://instagram.com/developer/

[6] http://dmolsen.com/2013/04/05/generating-access-tokens-for-instagram/

[7] https://developers.facebook.com

[8] https://gist.github.com/barkerd427/ff9e4a3e9a6f53a6923c#file-photo_culler-rb

[9] http://www.partytools.net/photolive/index_e.php#.U3tea1hdWwQ

A Welcome

Welcome to Daniel Barker's blog for dbdevs.com. This blog will mostly focus on small tutorials, prognosticating the future of technology, and anything else related to technology. I am starting this blog to better interact with others in my field. Through a collaborative effort of investigation, we all benefit in a way that isn't feasible on our own. I hope for this blog to provide insights to the future through analyses of current technologies and research. However, it will also serve as a practical repository for small tutorials and examples which I currently have no place for presentation. This blog will also serve to discuss current technology and trends with my perspective on them.