How to get java heap dump on RSA Authentication Manager 8.x
2 years ago
Originally Published: 2021-01-13
Article Number
000055584
Applies To
RSA Product Set: RSA SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.x
Platform: Linux
O/S Version: SUSE Linux 12
 
Issue
When troubleshooting some Authentication Manager problems such as increasing swap usage or console unavailable 503 errors / [STUCK] Thread errors, RSA Engineering may ask for the java heap dump output for one or more processes that consume the most Memory.

This article explains how to identify those top processes and collect Java heap dump information for them.
 
Tasks
  1.  Run Linux top command.
  2. Note the top three Java process IDs (PID), which will be input to the JCMD utility that is used to send diagnostic command requests to the JVM.
  3. Run the command for each PID to capture Java heap dump information to a file:
  4. Send the three output files to RSA Support for analysis by Engineering.
Engineering may ask for this process to be run again at a later date, but before any restart of Authentication Manager services or performing a reboot, monitor the growth of the memory allocations, e.g., problems monitoring increasing swap usage.
Resolution
Steps to get Java heap dumps on the PIDs using the highest Java memory must be done before restarting Authentication Manager services.
  1. Identify these Java processes with the Linux top command

rsaadmin@am83p:/opt/rsa/am/utils> top
top - 12:20:17 up 16 days, 18:20,  1 user,  load average: 0.02, 0.16, 0.70
Tasks: 126 total,   1 running, 125 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.5 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8175388 total,  4972332 used,  3203056 free,    35424 buffers
KiB Swap:  4193276 total,    28212 used,  4165064 free.   914172 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  366 root      20   0       0      0      0 S 0.332 0.000   4:10.77 jbd2/sda1-8
  690 rsaadmin  20   0 2862248 641904  22828 S 0.332 7.852   4:18.27 java
2071 rsaadmin  20   0 3924848 1.063g  38216 S 0.332 13.63   4:29.32 java
3938 rsaadmin  20   0 3763024 1.079g  46412 S 0.332 13.84   2:35.18 java
3961 rsaadmin  20   0 2554580 184568  39396 S 0.332 2.258   0:12.76 java
5595 rsaadmin  20   0   23564   2912   2424 R 0.332 0.036   0:00.10 top
16124 rsaadmin  20   0  108540   4788   3724 S 0.332 0.059   0:16.29 sshd
30482 rsaadmin  20   0 3197580 943232  29700 S 0.332 11.54  10:54.67 java
    1 root      20   0   37436   4000   2796 S 0.000 0.049   2:25.36 systemd
Java_heap_top.png

The screen shot above shows three Java PIDs (2071, 3938, and 30482).

The heapdump command syntax is /opt/rsa/am/appserver/jdk/bin/jcmd $PID GC.heap_dump /tmp/test123.hprof

For the three PIDs above run the command once for each PID. Be sure each file has a unique output file name as well.
/opt/rsa/am/appserver/jdk/bin/jcmd 2071 GC.heap_dump /tmp/java1.hprof
/opt/rsa/am/appserver/jdk/bin/jcmd 3938 GC.heap_dump /tmp/java2.hprof
/opt/rsa/am/appserver/jdk/bin/jcmd 30482 GC.heap_dump /tmp/java3.hprof

In the example above, files are saved to /tmp. 
 
Notes
Errors
I tested to try to get a 0-byte output file in /tmp, but was not exactly successful in doing that, saw other errors:
  • If I use a nonexistent or fake PID, I get an error and not an output file.
/opt/rsa/am/appserver/jdk/bin/jcmd 3938 GC.heap_dump /tmp/java2.hprof
3938:
java.io.IOException: No such process
  • If I use an empty, 0-byte /opt/rsa/am/appserver/jdk/bin/jcmd no output file is created.
  • If I try modifying the GC.heap_dump option after jcmd, no output file and error.
/opt/rsa/am/appserver/jdk/bin/jcmd 26692 GD.heap_dump /tmp/java6.hprof
26692:
java.lang.IllegalArgumentException: Unknown diagnostic command
/opt/rsa/am/appserver/jdk/bin/jcmd 26692 GC.heap_dump /tmp/java2.hprof
26692:
File exists.
Will not overwrite.
  • If I use a PID from a process that is not ava, like Adminwrapper, I get a different error.
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

Related Articles

Trending Articles

Don't see what you're looking for?