How to get java heap dump on RSA Authentication Manager Ver. 8.x

[#1]

Article Number

000039527

Applies To

RSA Product Set: RSA SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.4.0, 8.5, 8.6, 8.x
Platform: Linux
O/S Version: SUSE Linux 12
 

Issue

When troubleshooting some Authentication Manager problems such as increasing swap usage or console unavailable 503 errors / [STUCK] Thread errors, RSA Engineering may ask for the java heap dump output for one or more processes that consume the most Memory.

This Knowledge Base, KB article explains how to Identify those top processes and collect Java Heap Dump information for them.
 

Task

1.  Run Linux top command,
2.  Note the top three java process IDs or PIDs, which will be input into. 
3.  jcmd <PID> GC.heap_dump
   Which will output the Java heap dump information for that Process ID to a file,
4. Send the 3 output files to RSA Support for analysis by Engineering.

Engineering may ask you to repeat this process at a later date, but before any restart of AM services or reboot, to monitor the growth of the memory allocations, e.g., problems monitoring increasing swap usage.

Resolution

How to get Java heap dumps on the Process IDs, pids for the highest java memory using processes for a problem like increasing Swap usage or Console unavailable – before you restart AM services – are as follows.

Short answer – Run Linux top command, not top three java process IDs then run 3 jcmd 25460 GC.heap_dump commands against those PIDs with output to files, send files to RSA.

Details:
You identify these processes with the Linux top command

rsaadmin@am83p:/opt/rsa/am/utils> top
top - 12:20:17 up 16 days, 18:20,  1 user,  load average: 0.02, 0.16, 0.70
Tasks: 126 total,   1 running, 125 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.5 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8175388 total,  4972332 used,  3203056 free,    35424 buffers
KiB Swap:  4193276 total,    28212 used,  4165064 free.   914172 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  366 root      20   0       0      0      0 S 0.332 0.000   4:10.77 jbd2/sda1-8
  690 rsaadmin  20   0 2862248 641904  22828 S 0.332 7.852   4:18.27 java
2071 rsaadmin  20   0 3924848 1.063g  38216 S 0.332 13.63   4:29.32 java
3938 rsaadmin  20   0 3763024 1.079g  46412 S 0.332 13.84   2:35.18 java
3961 rsaadmin  20   0 2554580 184568  39396 S 0.332 2.258   0:12.76 java
5595 rsaadmin  20   0   23564   2912   2424 R 0.332 0.036   0:00.10 top
16124 rsaadmin  20   0  108540   4788   3724 S 0.332 0.059   0:16.29 sshd
30482 rsaadmin  20   0 3197580 943232  29700 S 0.332 11.54  10:54.67 java
    1 root      20   0   37436   4000   2796 S 0.000 0.049   2:25.36 systemd
Image description
 <screen shot - Java_heap_top.png>

I identified 3x java PIDs on my system above
java 2071, 3938, and  30482

The heapdump command syntax is like this:
/opt/rsa/am/appserver/jdk/bin/jcmd $PID GC.heap_dump /tmp/test123.hprof

So for my 3 PIDs, I ran like this, with unique output file names too.

/opt/rsa/am/appserver/jdk/bin/jcmd 2071 GC.heap_dump /tmp/java1.hprof
/opt/rsa/am/appserver/jdk/bin/jcmd 3938 GC.heap_dump /tmp/java2.hprof
/opt/rsa/am/appserver/jdk/bin/jcmd 30482 GC.heap_dump /tmp/java3.hprof

/opt/rsa/am/appserver/jdk/bin/jcmd 25460 GC.heap_dump /tmp/java4.hprof

Should get files in /tmp with a message showing PID and
<pid>:
Heap dump file created
Example screenshot for PID 25480 from ‘top’
Image description 
<screenshot - Java_heap_top_PID.png>
 

Notes

======errors======
I tested to try to get a 0-byte output file in /tmp, but was not exactly successful in doing that, saw other errors.

If I use a nonexistent or fake PID, I get an error and not output file.
/opt/rsa/am/appserver/jdk/bin/jcmd 3938 GC.heap_dump /tmp/java2.hprof
3938:
java.io.IOException: No such process

If I use an empty, 0-byte /opt/rsa/am/appserver/jdk/bin/jcmd no output file is created.

If I try modifying the GC.heap_dump option after jcmd, no output file and error.
/opt/rsa/am/appserver/jdk/bin/jcmd 26692 GD.heap_dump /tmp/java6.hprof
26692:
java.lang.IllegalArgumentException: Unknown diagnostic command

/opt/rsa/am/appserver/jdk/bin/jcmd 26692 GC.heap_dump /tmp/java2.hprof
26692:
File exists.
Will not overwrite.

If I use a PID from a process that is not java, like Adminwrapper, I get a different error.
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
====end====errors=====