000037358 - Increase biztier and console heapsizes to address console memory allocation errors for RSA Authentication Manager 8.3 and higher

Document created by RSA Customer Support Employee on Jun 5, 2019Last modified by RSA Customer Support Employee on Aug 24, 2020
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000037358
Applies ToRSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.3.0, 8.4.0
IssueThe primary RSA Authentication Manager Security Console is unreachable with a 503 (service unavailable) error. This error occurs because of stuck threads, internal errors that are attributed to group lookup failures that timed out, and other reasons.

In the /opt/rsa/am/server/logs/biztier.log, we see the following error:
Date&Time> <Error> <WebLogicServer> <securidadmin> <biztier> <[ACTIVE] ExecuteThread: '50' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <>
<1554957986183> <BEA-000337> <[STUCK] ExecuteThread: 

<Date&Time> <Info> <EJB> <securidadmin> <biztier> <BEA-010227> <EJB exception occurred during invocation from home or business: com.rsa.command.CommandServerEjb30_vraifm_Intf generated exception: COMMAND_EXECUTION_UNEXPECTED_ERROR 
Caused by: com.rsa.common.SystemException: com.rsa.common.UnexpectedDataStoreException: unable to select group from IMS_GROUP_DATA 
Caused by: java.sql.SQLException: The transaction is no longer active - status: 'Marked rollback. [Reason=weblogic.transaction.internal.TimedOutException:
Transaction timed out after 600 seconds 
BEA1-7296CB88F9924262E80E]'. No further JDBC access is allowed within this transaction.

The following error is seen in the /opt/rsa/am/server/logs/console.log:
<Date&Time> <Error> <WebLogicServer> <securidadmin> <console> <[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1554471588597> <BEA-000337> <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "620" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@6c4c9b83[GET /console-ims/DashBoardUserGroupMembership.do]" 

Reviewing the System Log report (Security Console > Reporting), we see:

 16042 GetPrincipalGroupsCommand ActivityMonitorCommand SQL No value specified for parameter 3 Could not get JDBC Connection; Rolled back. Transaction timed out after 599 seconds 
 16099 Administrator “<admin>” attempted to read a group ou=<group or ou> 
 16263 Find user across Identity Sources <LDAP Identity Source name>
 16294 Failed to connect to identity source <LDAP Identity Source name>


Messages in the /opt/rsa/am/server/imsTrace.log are:

Date&Time, [[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'], (SSOServiceImpl.java:229), trace.com.rsa.ims.sso.service.SSOServiceImpl, INFO, securidadmin.<company>.com,,,,Request URL = https://securidadmin.<company>.com:7004/console-ims/DashBoardUserGroupsList.do? 

019-04-11 08:20:06,647, [[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'], (DataObjectAccessSql.java:552), trace.com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql, ERROR, securidadmin.<company>.com,,,,failed to lookup domain object of class:class com.rsa.authmgr.internal.admin.principalmgt.dal.AMPrincipal by GUID:4x3b29bd0wdrk47bef99d5cf8fbbxx 

2019-04-11 08:20:43,461, [[ACTIVE] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'], (DataObjectAccessSql.java:552), trace.com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql, ERROR, securidadmin.<company>.com,,,,failed to lookup domain object of class:class com.rsa.authmgr.internal.admin.principalmgt.dal.AMPrincipal by GUID:4x3b29bd0wdrk47bef99d5cf8fbbxx 

INFO | jvm 1 | main | 2019/03/27 19:20:42 | Exception in thread "OARequestHandler Dispatcher Thread" java.lang.OutOfMemoryError: Java heap space 

INFO | jvm 1 | main | 2019/03/27 19:22:46 | Exception in thread "weblogic.GCMonitor" java.lang.OutOfMemoryError: Java heap space 
STATUS | wrapper | main | 2019/03/27 19:23:05 | TERM trapped. Shutting down. 

Caused by: java.lang.OutOfMemoryError: Java heap space 
at java.util.Arrays.copyOfRange(Arrays.java:2694) 
at java.lang.String.<init>(String.java:203) 
at java.lang.StringBuilder.toString(StringBuilder.java:405) 
at com.rsa.authmgr.internal.common.dal.hibernate.util.FilterHQL.createQuery(FilterHQL.java:543) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql$3.doInHibernate(DataObjectAccessSql.java:931) 
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:407) 
at org.springframework.orm.hibernate3.HibernateTemplate.executeFind(HibernateTemplate.java:344) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.executeFind(DataObjectAccessSql.java:902) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.executeSearch(DataObjectAccessSql.java:856) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.executeCiSearch(DataObjectAccessSql.java:821) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.search(DataObjectAccessSql.java:661) 
at com.rsa.authmgr.internal.admin.agentmgt.dal.sql.AgentAccessSQL.searchAccessibleAgentsByGroups(AgentAccessSQL.java:61) 
at com.rsa.authmgr.internal.admin.agentmgt.impl.AgentLocatorImpl.searchAccessibleAgentsByGroups(AgentLocatorImpl.java:183) 
at com.rsa.authmgr.admin.agentmgt.SearchAccessibleAgentsForPrincipalCommand$Executive.execute(SearchAccessibleAgentsForPrincipalCommand.java:25) 
at com.rsa.authmgr.admin.agentmgt.SearchAccessibleAgentsForPrincipalCommand.performExecute(SearchAccessibleAgentsForPrincipalCommand.java:217) 

The error unable to select group from IMS_GROUP_DATA seen in the biztier log is also in a system_log_report.
TasksTo resolve this issue,
  1. Increase both console and biztier heapsizes memory allocation to at least 3072m in the 8G, 16G and 32G sections of /opt/rsa/am/config/src/scripts/Config.groovy. 
    1. SSH to Linux as rsaadmin, do not su to root.
    2. Follow steps in 000039196 - Web Tier status offline/Reinstall status changes to pending connection for RSA Authentication Manager 8.4.
  2. Reboot the RSA Authentication Manager server.
Resolution1. Increase both console and biztier heapsizes memory allocation to 3072m in /opt/rsa/am/config/src/scripts/Config.groovy:
  1. Log on to the Appliance Operating System with SSH.
  2. Log in to the primary RSA Authentication Manager server as rsaadmin and enter the operating system password.

During Quick Setup, another username may have been selected. Use that username to log in.

  1. Repeat the login process to each replica, one at a time.
  2. Go to /opt/rsa/am/config/src/scripts/:

cd /opt/rsa/am/config/src/scripts/

  1. Back up the original Config.groovy file:

cp Config.groovy Config.groovy.orig   

  1. Edit the Config.groovy file.
    1. Under the heapsizes normal section, increase biztier as follows in the 8G, 16GB, and 32G sections.
  2. Edit the biztier in 8G, 16GB, and 32G sections increase console and biztier to at least 3072m:


        "8G" {
            opsconsole = "512m"
            biztier = "3072m"
            console = "2048m"
            radiusoc = "100m"
            quicksetup = "512m"
        "16G" {
            opsconsole = "512m"
            biztier = "3072m"
            console = "3072m"
            radiusoc = "100m"
            quicksetup = "512m"
        "32G" {
            opsconsole = "1024m"
            biztier = "8192m"
            console = "4096m"
            radiusoc = "256m"
            quicksetup = "512m"

  1. Save changes.
  2. Reboot the system.

There is no need to increase wrapper.java.additional numbers 35 and 36 in /opt/rsa/am/server/wrapper/BiztierServerWrapper.conf or ConsoleServerWrapper.conf as described in some older instructions. These files are updated by Config.groovy.after reboot. However, if you added the exact same changes to the two wrapper files that you did to the Config.groovy file, you could change settings with a restart of RSA Authentication Manager services in SSH Linux:

/opt/rsa/am/server/rsaserv restart all

After the restart, verify that the changes took by running the following commands:

ps -ef | grep biztier 
ps -ef | grep console

The output from running these commands should show 4096m for both the minimum and maximum values. For example,

ps -ef | grep biztier 
Dweblogic.management.server=https://dagsasrsa01.r1-core.r1.aig.net:7006 -Xms4096m -Xmx4096m -Dims.denial.of.service

NotesAppropriate memory allocation is based on two principles:
  • That you have adequate memory which can be allocated, and
  • That your users are accessing resources that need more memory.

These principles, in turn, indicate that there are different ways to address memory out of resources issues; 
  • You can allocate more memory, if you have it, 
  • You can access less resources, or
  • You can do both.

One task that can consume significant memory resources is a user dashboard search in the Security Console:

Because the query to populate the dashboard searches across all identity sources for a user and the user's associated group, along with their authentication history and accessible restricted agent information, you may see the message that data is loading in the User Dashboard screens: 

If your Help Desk administrators do not need all this information, or the resources constraints are so tight that you want to prevent your Help Desk administrators from displaying all this information in this resource-intensive manner, you can configure the LDAP group search to avoid fetching all this information in the identity source Map tab in the Operations Console.

Avoid searching all sublevels for group information and do not use the memberOf group search attribute if you have already allocated as much RAM as is recommended and available and still experience the out of memory errors, especially if the out of memory is due to a group search.

  1. Log in to the Operations Console.
  2. Go to Deployment Configuration > Identity Sources > Manage Existing.
  3. Click the context arrow next to the identity source and choose Edit.
  4. Click the Map tab.
  5. Scroll to Directory Configuration - User Groups.
  6. As shown in the images below,
    1. For Search Scope, change from Search all sublevels to Search only single level.
    2. Under Use MemberOf Attribute, clear the option to Enable the use of the MemberOf attribute:

LDAP group search

No Member of.

  1. When done, click Save or Save and Finish.