000037358 - How to increase biztier and console heapsizes for RSA Authentication Manager to address console memory allocation errors

Document created by RSA Customer Support Employee on Jun 5, 2019
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000037358
Applies ToRSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.3.0, 8.4.0
IssueThe primary Authentication Manager Security Console was unreachable with a 503 (service unavailable) error, which is usually caused by stuck threads as well as internal errors attributed to group lookup failures that timed out

In the /opt/rsa/am/server/logs/biztier.log we see the following error:
 
Date&Time> <Error> <WebLogicServer> <securidadmin> <biztier> <[ACTIVE] ExecuteThread: '50' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1554957986183> <BEA-000337> <[STUCK] ExecuteThread: 

<Date&Time> <Info> <EJB> <securidadmin> <biztier> <BEA-010227> <EJB exception occurred during invocation from home or business: com.rsa.command.CommandServerEjb30_vraifm_Intf generated exception: COMMAND_EXECUTION_UNEXPECTED_ERROR 
Caused by: com.rsa.common.SystemException: com.rsa.common.UnexpectedDataStoreException: unable to select group from IMS_GROUP_DATA 
Caused by: java.sql.SQLException: The transaction is no longer active - status: 'Marked rollback. [Reason=weblogic.transaction.internal.TimedOutException: Transaction timed out after 600 seconds 
BEA1-7296CB88F9924262E80E]'. No further JDBC access is allowed within this transaction.> 



The following error is seen in the /opt/rsa/am/server/logs/console.log:
 
<Date&Time> <Error> <WebLogicServer> <securidadmin> <console> <[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1554471588597> <BEA-000337> <[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "620" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@6c4c9b83[GET /console-ims/DashBoardUserGroupMembership.do]" 




In the system_log_report from the Security Console - Reporting we see:
 


 16042 GetPrincipalGroupsCommand ActivityMonitorCommand SQL No value specified for parameter 3 Could not get JDBC Connection; Rolled back. Transaction timed out after 599 seconds 
 16099 Administrator “<admin>” attempted to read a group ou=<group or ou> 
 16263 Find user across Identity Sources <LDAP Identity Source name>
 16294 Failed to connect to identity source <LDAP Identity Source name>


 


Messages in the /opt/rsa/am/server/imsTrace.log will be:
 


Date&Time, [[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'], (SSOServiceImpl.java:229), trace.com.rsa.ims.sso.service.SSOServiceImpl, INFO, securidadmin.<company>.com,,,,Request URL = https://securidadmin.<company>.com:7004/console-ims/DashBoardUserGroupsList.do? 

019-04-11 08:20:06,647, [[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'], (DataObjectAccessSql.java:552), trace.com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql, ERROR, securidadmin.<company>.com,,,,failed to lookup domain object of class:class com.rsa.authmgr.internal.admin.principalmgt.dal.AMPrincipal by GUID:4x3b29bd0wdrk47bef99d5cf8fbbxx 

2019-04-11 08:20:43,461, [[ACTIVE] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'], (DataObjectAccessSql.java:552), trace.com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql, ERROR, securidadmin.<company>.com,,,,failed to lookup domain object of class:class com.rsa.authmgr.internal.admin.principalmgt.dal.AMPrincipal by GUID:4x3b29bd0wdrk47bef99d5cf8fbbxx 


INFO | jvm 1 | main | 2019/03/27 19:20:42 | Exception in thread "OARequestHandler Dispatcher Thread" java.lang.OutOfMemoryError: Java heap space 

INFO | jvm 1 | main | 2019/03/27 19:22:46 | Exception in thread "weblogic.GCMonitor" java.lang.OutOfMemoryError: Java heap space 
STATUS | wrapper | main | 2019/03/27 19:23:05 | TERM trapped. Shutting down. 

Caused by: java.lang.OutOfMemoryError: Java heap space 
at java.util.Arrays.copyOfRange(Arrays.java:2694) 
at java.lang.String.<init>(String.java:203) 
at java.lang.StringBuilder.toString(StringBuilder.java:405) 
at com.rsa.authmgr.internal.common.dal.hibernate.util.FilterHQL.createQuery(FilterHQL.java:543) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql$3.doInHibernate(DataObjectAccessSql.java:931) 
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:407) 
at org.springframework.orm.hibernate3.HibernateTemplate.executeFind(HibernateTemplate.java:344) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.executeFind(DataObjectAccessSql.java:902) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.executeSearch(DataObjectAccessSql.java:856) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.executeCiSearch(DataObjectAccessSql.java:821) 
at com.rsa.authmgr.internal.admin.common.dal.sql.DataObjectAccessSql.search(DataObjectAccessSql.java:661) 
at com.rsa.authmgr.internal.admin.agentmgt.dal.sql.AgentAccessSQL.searchAccessibleAgentsByGroups(AgentAccessSQL.java:61) 
at com.rsa.authmgr.internal.admin.agentmgt.impl.AgentLocatorImpl.searchAccessibleAgentsByGroups(AgentLocatorImpl.java:183) 
at com.rsa.authmgr.admin.agentmgt.SearchAccessibleAgentsForPrincipalCommand$Executive.execute(SearchAccessibleAgentsForPrincipalCommand.java:25) 
at com.rsa.authmgr.admin.agentmgt.SearchAccessibleAgentsForPrincipalCommand.performExecute(SearchAccessibleAgentsForPrincipalCommand.java:217) 



The error unable to select group from IMS_GROUP_DATA seen in the biztier log will also show in a system_log_report
TasksTo resolve this issue,
  1. Increase both console and biztier heapsizes memory allocation to at least 3072m in the 8G, 16G and 32G sections of /opt/rsa/am/config/src/scripts/Config.groovy
  2. Restart the Authentication Manager services or reboot.
Resolution1. Increase both console and biztier heapsizes memory allocation to 3072m in /opt/rsa/am/config/src/scripts/Config.groovy
  1. Launch an SSH client, such as PuTTY.
  2. Login to the primary Authentication Manager server as rsaadmin and enter the operating system password.

Note that during Quick Setup another user name may have been selected. Use that user name to login.



  1. Repeat the login process to each replica, one at a time.
  2. Navigate to /opt/rsa/am/config/src/scripts/

cd /opt/rsa/am/config/src/scripts/


  1. Backup the original Config.groovy file:

cp Config.groovy Config.groovy.orig   


  1. Edit the Config.groovy file.
    1. Under the heapsizes normal section, increase biztier as follows in the 8G, 16GB and 32G sections
  2. Edit the biztier in 8G, 16GB and 32G sections increase console and bizier to at least 3072m.

ConfigGroovy

        "8G" {
            opsconsole = "512m"
            biztier = "3072m"
            console = "2048m"
            radiusoc = "100m"
            quicksetup = "512m"
        }
        "16G" {
            opsconsole = "512m"
            biztier = "3072m"
            console = "3072m"
            radiusoc = "100m"
            quicksetup = "512m"
        }
        "32G" {
            opsconsole = "1024m"
            biztier = "8192m"
            console = "4096m"
            radiusoc = "256m"
            quicksetup = "512m"
        }


  1. Save changes then reboot the system
 

There is no need to increase wrapper.java.additional numbers 35 and 36 in /opt/rsa/am/server/wrapper/BiztierServerWrapper.conf or ConsoleServerWrapper.conf per some older instructions. These files are updated by Config.groovy.

NotesAppropriate memory allocation is based on two principles:
  • That you have adequate memory which can be allocated, and
  • That your users are accessing resources that need more memory.

These principles, in turn, indicate there are different ways to address memory out of resources issues; 
  • You can allocate more memory, if you have it, 
  • You can access less resources, or
  • You can do both.

One task that can consume significant memory resources is a user dashboard search in the Security Console:
DashboardSearch.png


Because it searches across all identity sources for a user and the user's associated group, along with their authentication history and accessible restricted agent information, you may actually see the message Loading in the User Dashboard screens:

Loading...


If your Help Desk administrators do not need all this information, or the resources constraints are so tight that you want to prevent your Help Desk administrators from displaying all this information in this resource intensive manner, you can configure the LDAP group search to avoid fetching all this information via the Operations Console's identity source Map tab.

Avoid searching all sub-levels for group information and do not use the MemberOf group search attribute if you have already allocated as much RAM as is recommended and available and still experience the out of memory errors, especially if the out of memory is due to a group search.



  1. Login to the Operations Console.
  2. Navigate to Deployment Configuration > Identity Sources > Manage Existing.
  3. Click on the context arrow next to the identity source and choose Edit.
  4. Click on the Map tab.
  5. Scroll to Directory Configuration - User Groups.
  6. As shown in the images below,
    1. For Search Scope, change from Search all sub-levels to Search only single level.
    2. Under Use MemberOf Attribute, uncheck the option to Enable the use of the MemberOf attribute.

LDAP group search



No Member of.


  1. When done, click Save or Save and Finish.

Attachments

    Outcomes