RSA Identity Governance & Lifecycle aveksaServer.log file filling with warning messages "Logging event having message" and "is not logged as queue is full"

[#1]

Article Number

000035152

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle

Issue

The RSA Identity Governance and Lifecycle aveksaServer.log file is filling up with the following warning messages:

11/02/2016 16:45:34.291 WARN (ThreadPool-0) [com.aveksa.server.db.DatabaseAppender] Logging event having message
'Logging event having message 'Logging event having message 'Logging event having message 'Logging event having message
'Logging event having message 'Logging event having message 'Logging event having message 'Logging event having message
'Logging event having message 'Logging event having message 'Logging event having message 'Logging event having message
'Logging event having message 'Logging event having message 'Logging event having message 'Logging event having message
'Logging event having message 'Logging event having message 'Logging event having message 'Logging event having message
​'Logging event having message 'Logging event having message 'Logging event having message 'Logging event having

and

is not logged as queue is full' is not logged as queue is full' is not logged as queue is full' is not logged as queue is full'
is not logged as queue is full' is not logged as queue is full' is not logged as queue is full' is not logged as queue is full'
​is not logged as queue is full' is not logged as queue is full' is not logged as queue is full' is not logged as queue is full'

Cause

Information logged to the aveksaServer.log is logged both to the physical file and to database table T_AV_APPLICATION_LOG. These errors occur when so much information is being logged to the database that it cannot keep up with the volume of data being logged.

 

Resolution

The following are possible known ways that can cause the volume of data to increase and have been observed to date

1. DEBUG logging enabled.  In any environment that has a reasonable amount of load setting, setting the root level logging to DEBUG will cause this error and bring the system to a grinding halt since in addition to logging Aveksa DEBUG statements it will log system level DEBUG including all application server-wide statements.   Root level DEBUG logging should only be used on systems with low usage as a last resort. Instead, use the category log levels. If you have a more specific class level you want to DEBUG, you can use the Advanced tab (usually engineering will provide class specific debugging requirements).
2. Password Management is enabled, password expiration emails are configured to be sent, and hundreds to thousands of emails are failing to be sent causing exceptional logging activity. 

Make sure that DEBUG is not enabled:

1. DEBUG flag under Admin > System > Logs must be unchecked.

Image description

2. The root logger level under Admin > System > Logs > Settings is enabled, set it to INFO.

Image description

Password Management Settings

If unchecking the DEBUG flag does not resolve the issue, or if the DEBUG flag is not checked, check your Password Management settings, as follows:
 

1. Check if Password Management is enabled under Admin > System > Settings. 

Image description

2. If enabled, check your Password Policy and see if the setting to warn users is enabled. This setting sends an email to users whose passwords are expiring x days before their passwords expire  To do this, select Requests > Password Management > Password Policies tab and click on the name of the password policy.

Image description

 

3. Check the email log to see if emails are failing to be sent (Admin > Email > Log tab). Look under Processing Result to see if emails are not being sent.  For example:

Image description

 

Alternative solutions

1. Fix the reason the emails are not being sent (no server defined in the above example), or
2. Disable the sending of password expired emails, or
3. Disable Password Management.

Workaround

If ACM can be started:

When neither of these scenarios fits your situation, a workaround would be to disable logging altogether on the first page and set root logging level to OFF and contact RSA Identity Governance & Lifecycle Support for a root cause analysis.
 

Image description

Image description

If ACM will not start

When excessive logging is preventing ACM from starting, then it will not be possible to make any logging configuration changes via the RSA Identity Governance & Lifecycle administrative console.  Instead, the equivalent logging configuration changes must be made by editing the aveksa-log4j.properties file via the Linux shell. 

The aveksa-log4j.properties file location is:

• Standalone: $AVEKSA_WILDFLY_HOME/standalone/configuration/aveksa-log4j.properties
• Cluster: $AVEKSA_WILDFLY_HOME/domain/configuration/aveksa-log4j.properties

Steps to do this are:

1. SSH to the server where ACM is installed, and login as root.
2. Make a backup copy of the aveksa-log4j.properties file:

cp aveksa-log4j.properties aveksa-log4j.properties.bk

3. Edit the aveksa-log4j.properties file.  Any text editor, such as vi, can be used.
4. Modify the log4j.logger.com.aveksa line in the file to specify a more limited logging level.  In this example, the current setting is DEBUG.  The word DEBUG should be changed to one of the following:

• INFO
• WARN
• ERROR
• FATAL. 

INFO or WARN is normally recommended.  This change is case sensitive, so all uppercase text must be used to define the logging level.

5. Make sure all the other log4j.logger and log4j.rootlogger lines are set to INFO, as shown in bold below.

##
##  JBoss Log4J configuration file
##
##
log4j.rootLogger=INFO, aveksaServerLog

##
## Set default logging levels
##
log4j.logger.com.aveksa=DEBUG
log4j.logger.org.apache=INFO
log4j.logger.org.hibernate=INFO

Do not add, change or delete any other lines, words or punctuation in this file.

6. Save the file, then restart ACM.

# acm restart

If assistance is required, contact RSA Identity Governance & Lifecycle Support .