Large volume of log files generated under AFX folder in RSA Governance & Lifecycle
2 years ago
Article Number
000071973
Applies To

This is a known issue in the following versions:

  • RSA Governance & Lifecycle 8.0
  • SecurID Governance & Lifecycle 7.5.2
This issue also exists in all other versions but to a lesser extent.
Issue

A very large volume of log files may be generated in the AFX logs folder.

The following INFO, WARN and ERRO level log message are logged in a large number in aveksaServer.log file on the RSA Governance & Lifecycle server side:

11/21/2023 23:10:03.565 INFO  (AFX log import) [com.aveksa.server.db.persistence.PersistenceServiceProvider] executeCallableStatement giving up after hitting SQLException: ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00007: unexpected end-of-file encountered
ORA-06512: at "SYS.XMLTYPE", line 272
ORA-06512: at line 1
ORA-06512: at "AVUSER.AFX_PKG", line 121
ORA-06512: at "AVUSER.AFX_PKG", line 121
ORA-06512: at line 1

11/21/2023 23:10:03.567 ERROR (AFX log import) [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00007: unexpected end-of-file encountered
ORA-06512: at "SYS.XMLTYPE", line 272
ORA-06512: at line 1
ORA-06512: at "AVUSER.AFX_PKG", line 121
ORA-06512: at "AVUSER.AFX_PKG", line 121
ORA-06512: at line 1

11/21/2023 23:10:03.568 WARN  (AFX log import) [com.aveksa.afx.server.service.handler.AFXLoggingRequestHandler] Exception importing log data for server 2
com.aveksa.server.db.PersistenceException: java.sql.SQLException: ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00007: unexpected end-of-file encountered
ORA-06512: at "SYS.XMLTYPE", line 272
ORA-06512: at line 1
ORA-06512: at "AVUSER.AFX_PKG", line 121
ORA-06512: at "AVUSER.AFX_PKG", line 121
ORA-06512: at line 1

The following ERROR level log message related to failure to send the logs from AFX to ACM may be logged in the esb.AFX-INIT.log:

ERROR 2024-02-07 13:24:40,844 [[MuleRuntime].uber.35: [05-AFX-INIT].rollOverMoveComponent.BLOCKING @34ebb200] [processor: rollOverMoveComponent/processors/3; event: 28712e01-c5e6-11ee-8470-00155d3c360b] org.mule.runtime.core.internal.exception.DefaultSystemExceptionStrategy: 
********************************************************************************
Message               : Invocation of Method 'moveFiles()' from Class 'com.aveksa.afx.server.log.component.RolloverMoveComponent' without any argument resulted in an error.
Cause: org.apache.commons.io.FileExistsException - Destination 'D:\AFXServer\AFX\esb\logs\processing\esb.AFX-INIT-PERSISTED.log.20240202_115718_626' already exists

The following exception may be logged in the AFX mule_ee.log:

org.apache.commons.io.FileExistsException: Destination '/home/oracle/AFX/esb/logs/sent/esb.AFX-INIT-PERSISTED.log.20231105_010015_789' already exists


Note:

Prolonged inability of AFX to transfer log messages to the main server due to this issue causes these messages to be retained in the JVM memory of the AFX server.  This may subsequently lead to memory failures in AFX.  A variety of different log messages may be generated in the esb.AFX_INIT.log, esb.AFX-MAIN.log, and mule_ee.log files all with the following caused by statement:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

 

Cause

Unexpected XML formatted error messages from AFX causes a failure in the process that sends copies of the logs from AFX to ACM (RSA Governance & Lifecycle sever) that are used to display data under the "Logs" tab on the AFX Servers Menu. 

This causes corresponding failures to be logged in the avekaserver.log and may lead to a backlog of log files on the AFX server and ultimately JVM memory failures. 

A higher frequency of these types of messages in RSA Governance & Lifecycle 8.0.0 AFX increases the chance of this occurring.

Resolution
This issue is resolved in the following versions that prevents these messages from causing a failure:
  • RSA Governance & Lifecycle 8.0 P02
  • SecurID Governance & Lifecycle 7.5.2 P09
Workaround

Identify and resolve known AFX configuration issues that might be causing error messages in the AFX and aveksaServer.log files.  (See Notes)

Resolving the logging problem only stops additional logging, it does not clean up old logs.  The extra log data gathered in the database will be purged by the weekly Data Purging task after 30 days.  The log files themselves will roll over and be deleted after (default 10) rollover files are created.

If the logs have rolled over and no longer have relevant error messages in them (except for those related to the logging failure) then you may remove all the irrelevant log files and capture new log files to identify new issues.

To remove the problem log files:

  1. Stop AFX
  2. Delete all the logs/subfolders under the AFX/esb/logs folder
  3. Start AFX
Notes

The following issue is known to cause frequent log messages resulting in the failure noted above:
000071974 - AFX failure when Proxy is enabled without defining Proxy Host or Port values in RSA Governance & Lifecycle