000038075 - Runtime server takes up to one hour to start for RSA Authentication Manager 8.2 and 8.3

When a customer has a large amount of authentication activity logs built up in the database, and then performs an abnormal reboot without services shutdown properly, the archive log feature requires a full scan of the logs, which can last for more than a hour to do with the current index.

The following message is seen:

Primary instance Services failed to start due to Runtime server hanging
"Starting RSA Runtime Server:*****************************************************************

The following tables are large and causing the system scan during startup to take more than one hour.  To confirm this, follow the steps below:

1. Scan time is shown in the /opt/rsa/am/server/logs/BiztierServerWrapper.log file which was written when the system started., as shown below.  You may need to also check BiztierServerWrapper.log.1, BiztierServerWrapper.log.2 and BiztierServerWrapper.log.3 if they exist and are more current.

INFO | jvm 1 | main | 2019/09/10 19:11:49 | <Sep 10, 2019 7:11:49 PM UTC> <Notice> <Stdout> <BEA-000000> <ADMIN log table is scanned, scantime = 113775>
INFO | jvm 1 | main | 2019/09/10 19:32:08 | <Sep 10, 2019 7:32:08 PM UTC> <Notice> <Stdout> <BEA-000000> <RUNTIME log table is scanned, scantime = 1219823>
INFO | jvm 1 | main | 2019/09/10 20:32:14 | <Sep 10, 2019 8:32:14 PM UTC> <Notice> <Stdout> <BEA-000000> <SYSTEM log table is scanned, scantime = 3605873>

2. Follow the steps in 000027335 - Connect to the database using pgSQL in RSA Authentication Manager 8.x to connect to the database.
3. Run the following commands to examine the size:

4. Run the following SQL to show the table sizes:

relation | total_size

------------------------------------------------

rsa_logrep.ims_log_system             | 5954 MB

rsa_logrep.ims_log_audit_rt           | 3891 MB

rsa_rep.am_file_data                  | 1892 MB

rsa_logrep.ims_log_audit_adm          | 261 MB

rsa_rep.am_token                      | 221 MB

rsa_rep.am_token_attr_values          | 140 MB

rsa_rep.ims_principal_d_ext_att_value | 111 MB

rsa_rep.ims_principal_data            | 103 MB

rsa_norep.ims_report_rt_activity      | 75 MB

rsa_rep.am_principal                  | 54 MB

rsa_norep.ims_report_adm_activity     | 47 MB

rsa_rep.am_token_oob                  | 45 MB

rsa_rep.ims_principal_login_date      | 37 MB

rsa_norep.am_report_all_users         | 31 MB

rsa_norep.am_report_user_token_wc     | 28 MB

rsa_rep.am_sms_authenticators         | 23 MB

rsa_norep.ims_report_systemlog        | 21 MB

rsa_norep.am_sms_report               | 14 MB

rsa_norep.am_report_sms_custom_attr   | 5872 kB

rsa_rep_util.am_r2p_primarystatus     | 4296 kB

(20 rows)

1. Stop all services except the database:

2. Run the following SQL commands:

3. Restart services:

4. Log into the Security Console.
5. Navigate to  Administration > Archive Audit Logs > Schedule Log Archival.
6. Set log archive to 30 days in the database and 90 days offline then run the job. Changing the online archive logs retention time from 90 days to 30 days will help in the future.

To prevent the system from scanning the logs, shut the instance down, boot into single user mode, disable services from starting up so system logs can be reviewed:

1. Navigate to /opt/rsa/am/utils/etc.
2. Make a copy of /opt/rsa/am/utils/etc/ims_log_fullscan_needed.properties.
3. Open /opt/rsa/am/utils/etc/ims_log_fullscan_needed.properties in a text editor.
4. Set the following three true values shown below to false in order to skip the scan and see if services start normally.

5. Start services:

6. After system review, these values should be changed back to the original configuration.