Runtime server takes up to one hour to start for RSA Authentication Manager 8.2 and 8.3

[#1]

Article Number

000038075

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.2, 8.3

Issue

When a customer has a large amount of authentication activity logs built up in the database, and then performs an abnormal reboot without services shutdown properly, the archive log feature requires a full scan of the logs, which can last for more than a hour to do with the current index.

The following message is seen:

Primary instance Services failed to start due to Runtime server hanging
"Starting RSA Runtime Server:*****************************************************************

Cause

The following tables are large and causing the system scan during startup to take more than one hour.  To confirm this, follow the steps below:

1. Scan time is shown in the /opt/rsa/am/server/logs/BiztierServerWrapper.log file which was written when the system started., as shown below.  You may need to also check BiztierServerWrapper.log.1, BiztierServerWrapper.log.2 and BiztierServerWrapper.log.3 if they exist and are more current.

INFO | jvm 1 | main | 2019/09/10 19:11:49 | <Sep 10, 2019 7:11:49 PM UTC> <Notice> <Stdout> <BEA-000000> <ADMIN log table is scanned, scantime = 113775>
INFO | jvm 1 | main | 2019/09/10 19:32:08 | <Sep 10, 2019 7:32:08 PM UTC> <Notice> <Stdout> <BEA-000000> <RUNTIME log table is scanned, scantime = 1219823>
INFO | jvm 1 | main | 2019/09/10 20:32:14 | <Sep 10, 2019 8:32:14 PM UTC> <Notice> <Stdout> <BEA-000000> <SYSTEM log table is scanned, scantime = 3605873>

2. Follow the steps in 000027335 - Connect to the database using pgSQL in RSA Authentication Manager 8.x to connect to the database.
3. Run the following commands to examine the size:

SELECT COUNT(*) FROM rsa_logrep.ims_log_system;
SELECT COUNT(*) FROM rsa_logrep.ims_log_audit_rt;  
count
---------  
3183978
(1 row)  
count
---------  
3595064
(1 row)
--------------------------------------------------

4. Run the following SQL to show the table sizes:

SELECT nspname || '.' || relname AS "relation",pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' AND nspname !~ '^pg_toast' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 20;

relation | total_size

------------------------------------------------

rsa_logrep.ims_log_system             | 5954 MB

rsa_logrep.ims_log_audit_rt           | 3891 MB

rsa_rep.am_file_data                  | 1892 MB

rsa_logrep.ims_log_audit_adm          | 261 MB

rsa_rep.am_token                      | 221 MB

rsa_rep.am_token_attr_values          | 140 MB

rsa_rep.ims_principal_d_ext_att_value | 111 MB

rsa_rep.ims_principal_data            | 103 MB

rsa_norep.ims_report_rt_activity      | 75 MB

rsa_rep.am_principal                  | 54 MB

rsa_norep.ims_report_adm_activity     | 47 MB

rsa_rep.am_token_oob                  | 45 MB

rsa_rep.ims_principal_login_date      | 37 MB

rsa_norep.am_report_all_users         | 31 MB

rsa_norep.am_report_user_token_wc     | 28 MB

rsa_rep.am_sms_authenticators         | 23 MB

rsa_norep.ims_report_systemlog        | 21 MB

rsa_norep.am_sms_report               | 14 MB

rsa_norep.am_report_sms_custom_attr   | 5872 kB

rsa_rep_util.am_r2p_primarystatus     | 4296 kB

(20 rows)

Resolution

This issue has been reported in defects AM-30882 and AM-35390.  It was resolved in RSA Authentication Manager 8.4 patch 5.

If you are experiencing the above described symptoms, please do the following:

1. Stop all services except the database:

/opt/rsa/am/server/rsaserv stop all exclude db

2. Run the following SQL commands:

DROP INDEX rsa_logrep.idx_ims_log_system_serial;
CREATE INDEX idx_ims_log_system_serial
ON rsa_logrep.ims_log_system
USING btree (SERIAL, SERVER_NODE_IP);
DROP INDEX idx_ims_log_audit_rt_serial;
CREATE INDEX idx_ims_log_audit_rt_serial
ON rsa_logrep.ims_log_audit_rt
USING btree (SERIAL, SERVER_NODE_IP);

3. Restart services:

/opt/rsa/am/server/rsaserv restart all

4. Log into the Security Console.
5. Navigate to  Administration > Archive Audit Logs > Schedule Log Archival.
6. Set log archive to 30 days in the database and 90 days offline then run the job. Changing the online archive logs retention time from 90 days to 30 days will help in the future.

Workaround

To prevent the system from scanning the logs, shut the instance down, boot into single user mode, disable services from starting up so system logs can be reviewed:

1. Navigate to /opt/rsa/am/utils/etc.
2. Make a copy of /opt/rsa/am/utils/etc/ims_log_fullscan_needed.properties.
3. Open /opt/rsa/am/utils/etc/ims_log_fullscan_needed.properties in a text editor.
4. Set the following three true values shown below to false in order to skip the scan and see if services start normally.

ims.logging.system.fullscan_needed=true
ims.logging.audit.admin.fullscan_needed=true
ims.logging.audit.runtime.fullscan_needed=true
ims.logging.system.fullscan_needed.default=false

5. Start services:

/opt/rsa/am/server/rsaserv start all 

6. After system review, these values should be changed back to the original configuration.

Notes

Resolved in 8.4.0.5