Scheduled clean up job does not run but manual clean up works on Authentication Manager 8.x

[#1]

Article Number

000064473

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.x

Issue

A scheduled clean up job of an external  identity source will not run.  The job is listed in the Security Console under Administration > Batch Jobs, but the information in the Complete Date column never changes, so new scheduled clean up jobs do not run:
 

Image description

Running the Clean Up Unresolvable Users option completes successfully:
 

Image description

Cause

There is an old scheduled cleanup job stuck in the Scheduled Queue, which prevents new scheduled jobs of the same type from running.
 

Resolution

Before completing the steps below, login to the Operations Console and take a backup of the RSA Authentication Manager database (Maintenance > Backup and Restore > Backup Now).  Be sure to make a note of the backup password used for the backup.  If the backup is saved to the local Authentication Manager server, copy the backup off the server and store in a secure location.

1. Enable Secure Shell on the Appliance.
2. Log On to the Appliance Operating System with SSH.
3. Navigate to /opt/rsa/am/utils.
4. Capture the database password:

login as: rsaadmin 
Using keyboard-interactive authentication. 
Password: <enter operating system password> 
Last login: Fri Jan 10 12:33:27 2020 from jumphost.vcloud.local 
RSA Authentication Manager Installation Directory: /opt/rsa/am 
rsaadmin@am82p:~> cd /opt/rsa/am/utils 
rsaadmin@am82p:/opt/rsa/am/utils> ./rsautil manage-secrets -a get com.rsa.db.dba.password 
Please enter OC Administrator username: <enter Operations Console administrator user name> 
Please enter OC Administrator password: <enter Operations Console administrator password> 
com.rsa.db.dba.password: ckg2DBtNZLy80TADWcGqdF0NOJygAQ

5. Navigate to /opt/rsa/am/pgsql/bin
6. Login to the PostgreSQL database with the password captured above

rsaadmin@am82p:/opt/rsa/am/utils> cd ../pgsql/bin
rsaadmin@am82p:/opt/rsa/am/pgsql/bin> ./psql -h localhost -p 7050 -d db -U rsa_dba
Password for user rsa_dba: <enter the database password captured above>
psql.bin (9.4.1)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-SHA, bits: 256, compression: off)
Type "help" for help.

db=#

Note that to capture the database password you can highlight it then right click once to paste it to the session.  It does not display.

7. View all of the batch jobs with the following SQL statement:

db=# SELECT id, name, status, error_message, created_on FROM rsa_rep.ims_batch_job;
                id                |                name                | status | error_message |       created_on
----------------------------------+------------------------------------+--------+---------------+-------------------------
 ef4499f83202a8c03ad753e2c7391a69 | CleanupExpiredSMSAuthenticatorsJob | 4      |               | 2016-07-20 20:10:05.613
 d8f6a4a13202a8c00f4f1514e996d2b8 | LicenseCheckJob                    | 4      |               | 2016-07-20 20:10:05.238
 e01feeee3202a8c021898a6d0bf6ea6c | SYSTEM_CHECK_JOB                   | 4      |               | 2016-07-20 20:10:05.358
 16c9f5fb3202a8c062791fbd934a1417 | AA Maintenance Task                | 4      |               | 2016-07-20 20:10:06.276
 178fa9953202a8c04c8921b31ba08949 | RBA Device Management Job          | 4      |               | 2016-07-20 20:10:06.289
 1829d6e33202a8c07239eccb37def932 | Batch and Schedule Job Cleaner     | 4      |               | 2016-07-20 20:10:06.299
 18934ffl4a430e0234eef932ll13efef | Unresolveable Users and User Groups| 2      |               | 2016-07-20 20:10:06.303
 19ad1f8b3202a8c073cbed810a4649c2 | Archive Audit Logs Job             | 4      |               | 2016-07-20 20:10:06.324
 1b503b503202a8c07a40642da3af56a6 | Copy CLU Audit Logs Job            | 4      |               | 2016-07-20 20:10:06.351
 5d872ea43202a8c0233f34142c00079a | RBA Device Management Job          | 2      |               | 2016-07-20 20:10:06.289
 cb9049cd3202a8c03dc45f0e4c714b89 | Archive Audit Logs Job             | 2      |               | 2016-07-20 20:10:06.324
 1965c3cb3202a8c013aae95ff6138f19 | LicenseCheckJob                    | 2      |               | 2016-07-20 20:10:05.238
 2ce66b473202a8c0742b9b2ed98640e4 | AA Maintenance Task                | 2      |               | 2016-07-20 20:10:06.276
 1ca577a03202a8c0568d6938a3ab8977 | CleanupExpiredSMSAuthenticatorsJob | 2      |               | 2016-07-20 20:10:05.613
 d3f5731c3202a8c0113bd6f8a1571a62 | SYSTEM_CHECK_JOB                   | 2      |               | 2016-07-20 20:10:05.358
 e87fd1783202a8c05d009f10639814ed | LicenseCheckJob                    | 2      |               | 2016-07-20 20:10:05.238
 8a9426cb3202a8c03870a08b4c9efbc3 | Copy CLU Audit Logs Job            | 2      |               | 2016-07-20 20:10:06.351
 c6c264943202a8c07c51c9a6058196dd | Batch and Schedule Job Cleaner     | 2      |               | 2016-07-20 20:10:06.299
 49d5f9bf3202a8c02e7df59e59ad442f | LicenseCheckJob                    | 2      |               | 2016-07-20 20:10:05.238
(19 rows)

Note the Unresolveable Users and User Groups job. 

8. Optionally, the failed batch job for Unresolvable Users and User Groups Cleanup can be listed based on its ID.  Running a SELECT statement where id='<id captured above>' can be run first to verify that this is the correct batch job.

db=#  SELECT name, id, status, created_on, completed_on FROM rsa_rep.ims_batch_job where id='<id_captured_above>';

9. Now run the DELETE command to remove the scheduled batch job that never runs:

db=#  DELETE FROM rsa_rep.ims_batch_job WHERE id=<id_captured_above>';
DELETE 1

This SQL statement removes the particular scheduled batch job that never runs, now locate any stuck scheduled cleanup batch jobs.

10. Run a SELECT statement to locate all scheduled batch jobs:

db=# SELECT name, id, status, created_on, completed_on FROM rsa_rep.ims_batch_job;

11. Look for any scheduled Unresolvable User and User Group Cleanup jobs that do not have a status of 0.  Typically stuck jobs will have a status of 2.  These are the stuck scheduled cleanup jobs that are preventing the new scheduled cleanup from running,
12. Delete the stuck scheduled cleanup jobs, but test by running a SELECT on each job first, copying and pasting in the id for the specific scheduled batch job:

db=# SELECT name, id, status, created_on, completed_on FROM rsa_rep.ims_batch_job where id='<Unresolvable_Users_and_Groups_id>';

13. When ready, run the SQL command to delete the batch job:    

db=# DELETE FROM rsa_rep.ims_schedule_job WHERE id='<Unresolvable_Users_and_Groups_id>';

14. The stuck jobs have been removed from the database.  Try to run a new cleanup.