Authentication Manager Primary Replication Service Fails to Start with "Could Not Acquire Lock"
a day ago
Originally Published: 2018-01-16
Article Number
000041394
Applies To
RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.1 or later
Issue

The RSA Replication (Primary) service on the Authentication Manager primary server enters a SHUTDOWN state. Attempts to restart the service manually fail immediately.

Observable symptoms:

  • Replication failures are observed across the deployment
User-added image
  • When checking service status, all services show as RUNNING except RSA Replication (Primary), which shows as FAILED or SHUTDOWN:
  • Manually restarting the primary replication service returns a FAILED result .
cd /opt/rsa/am/server/
./rsaserv restart primary_replication
Starting RSA Replication (Primary) ********************
RSA Replication (Primary)                                   [FAILED]
  • The /opt/rsa/am/server/logs/PrimaryReplication.log reports the error:
@@@2018-01-15 23:14:48,897 FATAL [WrapperSimpleAppMain ] 
 Service.start(98) | <primary_hostname>,,,,Unhandled exception during service start.
java.lang.RuntimeException: Could not acquire lock. Another instance of this service may be running. Aborting...
	at com.rsa.replication.util.SingleProcessLock.<init>(SingleProcessLock.java:37)
	at com.rsa.replication.util.Service.start(Service.java:91)
	at com.rsa.replication.PrimaryReplicationService.main(PrimaryReplicationService.java:136)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:248)
	at java.lang.Thread.run(Thread.java:680)
Cause

When the Authentication Manager server runs out of disk space, the Primary Replication Service crashes unexpectedly and leaves a stale PrimaryReplicationService.lock file in the wrapper directory. On the next start attempt, the service detects the lock file, assumes another instance is already running, and aborts with the error Could not acquire lock.

This commonly occurs when oversized system log files accumulate in /opt/rsa/am/server/, consuming available disk space until the replication service can no longer operate. For guidance on identifying and resolving large log file accumulation, refer to 000041985 — RSA Authentication Manager 8.x: Large Disk Space Used by Logs — that article covers how to verify archive log placement and reclaim disk space.

Resolution

NOTE: If you have already confirmed there is no disk space issue, skip Part 1 and proceed directly to Part 2.

Part 1: Check and Free Disk Space 

  1. Log in to the Authentication Manager primary server via SSH as rsaadmin.
  2. Check current disk usage:Review the output. If the root filesystem (/) is at 90% or higher, disk space exhaustion is likely the cause. 
  3. Identify large log files in the server directory: 
    rsaadmin@am:~> ls -lah /opt/rsa/am/server | grep '[0-9][G|M]'
    total 910M
    -rw-------  1 rsaadmin rsaadmin  151M Mar  1 01:00 system_2017-11-19_0.log
    -rw-------  1 rsaadmin rsaadmin  159M Mar  2 01:00 system_2017-11-20_0.log
    -rw-------  1 rsaadmin rsaadmin  156M Mar  3 01:00 system_2017-11-21_0.log
    -rw-------  1 rsaadmin rsaadmin  175M Mar  4 01:00 system_2017-11-22_0.log
    -rw-------  1 rsaadmin rsaadmin  269M Mar  5 01:00 system_2017-11-23_0.log
  4. CAUTION: The following step permanently deletes log files. Confirm the files are no longer needed before proceeding. Consider archiving them to an external location if retention is required.
  5. Delete the oversized log files identified in Step 3. Replace the filename pattern with the actual files from your environment:
    rm /opt/rsa/am/server/system_2017-11-2[0-3]_0.log
  6. Re-run the disk check to confirm sufficient space has been freed: df -h 

Part 2: Delete the Stale Lock File and Restart the Replication Service

 

  1. Navigate to the wrapper directory:
    cd /opt/rsa/am/server/wrapper
  2. Delete the stale lock file:
    rm PrimaryReplicationService.lock
  3. Navigate to the server directory:
    cd /opt/rsa/am/server/
  4. Restart the Primary Replication service:
    ./rsaserv restart primary_replication
  5. Confirm the service starts successfully. Expected output:
    Starting RSA Replication (Primary)
    ********************
    RSA Replication (Primary)                                [RUNNING]
  6. Log in to the Operations Console.
  7. From the Home tab, click the Replication Status Report link.
  8. If replication is shown as out of sync, click the Sync link.

Verification: Confirm the Replication Status Report shows a status of Normal for all replica instances.

Notes