• The RSA Replication (Primary) Service on a primary instance shows a status of SHUTDOWN, as shown:

rsaadmin@primary:/opt/rsa/am/server> ./rsaserv status
RSA Database Server                                        [RUNNING]
RSA Administration Server with Operations Console          [RUNNING]
RSA RADIUS Server Operations Console                       [RUNNING]
RSA Runtime Server                                         [RUNNING]
RSA RADIUS Server                                          [RUNNING]
RSA Console Server                                         [RUNNING]
RSA Replication (Primary)                                  [SHUTDOWN]

• When trying to view the Replication Status Report from the Operations Console of the primary, the following error displays:

• When restarting the service using the command ./rsaserv start primary_replication, the service shows as RUNNING, then fails again after few seconds.
• There is no issue with disk space. 
• System date and time are correct.
• The /opt/rsa/am/server/logs/PrimaryReplication.log shows the following errors:

@@@2019-11-28 04:40:52,967 SYSTEM [WrapperSimpleAppMain ]
 Service.start(88) | XXXX,,,,Starting the service.
@@@2019-11-28 04:41:00,157 FATAL [WriteP2R nextSweepId: 22203619 mode: TMR minPrimaryChangeId: 257689041 primaryChangeIdRange: 221161 primaryChangeIdRangeHWM: 9223372036854775807]
 ServiceCallable.runMainLoop(83) | XXXX,,,,Unhandled exception during main loop. Shutting down all service threads.
java.lang.RuntimeException: unable to obtain primary changes from database
        at com.rsa.replication.WriteP2R.createSweepFile(WriteP2R.java:209)
        at com.rsa.replication.WriteP2R.sweepFromInsideTransaction(WriteP2R.java:191)
        at com.rsa.replication.WriteP2R.sweep(WriteP2R.java:239)
        at com.rsa.replication.WriteP2R.workIfNeccessary(WriteP2R.java:166)
        at com.rsa.replication.ReplicationRunnable.work(ReplicationRunnable.java:77)
        at com.rsa.replication.util.ServiceCallable.work(ServiceCallable.java:110)
        at com.rsa.replication.util.ServiceCallable.runMainLoopUnsafe(ServiceCallable.java:99)
        at com.rsa.replication.util.ServiceCallable.runMainLoop(ServiceCallable.java:79)
        at com.rsa.replication.util.ServiceCallable.call(ServiceCallable.java:42)
        at com.rsa.replication.util.ServiceCallable.call(ServiceCallable.java:1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL []; Ran out of memory retrieving query results.; nested exception is 
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:257)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:605)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:639)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:664)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
        at com.rsa.replication.util.WrappedJdbcTemplate.query(WrappedJdbcTemplate.java:62)
        at com.rsa.replication.WriteP2R$2.sweepUsingAppropriateQuery(WriteP2R.java:106)
        at com.rsa.replication.WriteP2R.createSweepFile(WriteP2R.java:207)
        ... 13 more
Caused by: org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1817)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
        at com.rsa.replication.util.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:64)
        at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:646)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:589)
        ... 19 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.postgresql.core.PGStream.ReceiveTupleV3(PGStream.java:364)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1814)
        ... 26 more
@@@2019-11-28 04:41:00,199 SYSTEM [Shutdown Hook ]
 Service.shutdownVm(198) | XXXX,,,,Stopped the service.

1. Before moving forward with the steps below, please login to the Operations Console on your primary and take a backup (Maintenance > Backup > Backup Now).
2. Launch an SSH client, such as PuTTY.
3. Login to the primary Authentication Manager server as rsaadmin and enter the operating system password.
4. Navigate to /opt/rsa/am/server/wrapper

cd /opt/rsa/am/server/wrapper

4. Take a backup of the PrimaryReplicationWrapper.conf as below:

cp PrimaryReplicationWrapper.conf PrimaryReplicationWrapper.conf.ORIG

5. Edit the PrimaryReplicationWrapper.conf using:

vim PrimaryReplicationWrapper.conf

6. Update the following parameters from

# Server-specific JVM params
wrapper.java.additional.18=-Xms20m
wrapper.java.additional.19=-Xmx128m

# Server-specific JVM params
wrapper.java.additional.18=-Xms128m
wrapper.java.additional.19=-Xmx256m

7. Save and exit the file using :wq!
8. Restart the primary_replication service:

/opt/rsa/am/server/rsaserv restart primary_replication