AFX Server and Connector failures if AFX is started as the root user in RSA Identity Governance & Lifecycle

[#1]

Article Number

000030656

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle
RSA Version/Condition: All
 

Issue

If an AFX server is started as the root user, problems can occur that will prevent correct management of the server and its connectors. 
 

EXAMPLES:

• When subsequently trying to manage the server as the less privileged afx user, the following errors may occur:

• afx stop may fail with:

ERROR: java.io.IOException: Operation not permitted

• afx start may fail with

Mule Enterprise Edition is already running

• If the AFX server is subsequently stopped as the root user and later started as the afx user, other problems may occur such as:

• afx start may fail with:

WARNING!! Timed out waiting for AFX applications to start. Please check AFX application log files for detailed status information

• The user interface (AFX > Server) may indicate that the AFX server is Running (green) but the AFX connectors (AFX > Connectors) may show with a status of Deployed (yellow) or Not Deployed (red)

• An AFX-related process is still running:

ps -ef | grep AFX
root     20019     1  0 09:57 pts/1  00:20:57 /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/bin/java 
-Xms512m -Xmx512m -Dorg.apache.activemq.UseDedicatedTaskRunner=true 
-Djava.util.logging.config.file=logging.properties 
-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true -XX:MaxMetaspaceSize=512m 
-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled 
-XX:+UseStringDeduplication -XX:InitiatingHeapOccupancyPercent=5 
-Dcom.sun.management.jmxremote.port=1099 
-Dcom.sun.management.jmxremote.password.file=/home/oracle/AFX/activemq/conf/jmx.password 
-Dcom.sun.management.jmxremote.access.file=/home/oracle/AFX/activemq/conf/jmx.access 
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote 
-Dactivemq.classpath=/home/oracle/AFX/activemq/conf; -Dactivemq.home=/home/oracle/AFX/activemq 
-Dactivemq.base=/home/oracle/AFX/activemq -Djava.security.egd=file:/dev/./urandom 
-jar /home/oracle/AFX/activemq/bin/run.jar start

• File permissions are incorrect as noted by executing the below as the afx user:

cd $AFX_HOME/bin
./setPerms.sh
Updating permissions for files in /home/oracle/AFX
chmod: changing permissions of `/home/oracle/AFX/esb/logs/active/esb.AFX-MAIN-PERSISTED.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/esb/logs/active/esb.AFX-CONN-AD-connector.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/esb/logs/active/esb.AFX-INIT-PERSISTED.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/esb/logs/sent/esb.AFX-CONN-AD-connector.log.20190624_095849_883': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/esb/logs/sent/log-batch-290214ce-e1e6-4759-b2ab-1e9392f24c30.xml': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/esb/conf/client.keystore': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/activemq/data/kahadb/lock': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/logs/manager.2019-06-24.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/logs/host-manager.2019-06-24.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/indexes_7p8q': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/segments.gen': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/segments_2': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/_0.cfs': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/cache.inSegmentParents': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/db/log/log1054.dat': Operation not permitted done

• Checking any one of the above files shows the file owned by root instead of by the afx user:

ll /home/oracle/AFX/esb/conf/client.keystore
-rw-r--r-- 1 root root 5329 Mar  2 15:07 /home/oracle/AFX/esb/conf/client.keystore

• Checking AFX ports such as 61616, 8585, or 8444 via netstat may show a port unexpectedly in use:

cd $AVEKSA_HOME/database/DBA/AVDB/scripts 
netstat -an | grep 61616
tcp        0      0 127.0.0.1:18212         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18207         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18206         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18213         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18208         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18166         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18214         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18167         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18168         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:61616         :::*                    LISTEN

 

Cause

Starting an AFX server as the root user will change some AFX file permissions to root which prevents successful startup by less privileged users such as the afx user. Additionally, even after stopping an AFX server as root, AFX processes may still be running and using required AFX ports. 
 

Resolution

To resolve this issue, stop AFX as the root user, remove any existing AFX processes, modify the AFX file ownership and restart AFX as the afx user.

1. As the root  user, login in to the server where AFX is installed.
2. Stop AFX

cd $AFX_HOME/bin
./afx stop

3. Check for any AFX processes that may still be running after afx has shut down. Kill any AFX processes found:

For example:

ps -ef | grep AFX
root     20019     1  0 09:57 pts/1  00:20:57 /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/bin/java 
-Xms512m -Xmx512m -Dorg.apache.activemq.UseDedicatedTaskRunner=true 
-Djava.util.logging.config.file=logging.properties 
-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true -XX:MaxMetaspaceSize=512m 
-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled 
-XX:+UseStringDeduplication -XX:InitiatingHeapOccupancyPercent=5 
-Dcom.sun.management.jmxremote.port=1099 
-Dcom.sun.management.jmxremote.password.file=/home/oracle/AFX/activemq/conf/jmx.password 
-Dcom.sun.management.jmxremote.access.file=/home/oracle/AFX/activemq/conf/jmx.access 
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote 
-Dactivemq.classpath=/home/oracle/AFX/activemq/conf; -Dactivemq.home=/home/oracle/AFX/activemq 
-Dactivemq.base=/home/oracle/AFX/activemq -Djava.security.egd=file:/dev/./urandom 
-jar /home/oracle/AFX/activemq/bin/run.jar start
kill -9 20019

4. Ensure that all AFX files and directories have the correct owner and group.  For example if the afx user is oracle, execute the following commands to set the owner and group as appropriate:

/home/oracle/AFX # chown oracle -R *
/home/oracle/AFX # chgrp oinstall -R *

5. After all AFX processes have been stopped and the file permissions and ownership corrected on the AFX files and directories, start AFX as the afx user:

afx start