000030656 - Multiple problems may prevent AFX Server startup if it has been started using root

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000030656
Applies Toall AFX versions
IssueIf an AFX server is started as root (via command line <path-to-AFX>/afx start) problems can occur that will prevent correct management of the server and its connectors. 
When subsequently trying to correctly manage the server as the less privileged "afxuser":
  • afx stop attempt may unexpectedly result in "ERROR: java.io.IOException: Operation not permitted"
  • afx start attempt may unexpectedly result in "Mule Enterprise Edition is already running."
If the AFX server is subsequently stopped as root and then started as the "afxuser" other problems may occur such as:

  • command line indicates "WARNING!! Timed out waiting for AFX applications to start. Please check AFX application log files for detailed status information."


  • The UI may indicate that the server is running (green) but connectors are showing status of Deployed (yellow) or Not Deployed (red)
CauseStarting an AFX server as root will change some AFX file permissions to root which prevents successful startup by less privileged users.  Additionally, even after stopping an AFX server as root AFX processes may still be running and using required AFX ports. 
Some problem indicators are:
Have executed afx stop but an AFX-related process is still running:
acm-691:~ # ps -ef |grep AFX
root     20019     1  0 09:57 pts/1    00:00:10 /etc/alternatives/java_sdk_1.6.0/bin/java -Xms256M -Xmx256M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/home/oracle/AFX/activemq/conf; -Dactivemq.home=/home/oracle/AFX/activemq -Dactivemq.base=/home/oracle/AFX/activemq -jar /home/oracle/AFX/activemq/bin/run.jar start
root     25849 16499  0 10:37 pts/1    00:00:00 grep AFX
File permissions are incorrect:
 
oracle@acm-691:~/AFX> bin/setPerms.sh
Updating permissions for files in /home/oracle/AFX
chmod: changing permissions of `/home/oracle/AFX/mule/logs/active/mule.AFX-MAIN-PERSISTED.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mule/logs/active/mule.AFX-CONN-AD-connector.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mule/logs/active/mule.AFX-INIT-PERSISTED.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mule/logs/sent/mule.AFX-CONN-AD-connector.log.20150624_095849_883': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mule/logs/sent/log-batch-290214ce-e1e6-4759-b2ab-1e9392f24c30.xml': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mule/conf/client.keystore': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/activemq/data/kahadb/lock': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/logs/manager.2015-06-24.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/logs/host-manager.2015-06-24.log': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/indexes_7p8q': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/segments.gen': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/segments_2': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/_0.cfs': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/workspaces/default/index/_7p8n/cache.inSegmentParents': Operation not permitted
chmod: changing permissions of `/home/oracle/AFX/mmc-console/mmc-data/db/log/log1054.dat': Operation not permitted
done

Checking one of the files above and it can be seen that it is incorrectly owned by root:
oracle@acm-691:~/AFX> ll /home/oracle/AFX/mule/conf/client.keystore
-rw-r--r-- 1 root root 5329 Mar  2 15:07 /home/oracle/AFX/mule/conf/client.keystore


Checking AFX ports such as 61616, 8585, or 8444 via netstat may show a port unexpectedly in use:
oracle@acm-691:~/database/DBA/AVDB/scripts> netstat -an | grep 61616
tcp        0      0 127.0.0.1:18212         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18207         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18206         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18213         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18208         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18166         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18214         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18167         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:18168         127.0.0.1:61616         ESTABLISHED
tcp        0      0 127.0.0.1:61616         :::*                    LISTEN

...

 
Resolution1) As root execute <path-to-afx>/afx stop.
2) Check if any AFX processes are still running via  ps -ef | grep AFX
3) If there are any AFX processes still running stop them using  kill -9 <pid>.  For example:
acm-691:~ # ps -ef |grep AFX
root    
20019     1  0 09:57 pts/1    00:00:10 /etc/alternatives/java_sdk_1.6.0/bin/java -Xms256M -Xmx256M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/home/oracle/AFX/activemq/conf; -Dactivemq.home=/home/oracle/AFX/activemq -Dactivemq.base=/home/oracle/AFX/activemq -jar /home/oracle/AFX/activemq/bin/run.jar start
root     25849 16499  0 10:37 pts/1    00:00:00 grep AFX

acm-691:~ # kill -9 20019
4) Ensure that all AFX files have the correct owner and group.  For example if the "afxuser" is oracle execute the following commands to set the owner and group as appropriate:
acm-691:/home/oracle/AFX # chown oracle -R *
acm-691:/home/oracle/AFX # chgrp oinstall -R *

5) Now with all AFX processes stopped and correct file permissions start the AFX server using the "afxuser" account
oracle@acm-691:~/AFX> ./afx start
 

Attachments

    Outcomes