000017528 - The RSA Security Analytics ESA service is immediately stopping after being started

The ESA service is found not to be running using the following command:

[root@sa-esa ~]# service rsa-esa status
RSA NetWitness ESA :: Server is not running.

Attempting to start the service is successful:

[root@sa-esa ~]# service rsa-esa start
Starting RSA NetWitness ESA :: Server...
[root@sa-esa ~]# service rsa-esa status
RSA NetWitness ESA :: Server is running (3718).

However, after 1-2 mins, the following 2 commands indicate that the service is no longer listening on port 50030/TCP as the service has stopped again.

[root@sa-esa ~]# netstat -anp | grep :50030
[root@sa-esa ~]# service rsa-esa status
RSA NetWitness ESA :: Server is not running.

The following message appears in the /opt/rsa/esa/logs/esa.log file:

2014-06-17 00:04:42,957 [WrapperSimpleAppMain] FATAL com.rsa.netwitness.esa.server.EsaCommandLine - Fatal throwable
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connections' defined in class path resource [META-INF/esa-handlers.xml]: Invocation of init method failed; nested exception is com.rsa.netwitness.carlos.transport.TransportException: com.rsa.netwitness.carlos.transport.TransportException: java.io.IOException: Failed to bind to server socket: socks://0.0.0.0:50030?transport.connectionTimeout=10000&transport.soTimeout=10000&transport.daemon=true&transport.keepAlive=true&transport.closeAsync=false&transport.soWriteTimeout=10000 due to: java.net.BindException: Address already in use

The issue is due to the fact that the service cannot bind to port 50030 because it believes the address is already in use.

The hostname specified in the loopback address specified in the /etc/hosts file must match the HOSTNAME specified in the /etc/sysconfig/network file, as shown below.

[root@sa-esa ~]# cat /etc/hosts
# Created by NetWitness Installer on Fri May 30 16:37:22 UTC 2014
127.0.0.1 sa-esa localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 sa-esa localhost localhost.localdomain localhost6 localhost6.localdomain6

[root@sa-esa ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=sa-esa.igloo.northpole.org
IPV6_DEFAULTGW=
NETWORKING=yes

1. Make the hostnames in the /etc/hosts and /etc/sysconfig/network files consistent.
   

   
   

   Example 1:
   In this case we can edit /etc/sysconfig/network to remove the domain suffix of igloo.northpole.org
   Edit /etc/sysconfig/network so contents appear as:

   
   
   

   
   

   
   NETWORKING=yes
   HOSTNAME=sa-esa
   IPV6_DEFAULTGW=
   NETWORKING=yes

   
   

   
   

   
    
   
   

   Example 2:
   Alternatively, we can include the FDQN in loopback specification in /etc/hosts
   So /etc/hosts becomes:

   
   
   

   
   

   # Created by NetWitness Installer on Fri May 30 16:37:22 UTC 2014
   127.0.0.1 sa-esa sa-esa.igloo.northpole.org localhost localhost.localdomain localhost4 localhost4.localdomain4
   ::1 sa-esa sa-esa.igloo.northpole.org localhost localhost.localdomain localhost6 localhost6.localdomain6

   
   

   
   
   

   It would probably be a good idea to specify the domain in /etc/sysconfig/network as well.
   So /etc/sysconfig/network becomes:

   
   
   

   
   

   NETWORKING=yes
   HOSTNAME=sa-esa.igloo.northpole.org
   IPV6_DEFAULTGW=
   NETWORKING=yes
   DOMAINNAME=igloo.northpole.org

   
   

   
   
2. Restart the ESA Service.
   

   
   

   [root@sa-esa ~]# service rsa-esa start
   Starting RSA NetWitness ESA :: Server...

   
   

   
   
3. Check ESA service log for the successful bind message:
   

   
   

   [root@sa-esa ~]# tail -n 100 /opt/rsa/esa/logs/esa.log | grep :50030
   2014-06-20 04:14:12,814 [WrapperSimpleAppMain] INFO  com.rsa.netwitness.carlos.transport.spring.MessageEndpointServiceExporter - Service on channel com.rsa.netwitness.esa.ESAProtocol$ServiceMessage bound to local endpoint jms://0.0.0.0:50030?carlos.dispatch.queue=256&carlos.dispatch.pool=32