000017528 - The RSA Security Analytics ESA service is immediately stopping after being started

The ESA service is found not to be running using the following command:

[root@sa-esa ~]# service rsa-esa status
RSA NetWitness ESA :: Server is not running.

Attempting to start the service is successful:

[root@sa-esa ~]# service rsa-esa start
Starting RSA NetWitness ESA :: Server...
[root@sa-esa ~]# service rsa-esa status
RSA NetWitness ESA :: Server is running (3718).

However after 1-2 mins, the following 2 commands indicate that the service is no longer listening on port 50030/TCP as the service has stopped again.

[root@sa-esa ~]# netstat -anp | grep :50030
[root@sa-esa ~]# service rsa-esa status
RSA NetWitness ESA :: Server is not running.

The following message appears in the /opt/rsa/esa/logs/esa.log file:

2014-06-17 00:04:42,957 [WrapperSimpleAppMain] FATAL com.rsa.netwitness.esa.server.EsaCommandLine - Fatal throwable
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connections' defined in class path resource [META-INF/esa-handlers.xml]: Invocation of init method failed; nested exception is com.rsa.netwitness.carlos.transport.TransportException: com.rsa.netwitness.carlos.transport.TransportException: java.io.IOException: Failed to bind to server socket: socks://0.0.0.0:50030?transport.connectionTimeout=10000&transport.soTimeout=10000&transport.daemon=true&transport.keepAlive=true&transport.closeAsync=false&transport.soWriteTimeout=10000 due to: java.net.BindException: Address already in use

The issue is due to the fact that the service cannot bind to port 50030 because it believes the address is already in use.

The hostname specified in the loopback address specified in the /etc/hosts file must match the HOSTNAME specified in the /etc/sysconfig/network file, as shown below.

[root@sa-esa ~]# cat /etc/hosts
# Created by NetWitness Installer on Fri May 30 16:37:22 UTC 2014
127.0.0.1 sa-esa localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 sa-esa localhost localhost.localdomain localhost6 localhost6.localdomain6

[root@sa-esa ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=sa-esa.igloo.northpole.org
IPV6_DEFAULTGW=
NETWORKING=yes

1. Make the hostnames in the /etc/hosts and /etc/sysconfig/network files consistent.

Example 1:
In this case we can edit /etc/sysconfig/network to remove the domain suffix of igloo.northpole.org
Edit /etc/sysconfig/network so contents appear as:

NETWORKING=yes
HOSTNAME=sa-esa
IPV6_DEFAULTGW=
NETWORKING=yes

Example 2:
Alternatively we can include the FDQN in loopback specification in /etc/hosts
So /etc/hosts becomes:

# Created by NetWitness Installer on Fri May 30 16:37:22 UTC 2014
127.0.0.1 sa-esa sa-esa.igloo.northpole.org localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 sa-esa sa-esa.igloo.northpole.org localhost localhost.localdomain localhost6 localhost6.localdomain6

It would probably be a good idea to specify domain in /etc/sysconfig/network as well.
So /etc/sysconfig/network becomes:

NETWORKING=yes
HOSTNAME=sa-esa.igloo.northpole.org
IPV6_DEFAULTGW=
NETWORKING=yes
DOMAINNAME=igloo.northpole.org

2. Restart the ESA Service.

[root@sa-esa ~]# service rsa-esa start
Starting RSA NetWitness ESA :: Server...

3. Check ESA service log for the successful bind message:

[root@sa-esa ~]# tail -n 100 /opt/rsa/esa/logs/esa.log | grep :50030
2014-06-20 04:14:12,814 [WrapperSimpleAppMain] INFO  com.rsa.netwitness.carlos.transport.spring.MessageEndpointServiceExporter - Service on channel com.rsa.netwitness.esa.ESAProtocol$ServiceMessage bound to local endpoint jms://0.0.0.0:50030?carlos.dispatch.queue=256&carlos.dispatch.pool=32