000037845 - The NwAppliance service cannot start in RSA Security Analytics 10.6.x

Document created by RSA Customer Support Employee on Aug 21, 2019
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000037845
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 10.6.6, 10.6.5, 10.6.4, 10.6.3, 10.6.2, 10.6.1, 10.6.0
 
Issuenwappliance is unable to start

/var/log/messages shows the following error:
 

Aug 13 22:24:47 PACLWAPOM01 NwAppliance[17330]: [Engine] [info] Server is shutting down 
Aug 13 22:24:47 PACLWAPOM01 NwAppliance[17330]: [Thread] [info] Stopped thread: Engine Stats id: 17378 
Aug 13 22:24:47 PACLWAPOM01 NwAppliance[17330]: [Engine] [info] Server 'PACLWAPOM01' has been shutdown 
Aug 13 22:24:47 PACLWAPOM01 NwAppliance[17330]: [Engine] [failure] Failed to start engine because std exception: bind: Address already in use 
Aug 13 22:24:47 PACLWAPOM01 init: nwappliance main process (17330) terminated with status 1 
Aug 13 22:24:47 PACLWAPOM01 init: nwappliance main process ended, respawning

 


Aug 13 22:25:24 PACLWAPOM01 NwAppliance[18228]: [Engine] [failure] Failed to start engine because std exception: bind: Address already in use


This usually means some other process(es) is(are) currently using its port(s). 

Looking at netstat.out: 

$ grep "56006\|50106\|50006" netstat.out 
tcp 0 0 127.0.0.1:5672 127.0.0.1:50006 ESTABLISHED 38222/beam.smp 
tcp 0 0 127.0.0.1:5672 127.0.0.1:50106 ESTABLISHED 38222/beam.smp 
tcp 0 0 ::ffff:127.0.0.1:50106 ::ffff:127.0.0.1:5672 ESTABLISHED 31264/java 
tcp 0 0 ::ffff:127.0.0.1:50006 ::ffff:127.0.0.1:5672 ESTABLISHED 31264/java


38222 is rabbitmq, 31264 is jetty: 

$ grep 38222 ps-ef.out 
rabbitmq 38222 1 27 Jul30 ? 3-23:28:34 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname sa@localhost -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/sa@localhost.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/sa@localhost-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/rsa_enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/sa@localhost-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/sa@localhost" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

$ grep 31264 ps-ef.out 
root 31264 1 22 2018 ? 75-09:03:31 /usr/bin/java -Djava.awt.headless=true -Dcom.rsa.netwitness.carlos.LOG_ENABLE_SYSOUT=true -Dcom.netwitness.platform.DB_DEFRAG_ALWAYS=false -Xms6G -Xmx8G -XX:MaxMetaspaceSize=256m -Djdk.tls.ephemeralDHKeySize=2048 -Djavax.net.ssl.keyStore=/opt/rsa/carlos/keystore -XX:+OptimizeStringConcat -XX:+UseLargePages -XX:+UseG1GC -Djetty.state=/opt/rsa/jetty9/jetty.state -Djetty.home=/opt/rsa/jetty9 -Djava.io.tmpdir=/tmp -jar /opt/rsa/jetty9/start.jar etc/jetty-logging.xml etc/jetty-started.xml
TasksSo, in this case, it looks like rabbitmq has established connections to jetty on 50006 and 50106, coincidentally nwappliance ports and hence the issue. 
A reboot on SA host will likely resolve the issue but you could also try restarting jetty first (since it appears to be the one "listening" on those ports) to see if it works (which should), if doesn't can try restarting rabbitmq as well.
Resolution

#restart jettysrv
#service rabbitmq-server restart


If restarting of jetty and rabbitmq service does not work then rebooting the Security Analytics host will resolve the issue.

Attachments

    Outcomes