000038232 - Global Audit Logging Stops after Rabbitmq Service Restart on the Admin Server in RSA NetWitness Platform

Document created by RSA Customer Support Employee on Dec 20, 2019
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000038232
Applies ToRSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.2.x, 11.3.x, 11.4.x
 
IssueGlobal Auditing could be configured on RSA NetWitness Platform as per -  NW Cfg: Configure Global Audit Logging 

However, when rabbitmq-server service is restarted on the Admin Server/Node Zero the Global Audit Logging stops and customers are unable to see the Audit logs in Investigation.

You may see below Error and Warning messages related to Logstash Service on the Admin Server  - 

[root@NWSERVER ~]# systemctl status logstash

Dec 13 18:22:48 NWSERVER logstash[1121]: Caught exception when recovering queue ls-audit
Dec 13 18:22:48 NWSERVER logstash[1121]: Caught exception when recovering consumer amq.xxxxx.xxxx.xxxxx


[root@NWSERVER ~]# tail -100 /var/log/logstash/logstash.log

{:timestamp=>"2018-04-26T11:50:07.386000+0000", :message=>"RabbitMQ connection error: Connection to localhost:5672 refused. Will reconnect in 10 seconds...", :level=>:error}
{:timestamp=>"2018-04-26T11:50:17.569000+0000", :message=>"RabbitMQ connection error: NOT_FOUND - no exchange 'carlos.audit' in vhost '/rsa/system'. Will reconnect in 10 seconds...", :level=>:error}
{:timestamp=>"2018-04-26T11:54:35.711000+0000", :message=>"RabbitMQ connection error: NOT_FOUND - no exchange 'carlos.audit' in vhost '/rsa/system'. Will reconnect in 10 seconds...", :level=>:error}


[root@NWSERVER ~]# tail -1000  /var/log/logstash/logstash-plain.log

[2019-12-13T18:22:48,580][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqps://logstash:XXXXXX@32c5b77d-309d-45ea-9134-9cd5c04791d8:5671/rsa/system", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)}


Customers with large environments may also see high utilization of disk space in /var/netwitness partition of the Admin Server and slowness in the Netwitness UI.

ls.audit queue in Rabbitmq on the Admin Server will be growing in messages and will not be consuming.

[root@NWSERVER ~]# rabbitmqctl list_queues -p /rsa/system | grep ls-audit
ls-audit 98491


 
CauseGlobal Audit Logging is mainly facilitated by the Logstash service and the Rabbitmq service acts as the reliable Message Bus.

However, when the Rabbitmq Service is restarted or crashed logstash service does not recover the connection to the relevant vhost (/rsa/system) and queue (ls.audit) in Rabbitmq. Therefore logstash service should also be manually restarted to reconnect to the RabbitMQ.
ResolutionCurrently this is a known issue and will be fixed in RSA Netwitness 11.5 Version.
WorkaroundThe workaround would be to override the systemd configuration of logstash to get restarted upon RabbitMQ service restart.

"PartOf" systemd option could be used to restart logstash service as a part of rabbitmq-server service restart.

Add the option "PartOf=rabbitmq-server.service" under the [Unit] section in the logstash.service systemd file.

[root@NWSERVER ~]# vi /etc/systemd/system/logstash.service

[Unit]
Description=logstash
PartOf=rabbitmq-server.service

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
ExecStartPre=/etc/systemd/system/logstash-prestart.sh
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target


After that restart the rabbitmq-server service.

# systemctl daemon-reload

# systemctl restart rabbitmq-server.service


 
NotesLog files to check are  - 

/var/log/logstash/logstash.log
/var/log/logstash/logstash-plain.log
/var/netwitness/logstash/logs/rsa-netwitness-audit.log
 

Attachments

    Outcomes