000032294 - Rate limiting errors in /var/log/messages on an RSA Security Analytics VLC due to collectd errors

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000032294
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics Server, Virtual Log Collector
RSA Version/Condition: 10.4.x,10.5.x
Platform: CentOS
O/S Version: EL6
IssueRate-limiting messages (as shown below) appear whenever there is a large number of log messages being recorded in the /var/log/messages file.
These errors can be stopped by setting appropriate parameters for $SystemLogRateLimitInterval and  $SystemLogRateLimitBurst in the /etc/rsyslog.conf file, as explained in the article entitled RSA Security Analytics Log Decoder is dropping system messages due to rate-limiting. However, adjusting these parameters may not help if a large number of log messages are logging.
Dec 17 05:05:10 DELGG7SIEMVLC7 rsyslogd-2177: imuxsock begins to drop messages from pid 2059 due to rate-limiting 
Dec 17 05:05:13 DELGG7SIEMVLC7 rsyslogd-2177: imuxsock lost 34281 messages from pid 2059 due to rate-limiting
CauseOne reason that rate-limiting messages may appear is due to a large number of collectd errors being logged, as shown in the example below.
Dec 24 11:42:53 DELGG7SIEMVLC7 collectd[2059]: NgNativeReader_NwLogCollector-FastUpdate: nwsdk failure: NwResponseData returned 0 and NwLastError failed 
Dec 24 11:42:53 DELGG7SIEMVLC7 collectd[2059]: NgNativeReader_NwLogCollector-FastUpdate: nwsdk failure: NwResponseData returned 0 and NwLastError failed
Dec 24 11:42:53 DELGG7SIEMVLC7 collectd[2059]: NgNativeReader_NwLogCollector-FastUpdate: nwsdk failure: NwResponseData returned 0 and NwLastError failed
Resolution
Follow below steps to stop the collectd errors from logging on the appliance.
  1. Connect to the appliance via SSH as the root user.
  2. Issue the command below and confirm that it executes successfully.
    puppet agent -t

  3. Stop and restart the collectd service.
    service collectd stop
    service collectd start

    Sample output of these commands is shown below.
    User-added image
  4. Verify the latest /var/log/messages do not have the collectd errors.
If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.

Attachments

    Outcomes