000034607 - Puppet agent failure alarm seen on the Health and Wellness module in RSA Security Analytics

Document created by RSA Customer Support Employee on Dec 29, 2016Last modified by RSA Customer Support on Jun 28, 2018
Version 6Show Document
  • View in full screen mode

Article Content

Article Number000034607
Applies ToRSA Product Set: RSA NetWitness Logs & Network (Security Analytics)
RSA Product/Service Type: Security Analytics UI, Health and Wellness
RSA Version/Condition: 10.4.x, 10.5.x, 10.6.x
Platform: CentOS
O/S Version: 6

 
IssueIn Health and Wellness on the Alarms tab, a Puppet Agent Failure alarm is seen.

User-added image
 
CauseThere can be several reasons for this alarm relating to both the puppet agent and the puppet master service.

1) Check that puppet agent service is running on the host by running the following command:

service puppet status


2) Issuing the command below on the affected host will provide further information on the error.

puppet agent -t


Due to a known issue with a slow memory leak in the puppetmaster module, when puppet agent -t is run on the node then the following error message may be seen:


Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 400 on Server: Cannot allocate memory - fork(2)


User-added image
 

3) Confirm if the puppetmaster service is running on the NetWitness Server:


service puppetmaster status
Resolution
  1. On the host, if `service puppet status` does not show the puppet agent service as running, the service may be restarted using the following command:

    service puppet restart

  2. On the NetWitness Server if `service puppetmaster status` does not show the puppetmaster service as running, the service may be restarted using the following command:

    service puppetmaster restart

WorkaroundIf restarting the puppetmaster service allows `puppet agent -t` to complete normally, a workaround for the puppet master memory leak is to schedule a cron job to restart this service on a regular basis. 

WARNING: This entry should be removed prior to commencing upgrade tasks to avoid upgrade disruption.

1. Add the following entry to /var/spool/cron/root which restarts the puppetmaster service every Saturday at 11:50 PM UTC

# Entry Description: Restart puppetmaster every Saturday at 11:50 PM UTC. Restarting puppet agent as well in case puppet master has subsys locked.
50 23 * * Sat (/sbin/service puppetmaster stop > /dev/null 2>&1) && (/sbin/service puppet stop > /dev/null 2>&1) && (/sbin/service puppetmaster start > /dev/null 2>&1) && rm -f /var/lib/puppet/state/agent_catalog_run.lock && (/sbin/service puppet start > /dev/null 2>&1)


You can do this either via editing this file or by running the following 2 commands:

printf '%s\n' '# Entry Description: Restart puppetmaster every Saturday at 11:50 PM UTC. Restarting puppet agent as well in case puppet master has subsys locked.' >> /var/spool/cron/root
printf '%s\n' '50 23 * * Sat (/sbin/service puppetmaster stop > /dev/null 2>&1) && (/sbin/service puppet stop > /dev/null 2>&1) && (/sbin/service puppetmaster start > /dev/null 2>&1) && rm -f /var/lib/puppet/state/agent_catalog_run.lock && (/sbin/service puppet start > /dev/null 2>&1)' >> /var/spool/cron/root


2. Execute the command below to reload the crond daemon and put the scheduled job into effect.

service crond restart


Example output:

# service crond restart
Stopping crond:                                            [  OK  ]
Starting crond:                                            [  OK  ]


3. The success of the cron daemon restart (and monitoring of run jobs) can be done by reviewing /var/log/cron. For example the last 10 entries:

tail /var/log/cron


Example entry in /var/log/cron

Jun 26 04:25:01  CROND[9664]: (root) CMD ((/sbin/service puppetmaster stop > /dev/null 2>&1) && (/sbin/service puppet stop > /dev/null 2>&1) && (/sbin/service puppetmaster start > /dev/null 2>&1) && rm -f /var/lib/puppet/state/agent_catalog_run.lock && (/sbin/service puppet start > /dev/null 2>&1) )
NotesTested by running script every 5 minutes using:

*/5 * * * * (/sbin/service puppetmaster stop > /dev/null 2>&1) && (/sbin/service puppet stop > /dev/null 2>&1) && (/sbin/service puppetmaster start > /dev/null 2>&1) && rm -f /var/lib/puppet/state/agent_catalog_run.lock && (/sbin/service puppet start > /dev/null 2>&1)

and


tailf /var/log/cron

Attachments

    Outcomes