000033574 - Adjust swappiness on RSA NetWitness Logs & Network 10.x hosts

Document created by RSA Customer Support Employee on Jul 20, 2016Last modified by RSA Customer Support on Jul 24, 2019
Version 5Show Document
  • View in full screen mode

Article Content

Article Number000033574
Applies ToRSA Product Set: NetWitness Logs & Network/Security Analytics
RSA Product/Service Type: All NetWitness Hosts (appliances & virtual)
RSA Version/Condition: 10.x
Platform: CentOS
O/S Version: 6
Issue
  • Receiving alarms from Health and Wellness about high swap memory utilization.
  • After disabling and re-enabling swap volumes using the following commands, the issue reappears:


sync; echo 3 > /proc/sys/vm/drop_caches   # Clear PageCache, dentries and inodes
swapoff -a
swapon -a


  • When looking in 'top' output can see significant swap volume usage even though physical memory shows a reasonable amount of memory free.
Cause
  • There is no exact way to determine why kernel chooses to page some of the RAM to swap.
  • Mostly are common tasks not required by memory for now, however, their processes are still running.
  • Kernel prefers to page these to swap to make room in the physical memory.
  • We can discourage kernel from paging physical memory to swap space frequently.
  • This can be controlled from a kernel parameter called vm.swappiness
  • Swappiness is a value between 0 and 100 which controls the priority of your system using RAM versus Swap.
  • A swappiness value of 0 means avoid swap as much as possible and only use RAM.
  • A swappiness value of 100 means avoid RAM as much as possible and only use swap.
Resolution
  • We can tune this parameter to discourage using swap as much as possible, but keep in mind, kernel uses other algorithms to choose which to be swapped and which not, tuning this value will only discourage it.
  • A swappiness of 10 means that 90% of the times kernel will choose RAM, and only 10% of the times it can go to swap.


NetWitness 10.x running on CentOS6



  • Run the following one-liner command which immediately changes the value in memory and allows the setting to persist after an OS reboot.


sourcefile=/etc/sysctl.conf;if [[ -f $sourcefile ]]; then grep -q -m1 swappiness "$sourcefile"; if [[ $? -eq 0 ]]; then printf "No change. Linux virtual memory swappiness has already been adjusted as per JIRA ASOC-23864.\n"; else tail -n1 "$sourcefile" | grep -q -E '^$'; if [[ $? -ne 0 ]]; then printf "\n" >> "$sourcefile"; fi; printf "#Change per RSA ASOC-23864\nvm.swappiness=10" >> "$sourcefile"; sysctl vm.swappiness=10; fi; else printf "FATAL error: Unable to locate %s\n" "$sourcefile"; fi


  • If the swappiness parameter was already set, it will print "No change. Linux virtual memory swappiness has already been adjusted as per JIRA ASOC-23864.", if not, then it will set it to 10.
  • When finished restart the affected NetWitness services, such as nwconcentrator.

Note: It is best practice to stop concentrator aggregation before restarting the service.
An example of the service restart commands would be:



stop nwconcentrator
start nwconcentrator


 

Notes

NetWitness 11.x running on CentOS7



  • This does NOT apply to NetWitness 11.x running on CentOS7(rsa-nw-config-management RPM contains the recipe to change). You can confirm the value is 10 by checking the following:

vm.swappiness Configuration in CentOS7 (value applied on next OS restart)



# grep vm.swappiness /etc/sysctl.d/100-nw-base.conf
vm.swappiness = 10


vm.swappiness running value (current value)



# sysctl vm.swappiness
vm.swappiness = 10

 

  • If you require assistance in performing these steps, please contact RSA Support and reference this article.

Attachments

    Outcomes