Security Analytics System Maintenance Checklist

Document created by RSA Information Design and Development on Dec 1, 2016Last modified by RSA Information Design and Development on Aug 2, 2017
Version 8Show Document
  • View in full screen mode
  

This checklist describes tasks to be performed daily and weekly for maintaining the health of your Security Analytics systems.

Introduction

It is important to perform daily maintenance checks on the Security Analytics Server (also known as the SA Head Unit) to keep it running smoothly. This checklist describes which items to check on a regular basis.

Audience

The primary audience for this guide is members of the Administration team who are responsible for maintaining Security Analytics.

Daily Tasks

                                                   
TaskTitleDescription

1.

Check
services
Security Analytics contains a robust Health and Wellness component. It is an excellent early warning system
and alert system for any issues that your deployment of
Security Analytics may face. To learn more about
health and wellness, read the Health and Wellness topic
in the System Maintenance Guide in RSA Link
(https://community.rsa.com/docs/DOC-61563).
 
2.Log
maintenance

It is a best practice to monitor service and systems logs for
content and physical size on a daily basis. It is important to
verify that logs are being rolled over to keep disk partitions
from getting full. (A log is rotated after it reaches a certain
size, for example, 50 MB, and a log control tool such as
logrotate creates a new file in its place for logging
purposes.) Some of the services might not function properly
if the root partition runs over 80%. Follow these steps:

  1. Check disk volume partition space and ensure
    that the root partition is not over 80%. Run the
    following command:
    df
  2. Check the size of the logs in the /etc/logrotate.conf and /etc/logrotate.d directories. Ensure that the logs are getting rolled over. Most services use logrotate
    to manage the logs. logrotate configurations are in
    the /etc/logrotate.conf and /etc/logrotate.d directories. The following list of logs should be monitored:
    /var/log/tokumx/
    /var/log/puppet/
    /var/log/logstash/
    /var/log/audit/
    /var/log/rabbitmq/
    /var/lib/netwitness/uax/logs
    /var/lib/netwitness/rsamalware/jetty/logs
    /opt/rsa/im/
    /opt/rsa/jetty9/logs
    /home/rsasoc/rsa/soc/reporting-engine/logs
    /opt/rsa/sms/
    /opt/rsa/sms/logs
    /var/lib/netwitness/rsamalware/spectrum/logs
  3. Pay special attention to the /var/lib/netwitness/uax/
    scheduler/
    directory. This is where Security Analytics stores all PCAPS that are generated from analysts using the Investigation module. Ensure that this directory does not fill up all the available space in the partition.

 
3.H2
Database

Security Analytics uses an in-memory H2 database. Check the size of the H2 database on a weekly basis. The
H2 database is located in var/lib/netwitness/uax/db.
Notifications and recurring jobs can increase the database
size to over 10 GB. Delete old notifications and unwanted
recurring jobs from the Security Analytics UI.
Recovery steps:

  1. Delete notifications from the Security Analytics
    UI by clicking the Notifications icon ( ) or by
    opening https://<sa_server_IP>/profile#notifications.

  2. Delete the recurring jobs that are not in use from
    https://<sa_server_IP>/profile#notifications,
    OR

  3. Delete the recurring jobs that are not in use from the
    platform.h2.db in https://<sa_server_IP
    >/profile#jobs by following these steps:

    1. Stop jettysrv
    2. cd /var/lib/netwitness/uax/db
    3. cp platform.h2.db platform.h2.db.backup_date
    4. wget http://repo1.maven.org/maven2/com/h2database/
      h2/1.2.147/h2-1.2.147.jar
    5. java -cp /<path to h2-1.2.147.jar org.h2.tools.Shell> -url jdbc:h2:file:platform
    6. Delete the following Quartz jobs from the database:
      1. DELETE FROM QRTZ_TRIGGERS WHERE JOB_NAME in (SELECT JOB_NAME FROM QRTZ_JOB_DETAILS where JOB_CLASS
        _NAME='com.rsa.smc.sa.esa.domain.
        repository.UploadCSVJob');
      2. DELETE FROM QRTZ_JOB_DETAILS where JOB
        _CLASS_NAME='com.rsa.smc.sa.esa.domain.
        repository.UploadCSVJob';
    7. Quit
    8. start jettysrv

    9. If jobs are recurring (such as UploadCSVJob), edit the configuration in the UI and save them. For example, in the previous step, if you deleted the UploadCSVJob, you would need to edit Recurring Enrichment Sources and save them without changes. Enrichment sources are located in the Security AnalyticsUI in Alerts > Configure > Settings > Enrichment Sources.
 
4.Reporting Engine

Monitor the Reporting Engine to ensure that it does not fill up the /home/rsasoc/ partition. Run a df command to determine if there
is an issue. If the command shows that the partition is getting full,
the most common directories that cause this are:

  • /home/rsasoc/rsa/soc/reporting-engine/formattedReports
  • /home/rsasoc/rsa/soc/reporting-engine/resultstore

Recovery steps: Open a ticket with Customer Support, in
case this indicates a unique situation that should be evaluated by Support.

 
5.Malware Colo
Service

The Malware Analysis colo service may fail if the spectrum.h2.db database size is over 10 GB. Avoid running the Malware Analysis colo service for continuous scans and check the size of the database frequently. This service is located on all Security Analytics servers. Do not confuse it with the stand-alone Malware Analysis appliance or virtual machine. If the service fails due to unavailable disk space, follow these steps to resolve the failure:

  1. stop rsaMalwareDevice

  2. Move the contents of /var/lib/netwitness/rsamalware/spectrum/db/
    to a backup location.
  3. start rsaMalwareDevice
 
6.RabbitMQ Server

The Security Analytics server uses the RabbitMQ service for features such as federation, Health and Wellness,
and Incident Management. Ensure that the RabbitMQ service is in a healthy state by running a report and looking for alarms, memory usage, and sockets used. To run this report:

  1. SSH to the Security Analytics server.
  2. Run rabbitmqctl status

Recovery Steps: If RabbitMQ is down, follow these steps:

  1. Collect the logs under /var/log/rabbitmq/
  2. service puppet stop
  3. service rsa-sms stop
  4. service rabbitmq-server stop
  5. service rabbitmq-server start
  6. service rsa-sms start
  7. service puppet start
 

Contact Customer Care

Use the following contact information if you have any questions or need assistance.

Email: support@rsa.com

                         

Preparing to Contact Customer Care

When you contact Customer Care, you should be at your computer. Be prepared to give the following information:

  1. The version number of the RSA Security Analytics product or application you are using.
  2. The type of hardware you are using.

 

Revision History

                                       
RevisionDateDescriptionAuthor
.0010-20-16Initial draftInformation Design and Development
.0110-28-16Second draftInformation Design and Development
.0211-04-16Third draftInformation Design and Development
.0311-21-16Final draftInformation Design and Development

 

You are here
Table of Contents > Security Analytics System Maintenance Checklist

Attachments

    Outcomes