1. | Check services | Security Analytics contains a robust Health and Wellness component. It is an excellent early warning system and alert system for any issues that your deployment of Security Analytics may face. To learn more about health and wellness, read the Health and Wellness topic in the System Maintenance Guide in RSA Link (https://community.rsa.com/). | |
2. | Log maintenance | It is a best practice to monitor service and systems logs for content and physical size on a daily basis. It is important to verify that logs are being rolled over to keep disk partitions from getting full. (A log is rotated after it reaches a certain size, for example, 50 MB, and a log control tool such as logrotate creates a new file in its place for logging purposes.) Some of the services might not function properly if the root partition runs over 80%. Follow these steps: - Check disk volume partition space and ensure
that the root partition is not over 80%. Run the following command: df - Check the size of the logs in the /etc/logrotate.conf and /etc/logrotate.d directories. Ensure that the logs are getting rolled over. Most services use logrotate
to manage the logs. logrotate configurations are in the /etc/logrotate.conf and /etc/logrotate.d directories. The following list of logs should be monitored: /var/log/tokumx/ /var/log/puppet/ /var/log/logstash/ /var/log/audit/ /var/log/rabbitmq/ /var/lib/netwitness/uax/logs /var/lib/netwitness/rsamalware/jetty/logs /opt/rsa/im/ /opt/rsa/jetty9/logs /home/rsasoc/rsa/soc/reporting-engine/logs /opt/rsa/sms/ /opt/rsa/sms/logs /var/lib/netwitness/rsamalware/spectrum/logs -
Pay special attention to the /var/lib/netwitness/uax/ scheduler/ directory. This is where Security Analytics stores all PCAPS that are generated from analysts using the Investigation module. Ensure that this directory does not fill up all the available space in the partition. | |
3. | H2 Database | Security Analytics uses an in-memory H2 database. Check the size of the H2 database on a weekly basis. The H2 database is located in var/lib/netwitness/uax/db. Notifications and recurring jobs can increase the database size to over 10 GB. Delete old notifications and unwanted recurring jobs from the Security Analytics UI. Recovery steps: -
Delete notifications from the Security Analytics UI by clicking the Notifications icon ( ) or by opening https://<sa_server_IP>/profile#notifications. -
Delete the recurring jobs that are not in use from https://<sa_server_IP>/profile#notifications, OR -
Delete the recurring jobs that are not in use from the platform.h2.db in https://<sa_server_IP >/profile#jobs by following these steps: - Stop jettysrv
- cd /var/lib/netwitness/uax/db
- cp platform.h2.db platform.h2.db.backup_date
- wget http://repo1.maven.org/maven2/com/h2database/
h2/1.2.147/h2-1.2.147.jar - java -cp /<path to h2-1.2.147.jar org.h2.tools.Shell> -url jdbc:h2:file:platform
- Delete the following Quartz jobs from the database:
- DELETE FROM QRTZ_TRIGGERS WHERE JOB_NAME in (SELECT JOB_NAME FROM QRTZ_JOB_DETAILS where JOB_CLASS
_NAME='com.rsa.smc.sa.esa.domain. repository.UploadCSVJob'); - DELETE FROM QRTZ_JOB_DETAILS where JOB
_CLASS_NAME='com.rsa.smc.sa.esa.domain. repository.UploadCSVJob';
- Quit
-
start jettysrv - If jobs are recurring (such as UploadCSVJob), edit the configuration in the UI and save them. For example, in the previous step, if you deleted the UploadCSVJob, you would need to edit Recurring Enrichment Sources and save them without changes. Enrichment sources are located in the Security AnalyticsUI in Alerts > Configure > Settings > Enrichment Sources.
| |
4. | Reporting Engine | Monitor the Reporting Engine to ensure that it does not fill up the /home/rsasoc/ partition. Run a df command to determine if there is an issue. If the command shows that the partition is getting full, the most common directories that cause this are:
- /home/rsasoc/rsa/soc/reporting-engine/formattedReports
- /home/rsasoc/rsa/soc/reporting-engine/resultstore
Recovery steps: Open a ticket with Customer Support, in case this indicates a unique situation that should be evaluated by Support. | |
5. | Malware Colo Service | The Malware Analysis colo service may fail if the spectrum.h2.db database size is over 10 GB. Avoid running the Malware Analysis colo service for continuous scans and check the size of the database frequently. This service is located on all Security Analytics servers. Do not confuse it with the stand-alone Malware Analysis appliance or virtual machine. If the service fails due to unavailable disk space, follow these steps to resolve the failure: -
stop rsaMalwareDevice - Move the contents of /var/lib/netwitness/rsamalware/spectrum/db/
to a backup location. - start rsaMalwareDevice
| |
6. | RabbitMQ Server | The Security Analytics server uses the RabbitMQ service for features such as federation, Health and Wellness, and Incident Management. Ensure that the RabbitMQ service is in a healthy state by running a report and looking for alarms, memory usage, and sockets used. To run this report: - SSH to the Security Analytics server.
- Run rabbitmqctl status
Recovery Steps: If RabbitMQ is down, follow these steps: - Collect the logs under /var/log/rabbitmq/
- service puppet stop
- service rsa-sms stop
- service rabbitmq-server stop
- service rabbitmq-server start
- service rsa-sms start
- service puppet start
| |