| 1. | Check
services | Security Analytics contains a robust Health and Wellness component. It is an excellent early warning system
and alert system for any issues that your deployment of
Security Analytics may face. To learn more about
health and wellness, read the Health and Wellness topic
in the System Maintenance Guide in RSA Link (https://community.rsa.com/). | |
| 2. | Log
maintenance | It is a best practice to monitor service and systems logs for
content and physical size on a daily basis. It is important to
verify that logs are being rolled over to keep disk partitions
from getting full. (A log is rotated after it reaches a certain
size, for example, 50 MB, and a log control tool such as
logrotate creates a new file in its place for logging
purposes.) Some of the services might not function properly
if the root partition runs over 80%. Follow these steps: - Check disk volume partition space and ensure
that the root partition is not over 80%. Run the
following command:
df - Check the size of the logs in the /etc/logrotate.conf and /etc/logrotate.d directories. Ensure that the logs are getting rolled over. Most services use logrotate
to manage the logs. logrotate configurations are in
the /etc/logrotate.conf and /etc/logrotate.d directories. The following list of logs should be monitored:
/var/log/tokumx/
/var/log/puppet/
/var/log/logstash/
/var/log/audit/
/var/log/rabbitmq/
/var/lib/netwitness/uax/logs
/var/lib/netwitness/rsamalware/jetty/logs
/opt/rsa/im/
/opt/rsa/jetty9/logs
/home/rsasoc/rsa/soc/reporting-engine/logs
/opt/rsa/sms/
/opt/rsa/sms/logs
/var/lib/netwitness/rsamalware/spectrum/logs -
Pay special attention to the /var/lib/netwitness/uax/
scheduler/ directory. This is where Security Analytics stores all PCAPS that are generated from analysts using the Investigation module. Ensure that this directory does not fill up all the available space in the partition. | |
| 3. | H2
Database | Security Analytics uses an in-memory H2 database. Check the size of the H2 database on a weekly basis. The
H2 database is located in var/lib/netwitness/uax/db.
Notifications and recurring jobs can increase the database
size to over 10 GB. Delete old notifications and unwanted
recurring jobs from the Security Analytics UI.
Recovery steps: -
Delete notifications from the Security Analytics
UI by clicking the Notifications icon (  ) or by
opening https://<sa_server_IP>/profile#notifications. -
Delete the recurring jobs that are not in use from
https://<sa_server_IP>/profile#notifications,
OR -
Delete the recurring jobs that are not in use from the
platform.h2.db in https://<sa_server_IP
>/profile#jobs by following these steps: - Stop jettysrv
- cd /var/lib/netwitness/uax/db
- cp platform.h2.db platform.h2.db.backup_date
- wget http://repo1.maven.org/maven2/com/h2database/
h2/1.2.147/h2-1.2.147.jar - java -cp /<path to h2-1.2.147.jar org.h2.tools.Shell> -url jdbc:h2:file:platform
- Delete the following Quartz jobs from the database:
- DELETE FROM QRTZ_TRIGGERS WHERE JOB_NAME in (SELECT JOB_NAME FROM QRTZ_JOB_DETAILS where JOB_CLASS
_NAME='com.rsa.smc.sa.esa.domain.
repository.UploadCSVJob'); - DELETE FROM QRTZ_JOB_DETAILS where JOB
_CLASS_NAME='com.rsa.smc.sa.esa.domain.
repository.UploadCSVJob';
- Quit
-
start jettysrv - If jobs are recurring (such as UploadCSVJob), edit the configuration in the UI and save them. For example, in the previous step, if you deleted the UploadCSVJob, you would need to edit Recurring Enrichment Sources and save them without changes. Enrichment sources are located in the Security AnalyticsUI in Alerts > Configure > Settings > Enrichment Sources.
| |
| 4. | Reporting Engine | Monitor the Reporting Engine to ensure that it does not fill up the /home/rsasoc/ partition. Run a df command to determine if there
is an issue. If the command shows that the partition is getting full,
the most common directories that cause this are:
- /home/rsasoc/rsa/soc/reporting-engine/formattedReports
- /home/rsasoc/rsa/soc/reporting-engine/resultstore
Recovery steps: Open a ticket with Customer Support, in
case this indicates a unique situation that should be evaluated by Support. | |
| 5. | Malware Colo
Service | The Malware Analysis colo service may fail if the spectrum.h2.db database size is over 10 GB. Avoid running the Malware Analysis colo service for continuous scans and check the size of the database frequently. This service is located on all Security Analytics servers. Do not confuse it with the stand-alone Malware Analysis appliance or virtual machine. If the service fails due to unavailable disk space, follow these steps to resolve the failure: -
stop rsaMalwareDevice - Move the contents of /var/lib/netwitness/rsamalware/spectrum/db/
to a backup location. - start rsaMalwareDevice
| |
| 6. | RabbitMQ Server | The Security Analytics server uses the RabbitMQ service for features such as federation, Health and Wellness,
and Incident Management. Ensure that the RabbitMQ service is in a healthy state by running a report and looking for alarms, memory usage, and sockets used. To run this report: - SSH to the Security Analytics server.
- Run rabbitmqctl status
Recovery Steps: If RabbitMQ is down, follow these steps: - Collect the logs under /var/log/rabbitmq/
- service puppet stop
- service rsa-sms stop
- service rabbitmq-server stop
- service rabbitmq-server start
- service rsa-sms start
- service puppet start
| |
on Mar 22, 2017