Storage: Prepare Virtual or Cloud Storage

Document created by RSA Information Design and Development Employee on Jun 5, 2020Last modified by RSA Information Design and Development Employee on Sep 8, 2020
Version 8Show Document
  • View in full screen mode
 

This section describes how to set up virtual or cloud storage for the following types of component hosts:

Decoder, Log Decoder, Concentrator, Archiver

Virtual or Cloud NetWitness hosts for Decoders, Log Decoders, Concentrators, and Archivers need block storage attached. Make sure that the allocated storage meets all of the storage requirements. Specifically, make sure that the required storage volumes are created (see "Required NetWitness Platform Storage Volumes" in Storage Requirements), and:

  • At least two Block Devices are created for Decoders (meta /session and packet volumes)
  • At least two block devices are created for Concentrators (index and meta volumes)
  • Ensure that block devices can meet the minimum IOPS for expected ingestion rates

Attach the allocated storage to the NetWitness host by following the hosting platforms native procedure.

  • VmWare – Vsphere Console (add disk to VM)
  • Hyper-V – Manager Console (add disk to VM)
  • Azure – Add Managed Disks to virtual instance
  • AWS – Add EBS Storage to virtual instance
  • Google Cloud Platform (GCP) - Add storage to virtual instance

After the storage is attached to the virtual host, proceed to "Task 3 - Allocate Block Devices to Partitions, Volume Groups, and Logical Volumes" in Configure Storage Using the REST API.

NW Server, ESA Primary, ESA Secondary and Malware Analysis

For an extension of /var/netwitness/ partition, attach an external volume.

Run lsblk to get the physical volume name.

If you attach 2 TB disk, run the following commands:

  1. pvcreate <pv_name> (for example, pv_name is /dev/sdc)
  2. vgextend netwitness_vg00 /dev/sdc
  3. lvextend –L 1.9T /dev/netwitness_vg00/nwhome
  4. xfs_growfs /dev/netwitness_vg00/nwhome

RSA recommends the following partition definitions. However, you can change these values based on the retention days.

                  
LVM Folder Block Storage
/dev/netwitness_vg00/nwhome /var/netwitness/Refer to the Cloud Provider Block Storage setup (storage) tables.

Log Collector

For an extension of /var/netwitness/ partition, attach an external volume

Run lsblk to get the physical volume name.

If you attach one 500 GB volume, run the following commands:

  1. pvcreate <pv_name> (for example, pv_name is dev/sdc)
  2. vgextend netwitness_vg00 /dev/sdc
  3. lvextend –L 600G /dev/netwitness_vg00/nwhome
  4. xfs_growfs /dev/netwitness_vg00/nwhome

RSA recommends the following partition definitions. However, you can change these values based on the retention days.

                  
LVM Folder Block Storage
/dev/netwitness_vg00/nwhome /var/netwitness/ Refer to the Cloud Provider Block Storage setup (storage) tables.

Endpoint Log Hybrid

The total disk size required depends on the data retention period. You can use the below per day disk usage indicative values to calculate the required disk size for your deployment. For example, to retain 30 days of data, multiply the below per day disk usage values with 30.

The following table provides disk usage for one full scan. The full scan disk usage values are based on the below event count:

  • Files count -1100

  • Processes count -100

  • Dlls count - 500
  • Drivers count -150

  • Services count - 500

  • Tasks count -100

                                              
 Endpoint Log Hybrid(50K Advance Agents - Disk usage per full scan)
 MetaDB PacketDB SessionDB Index

Total

Log Decoder

220 GB 12 GB5 GB

NA

237 GB
Concentrator 230 GBNA 5 GB 6 GB

241 GB

MongoDBNANANANA

35 GB (First full scan)

30 GB (Subsequent per scan increase)

The following tables provide per day disk usage for tracking data. The total tracking events per agent per day is 29000.

                                              
 Endpoint Log Hybrid (50K Advance Agents - Tracking data without Expanded Network Visibility)
 MetaDB PacketDB SessionDB Index

Total

Log Decoder

1500 GB 140 GB46 GB

NA

1,686 GB
Concentrator 1600 GBNA 46 GB 30 GB

1,676 GB

MongoDBNANANANA

35 GB (First full scan)

1.5 GB (Tracking data per day increase)

The following tables provide per day disk usage for tracking data. Total tracking events per agent per day is 33000

                                              
 Endpoint Log Hybrid (50K Advance Agents - Tracking data with Expanded Network Visibility)
 MetaDB PacketDB SessionDB Index

Total

Log Decoder

1800 GB 152 GB55 GB

NA

2007 GB
Concentrator 1900 GBNA 55 GB 36 GB

1991 GB

MongoDBNANANANA

35 GB (First full scan)

1.5 GB (Tracking data per day increase)

The following table provides per day disk usage for insight agents. The total tracking data per agent per day is 10800 plus 1 full scan daily.

                                              
 Endpoint Log Hybrid (50K Insights Agents with Expanded Network Visibility)
 MetaDB PacketDB SessionDB Index

Total

Log Decoder

500 GB 52 GB18 GB

NA

570 GB
Concentrator600 GBNA 18 GB 13 GB

631 GB

MongoDBNANANANA

35 GB (First full scan)

30 GB (Subsequent per scan increase)

The following table provides Endpoint Agents sizing based on the feature.

                                      
FeatureDescriptionAgent or Endpoint Server
Endpoint Only

Only scan and tracking data

Maximum 50K Endpoint Agents only
Windows Logs OnlyOnly Windows Logs from agents. Assuming 20K events per second supported by Hybrid.             
Maximum 20K Agents: 
  • Generates 20K log events per second
 
File Collection OnlyOnly File Collection from agents. Assuming 20K events per second supported by Hybrid              
Maximum 20K Agents : 
  • Generates 20K log events per second
 
Endpoint and Windows Logs         

Event per second per agent

  • (For Windows Logs) 1 event sent by 1 agent every second
  • (For Tracking Events) 0.4 event sent by 1 agent every second
  • 20K events per second supported by Hybrid

Note: Total agents should be calculated as below:
Hybrid events per second/
(Windows Logs Endpoint Server of 1 agent + Tracking Event Endpoint Server for 1 agent)
For example, 20000 / (1.0 + 0.4)

 

Maximum 15K (approximately) Agents:

  • Generates 15K (approximately) Windows log events

Plus

  • Generates 15K (approximately) Agents EDR data
Endpoint, Windows Logs and File Collection         

Event per second per agent:

  • (For Windows Logs) 1 event sent by 1 agent every second
  • (For Tracking Events) 0.4 event sent by 1 agent every second  
  • (For File Collection) 1 event sent by 1 agent every second
  • 20, 000 events per second supported by Hybrid

Note: Total agents should be calculated as below:
Hybrid events per second/
(Windows Logs Endpoint Server of 1 agent + Tracking Event Endpoint Server for 1 agent + File Collection)
For example, 20000 / (1.0 + 1.0 + 0.4)

 

Maximum 10K (approximately) Agents: 

  • Generates 10K (approximately) Windows log events

Plus

  • Generates 10K (approximately) Endpoint Agents data

Plus

  • Generates 10K (approximately) Agents File Collection data

Extending File Systems

For Endpoint Server, attach external disk for extension of /var/netwitness/ partition, create an external disk with suffix as nwhome.

Follow these steps:

  1. Ensure you have added a new disk. For more information, see "Task 1. Add New Disk" in the Virtual Hosts Installation Guide for RSA NetWitness Platform 11.5. Go to the Master Table of Contents to find all RSA NetWitness Platform 11.x documents.
  2. Execute lsblk and get the physical volume name, for example, if you attach one 6TB disk
  3. pvcreate <pv_name> suppose the PV name is /dev/sdc
  4. vgextend netwitness_vg00 /dev/sdc
  5. lvextend –L 5.9T /dev/netwitness_vg00/nwhome
  6. xfs_growfs /dev/mapper/netwitness_vg00-nwhome

RSA recommended partition for Endpoint Server (can be changed based on the retention days).

                     
LVMFolderSizeDisk Type
/dev/netwitness_vg00/nwhome /var/netwitness/ 6TBHDD

For Mongo DB, attach external disk for extension of /var/netwitness/mongo partition, create an external disk with suffix as nwhome.

Follow these steps:

  1. Ensure you have added a new disk. For more information, see "Task 1. Add New Disk" in the Virtual Hosts Installation Guide for RSA NetWitness Platform 11.5. Go to the Master Table of Contents to find all RSA NetWitness Platform 11.x documents.

  2. Execute lsblk and get the physical volume name, for example, if you attach one 6TB disk
  3. pvcreate <pv_name> suppose the PV name is /dev/sdc1
  4. vgextend hybrid /dev/sdc1
  5. lvextend –L 5.9T /dev/hybrid-vlmng
  6. xfs_growfs /dev/mapper/hybrid-vlmng

RSA recommended partition for Mongo DB (Can be changed based on the retention days). Minimum recommended size for var/netwitness is 500 GB.

                     
LVMFolderSizeDisk Type
/dev/hybrid-vlmng /var/netwitness/mongo 6TBHDD

Additional Endpoint Log Hybrid Partitions

The following partition should be on the volume group endpoint and should be in a single RAID 0 array.

                                 
Folder LVM Volume Group
/var/netwitness/mongo hybrid-mongoendpoint
/var/netwitness/concentrator concentrator-concroot

endpoint

/var/netwitness/concentrator/index

hybrid-concinde

endpoint

/var/netwitness/logdecoderhybrid-ldecroot endpoint

Run lsblk to get the physical volume name and run the following commands:

  1. pvcreate /dev/md0
  2. vgcreate –s 32 endpoint /dev/md0
  3. lvcreate –L <disk_size> -n <lvm_name> endpoint
  4. mkfs.xfs /dev/ endpoint /<lvm_name>
  5. Repeat the above steps for all the LVMs mentioned.

RSA recommends the following partitions. However, you can change these values based on the retention days.

                                      
LVM Folder Block Storage
/dev/netwitness_vg00/nwhome/var/netwitness/Refer to the Cloud Provider Block Storage setup (storage) tables.

/dev/endpoint/hybridmongo

/var/netwitness/mongoRefer to the Cloud Provider Block Storage setup (storage) tables.
/dev/endpoint/concentratorconcroot/var/netwitness/concentratorRefer to the Cloud Provider Block Storage setup (storage) tables.
/dev/endpoint/hybridconcinde/var/netwitness/concentrator/indexRefer to the Cloud Provider Block Storage setup (storage) tables.
/dev/endpoint/hybridldecroot/var/netwitness/logdecoder Refer to the Cloud Provider Block Storage setup (storage) tables.

UEBA

The following procedure attaches an external disk and extends the /var/netwitness/ partition. You must use nwhome as the eternal disk suffix. This procedure illustrates how to add a 2TB disk.

Note: /var/netwitness is the only partition that can reside on this volume.

  1. List the physical volume name.
    lsblk (for example, dev/mapper/sdc)
  2. Extend the /var/netwitness/ partition.
    pvcreate <pv_name>where pv_name is dev/mapper/sdc
    vgextend netwitness_vg00 /dev/mapper/sdc
    lvextend –L 1.9T /dev/mapper/netwitness_vg00/nwhome
    xfs_growfs /dev/mapper/netwitness_vg00-nwhome

This partition is the RSA recommended partition for UEBA. You can change it based on retention days.

Previous Topic:Storage Requirements
You are here
Table of Contents > Prepare Virtual or Cloud Storage
1 person found this helpful

Attachments

    Outcomes