000039439 - Recreate RSA NetWitness /var/netwitness/concentrator/index mount after two SSD failure

Document created by RSA Customer Support Employee on Nov 11, 2020
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000039439
Applies ToRSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.x
Platform: NetWitness Series 5 Hybrid appliance
O/S Version: CentOS 7
IssueBoth internal RAID 1 SSD disks are failed (Slot 12 and Slot 13) in the RSA NetWitness Series 5 Hybrid appliance.
User-added image

The /var/netwitness/concentrator/index mount is unavailable, and the NetWitness Series 5 Hybrid appliance is unable to boot.
User-added image
Cause
  1. There is a known issue with SanDisk D417 SSD disks that are used in some NetWitness Series 5 Hybrid appliances where they can prematurely be marked as Bad.
    There is a SanDisk D417 SSD firmware update to fix.
     
  2. SSD disks have a finite write life.
    Can see the remaining SSD write endurance in the NetWitness appliance iDRAC, Overview > Storage > Physical Disks, look at each appliance SSD disk at the field "Remaining Rated Write Endurance" which shows the remaining write endurance of the SSDs.
    When it drops to 0% the SSD disk becomes read-only.
    User-added image

    Reference: R730 Remaining Rated Write Endurance 4%
Resolution

Option after RMA replacing both SSD disks and need to manually recreate the NetWitness /var/netwitness/concentrator/index mount



After replacing both SSD disks the RAID 1 configuration for the NetWitness /var/netwitness/concentrator/index mount is lost and needs to be manually recreated.

  1. If the NetWitness appliance will not boot due to a Foreign configuration discovered.
    Choosing the option to import the Foreign configuration will likely create 2 separate Virtual Disks (VD4 & VD5) for the two new SSDs then boot into Single User Mode.

    Reference: Boot RSA NetWitness Platform 11.x appliance into Single User Mode
     
  2. Comment out the /var/netwitness/concentrator/index mount from the /etc/fstab file with the vi editor.
    For example:

    [root@hybrid ~]# grep concentrator/index /etc/fstab
    #/dev/mapper/VolGroup04-concinde /var/netwitness/concentrator/index xfs noatime,nosuid 1 2

  3. Show all Virtual Disks and confirm that both SSD disks are found.
    /opt/MegaRAID/perccli/perccli64 /c0/vall show

    For example:

    [root@hybrid ~]# /opt/MegaRAID/perccli/perccli64 /c0/vall show
    Controller = 0
    Status = Success
    Description = None

    Virtual Drives :
    ==============

    ---------------------------------------------------------------
    DG/VD TYPE  State Access Consist Cache Cac sCC       Size Name
    ---------------------------------------------------------------
    0/0   RAID1 Optl  RW     Yes     RFWBC -   OFF   931.0 GB
    1/1   RAID1 Optl  RW     Yes     RFWBC -   OFF   931.0 GB
    2/2   RAID5 Optl  RW     Yes     RFWBC -   OFF   5.456 TB
    3/3   RAID5 Optl  RW     Yes     RFWBC -   OFF   2.727 TB
    4/4   RAID0 Optl  RW     Yes     RFWBC -   OFF 744.625 GB
    5/5   RAID0 Optl  RW     Yes     RFWBC -   OFF 744.625 GB

    ---------------------------------------------------------------

    Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded
    Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked|
    Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
    FWB=Force WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
    Check Consistency

  4. If VD4 & VD5 exist for the two new 744.625 GB SSDs, then delete both.
    /opt/MegaRAID/perccli/perccli64 /c0/v5 delete
    /opt/MegaRAID/perccli/perccli64 /c0/v4 delete

     
  5. Import any foreign configuration disks, or confirm that there isn’t any.
    /opt/MegaRAID/perccli/perccli64 /c0/fall import

    For example:

    [root@hybrid ~]# /opt/MegaRAID/perccli/perccli64 /c0/fall import
    Controller = 0
    Status = Success
    Description = Couldn't find any foreign Configuration

  6. Re-create the RAID1 group with the two new SSDs.
    /opt/MegaRAID/perccli/perccli64 /c0 add vd r1 drives=32:12,32:13 wb ra cached Strip=128

    For example:

    [root@hybrid ~]# /opt/MegaRAID/perccli/perccli64 /c0 add vd r1 drives=32:12,32:13 wb ra cached Strip=128
    Controller = 0
    Status = Success
    Description = Add VD Succeeded

  7. Check that the "sde" disk (744.6G) exist for the SSDs, and it currently has no mounts.
    lsblk

    For example:

    [root@hybrid ~]# lsblk
    NAME                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    sda                           8:0    0   931G  0 disk
    ├─sda1                        8:1    0     1M  0 part
    ├─sda2                        8:2    0 930.5G  0 part
    │ ├─netwitness_vg00-root    253:0    0  29.3G  0 lvm  /
    │ ├─netwitness_vg00-swap    253:1    0     4G  0 lvm  [SWAP]
    │ ├─netwitness_vg00-nwhome  253:7    0 486.5G  0 lvm  /var/netwitness
    │ ├─netwitness_vg00-warec   253:8    0 390.6G  0 lvm  /var/netwitness/warehouseconnector
    │ ├─netwitness_vg00-varlog  253:9    0    10G  0 lvm  /var/log
    │ └─netwitness_vg00-usrhome 253:10   0    10G  0 lvm  /home
    └─sda3                        8:3    0   519M  0 part /boot
    sdb                           8:16   0   931G  0 disk
    └─sdb1                        8:17   0   931G  0 part
      └─VolGroup01-decometa     253:6    0   931G  0 lvm  /var/netwitness/decoder/metadb
    sdc                           8:32   0  16.4T  0 disk
    └─sdc1                        8:33   0  16.4T  0 part
      ├─VolGroup02-decoroot     253:2    0    30G  0 lvm  /var/netwitness/decoder
      ├─VolGroup02-decoinde     253:3    0    30G  0 lvm  /var/netwitness/decoder/index
      ├─VolGroup02-decosess     253:4    0   100G  0 lvm  /var/netwitness/decoder/sessiondb
      └─VolGroup02-decopack     253:5    0  16.2T  0 lvm  /var/netwitness/decoder/packetdb
    sdd                           8:48   0  10.9T  0 disk
    └─sdd1                        8:49   0  10.9T  0 part
      ├─VolGroup03-concroot     253:11   0    30G  0 lvm  /var/netwitness/concentrator
      ├─VolGroup03-concsess     253:12   0     1T  0 lvm  /var/netwitness/concentrator/sessiondb
      └─VolGroup03-concmeta     253:13   0   9.9T  0 lvm  /var/netwitness/concentrator/metadb
    sde                           8:64   0 744.6G  0 disk

  8. Create a new Physical Volume on the "sde" disk and confirm.
    pvcreate /dev/sde
    pvscan


    For example:

    [root@hybrid ~]# pvcreate /dev/sde
    WARNING: dos signature detected on /dev/sde at offset 510. Wipe it? [y/n]: y
      Wiping dos signature on /dev/sde.
      Physical volume "/dev/sde" successfully created.

    [root@hybrid ~]# pvscan
      PV /dev/sdd1   VG VolGroup03        lvm2 [<10.92 TiB / 0    free]
      PV /dev/sda2   VG netwitness_vg00   lvm2 [<930.47 GiB / 0    free]
      PV /dev/sdc1   VG VolGroup02        lvm2 [16.37 TiB / 0    free]
      PV /dev/sdb1   VG VolGroup01        lvm2 [<930.97 GiB / 0    free]
      PV /dev/sde                         lvm2 [744.62 GiB]
      Total: 5 [29.83 TiB] / in use: 4 [<29.11 TiB] / in no VG: 1 [744.62 GiB]

  9. Create Volume Group "VolGroup4" on the new "sde" disk.
    vgcreate VolGroup04 /dev/sde
    vgscan


    For example:

    [root@hybrid ~]# vgcreate VolGroup04 /dev/sde
      Volume group "VolGroup04" successfully created

    [root@hybrid ~]# vgscan
      Reading volume groups from cache.
      Found volume group "VolGroup03" using metadata type lvm2
      Found volume group "VolGroup04" using metadata type lvm2
      Found volume group "netwitness_vg00" using metadata type lvm2
      Found volume group "VolGroup02" using metadata type lvm2
      Found volume group "VolGroup01" using metadata type lvm2

  10. Create a Logical Volume on the new Volume Group.
    lvcreate -y -n concinde -l 100%FREE VolGroup04
    lvscan


    For example:

    [root@hybrid ~]# lvcreate -y -n concinde -l 100%FREE VolGroup04
      Wiping ntfs signature on /dev/VolGroup04/concinde.
      Logical volume "concinde" created.

    [root@hybrid ~]# lvscan
      ACTIVE            '/dev/VolGroup03/concroot' [30.00 GiB] inherit
      ACTIVE            '/dev/VolGroup03/concsess' [1.00 TiB] inherit
      ACTIVE            '/dev/VolGroup03/concmeta' [<9.89 TiB] inherit
      ACTIVE            '/dev/VolGroup04/concinde' [744.62 GiB] inherit
      ACTIVE            '/dev/netwitness_vg00/nwhome' [486.53 GiB] inherit
      ACTIVE            '/dev/netwitness_vg00/warec' [390.62 GiB] inherit
      ACTIVE            '/dev/netwitness_vg00/root' [29.31 GiB] inherit
      ACTIVE            '/dev/netwitness_vg00/varlog' [10.00 GiB] inherit
      ACTIVE            '/dev/netwitness_vg00/usrhome' [10.00 GiB] inherit
      ACTIVE            '/dev/netwitness_vg00/swap' [4.00 GiB] inherit
      ACTIVE            '/dev/VolGroup02/decoroot' [30.00 GiB] inherit
      ACTIVE            '/dev/VolGroup02/decoinde' [30.00 GiB] inherit
      ACTIVE            '/dev/VolGroup02/decosess' [100.00 GiB] inherit
      ACTIVE            '/dev/VolGroup02/decopack' [<16.22 TiB] inherit
      ACTIVE            '/dev/VolGroup01/decometa' [<930.97 GiB] inherit

  11. Make an xfs file system on the Logical Volume.
    mkfs.xfs /dev/VolGroup04/concinde

    For example:

    [root@hybrid ~]# mkfs.xfs /dev/VolGroup04/concinde
    meta-data=/dev/VolGroup04/concinde isize=512    agcount=4, agsize=48799488 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=0, sparse=0
    data     =                       bsize=4096   blocks=195197952, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal log           bsize=4096   blocks=95311, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0

  12. If any managed-values-* directories exist under the current /var/netwitness/concentrator/index directory, move them away.
    Otherwise, these directories and files will be hidden behind the new "/var/netwitness/concentrator/index" mount, causing lost disk space.

    For example:

    [root@hybrid ~]# du -sh /var/netwitness/concentrator/index
    159M    /var/netwitness/concentrator/index

    [root@hybrid ~]# mv /var/netwitness/concentrator/index /var/netwitness/concentrator/index.old

    [root@hybrid ~]# mkdir -p /var/netwitness/concentrator/index

  13. Un-comment the “/var/netwitness/concentrator/index” mount from the /etc/fstab file with the vi editor.
    For example:

    [root@hybrid ~]# grep concentrator/index /etc/fstab
    /dev/mapper/VolGroup04-concinde /var/netwitness/concentrator/index xfs noatime,nosuid 1 2

  14. Mount the re-built Concentrator index mount.
    mount -a
     
  15. Confirm that the /var/netwitness/concentrator/index mount exists.
    df -hP

    For example:

    [root@hybrid ~]# df -hP
    Filesystem                           Size  Used Avail Use% Mounted on
    /dev/mapper/netwitness_vg00-root      30G  3.4G   26G  12% /
    devtmpfs                              63G     0   63G   0% /dev
    tmpfs                                 63G   12K   63G   1% /dev/shm
    tmpfs                                 63G   18M   63G   1% /run
    tmpfs                                 63G     0   63G   0% /sys/fs/cgroup
    /dev/sda3                            516M  128M  389M  25% /boot
    /dev/mapper/netwitness_vg00-varlog    10G  3.4G  6.7G  34% /var/log
    /dev/mapper/netwitness_vg00-nwhome   487G  4.2G  483G   1% /var/netwitness
    /dev/mapper/VolGroup03-concroot       30G   30G   20K 100% /var/netwitness/concentrator
    /dev/mapper/VolGroup02-decoroot       30G  2.9G   28G  10% /var/netwitness/decoder
    /dev/mapper/VolGroup03-concsess      1.0T  972G   52G  95% /var/netwitness/concentrator/sessiondb
    /dev/mapper/VolGroup03-concmeta      9.9T  9.6T  361G  97% /var/netwitness/concentrator/metadb
    /dev/mapper/VolGroup02-decoinde       30G   40M   30G   1% /var/netwitness/decoder/index
    /dev/mapper/VolGroup02-decosess      100G   95G  5.4G  95% /var/netwitness/decoder/sessiondb
    /dev/mapper/VolGroup01-decometa      931G  882G   50G  95% /var/netwitness/decoder/metadb
    /dev/mapper/netwitness_vg00-usrhome   10G   33M   10G   1% /home
    /dev/mapper/VolGroup02-decopack       17T   16T  834G  95% /var/netwitness/decoder/packetdb
    /dev/mapper/netwitness_vg00-warec    391G   33M  391G   1% /var/netwitness/warehouseconnector
    tmpfs                                 13G     0   13G   0% /run/user/0
    /dev/mapper/VolGroup04-concinde      745G   33M  745G   1% /var/netwitness/concentrator/index

  16. Reboot the Hybrid appliance and confirm the Concentrator service is running and is writing into the rebuilt Concentrator index mount.
    reboot
Workaround

Preferred Option - Recover at least one of the two failed SSD SanDisk disks



  1. Pull and fully re-insert each failed SSD disks (Slot 12 and Slot 13).
  2. Download a copy of the new SanDisk D417 firmware.
    Reference: RSA NetWitness Availability of BIOS & iDRAC Firmware Updates
    Follow the link for SanDisk D417 for the model number(s) ..., under Series 5 to download the new SanDisk D417 firmware version, which is a Windows 64-bit .exe program that can be loaded via the appliance iDRAC.
    User-added image

    Try to update the SanDisk D417 firmware via the iDRAC, Overview > iDRAC Settings > Update and Rollback, choose the update SanDisk firmware Windows 64-bit .exe program and run it to update the firmware.

    Reference: How to upgrade the iDRAC firmware through the web interface on RSA NetWitness Platform appliances
     
  3. If at least one previously failed SSD can be recovered, then reboot the appliance and confirm the Concentrator index mount (/var/netwitness/concentrator/index) is recovered.
  4. RMA replaces any SSD that remains in a Bad state.

Attachments

    Outcomes