000026684 - RAID controller swap procedure for RSA Security Analytics and RSA NetWitness Platform appliances

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support on Sep 20, 2019
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000026684
Applies ToRSA Product Set:  Security Analytics, NetWitness Logs & Network
RSA Product/Service Type: Core appliance
RSA Version/Condition:  10.x, 11.x
Platform: CentOS 6, CentOS 7
IssueRAID controller swap procedure for RSA Security Analytics and RSA NetWitness appliances.
Due to an RMA, I need to swap the RAID controller in my appliance for a new one.  What steps do I take to accomplish this?
ResolutionPlease follow these steps:
  1. In /etc/fstab, comment out (by adding a # to the beginning of the line) all filesystems that mount under /var/netwitness.
  2. (optional but recommended) Edit /boot/grub/grub.conf and remove the following part of the active boot section’s ‘kernel’ line: ‘console=ttyS0,115200n8r’.  This will disable serial console redirection at boot should the appliance go into maintenance mode should it not be able to mount a RAID filesystem.  If one does not follow this step and the appliance goes into maintenance mode at boot, you will only be able to see the prompts and interact with the OS by attaching a console device to the serial port rather than by using VGA/Keyboard/Mouse.
  3. Using NetWitness Administrator, stop capture on the Decoder.  One should do this before shutting the service down to ensure all indexes are saved properly so as to prevent a need to reindex any sessions upon the next start of the service.
  4. Stop the service with the stop nwdecoder (SA10.x/CentOS6) or systemctl stop nwdecoder(NW11.x/CentOS7) command, based on the CentOS version.
  5. Run this command to shut down the appliance:
    shutdown -h now
  6. Take note of which port the DACs are plugged into, and which internal port the internal RAID enclosure is connected to.
  7. Open the case and perform the swap of the RAID controller, plugging the cables back into their original slots.  Move the cable for the old battery backup unit (BBU) from the old to the new controller.  Power the appliance back on.
  8. During boot, you should see a message that says ‘foreign configuration detected’ on the RAID controller, or words to that effect.  Hit ‘F’ to try to import.  It’s alright if you miss this part; the RAID configuration can be imported after the OS boots.
  9. The OS should now fully boot, since you commented out all the hardware RAID file systems in step 1.
  10. Run nwraidutil.pl on the appliance.  Check for the presence of all enclosures and DACs.  If all RAID disks are reported as being online, skip ahead to step 13.  If the RAID disks are in an Unconfigured(Good) state, skip ahead to step 12.  If you don't see all of your expected enclosures, check your cabling and ensure all cables are connected to the same ports as before the swap (see step 6).
  11. If the disks are in an Unconfigured(Bad) state, run this command:
           Note:  For newer version of NetWitness 11.x, you may need to substitute /opt/MegaRAID/MegaCli/MegaCli64  with /opt/MegaRAID/perccli/perccli64 if the commands return an error.

    /opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[ENCLOSURE:DISK,ENCLOSURE:SLOT] -a0 (substitute enclosure and slot number for each drive in an Unconfigured(Bad) state, and substitute correct adapter number (i.e, -a1), where appropriate.  If successful, proceed to step 12.

    E.g.:
    /opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[6:0,6:2,6:3,6:4,6:5,6:6,6:7,6:8,6:9,6:10,6:11,6:11] -a0

    /opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[25:1,25:2,25:3,25:4,25:5,25:6,25:7,25:8,25:9,25:10,25:11,25:12] -a1

    If successful, proceed to step 12.  If you cannot get past this step please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d to the case.
     
  12. If the disks in step are in a foreign state (run nwraidutil.pl again to verify).  Run these commands:
    /opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Import -aall

    If successful, proceed to step 13.  If you cannot get past this step please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d  to the case.
     
  13. Run the following commands:
    pvscan

    vgscan

    lvscan

    lvdisplay -C

    vgmknodes


     
  14. If the above commands indicate that physical volumes (PV) and volume groups (VG) are detected but 'lvdisplay -C' indicates the logical volumes are not online (the attributes will be '-wi---' if offline, '-wi-ao' if online), then please run these commands:
    lvchange -ay /dev/<VG>/<LV>

    vgmknodes


    E.g.
    lvchange -ay /dev/decodersmall/decoroot


    lvchange -ay /dev/decodersmall/index
    lvchange -ay /dev/decodersmall/metadb
    lvchange -ay /dev/decodersmall/sessiondb


    lvchange -ay /dev/decoder/packetdb

    lvchange -ay /dev/decoder0/packetdb

    lvchange -ay /dev/decoder1/packetdb
    vgmknodes

     
  15. If the output of steps 13/14 shows that all LVM RAID volumes are detected and online, you can remove the comments we added in step 1 to the /var/netwitness filesystems in /etc/fstab.
  16. Run the following command:
    mount -a

    If your filesystems do not mount, please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d to the case.
     
  17. If all went well and all of the RAID filesystems mounted, you may start the Decoder process back up with start nwdecoder (SA10.x/CentOS6) or systemctl start nwdecoder(NW11.x/CentOS7) and watch /var/log/messages for errors.  If the Decoder fails to load, please open a case with RSA NetWitness Support for assistance and attach the *full* output of nwtech.sh to the case.
Legacy Article IDa58908

Attachments

    Outcomes