000026684 - RAID controller swap procedure for RSA Security Analytics and RSA NetWitness appliances

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000026684
Applies ToRSA NetWitness NextGen
RSA NetWitness Decoder
RSA NetWitness Log Decoder
RSA NetWitness Concentrator
RSA NetWitness Hybrid
RSA NetWitness Broker
RSA Security Analytics
IssueRAID controller swap procedure for RSA Security Analytics and RSA NetWitness appliances.
Due to an RMA, I need to swap the RAID controller in my appliance for a new one.  What steps do I take to accomplish this?
ResolutionPlease follow these steps:
1.  In /etc/fstab, comment out (by adding a # to the beginning of the line) all filesystems that mount under /var/netwitness.
2.  (optional but recommended) Edit /boot/grub/grub.conf and remove the following part of the active boot section?s ?kernel? line: ?console=ttyS0,115200n8r?.  This will disable serial console redirection at boot should the appliance go into maintenance mode should it not be able to mount a RAID filesystem.  If one does not follow this step and the appliance goes into maintenance mode at boot, you will only be able to see the prompts and interact with the OS by attaching a console device to the serial port rather than by using VGA/Keyboard/Mouse.
3.  Using NetWitness Administrator, stop capture on the Decoder.  One should do this before shutting the service down to ensure all indexes are saved properly so as to prevent a need to reindex any sessions upon the next start of the service.
4.  Stop the service with the monit stop nwdecoder or stop nwdecoder command, based on the CentOS version.
5.  Run this command to shut down the appliance:
shutdown ?h now
6.  Take note of which port the JBOD?s are plugged into, and which internal port the internal RAID enclosure is connected to.
7.  Open the case and perform the swap of the RAID controller, plugging the cables back into their original slots.  Move the cable for the old battery backup unit (BBU) from the old to the new controller.  Power the appliance back on.
8.    During boot, you should see a message that says ?foreign configuration detected? on the RAID controller, or words to that effect.  Hit ?F? to try to import.  It?s alright if you miss this part; the RAID configuration can be imported after the OS boots.
9.  The OS should now fully boot, since you commented out all the hardware RAID filesystems in step 1.
10.  Run nwraidutil.pl on the appliance.  Check for the presence of all enclosures and JBODs.  If all RAID disks are reported as being online, skip ahead to step 13.  If the RAID disks are in an Unconfigured(Good) state, skip ahead to step 12.  If you don't see all of your expected enclosures, check your cabling and ensure all cables are connected to the same ports as before the swap (see step 6).
11.  If the disks are in an Unconfigured(Bad) state, run this command:
       Note:  For this and future invocations of MegaCli64, substitute /opt/MegaRAID/CmdTool2/CmdTool2 for the MegaCli64 command if your system has that instead; the rest of the command is the same.
/opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[ENCLOSURE:DISK,ENCLOSURE:SLOT] -a0 (substitute enclosure and slot number for each drive in an Unconfigured(Bad) state, and substitute correct adapter number (i.e, -a1), where appropriate.  If successful, proceed to step 12.
E.g.:
/opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[6:0,6:2,6:3,6:4,6:5,6:6,6:7,6:8,6:9,6:10,6:11,6:11] -a0

/opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[25:1,25:2,25:3,25:4,25:5,25:6,25:7,25:8,25:9,25:10,25:11,25:12] -a1
If successful, proceed to step 12.  If you cannot get past this step please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d to the case.
12.  If the disks in step are in a foreign state (run nwraidutil.pl again to verify).  Run these commands:
/opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Import -aall
If successful, proceed to step 13.  If you cannot get past this step please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d  to the case.
13.  Run the following commands:


pvscan

vgscan

lvscan

lvdisplay -C

vgmknodes


14.  If the above commands indicate that physical volumes (PV) and volume groups (VG) are detected but 'lvdisplay -C' indicates the logical volumes are not online (the attributes will be '-wi---' if offline, '-wi-ao' if online), then please run these commands:
lvchange -ay /dev/<VG>/<LV>

vgmknodes

E.g.
lvchange -ay /dev/decodersmall/decoroot


lvchange -ay /dev/decodersmall/index
lvchange -ay /dev/decodersmall/metadb
lvchange -ay /dev/decodersmall/sessiondb


lvchange -ay /dev/decoder/packetdb

lvchange -ay /dev/decoder0/packetdb

lvchange -ay /dev/decoder1/packetdb
vgmknodes

15.    If the output of steps 13/14 shows that all LVM RAID volumes are detected and online, you can remove the comments we added in step 1 to the /var/netwitness filesystems in /etc/fstab.
16.  run the following command:
mount ?a
If your filesystems do not mount, please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d fromto the case.
17.    If all went well and all of the RAID filesystems mounted, you may start the Decoder process back up with ?monit start nwdecoder? and watch /var/log/messages for errors.  If the Decoder fails to load, please open a case with RSA NetWitness Support for assistance and attach the *full* output of nwtech.sh to the case.
Legacy Article IDa58908

Attachments

    Outcomes