000026237 - How to fix RSA NetWitness Platform DAC drives that are in an Unconfigured(Bad) state

                        Physical Drive State Legend
-------------------------------------------------------------------------------------
B  Unconfigured(Bad)                                            O  Online
D  Dedicated Hotspare and associated virtual drive number       R  Rebuild
E  Hotspare prefers same enclosure                              S  Solid-State Drive
F  Foreign                                                      U  Unconfigured(Good)
G  Global hotspare                                              X  Offline
I  Hotspare is revertible                                       !  Failed
M  Missing                                                      ?  Unknown state
-------------------------------------------------------------------------------------
NOTE: 'E' does not prohibit a hotspare from being used in another enclosure, it is merely a preference

Adapter 1 (PERC H830 Adapter) enclosure 29 slots found: 15
Encl  Slot  State     P.Fail.Count  Raw Size       Inquiry Data
29     0    (O)       0             2.728 TB       YS18      SEAGATE ST330006CLAR3000YS18Z298AB01
29     1    (B)       0             0 KB           YS18      SEAGATE ST330006CLAR3000YS18Z298AB03
29     2    (O)       0             2.728 TB       YS18      SEAGATE ST330006CLAR3000YS18Z298AC04

1. Set the Unconfigured(Bad) disks back to Online.
2. Ensure the DAC disks are running with the latest disk firmware version.
3. If any disk remains in the Unconfigured(Bad) state, then RMA replace it.

It has been found that in some cases when there is a power outage or some other situation that causes the DAC or JBOD to reset or lose communication with the RSA NetWitness server, it will mark all drives as Unconfigured(Bad).

When in this state, the system will not boot properly due to the mismatch of what is in the /etc/fstab file, and what disks are available.

A. If disks are Unconfigured(Bad) for this reason, then the fix is to set all the drives back to good and import any foreign configuration.

1. To set the drives to a good state, first determine the list of drives and their enclosure ID. Get this from running the following command on an SSH session in the server in question:
   
   

   
   # /usr/sbin/nwraidutil.pl             # For RSA NetWitness 10.x
   # /opt/rsa/saTools/nwraidutil.pl      # For RSA NetWitness 11.x

   
   
   

   This will dump a verbose list of all the drives and their information.
   Extract the enclosure ID and a list of the drives which need to be configured as good.
    

   
   
2. Once the information has been obtained, enter the following command to configure them as well: 
   

   
   # /opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood PhysDrv[E#:S#,E#:S#,...] Force aN#

   
   
   Note: The list of drives is a combination of EnclosureID(E#):SlotID(S#) pairs separated by commas, for an Adapter(N#). For example:
   
   

   
   # /opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood PhysDrv[29:1] Force a1

   
    
3. After all the drives have been configured as good, you will likely need to import the foreign configuration. Enter the following command to import foreign configuration:

# /opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Import -aAll

At this point, a rerun of the nwraidutil.pl should show all drives in the "Online" state. If so, you should be able to reboot the appliance and the file system should be back to normal.
 

B. Ensure the DAC disks are running with the latest disk firmware version. This is useful as older disk firmware versions have been found to prematurely mark a good disk as faulty.

Refer to the RSA Security Analytics Availability of DAC Disk Drive Firmware Updates RSA Link document with links to the instructions and the disk firmware software for updating the DAC disk firmware.

Updating the DAC disk firmware of a disk which was prematurely marked as Unconfigured(Bad) will bring it back to an Online state.
 

C. If a disk remains in the Unconfigured(Bad) state, then it may truly have failed, and it should be RMA replaced. Contact RSA Support for further assistance.

In order for a RMA request to be created, please confirm/provide all the delivery information where the RMA should be sent:

• RSA Case Number:
• Company Name:
• Customer Contact (First and Last Name):
• Customer Contact Telephone Number:
• Customer Email Address:
• Customer Shipping Address:
• RSA NetWitness Version:
• Appliance Serial Number:
• DAC Serial Number (if the faulty HDD is in a DAC):
• For Failed Hard Drives (Product ID, Part Number, Drive Serial Number, Revision):
• Problem Description (and any action taken to try remedy):

Notes:

1. The appliance serial number and RSA NetWitness version can be shown by running the following commands on the server in question: 
   

   
   # dmidecode -t 1 | grep Serial
   
   # rpm -qa | grep nw

   
   
2. For any disk RMA, please provide the full nwraidutil.pl output after running the following command on an SSH session in the server in question: 
   

   
   # /usr/sbin/nwraidutil.pl             # For RSA NetWitness 10.x
   # /opt/rsa/saTools/nwraidutil.pl      # For RSA NetWitness 11.x

   
   

# /opt/MegaRAID/CmdTool2/CmdTool2 -PDMakeGood -PhysDrv[E#:S#,E#:S#,...] -Force -aN#

# /opt/MegaRAID/CmdTool2/CmdTool2 -CfgForeign -Import -aAll