Resolution | It has been found that in some cases when there is a power outage or some other situation that causes the DAC or JBOD to reset or lose communication with the RSA NetWitness server, it will mark all drives as Unconfigured(Bad).
When in this state, the system will not boot properly due to the mismatch of what is in the /etc/fstab file, and what disks are available.
- If disks are Unconfigured(Bad) for this reason, then the fix is to set all the drives back to good and import any foreign configuration.
- To set the drives to a good state, first determine the list of drives and their enclosure ID. Get this from running the following command on an SSH session in the server in question:
# /usr/sbin/nwraidutil.pl # For RSA NetWitness 10.x # /opt/rsa/saTools/nwraidtool.py # For RSA NetWitness 11.3 and higher
This will dump a verbose list of all the drives and their information. Extract the enclosure ID and a list of the drives which need to be configured as good. - Once the information has been obtained, enter the following command to configure them as well:
# /opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood PhysDrv[E#:S#,E#:S#,...] Force aN#
Note: The list of drives is a combination of EnclosureID(E#):SlotID(S#) pairs separated by commas, for an Adapter(N#). For example:
# /opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood PhysDrv[29:1] Force a1
- After all the drives have been configured as good, you will likely need to import the foreign configuration. Enter the following command to import foreign configuration:
# /opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Import -aAll
At this point, a rerun of the nwraidutil.pl should show all drives in the "Online" state. If so, you should be able to reboot the appliance and the file system should be back to normal.
- Ensure the DAC disks are running with the latest disk firmware version. This is useful as older disk firmware versions have been found to prematurely mark a good disk as faulty.
Refer to the RSA Security Analytics Availability of DAC Disk Drive Firmware Updates RSA Link document with links to the instructions and the disk firmware software for updating the DAC disk firmware. Updating the DAC disk firmware of a disk that was prematurely marked as Unconfigured(Bad) will bring it back to an Online state.
- If a disk remains in the Unconfigured(Bad) state, then it may truly have failed, and it should be RMA replaced. Contact RSA Support for further assistance.
In order for an RMA request to be created, please confirm/provide all the delivery information where the RMA should be sent:
- RSA Case Number:
- Company Name:
- Customer Contact (First and Last Name):
- Customer Contact Telephone Number:
- Customer Email Address:
- Customer Shipping Address:
- RSA NetWitness Version:
- Appliance Serial Number:
- DAC Serial Number (if the faulty HDD is in a DAC):
- For Failed Hard Drives (Product ID, Part Number, Drive Serial Number, Revision):
- Problem Description (and any action taken to try remedy):
Notes:
- The appliance serial number and RSA NetWitness version can be shown by running the following commands on the server in question:
# dmidecode -t 1 | grep Serial
# rpm -qa | grep nw
- For any disk RMA, please provide the full nwraidutil.pl output after running the following command on an SSH session in the server in question:
# /usr/sbin/nwraidutil.pl # For RSA NetWitness 10.x # /opt/rsa/saTools/nwraidutil.pl # For RSA NetWitness 11.x
|
Notes | For Legacy RSA NetWitness 9.x, instead of the above MegaCli64 commands use the below commands:
2. PDMakeGood the disk:
# /opt/MegaRAID/CmdTool2/CmdTool2 -PDMakeGood -PhysDrv[E#:S#,E#:S#,...] -Force -aN#
3. Import Foreign configuration:
# /opt/MegaRAID/CmdTool2/CmdTool2 -CfgForeign -Import -aAll
|