000038852 - How to fix all disks of DAC that are UGOOD(unconfigured Good) and DGs are frgn(foreign)

Document created by RSA Customer Support Employee on May 20, 2020
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000038852
Applies ToRSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: Decoder, Concentrator, Logdecoder
RSA Version/Condition: 11.3.1.0
 
Issue

The status of all disks of DAC is UGOOD(Unconfigured good) and DGs are frgn(foreign) state.
User-added image

In the MegaRAID firmware log, All disks in the DAC were reseted/removed/added. Because all the disks were removed, the VD was also degraded and removed. After that the disks were added, but their state were unconfigured good because the VD was removed.




05/01/20 20:03:19: EVT#17668-05/01/20 20:03:19: 114=State change on PD 0e(e0x0f/s2) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17672-05/01/20 20:03:19: 114=State change on PD 0b(e0x0f/s0) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17677-05/01/20 20:03:19: 114=State change on PD 10(e0x0f/s1) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17680-05/01/20 20:03:19: 114=State change on PD 01(e0x0f/s13) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17683-05/01/20 20:03:19: 114=State change on PD 02(e0x0f/s12) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17686-05/01/20 20:03:19: 114=State change on PD 03(e0x0f/s11) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17687-05/01/20 20:03:19: 114=State change on PD 04(e0x0f/s10) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17688-05/01/20 20:03:19: 114=State change on PD 05(e0x0f/s9) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17689-05/01/20 20:03:19: 114=State change on PD 06(e0x0f/s8) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17690-05/01/20 20:03:19: 114=State change on PD 07(e0x0f/s7) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17691-05/01/20 20:03:19: 114=State change on PD 08(e0x0f/s6) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17692-05/01/20 20:03:19: 114=State change on PD 09(e0x0f/s5) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17693-05/01/20 20:03:19: 114=State change on PD 0a(e0x0f/s4) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17694-05/01/20 20:03:19: 114=State change on PD 0c(e0x0f/s3) from ONLINE(18) to FAILED(11)

05/01/20 20:03:19: EVT#17669-05/01/20 20:03:19:  81=State change on VD 00/0 from OPTIMAL(3) to DEGRADED(2)
05/01/20 20:03:19: EVT#17673-05/01/20 20:03:19:  81=State change on VD 00/0 from DEGRADED(2) to OFFLINE(0)
05/01/20 20:03:20: EVT#17741-05/01/20 20:03:20: 139=Deleted VD 00/0



05/01/20 20:03:20: EVT#17726-05/01/20 20:03:20: 114=State change on PD 01(e0x0f/s13) from FAILED(11) to UNCONFIGURED_BAD(1)
05/01/20 20:04:29: EVT#17762-05/01/20 20:04:29: 114=State change on PD 01(e0x0f/s13) from UNCONFIGURED_BAD(1) to UNCONFIGURED_GOOD(0)

 


 

Resolution

Cause


Communication failure between Appliance and DAC.

Possible root cause


1. Bad cable connection between the RSA NetWitness appliance(Decoder or Concentrator) and DAC.
2. Bad RAID card on RSA NetWitness Appliance.
3. Bad Controller on DAC.
 

Resolution to recover the filesystems



  1. Stop capture(Decoder) or aggregation(Concentrator)
  2. Disable Capture autostart(Decoder) or aggregate autostart(Concentrator)
  3. Connect to the appliance via putty
  4. Edit /etc/fstab file to comment out /var/netwitness/<service> filesystems by placing # before each line. 

    Note: <service> will be decoder/logdecoder/concentrator

  5. Reboot the appliance in putty
  6. Import the foreign disks
    # /opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Import -a1 
     
  7. Run nwraidtool.py and check that all disks are online and DG is not a foreign state.
  8. Remove comments in /etc/fstab added in step 4.
  9. Mount all file systems.
    # mount -a
  10. Restart services. 
    # systemctl restart <service>

    Note: <service> will be nwdecoder.service/nwlogdecoder.service/nwconcentrator.service

  11. Start capture/aggregation and enable autostart which disabled in step 2.

Attachments

    Outcomes