In support we unfortunately only see the bad side of things. People don't call us when everything is working. However I'm surprised that some customers don't seem to have good contingency plans in place. Sorry if I am preaching to the converted or teaching people how to "suck eggs" but make sure that the following is configured before the worst happens.
What I would recommend:
- Make sure IDrac is configured and accessible in case you lose SSH access. Test console idrac access frequently
- Have a build stick ISO available in case a reimage is necessary. Reimaging over IDrac is possible.(https://rsaportal.force.com/customer/articles/How_To/How-to-re-image-an-RSA-Security-Analytics-appliance-via-IDRAC)
- Make sure that you have a backup of your system
- Consider what would happen in the case of a total hardware failure - is your deployment load balanced?
Don't wait for the worst to happen before
testing your DR plan!!!