000017310 - Kernel Panic while upgrading kernel to 2.6.32-358.18.1.el6 on an RSA Security Analytics appliance with an attached VNX

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000017310
Applies ToRSA Product Set: RSA Security Analytics
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 10.3
Platform (Other): EMC VNX
IssueKernel Panic while upgrading kernel to 2.6.32-358.18.1.el6 on an RSA Security Analytics appliance with an attached VNX.

Upgrading kernel from 2.6.32-358.11.1.el6.x64 to 2.6.32-358.18.1.el6.x86_64 causes OS to go into a kernel panic.


Error Message from console:



Error message: Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32-358.18.1.el6.x96_64 #1
Call Trace:
[<ffffffff8150d4f8>] ? panic+0xa7/0x16f
[<ffffffff81073ae2>] ? do_exit+0x862/0x870
[<ffffffff811829c5>] ? fput+0x25/0x30
[<ffffffff81073b48>] ? do_group_exit+0x58/0xd0
[<ffffffff81073bd7>] ? sys_exit_group+0x17/0x20
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b


Resolution

In order to resolve the issue, follow the steps below.


  1. After panic, do cold reboot from iDrac
  2. On grub, boot with the previous kernel.
  3. If the appliance is a decoder, stop capture on the device using the Security Analytics UI.  If the appliance is a concentrator, stop aggregation on the device.
  4. Stop the nwdecoder and/or nwconcentrator service (i.e. stop nwdecoder) depending on the appliance type.
  5. Remove the 61-persistent-storage-nw.rules file:  rm /etc/udev/rules.d/61-persistent-storage-nw.rules
  6. Move the lvm.conf file to the /root directory with the following command:  mv /etc/lvm/lvm.conf /root
  7. Move the lvm.conf.bck1 to /etc/lvm with the following command: mv /etc/lvm/lvm.conf.bck1 /etc/lvm/lvm.conf 
  8. Set the target kernel version variable with the following command:  targetKern=`rpm -q kernel | awk '{match($0, "kernel-([^$]+)$",n);print n[1]}' | sort | tail -n1`
    e.g. targetKern=2.6.32-358.18.1.el6.x86_64
  9. Make a backup of the target kernel image with the following command:  cp /boot/initramfs-$targetKern.img /boot/initramfs-$targetKern.img.orig
  10. Issue the following command(takes several minutes):  dracut --force /boot/initramfs-$targetKern.img $targetKern
  11. Reboot and on the grub boot with the target kernel.

 


If you are unsure of any of the steps above or experience any issues, contact RSA Support for further assistance.

Notes1.Issue conditions :
VNX attached SA configured with old ArrayCfg script (prior to 2.1) The issue was fixed in ArrayCfg 2.1
When upgrading kernel regardless of target version not only 2.6.32-358.18.1.
2. Note for those who upgrade kernel on SA with VNX
When upgrading kernel on every SA appliances with VNX configured with old ArrayCfg script, the kernel panic will occur.
In case of upgrading kernel under this circumstance, this solution must be added between yum upgrade and reboot. 
3. If lvm.conf.bck1 doesn't exist
Please search and replace the following

filter = [ "a/dellsd[a-z]+[0-9]*$/", "a/dev/mapper/mpath[a-z]*/", "r/sd.*/", "r/disk.*/" ]
types = [ "device-mapper", 1 ]
->
filter = [ "a/.*/" ]

 
Legacy Article IDa65989

Attachments

    Outcomes