000013828 - RKM process/command spawned by tnsmanager.sh is taking up 100% and Appliance hangs on reboot

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000013828
Applies ToRSA Key Manager Appliance 2.7
RSA Key Manager Appliance 2.6
Dell PowerEdge R710
IssueRKM process/command spawned by tnsmanager.sh is taking up 100% and Appliance hangs on reboot
issue with tnsmanager.sh taking up 100% of CPU. A process/command spawned by tnsmanager.sh is taking up 100% of CPU on primary of one cluster and on secondary of the other cluster. Have two clusters grouped together, each cluster has primary and secondary
Can't kill the process and the only way out is reboot
A soft reboot of RKM appliance through either reboot command run on shell prompt or through RKM Operations Console /rkmawa => Operations => Reboot option, hangs.  A hard reboot (power cycle) is required to recover from the appliance hung state.
Log entries similar to the following are logged to the RKM Appliance system log file /var/logs/messages around the time it hangs to soft-reboot:
Sep 7 16:15:15 rkmapp kernel: BUG: soft lockup - CPU#3 stuck for 61s! [cut:14859]
Sep 7 16:15:15 rkmapp kernel: Modules linked in: ipv6 ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_t
cpudp iptable_filter ip_tables x_tables binfmt_misc dm_mod video output sbs sbshc button battery ac rtc_cmos rtc_core rtc_lib
ehci_hcd uhci_hcd iTCO_wdt iTCO_vendor_support bnx2 sg sr_mod scsi_wait_scan ext3 jbd ata_piix libata megaraid_sas sd_mod crc_
t10dif scsi_mod
Sep 7 16:15:15 rkmapp kernel:
Sep 7 16:15:15 rkmapp kernel: Pid: 14859, comm: cut Not tainted (2.6.29.6-2.smp.gcc3.4.x86.i686 #1) PowerEdge R710
Sep 7 16:15:15 rkmapp kernel: EIP: 0060:[<c0422bb8>] EFLAGS: 00000246 CPU: 3
Sep 7 16:15:15 rkmapp kernel: EIP is at get_user_pages_fast+0xc1/0x145
Sep 7 16:15:15 rkmapp kernel: EAX: 00000001 EBX: b7f00000 ECX: b7f00000 EDX: 00000001
Sep 7 16:15:15 rkmapp kernel: ESI: b7eff000 EDI: b7f00000 EBP: f0669e70 ESP: f0669e50
Sep 7 16:15:15 rkmapp kernel:  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Sep 7 16:15:15 rkmapp kernel: CR0: 8005003b CR2: b7f0d416 CR3: 0641b000 CR4: 000006d0
Sep 7 16:15:15 rkmapp kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Sep 7 16:15:15 rkmapp kernel: DR6: ffff0ff0 DR7: 00000400
Sep 7 16:15:15 rkmapp kernel: Call Trace:
Sep 7 16:15:15 rkmapp kernel:  [<c0451720>] get_futex_key+0x8e/0x126
Sep 7 16:15:15 rkmapp kernel:  [<c0451f9e>] futex_wake+0x5b/0xf0
Sep 7 16:15:15 rkmapp kernel:  [<c041ed7e>] ? arch_flush_lazy_mmu_mode+0xd/0x3f
Sep 7 16:15:15 rkmapp kernel:  [<c04923aa>] ? __do_fault+0x3a1/0x3be
Sep 7 16:15:15 rkmapp kernel:  [<c0453542>] do_futex+0x9b/0x131
Sep 7 16:15:15 rkmapp kernel:  [<c04536a0>] sys_futex+0xc8/0xd4
Sep 7 16:15:15 rkmapp kernel:  [<c0431cb8>] mm_release+0x8e/0x99
Sep 7 16:15:15 rkmapp kernel:  [<c0435ad1>] exit_mm+0x1d/0xe4
Sep 7 16:15:15 rkmapp kernel:  [<c04361f7>] do_exit+0x170/0x25e
Sep 7 16:15:15 rkmapp kernel:  [<c04363b6>] sys_exit_group+0x0/0x18
Sep 7 16:15:15 rkmapp kernel:  [<c04363ca>] sys_exit_group+0x14/0x18
Sep 7 16:15:15 rkmapp kernel:  [<c0403e5f>] sysenter_do_call+0x12/0x34
Sep 7 16:19:46 rkmapp kernel: BUG: soft lockup - CPU#2 stuck for 61s! [cut:14859]
Sep 7 16:19:46 rkmapp kernel: Modules linked in: ipv6 ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_t
cpudp iptable_filter ip_tables x_tables binfmt_misc dm_mod video output sbs sbshc button battery ac rtc_cmos rtc_core rtc_lib
ehci_hcd uhci_hcd iTCO_wdt iTCO_vendor_support bnx2 sg sr_mod scsi_wait_scan ext3 jbd ata_piix libata megaraid_sas sd_mod crc_
t10dif scsi_mod
Sep 7 16:19:46 rkmapp kernel:
Sep 7 16:19:46 rkmapp kernel: Pid: 14859, comm: cut Not tainted (2.6.29.6-2.smp.gcc3.4.x86.i686 #1) PowerEdge R710
Sep 7 16:19:46 rkmapp kernel: EIP: 0060:[<c0422bb8>] EFLAGS: 00000246 CPU: 2
Sep 7 16:19:46 rkmapp kernel: EIP is at get_user_pages_fast+0xc1/0x145
Sep 7 16:19:46 rkmapp kernel: EAX: 00000001 EBX: b7f00000 ECX: b7f00000 EDX: 00000001
Sep 7 16:19:46 rkmapp kernel: ESI: b7eff000 EDI: b7f00000 EBP: f0669e70 ESP: f0669e50
Sep 7 16:19:46 rkmapp kernel:  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Sep 7 16:19:46 rkmapp kernel: CR0: 8005003b CR2: 080db67c CR3: 0641b000 CR4: 000006d0
Sep 7 16:19:46 rkmapp kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Sep 7 16:19:46 rkmapp kernel: DR6: ffff0ff0 DR7: 00000400
Sep 7 16:19:46 rkmapp kernel: Call Trace:
Sep 7 16:19:46 rkmapp kernel:  [<c0451720>] get_futex_key+0x8e/0x126
Sep 7 16:19:46 rkmapp kernel:  [<c0451f9e>] futex_wake+0x5b/0xf0
Sep 7 16:19:46 rkmapp kernel:  [<c041ed7e>] ? arch_flush_lazy_mmu_mode+0xd/0x3f
Sep 7 16:19:46 rkmapp kernel:  [<c04923aa>] ? __do_fault+0x3a1/0x3be
Sep 7 16:19:46 rkmapp kernel:  [<c0453542>] do_futex+0x9b/0x131
Sep 7 16:19:46 rkmapp kernel:  [<c04536a0>] sys_futex+0xc8/0xd4
Sep 7 16:19:46 rkmapp kernel:  [<c0431cb8>] mm_release+0x8e/0x99
Sep 7 16:19:46 rkmapp kernel:  [<c0435ad1>] exit_mm+0x1d/0xe4
Sep 7 16:19:46 rkmapp kernel:  [<c04361f7>] do_exit+0x170/0x25e
Sep 7 16:19:46 rkmapp kernel:  [<c04363b6>] sys_exit_group+0x0/0x18
Sep 7 16:19:46 rkmapp kernel:  [<c04363ca>] sys_exit_group+0x14/0x18
Sep 7 16:19:46 rkmapp kernel:  [<c0403e5f>] sysenter_do_call+0x12/0x34
CauseIssue with O/S kernel; applies to only RSA Key Manager Appliance versions 2.6 and 2.7 on Dell PowerEdge R710 hardware
ResolutionRKM 2.7 SP1 fixes the issue for 2.7 appliances.
For 2.6 appliance, contact RSA Customer Support for fix.
NotesKMA-618
KMA-845
Legacy Article IDa51619

Attachments

    Outcomes