000036778 - After upgrading to RSA  NetWitness 11.2.0.0, Log Decoder and Packet Decoder services continuously restarting

Document created by RSA Customer Support Employee on Sep 27, 2018
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000036778
Applies ToRSA Product Set: NetWitness Logs & Packets
RSA Product/Service Type: Packet Decoder,
Log Decoder
RSA Version/Condition: 11.2.0
Platform: CentOS
O/S Version: EL7
IssueAfter upgrading to 11.2.0.0, Log Decoder and Packet Decoder services are continuously restarting.
Due to the restarts, the service is always in a startup state and so unable to manage the services via the NetWitness v11.X Web UI.

An example of the service crash as seen in /var/log/messages:

Sep 21 04:01:59 LOGDECODER1 kernel: [ 2862.491161] traps: NwLogDecoder[9262] trap divide error ip:895907 sp:7ffd055741c0 error:0 in NwLogDecoder[400000+162b000]
Sep 21 04:02:04 LOGDECODER1 NwLogDecoder[9353]: [Engine] [warning] Warning, PID path /var/run/nwlogdecoder.pid already exists.  Is another instance already running?
Sep 21 04:02:04 LOGDECODER1 NwLogDecoder[9353]: [Engine] [info] Running logdecoder in console
Sep 21 04:02:04 LOGDECODER1 NwLogDecoder[9353]: [Engine] [info] RSA NetWitness Service, Log Decoder 11.2.0.0 (Aug  7 2018) 64 bit Starting


Within /var/log/messages can also see service Process ID (PID) continually changing.

# tail -n10000 /var/log/messages | grep NwLogDecoder | grep Copyright
Sep 21 04:07:06 LOGDECODER1 NwLogDecoder[10124]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:14:53 LOGDECODER1 NwLogDecoder[10840]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:15:26 LOGDECODER1 NwLogDecoder[10927]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:15:58 LOGDECODER1 NwLogDecoder[11023]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:16:29 LOGDECODER1 NwLogDecoder[11098]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:17:01 LOGDECODER1 NwLogDecoder[11191]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:17:32 LOGDECODER1 NwLogDecoder[11251]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:18:03 LOGDECODER1 NwLogDecoder[11340]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:18:34 LOGDECODER1 NwLogDecoder[11403]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.
Sep 21 04:19:05 LOGDECODER1 NwLogDecoder[11572]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc.  All Rights Reserved.


 
CauseA software defect introduced in 11.2.0 which incorrectly attempts to use the value of auto for save.session.count as an integer.
ResolutionAs per JIRA SACE-10191, this issue will be resolved in the next release of 11.2.X
 
WorkaroundThe workaround is to change the value of /index/config/save.session.count to the equivalent of auto which is 600M in service config file.

On Log Decoders:


systemctl stop nwlogdecoder.service
cp /etc/netwitness/ng/NwLogdecoder.cfg /etc/netwitness/ng/NwLogdecoder.cfg.backup.$(date +"%Y%m%d_%H%M")
sed -ri 's/(<config name="save.session.count" prettyName="Save Session Count" value=")auto/\1600000000/g' /etc/netwitness/ng/NwLogdecoder.cfg
rm -f /var/netwitness/logdecoder/metadb/core.*
systemctl start nwlogdecoder.service



On Packet Decoders:


systemctl stop nwdecoder.service
cp /etc/netwitness/ng/NwDecoder.cfg /etc/netwitness/ng/NwDecoder.cfg.backup.$(date +"%Y%m%d_%H%M")
sed -ri 's/(<config name="save.session.count" prettyName="Save Session Count" value=")auto/\1600000000/g' /etc/netwitness/ng/NwDecoder.cfg
rm -f /var/netwitness/decoder/packetdb/core.*
systemctl start nwdecoder.service

Attachments

    Outcomes