Issue | After upgrading to 11.2.0.0, Log Decoder and Packet Decoder services are continuously restarting. Due to the restarts, the service is always in a startup state and so unable to manage the services via the NetWitness v11.X Web UI.
An example of the service crash as seen in /var/log/messages:
Sep 21 04:01:59 LOGDECODER1 kernel: [ 2862.491161] traps: NwLogDecoder[9262] trap divide error ip:895907 sp:7ffd055741c0 error:0 in NwLogDecoder[400000+162b000] Sep 21 04:02:04 LOGDECODER1 NwLogDecoder[9353]: [Engine] [warning] Warning, PID path /var/run/nwlogdecoder.pid already exists. Is another instance already running? Sep 21 04:02:04 LOGDECODER1 NwLogDecoder[9353]: [Engine] [info] Running logdecoder in console Sep 21 04:02:04 LOGDECODER1 NwLogDecoder[9353]: [Engine] [info] RSA NetWitness Service, Log Decoder 11.2.0.0 (Aug 7 2018) 64 bit Starting
Within /var/log/messages can also see service Process ID (PID) continually changing.
# tail -n10000 /var/log/messages | grep NwLogDecoder | grep Copyright Sep 21 04:07:06 LOGDECODER1 NwLogDecoder[10124]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:14:53 LOGDECODER1 NwLogDecoder[10840]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:15:26 LOGDECODER1 NwLogDecoder[10927]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:15:58 LOGDECODER1 NwLogDecoder[11023]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:16:29 LOGDECODER1 NwLogDecoder[11098]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:17:01 LOGDECODER1 NwLogDecoder[11191]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:17:32 LOGDECODER1 NwLogDecoder[11251]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:18:03 LOGDECODER1 NwLogDecoder[11340]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:18:34 LOGDECODER1 NwLogDecoder[11403]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved. Sep 21 04:19:05 LOGDECODER1 NwLogDecoder[11572]: [Engine] [info] RSA NetWitness Service Copyright 2001-2018, RSA Security Inc. All Rights Reserved.
|
Workaround | The workaround is to change the value of /index/config/save.session.count to the equivalent of auto which is 600M in service config file.
On Log Decoders:
systemctl stop nwlogdecoder.service cp /etc/netwitness/ng/NwLogdecoder.cfg /etc/netwitness/ng/NwLogdecoder.cfg.backup.$(date +"%Y%m%d_%H%M") sed -ri 's/(<config name="save.session.count" prettyName="Save Session Count" value=")auto/\1600000000/g' /etc/netwitness/ng/NwLogdecoder.cfg rm -f /var/netwitness/logdecoder/metadb/core.* systemctl start nwlogdecoder.service
On Packet Decoders:
systemctl stop nwdecoder.service cp /etc/netwitness/ng/NwDecoder.cfg /etc/netwitness/ng/NwDecoder.cfg.backup.$(date +"%Y%m%d_%H%M") sed -ri 's/(<config name="save.session.count" prettyName="Save Session Count" value=")auto/\1600000000/g' /etc/netwitness/ng/NwDecoder.cfg rm -f /var/netwitness/decoder/packetdb/core.* systemctl start nwdecoder.service
|