Packet Concentrator not working

[SeanKoniarz]

Jan 24 18:08:03 avsesa-rsasa-pkt-p01 nw[3057]: [Index] [failure] Unexpected Query Exception of type N5boost12interprocess22interprocess_exceptionE: Too many open files

Any ideas?  This is what happens when I try to use investigator.

[huanzhou1]

what's the SA version? restart service same? reset index?

[RSAAdmin · Accepted solution]

This problem is a sympton of the refactored 10.3 index on a Concentrator/Decoder service using a lot more open file handles for performance and hitting the default Linux file process limit.  Fortunately, the fix is easy:

On CentOS 6:

1) Stop the decoder process (stop nwdecoder)

2) Edit /etc/init/nwdecoder.conf

3) Add "limit nofile 65536 65536" right before "exec /usr/sbin/NwDecoder --stopwhenready"

4) start nwdecoder

When you install 10.3.2, it will perform this process automatically, if it hasn't been done already.  Replace decoder with concentrator for a Concentrator service.

The full file should look something like this:

start on runlevel [35] and stopped rc

stop on runlevel [!35]

respawn

respawn limit 10 300

console none

kill timeout 60

chdir /var/netwitness/decoder/packetdb

limit core unlimited unlimited

limit nofile 65536 65536

exec /usr/sbin/NwDecoder --stopwhenready

expect stop

[huanzhou1]

thats good. Thanks for sharing

[SeanKoniarz]

That worked

Thanks!

[BrianHoward]

What would solution be for the same error on a CentOS 5 device?

[RSAAdmin]

Add the line:

ulimit -n 65536

above or below the other ulimit line in the monit config file(s) found in /etc/rc.d/init.d

[BrianHoward]

Here is my monit.conf file contents for one of my decoders.

# NetWitness configuration, please do not edit

#

set daemon 30   # check services at 30 second intervals

#

set logfile /var/log/monit.log

#

set idfile /var/lib/monit/monit.id

#

set statefile /var/lib/monit/monit.state

#

set httpd port 2812 and use the address 127.0.0.1

    allow 127.0.0.1 #Restrict Daemon to LocalNet

#

include /etc/monit.d/*

[RSAAdmin]

Hmm, I just checked and ulimit -n 100000 should already be part of the monit file for Decoder/Concentrator.  I wouldn't change it.  Are you specifically having issues on CentOS 5?

If you don't see that line in the monit file, then add it as the first line after the comments at the top of the file.

[BrianHoward]

Ok, below is the content from my nwdecoder file in /etc/rc.d/init.d/

#!/bin/bash

#

#       /etc/rc.d/init.d/nwdecoder

#

#       NetWitness Decoder Service

#

# chkconfig: 35 60 40

# description: NetWitness Decoder Service

# processname: NwDecoder

# config: /etc/netwitness/9.0/NwDecoder.cfg

#

# Source function library.

. /etc/init.d/functions

start() {

        echo -n "Starting nwdecoder: "

  cd /var/netwitness/decoder/packetdb

        DAEMON_COREFILE_LIMIT=unlimited daemon "/usr/sbin/NwDecoder >/dev/null " &

  local result=$?

  if ! (( $result ))

  then

  success

  else

  failure

  fi

  echo ""

        touch /var/lock/subsys/nwdecoder

        return $result

}

stop() {

        echo -n "Shutting down nwdecoder: "

  killproc "/usr/sbin/NwDecoder"

  echo ""

  local result=$?

        rm -f /var/lock/subsys/nwdecoder

        return $result

}

restart() {

  stop

  start

}

case "$1" in

    start)

        start

       ;;

    stop)

        stop

        ;;

    status)

        status /usr/sbin/NwDecoder

        ;;

    restart|reload)

        restart

        ;;

    condrestart)

        [ -f /var/lock/subsys/nwdecoder ] && restart || :

  ;;

    *)

        echo "Usage: nwdecoder {start|stop|status|reload|restart}"

        exit 1

        ;;

esac

exit $?