000029407 - Concentrator aggregation immediately stops after starting aggregation in RSA Security Analytics 10.4.0.2

Document created by RSA Customer Support Employee on Mar 14, 2017Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000029407
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Concentrator
RSA Version/Condition: 10.4.0.2, 10.X
Platform: CentOS
O/S Version: EL6
IssueConcentrator aggregation stops immediately after start aggregation. 
The following error message will be observed in /var/log/messages:


Dec 16 22:54:28 NWAPPLIANCE nw[39892]: [Aggregation] [failure] There was a problem at initialization for device
'127.0.0.1:56002'. Newest remote session was 524,505 but last local session was 6,845,194,697. Consumption has been stopped.

 
CausePossible cause may include but not limited to, the appliance is RMAed and old data (metadb,sessiondb,index,etc) are restored from backup.
The problem is the Concentrator has very large session numbers in its old database and the newly consumed sessions are much lower and it seems to be confusing the issue.
ResolutionOne option is to change hostname and reboot the Packet/Log Decoder, at which point the Concentrator will recognize the Decoder as a new one and accept the smaller session.
Note: It is first worth entering the existing hostname of the Packet or Log Decoder in the service.name.override and following the steps below as this will avoid the need to actually change the hostname of the device.
WorkaroundThe easiest way to workaround this is to change the Log Decoder or Decoder service name.
  1. In the Packet/Log Decoder Explore view, navigate to sys -> config -> service.name.override and add the new hostname to its value.
  2. Restart the nwlogdecoder /nwdecoder service.
  3. Restart Concentrator aggregation.
  4. Verify that the errors in /var/log/messages are no longer occurring.
  5. The old data should be populated under the old Packet/Log Decoder name in investigations.
  6. The new data will be populated under new Packet/Log Decoder name in investigations.
Another workaround is to delete the entries specific to the Packet or Log Decoder from the NwConcentrator.cfg file
  1. Stop the nwconcentrator service by SSHing to the stop nwconcentrator
  2. Make a backup of the /etc/netwitness/ng/NwConcentrator.cfg file
  3. Edit the /etc/netwitness/ng/NwConcentrator.cfg to remove the two entries where the Decoder is mentioned. This is highlighted below.
  4. Start the nwconcentrator service with start nwconcentrator
  5. Readd the Log decoder or decoder via the GUI
Example of changes made in NwConcentrator.cfg file.
The decoder entry needs to be deleted from

1) Under: 


<folder instance="folder" name="recovery" prettyName="recovery">

2) Under 


<folder instance="folder" name="devices" prettyName="devices">

Here it the file with the lines to be deleted in RED. In this case we are deleting a packet decoder 192.168.123.2 on port 56004
 
<?xml version="1.0" encoding="UTF-8"?>
<root date="2017-Mar-06 10:43:54" doc-version="1" nw-version="10.6.3.0">
    <folder instance="folder" name="concentrator" prettyName="concentrator">
       
... snip ...
            <folder instance="folder" name="recovery" prettyName="recovery">
                <folder instance="folder" name="192.168.123.242:50002" prettyName="192.168.123.242:50002">
                    <config getRoles="" instance="config" maxLength="2048" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="192.168.123.249:50003" prettyName="192.168.123.249:50003">
                    <config getRoles="" instance="config" maxLength="2048" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="192.168.123.2:50004" prettyName="192.168.123.2:50004">
                    <config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value="7463934:507114659-507139537"/>
                </folder>
                <folder instance="folder" name="192.168.123.2:56004" prettyName="192.168.123.2:56004">
                    <config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>

                <folder instance="folder" name="192.168.123.3:50002" prettyName="192.168.123.3:50002">
                    <config getRoles="" instance="config" maxLength="2048" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="192.168.123.3:56002" prettyName="192.168.123.3:56002">
                    <config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="192.168.123.44:50002" prettyName="192.168.123.44:50002">
                    <config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="BROKER:50003" prettyName="BROKER:50003">
                    <config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="NWAPPLIANCE9201:50003" prettyName="NWAPPLIANCE9201:50003">
                    <config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
                </folder>
                <folder instance="folder" name="broker:50003" prettyName="broker:50003">
                    <config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="malware:50003" prettyName="malware:50003">
                    <config getRoles="" instance="config" maxLength="4096" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="nwappliance16112" prettyName="nwappliance16112">
                    <config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
                </folder>
                <folder instance="folder" name="nwappliance20886" prettyName="nwappliance20886">
                    <config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
                </folder>
                <folder instance="folder" name="packetconc:50005" prettyName="packetconc:50005">
                    <config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
                </folder>
                <folder instance="folder" name="sa" prettyName="sa">
                    <config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
                </folder>
            </folder>
            <folder instance="folder" name="rules" prettyName="rules">
... snip ...
                <folder instance="folder" name="correlation" prettyName="correlation">
                    <config getRoles="rules.manage" instance="config" maxLength="8192" name="0001" prettyName="0001" setRoles="rules.manage"                 </folder>
            </folder>
        </folder>
        <folder instance="folder" name="devices" prettyName="devices">
            <folder instance="folder" name="192.168.123.2:56004" prettyName="192.168.123.2:56004">
                <folder instance="device" name="config" prettyName="config">
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="backup" prettyName="Backup" setRoles="concentrator.manage" value="no"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="name" prettyName="Name" setRoles="concentrator.manage" value=""/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="8192" name="options" prettyName="Options" setRoles="concentrator.manage" value=""/>
                    <config getRoles="concentrator.manage" instance="device.password.config" maxLength="255" name="password" prettyName="Password" setRoles="concentrator.manage" value="74BFC2D02C282038C1090DD08E1A4402"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="reconnect" prettyName="Reconnect" setRoles="concentrator.manage" value="yes"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="ssl" prettyName="SSL" setRoles="concentrator.manage" value="yes"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="username" prettyName="Username" setRoles="concentrator.manage" value="admin"/>
                </folder>
            </folder>

            <folder instance="folder" name="192.168.123.3:56002" prettyName="192.168.123.3:56002">
                <folder instance="device" name="config" prettyName="config">
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="backup" prettyName="Backup" setRoles="concentrator.manage" value="no"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="name" prettyName="Name" setRoles="concentrator.manage" value=""/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="1024" name="options" prettyName="Options" setRoles="concentrator.manage" value=""/>
                    <config getRoles="concentrator.manage" instance="device.password.config" maxLength="255" name="password" prettyName="Password" setRoles="concentrator.manage" value="74BFC2D02C282038C1090DD08E1A4402"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="reconnect" prettyName="Reconnect" setRoles="concentrator.manage" value="yes"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="ssl" prettyName="SSL" setRoles="concentrator.manage" value="yes"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="username" prettyName="Username" setRoles="concentrator.manage" value="admin"/>
                </folder>
            </folder>
            <folder instance="folder" name="192.168.123.44:50002" prettyName="192.168.123.44:50002">
                <folder instance="device" name="config" prettyName="config">
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="backup" prettyName="Backup" setRoles="concentrator.manage" value="no"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="name" prettyName="Name" setRoles="concentrator.manage" value=""/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="8192" name="options" prettyName="Options" setRoles="concentrator.manage" value=""/>
                    <config getRoles="concentrator.manage" instance="device.password.config" maxLength="255" name="password" prettyName="Password" setRoles="concentrator.manage" value="74BFC2D02C282038C1090DD08E1A4402"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="reconnect" prettyName="Reconnect" setRoles="concentrator.manage" value="yes"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="ssl" prettyName="SSL" setRoles="concentrator.manage" value="no"/>
                    <config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="username" prettyName="Username" setRoles="concentrator.manage" value="admin"/>
                </folder>
            </folder>
        </folder>
    </folder>
    <folder instance="folder" name="database" prettyName="database">
        <folder instance="folder" name="config" prettyName="config">
... snip ...
        </folder>
    </folder>
</root>
NotesThe old log will be rolled out when DB maximum size is reached.

Attachments

    Outcomes