000031068 - Large CSV feed files not working with custom feeds in RSA Security Analytics 10.5

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000031068
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics UI, Security Analytics Server, Decoder, Log Decoder
RSA Version/Condition: 10.5.0.0
Platform: CentOS
O/S Version: EL6
IssueWhen using large csv files for custom feeds, the time taken for them to compile can be too great and result in a "Failed" message under the Feeds screen. This article explains a workaround.
The issue has been seen with csv feed files over 20MB. These may take over 20 minutes to compile on some systems.
Failed Feed
The steps below explain how to reproduce the issue.
  1. Create your custom feed as normal.
  2. Notice that for large CSV files the feed will fail to apply to the decoder.
ResolutionRun the following script on the SA Server. This can be run as a cronjob if desired.
The script will look through all your scheduled feeds, recompile them and then apply them. In the example script below
  • 192.168.123.2 is the IP address of a Packet Decoder
  • 192.168.123.3 is the IP address of a Log Decoder
Create a file with the following content and make it executable.
Be aware that the service account passwords are exposed in this file.
Make sure that password-less ssh connections have been set up between the SA Server and the log and packet decoders, so that the feeds can be copied over. For more information google the ssh-copy-id command.
find /var/lib/netwitness/uax/scheduler/ |grep xml >/tmp/feeds
for feed in $(cat /tmp/feeds)
do
  FEEDDIR=$(dirname $feed)
  FEEDNAME=$(basename $feed)
  echo $FEEDDIR
  echo $FEEDNAME
  cd $FEEDDIR
  NwConsole -c "feed create $FEEDNAME" -c "exit"
  scp *.feed root@192.168.123.3:/etc/netwitness/ng/feed
  scp *.feed root@192.168.123.2:/etc/netwitness/ng/feed
  NwConsole -c "login 192.168.123.2:50004 admin netwitness" -c "/decoder/parsers feed op=notify" -c "exit"
  NwConsole -c "login 192.168.123.3:50002 admin netwitness" -c "/decoder/parsers feed op=notify" -c "exit"
done

 
NotesAn example of the script running on my test system:
 
[root@rsareNsa ~]# ./ManualDeployFeeds.sh
/var/lib/netwitness/uax/scheduler/a5dc6f31-9924-4365-be5f-22e93b62b0f5
Websense.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>feed create Websense.xml
Creating feed Websense...
done.  167 entries, 0 invalid records
All feeds complete.
>exit
Websense.feed                                                                     100% 7830     7.7KB/s   00:00
Websense.feed                                                                     100% 7830     7.7KB/s   00:00
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>login 192.168.123.2:50004 admin netwitness
Successfully logged in as session 128854
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:10 [ChannelManager::messageHandler]  Socket Error: Operation canceled
Logged out of 192.168.123.2:50004
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>login 192.168.123.3:50002 admin netwitness
Successfully logged in as session 132591
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:10 [ChannelManager::messageHandler]  Socket Error: Operation canceled
Logged out of 192.168.123.3:50002
/var/lib/netwitness/uax/scheduler/ca159f3e-b80d-4252-9a76-5573209fa3da
ECAT40.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>feed create ECAT40.xml
Creating feed ECAT40...
done.  150 entries, 0 invalid records
All feeds complete.
>exit
ECAT40.feed                                                                       100% 4657     4.6KB/s   00:00
ECAT40.feed                                                                       100% 4657     4.6KB/s   00:00
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>login 192.168.123.2:50004 admin netwitness
Successfully logged in as session 128885
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:43 [ChannelManager::messageHandler]  Socket Error: Operation canceled
Logged out of 192.168.123.2:50004
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>login 192.168.123.3:50002 admin netwitness
Successfully logged in as session 132622
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:43 [ChannelManager::messageHandler]  Socket Error: Operation canceled
Logged out of 192.168.123.3:50002
/var/lib/netwitness/uax/scheduler/f84130cc-db1c-4bed-8c2a-3defca1f80a4
NetworkNamesCIDR.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>feed create NetworkNamesCIDR.xml
Creating feed NetworkNamesCIDR...
done.  16 entries, 0 invalid records
All feeds complete.
>exit
NetworkNamesCIDR.feed                                                             100%  732     0.7KB/s   00:00
NetworkNamesCIDR.feed                                                             100%  732     0.7KB/s   00:00
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>login 192.168.123.2:50004 admin netwitness
Successfully logged in as session 128918
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:24:17 [ChannelManager::messageHandler]  Socket Error: Operation canceled
Logged out of 192.168.123.2:50004
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>login 192.168.123.3:50002 admin netwitness
Successfully logged in as session 132654
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:24:17 [ChannelManager::messageHandler]  Socket Error: Operation canceled
Logged out of 192.168.123.3:50002
/var/lib/netwitness/uax/scheduler/e3fadf43-0de7-4783-b576-482ac9b773f1
CollectionTypeFeed.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc.  All Rights Reserved.
>feed create CollectionTypeFeed.xml
Creating feed CollectionTypeFeed...
done.  9 entries, 0 invalid records
All feeds complete.
>exit
CollectionTypeFeed.feed                                                           100%  391     0.4KB/s   00:00
CollectionTypeFeed.feed                                                           100%  391     0.4KB/s   00:00

More information on creating a custom feed can be found in the Security Analytics documentation.

For instance in 10.5 this would be done using the instructions in the Security Analytics 10.5 User Guide.

Attachments

    Outcomes