AnsweredAssumed Answered

Applying custom feeds to already parsed decoder data

Question asked by Ben May on Jun 4, 2013
Latest reply on Jun 10, 2013 by RSA Admin

Like • Show 0 Likes0
Comment • 5

I am wondering if anyone has any real world examples where they have the ability to be able to apply a single feed to already parsed data on a decoder.

The use case is that we have custom feeds that are produced and want to have them applied to all data on the decoder, not just data that has not been parsed. I understand that the only way this can be accomplished is to extract the already parsed data from the decoder to a pcap and then have it replayed back through a decoder. The main issue with this approach is that all the data being replayed would be parsed as opposed to just the single feed.

No one else has this question

Outcomes

5 Replies

RSA Admin Jun 4, 2013 10:04 AM
Mark Correct
Correct Answer
I assume what you are talking about is retroactively applying a newly added/updated feed to historical pcaps/metadata.

There is no existing functionality in NW that I know of to do that. Basically yes, you would need to do full packet replay against a whole new collection to achieve that.

However, depending on your goals/needs for doing a retroactive check there are several other possible solutions you could try that I'll suggest:
1. For updates to a custom feed, programatically generate a list of new elements added to the feed and then query all data for those new elements as a one-time "catch-up" operation. The query could be done automatically using NW API calls.
2. If there is a specific area of concern, such as a specific subnet, service or user, you can extract pcaps for just that period and then create a local collection in Investigator and import those packets. This would require adding the custom feed to your local Investigator.
Like • Show 0 Likes0
Actions
- Ben May @ RSA Admin on Jun 4, 2013 7:57 PM
  Mark Correct
  Correct Answer
  Option 1 was something I was hoping may be feasible as it is a single custom feed that only needs to be applied and only the new elements from that.
  
  I will look into developing some proof of concept code and post back what I find.
  
  If anyone has anything that they have put together to do something similar that could point this in the right direction it would be much appreciated.
  Like • Show 0 Likes0
  Actions
  - RSA Admin @ Ben May on Jun 10, 2013 12:29 PM
    Mark Correct
    Correct Answer
    I uploaded my URL checking code - that may help you with your implementation:
    [DEAD LINK /docs/DOC-25152]https://community.emc.com/docs/DOC-25152
    
    The following GIT command is quite to get the list of recent additions to the feeds:
    git.exe whatchanged --since="1 day ago" -p -- urlquery.csv
    Like • Show 0 Likes0
    Actions
RSA Admin Jun 10, 2013 4:11 AM
Mark Correct
Correct Answer
Jedi Mercer's solution is excellent. I'm doing exactly that. First off, I highly recommend that you put all your feeds under version control if you haven't already. Among many other benefits this allows you to do diffs very easily,
```
def _diff(feed_folder, csv_file):
    csv_path = os.path.join(feed_folder,csv_file)
    cmd = 'git.exe diff --ignore-space-at-eol master ' + csv_path
   
    p = subprocess.Popen(cmd , shell=True, stdout=subprocess.PIPE,
        cwd=feed_folder, stderr=subprocess.PIPE)
    (std, err) = p.communicate()  
       
    changes = {}
    for line in std.split('\n'):
        for mode in ('+', '-'):
            if line.startswith(mode) and not line.startswith(3*mode):               
                changes.setdefault(mode,[]).append( line[1:] )
    return changes
```
Then we build API calls to see if there any matches in the past week and how many.
Like • Show 0 Likes0
Actions
- RSA Admin @ RSA Admin on Jun 10, 2013 10:25 AM
  Mark Correct
  Correct Answer
  Yes! Put custom feeds (and other customizations you do) for Netwitness under a version control system like SVN or git or whatever you prefer. Especially good if you have more than one person making updates or an automated tool adding to feeds.
  Like • Show 0 Likes0
  Actions

Applying custom feeds to already parsed decoder data

Outcomes

Related Content

Recommended Content