Skip navigation

Log in to create and rate content, and to follow, bookmark, and share content with other members.

AnsweredAssumed Answered

How to alert based on multiple meta keys?

Question asked by Uma Mahesh Padisetty on Jul 27, 2017
Latest reply on Jul 31, 2017 by Christopher Ahearn

Like • Show 0 Likes0
Comment • 12

Hi,

Scenerio:

A log is parsed into below meta keys

aba = 'xxxx'

dda = 'yyyy'

.....

I have blacklist of aba,dda pair

blacklisted_account,aba1,dda1

blacklisted_account,aba2,dda2

I need to alert if any log has aba=aba1 and dda=dda1

OR aba=aba2 and dda=dda2

I should not alert when aba=aba1 and dda=dda2

-----------
One solution i have in mind is as below which i am not sure works.

To create custom meta key which is created while log is parsed by SA

abadda = aba||'!'||dda

Feed is

tags,abadda

blacklisted_account,aba1!dda1
blacklisted_account,aba2!dda2

ESPER Alert

'blacklisted_account'=ANY(tags)

-------------

Hence i am looking for a way to create abadda (a custom meta key carved out of existing metakeys)

Or

Alternative way to accomplish above alerting.

Thanks,

Uma Mahesh | 7068402149

ADP

No one else has this question

Outcomes

12 Replies

Christopher Ahearn Jul 28, 2017 4:21 PM
Mark Correct
Correct Answer
Would an app rule work in this case?

name=blacklisted_account rule= aba = 'aba1' && dda = 'dda1'

This assumes that both 'aba' and 'dda' were meta keys.
Like • Show 0 Likes0
Actions
- Uma Mahesh Padisetty @ Christopher Ahearn on Jul 28, 2017 4:25 PM
  Mark Correct
  Correct Answer
  Hi Christopher,
  I have 10k+ blacklisted (aba,dda) pairs and growing.
  So they have to go to a feed that has to be matched against meta key pair aba,dda in the log for the alert to fire.
  hence manual addition in app rule doesn't work.
  Like • Show 0 Likes0
  Actions
  - Christopher Ahearn @ Uma Mahesh Padisetty on Jul 28, 2017 4:54 PM
    Mark Correct
    Correct Answer
    In that case, you could look at a multi-indexed feed. This is a feed that would match against multiple meta-keys before writing meta. In a sense, it is like the app rule I wrote above...just a lot bigger.
    
    Playing with this in my lab, I used the following:
    
    ===================multiblacklist.xml==================
    <?xml version="1.0" encoding="utf-8"?>
       <FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="feed-definitions.xsd">
          <FlatFileFeed name="multiblacklist"
          path="multiblacklist.csv"
          separator=","
          comment="#">
    
          <MetaCallback name="DeviceSource">
             <Meta name="nethost.src" valuetype="Text" ignorecase="true"/>
          </MetaCallback>
    
          <MetaCallback name="RootHost">
             <Meta name="root.host" valuetype="Text"/>
          </MetaCallback>
    
          <LanguageKeys>
             <LanguageKey name="alert" valuetype="Text"/>
          </LanguageKeys>
    
          <Fields>
             <Field index="1" type="index" key="DeviceSource"/>
             <Field index="2" type="index" key="RootHost"/>
             <Field index="3" type="value" key="alert" />
          </Fields>
       </FlatFileFeed>
    </FDF>
    
    ===================multiblacklist.csv==================
    #DeviceSource,RootHost,alert
    iPhone6-Chris,google.com,blacklist_iphone6_google
    MacbookPro-Chris,shodan.io,blacklist_mbp_shodan
    iPadMini-Kailynn,googlevideo.com,blacklist_kk_youtube
    iPadMini-Kailynn,youtube.com,blacklist_kk_youtube
    
    This produced the following alert meta:
    It sounds like this is a straight forward key/value pairing which is all the feed was told to do. It is also possible to have multiple keys be checked for matches but it doesn't sound like that is needed for this specific use case.
    
    In your use case, column 1 might be the value in 'aba'. Column 2 might be the value in 'dda'. Column 3 would be the meta you wish to write such as 'blacklisted_account' or something similar.
    
    Chris
    Attachments
    multiblacklist.csv.zip287 bytes
    multiblacklist.xml.zip521 bytes
    1 person found this helpful
    Like • Show 1 Like1
    Actions
    - Uma Mahesh Padisetty @ Christopher Ahearn on Jul 28, 2017 5:01 PM
      Mark Correct
      Correct Answer
      oh wow.
      the UI doesn't let me have multi index feed so i assumed it is not possible.
      just realized editing xml will work to have multi index feed.
      I will try and let you know.
      
      Thanks
      Have a good weekend
      Like • Show 0 Likes0
      Actions
      - Christopher Ahearn @ Uma Mahesh Padisetty on Jul 28, 2017 5:05 PM
        Mark Correct
        Correct Answer
        Correct...and I should have mentioned that. This is not something that can be done within the Feed Wizard in the UI. I had edited the XML by hand and deployed the feed manually. It does appear we have that on the list of feature enhancements though.
        
        I haven't tried deploying the compiled feed with the wizard but can try that out.
        Like • Show 0 Likes0
        Actions
      - Christopher Ahearn @ Uma Mahesh Padisetty on Jul 28, 2017 5:17 PM
        Mark Correct
        Correct Answer
        Yeah..feed wizard said it was not a valid feed xml. I wound up compiling the feed on the decoder manually.
        
        To do so, I ran the following command from the console in the same directory as my XML and CSV files:
        
        NwConsole -c feed create multiblacklist.xml
        
        Make sure you have 0 or very few invalid records.
        
        Then, I copied the multiblacklist.feed file into the /etc/netwitness/ng/feeds directory and issued a reload of the feeds.
        
        To check your work, you can review /var/log/messages and look for the feed being loaded.
        
        Chris
        Like • Show 0 Likes0
        Actions
        Uma Mahesh Padisetty @ Christopher Ahearn on Jul 28, 2017 5:43 PM
        Mark Correct
        Correct Answer
        me too got Invalid XML Error.
        NwConsole - is this a executable on SA Appliance? (so far i only have access to UI based admin view)
        
        <?xml version="1.0" encoding="UTF-8"?>
        <FDF>
        <FlatFileFeed comment="#" separator="," path="blacklistabadda.csv" name="blacklistabadda">
        <MetaCallback valuetype="Text" name="aba" ignorecase="true"/>
        <MetaCallback valuetype="Text" name="dda" ignorecase="true"/>
        <LanguageKeys>
        <LanguageKey valuetype="Text" name="alert_tags"/>
        </LanguageKeys>
        <Fields>
        <Field type="index" index="1" key="aba"/>
        <Field type="index" index="2" key="dda"/>
        <Field key="alert_tags" type="value" index="3"/>
        </Fields>
        </FlatFileFeed>
        </FDF>
        Like • Show 0 Likes0
        Actions
        Christopher Ahearn @ Uma Mahesh Padisetty on Jul 28, 2017 5:52 PM
        Mark Correct
        Correct Answer
        Yes.
        
        Sent from my mobile device
        Like • Show 0 Likes0
        Actions
        Uma Mahesh Padisetty @ Christopher Ahearn on Jul 31, 2017 11:05 AM
        Mark Correct
        Correct Answer
        Hi Chris,
        In The Feed create above, how to make the feed recurrent?
        Like • Show 0 Likes0
        Actions
        Christopher Ahearn @ Uma Mahesh Padisetty on Jul 31, 2017 11:09 AM
        Mark Correct
        Correct Answer
        Since this is being done outside of the feed wizard, you'd likely need to script the process. Once the xml is created, a cron job could be created to retrieve the updated CSV and create the feed file using the NwConsole syntax. Once created, that file could be pushed to the decoders and a feed reload issued.
        Like • Show 0 Likes0
        Actions
        Uma Mahesh Padisetty @ Christopher Ahearn on Jul 31, 2017 11:28 AM
        Mark Correct
        Correct Answer
        So a cron job has to run on SA Broker
        
        That does
        
        1. Execute below commands in sequence
        
        CURL http://application/feeds/multiblacklist.csv
        
        NwConsole –c feed create multiblacklist.xml
        
        cp *.feed /etc/netwitness/ng/feeds
        
        NwConsole –c parsers feed op=notify
        
        Uma
        Like • Show 0 Likes0
        Actions
        Christopher Ahearn @ Uma Mahesh Padisetty on Jul 31, 2017 11:31 AM
        Mark Correct
        Correct Answer
        That would probably work. SA server is one place. NwConsole could be installed anywhere really. As long as it can get the updated CSV and perform the other functions, it could be run from any system.
        Like • Show 0 Likes0
        Actions

Recommended Content