AnsweredAssumed Answered

REST API to reload parsers / feeds?

Question asked by Don Radick on May 5, 2014
Latest reply on May 21, 2014 by Christopher Ahearn

Like • Show 0 Likes0
Comment • 9

Hi all,

I inherited a Netwitness infrastructure (now at version 10.3.2) - and part of the desired processing is

downloading new feed CSV files, massaging them and creating new custom feeds. We are using

Python to process this on a Windows server.

After the feeds have been uploaded to the correct Informer directory, the script sends a reload message

to the decoders. This doesn't look like it's working - here's the line of Python code:

urllib.urlopen('http://' + adminID + ':' + adminPW + '@' + ip_address + ':50104/parsers?msg=feed&op=reload')

I've spent some time in a browser going through decoder/parser/ REST API, and not getting anywhere.

I think the "msg=feed" parameter is correct, but I'm not sure about the "op" parameter.

Here is what the GUI is showing for "op".

op - <string, optional, {enum-one:messages|parameters|description|values|roles|extra}> The specific help operation to perform

I can't seem to get a list of permissible values for this.

Any tips for further GUI exploration of REST or permissible values for "op" ?

thanks.

Christopher Ahearn

May 9, 2014 12:11 PM

Correct Answer

In that case, you could just issue the 'help' call.

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=help

The output:

Bad Request

Parameter op: The value 'help' is invalid, must be one of the values 'notify|reload|remove|delete'

The possible values are listed in the message.

I also do this from NwConsole. I haven't tried it from EVERY node. Much of what I had done via NwConsole, I do now via REST. Much of what I had done via REST I stored in a text file and use as a reference. This was done based on trial and error.

I am attaching that notes file here. It isn't 'documentation'...just a brain dump of what worked for me.

Chris

See the reply in context

No one else had this question

Outcomes

huan zhou @ Christopher Ahearn on May 9, 2014 11:38 AM

[root@NWAPPLIANCE30922 ~]# curl -v http://admin:netwitness@127.0.0.1:50104/decoder/parsers?msg=reload

* About to connect() to 127.0.0.1 port 50104 (#0)

* Trying 127.0.0.1... connected

* Connected to 127.0.0.1 (127.0.0.1) port 50104 (#0)

* Server auth using Basic with user 'admin'

> GET /decoder/parsers?msg=reload HTTP/1.1

> Authorization: Basic YWRtaW46bmV0d2l0bmVzcw==

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: 127.0.0.1:50104

> Accept: */*

< HTTP/1.1 200 OK

< Content-Length: 130

< Connection: Keep-Alive

< Pragma: no-cache

< Expires: -1

< Cache-Control: no-cache, no-store, must-revalidate

< Content-Type: text/xml; charset=utf-8

<?xml version="1.0" encoding="utf-8"?>

<string>The parsers have been reloaded</string>

</response>

* Connection #0 to host 127.0.0.1 left intact

* Closing connection #0

[root@NWAPPLIANCE30922 ~]#

Actions

huan zhou @ huan zhou on May 9, 2014 11:41 AM

[root@NWAPPLIANCE30922 ~]# curl -v 'http://admin:netwitness@127.0.0.1:50104/decoder/parsers?msg=feed&op=reload'

* About to connect() to 127.0.0.1 port 50104 (#0)

* Trying 127.0.0.1... connected

* Connected to 127.0.0.1 (127.0.0.1) port 50104 (#0)

* Server auth using Basic with user 'admin'

> GET /decoder/parsers?msg=feed&op=reload HTTP/1.1

> Authorization: Basic YWRtaW46bmV0d2l0bmVzcw==

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: 127.0.0.1:50104

> Accept: */*

< HTTP/1.1 200 OK

< Content-Length: 133

< Connection: Keep-Alive

< Pragma: no-cache

< Expires: -1

< Cache-Control: no-cache, no-store, must-revalidate

< Content-Type: text/xml; charset=utf-8

<?xml version="1.0" encoding="utf-8"?>

<string>All feeds flagged to be reloaded.</string>

</response>

* Connection #0 to host 127.0.0.1 left intact

* Closing connection #0

[root@NWAPPLIANCE30922 ~]#

Actions

Don Radick May 9, 2014 11:52 AM

Thanks guys - both helpful answers, but my original question was about REST documentation.

The RSA folks all say "browse the rest api on a machine and click on the asterisk for help" -

but I couldn't seem to get the valid "parameters" or "values".

The same thing happens in the Python/TK REST explorer that some Netwitness engineer coded.

re:Patriot3w -

Thanks for the CURL example! I'm having a heck of a time trying to get pycurl to work on my Windows Informer

box, but I can just call "curl.exe" from Python, and it should work perfectly to reload the feeds.

Actions

Christopher Ahearn

May 9, 2014 12:11 PM

In that case, you could just issue the 'help' call.

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=help

The output:

Bad Request

Parameter op: The value 'help' is invalid, must be one of the values 'notify|reload|remove|delete'

The possible values are listed in the message.

I am attaching that notes file here. It isn't 'documentation'...just a brain dump of what worked for me.

Chris

Attachments

rest_notes.txt16.7 KB

Actions

Don Radick @ Christopher Ahearn on May 9, 2014 1:16 PM

ahearc - you da man!

Thanks SO much for the REST doc.

Of course I plan to keep and expand it as I explore more areas of NW.

Always trying to automate more and more to speed up our IR process.

Gotta love curl command line.

I solved a sticky problem in REST by calling curl command line from Python.

warm regards,

don

Actions

Christopher Ahearn

May 21, 2014 2:34 PM

op=notify

Chris Ahearn

Actions

Christopher Ahearn

May 21, 2014 2:44 PM

http://192.168.99.51:50104/decoder/parsers?msg=reload

parser reload via rest

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=notify

feed reload via rest

Chris Ahearn

Senior Security Practice Consultant

RSA Security Analytics

Cell: 845.461.5985

Actions

Christopher Ahearn

May 21, 2014 2:48 PM

Actually…as I re-read this a third time, are the feeds first compiled as a feed or are they just CSV’s?

You could create a recurring feed task in SA Live custom feed wizard to point to the CSV via an http url.

In order for the feed op=notify to really work, the compiled feed (filename.feed) needs to be in the /etc/netwitness/ng/feeds directory first.

It wasn’t clear from the original post if the file was on the decoder or not.

Chris

Chris Ahearn

Senior Security Practice Consultant

RSA Security Analytics

Cell: 845.461.5985

Actions

9 Replies

Christopher Ahearn May 9, 2014 8:38 AM
Mark Correct
Correct Answer
http://192.168.99.51:50104/decoder/parsers?msg=reload
parser reload via rest

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=notify
feed reload via rest
Like • Show 0 Likes0
Actions
- huan zhou @ Christopher Ahearn on May 9, 2014 11:38 AM
  Mark Correct
  Correct Answer
  [root@NWAPPLIANCE30922 ~]# curl -v http://admin:netwitness@127.0.0.1:50104/decoder/parsers?msg=reload
  * About to connect() to 127.0.0.1 port 50104 (#0)
  * Trying 127.0.0.1... connected
  * Connected to 127.0.0.1 (127.0.0.1) port 50104 (#0)
  * Server auth using Basic with user 'admin'
  > GET /decoder/parsers?msg=reload HTTP/1.1
  > Authorization: Basic YWRtaW46bmV0d2l0bmVzcw==
  > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
  > Host: 127.0.0.1:50104
  > Accept: */*
  >
  < HTTP/1.1 200 OK
  < Content-Length: 130
  < Connection: Keep-Alive
  < Pragma: no-cache
  < Expires: -1
  < Cache-Control: no-cache, no-store, must-revalidate
  < Content-Type: text/xml; charset=utf-8
  <
  <?xml version="1.0" encoding="utf-8"?>
  <response flags="1073872897">
  <string>The parsers have been reloaded</string>
  </response>
  * Connection #0 to host 127.0.0.1 left intact
  * Closing connection #0
  [root@NWAPPLIANCE30922 ~]#
  Like • Show 1 Like1
  Actions
  - huan zhou @ huan zhou on May 9, 2014 11:41 AM
    Mark Correct
    Correct Answer
    [root@NWAPPLIANCE30922 ~]# curl -v 'http://admin:netwitness@127.0.0.1:50104/decoder/parsers?msg=feed&op=reload'
    * About to connect() to 127.0.0.1 port 50104 (#0)
    * Trying 127.0.0.1... connected
    * Connected to 127.0.0.1 (127.0.0.1) port 50104 (#0)
    * Server auth using Basic with user 'admin'
    > GET /decoder/parsers?msg=feed&op=reload HTTP/1.1
    > Authorization: Basic YWRtaW46bmV0d2l0bmVzcw==
    > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
    > Host: 127.0.0.1:50104
    > Accept: */*
    >
    < HTTP/1.1 200 OK
    < Content-Length: 133
    < Connection: Keep-Alive
    < Pragma: no-cache
    < Expires: -1
    < Cache-Control: no-cache, no-store, must-revalidate
    < Content-Type: text/xml; charset=utf-8
    <
    <?xml version="1.0" encoding="utf-8"?>
    <response flags="1073872897">
    <string>All feeds flagged to be reloaded.</string>
    </response>
    * Connection #0 to host 127.0.0.1 left intact
    * Closing connection #0
    [root@NWAPPLIANCE30922 ~]#
    Like • Show 0 Likes0
    Actions
Don Radick May 9, 2014 11:52 AM
Mark Correct
Correct Answer
Thanks guys - both helpful answers, but my original question was about REST documentation.
The RSA folks all say "browse the rest api on a machine and click on the asterisk for help" -
but I couldn't seem to get the valid "parameters" or "values".

The same thing happens in the Python/TK REST explorer that some Netwitness engineer coded.

re:Patriot3w -
Thanks for the CURL example! I'm having a heck of a time trying to get pycurl to work on my Windows Informer
box, but I can just call "curl.exe" from Python, and it should work perfectly to reload the feeds.
Like • Show 0 Likes0
Actions
Christopher Ahearn May 9, 2014 12:11 PM
Unmark Correct
Correct Answer
In that case, you could just issue the 'help' call.

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=help

The output:

Bad Request
Parameter op: The value 'help' is invalid, must be one of the values 'notify|reload|remove|delete'

The possible values are listed in the message.

I also do this from NwConsole. I haven't tried it from EVERY node. Much of what I had done via NwConsole, I do now via REST. Much of what I had done via REST I stored in a text file and use as a reference. This was done based on trial and error.

I am attaching that notes file here. It isn't 'documentation'...just a brain dump of what worked for me.

Chris
Attachments
- rest_notes.txt16.7 KB
Like • Show 1 Like1
Actions
- Don Radick @ Christopher Ahearn on May 9, 2014 1:16 PM
  Mark Correct
  Correct Answer
  ahearc - you da man!
  
  Thanks SO much for the REST doc.
  Of course I plan to keep and expand it as I explore more areas of NW.
  Always trying to automate more and more to speed up our IR process.
  
  Gotta love curl command line.
  I solved a sticky problem in REST by calling curl command line from Python.
  
  warm regards,
  
  don
  Like • Show 1 Like1
  Actions
Christopher Ahearn May 21, 2014 2:34 PM
Mark Correct
Correct Answer
op=notify

--
Chris Ahearn
Like • Show 0 Likes0
Actions
Christopher Ahearn May 21, 2014 2:44 PM
Mark Correct
Correct Answer
http://192.168.99.51:50104/decoder/parsers?msg=reload
parser reload via rest

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=notify
feed reload via rest

--
Chris Ahearn
Senior Security Practice Consultant
RSA Security Analytics
Cell: 845.461.5985
Like • Show 0 Likes0
Actions
Christopher Ahearn May 21, 2014 2:48 PM
Mark Correct
Correct Answer
Actually…as I re-read this a third time, are the feeds first compiled as a feed or are they just CSV’s?

You could create a recurring feed task in SA Live custom feed wizard to point to the CSV via an http url.

In order for the feed op=notify to really work, the compiled feed (filename.feed) needs to be in the /etc/netwitness/ng/feeds directory first.

It wasn’t clear from the original post if the file was on the decoder or not.

Chris

--
Chris Ahearn
Senior Security Practice Consultant
RSA Security Analytics
Cell: 845.461.5985
Like • Show 0 Likes0
Actions

REST API to reload parsers / feeds?

Bad Request

Outcomes

Bad Request

Attachments

Related Content

Recommended Content