AnsweredAssumed Answered

Disabling automatic parser reload for REST file upload in 9.8

Question asked by RSA Admin

on Oct 29, 2013
Latest reply on Nov 3, 2013 by RSA Admin

Like • Show 0 Likes0
Comment • 6

Hello,

I'm in the process of automating the deployment of some of our custom parsers and feeds. I see that the REST API apparently allows for the automatic reload to be disabled, but I cannot make that work. Even if using the NW-built web form at <decoder>:50104/decoder/parsers/upload and leaving the "reload" checkbox turned off, the device still logs "Issuing automatic parser reload" and reloads all parsers.

I've tried to find a real REST API document, but haven't found anything that details the things I need. I first thought that I could use /decoder/parsers?msg=upload&op=start, sending the file in the POST body... but that didn't work. I *can* successfully send the file to /decoder/parsers/upload but then all attempts to disable the auto reload have failed.

Can someone please point me to a real API document, or perhaps an example of how to disable this automatic reload?

Thanks!

Josh

RSA Admin

Nov 1, 2013 2:55 PM

Correct Answer

Hi Josh-

What version of 9.8 are you using? I have verified that 9.8.5.14 works as intended using /decoder/parsers/upload with out setting the reload argument.

To be a little more specific, once a file is received (upload completed), decoder will delay the reload by 3 seconds to provide a window in which additional content can be received. Each subsequent upload within this window will push the delay another 3 seconds from the completion of the upload. This can continue up until 1 minute from the initial upload at which point a reload is issued regardless of subsequent content upload.

There is no way to disable or alter the parser reload time windows.

The /decoder/parsers/upload REST interface accepts content as multipart/form-data (as used by the html page) or application/zip. The multipart/form-data has a single part with the following headers:

Content-Disposition: form-data; name="feed/parser"; filename="test.lua"

Content-Type: application/octet-stream

The application/zip requires that your content be placed within a flat zip file (no directory structure within the archive) and posted with a content type of application zip. The posted zip file can only contain a single piece of content (feed, parser, etc...). The following demonstrates using curl to upload a zip file:

curl -u admin:netwitness --verbose --header "Content-Type: application/zip" --data-binary @test.zip "http://localhost:50104/decoder/parsers/upload"

The following logs show the upload process for the above curl command including the 3 second reload delay:

id=495493 time=1383330807 level=audit module=Engine msg=User admin (session 596, 127.0.0.1:49855) has logged in

id=495494 time=1383330807 level=info module=Rest msg=Received 3971-42cb-0513-22a8.zip of size 586 for upload

id=495495 time=1383330807 level=audit module=Parse msg=User admin (session 596, 127.0.0.1:49855) has started uploading file '3971-42cb-0513-22a8.zip'

id=495496 time=1383330807 level=audit module=Parse msg=User admin (session 596, 127.0.0.1:49855) has finished uploading file '3971-42cb-0513-22a8.zip'

id=495497 time=1383330807 level=audit module=Parse msg=User admin (session 596, 127.0.0.1:49855) has finished uploading file 'test.lua'

id=495498 time=1383330807 level=audit module=Engine msg=User admin (session 596, 127.0.0.1:49855) has logged out

id=495499 time=1383330810 level=audit module=Parse msg=Issuing automatic parser reload

I hope that helps. Please let me know if you have any additional questions.

Edward Quackenbush

See the reply in context

No one else had this question

Outcomes

RSA Admin

Oct 29, 2013 5:35 PM

And before anyone replies to look at the REST API PDF from the User Conference, I already have. It doesn't cover this subject, unfortunately. It also does not cover the usage of /decoder/parser?msg=upload (nor does anything else I can find). Thanks in advance!

Actions

RSA Admin

Oct 31, 2013 2:56 PM

Does anyone have any input on this? This REST API seems very powerful but so much of it is apparently completely undocumented. I can't tell if this piece is working as intended or if it's broken... because I have no way to know exactly how it's supposed to be used. Surely someone involved with development of the REST API read these posts...

Actions

RSA Admin

@ RSA Admin on Oct 31, 2013 9:57 PM

It's working as intended. The reload command is meant to reload immediately. If you don't issue a reload, it will wait to see if more parsers are uploaded within a small amount of time. If no additional feeds/parsers are uploaded within the time window, it will issue the reload automatically.

There's no point to updating parsers and feeds on the decoders if they aren't reloaded. The automatic reload is to prevent customers from adding new content but forgetting to reload. It's also meant to keep reloads to a minimum (reloads are expensive), which is why there's a timer to automatically reload after uploads stop.

Actions

RSA Admin

@ RSA Admin on Oct 31, 2013 10:08 PM

That timer must be less than a second. If I upload two files in a row, it issues an "automatic reload" after each. What I'd like to do is *disable* that automatic reload. There is a point to doing that, since I want to perform several operations at once, including uploading several parsers/feeds. After which I'll manually issue one reload.

The help message says that you can set reload to 0 on a "finished" op... but then I can't find a way to successfully issue a finished op. I can issue a start op and get a "Success" response, but any finished op results in an error message. Something to the tune of "no files were uploaded on pipe xxxxxxxx".

Actions

RSA Admin

@ RSA Admin on Nov 1, 2013 2:55 PM