Help to make Fortiweb parser

[RSAAdmin]

Hi everybody !

We have to make a parser for the appliance Fortiweb 5.0.2 (Web Application Firewall).

I began to build it with ESI but I don't succeed to understand everything, so I would need some help/advice

Extract of brut log :

Oct 22 19:29:24 [10.200.190.166] date=2013-10-22 time=19:35:12 devname=ap-b1-fortinet-1 log_id=20000002 msg_id=000000124644 type=attack subtype="none" pri=alert device_id=FV-1KC3R13700032 timezone="(GMT+1:00)Brussels,Copenhagen,Madrid,Paris"trigger_policy="" severity_level=Low proto=tcp service=http action=Alert_Deny policy="policy-wildcard-extranet" src=XXX.XX.XXX.XXX src_port=56962 dst=10.200.190.65 dst_port=80 http_method=get http_url="/w00tw00t.at.ISC.SANS.DFind:)" http_host="none" http_agent="none" http_session_id=none msg="HTTP Host Violation"

Orange -> header part

Green -> payload part

For Message_ID, I chose the number of the variable log_id, so in this exemple the Message_ID is 20000002.

Header :

<HEADER

        id1="0001"

        id2="0001"

        content="date=&lt;hdate&gt; time=&lt;htime&gt; devname=&lt;hdevice&gt; log_id=&lt;messageid&gt; msg_id=&lt;hmessageid&gt; type=&lt;htype&gt; subtype=&lt;hsubtype&gt; pri=&lt;hpriority&gt; device_id=&lt;hdeviceid&gt; timezone=&lt;hzone&gt; &lt;!payload&gt;"/>

Message for the extract of brut log above :

<MESSAGE

        level="3"

        parse="1"

        parsedefvalue="1"

        tableid="75"

        id1="HTTP_Host_Violation"

        id2="20000002"

        eventcategory="1204000000"

        content="trigger_policy=&quot;&quot; severity_level=&lt;severity&gt; proto=&lt;protocol&gt; service=&lt;network_service&gt; action=&lt;action&gt; policy=&lt;policyname&gt; src=&lt;saddr&gt; src_port=&lt;sport&gt; dst=&lt;daddr&gt; dst_port=&lt;dport&gt; http_method=&lt;web_method&gt; http_url=&lt;web_query&gt; http_host=&lt;hostid&gt; http_agent=&lt;user_agent&gt;&quot; http_session_id=none msg=&lt;message_body&gt;"/>

This couple header/message works well with ALL the logs which have the same log_id 🙂

Now, take this extract of brut logs :

Oct 22 18:05:52 [10.200.190.166] date=2013-10-22 time=18:11:40 devname=ap-b1-fortinet-1 log_id=20000008 msg_id=000000124625 type=attack subtype="none" pri=alert device_id=FV-1KC3R13700032 timezone="(GMT+1:00)Brussels,Copenhagen,Madrid,Paris" trigger_policy="" severity_level=Low proto=tcp service=https action=Alert_Deny policy="policy-wildcard-extranet" src=XX.XXX.XXX.XX src_port=28018 dst=10.200.185.249 dst_port=80 http_method=get http_url="/Admin/redirectmail04.nsf/Login.js" http_host="mailagent.extranet.blabla.com" http_agent="Mozilla/5.0 (Windows NT 5.1; rv:24.0) Gecko/20100101 Firefox/24.0" http_session_id=AT3BN5IXDY2WQEOAP3CEUNDDENJJITRJ msg="url-deny-mailagent : URL Access Violation"

This log have the same fields than the first log, so I can use the existing header right ? Only the log_id change (that's normal, it's another type of attack message)

I just have to create a new message and it should work right ?

Header (the same than before) :

<HEADER

        id1="0001"

        id2="0001"

        content="date=&lt;hdate&gt; time=&lt;htime&gt; devname=&lt;hdevice&gt; log_id=&lt;messageid&gt; msg_id=&lt;hmessageid&gt; type=&lt;htype&gt; subtype=&lt;hsubtype&gt; pri=&lt;hpriority&gt; device_id=&lt;hdeviceid&gt; timezone=&lt;hzone&gt; &lt;!payload&gt;"/>

Message (with the log_id 20000008)  :

<MESSAGE

        level="3"

        parse="1"

        parsedefvalue="1"

        tableid="75"

        id1="URL_Access_Violation"

        id2="20000008"

        eventcategory="1201000000"

        content="trigger_policy=&quot;&quot; severity_level=&lt;severity&gt; proto=&lt;protocol&gt; service=&lt;network_service&gt; action=&lt;action&gt; policy=&lt;policyname&gt; src=&lt;saddr&gt; src_port=&lt;sport&gt; dst=&lt;daddr&gt; dst_port=&lt;dport&gt; http_method=&lt;web_method&gt; http_url=&lt;web_query&gt; http_host=&lt;hostid&gt; http_agent=&lt;user_agent&gt; http_session_id=RT1JSEM1QJ1K9SXOK9ALUS8QBHKPDSGJ msg=&lt;message_body&gt;"/>

This couple header/message works well with ONLY ONE message which have the same log_id

Take this other  line of log :

Oct 23 12:00:55 [10.200.190.166] date=2013-10-23 time=12:06:44 devname=ap-b1-fortinet-1 log_id=20000008 msg_id=000000124681 type=attack subtype="none" pri=alert device_id=FV-1KC3R13700032 timezone="(GMT+1:00)Brussels,Copenhagen,Madrid,Paris" trigger_policy="" severity_level=Low proto=tcp service=https action=Alert_Deny policy="policy-wildcard-extranet" src=XX.XXX.XXX.XX src_port=41369 dst=10.200.190.65 dst_port=443 http_method=get http_url="/names.nsf" http_host="mail04.extranet.blabla.com" http_agent="Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.3072" http_session_id=none msg="url-deny-mail0X : URL Access Violation"

She's the same right (same log_id) ? But she's not well parsed...

Have you an idea of what's going wrong ?

EDIT : Variable session_id was missing in the message definition, after have integrated this variable it works well 🙂

OK for the first problem, but I 'm still blocked for the second :

Second problem : For some logs, I can't use only the field log_id as message_id because one log_id means diffferent security message. For exemple the log_id 20000010 means these messages :

[Signatures name: WebMail] [main class name: Information Disclosure] [sub class name: Application Availability/Errors]: 080080003
[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Email Injection]: 050110001
[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: HTTP Response Splitting]: 050130001
[Signatures name: WebMail] [main class name: Cross Site Scripting]: 010000107
[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Command Injection]: 050050052
[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: PHP Injection]: 050080033
[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: HTTP Response Splitting]: 050130001
[Signatures name: WebMail] [main class name: Information Disclosure] [sub class name: Microsoft Office Document Properties Leakage]: 080050002
[Signatures name: WebMail] [main class name: Known Exploits] [sub class name: Struts 2 Vulnerability]: 090390001
[Signatures name: WebMail] [main class name: Information Disclosure] [sub class name: CF Source Code Leakage]: 080060001
[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: OS Command Injection Attacks]: 050010001

So I have to create a Message_id ; I tried log_id & the number at the end of the log, for exemple with this line of log :

Oct 23 13:54:57 [10.200.190.166] date=2013-10-23 time=14:00:46 devname=ap-b1-fortinet-1 log_id=20000010 msg_id=000000124685 type=attack subtype="none" pri=alert device_id=FV-1KC3R13700032 timezone="(GMT+1:00)Brussels,Copenhagen,Madrid,Paris" trigger_policy="" severity_level=High proto=tcp service=https action=Alert policy="policy-wildcard-extranet" src=XX.XXX.XXX.XXX src_port=56713 dst=10.200.185.250 dst_port=80 http_method=post http_url="/traveler" http_host="m-services.extranet.blabla.com" http_agent="Lotus Traveler Android 9.0" http_session_id=EHVJZ0KJNI76A8HNPKOVVANLTW3OKFUP msg="[Signatures name: WebMail] [main class name: Cross Site Scripting]: 010000107"

Message_id is [20000010][010000107], but with this header, the header of logs is not parsed

Any help/advice is welcome, thx in advance 🙂

[RSAAdmin]

I up my topic because 50% is solved 😉 nobody can help me for the second problem ??

Thx

[FernandoAllende]

First of all, you need to understand what "Message ID" stand for and then you must fully understand the log itself.

In this case, I'm not really sure that your Message ID selection is the best one. You should get more info from the log owner/creator, in this case Fortinet for its FortiWeb product. Maybe form FortiWeb manual or knowledge base you can find a better Message ID.

For your second problem, probably you can only use the number at the end of the log, in your example "010000107".

Greetings from Chile,

Fernando Allendes.

[RSAAdmin]

Hi Fernando,

Thx for you response !:-)

What I understood for Message ID : this is a unique identifier for the event.

What is saying the fortinet documentation about log_id :

The ID (log_id) is an 8-digit field located in the header, immediately following the time and date

fields.

The log_id field is a number assigned to all permutations of the same message. It classifies a

log message by the nature of the cause of the log message, such as administrator

authentication failures or traffic. Other log messages that share the same cause will share the

same log_id.

For example, creating an administrator account always has the log ID 00003401.

So I think that log_id is a good choice, but If you know tell me what would be better to differentiate the messages ?

For the second problem : the number at the end of the log (for exemple "010000107") could be a good message_id. But I saw in the logs that many numbers refers to the same attack type :

[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Command Injection]: 050050052

[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Command Injection]: 050050006

[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Command Injection]: 050050004

[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Command Injection]: 050050045

[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Command Injection]: 050050027

[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: Command Injection]: 050050051

The number at the end of the log, for exemple (for exemple "010000107") refers to a signature ID. So using this method, I would have to create as many message definitions as theres is many different signature ID....this would say that I would never have finished to create the parser because there will be always new signature ID right ?

[FernandoAllende]

You said it ... "Message ID : this is a unique identifier for the event." ... and ... about log_id "Other log messages that share the same cause will share the same log_id." ... then log_id is not the best candidate to be the Message ID.

If the number at the end of the log (for example "010000107") always changes, then probably does not refer to a signature ID and then it's not the best candidate to be the Message ID. This is why you must fully understand the log itself.

Tell me if you need professional help.

Greetings from Chile,

Fernando Allendes.

[RSAAdmin]

Fernando Allendes Fdez. a écrit:

You said it ... "Message ID : this is a unique identifier for the event." ... and ... about log_id "Other log messages that share the same cause will share the same log_id." ... then log_id is not the best candidate to be the Message ID.

I agree with you but what can I choose ? I can't use the field msg_id (for exemple msg_id=000000124644)  because this number is different for every event so I would have to make as many message definition as there is many different message :

"The MSG ID(msg_id) field is an 12-digit number located in the header, incremented with each

individual log message generated by the FortiWeb appliance. It is used only for numbering each

entry in the database, and does not necessarily reflect its cause.

Each msg_idnumber is a unique identifier for that specific log entry. No other log messages,

regardless of cause, share the same msg_id."

Fernando Allendes Fdez. a écrit:

If the number at the end of the log (for example "010000107") always changes, then probably does not refer to a signature ID and then it's not the best candidate to be the Message ID. This is why you must fully understand the log itself.

I read all the documentation I found

Exemple of a msg :

[Signatures name: WebMail] [main class name: Generic Attacks] [sub class name: SQL Injection]: 050050052

What say the manual :

"SQL Injection : Signature ID n where n is the index number of the specific predefined attack or data leak signature"

So, in my understanding, the number at the end of the log is the index number of the signature

Source :http://docs.fortinet.com/fweb/5-0-0/FortiWeb_5_0_Log_Reference_Revision1.pdf

Tell me if you need professional help.

I would need professionnal help with pleasure but by professionnal you means I have to pay right ? We just have RSA support and we can't pay for this 😕

Anyway thx for your help 😉