AnsweredAssumed Answered

ESM CEF Syslog Template for report/dashlet creation

Question asked by Marinos Roussos on Feb 16, 2018
Latest reply on Feb 19, 2018 by Marinos Roussos

Like • Show 0 Likes0
Comment • 8

I am trying to use what used to be “ESM CEF Syslog Template” for Syslog notifications in admin->event sources-> monitoring policies.

Ultimately, I would like to create reports from the META values from that Syslog notification like it was possible previously. RSA haven’t provided a proper method of having event source notifications in a dashlet so I am trying to improvise to see what I can put on a dashlet.

This template is not there anymore and used to generate rsa_security_analytics_event_source_monitoring device.type.

The current “ESM Default Syslog Template” also generates that device.type but the syslog payload is not parsed, therefore useless.

I tried using legacy CEF Syslog template that was ported from 10.4 and getting these errors:

CEF:0|RSA|Security Analytics ESA|10.4|${statement}|${moduleName}|${severity}|rt=${time?datetime} id=${id} source=${eventSourceId}

2018-02-16 14:49:29,556 [scheduler_Worker-1] ERROR freemarker.runtime - Template processing error: "Expression statement is undefined on line 1, column 41 in 5a86bce2f280b718fc2132dd."

Expression statement is undefined on line 1, column 41 in 5a86bce2f280b718fc2132dd.

The problematic instruction:

----------

==> ${statement} [on line 1, column 39 in 5a86bce2f280b718fc2132dd]

----------

Java backtrace for programmers:

----------

freemarker.core.InvalidReferenceException: Expression statement is undefined on line 1, column 41 in 5a86bce2f280b718fc2132dd.

at freemarker.core.TemplateObject.assertNonNull(TemplateObject.java:125)

at freemarker.core.Expression.getStringValue(Expression.java:118)

at freemarker.core.Expression.getStringValue(Expression.java:93)

at freemarker.core.DollarVariable.accept(DollarVariable.java:76)

at freemarker.core.Environment.visit(Environment.java:221)

at freemarker.core.MixedContent.accept(MixedContent.java:92)

at freemarker.core.Environment.visit(Environment.java:221)

at freemarker.core.Environment.process(Environment.java:199)

at freemarker.template.Template.process(Template.java:259)

at com.rsa.netwitness.carlos.notification.Notification.resolve(Notification.java:198)

at com.rsa.netwitness.carlos.notification.NotificationEngine.resolve(NotificationEngine.java:558)

at com.rsa.netwitness.carlos.notification.NotificationEngine.dispatch(NotificationEngine.java:448)

at com.rsa.smc.esm.core.alert.notification.NotificationService.dispatch(NotificationService.java:135)

at com.rsa.smc.esm.core.alert.notification.NotificationService.dispatchAlarm(NotificationService.java:151)

at com.rsa.smc.esm.core.alert.notification.NotificationService.processNotifications(NotificationService.java:89)

at com.rsa.smc.esm.core.jobs.NotificationDispatchJob.executeJob(NotificationDispatchJob.java:26)

at com.rsa.netwitness.carlos.scheduling.jobs.AbstractJob.execute(AbstractJob.java:61)

at org.quartz.core.JobRunShell.run(JobRunShell.java:213)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

I then tried to use some of the variables from 10.5 Default Audit CEF templates and got a similar error like before:

CEF:0|${deviceVendor}|${deviceProduct}|${deviceVersion}|${category}|${operation}|${severity}|rt=${timestamp} src=${sourceAddress} spt=${sourcePort} suser=${identity} sourceServiceName=${deviceService} deviceExternalId=${deviceExternalId} deviceProcessName=${deviceProcessName} outcome=${outcome} msg=${text}

This worked perfectly before; I don’t know what RSA have done in the meantime. All the variables where previously defined and worked OOTB so I assumed that I wouldn’t have to start hacking XMLs and break parsers for making this to work.

In attach, you can see a screenshot and a sample report of how it used to work in the past.

Attachments

Device down report.pdf41.8 KB
report.jpg123.6 KB

No one else has this question

Outcomes

Marinos Roussos @ Guy Bruneau on Feb 16, 2018 11:07 AM

In my case, even if I copy the variables from "10.5 Default Audit CEF Template" that all of them should be parse-able based on SA Cfg: Supported CEF Meta Keys, I'm still getting the same errors as if they were undefined.

CEF:0|${deviceVendor}|${deviceProduct}|${deviceVersion}|${category}|${operation}|${severity}|rt=${timestamp} src=${sourceAddress} spt=${sourcePort} suser=${identity} sourceServiceName=${deviceService} deviceExternalId=${deviceExternalId} deviceProcessName=${deviceProcessName} outcome=${outcome} msg=${text}

sms.log:

2018-02-16 16:05:29,624 [scheduler_Worker-1] ERROR freemarker.runtime - Template processing error: "Expression deviceVendor is undefined on line 1, column 9 in 5a86bce2f280b718fc2132dd."

Expression deviceVendor is undefined on line 1, column 9 in 5a86bce2f280b718fc2132dd.
The problematic instruction:
----------
==> ${deviceVendor} [on line 1, column 7 in 5a86bce2f280b718fc2132dd]

thanks though:)

Actions

Josh Randall

@ Marinos Roussos on Feb 16, 2018 11:58 AM

Hi Marinos,

When you say the syslog payload is not parsed, can you be more specific?

I've been working on a fix with engineering for the default template that addresses the need for spaces and identifiers between <staticValue>=$<variableValue> fields when there is more than one event source in the event; e.g.:

In the fixed template, this becomes:

Is this the issue you're facing?

Actions

Marinos Roussos @ Josh Randall on Feb 16, 2018 12:14 PM

This is with the OOTB ESM Default Syslog Template. Any other template or custom fields that I tried, doesn't work at all. ie It does not generate rsa_security_analytics_event_source_monitoring and therefore useless for what I'm trying to achieve. It is failing completely with the errors I attached earlier.

This is the first part of the RAW log, and there are more entries in the Syslog but nothing is parsed after the header. See screenshot.

Actions

Josh Randall

@ Marinos Roussos on Feb 16, 2018 1:10 PM

First, to the issue of other templates not working or throwing exceptions - the ESM template has "${deviceVendor}|${deviceProduct}|" hardcoded as static values instead of freemarker variables, and the ${deviceVendor} variable is instead ${vendor} in the ESM template, which leads me to assume that those values are either not present or named differently in ESM.

For the syslog payload not parsing, it looks like your "src=" values aren't registering with the CEF parser at all. Has that parser been modified from the OOTB version? I have a couple older OOTB CEF revisions (99, 110) that properly parse out the "src=" values, which is why I'm leaning towards that as the issue.

Could you download the current revision (115, released yesterday, actually) from Live and see if that changes anything?

Also, since your log does have multiple "src=" values, you can try out the new default template that will separate those out into distinct host.src metavalues (once the parsing issue is fixed, of course).

<@compress single_line=true>CEF:0|RSA|NetWitness Suite Event Source Monitoring|${version}|
<#if highAlarmsCount > 0>HighThresholdAlert|ThresholdExceeded|1|cat=${group}|Devices|
<#list 0..highAlarmEventSources?size-1 as es>
<#assign highAlarms=highAlarmEventSources[es]?split("^")>src=${highAlarms[0]}
<#if highAlarms?size > 1><#list 1..highAlarms?size-1 as i>cs${i}=${highAlarms[i]} </#list></#if>|
</#list>
</#if>
<#if lowAlarmsCount > 0>LowThresholdAlert|ThresholdViolated|1|cat=${group}|Devices|
<#list 0..lowAlarmEventSources?size-1 as es>
<#assign lowAlarms=lowAlarmEventSources[es]?split("^")>src=${lowAlarms[0]}
<#if lowAlarms?size > 1><#list 1..lowAlarms?size-1 as i>cs${i}=${lowAlarms[i]} </#list></#if>|
</#list>
</#if>
</@compress>

Actions

Marinos Roussos @ Josh Randall on Feb 19, 2018 11:29 AM

Hi, please find my responses in-line:

First, to the issue of other templates not working or throwing exceptions - the ESM templates has the "${deviceVendor}|${deviceProduct}|" values hardcoded as static values instead of freemarker variables, and the ${deviceVendor} variable is instead ${vendor} in the ESM template, which leads me to assume that those values are either not present or named differently in ESM.

Using variables and then hard-coding values defeats the purpose, doesn’t it?

Where would the customers need to go for getting the correct values?

Why has RSA not “updated” their documentation with the new values but keep producing new documents with the wrong values? It is more and more confusing when there is no accurate piece of documentation to refer to.

https://community.rsa.com/docs/DOC-84388

https://community.rsa.com/docs/DOC-84407

System Configuration Guide for Version 11.0

For the syslog payload not parsing, it looks like your "src=" values aren't registering with the CEF parser at all. Has that parser been modified at all from the OOTB version? I have a couple older OOTB CEF revisions (99, 110) that properly parse out the "src=" values, which is why I'm leaning towards that as the issue.

No, the parser is the OOTB from Live and subscribed to it. The fact that you are saying that it works on some older versions proves that RSA have been breaking it by updating the parser.

Could you download the current revision (115, released yesterday, actually) from Live and see if that changes anything?

It is already installed. Based on your above comment, it seems that it exactly the reason why it’s broken.

It’s also confusing that after you’ve confirmed that the older parser works, you suggest me to install the latest one?

Thank you. I’ve tried that “new template” and the errors are bellow

These are the errors from the “New template”:

2018-02-19 11:45:31,596 [scheduler_Worker-1] ERROR freemarker.runtime - Template processing error: "Error on line 2, column 6 in 5a8ab38df280b718fc2132de\nExpecting a boolean (true/false) expression here\nExpression highAlarmsCount does not evaluate to true/false \nit is an instance of freemarker.template.SimpleNumber"

Error on line 2, column 6 in 5a8ab38df280b718fc2132de

Expecting a boolean (true/false) expression here

Expression highAlarmsCount does not evaluate to true/false

it is an instance of freemarker.template.SimpleNumber

The problematic instruction:

----------

==> if highAlarmsCount [on line 2, column 1 in 5a8ab38df280b718fc2132de]

in user-directive compress [on line 1, column 1 in 5a8ab38df280b718fc2132de]

----------

Java backtrace for programmers:

----------

freemarker.core.NonBooleanException: Error on line 2, column 6 in 5a8ab38df280b718fc2132de

Expecting a boolean (true/false) expression here

Expression highAlarmsCount does not evaluate to true/false

it is an instance of freemarker.template.SimpleNumber

at freemarker.core.Expression.isTrue(Expression.java:150)

at freemarker.core.ConditionalBlock.accept(ConditionalBlock.java:77)

at freemarker.core.Environment.visit(Environment.java:221)

at freemarker.core.MixedContent.accept(MixedContent.java:92)

at freemarker.core.Environment.visit(Environment.java:221)

at freemarker.core.Environment.visit(Environment.java:310)

at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)