AnsweredAssumed Answered

ESA alert is flooding when using sliding time window

Question asked by Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= on May 11, 2016
Latest reply on Jun 1, 2016 by Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE=

Like • Show 0 Likes0
Comment • 10

Hello Community,

I have the following ESA rule to detect portscan :

@RSAAlert(oneInSeconds=0)
SELECT * FROM Event
(
device_type='snort'
AND ip_dstport=137
)
.std:groupwin(ip_src)
.std:unique(ip_dst)
.win:time(60 sec)
GROUP BY ip_src HAVING count(ip_dst) > 3;

But when I do a test (a portscan, from the same source, on 4 different IP and on the port 137) , I have lots of ESA alerts : something like 10 alerts. These alerts match the same ip_src and the same ip_dst

However, if I choose to use a batch time windows (win:time_batch(60 sec) , I'm not flooded by ESA alerts. I have only one alert which corresponds to my test.

Do you know if this is normal ? And would you be aware of any way to avoid this flood when using sliding time windows ?

I would prefer use sliding time windows to avoid to miss some alerts if events trigger between 2 batch time windows...

Thanks for your help ! :-)

David Waugh

May 11, 2016 4:44 AM

Correct Answer

Hi this is normal. I would recommend putting some suppression on your rule to limit the amount of alerts it will generate. For example

https://rsaportal.force.com/customer/articles/How_To/Example-Advanced-ESA-Rule-which-shows-suppression-on-multiple-variables-in-RSA-Security-Analytics

Create an Advanced Event Stream Analysis rule with the following content:

module Module_564eeceae4b06807f34ebd2d; 
@RSAPersist @Name('Module_564eeceae4b06807f34ebd2d_Alert') 
@Description('') @RSAAlert(oneInSeconds=0)  
SELECT  ip_src, ip_dstport, device_type, ip_dst FROM Event( 
/* Statement: CheckPoint */ (device_type IN ( 'checkpointfw1' ) 
AND ip_src is not null AND ip_dstport is not null AND ip_dst is not null)  )
.std:groupwin(ip_src,ip_dstport)
.win:time(3600 seconds)
.std:firstunique(ip_src,ip_dstport) retain-intersection

The rule does the following:

Group events into distinct ip_src and ip_dstport combinations.
Has a sliding window of 1 hour.
Only sends the first unique ip_src and ip_dst port combination.

The retain-intersection command ensures that only events matching ALL three of these criteria are forwarded.

See the reply in context

No one else had this question

Outcomes

David Waugh

May 11, 2016 4:44 AM

Hi this is normal. I would recommend putting some suppression on your rule to limit the amount of alerts it will generate. For example

https://rsaportal.force.com/customer/articles/How_To/Example-Advanced-ESA-Rule-which-shows-suppression-on-multiple-variables-in-RSA-Security-Analytics

Create an Advanced Event Stream Analysis rule with the following content:

module Module_564eeceae4b06807f34ebd2d; 
@RSAPersist @Name('Module_564eeceae4b06807f34ebd2d_Alert') 
@Description('') @RSAAlert(oneInSeconds=0)  
SELECT  ip_src, ip_dstport, device_type, ip_dst FROM Event( 
/* Statement: CheckPoint */ (device_type IN ( 'checkpointfw1' ) 
AND ip_src is not null AND ip_dstport is not null AND ip_dst is not null)  )
.std:groupwin(ip_src,ip_dstport)
.win:time(3600 seconds)
.std:firstunique(ip_src,ip_dstport) retain-intersection

The rule does the following:

Group events into distinct ip_src and ip_dstport combinations.
Has a sliding window of 1 hour.
Only sends the first unique ip_src and ip_dst port combination.

The retain-intersection command ensures that only events matching ALL three of these criteria are forwarded.

1 person found this helpful

Actions

Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ David Waugh on May 19, 2016 5:26 AM

Hello David,

Thanks for this feedback

I tried your solution and it looks like ok. I'm still doing some tests.

You say that this is a normal behaviour, could you explain me why?

Thanks !

Actions

David Waugh

@ Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= on May 19, 2016 5:34 AM

Lets take a look at your original rule and play some events through it:

@RSAAlert(oneInSeconds=0)

SELECT * FROM Event

(

device_type='snort'

AND ip_dstport=137

)

.std:groupwin(ip_src)

.std:unique(ip_dst)

.win:time(60 sec)

GROUP BY ip_src HAVING count(ip_dst) > 3;

Okay,

lets us have (time in seconds)

time=1 event1={device.type=snort,ip.dstport=137,ip.src=1.2.3.4,ip.dst=2.3.4.5}

time=2 event2={device.type=snort,ip.dstport=137,ip.src=1.2.3.4, ip.dst=2.3.4.5}

time =3 event3={device.type=snort,ip.dstport=137,ip.src=1.2.3.4, ip.dst=2.3.4.5}

so in general we at time n seconds, we get eventn.

(for simplicity assume one every second)

For a sliding window,

we get an alert at time =4,5,6,7,8 ...and every second onwards

For a batch window of size 60 seconds we get an alert at time =4,5,6 .. 60

Then at time 61, the batch window will get destroyed and all events dropped, so in effect we start again.

so will get alerts at 64,64,66....up to 120 and the process repeats.

Hope that clears it up.

1 person found this helpful

Actions

Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ David Waugh on May 19, 2016 5:49 AM

Yes perfect thanks for your explanation !

Actions

Nikolay Klender @ David Waugh on May 27, 2016 2:58 AM

David your rule may cause out of memory!

Beacuse events from Event stream will go to .std:groupwin(ip_src,ip_dstport) so all coombination of ip_src,ip_dstport will be in this group window!

In rules where std:groupwin is combined with time window on high amount of distinct values by wich events are grouped (ip_src,ip_dstport in your example) special hind must be used!

Chapter 13. EPL Reference: Views

The @Hint("reclaim_group_aged=age_in_seconds") hint instructs the engine to discard grouped data window state that has not been updated for age_in_seconds seconds. The optional @Hint("reclaim_group_freq=sweep_frequency_in_seconds") can be specified in addition to control the frequency at which the engine sweeps data window state. If the hint is not specified, the frequency defaults to the same value as age_in_seconds. Use the hints when your group criteria returns a changing or unlimited number of values. By default and without hints the view does not reclaim or remove data windows for group criteria values.

1 person found this helpful

Actions

Nikolay Klender May 27, 2016 2:58 AM

Hi!

First of all I should note that this rule may cause high memory usage, beacuse if you have lots of ip_src (for example you have snort which listen internet trafic) than .std:groupwin(ip_src) will hold all this information and after some time will cauase out of memory (please look at Chapter 13. EPL Reference: Views unfortunately RSA use such aproach in default rules without specail hint usage like this:

@Hint('reclaim_group_aged=10,reclaim_group_freq=30')

If you want alert supression use "output first every 1 hour" instruction. So please try following rule

@RSAAlert

SELECT window(*) FROM Event

(

device_type='snort'

AND ip_dstport=137

).win:time(60 sec)

GROUP BY ip_src

HAVING count(ip_dst) > 3

output first every 30 min;

Also I advice you to debug Esper rules with help of EsperTech Esper EPL Online

(do not paste @RSAAlert annotation becuase it is made by RSA and esper knows nothing about it)

1 person found this helpful

Actions

Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ Nikolay Klender on Jun 1, 2016 4:46 AM

Nikolay Klender : Does the argument select window(*) is mandatory ? What is the differrence with select * ?

If I don't use a HAVING COUNT in the alert, is it usefull ?

Actions

Nikolay Klender @ Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= on Jun 1, 2016 5:04 AM

window(*) will select all events which are in sliding window win:time(60 sec) so in alerts page you will see all events corresponding to alerts. Please made two types of rules with * and window(*) and you will see difference.

having - is operator which can be used to filter aggregated functions results, because you cannot use something like "where count(ip_dst)>3". I think it is better to google some sql examples to understood "having" usage because esper epl is very similar to sql.

1 person found this helpful

Actions

Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ Nikolay Klender on Jun 1, 2016 10:11 AM

OK understood ! :-) Thank you Nikolay !

Actions

Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= May 27, 2016 11:01 AM

David Waugh I made more tests with retain syntax and I had flood again...

Nikolay Klender Thanks for your comment ! I found the same solution using the pdf ESA_Rule_Development available on the RSA Knowledge Base https://rsaportal.force.com/customer/servlet/fileField?retURL=%2Fcustomer%2Farticles%2FHow_To%2FHelpful-information-for-developing-ESA-rules-in-RSA-Security-Analytics%3Fpopup%3Dfalse%26navBack%3DH4sIAAAAAAAAAIuuVipWslLyzssvz0lNSU_1yM9NVdJRygaKFSSmp4ZkluSA-KVAvn5yaXEJULpIP740Uz8bpkMfyEHRbl-cmliUnGGbWpyoVBsLANunma1hAAAA&entityId=ka170000000ThE4AAK&field=Attachment__Body__s

Anyway thanks for the advices ! :-)

@RSA Link Moderator : please change Corret Answer flag to Nikolay comment

Actions

10 Replies

David Waugh May 11, 2016 4:44 AM
Unmark Correct
Correct Answer
Hi this is normal. I would recommend putting some suppression on your rule to limit the amount of alerts it will generate. For example
https://rsaportal.force.com/customer/articles/How_To/Example-Advanced-ESA-Rule-which-shows-suppression-on-multiple-variables-in-RSA-Security-Analytics

Create an Advanced Event Stream Analysis rule with the following content:
```
module Module_564eeceae4b06807f34ebd2d; 
@RSAPersist @Name('Module_564eeceae4b06807f34ebd2d_Alert') 
@Description('') @RSAAlert(oneInSeconds=0)  
SELECT  ip_src, ip_dstport, device_type, ip_dst FROM Event( 
/* Statement: CheckPoint */ (device_type IN ( 'checkpointfw1' ) 
AND ip_src is not null AND ip_dstport is not null AND ip_dst is not null)  )
.std:groupwin(ip_src,ip_dstport)
.win:time(3600 seconds)
.std:firstunique(ip_src,ip_dstport) retain-intersection
```
The rule does the following:
1. Group events into distinct ip_src and ip_dstport combinations.
2. Has a sliding window of 1 hour.
3. Only sends the first unique ip_src and ip_dst port combination.
The retain-intersection command ensures that only events matching ALL three of these criteria are forwarded.
1 person found this helpful
Like • Show 0 Likes0
Actions
- Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ David Waugh on May 19, 2016 5:26 AM
  Mark Correct
  Correct Answer
  Hello David,
  
  Thanks for this feedback
  
  I tried your solution and it looks like ok. I'm still doing some tests.
  
  You say that this is a normal behaviour, could you explain me why?
  
  Thanks !
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= on May 19, 2016 5:34 AM
    Mark Correct
    Correct Answer
    Lets take a look at your original rule and play some events through it:
    
    @RSAAlert(oneInSeconds=0)
    SELECT * FROM Event
    (
    device_type='snort'
    AND ip_dstport=137
    )
    .std:groupwin(ip_src)
    .std:unique(ip_dst)
    .win:time(60 sec)
    GROUP BY ip_src HAVING count(ip_dst) > 3;
    
    Okay,
    lets us have (time in seconds)
    time=1 event1={device.type=snort,ip.dstport=137,ip.src=1.2.3.4,ip.dst=2.3.4.5}
    time=2 event2={device.type=snort,ip.dstport=137,ip.src=1.2.3.4, ip.dst=2.3.4.5}
    time =3 event3={device.type=snort,ip.dstport=137,ip.src=1.2.3.4, ip.dst=2.3.4.5}
    
    so in general we at time n seconds, we get eventn.
    (for simplicity assume one every second)
    
    For a sliding window,
    we get an alert at time =4,5,6,7,8 ...and every second onwards
    
    For a batch window of size 60 seconds we get an alert at time =4,5,6 .. 60
    Then at time 61, the batch window will get destroyed and all events dropped, so in effect we start again.
    so will get alerts at 64,64,66....up to 120 and the process repeats.
    
    Hope that clears it up.
    1 person found this helpful
    Like • Show 0 Likes0
    Actions
    - Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ David Waugh on May 19, 2016 5:49 AM
      Mark Correct
      Correct Answer
      Yes perfect thanks for your explanation !
      Like • Show 0 Likes0
      Actions
- Nikolay Klender @ David Waugh on May 27, 2016 2:58 AM
  Mark Correct
  Correct Answer
  David your rule may cause out of memory!
  Beacuse events from Event stream will go to .std:groupwin(ip_src,ip_dstport) so all coombination of ip_src,ip_dstport will be in this group window!
  In rules where std:groupwin is combined with time window on high amount of distinct values by wich events are grouped (ip_src,ip_dstport in your example) special hind must be used!
  Chapter 13. EPL Reference: Views
  
  The @Hint("reclaim_group_aged=age_in_seconds") hint instructs the engine to discard grouped data window state that has not been updated for age_in_seconds seconds. The optional @Hint("reclaim_group_freq=sweep_frequency_in_seconds") can be specified in addition to control the frequency at which the engine sweeps data window state. If the hint is not specified, the frequency defaults to the same value as age_in_seconds. Use the hints when your group criteria returns a changing or unlimited number of values. By default and without hints the view does not reclaim or remove data windows for group criteria values.
  1 person found this helpful
  Like • Show 1 Like1
  Actions
Nikolay Klender May 27, 2016 2:58 AM
Mark Correct
Correct Answer
Hi!
First of all I should note that this rule may cause high memory usage, beacuse if you have lots of ip_src (for example you have snort which listen internet trafic) than .std:groupwin(ip_src) will hold all this information and after some time will cauase out of memory (please look at Chapter 13. EPL Reference: Views unfortunately RSA use such aproach in default rules without specail hint usage like this:
```
@Hint('reclaim_group_aged=10,reclaim_group_freq=30')
```
If you want alert supression use "output first every 1 hour" instruction. So please try following rule

@RSAAlert
SELECT window(*) FROM Event
(
device_type='snort'
AND ip_dstport=137
).win:time(60 sec)
GROUP BY ip_src
HAVING count(ip_dst) > 3
output first every 30 min;

Also I advice you to debug Esper rules with help of EsperTech Esper EPL Online
(do not paste @RSAAlert annotation becuase it is made by RSA and esper knows nothing about it)
1 person found this helpful
Like • Show 1 Like1
Actions
- Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ Nikolay Klender on Jun 1, 2016 4:46 AM
  Mark Correct
  Correct Answer
  Nikolay Klender : Does the argument select window(*) is mandatory ? What is the differrence with select * ?
  
  If I don't use a HAVING COUNT in the alert, is it usefull ?
  Like • Show 0 Likes0
  Actions
  - Nikolay Klender @ Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= on Jun 1, 2016 5:04 AM
    Mark Correct
    Correct Answer
    window(*) will select all events which are in sliding window win:time(60 sec) so in alerts page you will see all events corresponding to alerts. Please made two types of rules with * and window(*) and you will see difference.
    
    having - is operator which can be used to filter aggregated functions results, because you cannot use something like "where count(ip_dst)>3". I think it is better to google some sql examples to understood "having" usage because esper epl is very similar to sql.
    1 person found this helpful
    Like • Show 0 Likes0
    Actions
    - Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= @ Nikolay Klender on Jun 1, 2016 10:11 AM
      Mark Correct
      Correct Answer
      OK understood ! :-) Thank you Nikolay !
      Like • Show 0 Likes0
      Actions
Bwy4GgAhxvlUflMJfjPdriPLUqjZDfCRNZju5x6WmBE= May 27, 2016 11:01 AM
Mark Correct
Correct Answer
David Waugh I made more tests with retain syntax and I had flood again...

Nikolay Klender Thanks for your comment ! I found the same solution using the pdf ESA_Rule_Development available on the RSA Knowledge Base https://rsaportal.force.com/customer/servlet/fileField?retURL=%2Fcustomer%2Farticles%2FHow_To%2FHelpful-information-for-developing-ESA-rules-in-RSA-Security-Analytics%3Fpopup%3Dfalse%26navBack%3DH4sIAAAAAAAAAIuuVipWslLyzssvz0lNSU_1yM9NVdJRygaKFSSmp4ZkluSA-KVAvn5yaXEJULpIP740Uz8bpkMfyEHRbl-cmliUnGGbWpyoVBsLANunma1hAAAA&entityId=ka170000000ThE4AAK&field=Attachment__Body__s

Anyway thanks for the advices ! :-)

@RSA Link Moderator : please change Corret Answer flag to Nikolay comment
Like • Show 0 Likes0
Actions

ESA alert is flooding when using sliding time window

Outcomes

Related Content

Recommended Content