AnsweredAssumed Answered

Decoder stopping with disk space issues

Question asked by RSA Admin

on Sep 1, 2015
Latest reply on Sep 9, 2015 by Jesse Carleton

Like • Show 1 Like1
Comment • 6

We're having problems with RSA Security Analytics stopping decoders with full disk space. First, a decoder stopped due lack of space on a partition with dumps generated by errors.

Today, a log decoder stopped with full /var/log due several messages going to /var/log/messages:

Sep 1 19:30:17 sa105sas01 ...ere='time=\"2015-08-12 08:10:00\"-\"2015-08-12 08:50:59\"', options=InvestigationOptions{options={date_range=com.netwitness.platform.server.common.domain.model.DateRange@25fca493, total_by=SESSION_COUNT, order_by=TOTAL, time_range_type=CUSTOM, sort_order=DESCENDING}, dateRange=com.netwitness.platform.server.common.domain.model.DateRange@25fca493, orderBy=TOTAL, sortOrder=DESCENDING, timeRangeType=CUSTOM, totalBy=SESSION_COUNT}, metaAliases={}, aggregateFunction='null', aggregateFieldName='null', min=null, max=null}","severity":6,"userRole":"Administrators+Administrators+PRIVILEGED_CONNECTION_AUTHORITY"}

Sep 1 19:30:17 sa105sas01 ...ere='time=\"2015-08-12 08:10:00\"-\"2015-08-12 08:50:59\"', options=InvestigationOptions{options={date_range=com.netwitness.platform.server.common.domain.model.DateRange@66356e87, total_by=SESSION_COUNT, order_by=TOTAL, time_range_type=CUSTOM, sort_order=DESCENDING}, dateRange=com.netwitness.platform.server.common.domain.model.DateRange@66356e87, orderBy=TOTAL, sortOrder=DESCENDING, timeRangeType=CUSTOM, totalBy=SESSION_COUNT}, metaAliases={}, aggregateFunction='null', aggregateFieldName='null', min=null, max=null}","severity":6,"userRole":"Administrators+Administrators+PRIVILEGED_CONNECTION_AUTHORITY"}

Sep 1 19:30:29 sa105sas01 ...='time=\"2015-08-12 08:10:00\"-\"2015-08-12 08:50:59\"', options=InvestigationOptions{options={date_range=com.netwitness.platform.server.common.domain.model.DateRange@7348515b, total_by=SESSION_COUNT, order_by=TOTAL, time_range_type=CUSTOM, sort_order=DESCENDING}, dateRange=com.netwitness.platform.server.common.domain.model.DateRange@7348515b, orderBy=TOTAL, sortOrder=DESCENDING, timeRangeType=CUSTOM, totalBy=SESSION_COUNT}, metaAliases={}, aggregateFunction='null', aggregateFieldName='null', min=null, max=null}","severity":6,"userRole":"Administrators+Administrators+PRIVILEGED_CONNECTION_AUTHORITY"}

Is there a way to implement a fix or some configuration to avoid these issues?

Regards

No one else has this question

Outcomes

Jesse Carleton Sep 2, 2015 6:48 PM

We experienced a similar issue to this. Our metaDB was filling up past 95% disk space and wasn't able to roll the logs quickly enough, resulting in the service crashing.

What's your disk usage like? In the CLI, do "df -lah" to see your usage.

Actions

RSA Admin

@ Jesse Carleton on Sep 3, 2015 6:50 PM

We got 100% of use in /var/log recently due several messages on messages and the rotation did not work fast enough.

Did you contacted support or just remove content? We accessed ssh and removed old files.

Fernando José Karl

AMBCI, CISSP, CISM, MBA, MSc

Defenda Business Protection Services & Solutions

Phone: +55 51 3091 3337 / +1 415 656 8337 / +55 51 8052 8034

Site: www.defenda.com.br

Em 03/09/2015, às 03:51, moar_powah! <emc-community-network@emc.com> escreveu:

ECN

Decoder stopping with disk space issues
reply from moar_powah! in RSA Security Analytics - View the full discussion

We experienced a similar issue to this. Our metaDB was filling up past 95% disk space and wasn't able to roll the logs quickly enough, resulting in the service crashing.

What's your disk usage like? In the CLI, do "df -lah" to see your usage.

Reply to this message by replying to this email, or go to the message on ECN
Start a new discussion in RSA Security Analytics by email or at ECN
Following Decoder stopping with disk space issues in these streams: Inbox

Actions

Md Salleh Mohd Nur @ RSA Admin on Sep 9, 2015 3:52 AM

is your decoder generating the core dump files? if you remove the core dump decoder will start again. but you need to contact support to check why core dump created.

Actions

Jesse Carleton @ RSA Admin on Sep 9, 2015 6:28 PM

We applied a fix to our partitions with support. They noted there was a bug in the AIO appliance, fixed little over a year ago. The fix involved running a script in CLI, which backed up the data, resized partitions, and restored it once done. We no longer have the issue any longer since applying this fix.

Actions

Md Salleh Mohd Nur Sep 14, 2015 2:38 AM

could you share the script? would like to know about this if in future installing AIO. last time on june i install AIO but didn't saw this issue.

Actions

Jesse Carleton @ Md Salleh Mohd Nur on Sep 14, 2015 2:57 PM

The issue isn't present on current releases of AIO boxes. It only happened on the first batch of boxes with version 10.

I didn't keep the script, as we didn't need to run it again. You should be able to find it on SCOL.

Actions

6 Replies

Jesse Carleton Sep 2, 2015 6:48 PM
Mark Correct
Correct Answer
We experienced a similar issue to this. Our metaDB was filling up past 95% disk space and wasn't able to roll the logs quickly enough, resulting in the service crashing.

What's your disk usage like? In the CLI, do "df -lah" to see your usage.
Like • Show 0 Likes0
Actions
- RSA Admin @ Jesse Carleton on Sep 3, 2015 6:50 PM
  Mark Correct
  Correct Answer
  We got 100% of use in /var/log recently due several messages on messages and the rotation did not work fast enough.
  
  Did you contacted support or just remove content? We accessed ssh and removed old files.
  
  --
  Fernando José Karl
  AMBCI, CISSP, CISM, MBA, MSc
  
  Defenda Business Protection Services & Solutions
  Phone: +55 51 3091 3337 / +1 415 656 8337 / +55 51 8052 8034
  Site: www.defenda.com.br
  
  Em 03/09/2015, às 03:51, moar_powah! <emc-community-network@emc.com> escreveu:
  
  ECN
  
  Decoder stopping with disk space issues
  reply from moar_powah! in RSA Security Analytics - View the full discussion
  
  We experienced a similar issue to this. Our metaDB was filling up past 95% disk space and wasn't able to roll the logs quickly enough, resulting in the service crashing.
  
  What's your disk usage like? In the CLI, do "df -lah" to see your usage.
  
  Reply to this message by replying to this email, or go to the message on ECN
  Start a new discussion in RSA Security Analytics by email or at ECN
  Following Decoder stopping with disk space issues in these streams: Inbox
  Like • Show 0 Likes0
  Actions
  - Md Salleh Mohd Nur @ RSA Admin on Sep 9, 2015 3:52 AM
    Mark Correct
    Correct Answer
    is your decoder generating the core dump files? if you remove the core dump decoder will start again. but you need to contact support to check why core dump created.
    Like • Show 0 Likes0
    Actions
  - Jesse Carleton @ RSA Admin on Sep 9, 2015 6:28 PM
    Mark Correct
    Correct Answer
    We applied a fix to our partitions with support. They noted there was a bug in the AIO appliance, fixed little over a year ago. The fix involved running a script in CLI, which backed up the data, resized partitions, and restored it once done. We no longer have the issue any longer since applying this fix.
    Like • Show 0 Likes0
    Actions
Md Salleh Mohd Nur Sep 14, 2015 2:38 AM
Mark Correct
Correct Answer
could you share the script? would like to know about this if in future installing AIO. last time on june i install AIO but didn't saw this issue.
Like • Show 0 Likes0
Actions
- Jesse Carleton @ Md Salleh Mohd Nur on Sep 14, 2015 2:57 PM
  Mark Correct
  Correct Answer
  The issue isn't present on current releases of AIO boxes. It only happened on the first batch of boxes with version 10.
  
  I didn't keep the script, as we didn't need to run it again. You should be able to find it on SCOL.
  Like • Show 0 Likes0
  Actions

Decoder stopping with disk space issues

Outcomes

Related Content

Recommended Content