RSA Admin

UNIQUE LOG FILE NAMES

Discussion created by RSA Admin Employee on Sep 8, 2011

The enVision script that captures files and sends them to the collecter's file reader service relies on pattern matching to decide if a file needs to be monitored.

But what if an application creates a new file every day, like FILE_MMDD.LOG and the app is not amicable to archiving them into a new directory or new name?

Well, if they keep a year’s worth on line, envision will keep pointers for all 365 files and will dutifully check them every time the script is called.

This can cause performance issues ( In our case, the apps kept over 20K files!! Heck, disc space is cheap) and it’s just inefficient to check 364 files that will never change just to get to the one that is current.

In order to adjust the script to handle this type of situation, there are 2 things that need to be done.

You must tell the portion of the script that checks the files to ignore files that aren’t current

You must tell the cleanup portion of the code to remove any .time and .line files for any non current logfiles.

I’ve made these adjustments and the collection is working perfectly.  I’ll share the start of the solution and if there is interest, I’ll add the next steps to completion.

Step 1 – you need to define the criteria.  In the case above, FILE_MMDD.LOG, if I use a FILESPEC=FILE_*.LOG, I’ll get every file matching that variable, that is, all 365 files.  What I really want is just today’s file and yesterday’s file.  I want yesterday’s file because we can’t count on cron and the logging to be working perfectly in sync.  So we eat the little performance hit and gather files for today and yesterday.  I did this via a function.

function DateSpan {

  # create array of MMDD dates from today (0) until number of days sent ($1)

  for (( i=0; i<=$1; i++ ))

  do

    array[$i]=`date --date "$dte -$i days" +%m%d`;

  done

}

 

The function is called with the number of days past today that you wish to keep the files.  That is, 0 means just keep today’s file, 1 means today and yesterday’s, etc.

So making the call

DateSpan 1

Will create an array with two elements one with today’s month and date, 0908 and yesterday’s 0907

 

The function should be put at the top of the script.  Ok, if there is interest, I’ll keep going.

Outcomes