RSA Admin

Using LastWriteTime to Detect Devices Not Sending Logs

Discussion created by RSA Admin Employee on Oct 3, 2008
Latest reply on Mar 27, 2013 by RSA Admin

I noticed several posts across the forum asking for ways to detect on devices not sending logs.  I have bit different way of reporting on this, that might be of use to others.  Albeit a cluge, and I admit this is a very VERY temporary fix till something more offical comes out in the product...  it works suprisingly well enough, for now.



Consider this...  all events that come into envision are processed by the collector and dropped where?   The D:\tmp\nuggets directory, right?   I really like how the subbordinate folder structure is layed out within D:\tmp\nuggets and how the collector service makes use of it, well, for three primary reasons.

1. I like how it's well organized, first by known device type (ciscorouter, rhlinux, winevent_nic, etc) then by IP address of the source device.  So with a quick glance you can easily tell if you've ever collected anything of a certain device type (to include Unknown), and also what IP address it had at the time.  Now that doesn't necessarily mean you're collecting logs from that IP right now this minute... but the fact that the folder exists means that you've gotten logs from it at some point.

2.  I like it because in the grand scheme of things it's somewhat "upstream" of the IPDB still, and the whole mess of other processes that take place on or with that log data after it leaves the nuggets dir.

3.  It's updated very very quickly.  Not only is the log data put in there quickly by the collector service, it's also moved on down the line (typically within 60 seconds) by the packager I believe.  The key point here is that things are coming and going in-and out of this directory structure very very quickly.



It dawned on me one day that this is would be the ideal choke point to collect some metric type info from.  Read-only of course.  I too wanted to know if we had stopped receiving log data from a device.  And yes, I'm aware of the NIC System 508100 event, but without the ability to report on sum(count) off the NIC SYSTEM table, it's quite frankly useless to me.   Which led me to write the attached powershell script ReportNuggetsTimestamp.ps1.  Which basically reports on the last modified timestamp of these directories to help identify what IP's are and are not sending logs.   There is no help file, so see below for how it works.  There are also a lot of assumptions... like enVision might change how their platform works or how the nuggets dir is layed out, etc, etc, but beyond all that, it works well today.



In just one long line of code, here's what it does...
-Walks down the D:\tmp\nuggets directory an recursively reads the contents.  Same thing as you doing a [ Dir /A:smileyvery-happy: /S ] but not as messy.  Just reads.. doesn't do anything to your data.
-Filters on only Directories.  This way all we get back are foldernames like "111.222.333.444" and not .tmp or .nug files.
-Also Filters specifically on the directory name, it's parent, and the date/time stamp of when it was last modified (aka LastWriteTime) which is the key to it all.
-Then it does a little cleanup, ingnores folders who's parent is 'nuggets', because I don't care when the device type folders we're last modified, only the subordinate IP address folders themselves.
-Then lastly sorts all the entries by LastWriteTime and spits them out on screen, so that the oldest directories are at the top for you to review.

Results look something like this when i ran it at 4:09pm today The list is truncated and IPs obfuscated, it appears the forum chewed up the alignment a bit, but you get the idea:

Parent                          Name                                LastWriteTime
------                          ----                                  -------------

hpux                                     4/16/2008 6:18:00 PM
winevent_nic                           5/15/2008 8:45:00 AM
solaris                                      5/28/2008 3:05:26 PM
ciscorouter                            6/4/2008 4:02:00 AM
checkpointfw1                           6/7/2008 1:44:00 AM
checkpointfw1                        10/3/2008 4:09:04 PM
solaris                                   10/3/2008 4:09:04 PM
winevent_nic                          10/3/2008 4:09:04 PM
rhlinux                                    10/3/2008 4:09:04 PM
winevent_nic                           10/3/2008 4:09:04 PM
winevent_nic                           10/3/2008 4:09:04 PM
checkpointfw1                           10/3/2008 4:09:04 PM
netapp                                    10/3/2008 4:09:04 PM
netapp                                    10/3/2008 4:09:04 PM
netapp                                    10/3/2008 4:09:04 PM
checkpointfw1                          10/3/2008 4:09:04 PM
checkpointfw1                        10/3/2008 4:09:04 PM
rhlinux                                   10/3/2008 4:09:04 PM
unknown                                   10/3/2008 4:09:07 PM
winevent_nic                          10/3/2008 4:09:08 PM
winevent_nic                           10/3/2008 4:09:09 PM
winevent_nic                           10/3/2008 4:09:12 PM
unknown                               10/3/2008 4:09:13 PM
unknown                               10/3/2008 4:09:19 PM

At first glance you can tell real quickly what IP's I've processed logs from and which one's I haven't.   Because this is done from command line you can then pipe it into all your other favorite tools, like maybe findstr, to key in on the windows boxes, like this:


PS C:\ReportNuggetsTimestamp.ps1 | Findstr "winevent_nic"


winevent_nic                          5/15/2008 8:45:00 AM
winevent_nic                        10/3/2008 4:09:04 PM
winevent_nic                          10/3/2008 4:09:04 PM
winevent_nic                          10/3/2008 4:09:04 PM
winevent_nic                        10/3/2008 4:09:08 PM
winevent_nic                          10/3/2008 4:09:09 PM
winevent_nic                          10/3/2008 4:09:12 PM



To schedule it on the appliance use a cmd file to call it out...
@ECHO off
c:\powershell.exe c:\(path_to_script)\reportnuggetstimestamp.ps1 > c:\(path_to_wherever)\outputfile.txt



I realize there are tons of other things that can be done with powershell, and I only scratched the surface. At this point, I'm just running it manually as needed. A lot of other forum posters want to report on devices that haven't logged in the past X minutes or hours, etc. Or have it up on the dashboard.  I think you'd find with a little effort you could do this with the new-timespan cmdlet, and just compare $date to LastWriteTime.   Please understand that I got it to the point where it served my immediate needs... and I stopped.   If you want to expand on this and add your own functionality, go for it.  You can run this manually or schedule it, although you'd probably want to output it to somewhere... like maybe a txt file, maybe a text file that gets pulled in by file reader... or output it to an email or whatever...     I had half a thought to get it read back into envision so I can alert, report, correlate, and paint in on my dashboard, like some forum posters before me have wanted to do.  But alas... I do not work for RSA enVision... I bought enVision so it would work for me.  There's also been roadmap promises of better device management, and tracking, so with that in mind I'll see what pans out.



More importantly... Just knowing I have powershell at my disposal now opens the door to a whole LOT of other stuff that we can do, and share here on the forum.


<Disclaimer>...   MS Powershell 1.0 x64 for server 2003 installed just fine.  don't forget .NET 2.0 Framework x64 and it's service pack 1.  I stopped all NIC services just to be safe.  Oh yes, almost forgot... RSA enVision Support (and myself included) will probably tell you installing powershell, the .NET 2.0 Framework, running this script, and probably everything in this forum post, along with most things that come out of my mouth on a friday night, are entirely unsupported configurations and in most cases not recommended.  Use at your own risk, etc, etc. </disclaimer> 

Message Edited by Ryan on10-07-200810:42 AM