Is there any documentation on the default RSA Meta keys/fields and what they mean? I think it would be pretty beneficial for RSA to provide this to the end users to clarify what they can search for within the product.
Not only is there no documentation, there's often inconsistency in how these fields are used.
For example, user.dst is normally the user on which the action is performed (account logged into, account manipulated), but it's also used for the user initiating an action by the bluecoat parser. Similarly firewall parsers often populate ip.srcport and ip.dstport but the network packet parsers populate udp.dstport/tcp.dstport or the srcport equivalents. Similarly there's no resolution from name to IP so you're stuck with whatever happened to be in the logs with little or no enrichment.
The other thing that's missing is clear documentation of the log taxonomy model.
After 18 months with SA, and a couple of years with EnVision beforethat, I generally have a pretty good idea of what goes where, but the inconsistencies in the standard parsing drive my users crazy.
If you've dug down to the level of writing your own parsers, then it's worth spending some time looking at table-map.xml and index-concentrator.xml. We've made extensive modifications to these (via the -custom.xml) files to ensure the right metadata is stored and indexed in the way we need it. I'm not sure many organisations could justify the amount of time we've spent on it, though.
Everything Andy said is correct. I do not use any of the default parsers anymore. I will normally do a quarterly merge of any updates RSA has implemented but changing where parsers go helps a lot in making everything normalized.
Thanks guy, would any of you happen to have documentation on how to write log parsers? I have found little to no information how to write log parsers, let alone even read them.
I'm not sure if this fits the bill, but The specified item was not found. posted this last month: Parsers Book.zip
Thanks, It appears that this documentation only covers packet parsers.
you can use the event source integrator (ESI Tool), that's used for envision.to create custom parsers. and the install the parser into the log decoder (there are some posts on this) you can check the Security Analytics parser so you can have an idea on how to do it. Just a heads up... it's not that easy
Retrieving data ...