I have attached a recently created guide for tuning the SA Core 10.3 databases ([Log/Packet] Decoder, Concentrator, Archiver). This guide will eventually be part of the online documentation (Home - RSA Security Analytics Documentation).
If you have any questions or find errors in the documentation, please let me know.
Thanks,
Scott
Thank you for the documentation it is really what we are looking for.
Thank you for sharing; such a great document.
However, I have two follow-up questions and I hope someone has the knowledge to answer them:
1. Early last year I was taught that custom meta keys should be entered into both index-concentrator-custom.xml and index-decoder-custom.xml with the slight difference that the latter has its keys set to IndexNone. At some point this changed so that we were told that it was no longer necessary to make the changes to index-decoder-custom.xml. Similarly the SCOL KB teaches that one should manipulate the index-concentrator-custom.xml and table-map-custom.xml but not the index-decoder-custom.xml. And for the most part this seems to working great: custom meta keys are visible when investigating and so on. But what I cannot figure out is that how does the decoder know of the custom meta keys created without manipulating index-decoder-custom.xml? For example, if one creates application rules or feeds he/she needs to reference to those meta keys when creating the rules or mapping feed source files to meta keys. For most of the meta keys (defaults) the text auto-complete works (the GUI looks up meta keys that contain the substring you just wrote) but usually not for the custom ones. You can still type the custom meta keys into the fields and for example in the case of an application rule hit the "Validate Rule" button and it says it will validate.
For clarification it needs to be said that we are running a log hybrid appliance.
2. How to make concrete measurements about performance gains or losses when regarding the index slice slices? This document makes references to "light workloads" but this can be seen as subjective. How to empirically make sure that it would benefit us to either increase or decrease the index save interval away from the default of 8 hours?
Tom-- I'm glad you enjoyed my document. :-)
To answer your questions--
The decoder is very flexible about what meta keys parsers are allowed to create. A parser is allowed to populate any meta key it likes, even if it's not in the decoder's custom index definition. The decoder keeps track of the meta key names generated by the parsers while capture is running, and uses that to provide the list of meta keys. As you have found, the auto-complete functionality is probably the only area where you might find some benefit from explicitly defining meta keys in the decoder's index configuration. As you can see, if a parser hasn't generated the meta key, it won't be able to provide the meta key to the auto-complete functionality.
Regarding the number of index slices-- this is a good question. The answer is that it depends on how much retention your index has. For example, on a concentrator the index has a fixed size and automatically rolls over after a period of time. A lightly loaded index will take a long time to roll over (maybe months or even years) while a heavily loaded index will rollover in a matter of weeks or days. If your index is heavily loaded and rolls out quickly, then you will see that the number of slices in the index always stays low; perhaps 100 or 200. On a lightly loaded index the number of slices may be in the thousands. For Concentrator 10.3, my general rule of thumb would be to try to keep the number of index slices below 500.
Thanks Buddy ....
Really a good documentation for understanding the Database.
Thanks for sharing this doc, it is really helpful..
FYI, the Database Tuning Guide has been greatly expanded for 10.4 and is now posted online here: Security Analytics Core Database Introduction - RSA Security Analytics Documentation
Scott
Thanks Scott!
Very nice guide.
The link you posted to the 10.4 Database guide is sort of messed up.
I want to download the entire guide, not have to page through 2 page chunks to get
to what I need.
Don
You can download a PDF of the whole guide by going to the root page and pressing the PDF icon in the right column.
Right above the words "Support and Service".
Hi,
I have a test environment with a Log Decoder, Concentrator and SA Server, and recently I configured an Archiver and I would like to test it.
As my test environment does not have much traffic, is there any way to force Log Decoder forward the logs to Archiver before it reaches his threshold??
Regards.
The only way to get logs into Archiver is by aggregation. You need to add the LD to the Archiver as an aggregation device and then start aggregation.
Thanks for your attention Scott.
After starting the aggregation, all logs that arrive in Log Decoder are forwarded to Archiver? Or these logs are forwarded after Log Decoder reaches his threshold?
Regards.
Yes, unless you add an aggregation filter. I'm not sure what you mean by "Log Decoder reaches his threshold"? What threshold?
By default, Archivers are configured to perform "nice" aggregation, which is meant to alleviate stress on the Log Decoder, especially if multiple devices are aggregating from it. What this means is it will fall behind a bit to alleviate write contention on the LD.
If your Log Decoder is well under 30K EPS, then you can get logs in near real time by turning nice off (/archiver/config/aggregate.nice=false).
Does that answer your question or am I missing something?
So, added the Log Decoder to Aggregation in Archiver and it shows as online, but I dont see any Logs when I choose Archiver in Investigation Tab.
I can see the Logs just when I choose Log Decoder in Investigation Tab.
I appreciate your attention.
Regards.
Did you start aggregation on the Archiver? Try monitoring the /database/stats on the archiver and make sure the packet.total stat is increasing. If it's not, but the session stat is increasing, that means it is aggregating, but the current session on the LD does not have a log. In other words, the logs are in rollover on the LD.
Once Archiver catches up to the sessions that have logs, everything should start working.
If this isn't the problem and it is aggregating, then you will need to look at the logs on the Archiver to determine what the problem is. Open a support ticket in that case.
on Jul 19, 2014
great, thanks for sharing.