Security Analytics Core Database Tuning Guide
I have attached a recently created guide for tuning the SA Core 10.3 databases ([Log/Packet] Decoder, Concentrator, Archiver). This guide will eventually be part of the online documentation (Home - RSA Security Analytics Documentation).
If you have any questions or find errors in the documentation, please let me know.
- Community Thread
- Forum Thread
- RSA NetWitness
- RSA NetWitness Platform
Thank you for sharing; such a great document.
However, I have two follow-up questions and I hope someone has the knowledge to answer them:
1. Early last year I was taught that custom meta keys should be entered into both index-concentrator-custom.xml and index-decoder-custom.xml with the slight difference that the latter has its keys set to IndexNone. At some point this changed so that we were told that it was no longer necessary to make the changes to index-decoder-custom.xml. Similarly the SCOL KB teaches that one should manipulate the index-concentrator-custom.xml and table-map-custom.xml but not the index-decoder-custom.xml. And for the most part this seems to working great: custom meta keys are visible when investigating and so on. But what I cannot figure out is that how does the decoder know of the custom meta keys created without manipulating index-decoder-custom.xml? For example, if one creates application rules or feeds he/she needs to reference to those meta keys when creating the rules or mapping feed source files to meta keys. For most of the meta keys (defaults) the text auto-complete works (the GUI looks up meta keys that contain the substring you just wrote) but usually not for the custom ones. You can still type the custom meta keys into the fields and for example in the case of an application rule hit the "Validate Rule" button and it says it will validate.
For clarification it needs to be said that we are running a log hybrid appliance.
2. How to make concrete measurements about performance gains or losses when regarding the index slice slices? This document makes references to "light workloads" but this can be seen as subjective. How to empirically make sure that it would benefit us to either increase or decrease the index save interval away from the default of 8 hours?
Tom-- I'm glad you enjoyed my document. 🙂
To answer your questions--
The decoder is very flexible about what meta keys parsers are allowed to create. A parser is allowed to populate any meta key it likes, even if it's not in the decoder's custom index definition. The decoder keeps track of the meta key names generated by the parsers while capture is running, and uses that to provide the list of meta keys. As you have found, the auto-complete functionality is probably the only area where you might find some benefit from explicitly defining meta keys in the decoder's index configuration. As you can see, if a parser hasn't generated the meta key, it won't be able to provide the meta key to the auto-complete functionality.
Regarding the number of index slices-- this is a good question. The answer is that it depends on how much retention your index has. For example, on a concentrator the index has a fixed size and automatically rolls over after a period of time. A lightly loaded index will take a long time to roll over (maybe months or even years) while a heavily loaded index will rollover in a matter of weeks or days. If your index is heavily loaded and rolls out quickly, then you will see that the number of slices in the index always stays low; perhaps 100 or 200. On a lightly loaded index the number of slices may be in the thousands. For Concentrator 10.3, my general rule of thumb would be to try to keep the number of index slices below 500.