In RSA NetWitness, when does it make sense to create a new index key instead of using an existing one to store more heterogeneous data?
My example use case is a key named 'hash'. If hashes from logs and other sources can be either MD5, SHA-1, SHA-256, etc., would it make more sense to create separate keys for each of them instead of using a single key called 'hash'? First of all, these hashes have different lengths so putting them all under a single key would make the size of the index bigger. Secondly, if I can extract feeds per hash type from external systems, wouldn't it make sense to only match MD5 hashes to MD5 hashes, SHA-256 hashes to SHA-256 hashes, and so on?
I plan on keeping the hash keys closed by default in the investigation view, so would there be any real and impactful downside of creating all these keys instead of just using a single key for all hashes?
Hello Tomi,
It's a balance between data organization and query/system performance...
When it comes to NW Indexing, uniqueness matters -- in that, if you have a variety of unique values (as you would expect with hashes), the high number of unique values in any given "index slice", could be overrun if your valueMax setting for that key is not set appropriately for your given environment.
I can certainly elaborate on the topic above as I'm not sure how familiar you are with the way indexing works in NetWitness and its relationship of unique values stored per slice for a given key in that slice. Please let me know if you need me to dive deeper on this topic.
Given the information you have provided, it would make sense to make separate keys for each subset of data i.e. hash.md5, hash.sha256 etc.. to avoid index overruns and/or lowering the index save frequency to accommodate the high number of unique values stored in those keys. Particularly, as you mentioned that you are doing matching against some feed content, it could certainly help keep down the query times as you use this data for reporting, charts etc...