I am attempting to use custom feeds but they just seem really slow to search on. Due to the nature of our company we use almost every 10.1-255.0.0 cidr range. The feed is designed just to segment out each range so I can search on it. But when I attempt to do a search on the feed or just view the meta of feed.name, it takes more than ten minutes to load an hour. Anyone else seeing this?
i think it depends how much data you viewing. check the query finished time.
What you have described sounds like an index problem during investigation rather than the feeds themslevs.
Check the index settings for the meta key that you are writing the data to. Any changes to index settings, and any custom meta keys you have created, should be in the index-concentrator-custom.xml on the concentrator. Make sure it is set to level="IndexValues" and give it an appropriate ValueMax setting. Here is an example of what the entry should look like.
<key description="Threat Source" format="Text" level="IndexValues" valueMax="250000" name="threat.source"/>
Do you know what the difference is between indexValues and indexKeys? Just curious, by switching it to indexValues it did work.
One of the RSA guys may be able to provide a more technically accurate answer, but in general IndexValues actually indexes the unique values in each meta key/table. This means you can search for specific values and leverage the benefit of the index. At a really abstracted level, an index contains pointers to where the different unique values are, so when you search for one it can find it in the index which is generally much smaller than the whole table.
IndexKeys doesn't index the individual values, but only whether a value exists for that key. So you can do "exists" comparisons efficiently (like "alias.host exists"), but hunting for specific values still requires iterating over every record in the table and comparing against each one, which is very slow.
Indexing each individual value is more resource intensive to create and maintain as the database engine has more work to do at consumption time because it has to update the index ... but the indexing allows you to search through data faster after the fact. So it's a trade-off... if you try to set IndexValues for everything, you may have performance problems depending on how much data you're consuming.
I actually got a response from RSA support on this. The article is for 9.8 and lower so it might be different for SA but I think it helps understand them more.
https://knowledge.rsasecurity.com/scolcms/knowledge.aspx#a59809
The main difference between IndexValues and IndexKeys is as follows:
Hence there is space saving in the index and some increased performance for initial drills if you set a key to IndexKeys over using IndexValues.
Where you will see the most difference is in Investigator where the keys with IndexKeys will always come up in a closed state regardless if there are values or not.
Patriot, if set to indexvalues, for action, everything in blue is indexed. This means that queries that specify something in the action key will return very quickly. For instance action=put
If that meta key was only indexkey, any queries for matches will take a while to return, but the EXISTENCE of the key is what will return quickly. For instance, service=80 && action EXISTS
So why the distinction?
There are tons of meta keys that just don't need to be queried by an analyst. For instance, with the GEOIP integration, most analysts don't care about latitude and longitude, or city, but those values are still created. In most instances, the SQL key is not indexed, but if I had SA near to my SQL servers, I'd want to turn indexing on to capture and alert against certain SQL commands and queries.
The reason behind these settings is that indexing is rarely turned on for everything. The index file takes up quite a bit of room on the disk, and in fact, many concentrators have fast reading solid state drives that only host the index, and if the index grows beyond that SSD's capacity, it will ruin the performance gain of having the SSD in the first place.
Of course, technology improves and one day this distinction might not be necessary. Everything will eventually be able to be indexed.
But for now, make sure your indexing supports your use cases. Need to analyze malware and hunt for malicious known filenames? Gotta index filenames. Need to be on the lookout for known malicious referral strings? Gotta index referral.
Thanks guys.
So if I understand correctly:
IndexValues means index the blue color values , which make query look for the green color values faster?
To use your example – IndexValues means index the blue colour values which makes queries for the blue colour values faster.
That means when we click the green number, the concentrator ask decoder to load the actual data to display?
Where the green number came from then?
Thanks.
That is simply the number of times that particular meta value has been seen. They are distinct sessions.
So that means it's in the index?
not exactly. When we talk about indexing, it is meta only. The sessions in green are tagged with meta, but those sessions are tracked on the decoders by sessionID numbers.
The decoder captures traffic.
It creates the session in memory.
It creates meta tags according to its parsers for each session, and links its session ID to those meta values
It dumps those meta values locally to await pickup by the concentrator.
The decoder then writes the session to disk for safekeeping until it is needed later.
Concentrator logs into decoder and asks if it has new meta.
Concentrator pulls meta values with linked session IDs to store in tables on the concentrator
If the table key is indexKeys, it stores those values in the key without indexing the values.
If the table key is IndexValues, it creates a searchable index for each of the values within that key up to its ValueMax setting.
If the ValueMax setting for a key becomes full, the concentrator drops the least used 10% of values out of its index to make room for more values.
Then wash, rinse and repeat.
The way I see it, fielder can comment if I am wrong.
You are writing a rule to look for an action from an IP source.
Scenario one action is using indexkeys and ip.src is using indexvalues
ip.src = '1.1.1.1' && action = 'get'
because action is only indexkeys it is going to go "Okay, let me get the ip.src and any log/packet that has action attached to it" It does not care that you want it to equal 'get'
Scenario two action is indexvalues and ip.src is indexvalues
ip.src = '1.1.1.1' && action = 'get'
because the action is now index values it looks for get and the ip.src and only pulls those values.
Using index keys can be very help if you really just want it to be useful information for an investigation but you don't really care about querying on that key.
So maybe you want to label your 1.1.1.0/24 network as Critical Web Servers
You might not want to query on that ever but during an investigation if you see that you are dealing with a critical web server it can be helpful but you are not wasting your index for querying purposes.
So Sean brings up a good point- querying for values some of which are indexed and others are not.
In Sean's scenario, he is querying IP address first, which is indexed, followed by an action that is not.
ip.src=1.1.1.1 && action = get. Both values get queried, but the IP goes first, followed by the action. Since IP is indexed, the first portion of the query returns quickly, and the action of get takes a little longer, but the initial drill into the IP narrows down the search parameters quite a bit.
Now let's look at the obverse:
Action=get && ip.src=1.1.1.1
The first query will be run which will take a very long time since it is not indexed. Once you get the first part of the result, the action, then the IP gets queried.
So whenever you craft your drills and queries, be sure to always use indexed items up front and only use non-indexed values at the very last. If it becomes a common query you need to run often, you need to look at indexing the key's values to get results faster.
thanks guys for all the helpful reply. Now i understand what does indexkeys and indexvalues means.
We had an issue on the filename meta, it take long time load, below is the benchmark.
Last 24 hours data – click Show more
SA UI: 2 minutes and 25 seconds
Nw Investigator: 2 minutes and 30 seconds.
I tried change from indexkeys to indexvalues, then reset the index, but the performance(display in SA UI) is similar.
Wondering is there anything we can do to make it fast?
I understand the filename is volatile, only indexkeys will be recommended, but why so slow?
Thanks.
Check the index level of the feed.name key. It's probably set to IndexKeys, which means accessing or searching it for specific values on that key will be slow. If this is something you will be using heavily or doing frequently, you might consider changing the indexing level of this key to IndexValues instead..
Something I also use are custom meta keys for my own network ranges. So, I create a key called netname.src and netname.dst and then slice up the csv with cidr notation and some meta. I try to keep the meta value small to save space on the database.
Ex.
192.168.1.1/24,internal
10.1.1.1/24,dmz
172.16.0.1/24,vpn
Then, I create some app rules around this to figure out direction to either include or exclude. Furthermore, you could create app rules to fire when certain segments are observed communicating with each other.
It does sound like an indexing problem, but I think there may be some other ways around getting the data you want to get at.
Chris
What you have described sounds like an index problem during investigation rather than the feeds themslevs.
Check the index settings for the meta key that you are writing the data to. Any changes to index settings, and any custom meta keys you have created, should be in the index-concentrator-custom.xml on the concentrator. Make sure it is set to level="IndexValues" and give it an appropriate ValueMax setting. Here is an example of what the entry should look like.
<key description="Threat Source" format="Text" level="IndexValues" valueMax="250000" name="threat.source"/>