question on indexkeys and index values again

[huanzhou1]

Hi Guys,

I have few questions on the index xml file again:

1. if now i change from indexkeys to indexvalues, will it apply to the existing index files? Or only the future index files?

Need to reset index to recreate the index files? Last time i did reset, for 1 DAC raw packets, it took a few hours to finish re-indexing.

2. Is there any benchmark available for the indexkeys or indexvalues?

The day i did some testing, for filenames, indexkeys or indexvalues got the same query time.

3. Today in one customer, the customer has around 8TB data per decoder, total 4 decoders, that means around 30TB per day. Customer complaint query sloness, I did some testing, for example ip.src=xxx.xxx.xxx.xxx, for 24 hours data, it took 20-30 seconds to finish. To me it's normal, but to customer, it's slow.

Any comments?

Thank you.

[SeanKoniarz]

That query time seems correct to me.  Is that for all of the meta fields?

[huanzhou1]

more or less. some maybe like 60 seconds. Mostly are 20-30 seconds.

but if you have 5 meta keys open, then add on already 1 and half minutes. that's why the customer not happy!

[SeanKoniarz]

Honestly I think this is another RSA oversight that they have missed the mark on.  This is very similar, and by very I mean the same problem as, the lookup_and_add() issue.  When the broker goes out and asks for data, it does it sequentially, when it should query as many as possible.  If your max amount of queries is say 20, it should load 20 meta keys at once.  This would in turn make it seem that all the data comes back in 20seconds.

Feature request? 

[RSAAdmin]

When a query comes into a broker, it does not query each device sequentially, it queries all devices concurrently.  The query responsiveness is as fast as your slowest concentrator.

Also, to answer the other question:

IndexKeys is used to query sessions that have a particular meta key ("exists or !exists").  If you query for a particular value, the query will work but will be slower because it has to filter out all the sessions that do not match.

IndexValues is used to find sessions with a particular value.  If you do this type of query often, you should change from IndexKeys to IndexValues

For 10.3, you can make this change and it will automatically take effect on the next index slice without having to restart the service.  For earlier index slices, it will continue to use the previous setting.  But since index slices roll out, it will eventually take effect on the whole index.

[RSAAdmin]

Changing from Index Keys to Index Values only applies to indexing the values from that time forward.  It will not go back and retro-actively index what is already stored on the decoder. 

So when you say you changed to index the values and got the same query time, you were still looking at the unindexed values based on time.  Had you waited a few hours, and queries only over the indexed time, the values should have returned faster.

[huanzhou1]

we're just doing drill, nothing specical.

[SeanKoniarz]

I am not talking about the devices being queried sequentially, I am talking about the meta keys.

Say you have device.ip, ip.src, ip.dst, device.type and msg.id.  From looking at logs on SA.  It goes out and says, packet concentrator and log concentrator, give me the device.ip for the given time frame, returns data (slowest concentrator is the bottle neck here).  After the data is returned it does the same for ip.src, and so on.

A more efficient way it to give us the option to say how many we want to query at once.  So we set the limit to say 3 at once.  The broker could then go out and do 3 queries at once for device.ip, ip.src and ip.dst.  Then keep going from there, so if 2 queries came back in 5 seconds, it could then do 2 more queries while the third finished.   

[huanzhou1]

I've done the index reset after chaning the indexkeys to indexvalues.  Then i tried to query again, the performance turned out simillar. It's for the filename meta key.

[RSAAdmin]

That isn't a broker issue, that's a client issue.

I believe that is being addressed in SA UI for 10.4.  Multiple keys can be queried concurrently.

Also, this can be enabled in Investigator 9.8 by editing the NwInvestigator9.settings file and changing MaxConcurrentQueries="1" on the Navigation node.  Set it to something like 3 or 4 and it will query that number of keys concurrently.