This website uses cookies. By clicking OK, you consent to the use of cookies. Click Here to learn more about how we use cookies.
OK
  • RSA.com
  • Products
    • Archer®
      • Archer®
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Knowledge Base
      • Archer® Exchange
      • Training
      • Upcoming Events
      • Videos
    • RSA® Fraud & Risk Intelligence Suite
      • RSA® Fraud & Risk Intelligence Suite
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Web Threat Detection
      • Upcoming Events
      • Videos
    • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Cloud
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Events
      • Ideas
      • Knowledge Base
      • Training
      • Upcoming Patch Content
      • Videos
    • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication Mobile SDK
      • Advisories
      • Events
      • Ideas
      • Knowledge Base
      • Request Access
      • Training
    • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Events
      • Ideas
      • Knowledge Base
      • Training
      • Videos
    • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 7.x
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Events
      • Ideas
      • Knowledge Base
      • Training
      • Videos
    • RSA® Adaptive Authentication for eCommerce
      • RSA® Adaptive Authentication for eCommerce
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Ideas
      • Knowledge Base
      • Training
      • Videos
    • RSA® FraudAction Services
      • RSA® FraudAction Services
      • Advisories
      • Discussions
      • Documentation
      • Ideas
      • Videos
    • RSA® Web Threat Detection
      • RSA® Web Threat Detection
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Knowledge Base
      • Videos
    • RSA NetWitness® Platform
      • RSA NetWitness® Platform
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Integrations
      • Knowledge Base
      • Training
      • Upcoming Events
      • Videos
    • RSA NetWitness® Detect AI
      • RSA NetWitness® Detect AI
      • Documentation
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
    • RSA NetWitness® Investigator
      • RSA NetWitness® Investigator
      • Documentation
      • Download the Client
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
    • RSA NetWitness® Orchestrator
      • RSA NetWitness® Orchestrator
      • Overview
      • Documentation
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
    • RSA SecurID® Suite
      • RSA SecurID® Suite
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Knowledge Base
      • Ideas
      • Integrations
      • Training
      • Videos
    • RSA® Identity Governance & Lifecycle
      • RSA® Identity Governance & Lifecycle
      • Advisories
      • Blog
      • Community Exchange
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Knowledge Base
      • Training
      • Upcoming Events
      • Videos
    • RSA SecurID® Access
      • RSA SecurID® Access
      • Advisories
      • Blog
      • Discussions
      • Documentation
      • Downloads
      • Ideas
      • Integrations
      • Knowledge Base
      • Training
      • Upcoming Events
      • Videos
    • Other RSA® Products
      • Other RSA® Products
      • RSA® Access Manager
      • RSA® Data Loss Prevention
      • RSA® Digital Certificate Solutions
      • RSA enVision®
      • RSA® Federated Identity Manager
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
      •  
  • Resources
    • Advisories
      • Product Advisories on RSA Link
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Hosted
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Product Advisories
    • Blogs
      • Blogs on RSA Link
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Blogs on RSA Link
    • Discussion Forums
      • Discussion Forums
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Discussion Forums on RSA Link
    • Documentation
      • Product Documentation
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Documentation on RSA Link
    • Downloads
      • Product Downloads
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Downloads on RSA Link
    • Ideas
      • Idea Exchange
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® FraudAction Services
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Documentation on RSA Link
    • Knowledge Base
      • Knowledge Base
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication Mobile SDK
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Knowledge Base Pages on RSA Link
    • Upcoming Events on RSA Link
      • Upcoming Events
    • Videos
      • Videos on RSA Link
      • Archer®
      • RSA® Adaptive Authentication Cloud
      • RSA® Adaptive Authentication On-Premise 7.x
      • RSA® Adaptive Authentication On-Premise 14.x
      • RSA® Adaptive Authentication for eCommerce
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
      • RSA® Web Threat Detection
      • All Videos on RSA Link
  • Support
    • RSA Link Support
      • RSA Link Support
      • News & Announcements
      • Getting Started
      • Support Forum
      • Support Knowledge Base
      • Ideas & Suggestions
    • RSA Product Support
      • RSA Product Support
      • General Security Advisories and Statements
      • Product Life Cycle
      • Support Information
      •  
      •  
      •  
      •  
      •  
  • RSA Ready
  • RSA University
    • Certification Program
      • Certification Program
    • Course Catalogs
      • Course Catalogs
      • Archer®
      • RSA NetWitness® Platform
      • RSA SecurID® Suite
    • On-Demand Subscriptions
      • On-Demand Subscriptions
      • Archer®
      • RSA NetWitness® Platform
      • RSA SecurID® Suite
    • Product Training
      • Product Training
      • Archer®
      • RSA® Fraud & Risk Intelligence Suite
      • RSA® Identity Governance & Lifecycle
      • RSA NetWitness® Platform
      • RSA SecurID® Access
    • Student Resources
      • Student Resources
      • Access On-Demand Learning
      • Access Virtual Labs
      • Contact RSA University
      • Enrollments & Transcripts
      • Frequently Asked Questions
      • Getting Started
      • Learning Modalities
      • Payments & Cancellations
      • Private Training
      • Training Center Locations
      • Training Credits
      • YouTube Channel
    • Upcoming Events
      • Upcoming Events
      • Full Calendar
      • Conferences
      • Live Classroom Training
      • Live Virtual Classroom Training
      • Webinars
Sign In Register Now
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements

RSA NetWitness® Platform Discussions

Discussions about the RSA NetWitness Platform.
  • RSA Link
  • :
  • Products
  • :
  • RSA NetWitness Platform
  • :
  • Discussions
  • :
  • question on indexkeys and index values again
  • Options
    • Subscribe to RSS Feed
    • Mark Topic as New
    • Mark Topic as Read
    • Float this Topic for Current User
    • Bookmark
    • Subscribe
    • Mute
    • Printer Friendly Page
huanzhou1
huanzhou1 Beginner
Beginner
‎2014-04-08 08:49 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

question on indexkeys and index values again

Hi Guys,

I have few questions on the index xml file again:

1. if now i change from indexkeys to indexvalues, will it apply to the existing index files? Or only the future index files?

Need to reset index to recreate the index files? Last time i did reset, for 1 DAC raw packets, it took a few hours to finish re-indexing.

2. Is there any benchmark available for the indexkeys or indexvalues?

The day i did some testing, for filenames, indexkeys or indexvalues got the same query time.

3. Today in one customer, the customer has around 8TB data per decoder, total 4 decoders, that means around 30TB per day. Customer complaint query sloness, I did some testing, for example ip.src=xxx.xxx.xxx.xxx, for 24 hours data, it took 20-30 seconds to finish. To me it's normal, but to customer, it's slow.

 

Any comments?

 

Thank you.

  • Tags:
  • Community Thread
  • Decoder
  • Discussion
  • Forum Thread
  • index
  • NetWitness
  • NW
  • NWP
  • packets
  • Query
  • RSA NetWitness
  • RSA NetWitness Platform
  • sa
0 Likes
Share
Reply
  • All forum topics
  • Previous Topic
  • Next Topic
16 Replies
SeanKoniarz
SeanKoniarz Beginner
Beginner
‎2014-04-08 09:24 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

That query time seems correct to me.  Is that for all of the meta fields?

0 Likes
Share
Reply
huanzhou1
huanzhou1 Beginner
Beginner
In response to SeanKoniarz
‎2014-04-08 09:30 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

more or less. some maybe like 60 seconds. Mostly are 20-30 seconds.

but if you have 5 meta keys open, then add on already 1 and half minutes. that's why the customer not happy!

0 Likes
Share
Reply
SeanKoniarz
SeanKoniarz Beginner
Beginner
In response to huanzhou1
‎2014-04-08 09:33 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Honestly I think this is another RSA oversight that they have missed the mark on.  This is very similar, and by very I mean the same problem as, the lookup_and_add() issue.  When the broker goes out and asks for data, it does it sequentially, when it should query as many as possible.  If your max amount of queries is say 20, it should load 20 meta keys at once.  This would in turn make it seem that all the data comes back in 20seconds.

 

Feature request? 

0 Likes
Share
Reply
RSAAdmin
RSAAdmin Beginner
Beginner
In response to SeanKoniarz
‎2014-04-08 10:11 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

When a query comes into a broker, it does not query each device sequentially, it queries all devices concurrently.  The query responsiveness is as fast as your slowest concentrator.

 

Also, to answer the other question:

 

IndexKeys is used to query sessions that have a particular meta key ("exists or !exists").  If you query for a particular value, the query will work but will be slower because it has to filter out all the sessions that do not match.

 

IndexValues is used to find sessions with a particular value.  If you do this type of query often, you should change from IndexKeys to IndexValues

 

For 10.3, you can make this change and it will automatically take effect on the next index slice without having to restart the service.  For earlier index slices, it will continue to use the previous setting.  But since index slices roll out, it will eventually take effect on the whole index.

1 Like
Share
Reply
RSAAdmin
RSAAdmin Beginner
Beginner
‎2014-04-08 10:11 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

Changing from Index Keys to Index Values only applies to indexing the values from that time forward.  It will not go back and retro-actively index what is already stored on the decoder. 

 

So when you say you changed to index the values and got the same query time, you were still looking at the unindexed values based on time.  Had you waited a few hours, and queries only over the indexed time, the values should have returned faster.

0 Likes
Share
Reply
huanzhou1
huanzhou1 Beginner
Beginner
In response to SeanKoniarz
‎2014-04-08 10:16 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

we're just doing drill, nothing specical.

0 Likes
Share
Reply
SeanKoniarz
SeanKoniarz Beginner
Beginner
In response to RSAAdmin
‎2014-04-08 10:17 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

I am not talking about the devices being queried sequentially, I am talking about the meta keys.

 

Say you have device.ip, ip.src, ip.dst, device.type and msg.id.  From looking at logs on SA.  It goes out and says, packet concentrator and log concentrator, give me the device.ip for the given time frame, returns data (slowest concentrator is the bottle neck here).  After the data is returned it does the same for ip.src, and so on.

 

A more efficient way it to give us the option to say how many we want to query at once.  So we set the limit to say 3 at once.  The broker could then go out and do 3 queries at once for device.ip, ip.src and ip.dst.  Then keep going from there, so if 2 queries came back in 5 seconds, it could then do 2 more queries while the third finished.   

0 Likes
Share
Reply
huanzhou1
huanzhou1 Beginner
Beginner
In response to RSAAdmin
‎2014-04-08 10:20 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

I've done the index reset after chaning the indexkeys to indexvalues.  Then i tried to query again, the performance turned out simillar. It's for the filename meta key.

0 Likes
Share
Reply
RSAAdmin
RSAAdmin Beginner
Beginner
In response to SeanKoniarz
‎2014-04-08 10:21 AM
  • Mark as New
  • Bookmark
  • Subscribe
  • Mute
  • Subscribe to RSS Feed
  • Permalink
  • Print
  • Email to a Friend
  • Report Inappropriate Content

That isn't a broker issue, that's a client issue.

 

I believe that is being addressed in SA UI for 10.4.  Multiple keys can be queried concurrently.

 

Also, this can be enabled in Investigator 9.8 by editing the NwInvestigator9.settings file and changing MaxConcurrentQueries="1" on the Navigation node.  Set it to something like 3 or 4 and it will query that number of keys concurrently.

0 Likes
Share
Reply
  • « Previous
    • 1
    • 2
  • Next »
  • « Previous
    • 1
    • 2
  • Next »
Powered by Khoros
  • Products
  • Resources
  • Solutions
  • RSA University
  • Support
  • RSA Labs
  • RSA Ready
  • About RSA Link
  • Terms & Conditions
  • Privacy Statement
  • Provide Feedback
© 2021 RSA Security LLC or its affiliates.
All rights reserved.