It appears we have gotten too big for our britches. We have a server dedicated to just indexing and our Search has broken. We have too much data! We have tried the following:
Reduced history log fields and their retention period
We are afraid to enable verbose logging for fear of increase CPU usage
We've tried running the search index from another server completely
Finally we have changed our indexing server to an XL server and are being told it could take a week to do a full index rebuild
Anyone have any ideas?
Anyone have any luck with clustering or load balancing index servers?
How much data do you have? What is your instance DB size and record count? How big is the largest file in your index folder?
I ask these things, as I find it hard to believe it will take a week to rebuild your indices unless your services server is drastically underpowered. We have a nearly 400 Gb instance DB with just shy of 5 million records in it, which has our search indices files at just under 2 Gb for the largest file, and we can do a full rebuild in just shy of 36 hours (a weekend).
Okay - Can definitely see a weeks time to rebuild then. Out of curiosity, how big is the biggest file within your index folder? I mentioned ours was 2Gb, curious what your increased scale from ours does to this as we aren't far behind you in growth.
When you say your search has broken, what does that mean? Is it returning false negatives (i.e. saying No Records Found when you know it exists, and repeat searches ultimately return the proper results set)? Is it consistently behind, in that new records won't show in results no matter how many times you search until a day or two later?
Also, it is known that the more history log data you get the more content needs indexed. Check and see how big your history table is and let us know. I would bet money that your history logs have gotten pretty large and now the jobs that clean up that content will timeout because of too much content to clean up especially with the effort you put in to reducing your history.
I am Sanchitha who is part of Laura's team. Thought of responding on her behalf.
1. Search is broken - display "No Records Found"
2. Yes, we got rid of the History Log field of some of our applications based on the query which returns the applications causing an overhead. That said, we cannot get rid of HL for some of our other apps (that contain data from past 2 years or so) for various reasons.
I agree the it's the every growing History Log. Thanks for all your responses, I shall let you know if we make a good progress.
Not sure if you are currently using multi-lingual at all now, but if you are not you could make sure that "Index design language only" is checked within your ACP. This should cut down your index sizes a wee bit.
You can additionally check all of your history log fields, to ensure the option of "Keyword Searching" is de-selected within all of your HL fields. This should prevent HL contents from being indexed for search.
We're going down the road of excluding History Log from Indexing, however the scripts that we are getting from RSA are not working. Anyone know a tried/true way of excluding them?