000038772 - Frequent Search Index Rebuilds needed for RSA Archer

• The RSA Archer Search Index breaks frequently or is often slow.
• Quick Searches are no longer working for new records.

• a recent upgrade
• whenever backing up and restoring an instance database
• locks on the Search Index file path and more.

1. Quick Searches are no longer working for new records
2. tblSearchMessageQueue is backlogged:
   
   • Run the following SQL query and observe the count returned: 

SELECT COUNT(*) FROM dbo.tblSearchMessageQueue smq WHERE smq.priority_type_id = 3

• If the count returned is above 0, run the query again in 5 minutes. If the count returned is higher after 5 minutes, this could point to a broken Search Index

3. Check the Search Index file path specified in the ACP and make sure the timestamps are being updated. If not, this could also point to a broken Search Index. 

1. On the Queuing Server, verify the virus protection software excludes the Search Index File Location from scanning.
2. Have the Search Index File Location local to the Queuing Server like C:\ArcherFiles\Index.  Share this folder path like \\MyServer\Archer\Index and update the Search Index Path in the ACP with the UNC path.  This prevents issues from things like DNS name resolution, share permissions, network latency, etc.
3. If the Queuing Server also handles the Job Engine and/or Web processes, you may want to consider dedicating a server for the Queuing Service.
4. Ensure the Service Account running the Queuing Service has full access to the Search Index file path
5. Ensure no two RSA Archer Instances are using the same Search Index file path

1. write.lock errors continue to occur in Queuing Service logs (example below), despite following all the above noted recommendations.
   

   
   Error closing the index path: \\ServerName\archer\indexes\50000\quicksearch\, Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@\\ServerName\archer\indexes\50000\quicksearch\write.lock
   at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
   at Lucene.Net.Index.DirectoryReader.AcquireWriteLock()
   at Lucene.Net.Index.IndexReader.DeleteDocument(Int32 docNum)
   at Lucene.Net.Index.IndexReader.DeleteDocument(Int32 docNum)
   at Lucene.Net.Index.IndexReader.DeleteDocuments(Term term)
   at ArcherTech.Services.Queuing.IndexWriterFacade.DeleteAllDocumentsByContentId()
   at ArcherTech.Services.Queuing.IndexWriterFacade.Flush(Boolean deleteBeforeFlush)
   at ArcherTech.Services.Queuing.IndexFactory.Close(String indexPath, Boolean deleteBeforeFlush)
   7/24/2019 1:59:31 PM - Started Optimizing \\ServerName\archer\indexes\50000\quicksearch\
   7/24/2019 1:59:32 PM - Error optimizing the index path: \\ServerName\archer\indexes\50000\quicksearch\, Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@\\ServerName\archer\indexes\50000\quicksearch\write.lock
   at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, Boolean closeDir, IndexDeletionPolicy deletionPolicy, Boolean autoCommit, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit)
   at Lucene.Net.Index.IndexWriter..ctor(String path, Analyzer a, Boolean create)
   at ArcherTech.Services.Queuing.IndexFactory.Optimize(String fullIndexPath)
   7/24/2019 1:59:33 PM - Finished Optimizing \\ServerName\archer\indexes\50000\quicksearch\

   
   
2. Lower the Flush Factor value from 4000 to 1000 in \\ArcherInstallDirectory\Services\Archer.Services.Queuing.exe.config file. This will allow the index files to merge. Lowering the flushFactor setting forces the segment files to be smaller in size to allow smaller files to be merged at any given point, reducing memory and performance issues.