000036222 - Blocking tab error "Execution Timeout Expired. The timeout period elapsed prior to completion of the operation" in RSA NetWitness Endpoint

Document created by RSA Customer Support Employee on Apr 19, 2018Last modified by RSA Customer Support Employee on Apr 21, 2018
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000036222
Applies ToRSA Product Set: NetWitness Endpoint
RSA Product/Service Type: User Interface
RSA Version/Condition: 4.2, 4.3, 4.4
IssueThe UI errors to start occasionally at launch with a timeout error. Alternatively, and in fact more commonly assuming both conditions are met, the blocking tab will never succeed to load when selected and will instead error out with the following error message crashing the UI:


System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.TryNextResult(Boolean& more)
   at System.Data.SqlClient.SqlDataReader.NextResult()
   at ECatDataModel.LoaderParameters.GetBlockingSentinelEvents.DataReader(SqlCommand& sqlCommand, SqlDataReader dataReader, PagedThreadParameters& threadParameters, Int64& itemCounter)
   at DatabaseTools.AsyncPagedDataLoad`1.LoadDataThreadProc(PagedThreadParameters aThreadParameter)
ClientConnectionId:905fc8f5-a18e-408a-9a01-0f27921d0966
Error Number:-2,State:0,Class:11

CauseThere is a five minute timeout used with running SQL queries that can hit depending on a number of factors, which includes hardware, IOPS of the disk used for the SQL database, and the number of entries related to blocking and the total number of tracking event items in the tracking event global blocking view query can take long enough to cause this error.

There are two key takeaways:
  1. Total number of blocked modules. This has nothing to do with the actual modules still in the environment, but in the SQL database referenced.  If there are hundreds, or worse, thousands of these entries, the blocking can cause unacceptable long runtimes in SQL
  2. The P0 and P1 tracking tables. These tables are queried along with the modules to determine the modules that have blocking enabled. When tracking events reach a very high count, such as increasing tracking events beyond the default 30 days, the two tables can increase to a point where the query exceeds the 5 minute threshold.
ResolutionRolling off both the blocked modules, and the tracking events is critical to resolving this issue. Suggested methods:
  1. Roll off tracking event data. This is done in the Configure > Database Maintenance > Remove Tracking Events Older Than section of the UI and should be changed from its current value to 7 days. It will require about two weeks for the bulk of these entries to be rolled off due to how data rolls from P0 to P1 and then rolls off completely during the weekend maintenance database runs.
  2. Delete entries from the Blocking tab to reduce the total number of blocked modules. A normal environment shouldn't need hundreds, or worse thousands, of blocked modules, which can impact performance heavily in loading and processing those entries in the database. If this hangs and fails the Blocking tab in the UI, it will require table truncation by engineering to remove enough entries but the first step should be tried to see if this is enough to get the tab to load. If so, removing entries will improve performance.

Attachments

    Outcomes