000036843 - RSA NetWitness Endpoint - The Specified Network Name is no longer available/error with agent xxx connecting from xxx

Document created by RSA Customer Support Employee on Oct 23, 2018
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000036843
Applies ToRSA Product Set: NetWitness Endpoint
RSA Product/Service Type: NetWitness Endpoint
RSA Version/Condition: 4.3..x, 4.4.x
Platform: Windows
O/S Version: Windows
IssueIn the ConsoleServer-error.log file, the following is seen often repeated endlessly in the error log:

 
---------- --:--:--,--- [44] ERROR - [44] a.b.g.X?:There was an error with agent xxx connecting from xxx
   at EConsole.Server.ServerLogic.OnHttpGetCommandHandler(Int64 requestId, X? input, X? output)
   at a.b.X?.?(X? A_0, X? A_1)

Below is the most common Inner-Exception example of this error (#1):

   at a.b.X?.?.?()Inner-Exception:
[44] a.b.g.X?:The specified network name is no longer available
   at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
   at EConsole.Server.ServerLogic.OnHttpGetCommandHandler(Int64 requestId, X? input, X? output)

Inner-Exception error #2:

---------- --:--:--,982 [Auto Restart (?)] ERROR - [3] System.Net.Sockets.SocketException:The connection has been broken due to keep-alive activity detecting a failure while the operation was in progress
   at System.Net.Sockets.Socket.ReceiveFrom(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint& remoteEP)
   at a.b.X?.?(Object A_0)
   at a.b.X?.?(Object A_0

Inner-Exception semaphore error #3:

Inner-Exception:
[40] a.b.e.Xᜀ:
The semaphore timeout period has expired
   at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
   at a.b.Xᜁ.ᜆ(Int64 A_0, Xᜳ A_1, X᜿ A_2)


Inner-Exception error #4:
 

Inner-Exception:
[48] a.b.e.Xᜀ:
An operation was attempted on a nonexistent network connection
   at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
   at a.b.Xᜁ.ᜆ(Int64 A_0, Xᜳ A_1, X᜿ A_2)
CauseThe cause of this error is, effectively, a loss in the communication of the existing connection between the agent and the server. There are numerous possible reasons why this would be:

1. Endpoint machine reboots
2. Endpoint machine enters hibernation
3. Network disruption(switch is disabled, cable breaks)
4. Hardware failure on the endpoint machine(NIC failure or intermittent signal over the NIC)
5. Network latency(too many dropped packets)
6. SQL latency, connection issues, blocking

 
ResolutionIt is first important to understand what message is being displayed by the Inner Exception field. In this instance, we are getting "The specified network name is no longer available" but there are other types as well. Generally, this exception is thrown when the opened connection to the client is disconnected or disconnects in general, and could be a false error, or the result of an error that forcefully disconnected the ongoing stream. The cause is actually obfuscated in the code so in order to determine the exact error that is being propagated, its necessary to request a special debug version of the ConsoleServer.exe that will produce unobfuscated output in the error log.

Generally, the issue becomes more pronounced the larger the deployment is; database responsiveness is noted as a major factor, so resolving database issues can also eliminate or greatly reduce the number of instances of this error message appearing.

For the most part, a moderate volume of these errors is considered acceptable in larger deployments; generally, unless a specific agent is repeated consistently in the error logs, it can be ignored. Below are steps to resolve when this error becomes very pronounced and needs specific remediation to address it:

Resolution Steps:
1. Check the ConsoleServer-error.log file for error messages and record each type of error.
2. Check the SQL server for errors, poor processing, or network connection timeouts/errors. If there are any known issues, resolve these first.
3. If no errors in SQL processing observed or all other errors resolved, request support provide an unobfuscated debug build of the ConsoleServer.exe file and send back the new error log for analysis
WorkaroundOnce the root cause of each Inner-Exception error is fully documented and remediated, kb's should be linked here for those specific instances.
NotesIDENTIFICATION STEPS:

The below steps will assist in determining the number of these agentID's, and how frequent specific agents experience this issue. Make sure for the first step to provide the correct path to the error log file in the -Path section, in this example its local to the directory.
  1. $output = Select-String -Path "cserrorlog.txt" -Pattern "There was an error with agent"
    //Strip out the the list to just AgentID's only
    $output -split '\s+' -match '-'
  2.  //Get the total number of unique Agent's with this error
    $output -split '\s+' -match '-' | Get-Unique | Measure-Object -Line
  3. //Get the list of unique AgentID's
    $output -split '\s+' -match '-' | Get-Unique

Attachments

    Outcomes