Article Content
Article Number | 000036843 |
Applies To | RSA Product Set: NetWitness Endpoint RSA Product/Service Type: NetWitness Endpoint RSA Version/Condition: 4.3..x, 4.4.x Platform: Windows O/S Version: Windows |
Issue | In the ConsoleServer-error.log file, the following is seen often repeated endlessly in the error log: ---------- --:--:--,--- [44] ERROR - [44] a.b.g.X?:There was an error with agent xxx connecting from xxx at EConsole.Server.ServerLogic.OnHttpGetCommandHandler(Int64 requestId, X? input, X? output) at a.b.X?.?(X? A_0, X? A_1) Below is the most common Inner-Exception example of this error (#1): at a.b.X?.?.?()Inner-Exception: [44] a.b.g.X?:The specified network name is no longer available at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize) at EConsole.Server.ServerLogic.OnHttpGetCommandHandler(Int64 requestId, X? input, X? output) Inner-Exception error #2: ---------- --:--:--,982 [Auto Restart (?)] ERROR - [3] System.Net.Sockets.SocketException:The connection has been broken due to keep-alive activity detecting a failure while the operation was in progress at System.Net.Sockets.Socket.ReceiveFrom(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint& remoteEP) at a.b.X?.?(Object A_0) at a.b.X?.?(Object A_0 Inner-Exception semaphore error #3:
Inner-Exception error #4:
|
Cause | The cause of this error is, effectively, a loss in the communication of the existing connection between the agent and the server. There are numerous possible reasons why this would be: 1. Endpoint machine reboots 2. Endpoint machine enters hibernation 3. Network disruption(switch is disabled, cable breaks) 4. Hardware failure on the endpoint machine(NIC failure or intermittent signal over the NIC) 5. Network latency(too many dropped packets) 6. SQL latency, connection issues, blocking |
Resolution | It is first important to understand what message is being displayed by the Inner Exception field. In this instance, we are getting "The specified network name is no longer available" but there are other types as well. Generally, this exception is thrown when the opened connection to the client is disconnected or disconnects in general, and could be a false error, or the result of an error that forcefully disconnected the ongoing stream. The cause is actually obfuscated in the code so in order to determine the exact error that is being propagated, its necessary to request a special debug version of the ConsoleServer.exe that will produce unobfuscated output in the error log. Generally, the issue becomes more pronounced the larger the deployment is; database responsiveness is noted as a major factor, so resolving database issues can also eliminate or greatly reduce the number of instances of this error message appearing. For the most part, a moderate volume of these errors is considered acceptable in larger deployments; generally, unless a specific agent is repeated consistently in the error logs, it can be ignored. Below are steps to resolve when this error becomes very pronounced and needs specific remediation to address it: Resolution Steps: 1. Check the ConsoleServer-error.log file for error messages and record each type of error. 2. Check the SQL server for errors, poor processing, or network connection timeouts/errors. If there are any known issues, resolve these first. 3. If no errors in SQL processing observed or all other errors resolved, request support provide an unobfuscated debug build of the ConsoleServer.exe file and send back the new error log for analysis |
Workaround | Once the root cause of each Inner-Exception error is fully documented and remediated, kb's should be linked here for those specific instances. |
Notes | IDENTIFICATION STEPS: The below steps will assist in determining the number of these agentID's, and how frequent specific agents experience this issue. Make sure for the first step to provide the correct path to the error log file in the -Path section, in this example its local to the directory.
|