000034166 - Roaming Agent Relay Server error: ERROR: System.IO.EndOfStreamException: SharedQueue closed for RSA Netwitness Endpoint 4.1.2

Document created by RSA Customer Support Employee on Oct 11, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000034166
Applies ToRSA Product Set: Netwitness Endpoint (formerly ECAT)
RSA Product/Service Type: Netwitness Endpoint (formerly ECAT)
RSA Version/Condition: 4.1.2
Platform: Windows
 
IssueWhen commissioning a machine in the Netwitness Endpoint (formerly ECAT) UI, a unique numerical key is generated which is used to create both the exchange queue in rabbitmq and the queue's used by said exchange as part of their naming convention. When this doesn't match the expected value in the consoleserver, or when one or more queue's or the exchange are missing, errors are seen in both the ConsoleServerOutput as well as in the rabbitmq log.
CauseThe current causes of why this can happen are only speculative at this time. Possible causes though can include:
  1. Manually deleting or modifying the existing exchange, or any of its bound queue's.
  2. Recommissioning a Roaming Agent Relay server. This can lead to the consoleserver generating a new numerical value, and no longer communicating with the Roaming Agent Relay due to having an exchange mismatch.
  3. Incorrect exchange and queue's being created following commissioning. It is believed this can possibly happen following initial creation of the exchange and its queue's where the system will fail to overwrite incorrect queue and exchange values causing a mismatch
  4. Missing any of the mandatory queue's or exchanges during commissioning will also cause these kinds of errors.
Resolution
Confirming the Roaming Agent Relay Error

NOTE: The below mainly applies to single server environments. With multi-server, there may be differences not covered here.
It is important to first identify this error message by checking both the console server error messages, and the messages seen in the rabbitmq logfile. The rabbitmq log for the Roaming Agent Relay is typically found in:
C:\ECAT\Relay\logs


This is the typical location but can be changed. The format of the log is usually rabbit@<name_of_server>, i.e. rabbit@W10x64. When checking the log, the following errors will be seen in this order. NOTE - Use a better application to open this than Notepad, like Notepad++ so it gets parsed in a more readable way:
 


06 10:49:25:1970 RabbitMQ connection established.
06 10:50:05:6502 ERROR: RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, ini
tiated by Peer, code=404, text="NOT_FOUND - no exchange 'cs_F492ABC8-B9E5-4687-A898-05CB16BBA3C3.exchange' in vhost 'ecat'", classId=60, met
hodId=40, cause=
UDP Beacon received from [::ffff:104.100.100.70]:2220!
06 10:50:05:6502 WARNING: RabbitMQ connection is down.
UDP Beacon received from [::ffff:74.38.100.138]:54395!
06 10:50:10:9313 ERROR: System.IO.EndOfStreamException: SharedQueue closed


 

Other errors may be listed such as all of the rabbitmq Queue's related to the exchange. By listing them all out, it becomes clear which are needed to be added or modified. Fixing this has two methods:

Resolution Steps

  1. Delete the cs_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.exchange and its associated queue's and recommission the server in the ConsoleServer UI to regenerate a new exchange and the queue bindings associated with it in the Roaming Agent Relay(currently this is untested to work but is technically harmless since it isn't functional under the listed queue's if the above error is being seen).
  2. Delete the old exchange and its queue's and then generate a new exchange and set of associated queue's with the appropriate settings and Queue bindings to direct rabbitmq traffic to the correct queue.(Tested) See the workaround steps below.
Workaround
  1. Confirm the actual exchange being used by the consoleserver by grabbing the rabbit@<server_name> logfile from the RAR in C:\ECAT\RELAY\logs and confirm that the exchange is incorrect by comparing the number, i.e. cs_F492ABC8-B9E5-4687-A898-05CB16BBA3C3.exchange is in the error log as NOT FOUND but the actual cs queue in the rabbitmq web page (http://localhost:15672/#/exchanges) is a different number, i.e. cs_EE159188-903B-4957-ADF8-58B5E943E112.exchange
  2. Once confirmed, add the missing exchange as well as its missing associated queue's. The queue's are listed below for reference along with the associated exchange:
    1.   cs_EE159188-903B-4957-ADF8-58B5E943E112.get_command_queue
    2. cs_EE159188-903B-4957-ADF8-58B5E943E112.post_data_queue
    3. cs_EE159188-903B-4957-ADF8-58B5E943E112.post_file_queue
    4. cs_EE159188-903B-4957-ADF8-58B5E943E112.post_processed_queue
    5. cs_EE159188-903B-4957-ADF8-58B5E943E112.post_upload_confirm_queue
    6. cs_EE159188-903B-4957-ADF8-58B5E943E112.udp_beacon_queue
    7. cs_EE159188-903B-4957-ADF8-58B5E943E112.exchange (this is the exchange, and is not a queue)
list of queue's

  1. Create the exchange:
    1. Open the page in the listed link here: http://localhost:15672/#/exchanges. You may need the login user and password, which are default ecat for both user and pass
    2. Click Add new exchange
    3. Under Virtual host select the vhost user, typically this is ecat
    4. For name, you must give it the missing exchange name from the rabbit log output, i.e. cs_F492ABC8-B9E5-4687-A898-05CB16BBA3C3.exchange in this example
    5. Type=topic
    6. Durability=Durable
    7. Click the Add Exchange button
adding an exchange queue

  1. Create the missing Queue's:
    1. Go to the Queue page: http://localhost:15672/#/queues
    2. Click Add a new queue
    3. For Virtual host, the default user is ecat, otherwise set it to the designated vhost user
    4. Under Name, insert the name of the designated queue starting with the first, which in our example from step 3 would look similar to our exchange created earlier: cs_F492ABC8-B9E5-4687-A898-05CB16BBA3C3.get_command_queue
    5. Leave the Durability setting as Durable, and click Add Queue
    6. Repeat this process for the first 5 queue's
    7. When you get to the 6th queue (udp_beacon_queue), there is an additional step; this queue must have the following argument added to the Arguments field. A screenshot is provided of the udp_beacon_queue below; this is the only queue that has arguments:
       
      1. In the first field enter x-message-ttl
      2. In the second field(after the equals sign) enter the value 10000
      3. Change the type from String to Number
       
adding a queue

  1. The exchanges and queue's exist, now the queue's must be bound to the respective exchange created in step 3 using rabbitmq's bindings:
    1. Open the Exchange page: http://localhost:15672/#/exchanges
    2. Select the cs_exchange created in step 3 to bring up its options
    3. Under Bindings there is a subsection called 'Add binding from this exchange'
    4. Insert the name of the first queue(the full name) into the field next to the dropdown listed as 'To queue'
    5. Under Routing key, add the listed queue's routing key. Below is a summary list of each queue and below that is the routing key:
       
      1. get_command_queue
          
        1. get_command
          
      2. post_data_queue
          
        1. post_data
          
      3. post_file_queue
          
        1. post_file
          
      4. post_processed_queue
          
        1. post_processed
          
      5. post_upload_confirm_queue
          
        1. post_upload
          
      6. udp_beacon_queue
          
        1. udp_beacon
          
       
binding queue's to an exchange

  1. Once this is complete restart the Primary ConsoleServer and then observe if any errors related to incorrect configuration of the Relay server are gone. Additionally, the error messages logged to the Rabbitmq logfile should no longer be present. This will confirm that rabbit is running correctly. Testing should show that after a few minutes the agents are connecting in through the Roaming Agent Relay server's although you may have to connect them to the corporate network first to get an updated Roaming Agent Relay key if they have not connected for a long time. This can be checked under Agent Log for the agent if a Get Key command has been queued but not processed.

Attachments

    Outcomes