Problem after update to 10.4

[AlexFedorov]

Hello,

After update from 10.3.4 to 10.4 I receive some problem. My Log Collector not work. In first time I receive many like:

Sep 23 09:36:00 SA-LogDecoder nw[2196]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: a socket error occurred

Sep 23 09:36:00 SA-LogDecoder nw[2196]: [LogdecoderProcessor] [failure] [queue.odbc] [idle] Failed during getWork: a socket error occurred

I can see only logs collect via log decoder (protocol Syslog). Then I try find solution at forum and find action rekey (). I'm also reinstall nwlogcollector, rabbit-mq server. Now I can access to Log Collector (Service x.x.x.x host LogCollector is unreachable).

[AlexFedorov]

How I guess problem in rabbitmq-server. NwLogCollector not start automatically.

BOOT FAILED

===========

Error description:

   {could_not_start,nw_admin,

       {{shutdown,

            {failed_to_start_child,nw_admin_worker,

                {{badmatch,false},

                 [{nw_admin_worker,init,1,[]},

                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},

                  {proc_lib,init_p_do_apply,3,

                      [{file,"proc_lib.erl"},{line,239}]}]}}},

        {nw_admin,start,[normal,[]]}}}

=INFO REPORT==== 23-Sep-2014::12:30:19 ===

node           : rabbit@SA-LogDecoder

home dir       : /var/lib/rabbitmq

config file(s) : (none)

cookie hash    : Fb/V18D0IfbwgiH0zgGgvw==

log            : /var/log/rabbitmq/rabbit@SA-LogDecoder.log

sasl log       : /var/log/rabbitmq/rabbit@SA-LogDecoder-sasl.log

database dir   : /var/lib/rabbitmq/mnesia/rabbit@SA-LogDecoder

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

Memory limit set to 3149MB of 7872MB total.

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

Disk free limit set to 50MB

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 23-Sep-2014::12:30:22 ===

msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

Adding vhost '/'

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

Creating user 'guest'

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

Setting user tags for user 'guest' to [administrator]

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

Setting permissions for 'guest' in '/' to '.*', '.*', '.*'

=INFO REPORT==== 23-Sep-2014::12:30:22 ===

started TCP Listener on [::]:5672

=INFO REPORT==== 23-Sep-2014::12:30:23 ===

Management plugin started. Port: 15672

=INFO REPORT==== 23-Sep-2014::12:30:23 ===

Statistics database started.

=INFO REPORT==== 23-Sep-2014::12:30:23 ===

nw_admin_worker:init: [{included_applications,[]}]

=INFO REPORT==== 23-Sep-2014::12:30:23 ===

nw_admin initialized.

=INFO REPORT==== 23-Sep-2014::12:30:23 ===

stopped TCP Listener on [::]:5672

=INFO REPORT==== 23-Sep-2014::12:30:23 ===

Error description:

   {could_not_start,nw_admin,

       {{shutdown,

            {failed_to_start_child,nw_admin_worker,

                {{badmatch,false},

                 [{nw_admin_worker,init,1,[]},

                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},

                  {proc_lib,init_p_do_apply,3,

                      [{file,"proc_lib.erl"},{line,239}]}]}}},

        {nw_admin,start,[normal,[]]}}}

{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,nw_admin,{{shutdown,{failed_to_start_child,nw_admin_worker,{{badmatch,false},[{nw_admin_worker,init,1,[]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}},{nw_admin,start,[normal,[]]}}}}}

[60G[ [0;31mFAILED [0;39m]

=CRASH REPORT==== 23-Sep-2014::12:30:23 ===

  crasher:

    initial call: nw_admin_worker:init/1

    pid: <0.361.0>

    registered_name: []

    exception exit: {{badmatch,false},

                     [{nw_admin_worker,init,1,[]},

                      {gen_server,init_it,6,

                                  [{file,"gen_server.erl"},{line,304}]},

                      {proc_lib,init_p_do_apply,3,

                                [{file,"proc_lib.erl"},{line,239}]}]}

      in function  gen_server:init_it/6 (gen_server.erl, line 328)

    ancestors: [nw_admin_sup,<0.358.0>]

    messages: []

    links: [<0.359.0>]

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 1598

    stack_size: 27

    reductions: 4311

  neighbours:

=SUPERVISOR REPORT==== 23-Sep-2014::12:30:23 ===

     Supervisor: {local,nw_admin_sup}

     Context:    start_error

     Reason:     {{badmatch,false},

                  [{nw_admin_worker,init,1,[]},

                   {gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},

                   {proc_lib,init_p_do_apply,3,

                             [{file,"proc_lib.erl"},{line,239}]}]}

     Offender:   [{pid,undefined},

                  {name,nw_admin_worker},

                  {mfargs,

                      {nw_admin_worker,start_link,

                          [[{included_applications,[]}]]}},

                  {restart_type,permanent},

                  {shutdown,10000},

                  {child_type,worker}]

=CRASH REPORT==== 23-Sep-2014::12:30:23 ===

  crasher:

    initial call: application_master:init/4

    pid: <0.357.0>

    registered_name: []

    exception exit: {{shutdown,

                         {failed_to_start_child,nw_admin_worker,

                             {{badmatch,false},

                              [{nw_admin_worker,init,1,[]},

                               {gen_server,init_it,6,

                                   [{file,"gen_server.erl"},{line,304}]},

                               {proc_lib,init_p_do_apply,3,

                                   [{file,"proc_lib.erl"},{line,239}]}]}}},

                     {nw_admin,start,[normal,[]]}}

      in function  application_master:init/4 (application_master.erl, line 133)

    ancestors: [<0.356.0>]

    messages: [{'EXIT',<0.358.0>,normal}]

    links: [<0.356.0>,<0.7.0>]

    dictionary: []

    trap_exit: true

    status: running

    heap_size: 376

    stack_size: 27

    reductions: 124

  neighbours:

[AlexFedorov]

How I can understand - I lost my keys in folder /etc/netwitness/ng/rabbitmq/ssl/keys at LogCollector/Decoder. How I can generate new keys?

[RSAAdmin]

I am seeing the same issue. you mention that you upgraded to 10.4 and did you reinstall nwlogcollector and rabbitmq ? did it resolve the issue ?

[AlexFedorov]

I make same actions like reinstall erlang*, rabbitmq-server, nwlogcollector... nothing happend.

But I have same progress...

1. service rabbitmq-server stop, stop nwlogcollector

2. rm -rf /etc/netwitness/ng/rabbitmq/legacy-ssl

3. cp /opt/netwitness/etc/rabbitmq-base-config /etc/rabbitmq/config/rabbitmq.config

4. ln -s /etc/netwitness/ng/rabbitmq/config/rabbitmq.config /etc/rabbitmq/config/rabbitmq.config

5. rm /etc/rabbitmq/ssl/truststore.pem

6. service rabbitmq-server start, start nwlogcollector

7. Administration --> Devices --> Select Log Collector --> Explore --> event-broker --> ssl --> (Right-Click) Properties --> (from drop down) rekey

After this actions I reboot log collector and links in my folder /etc/netwitness/ng/rabbitmq/ssl recovered. But this links to:

cert.pem /etc/rabbitmq/ssl/server/cert.pem

privkey.pem /etc/rabbitmq/ssl/server/key.pem

This files and folder "server" not exist at my log collector. I look at similar files at SA Server, this file - links to

/var/lib/puppet/ssl/certs/id.pem

/var/lib/puppet/ssl/private_keys/id.pem

I make similar links, but this not happend 😞

[ChandlerHall]

I am also experiencing this problem after the 10.4 update.

[RSAAdmin]

Please also check the host file which is etc/host and make sure you see a loop back address.I had same issue and seems my host file got corrupted. and fixing the host file thing are normal, but see a different error now.

Keyur

[AlexFedorov]

Hosts file correct. I guess problem in bad links in folder /etc/netwitness/ng/rabbitmq/ssl/keys/:

cert.pem -> /etc/rabbitmq/ssl/server/cert.pem

privkey.pem -> /etc/rabbitmq/ssl/server/key.pem

This files not exist. All ssl keys from previously version go to folder /etc/netwitness/ng/rabbitmq/legacy-ssl/keys/, but if we look at View/Explorer we see links to /etc/netwitness/ng/rabbitmq/ssl/keys/. I try cope files from folder ../legacy-ssl/keys/ to ../ssl/keys/ - this not happend. I guess new keys - puppet keys like in SA Server. I make similar links, but this not happed. Need contact to RSA Support... or need contact with customer who correctly update to 10.4.

[SeanKoniarz]

Anyone figure this out?  I contacted support today but curious. 

[AlexFedorov]

Please, share answer RSA Support here.

I not have more time to wait solution. I reimage my demo instance. But I not "Enable" at tab "Administrator -> Appliance" my Log Decoder/Collector before I try solve problem. After "Enable" - this not helped me, maybe because I done more same operations.. Maybe If I done all step by step from update guide - update be success.