000037370 - Install/Upgrade fails in RSA NetWitness Logs and Network because Resolv::ResolvError: no address for a particular host

Document created by RSA Customer Support Employee on May 7, 2019
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000037370
Applies ToRSA Product Set: NetWitness Logs and Network
RSA Product/Service Type: Orchestration/Chef
RSA Version/Condition:  11.3
Platform: CentOS
O/S Version: 7
IssueWhile updating/installing a device to version 11.3, the following error can occur and be found in /var/log/netwitness/config-management/chef-solo.log:

.......
[2019-04-16T20:55:32+00:00] ERROR: Running exception handlers
[2019-04-16T20:55:32+00:00] ERROR: Exception handlers complete
[2019-04-16T20:55:32+00:00] FATAL: Stacktrace dumped to /var/lib/netwitness/config-management/cache/chef-stacktrace.out
[2019-04-16T20:55:32+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-04-16T20:55:32+00:00] ERROR: ruby_block[resolve ips] (nw-dns-client::config line 69) had an error: Resolv::ResolvError: no address for 889e5752-6ae3-4286-a944-c182
33f4ccbc
[2019-04-16T20:55:32+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)


Thus, the install/upgrade fails.
CauseThe reason can be because the target host is unable to communicate to the Admin Server on port 53 as it is attempting to use the dnsmasq service on the Admin Server to resolve, in this case, 889e5752-6ae3-4286-a944-c182 33f4ccbc. This is the salt minion id of the admin server. You can see this by running "cat /etc/salt/minion" on the Admin Server to compare.
Example output:

[root@S5-NWAdmin ~]# cat /etc/salt/minion
master: localhost
hash_type: sha256
log_level: info
id: 889e5752-6ae3-4286-a944-c18233f4ccbc


 
ResolutionIf possible, configure any firewalls between the target host and the Admin Server host to be able to communicate on port 53.

If this is not possible, the workaround is to include the minion id in the /etc/host file on the component hosts.
WorkaroundTake the example /etc/hosts file from an Endpoint Hybrid host.
 

[root@S5-ENDPOINTHYB ~]# cat /etc/hosts
127.0.0.1   S5-ENDPOINTHYB localhost localhost.localdomain localhost4 localhost4.localdomain4 500081a7-f678-45ef-8def-0d416a10e415
::1         S5-ENDPOINTHYB localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.102    nw-node-zero


Edit /etc/hosts and add the node id, just like you saw in the error, next to nw-node-zero

[root@S5-ENDPOINTHYB ~]# cat /etc/hosts
127.0.0.1   S5-ENDPOINTHYB localhost localhost.localdomain localhost4 localhost4.localdomain4 500081a7-f678-45ef-8def-0d416a10e415
::1         S5-ENDPOINTHYB localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.102    nw-node-zero 889e5752-6ae3-4286-a944-c18233f4ccbc

Then, you may attempt to the upgrade once more while tailing the /var/log/netwitness/config-management/chef-solo.log and see if you bypass this error.

A Very Special Note about resolv.conf:
In the 11.3 Release, we are making the /etc/resolv.conf an immutable file. If you are unable to reach the Admin Server on port 53 or your component host uses a different DNS Server from your Admin Server, you will need to edit the local resolv.conf on the component host. To be able to edit the file to change what DNS Servers you query, you must undo this change.

chattr -i /etc/resolv.conf

Once this is done, you can restore your DNS server settings by vi-ing the file. If you are unsure what they were prior to your upgrade, you can check the backup files that chef creates as it goes through it's upgrade run. They will be date-stamped in the file name


[root@S5-ENDPOINTHYB ~]# locate resolv.conf
/etc/resolv.conf
/var/netwitness/config-management/cache/cookbooks/nw-dns-client/templates/default/resolv.conf.erb
/var/netwitness/config-management/cookbooks/nw-dns-client/templates/default/resolv.conf.erb
/var/netwitness/config-management/local-mode-cache/backup/etc/resolv.conf.chef-20181016174034.809106
/var/netwitness/config-management/local-mode-cache/backup/etc/resolv.conf.chef-20190415152127.680013

Please also note that the Admin Server is different. The options in /etc/resolv.conf are being overwritten by what is defined in /etc/netwitness/platform/resolv.dnsmasq. If you want to change the Admin Server's DNS Servers, you will need to modify it there.

 
NotesIf this solution does not work for you and you are still experiencing issues with the /var/log directory filling up, please open a case with RSA Technical Support quoting this KB article.

Attachments

    Outcomes