Running upgrade-cli-client command failed within few seconds without any specific error while upgrading a host from 22.214.171.124 to 126.96.36.199. No event is logged in /var/log/netwitness/config-management/chef-solo.log on the target host.
/var/log/messages showed the salt-minion service failing to connect to the Salt Master.
Aug 13 01:17:02 NWVLC salt-minon: [ERROR ] Minion unable to successfully connect to a Salt Master.
curl -v nw-node-zero:4505 and curl -v nw-node-zero:4506 confirmed no connectivity issue to node zero.
Re-provisioning the host by following KB36443 failed as the enabling process for the discovered host appeared to be stuck.
/var/log/salt/master.log showed messages like below.
2021-08-13 02:18:17,010 [salt.transport.mixins.auth:138 ][ERROR ] AES key not found
2021-08-13 02:52:16,859 [salt.master :1365][WARNING ] Salt minion claiming to be 39a63bb1-1523-45bb-8bc1-a38d6a884b12 attempted to communicate with master, but key could not be read and verification was denied.
2021-08-13 03:19:10,590 [salt.transport.mixins.auth:388 ][INFO ] Authentication failed from host 39a63bb1-1523-45bb-8bc1-a38d6a884b12, the key is in pending and needs to be accepted with salt-key -a 39a63bb1-1523-45bb-8bc1-a38d6a884b12
The issue can occur when /etc/hosts of the target host contain an incorrect/not current hostname for the localhost.
For example, having the old host name, SAVLC, in /etc/hosts can cause the Salt Master service to deny the authentication request from the Salt Minion service.
In order to resolve the issue, please ensure /etc/hosts, /etc/hosts.netwitness and /etc/hosts.user contain the current hostname. After the files are being updated, run nwsetup-tui and discover/enable the host.
If the issue continues, try the steps below and discover again or follow KB36443 to completely remove the host and start fresh.
Move the /etc/salt/pki/minion/minion_master.pub file to the /tmp directory.