How to Decomission a Host from the Puppet Trust Model

3 years ago

Originally Published: 2017-06-14

Article Number

000065064

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 10.4.x, 10.5.x, 10.6.x
Platform: CentOS
O/S Version: EL6
Component Name: Puppet

Issue

How to manually remove host if the "Remove and Repurpose Host" feature in Administration \ Appliances (SA 10.4.X) or Administration \ Hosts (SA 10.5.X and later) is not an option.

Resolution

On the Host to be Removed (if it is still available)
1. Either SSH or use a Console (such as iDRAC console) to connect to node you are removing from the trust model

2. Obtain the puppet node_id
On that Node, obtain the node_id

/etc/puppet/scripts/node_id.py

Example Output:

# /etc/puppet/scripts/node_id.py
b93bdb62-b0e4-4141-a03b-dac803da73a7

Other ways of obtaining puppet node_id

echo $(</var/lib/puppet/node_id)
facter -p node_id

3. Stop services

service collectd stop
service mcollective stop
puppet resource service puppet ensure=stopped
service rabbitmq-server stop

4. Clean-up puppet

rm -rf /var/lib/puppet/ssl
rm -f /var/lib/puppet/state/agent_catalog_run.lock
rm -f /var/lib/puppet/client_data/catalog/$(/etc/puppet/scripts/node_id.py).json
rm -f /var/lib/puppet/yaml/facts/$(/etc/puppet/scripts/node_id.py).yaml
rm -f /var/lib/puppet/yaml/node/$(/etc/puppet/scripts/node_id.py).yaml
rm -rf /var/lib/puppet/reports/$(/etc/puppet/scripts/node_id.py)

5. Remove trustpeers

find /etc/netwitness/ng -name 'trustpeers' -type d -exec rm -rf {} \; 2>/dev/null

On the NetWitness Head Unit Server/SA Server which hosts the puppetmaster:
6. The following 3 commands are optional however may improve the likelihood of success

service puppetmaster restart
service mcollective restart
service rabbitmq-server restart

7. Run delNode.py
Syntax:

/etc/puppet/scripts/delNode.py <node_id>

Example:

/etc/puppet/scripts/delNode.py b93bdb62-b0e4-4141-a03b-dac803da73a7

8. The previous step runs puppet cert clean <node_id>
Check that there are no current requests for this node_id:

ls /var/lib/puppet/ssl/ca/requests -l
ls /var/lib/puppet/ssl/ca/signed -l

9. Remove from puppetmaster inventory.txt

mv /var/lib/puppet/ssl/ca/inventory.txt /var/lib/puppet/ssl/ca/inventory.old

Syntax:

grep -v <node_id> /var/lib/puppet/ssl/ca/inventory.old > /var/lib/puppet/ssl/ca/inventory.txt

Example:

grep -v b93bdb62-b0e4-4141-a03b-dac803da73a7 /var/lib/puppet/ssl/ca/inventory.old > /var/lib/puppet/ssl/ca/inventory.txt

10. Remove references from mongoDB
Syntax:

node_target=<node_id>
applianceId=$(echo 'db.appliance_info.find( { "identity" : "'$node_target'" },{"_id":1} )' | mongo asg | grep -o "ObjectId(.*" | cut -d\" -f2)
echo 'db.appliance_info.remove( { "identity" : "'$node_target'" } )' | mongo asg
echo 'db.service_info.remove( { "applianceId" : "'$applianceId'" } )' | mongo asg
echo 'db.appliance_update.remove( { "identity" : "'$node_target'" } )' | mongo puppet
echo 'db.entitlement.remove( { "identity" : "'$node_target'" } )' | mongo sa
echo 'db.service_info.remove( { "hostId" : "'$node_target'" } )' | mongo les

Example:

node_target='b93bdb62-b0e4-4141-a03b-dac803da73a7'
applianceId=$(echo 'db.appliance_info.find( { "identity" : "'$node_target'" },{"_id":1} )' | mongo asg | grep -o "ObjectId(.*" | cut -d\" -f2)
echo 'db.appliance_info.remove( { "identity" : "'$node_target'" } )' | mongo asg
echo 'db.service_info.remove( { "applianceId" : "'$applianceId'" } )' | mongo asg
echo 'db.appliance_update.remove( { "identity" : "'$node_target'" } )' | mongo puppet
echo 'db.entitlement.remove( { "identity" : "'$node_target'" } )' | mongo sa
echo 'db.service_info.remove( { "hostId" : "'$node_target'" } )' | mongo les

On the Host to be Removed
11. Stop services in upstart

declare -a services_to_stop=( $(find /etc/init \( -name "nw*" -o -name "rsa*" \) -type f -exec basename {} \; | sed 's/.conf$//g' | grep -v -E "(bad|disable|old)" | sort -r) )
for target in "${services_to_stop[@]}"; do $(stop "$target") 2>/dev/null; done

12. Restart/Reboot OS

shutdown -r now

Don't see what you're looking for?

Related Articles

Trending Articles