Joshua Randall

Automating Meta Key Updates with UDM

Blog Post created by Joshua Randall Employee on Apr 9, 2018

While the release of the Unified Data Model (UDM) has given us a unified meta key foundation on which to build moving forward (awesome!), it has also opened an administrative can of worms (not so awesome...).

 

With these new and/or modified meta keys comes the challenge of combing through your NetWitness architecture to find all the places that the discontinued meta exist, identifying the discontinued keys that you want to change, and then actually changing them. We can’t automate this entire process yet, but we can still automate some to make our lives easier.

 

One of the primary places that meta keys live within NetWitness is the custom XML file that allows for tuning and adding to the default out-of-the-box meta. In the UI, these files are accessible at Admin (or Administration) → Services → <serviceName> → Config → Files:

Custom XMLs in the UI

 

And on disk at /etc/netwitness/ng/index-<serviceName>-custom.xml, (Log Decoders have an additional custom XML at /etc/netwitness/ng/envision/etc/table-map-custom.xml):

Custom XMLs in the Filesystem

Custom XMLs in the Filesystem (Table Map)

 

We could search through and update these files manually for every discontinued meta key...but frankly, that would be an enormous headache and a waste of time, which is why I put together this script to do it instead.

 

Before running the script, go to the UDM page on RSA Link (https://community.rsa.com/community/products/netwitness/rsa-content/udm) and check out the table of Discontinued Meta (https://community.rsa.com/community/products/netwitness/rsa-content/udm#deprecated). Copy the contents of this table (with or without the header – the script will omit that line if you do include it) into a text file. No modification of this copied table is necessary – again, the script will take of that for us.

Discontinued Meta - UDM

 

Any discontinued meta keys from this table that do not have a specific 1-to-1 replacement meta key, such as orig_ip or any of the risk.* keys, will also be omitted when the script runs.

 

Next, copy this text file and the script to the filesystem of the appliance that you want to run it on (Log/Decoder, Log/Concentrator, Log/Hybrid, Archiver, or Broker), and make the script executable.

 

The script will require two arguments – the name of the text file that you copied the Discontinued Meta table into, and the name of the custom XML that you want to modify:

 

# python <scriptName.py> <text_file_with_copied_table.txt> <target_custom_file.xml>

 

For example:

 

# python replaceDiscontinuedMeta.py filename.txt index-concentrator-custom.xml

# python replaceDiscontinuedMeta.py filename.txt index-archiver-custom.xml

# python replaceDiscontinuedMeta.py filename.txt table-map-custom.xml

 

The script will ask whether to perform a dry run replacement or to do it for real. If run as a dry-run, you will get an output of all the discontinued meta keys that were identified within the target custom XML, as well as the new meta key that replaces it in the UDM.

 

If you do not choose the dry-run option, the script will give you the option to view each discontinued meta key and the corresponding new meta key and accept or deny its replacement, or to simply replace everything without any further prompts.

Script Options

Script Options 2

If the actual replacement(s) are accepted, the script will backup the original custom XML before making any changes.

 

Once complete, I recommend that you compare the new and original files using your diff tool or utility of choice to verify that everything proceeded without error. And as a reminder, you will need to restart the service for these changes to take effect.

 

Happy UDM'ing!

Outcomes