000036657 - Running SOS on RSA NetWitness Version 11.x

Document created by RSA Customer Support Employee on Aug 27, 2018Last modified by RSA Customer Support Employee on Jul 17, 2019
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000036657
Applies ToRSA Product Set: RSA Security Analytics
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.x
Platform: CentOS 7
IssueThe customer would like to get diagnostic information on their 11.x device.
ResolutionRequirements: This utility requires two RPM's. See the "Notes" section of this knowledge-based article before performing resolution steps.

  • The following command can be used to execute the script from the command line: 

    OWB_ALLOW_NON_FIPS=1 sosreport

  • The following command can be used to view the on-screen help menu: 

    OWB_ALLOW_NON_FIPS=1 sosreport --help

  • The following command can be used to skip a plugin (only at customer support request) :


Select Examples:

  • The following command can be used to skip only the plugins 'rsa_nw_mongo' and 'rsa_nw_rest': 

    OWB_ALLOW_NON_FIPS=1 sosreport --skip-plugins=rsa_nw_mongo,rsa_nw_rest

  • The following command can be used to run only the plugin 'rsa_nw_mongo': 

    OWB_ALLOW_NON_FIPS=1 sosreport -o rsa_nw_mongo

  • The following command can be used to make sosreport limit log sizes to 5mb (default: 10mb):

    OWB_ALLOW_NON_FIPS=1 sosreport --log-size=5

    • However, this will only apply to those log files specifically set up to follow this limit.
  • The following command can be used to find a list of active and inactive plugins: 

    OWB_ALLOW_NON_FIPS=1 sosreport -l

  • The following command can be used to get the help menu: 

    [root@saserver1857 ~]# OWB_ALLOW_NON_FIPS=1 sosreport --help
    Usage: sosreport [options]

      -h, --help            show this help message and exit
      -l, --list-plugins    list plugins and available plugin options
      -n NOPLUGINS, --skip-plugins=NOPLUGINS
                            disable these plugins
      --experimental        enable experimental plugins
      -e ENABLEPLUGINS, --enable-plugins=ENABLEPLUGINS
                            enable these plugins
      -o ONLYPLUGINS, --only-plugins=ONLYPLUGINS
                            enable these plugins only
      -k PLUGOPTS, --plugin-option=PLUGOPTS
                            plugin options in plugname.option=value format (see
      --log-size=LOG_SIZE   set a limit on the size of collected logs (in MiB)
      -a, --alloptions      enable all options for loaded plugins
      --all-logs            collect all available logs regardless of size
      --batch               batch mode - do not prompt interactively
      --build               preserve the temporary directory and do not package
      -v, --verbose         increase verbosity
      --verify              perform data verification during collection
      --quiet               only print fatal errors
      --debug               enable interactive debugging using the python debugger
                            specify ticket number
      --case-id=CASE_ID     specify case identifier
      -p PROFILES, --profile=PROFILES
                            enable plugins selected by the given profiles
      --list-profiles       display a list of available profiles and plugins that
                            they include
      --name=CUSTOMER_NAME  specify report name
                            specify alternate configuration file
      --tmp-dir=TMP_DIR     specify alternate temporary directory
      --no-report           disable HTML/XML reporting
      -s SYSROOT, --sysroot=SYSROOT
                            system root directory path (default='/')
      -c CHROOT, --chroot=CHROOT
                            chroot executed commands to SYSROOT [auto, always,
                            never] (default=auto)
      -z COMPRESSION_TYPE, --compression-type=COMPRESSION_TYPE
                            compression technology to use [auto, gzip, bzip2, xz]

    • Examples of the Help Menu options: 
      • enable dlm plugin only and collect dlm lockdumps: 


        # OWB_ALLOW_NON_FIPS=1 sosreport -o dlm -k dlm.lockdump

      • disable memory and samba plugins, turn off rpm -Va collection: 

        # OWB_ALLOW_NON_FIPS=1 sosreport -n memory,samba -k rpm.rpmva=off


Running the sosreport out-of-the-box will yield many errors and force SOS to run in debug mode, due to the BSAFE package on RSA NetWitness 11.0 devices.

Example errors include:

# sosreport
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/hashlib.py", line 129, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib64/python2.7/hashlib.py", line 98, in __get_openssl_constructor
ValueError: error:3207A06D:lib(50):B_HASH_init:cr new
DEBUG:sos:set sysroot to '/' (default)

Other errors include:

plugin rsa_nw_rest does not install, skipping: No module named 'requests.packages.urllib3'
WARNING:sos:plugin rsa_nw_rest does not install, skipping: No module named 'requests.packages.urllib3'

A similar message can also be seen in /sos_logs/sos.log in sos report output.

2018-10-18 01:54:52,515 WARNING: plugin rsa_nw_rest does not install, skipping: No module named 'requests.packages.urllib3'

Note: Above rsa_nw_rest plugin issue is resolved by Step 1. below.

The following fixes will cause SOS to run much more cleanly. If the customer does not wish to perform the following steps, then the SOS package will not generate a .sha256 file (not required) accompanying the .tar.xz file, and the contents of the .tar.xz file may have some items missing. However, the report will run.

To resolve the errors, the customer should perform the following steps:

  1. First, add 'OWB_ALLOW_NON_FIPS=1' in front of the command to execute the report, which removes nearly all the errors being printed to the screen.

  • Example of sosreport run:

    OWB_ALLOW_NON_FIPS=1 sosreport

  • To make this a permanent fix, create a file /etc/profile.d/sos.sh with the following contents: 

    alias sosreport='OWB_ALLOW_NON_FIPS=1 sosreport' 2>/dev/null

    Example Command to create file /etc/profile.d/sos.sh

    printf "alias sosreport='OWB_ALLOW_NON_FIPS=1 sosreport' 2>/dev/null" > /etc/profile.d/sos.sh

  1. The next error is how the RSA system has enabled FIPS. The open-source SOS package is looking in /proc/sys/crypto/fips_enabled to see if it is running on a FIPS enabled system. Our devices, by default, have a ‘0’ in that file and the SOS package wrongly assumes FIPS isn’t enabled. Because of this, when SOS goes to create a checksum file for the generated report, it will fail. Because SOS is recognizing our appliances as non-FIPS enabled, it uses md5 instead of sha256 to create that checksum.

    • The fix for this is to manually modify the SOS code. Though RSA is unable to modify the open-source code, we are able to provide the customers with the necessary information. Customers will have to repeat the following step each time they update the sos RPM (e.g. sos-3.3-5.el7.centos.noarch.rpm) : 


      sed -i.bak 's/"md5"/"sha256"/g' /usr/lib/python2.7/site-packages/sos/policies/__init__.py


Note: Above command creates backup file /usr/lib/python2.7/site-packages/sos/policies/__init__.py.bak to allow for change reversal.

  1. The customer should log out of the ssh session and then log back in.

If the customer does not wish to perform the above steps, at the end of the debug output it will place the following in stdout:

Creating compressed archive...
INFO:sos:[archive:TarFileArchive] finalizing archive '/var/tmp/sos.QX5bXA/sosreport-df.323322345676645-20170915152056' using method 'auto'
INFO:sos:[archive:TarFileArchive] built archive at '/var/tmp/sos.QX5bXA/sosreport-df.323322345676645-20170915152056.tar' (size=20480)
Traceback (most recent call last):
  File "/usr/sbin/sosreport", line 25, in <module>
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1637, in main
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1616, in execute
    return self.final_work()
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1529, in final_work
    checksum = self._create_checksum(archive, hash_name)
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1468, in _create_checksum
    digest = hashlib.new(hash_name)
  File "/usr/lib64/python2.7/hashlib.py", line 111, in __hash_new
    return _hashlib.new(name, string, usedforsecurity)
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips

The customer can find the .tar.xz file in the location mentioned in the above log. In this case, it is /var/tmp/sos.QX5bXA/sosreport-df.323322345676645-20170915152056.tar, however, the customer's location may differ slightly if --tmp-dir parameter is used.