KEVIN DIENST

Python Script to Email Report on Concentrator Indexes

Discussion created by KEVIN DIENST on Dec 28, 2016
Latest reply on Jan 26, 2017 by KEVIN DIENST

I improved an existing Python script that RSA Professional Services provided me a while back. John Snider

 

I am pasting here for your reference. It is useful to run against your concentrators from say the SA head unit to collect all the indexes max meta values and email you a report. 

 

It has built@ in help via --help and requires modification before use in your environment. 

1. Update lines 18 and 19 to set the sender and receiver email addresses. 

2. Update line 27 so it points to a flat file (one per line) of either the IP address, hostname or alias of your concentrators. 

3. Update line 28 if you'd like to set the appropriate location for your log file to be written out to. 

4. Update line 29, 30 to set the service account credentials used to authenticate. Yes it is in plaintext on this server, no I don't have an alternative unless RSA's API will accept tokens and offer a way to generate them. 

5. Update line 233 to set your internal SMTP server or set an open relay if you use one. This script doesn't support SMTP authentication to send mail. Assumes open internal relays. 

 

Two ways to run it. 

 

1. Individual concentrator as such:

index_profile.py --host sa-concentrator1

2. A newline/carriage return delimited file of IP addresses/hostnames:

index_profile.py --file

 

If you want to improve the logging level of the script you can update line 49:

handler.setLevel(logging.INFO)

 

Update the INFO part to DEBUG or WARN for example. 

 

Once I've verified on an individual host it works, I setup in crontab to run every 4 hours. I check the emails daily and see if any meta keys pop out. Keep in mind unless you know exactly how frequently your indexes roll, it can be hard to determine how frequently to run the script. 

 

#!/usr/bin/env python
# Author: kddiens
# Changelog:
# 03/29/2016 - kdd - created off sample code from RSA, added email function and formatting
# 03/30/2016 - kdd - restructing into a loop for main so we can go across multiple hosts
# 03/31/2016 - kdd - fixes and cleanup, adding script location to html output
# 12/21/2016 - kdd - Adding internal script logging to ease updating, writing, etc
###########################################################################################

import requests, csv, sys, argparse, json, smtplib, time, datetime, os, logging
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

########################
### Global Variables ###
########################

sender = 'sample@domain.com'
receiver = 'sample@domain.com'
today = datetime.datetime.today()
now = datetime.datetime.now()
t1 = now.strftime("%H:%M:%S")
t2 = now.strftime("%H")
d1 = today.strftime("%Y-%m-%d")
body = []
hosts = []
filename = os.path.join("/root/scripts/applianceList/concentratorlist")
log_file = '/root/scripts/check-indexes/index_profile_concentrators' + str(d1) + '_' + str(t2) + '.log'
user = 'serviceaccountname'
password = 'SECRET PASSWORD'
port = "50105"
language_payload = { 'msg' : 'language', 'force-content-type' : 'text/plain', 'expiry' : '600' }
inspect_payload = { 'msg' : 'inspect', 'force-content-type' : 'application/json', 'expiry' : '600' }

###############
### Logging ###         
###############

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

#set debug logging
#logging.basicConfig(level=logging.DEBUG)
#logger.setLevel(logging.DEBUG)

# create file handler for logger

handler = logging.FileHandler(log_file)
handler.setLevel(logging.INFO)

# create logging format

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# add the handles to the logger

logger.addHandler(handler)

###############
#     MAIN    #
###############

def main():

    parser = argparse.ArgumentParser()
    parser.add_argument("--host", help="Host IP or Name", dest="host", action="store")
    parser.add_argument("--file", help="Input File", dest="fflag", action="store_true")
    args = parser.parse_args()

    if not args.host and not args.fflag:
        logger.info('Hostname not specified and file flag not set, unable to continue...')
        sys.exit("Hostname not specified with --host and input file flag not defined via --file")

    if args.fflag and args.host:
        logger.info('Hostname and file flag not null, these are exclusive arguments, unable to continue..')
        sys.exit("--file and --host parameters are exclusive, they cannot be combined, try again")

    if args.host:
        logger.info('Hostname exists, passing param to function for individual host %s...' % (args.host))
        body.append("<br>The following metakeys on <b>%s</b> are over used and require index config custom updates: <br>" % (args.host))
        #collect_meta(args.host,args.ssl) potential to add SSL flag during runtime to toggle if HTTP or HTTPS is used
        collect_meta(args.host)

    elif args.fflag:
        logger.info('File flag set, using file declared in global variable %s...' % (filename))
        logger.info('Changing state to build_hosts() function.')

        build_hosts()
        filtered_hosts = filter(None, hosts)
        for host in filtered_hosts:
            body.append("<br>The following metakeys on <b>%s</b> are over used and require index config custom updates: <br>" % (host))
            #collect_meta(host,args.ssl)
            collect_meta(host)
   
    email(body)

############################################################
## Read in concentrator list line by line and create list ##
############################################################

def build_hosts():
    logger.info('Attempting to open file %s to build hosts', filename)
    f = open(filename, 'r')
    for line in f.read().split('\n'):
        logger.info('Adding host %s from file into hosts array', line)
        hosts.append(line)
    f.close()

###########################################
## Function to collect the meta per host ##
###########################################

def collect_meta(host):
    logger.info('Host passed to function is %s', host)

    # Construct the URI, will add path later
    host_uri = "http://" + host + ":" + port

    logger.info('Host URI constructed is %s', host_uri)

    # Do a try catch and get a HTTP response from the system
   
    try:
        index_language = requests.get(host_uri + '/index',
                verify=False,
                auth=(user, password),
                params=language_payload,
                timeout=5)
        logger.info('Full URL to locate index langauge is %s', index_language.url)

        if index_language.status_code == 200:
            logger.info('w00t! HTTP 200 response!')
        elif index_language.status_code != 200:
            logger.warn('Program received status code %s for host %s using URL %s', index_language.status_code, host, index_language.url)
            body.append("WARNING: Cannot reach URL: %s on host %s <br>" % (index_language.url,host))

    except requests.exceptions.Timeout as e:
        logger.warn('Network timeout trying to reach host %s, this may be due to firewall restrictions or an issue with the service we are trying to contact.', host)
        body.append("WARNING: Host %s is unreachable! <br>" % (host))
        return
   
    try:
        index_inspect = requests.get(host_uri + '/index',
                verify=False,
                auth=(user, password),
                params=inspect_payload,
                timeout=5)
        logger.info('Full URL to locate index inspect page is %s', index_inspect.url)

        if index_inspect.status_code == 200:
            logger.info('w00t! HTTP 200 response!')
        elif index_inspect.status_code != 200:
            logger.warn('Program received status code %s for host %s using URL %s', index_inspect.status_code, host, index_inspect.url)
            body.append("WARNING: Cannot reach URL: %s on host %s <br>" % (index_inspect.url,host))

    except requests.exceptions.Timeout as e:
        logger.warn('Network timeout trying to reach host %s, this may be due to firewall restrictions or an issue with the service we are trying to contact.', host)
        body.append("WARNING: Host %s is unreachable! <br>" % (host))
        return

    # Correctly splits and formats the text we get from the index_language HTTP response

    langreader = csv.DictReader(index_language.text.split("\n"),
            delimiter=",",
            fieldnames=['key', 'description', 'format', 'level', 'valueMax'])

    # For loop over our new list of meta and update metakey and metamax values from it into an array.

    for meta in langreader:
        metakey = meta['key']
        metamax = meta['valueMax']

    # Iterate over JSON we get from index_inspect page

    inspectreader = json.loads(index_inspect.text)
    for line in inspectreader['params']:
        metavalues = line.get('pages')
        if str(line.get('key')) == str(metakey):
            if metavalues > "0" and metamax > "0":
                if metamax.isdigit():
                    percent = int(metavalues) * 100 / int(metamax)
                #DEBUG purposes print str(metakey) + " : " + str(metamax) + " : " + str(metavalues) + " : " + str(percent) + "% Used"

                ###### Call warning function ######
                warning(metakey,percent,host,metamax,metavalues)

##############################################
## Function to check values and build array ##
##############################################

def warning(metakey,percent,host,metamax,metavalues):
    metakey = str(metakey)
    metamax = str(metamax)
    metavalues = str(metavalues)

    if percent >= 100:
        percent = str(percent)
        body.append("%s, Max Value: %s, Actual Value: %s, %s percent over <br>" % (metakey, metamax, metavalues, percent))
        logger.info('%s : %s : %s : %s % Used', metakey, metamax, metavalues, percent)

#################################################
## Function to build email message and send it ##
#################################################

def email(body):
    msg = MIMEMultipart('alternative')
    msg['Subject'] = 'Max index report - ' + d1 + ' ' + t1
    msg['From'] = sender
    msg['To'] = receiver
    msg.preamble = 'Max index report'

    html = """\
    <html>
      <head><b> Max Index Report </b></head>
      <body>
        <p>
        <b> Location: /root/scripts/check-indexes/index_profile.py </b>
        <br>
        <b> Report Generated on %s at %s </b>
        %s
        <br>
        </p>
      </body>
    </html>
    """
%(d1, t1, ''.join(body))

    message = MIMEText(html, 'html')
    msg.attach(message)

    try:
        logger.info('Creating SMTP object...')
        smtpObj = smtplib.SMTP('internal.smtp.server.domain.com')
        logger.info('Send the mail object over to default SMTP server...')
        smtpObj.sendmail(sender, receiver, msg.as_string())
        logger.info('Successfully sent email')
    except SMTPException:
        logger.warn('Error: unable to send email')

## Run main ##

if __name__=='__main__':
    main()

Outcomes