AnsweredAssumed Answered

Is the ESA Advanced Threat Detection Who IS Service Down?

Question asked by David Waugh on Nov 1, 2018
Latest reply on Nov 26, 2018 by Marinos Roussos

Like • Show 0 Likes0
Comment • 15

Hi I check every hour if the whois service is returning a response for google.com

Sometime between 04:14 and 005:14 UTC on 1st November 2018, the service no longer seems to be working.

I can request an auth token, but dont get any response:

./cloud-whois-bank.sh google.com
Authenticate:
curl -sk -H "Content-Type: application/json" -X POST -d "{"X-Auth-Username":"BLAH","X-Auth-Password":"BLAH"}" "https://cms.netwitness.com/authlive/authenticate/WHOIS" -D /tmp/resp_headers.yQzaET -o /dev/null
Query: /usr/bin/curl -sk -H "Content-Type: application/json" -H "X-Auth-Token: eyJpc3MiOiJMSVZFX0FVVEhFTlRJQ0FUSU9OIiwic3ViIjoiV0hPSVMiLCJhdWQiOnsiTGl2ZS1Vc2VybmFtZSI6ImJhbmtvZmVuZ2xhbmRsaXZlQGJhbmtvZmVuZ2xhbmQuY28udWsifSwiZXhwIjoxNTQxMDc0MDg3MTA3LCJhbGciOiJIbWFjU0hBMjU2In0=.IfBp2rdB4oMJOCyxo42E8925g0iI+KC3jfvW12hxd/k=" "https://cms.netwitness.com/whois/query/google.com" | tr -d '\r' | python -m json.tool
No JSON object could be decoded
[328779@HO-SA-ESA ~]$ ./cloud-whois-bank.sh google.com/usr/bin/curl -sk -H "Content-Type: application/json" -H "X-Auth-Token: eyJpc3MiOiJMSVZFX0FVVEhFTlRJQ0FUSU9OIiwic3ViIjoiV0hPSVMiLCJhdWQiOnsiTGl2ZS1Vc2VybmFtZSI6ImJhbmtvZmVuZ2xhbmRsaXZlQGJhbmtvZmVuZ2xhbmQuY28udWsifSwiZXhwIjoxNTQxMDc0MDg3MTA3LCJhbGciOiJIbWFjU0hBMjU2In0=.IfBp2rdB4oMJOCyxo42E8925g0iI+KC3jfvW12hxd/k=" "https://cms.netwitness.com/whois/query/google.com"

Can anyone confirm if the whois service is actually working for them?

No one else has this question

Outcomes

15 Replies

David Waugh Nov 2, 2018 4:06 AM
Mark Correct
Correct Answer
It still seems down.....
Like • Show 0 Likes0
Actions
Edward Darnell Nov 2, 2018 8:25 AM
Mark Correct
Correct Answer
We have had issues with the WHOIS service being up and down since early October and submitted a ticket Oct 11th about same issue. No resolution really ever came out of it for the issue.

Service always seemed to be up more than down, this one seems prolonged.
1 person found this helpful
Like • Show 1 Like1
Actions
- David Waugh @ Edward Darnell on Nov 2, 2018 9:33 AM
  Mark Correct
  Correct Answer
  Agreed. I opened a ticket previously and was told that this would be monitored.
  
  This was the response I got in ticket 01181735 on 14th June 2018
  
  "The Whois service experienced issues during that time frame. Due to malfunction of the monitoring system, the issue has not been identified immediately. The SaaS team addressed the issue with Whois service, and then made sure the monitoring system monitors the service properly."
  
  These things happen, but being down for over 24 hours without anyone doing anything is a little troubling. It is also a design flaw that I have previously highlighted, that if the WhoIS service is down the ESA should not get increasingly behind in sessions until it eventually stops.
  
  Anyone from RSA care to comment?
  Like • Show 1 Like1
  Actions
David Waugh Nov 2, 2018 11:41 AM
Mark Correct
Correct Answer
Just had the first confirmation from RSA Support that the WhoIs service is down.
Like • Show 0 Likes0
Actions
David Waugh Nov 2, 2018 12:00 PM
Mark Correct
Correct Answer
Its back!!!
Like • Show 1 Like1
Actions
Marinos Roussos Nov 5, 2018 4:38 AM
Mark Correct
Correct Answer
I feel sorry for you that you have been trying to get answers. I've noticed , every time you've faced issues with the Whois service and asking for feedback, then it becomes clear that this is not monitored by RSA and that the phone is dead on the other end.

I would understand RSA's argument that this is a free service but if it's not reliable then it can only be used in a lab.

To PMs: In security, there is a big difference when it comes to software, you cannot use untested/unreliable/not-monitored/toy software for conducting security operations. Scoring high on Gartner is not enough to sell if customers can't trust the software and the people supporting it.
1 person found this helpful
Like • Show 1 Like1
Actions
- David Waugh @ Marinos Roussos on Nov 5, 2018 4:43 AM
  Mark Correct
  Correct Answer
  Not only is the fact that it is not sufficiently monitored that is disappointing, it is the fact that when it is down, the ESA seems to wait until it gets a response. This causes a backlog of sessions and the ESA quickly get behind by many millions of sessions. This then takes many hours for it to catch back up.
  
  The ESA is a critical part of the solution as it generated alerts on suspicious activity.
  
  I want to know that I can rely on the ATD service. How long is it acceptable for it to be down? 1 hour, 2 hours? 24 hours?
  1 person found this helpful
  Like • Show 0 Likes0
  Actions
  - Marinos Roussos @ David Waugh on Nov 5, 2018 5:01 AM
    Mark Correct
    Correct Answer
    The fact that ESA is stopping when Whois goes down is a bug. If RSA don't want to admit and fix this and as a workaround suggest you to stop using the Whois service then they are worthy or their fate.
    
    Monitoring a free service that an expensive paid service is depending on(not to mention that is crucial for security operations), should be a given and not having to cry for help on a forum or raise a RFE or other silly things like that.
    Like • Show 1 Like1
    Actions
David Waugh Nov 19, 2018 4:29 AM
Mark Correct
Correct Answer
Its down again. Last successful check was 04:13 on Saturday 17th November UTC
Like • Show 0 Likes0
Actions
- Marinos Roussos @ David Waugh on Nov 19, 2018 8:39 AM
  Mark Correct
  Correct Answer
  The fact that noone from RSA have ever acknowledged or replied to this ongoing issue with their software is a reflection of their idea of business critical software and after-sales Support. Actions speak louder than words.
  
  I would personally disable Whois and stop bothering with it especially if it can take ESA down for which I might be responsible to maintain. Not to mention the reliance of the SOC Analysts on a reliable alerting mechanism.
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Marinos Roussos on Nov 19, 2018 11:03 AM
    Mark Correct
    Correct Answer
    Hi Marinos, yes after the last time it went down we disabled it as we can't rely on it. If we hadn't, then as the latest outage happened early on a Saturday morning we would have come in today to find that our ESA is many millions of sessions behind. I have support ticket 01273363 open for this issue.
    
    This is particularly disappointing as I have been told twice that the service is being monitored.I'm not sure how I am able to detect that the service is down, but RSA is not able to?
    
    At the time of writing it is still down. I can obtain an X-Auth-Token but not get a response when I lookup google.com.
    
    Here is my script for checking that this service is down. I run the script hourly and it results in an email being sent to me.
    
    cloud-check.sh
    
    cd /root/Scripts
    echo "ESA Sessions Behind (Should be Zero)" >output.txt
    ./check_esa_sessions_behind.sh >>output.txt
    echo "WhoIS Test Lookup to Google" >>output.txt
    ./cloud-whois-bank.sh google.com >>output.txt 2>&1
    python email-whois.py
    
    more check_esa_sessions_behind.sh
    
    #!/bin/bash
    #David Waugh ex RSA
    #This script will display the current maximum value of the sessionsBehind for an
    y aggregating source of the ESA
    E_OK=0
    E_WARNING=1
    E_CRITICAL=2
    E_UNKNOWN=3
    
    #Get the stream names
    Perfdata=""
    Output=""
    behind=$(echo -e "cd /Workflow/Source/nextgenAggregationSource\ndump\nexit" |
    /opt/rsa/esa/client/bin/esa-client |grep sessionsBehind | cut -d ":" -f 2 |cut
    -d " " -f 2 |cut -d "," -f 1 |sort -nr |head -n 1)
    Output+="Sessions behind: "
    Output+=": "
    Output+=$behind
    Output+=" "
    Perfdata+="Sessions_behind"
    Perfdata+=\=
    Perfdata+=$behind
    Perfdata+=""
    #echo $Output\|$Perfdata\;\;
    #exit 0
    show_help() {
    echo "$0 -w VALUE -c VALUE | -h"
    echo
    echo "This plug-in is used to be alerted when maximum ESA behind session
    s is reached"
    echo
    echo " -w/c Sessions behind integer"
    echo " To warn when 200 sessions behind and critical when 300 sessions b
    ehind"
    echo " example: $0 -w 200 -c 300"
    }
    # process args
    while [ ! -z "$1" ]; do
    case $1 in
    -w) shift; WARNING=$1 ;;
    -c) shift; CRITICAL=$1 ;;
    -h) show_help; exit 1 ;;
    esac
    shift
    done
    # check input parameters so we can continue !
    sanitize() {
    # check thresholds
    if [ -z "$WARNING" ]; then
    WARNING=500
    #echo "Need warning threshold"
    #exit $E_UNKNOWN
    fi
    if [ -z "$CRITICAL" ]; then
    CRITICAL=1000
    #echo "Need critical threshold"
    #exit $E_UNKNOWN
    fi
    }
    # check args
    sanitize
    OUTPUT=""
    EXITCODE=$E_OK
    # check behind
    if [ $behind -gt $WARNING ]; then
    if [ $behind -gt $CRITICAL ]; then
    OUTPUT="CRITICAL behind (>$CRITICAL), "
    EXITCODE=$E_CRITICAL
    else
    OUTPUT="WARNING behind (>$WARNING), "
    EXITCODE=$E_WARNING
    fi
    else
    OUTPUT="OK"
    fi
    
    #echo "${OUTPUT} ${Output}|${Perfdata};;;;"
    echo "${behind}"
    exit $EXITCODE
    
    more cloud-whois-bank.sh
    #!/bin/bash
    #set -x
    export http_proxy=http://1.2.3.4:8080/
    export https_proxy=https://4.5.6.7:8080/
    if [[ $# -eq 0 ]] ; then
    echo "$0 requires at least the name of the domain to look up"
    exit 1
    fi
    CMD="/usr/bin/curl -sk"
    domain=$1
    username=${2:-"myliveusername"}
    password=${3:-"mylivepassword"}
    whois_host=${4:-"https://cms.netwitness.com"}
    debug=${5:-"Y"}
    resp_headers_file=`mktemp --tmpdir resp_headers.XXXXXX`
    auth_path="authlive/authenticate/WHOIS"
    auth_header="Content-Type: application/json"
    auth_data="{\"X-Auth-Username\":\"${username}\",\"X-Auth-Password\":\"${password
    }\"}"
    query_path="whois/query/$domain"
    getToken() {
    # Authorize your access using a POST of the login parameters
    if [ "$debug" == "Y" ];
    then
    echo -e "Authenticate:\ncurl -sk -H \"${auth_header}\" -X POST -
    d @auth.json -D ${resp_headers_file} -o /dev/null"
    echo -n
    fi
    $CMD -H "${auth_header}" -X POST -d @auth.json "${whois_host}/${auth_pat
    h}" -D ${resp_headers_file} -o /dev/null
    token_header=$(cat ${resp_headers_file} | tr -d '\r' | grep '^X-Auth-Tok
    en:')
    if [ -z "$token_header" ]; then
    echo "X-Auth-Token is empty."
    else
    echo $token_header > ~/.cloud-whois
    fi
    }
    query() {
    # Do the query now
    if [ "$debug" == "Y" ];
    then
    echo "Query: $CMD -H \"${auth_header}\" -H \"${token_header}\" \
    "${whois_host}/${query_path}\" | tr -d '\r' | python -m json.tool"
    echo -n
    fi
    if [ -z "$token_header" ]; then
    echo "Skipping query because no X-Auth-Token."
    else
    $CMD -H "${auth_header}" -H "${token_header}" "${whois_host}/${query_pat
    h}" -D ${resp_headers_file} | tr -d '\r' | python -m json.tool
    rm -f ${resp_headers_file}
    fi
    }
    if [ ! -f ~/.cloud-whois ] || test `find ~/.cloud-whois -mmin +60`
    then
    getToken
    else
    token_header=`cat ~/.cloud-whois`
    echo -e "\n"
    echo "Using existing X-Auth-Token from ~/.cloud-whois."
    echo -e "\n"
    fi
    #query
    query
    
    email-whois.py
    # Import smtplib for the actual sending function
    import smtplib
    # Import the email modules we'll need
    from email.mime.text import MIMEText
    # Open a plain text file for reading. For this example, assume that
    # the text file contains only ASCII characters.
    fp = open('output.txt', 'rb')
    # Create a text/plain message
    msg = MIMEText(fp.read())
    fp.close()
    # me == the sender's email address
    # you == the recipient's email address
    msg['Subject'] = 'Cloud WhoIs Check'
    msg['From'] = 'siem@mydomain.com'
    msg['To'] = 'email1@mydomain.com'
    # Send the message via our own SMTP server, but don't include the
    # envelope header.
    s = smtplib.SMTP('1.2.3.4')
    s.sendmail('siem@mydomain.com', ['email1@mydomain.com','email2@mydomain.com'], msg.as_string())
    s.quit()
    Like • Show 1 Like1
    Actions
    - Marinos Roussos @ David Waugh on Nov 19, 2018 11:11 AM
      Mark Correct
      Correct Answer
      Thanks for sharing the script, even though I would only use Whois in test environment.
      
      Well, saying that they do vs providing evidence that it is actually monitored are two different things. I think someone would have replied if they did. Not to mention that there should have been an e-mail going out to customers regarding the outage or at least some note on Link homepage. So again, actions speak louder than words:)
      
      Anyway, just to play ball with RSA's fixation about RFEs, I created an Idea to make a Service Status page Create a Service Status page for all RSA hosted services
      Like • Show 0 Likes0
      Actions
David Waugh Nov 20, 2018 4:02 AM
Mark Correct
Correct Answer
Still down.....
1 person found this helpful
Like • Show 0 Likes0
Actions
David Waugh Nov 21, 2018 4:14 AM
Mark Correct
Correct Answer
And its backup again on Tuesday 20th November sometime around 16:13.

So the service was down for 4 and half days!!!!!
1 person found this helpful
Like • Show 1 Like1
Actions
- Marinos Roussos @ David Waugh on Nov 26, 2018 4:15 AM
  Mark Correct
  Correct Answer
  Perfect for production environments
  Like • Show 0 Likes0
  Actions