Eric Partington

Threat Intel Integration with MISP and Minemeld

Blog Post created by Eric Partington Employee on Apr 5, 2019

RSA NetWitness has a number of integrations with threat intel data providers but two that I have come across recently were not listed (MISP and Minemeld) so I figured that it would be a good challenge to see if they could be made to provide data in a way that NetWitness understood.

 

Current RSA Ready Integrations

https://community.rsa.com/community/products/rsa-ready/rsa-ready-documentation/content?filterID=contentstatus%5Bpublished%5D~objecttype~objecttype%5Bdocument%5D&filterID=contentstatus%5Bpublished%5D~tag%5Bthreat+intel%5D&sortKey=contentstatus%5Bpublished%5D~subjectAsc&sortOrder=1

 

MISP

Install the MISP server in a few different ways

https://www.misp-project.org/

 

VMWare image, Docker image or on an OS are all available (VMware image worked the best for me)

https://www.circl.lu/misp-images/latest/

 

Authenticate and setup the initial data feeds into the platform

Set the schedule to get them polling for new data

 

Once created and feeds are being pulled in you can look at the attributes to make sure you have the data you expect

 

Test the API calls using PyMISP via Jupyter Notebook

https://github.com/epartington/rsa_nw_misp/blob/master/get-misp.ipynb

  • you can edit the notebook code to change the interval of data to pull back (last 30 days, all data or such to limit impact on the MISP server)
  • You can change the indicator type (ip-dst, domain etc.) to pull back the relevant columns of data
  • You can change the column data to make sure you have what you need as other feed data

 

Once that checks out and you have the output data you want via the notebook you can add the python script to the head server of NetWitness

 

Install PyMISP on the head server of the NetWitness system so that you can crontab the query.

  • Install PyMISP using PIP

(keep in mind that updating the code on the head server could break things so be careful and test early and often before committing this change in production)

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install python-pip
OWB_FORCE_FIPS_MODE_OFF=1 python
OWB_FORCE_FIPS_MODE_OFF=1 pip install pymisp
OWB_FORCE_FIPS_MODE_OFF=1 pip install --upgrade pip
OWB_FORCE_FIPS_MODE_OFF=1 ./get-misp.py
yum repolist
vi /etc/yum.repos.d/epel.repo
change enabled from 1 to 0

Make sure you disable the epel repo after installing so that you don't create package update issues later

 

Now setup the query that is needed in a script (export the Jupyter notebook as python script)

https://github.com/epartington/rsa_nw_misp/blob/master/get-misp.py

 

Crontab the query to schedule it (the OWB is required to work around FIPS restrictions that seem to break a number of script related items in python)

23 3 * * * OWB_FORCE_FIPS_MODE_OFF=1 /root/rsa-misp/get-misp.py > /var/lib/netwitness/common/repo/misp-ip-dst.csv

 

Now setup the NetWitness recurring feed to pull from the local feed location

map the ip-dst values (for this script) to the 3rd column and the other columns as required

 

 

Minemeld

logo

Minemeld is another free intel aggregation tool from Palo Alto Networks and can be installed many ways (i tried a number of installs on different Ubuntu OSes and had difficulties), the one that worked the best for me was via a docker image.

https://www.paloaltonetworks.com/products/secure-the-network/subscriptions/minemeld

https://github.com/PaloAltoNetworks/minemeld/wiki

 

Docker image that worked well for my testing

https://github.com/jtschichold/minemeld-docker

 

docker run -it --tmpfs /run -v /somewhere/minemeld/local:/opt/minemeld/local -p 9443:443 jtschichold/minemeld

to make it run as daemon after testing add the -d command to have it continue running after you exit the terminal

 

After installing (if you do this right you can get a certificate included in the initial build of the container that will help with the Certificate trust to NW) you will log in and set up a new output action to take your feeds and map them to a format and output that can be used with RSA NetWitness.

 

This is the pipeline that we will create which will map a sample threat intel list to an output action so that NetWitness can consume that information

And it gets defined by editing the yml configuration file (specifically this section creates the outboundhcvalues section that NetWitness reads)

https://github.com/epartington/rsa_nw_minemeld/blob/master/minemeld-netwitness-hcvalues.yml

outboundfeedhcvalues:
inputs:
- aggregatorIPv4Outbound-1543370742868
output: false
prototype: stdlib.feedHCGreenWithValue

This is a good start for how to create custom miners

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Using-MineMeld-to-Create-a-Custom-Miner/ta-p/227694

 

Once created and working you will have a second miner listed and the dashboard will update

 

You can test the feed output using a direct API call like this via the browser

https://192.168.x.y:9443/feeds/"$feed_name"?tr=1&v=csv&f=indicator&f=confidence&f=share_level&f=sources

the  query parameters are explained here:

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

 

in this case:

tr=1

translate IP ranges into CIDRs. This can be used also with v=json and v=csv.

v=csv

returns the indicator list in CSV format.

 

The list of the attributes is specified by using the parameter f one or more times. The default name of the column is the name of the attribute, to specify a column name add |column_name in the f parameter value.

 

The h parameter can be used to control the generation of the CSV header. When unset (h=0) the header is not generated. Default: set.

 

Encoding is utf-8. By default no UTF-8 BOM is generated. If ubom=1 is added to the parameter list, a UTF-8 BOM is generated for compatibility.

 

F are the column names from the feed

This command testing drops a file in your browser to look at and make sure you have the data and columns that you want

 

Now once you are confident in the process and the output format you can script and crontab the output to drop into the local feed location on the head server (I did this as i couldn't figure out how to accept the self signed certificate from the docker image).

https://github.com/epartington/rsa_nw_minemeld/blob/master/script-rsa-minemeld.sh

# 22 3 * * * /root/rsa-minemeld/script-rsa-minemeld.sh

Now create the same local recurring feed file to pull in the information as feed data on your decoders.

Define the column to match column 1 for the IP in CIDR notation and map the other columns as required

 

Done

 

Now we have a pipeline for two additional threat data aggregators that you may have a need for in your environment.

Outcomes