Using RSA Logs and/or Packets to Send or Receive Data from/to LogStash – Putting it all together - Demonstration

Document created by Thomas Jones Employee on Jan 7, 2021Last modified by Thomas Jones Employee on Jan 7, 2021
Version 7Show Document
  • View in full screen mode

What is LogStash:

LogStash is an Elastic product that can collect, parse, and transform logs to be presented to some type of output such as an Elastic Stack or a RSA Decoder or Virtual Log Collector.

https://www.elastic.co/downloads/logstash-oss

Why LogStash:

Depending on the environment, LogStash can act as an intermediary to process data from various sources and send it to a specified destination, called an output.  I can think of three simple examples for the RSA use case.  The first example is sending data from the data lake to Netwitness. The second is sending data from Netwitness to other sources. Finally, it can also be configured to collect data from various sources and send that data to Netwitness or the data lake.

https://en.wikipedia.org/wiki/Data_lake

RSA LogStash Components:

As mentioned above there are several scenarios.  Depending on the configuration ether the Codec or Export Connector will be used.

LogStash Codec:

https://community.rsa.com/docs/DOC-113777

The Codec is used forward Logstash events to the NetWitness Platform  in RFC-5424 format, you need to install the NetWitness codec on your system and refer to it in your output plugin configuration.

Example:

         Data Lake/Warehouse, Syslog, etc --> LogStash --> RSA Decoder

LogStatsh Export Connector:

             https://community.rsa.com/docs/DOC-114533

             Logstash Output plugin to send the input events to a data warehouse destination.

             Example:

                     RSA Decoder --> LogStash --> Data Lake/Warehouse

Supporting Documentation:

RSA NetWitness 11.5 Logstash Codec Download 

Export Connector Install: Configure Logstash Output Plugin

Logstash:Install Logstash 

Logstash Integration Guide for RSA NetWitness® Platform 11.5 

The Process:

The process is fairly simple but LogStash is a product external to RSA and managed by Elastic.  Because of this, it will be necessary to download LogStash and create its own virtual machine.  This demonstration will provide a foundation to learn more about the process and how everything interacts.

Preparation for the demonstration:

Download the CentOS 7 iso.

      CentOS Mirrors List 

Download LogStash if you want to manually install it.  This demo will pull it directly using the rpm –import method.

      Download Logstash Free • Get Started Now | Elastic 

Once CentOS is installed, the following items are going to be required for the Demo.  The actual install, may be different, so please note this.

Minimal Install of Centos 7

               The minimal install is all that the demo requires

add jre for keytool

               Keytool is required for the demo certificate process

yum install java-11-openjdk-devel

yum install mlocate

               mlocate is used to quickly search for files

----> Begin initial setup

Demonstration

 

Download and install the public signing key:Insert into logstash.repo

   rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

 

   vi /etc/yum.repos.d/logstash.repo

Copy and paste into the new file

[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

 

The repository is ready for use. You can install it with:

yum install logstash
systemctl enable logstash.service #Adds to startup
reboot

Check the LogStash status to make sure the LogStash service automatically starts after the reboot

systemctl status logstash

Download and copy the connector to tmp (WinSCP) - netwitness-export-connector-1.1.0.zip

systemctl stop logstash
cd /usr/share/logstash
bin/logstash-plugin install file:///tmp/netwitness-export-connector-1.1.0.zip

Validate the connector is installed

bin/logstash-plugin list

Add the following to the conf file (this initiates data pull via the decoder api on the decoder 50104)

vi /etc/logstash/conf.d/netwitness-192.168.2.121-input.conf
input {
netwitness_export_connector {
host => "192.168.x.x" #Decoder IP Address
username => "admin" #API username
password => "your_password" #API password
decoder_type => "decoder"
}
}

Clean up the input.conf to make sure there are no characters that cannot be seen                       

sed -i -e 's/\r$//' /etc/logstash/conf.d/netwitness-192.168.x.x-input.conf

Restart LogStash to load the input.conf

systemctl start logstash
tailf /var/log/logstash/logstash-plain.log... if you see regex errors check E.3 again
Look for a different sessionid

     

Test with TCPDump

cd /tmp
tcpdump -i eth0 -A -nvvv port 50004 -s 65535 -w logstashDump.txt
tcpdump -r logstashDump.txt

     

Example History for the export connector

[root@localhost conf.d]# history
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vi /etc/yum.repos.d/
vi /etc/yum.repos.d/logstash.repo
yum install logstash
systemctl enable logstash.service
reboot
systemctl status logstash
systemctl stop logstash
cd /usr/share/logstash
bin/logstash-plugin install file:///tmp/netwitness-export-connector-1.1.0.zip
systemctl start logstash

 

----> Begin with the certificate section 

Demonstration: No Audio

 

On the LogStash Server

If there is an existing certificate authority proceed to step 2. (Step 1) - The certificates can be called what ever fits the environment.

openssl genrsa -out CA-key.pem 2048 
openssl req -new -key CA-key.pem -x509 -days 2000 -out CA-cert.pem

Resume Certificate Process (Step 2)

openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -out signingReq.csr (set a password)
openssl x509 -req -days 1000 -in signingReq.csr -CA CA-cert.pem -CAkey CA-key.pem -CAcreateserial -out server-cert.pem
openssl pkcs12 -export -in server-cert.pem -inkey server-key.pem -certfile CA-cert.pem -out logstash-input-netwitness.p12 (set export password)
mkdir /etc/pki/logStashDecoder (on the logstash server)

On the decoder

scp /etc/pki/nw/trust/truststore.pem root@192.168.x.x:/etc/pki/logStashDecoder

if known_host contains a key for this host you may want to delete it.

vi ~/.ssh/known_hosts

LogStash Server

keytool -importcert -keystore logstash-input-netwitness.p12 -trustcacerts -alias nw-inter -file /etc/pki/logStashDecoder/truststore.pem -storetype PKCS12 (enter password)
Trust this certificate: yes

 

cp logstash-input-netwitness.p12 /etc/logstash
chown logstash:logstash /etc/logstash/logstash-input-netwitness.p12

Curl to the Decoder API

curl -X POST -d server-cert.pem https://192.168.2.121:50104/sys/trustpeer
curl -X POST -d server-cert.pem https://192.168.2.121:50104/sys/caupload

alternative to the curl command above - If using the rest interface is preferred (trustpeer and caupload)

https://192.168.x.x:50104/sys/trustpeer

https://192.168.x.x:50104/sys/caupload

copy and paste the server-cert.pem

Check the certificate exists on the Decoder

cd /etc/netwitness/ng/decoder/trustpeers

Change the keystore password on logstash - current password is changeit

keytool -storepasswd -keystore /usr/lib/jvm/java-11-openjdk-11.0.9.11-0.el7_9.x86_64/lib/security/cacerts
Untrusted certificate from above (ex. your_password)
keytool -importcert -file /etc/pki/logStashDecoder/truststore.pem -keystore /usr/lib/jvm/java-11-openjdk-11.0.9.11-0.el7_9.x86_64/lib/security/cacerts -alias nw-core-cert -storepass your_password


   systemctl restart logstash
   tailf /var/log/logstash/logstash-plain.log

              

 

Please let me know your thoughts or suggestions, as this is a work in progress.

Attachments

    Outcomes