LC AWS:Step 3 - Deploy Remote Log Collector Service in AWS

Document created by RSA Information Design and Development on Nov 22, 2016Last modified by RSA Information Design and Development on May 4, 2017
Version 10Show Document
  • View in full screen mode
  

This topic tells you how to deploy your remote log collection service in an AWS environment using automated scripts, as well as deploying your remote log collection service manually.

Option 1: Using Scripts to Deploy Remote Log Collector Service

After you configure your Remote Log Collector Service, you need to deploy your Remote Log Collector Service onto a virtual machine (VM) by following the steps provided below.

Note: Using scripts is recommended. If you are using scripts, follow the steps below. Or, to manually deploy your remote log collector service, refer to the steps in the Option 2: Manually Deploy Remote Log Collector Service section.

Log in Using SSH

Before you download scripts, you need to SSH to your virtual machine after you find your IP Address by following the steps below.

  1. To obtain the IP address, select the name of the deployed virtual instance.
  2. In the Description tab, enter the Private IP address of that particular virtual instance.

    The following screen is displayed.

  3. SSH to your virtual instance using an SSH client such as PuTTy. Instructions are provided for PuTTy, but you can use any SSH client.
  4. If you are using PuTTy, run the following command to connect to a remote log collector:

    putty.exe

    The following screen is displayed.

  5. Enter the IP Address using the Private IP Address from the previous screen.
  6. Under Connection type, select SSH.
  7. Click Open.

    The following screen is displayed.

  8. Select Connection > SSH > Auth.
  9. Select Browse to find your Private Key file.
  10. Click Open to connect to your virtual instance.

Disable CentOS Base Repo

To disable the CentOS Base Repo:

  1. Run the following command:

    sudo vi /etc/yum.repos.d/CentOS-Base.repo

  2. Ensure that the contents of the CentOS-Base.repo file in the /etc/yum.repos directory have sections with enabled=0

    File contents should be similar to the example below:

    base
    name=CentOS-$releasever - Base
    mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
    #baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
    enabled=0

Download AWS Scripts

Note: Each script requires you to enter your username and password credentials in https://community.rsa.com.

You can download the AWS scripts onto your machine from the following locations:

WinSCP Instructions to Copy Downloaded Scripts

Once you have downloaded the AWS scripts onto your local machine, you need to SFTP to your virtual machine using an SFTP client such as WinSCP so that you can transfer the scripts to your AWS Log Collector.

To connect to a virtual machine instance with SFTP, start WinSCP by following the instructions below.

In the WinSCP log in dialog box:

  1. Make sure New site node is selected.
  2. In the New site node field, select the SFTP protocol.
  3. Enter the Host name.
  4. Enter the default username.

  5. Leave the password field blank.
  6. Click the Save button to save your settings.
  7. Click the Login button to log in.
  8. Verify the host key by referring to the key that you created in the Create a Key Pair section in Step 1 - Log in to AWS and Create an Instance.

  1. Copy the scripts that you downloaded from your laptop to the home directory of the Remote Log Collector.

Providing Execute Permissions to the Scripts

  1. Use the SSH instructions to SSH in to your AWS machine described in the section Log in Using SSH.
  2. Change the user to the home directory using the following command:

    cd ~

  3. Use the following command to provide execute permissions before running these scripts:

    sudo chmod +x aws_vlc_preinstall.sh
    sudo chmod +x aws_vlc_preinstall.sh
    sudo chmod +x aws_vlc_start_services.sh

Configure Firewall Permissions to Allow Communication

Note: You need to open ports between your Remote Log Collector and your Security Analytics Server, and you also need to open ports between your Log Collector on your Log Decoder.

Configure your firewall(s) to allow communication between the Remote Log Collector and AWS, and Security Analytics components listed in the following table.

                                                                                                                           
From HostTo HostTo Ports (Protocol) 

Security Analytics Server

Remote Log Collector

56001 (TCP) or

50001 (TCP)

SSL or

Non-SSL

Security Analytics Server

Remote Log Collector

50101 (TCP)

REST (Optional)

Security Analytics Server

Remote Log Collector

5672 (TCP)

RabbitMQ

Security Analytics Server

Remote Log Collector

50055 (TCP)

RSA-SMS

Security Analytics Server

Remote Log Collector

50056 (TCP)RSA-SMS

Remote Log Collector

Security Analytics Server

8140 (TCP)

Puppet

Security Analytics Server

Remote Log Collector

61614(TCP)

MCollective

Remote Log Collector

Security Analytics Server

61614(TCP)

MCollective

Security Analytics Server

Remote Log Collector

15671(TCP)

RabbitMQ

Remote Log Collector

Security Analytics Server

15671(TCP)

RabbitMQ

Security Analytics Server

Remote Log Collector

61614 (TCP)

MCollective

Remote Log Collector

Security Analytics Server

61614 (TCP)

MCollective

Security Analytics Server

Remote Log Collector

15671 (TCP)

RabbitMQ

Remote Log Collector

Security Analytics Server

15671 (TCP)

RabbitMQ

In Pull Mode:    

Log Collector (on Log Decoder)

Remote Log Collector

5671 (TCP)

RabbitMQ

In Push Mode:    

Remote Log Collector

Log Collector (on Log Decoder)5671 (TCP)

RabbitMQ

Run AWS Scripts

To use scripts to set up your Remote Log Collector service in AWS, complete the following steps.

  1. Change to the home directory using the following command:

    cd ~

    Note: Each script requires you to enter your RSA Community username and password credentials as you would in https://community.rsa.com.

  2. Set up a root user using the following commands:

    sudo password root
    Passwd
    (enter your password and re-type it)

  3. Log in as root and enter the following command:

    su root

  4. Run the aws_vlc_preinstall.shscript.

    This script creates an sa.repo file that installs all required dependencies and packages. You are required to enter three parameters as input once you execute the script:

    • Live Account Username
    • Live Account Password
    • Version of the Log Collector you want to install.

    Run the following command to execute the script:

    ./aws_vlc_preinstall.sh

  5. Run the aws_vlc_postinstall.sh script. This script changes the Hostname and allows you to configure the Security Analytics server address.

    You are required to pass two parameters as input: Hostname and Security Analytics IP Address. Be sure to use input parameters inside a single quote separated by a space, as shown in the following command:

    ./aws_vlc_postinstall.sh '<hostname>' '<IP Address of the Security Analytics Server>

  6. Log back in to the Remote Log Collector as your virtual machine will be automatically restarted after the aws_vlc_postinstall.sh script is executed.

  7. After your virtual machine reboots, you need to synchronize the time between the AWS Remote Log Collector and the Security Analytics Server.

    1. SSH in to your virtual machine following the instructions provided in Log in Using SSH.
    2. Change users to the home directory using the following command:

      cd ~

    3. Log back in and run the following command:

      su root

    4. Get the current date and time on both the Security Analytics server and the remote log collector by running the following command:

      date

    5. If the date and time are not in sync with each other, run the following commands to set the date and time on the AWS remote log collector:

      date --set="MM/DD/YYYY"
      date --set="HH:MM:SS"

  8. Run the aws_vlc_start_services.sh script. This script starts all the required services.

    ./aws_vlc_start_services.sh

  9. Log in to the Security Analytics Server. In the Security Analytics menu, go to Administration > Hosts.
  10. Click Discover.

    The following screen is displayed after you click Discover.


  11. Select an appliance and host, then click Enable to enable your appliance.

Note: Refer to the Troubleshooting using AWS section if errors occur during provisioning.

On the Security Analytics Server, you need to deploy the Log Collector nwlogcollectorcontent to the Remote Log Collector (AWS) using Live.

  1. In the Security Analytics menu, select Live > Search.
  2. Under the Resource Types drop-down menu, select RSA Log Collector.
  3. Select Search.

The Remote Log Collector (AWS) should now be operational, and can be verified on the Administration > Services page.

For more information, refer to the Live Services Guide.

Option 2: Manually Deploy Remote Log Collector Service

To manually deploy your remote log collector service, follow these steps.

Note: Script deployment is recommended.

Log in Using SSH

Before you download scripts, you need to SSH to your virtual machine after you find your IP Address by following the steps below.

  1. To obtain the IP address, select the name of the deployed virtual instance. On the next tab, select Properties to obtain the Private IP Address of that particular virtual machine.

    The following screen is displayed.

  2. SSH to your virtual instance using an SSH client such as PuTTY. Instructions are provided for PuTTY, but you can use any SSH client.
  3. If you are using PuTTy, run the following command to connect to a Linux machine:

    putty.exe

    The following screen is displayed.

  4. Enter the IP Address using the Private IP Address from the previous screen.
  5. Under Connection type, select SSH.
  6. Under Close window on exit, select one of the available options.
  7. Click Open.

    The following screen is displayed.

  8. Select Connection > SSH > Auth.
  9. Select Browse to find your Private Key file.
  10. Click Open to connect to your virtual instance.

Disable CentOS Base Repo

To disable the CentOS Base Repo:

1. Run the following command:

sudo vi /etc/yum.repos.d/CentOS-Base.repo

2. Ensure that the contents of the CentOS-Base.repo file in the /etc/yum.repos directory have sections with

enabled=0.

File contents should be similar to the example below:

[base]

name=CentOS-$releasever - Base

mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os

#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

enabled=0

Set up Root User

To manually deploy your remote log collector, follow these steps:

1. Set up a root user using the following commands:

sudo passwd root

password (Enter your password and re-type it.)

2. Log in as root using the following command:

su root

3. Use the following commands to set up a repo file:

cd /etc/yum.repos.d

vi sa.repo

Note: Copy the contents below and paste into the Security Analytics repo file below, replacing the Live Account username and password with your Live Account credentials and Remote Log Collector version (10.6.2).

[sa]

name=SA Yum Repo

https://<LiveAccountUsername>:<LiveAccountPassword>@smcupdate.emc.com/RSA/<version>/

enabled = 1

protect = 0

gpgcheck = 0

sslVerify = 1

metadata_expire = 1d

failovermethod=priority

4. Save the sa.repo file.

Press <Escape>:wq! to save your changes to the file.

5. Run the following commands to download and install dependencies.

yum install nwconsole

yum install nwappliance

yum install nwsdk

yum install nwlogcollector

yum install nwsupport-script

yum install (res-protobuffs,rsa-audit-rt,rsa-collectd,rsasa-sshconfig,rsa-sms-runtime-rt,rsa-satools,

rsa-gpgpubkeys,rsa-mcollective-agents)

yum install mcollective-*

yum install puppet

Set up Communication (AWS Side)

To set up communication on the AWS side:

  1. Edit the hostname file to add the hostname in three places, as shown below.

    vi /etc/hostname
    vi /etc/sysconfig/network (Update the hostname that you added.)
    vi /etc/sysconfig/network-scripts/ifcfc-etho0 (Update the DHCP hostname that you added.)

  2. Press <Escape>:wq! to save your changes to the files.
  3. Edit the Hosts file and add the hostname that you added in the Hostname file. Also add the Security Analytics IP Server as puppetmaster.local by running the following command:

    vi /etc/hosts

    File contents should look similar to those shown in the following example:

    127.0.0.1 <hostname> localhost.localdom localhost
    ::1 < hostname> localhost.localdom localhost ip6-localhost ip6-loopback
    <SA IP> puppetmaster.local

  4. In the following steps you will need the node ID. Run the following command to get the node ID:

    /etc/puppet/scripts/node_id.py

  5. Remove the existing puppet.conf file by running the following command:

    rm -rf /etc/puppet/puppet.conf

  6. Edit the puppet.conf file and add the certname=node_id and server=puppetmaster.local lines, replacing the node_id with the one from Step 4.

    vi /etc/puppet/puppet.conf

    File contents should look similar to those shown in the following example:

    [main]
    rundir = /var/run/puppet
    logdir = var/log/puppet
    ssldir = $vardir/ssl
    certname = <node_id>
    [agent]
    localconfig = $vardir/localconfig
    classfile = $vardir/classes.txt
    server = puppetmaster.local

  7. Create a csr_attributes.yaml file by running the following command, replacing the hostname and IP address of the remote log collector:

    vi /etc/puppet/csr_attributes.yaml

    Note: Make sure the second line in the file contents is indented one space.

    File contents should be exactly what is shown in the following example:

    custom_attributes:

     1.2.840.113549.1.9.7: fqdn:<hostname>,ipaddress=<IP of Remote Log Collector>,type=base

  8. Run the following command to restart your system.

    reboot

  9. Log in to the Remote Log Collector using the SSH instructions in the Log in Using SSH section above.
  10. Log in as root and enter the following command:

    su root

  11. Run the following command to ensure that the Log Collector is set as a Remote Log Collector and all services are running:

    vi / etc/netwitness/ng/logcollection/logCollectionType

    File contents should be the same as the following line:

    RC

  12. To synchronize the time and date between the AWS Remote Log Collection Service and the Security Analytics Server, follow these steps:

    1. Get the current date and time on both the Security Analytics server and the remote log collector by running the following command.

      date

    2. If the date and time are not in sync with each other, run the following commands to set the date and time on the AWS remote log collector:

      date --set="MM/DD/YYYY"
      date --set="HH:MM:SS"

  13. Run the following commands to start the required services:

    service rabbitmq-server start
    service puppet start
    service mcollective start
    start nwlogcollector

Configure Firewall Permissions

Note: You need to open ports between your Remote Log Collector and your Security Analytics Server, and you also need to open ports between your Log Collector on your Log Decoder.

Configure your firewall(s) to allow communication between the Remote Log Collector and AWS, and the Security Analytics components listed in the following table.

                                                                                                   
From HostTo HostTo Ports (Protocol)Comments
From HostTo HostTo Ports (Protocol)Comments

Security Analytics Server

Remote Log Collector

56001 (TCP) or

50001 (TCP)

SSL or

Non-SSL

Security Analytics Server

Remote Log Collector

50101 (TCP)

REST (Optional)

Security Analytics Server

Remote Log Collector

5672 (TCP)

RabbitMQ

Security Analytics Server

Remote Log Collector

50055 (TCP)

RSA-SMS

Security Analytics Server

Remote Log Collector

50056 (TCP)RSA-SMS

Remote Log Collector

Security Analytics Server

8140 (TCP)

Puppet

Security Analytics Server

Remote Log Collector

61614 (TCP)

MCollective

Remote Log Collector

Security Analytics Server

61614 (TCP)

MCollective

Security Analytics Server

Remote Log Collector

15671 (TCP)

RabbitMQ

Remote Log Collector

Security Analytics Server

15671 (TCP)

RabbitMQ

In Pull Mode:

Log Collector (on Log Decoder)

Remote Log Collector

5671 (TCP)

RabbitMQ

In Push Mode:
Remote Log Collector Log Collector (on Log Decoder)5671 (TCP)

RabbitMQ

The general link that lists which ports need to be open on which appliance can be found here: https://community.rsa.com/docs/DOC-54917.

Enable Remote Log Collector on Security Analytics Server

Run the following command on Remote Collector to enable Remote Log collector on the Security Analytics Server:

puppet agent -t --waitforcert 30

Set up Communication (Security Analytics Side)

To set up communication on the Security Analytics side, follow these steps:

  1. Log in to the Security Analytics Server.

    In the Security Analytics menu, go to Administration > Hosts.

  2. Click Discover.

    The following screen is displayed.

  3. Select an Appliance and Host, then click Enable.

On the Security Analytics Server, you need to deploy the Log Collector nwlogcollectorcontent to the Remote Log Collector (AWS) using Live.

  1. In the Security Analytics menu, select Live > Resource Types.
  2. Under the Resource Types drop-down menu, select RSA Log Collector.
  3. Select Search.

The Remote Log Collector (AWS) should now be operational, and can be verified on the Administration > Services page.

For more information, refer to the Live Services Guide.

Next Topic:Troubleshooting
You are here
Table of Contents > Configure and Deploy Remote Log Collector Service into AWS > Step 3 - Deploy Remote Log Collector Service in AWS

Attachments

    Outcomes