Configure SFTP Shell Script File Transfer

Document created by RSA Information Design and Development on May 19, 2016Last modified by RSA Information Design and Development on May 4, 2017
Version 6Show Document
  • View in full screen mode
  

Overview

Use the sasftpagent.sh shell script to transfer text-based log data from Linux systems. This script takes data slices from active log files, but only transfers the new data each time the script runs.

Schedule the script in cron to run as often as you want log data sent to the RSA Security Analytics Log Collector. The script uses the SFTP or SCP protocol to transfer the data.

Note the following:

  • All connections are initiated from the system to Security Analytics.
  • The script runs on all POSIX compliant Unix/Linux systems and shells.

    Note: You must use OpenSSH version 4.4p1 or later.

  • RSA recommends that you set up a cron job to run the script at specified time intervals. However, if you do set up a cron job, make sure to run it as a user that has access to the logs that need to be sent to Security Analytics.

This topic contains the following information:

  • Enhancements for version 3 of the Agent
  • Instructions for Upgrading the Agent: follow these steps if you are currently running version 2.7 of the agent
  • Instructions for Installing the Agent: follow these steps if you are downloading the agent for the first time
  • Details for the Shell Script Parameters
  • Instructions to Configure RSA Security Analytics Log Collector to Receive Log Files

You must perform the following steps to complete the installation and configuration of the agent:

  1. Install or Upgrade the agent, depending on whether or not you are running it currently.
  2. Configure the RSA Security Analytics Log Collector to Receive Log Files.

Enhancements for Version 3

  • The script runs on all POSIX compliant Unix/Linux systems and shells.
  • Expects configuration at /etc/rsa/sasftpagent.conf.
  • Encourages the user to keep configuration separate from script source. Warning is logged if the user does not do so.
  • Persistent state is written to /var/lib/rsa by default.
  • Enhanced user interaction, such as:

    • Support for running the script without root privileges.
    • The agent has been made highly configurable so that it can be configured to run from anywhere, and create a persistent state directory anywhere. For example, non-root users can persist state information to their home directory, by specifying an alternate persistent state directory in the configuration file.
    • A configuration specified at a non-root user's home directory (e.g. ~/sasftpgent.conf) is automatically picked up while running as a non-root user.
    • Command line options (-C or --config) have been added to point the agent to custom configuration.
  • Logs are written to /var/log/rsa/sasftpagent.log. Logging levels have been introduced so that the logs may be filtered for WARN, ERROR and FATAL entries, to help troubleshoot issues. These log entries can now be used to perform troubleshooting after the fact.
  • If the user forgets to edit the configuration, a FATAL log entry is generated. The entry contains a clear message that the user needs to edit the configuration before the script can be used.

Upgrade the Agent

If you have used a version of sasftpagent.sh prior to version 3, then you should follow the instructions in this section to upgrade to the latest version.

The major steps are as follows:

  1. Download the new Agent from SCOL.
  2. Move configuration information to /etc/rsa/sasftpagent.conf.
  3. Download the mvpersinfo.sh script.
  4. Run the mvpersinfo.sh script to move persistent information to the location used by version 3.
  5. Run version 3 of the agent.

Move Configuration Information

In version 2.7, the user configuration was specified in one of the following two locations:

  • The configuration may have been edited inline within /usr/local/sa/sasftpapgent.sh (or wherever you placed the script)
  • The configuration may have been specified separately; wherever the CONFIG_FILE parameter in the sasftpagent.sh script is set.

For any parameters that you edited within the script or that you specified in the separate configuration file, you need to move them to a separate file in the following location: /etc/rsa/sasftpagent.conf.

The following parameters are the ones that users often change during configuration:

  • SA
  • DATA_DIRECTORY
  • FILESPEC
  • FLAG_REMOVE_AFTER_SEND (only set if you wanted to remove data files automatically after transferring them to the Log Collector).

Move Persistent Information

In version 2.7, the persistent state information is maintained in the /usr/local/sa directory by default. It can also be specified in the PERSINFO_DIRECTORY parameter.

The persistent information directory contains tracking files that contain the number of lines, for each data file, that have already been transferred to the Log Collector. The agent uses this information to figure out which lines in each file are new, since the last time it ran. It then transfers only the new data, and updates the tracking files accordingly.

It is important to move these files into the new location before you run version 3.0.1 of the agent. RSA provides a script, mvpersinfo.sh, for moving the persistent information.

Run the Move Script

To move the persistent information to its new location, perform the following steps:

  1. Copy mvpersinfo.sh to the system where you run version 2.7 of the agent.
  2. Open mvpersinfo.sh with a text editor, and confirm that OLD_PERSINFO_DIRECTORY is set to the value set for PERSINFO_DIRECTORY in your 2.7 configuration.
  3. Run the script using the following command:

    sh mvpersinfo.sh

If there are no errors, and the script runs successfully, it does not generate any output.

Confirm the Files Were Moved

After you run the script, you can confirm the successful movement by using the following procedure:

  1. Run the following command to get a list of the old tracking files:

    find /usr/local/sa -name "*-*last.line"

  2. Run the following command to get a list of the new tracking files:

    find /var/lib/rsa/sasftpagent -name "*-*last.line"

  3. Compare the output from the two commands. The output should be similar, with the only difference being the paths to the files. Here is a sample of the results of running these commands after moving the persistent files:

$ find /usr/local/sa -name "*-*last.line"

/usr/local/sa/opt/log/bar.log-sa.last.line
/usr/local/sa/opt/log/foo.log-sa.last.line
usr/local/sa/var/log/foo.log-sa.last.line
/usr/local/sa/var/log/fob.log-sa.last.line

$ find /var/lib/rsa/sasftpagent -name "*-*last.line"

/var/lib/rsa/sasftpagent/track/opt/log/bar.test-last.line
/var/lib/rsa/sasftpagent/track/opt/log/foo.test-last.line
/var/lib/rsa/sasftpagent/track/var/log/foo.test-last.line
/var/lib/rsa/sasftpagent/track/var/log/fob.test-last.line

Install and Configure the Agent

If you are not upgrading from a previous version of the agent, follow the steps in this section to download, install, and configure the agent.

  1. Download the Agent
  2. Create or Configure a User Account to Run the Agent
  3. Create and Update the Configuration File
  4. Schedule the Agent to Run Periodically: Configure cron or your OS scheduler to automate running the script at your desired interval.

Download the Agent

Follow these steps to download the SA SFTP Agent (sasftpagent.sh) from SCOL.

  1. Log on to RSA SecurCare Online (SCOL).
  2. In the Search box, enter RSA Security Analytics SFTP Agents.
  3. Select the page that is returned from the Search, RSA Security Analytics SFTP Agents.
  4. Click RSA Security Analytics Unix SFTP Agent and save the file anywhere on your file system.
  5. Set execute permissions on the sasftpagent.sh file. For example, run the following command:

    chmod 755 /usr/local/sa/sasftpagent.sh

Create or Configure a User Account to Run the Agent

The user account that runs the agent needs read permission to the DATA_DIRECTORY that holds the logs, and read/write access to the PERSINFO_DIRECTORY that the agent uses for persistent storage.

Create and Update the Configuration File

Create the configuration file here: /etc/rsa/sasftpagent.conf. If you do not have a configuration file, copy the script file (sasftpagent.sh) and remove everything but the configuration parameters. Update the configuration file with the information for your environment. For reference, see the Parameters table below.

If you are running the script for the first time, run the following command, where collector-IP is the IP address of your Security Analytics Log collector:

sftp collector-IP

Caution: It is important to run this command as the same user who will run it when it is automated.

Shell Script Parameters

The following table describes the most important parameters that you need to set when configuring the script.

                                                             
ParameterValuesDescription
SAName or IP addressThe name or IP address of your RSA Security Analytics Log Collector host.
DATA_DIRECTORYDirectory path or paths, separated by colons (:).

The local source for the log data. For example:

DATA_DIRECTORY=/var/log:/var/log/audit

You can specify one or more folders.

Note: All folders that you specify are searched for the file names that you specify in the FILESPEC parameter.

FILESPECFile name or names, separated by colons (:).

File mask that matches the log files to be processed by the script.

Note: The script supports line-by-line text data. Thus, .xml, .zip, .gz, .exe and other non-text formats are not supported.

For example, to process all files in the folder:

FILESPEC=isi_webui.log:smb.log:/etc/logs/*.*

Note: The files that you specify can reside in different folders. Make sure to list all of the necessary folders in the DATA_DIRECTORY parameter.

SA_DIRECTORYThe directory name of your Security Analytics Log Collector host

The destination folder name. For example:

/upload/apache/muditapache

TRANSFER_METHODSFTP, or SCPSFTP is the default (and recommended) transfer protocol
USEHEADA non-negative integer, representing the number of header lines

The number of lines in each log file to be considered as a header that must be transferred to Log Collector in each transfer.

You can set this to 0 to indicate that there are no header lines.

DEPTHA positive integer, representing number of folder levels

Governs how many levels deep the script searches to find logs under the configured DATA_DIRECTORY.

Defaults to DEPTH=1, which causes the script to search for data files directly under the directories configured in DATA_DIRECTORY, but not in any sub-folders.

SFTP and SCP Settings
USERNAMEsftp Default setting for SSH daemon on the Security Analytics platform.
IDENTITYFile path

Location of the private key used to connect to Security Analytics. For instructions on generating keys, see Install and Update the SFTP Agent.

The default value is the following:

$HOME/.ssh/id_rsa

Configuration Script Information

All configuration settings can be loaded using a configuration file that is separate from the script. This file should contain one setting and one value per line (except for DATA_DIRECTORY and FILESPEC, which can contain multiple colon-separated entries).

The configuration file must be located in the directory assigned by SA_DIRECTORY in the shell script or in the path of the shell that calls the script. The SA_DIRECTORY can be overridden in the configuration file, although the shell script will try to use its own SA_DIRECTORY setting to open the configuration file.

For example, a configuration file could contain the following information:

SA=10.31.246.168
DATA_DIRECTORY=/var/log/httpd
SA_DIRECTORY=/upload/apache/muditapache
USERNAME=sftp
FILESPEC=*_log*
FLAG_REMOVE_FILE_AFTER_SEND=no

Configure RSA Security Analytics Log Collector to Receive Log Files Edit section

Before you configure the Log Collector, remember to run the following command on any new event source, from which Security Analytics has not previously collected logs using SFTP or SCP:

sftp collector-IP

where collector-IP is the IP address of your Security Analytics Log collector.

To configure Security Analytics to receive the log files:

  1. On the Linux or Unix event source, run the following command to generate the public/private key pair:

    ssh-keygen -b 1024 -t rsa

    This command creates id_rsa in OpenSSH format, which is used by RSA Security Analytics. If your Linux system creates IETF SECSH format by default, run the following command to convert it:

    ssh-keygen -f ~/.ssh/id_rsa.pub -i

  2. Add the public key into the log collector, as described in Install and Update the SFTP Agent.

You are here
Table of Contents > Configure SFTP Shell Script File Transfer

Attachments

    Outcomes