Skip navigation
All Places > Products > RSA NetWitness Platform > Blog
1 2 3 Previous Next

RSA NetWitness Platform

542 posts

Overview

The RSA NetWitness is run by many of our customers on RSA's physical appliances, but the entire stack can run in AWS, Azure, VMware, or Hyper-V just fine. You can even mix-and-match hardware between physical and virtual hosts however you prefer. Our Virtual Host Installation Guide does a great job outlining the steps to building a virtual RSA NetWitness Platform host.

 

However, there is frequently a need to build smaller hosts to gather data in smaller remote locations. Small issues that don't apply to larger hosts can cause RSA NetWitness Platform folders to overrun their allotments and cause NetWitness to stop capture or aggregation. This post will primarily focus on the settings to focus on when building smaller virtual hosts. It will also include some tricks to monitor your NetWitness hosts to make sure they don't reach unhealthy levels of storage. Of course, many of these tips will also apply to virtual hosts of all sizes, so hopefully you can benefit regardless of your particular virtual implementation.

 

To ISO or Not to ISO

RSA provides both an ISO and an OVA (and a Hyper-V VHD) to use to build your virtual hosts. Which should you use? If you are building a full RSA NetWitness Platform implementation virtually, you will have to use the ISO to build your Admin Server because the OVA does not come with all of the required RPMs. As for the other hosts, using the OVA isn't a bad idea. The OVA is a much smaller file to deal with (~450MB OVA vs ~6GB ISO) and it has already completed the bootstrap, which is one of the longest steps of the installation. However, the OVA has already provisioned the logical volumes for a 195GB host. That is the recommended size for the OS drive, but if you're wanting to give more than that, the ISO is the easiest option - and I say that as someone who rather enjoys partitioning Linux file systems! As for assigning less than the 195GB, I would recommend you thin provision your host's OS drive before you install with less than what RSA recommends.

 

Keep in mind that your log, network, and endpoint data stores will be separate from this. The OS drive is strictly for holding OS files, NetWitness internal service log entries, temporary data, and some other miscellaneous data. You will add disks to accommodate storing your log, network, and/or endpoint data in the step.

 

Installing the ISO is extremely simple: create your virtual host, give it the CPU, RAM, and HDD storage as recommended in the installation guide or by your RSA engineer (different requirements for different services and different levels of throughput), attach the ISO, and turn on the VM. It will boot to the blue installation screen where you will hit <Enter>. Once you get to the following screen...

...make sure you enter "y" or "Y" and hit <Enter>. Once the bootstrap is complete, the system will reboot to the login prompt. After logging in, you will run "nwsetup-tui" and you can refer to the installation guide for instructions on how to properly orchestrate a host from there.

 

VM Host Sizing

In the previous step, you installed the bootstrapped host via the ISO or the OVA and possibly orchestrated the services as well. In the case of any host that will retain data - Decoders (network / log), Hybrids (endpoint / network / log), Concentrators, or Archivers - you will need to also provision storage for that data. Sizing that can be difficult, but I have a calculator that can help size most of those appropriately.

 

...except Archivers. Why not Archivers? Archivers are employed, generally, for regulatory purposes. You should engage your RSA Engineer to make sure you size them appropriately so that you don't run into issues with auditors. You might be logging especially large logging sources, while the calculator only uses a static 600 bytes per message. You can also retain more or less meta keys which can drastically affect how much storage to assign. And after all, while the "[Small]" in the title of this post was in hard brackets, this guide is generally geared towards smaller deployments / hosts. The sole reason to use an Archiver is because the amount of storage has reached significantly beyond any definition of the word "small".

 

To use the calculator, there are a number of things to understand:

  • The calculator is used to calculate Hybrid storage, because most "small" environments will use Hybrids rather than discrete Decoder and Concentrator pairs. If you are using separate Decoders and Concentrators, you can simply break up the calculated storage per service and split up the provisioning commands. NOTE: There is no such thing as a "discrete Endpoint Decoder". Endpoint servers only come as Hybrids, whether virtual or physical.
  • When you enter information to size up your storage, at the bottom of the calculator you will get provisioning commands to setup your hosts. If you have any Hosts entered in rows 6 or 7, you'll get commands to provision storage for an Endpoint Log Hybrid. If you don't have any Hosts, but you have Log Events >0 GB/day, you will get commands to provision storage for a Log Hybrid. If you have Log Events at 0 GB/day and you have 0 Hosts but your network traffic is >0 GB/day, you will get commands to provision storage for a Network Hybrid.
    • If you are sizing an Endpoint Log Hybrid, keep in mind that you cannot currently download modules automatically, download memory dumps, or download Master File Tables from hosts. Those features which were in ECAT 4.x will be back in the product as of 11.4, and I've included commands to provision them. However, the amount of storage you provision for those purposes is entirely up to you, so you will need to just type the numbers into that cell. They can both be relatively small (10 - 30GB) if you don't plan to auto-download unsigned, new modules. However, once the feature is back, we do highly recommend that you automatically have NetWitness Endpoint download any unsigned, unknown modules less than 5MB - 10MB, and estimate storage for your environment appropriately.
  • Once storage is provisioned for each of the given volumes, the last provisioning command is to give 100% of the remaining space to the MetaDB on the Concentrator. That is done on purpose because if I have any extra space left over, that is where I want it. However, you also must make sure (likely with df -h) that you enough storage in that logical volume. If not, you likely didn't give the entire partition enough space.
    • For this same reason, if you end up using this calculator to build a discrete Decoder, you'll likely want to change the command that would provision your PacketDB to use the "100%FREE" version of the lvcreate command. The syntax would be the same as the one I use for the Concentrator's MetaDB.
  • When you enter the scale information for Network Traffic, you might wonder, "But I don't know how many GB/day of network traffic I plan to send to NetWitness!" The easiest rule of thumb is that if you expect to see 100Mbps on average for a 24-hour period (that would mean ~175Mbps over the peak hour and 10Mbps overnight), that is 1TB/day of traffic. If you expect to see 10Mbps because it's a small office or home environment, assume 100GB/day. If you have absolutely no idea, just throw a number in there.
  • For logs, in a small environment, if you had any log management system you can probably figure out how many GB/day of day you were generating before. If you expect a certain number of Events per Second, I put a handy calculator to turn that into GB/day on row 10. If you have no idea, then once again, I suggest you just throw something in there.
  • You can edit the calculator if you like. The password is just "rsa". I only password protect it to make sure that first-time users aren't editing cells they shouldn't and break it.

 

The calculator is called NW Virtual Hybrid Sizing Calculator v1.0.xlsx. PLEASE, if you find any errors, leave a comment below or contact me somehow so that I can fix it for others.

 

Raw Event Data Storage

The Virtual Host Installation Guide covers how to add storage for the various RSA NetWitness Platform databases in Step 3. It also covers how to calculate the amount of storage you'll need to allocate to each database for any given host/service. For the Admin Server, Archiver, Broker, ESA, Log Collector, and UEBA hosts, all storage will get dumped into the /var/netwitness/ folder. The instructions for extending that volume group and logical volume are in the installation guide and generally involve: pvcreate, then vgextend, then lvextend, and finally xfs_growfs.

 

For Decoders, Concentrators, and Hybrids, I've put together the commands that you need in the attached

 *Commands.txt text files to setup the storage for those hosts. I recommend running all of these scripts to build the partitions, volume groups, and logical volumes after you run nwsetup-tui, but *BEFORE* you install the services on the hosts. A few things to note:

  • I name the volume group "vg01" for the sake of brevity. The name you assign does not matter at all.
  • In Step 5, I assign storage to the "root" folder for each respective service; /var/netwitness/decoder for Network Decoders, /var/netwitness/concentrator for Concentrators, and /var/netwitness/logdecoder for Log Decoders. This is not required, but I prefer to create these volumes so that I can monitor them in case they fill up. Note: they must have at least 5GB of storage assigned, but larger VMs can have as much as 30GB.
  • Also in Step 5, you will need to replace the lv sizes with the proper sizes based on the Installation Guide and/or your RSA NetWitness Platform engineer. In my scripts, I assign specific sizes to every volume except the last one, which I then assign whatever free space is left with the "100%FREE" command.
  • For Step 10, I wrote that so that you can copy and paste it directly into an SSH session into the /etc/fstab file on the host. You can paste that directly to the bottom of the existing file. Once that is done, before you install services, make sure to reboot the host to make sure there aren't any errors in that fstab file. The syntax is very particular and any errors will cause the system to fail to come up. If that happens, just open a Console window to the machine, hit CTRL+D to enter maintenance mode, and then fix the fstab file.
  • I want to say this again because it's very important: after adding your changes to the fstab file, reboot the machine and make sure your syntax was correct!

Just view the *Commands.txt file attached to this post that corresponds to the type of host you're trying to install.

 

Install Services

This step is straightforward. If you haven't already, go to Admin --> Hosts and enable the host. Then install the services just as outlined in the Installation Guide.

 

Validate Folder Sizes - RSA NetWitness Platform Databases

In order to properly roll off the oldest entries in NWDB (NetWitness Database, our proprietary database format), we have to make sure that the RSA NetWitness Platform knows how much storage each database has to fill. Navigate to Admin --> Services, and for any Concentrator or Decoder/Log Decoder service, go to the Explore page. Expand the "database" menu item on the left-hand side, and click on "config". Here I show the page for an RSA Log Decoder service on a physical Endpoint Log Hybrid:

The sizes you see there are 95% of the corresponding folders we built using the provisioning commands, measured in 1,073,741,824 byte blocks. If you want to get to exact, you can run "df --block-size=G", multiply a folder by 95%, and round to the nearest two digits to get the value RSA NetWitness Platform will place in the corresponding line above. Once the data in one of these folders exceeds these limits, RSA NetWitness Platform rolls off data.

 

If you followed this guide and the Virtual Host Installation Guide, you will see folder sizes here that match what you provisioned. But what if they don't match or you made a mistake? Well, you can reset those by right-clicking on the "database" menu item and clicking "Properties":

 

At the bottom-right of the window, the Properties pane will open up. Select "reconfig" from the drop-down and click the Send button:

You can see that these values match what we saw in the previous screen. If these values still don't look correct - usually, if they are all the same - then your folders aren't mounted to separate logical volumes. If these values do look correct, you can remove the "=xx.xxTB" or "=xx.xxGB" from the entries on the previous screen. Then, back in the Properties pane, in the Parameters box, type update=1 and click Send again. It will append those values to the appropriate entries at the top, though you'll have to refresh the screen to see the update.

 

The indexes for each of these services has a separate entry. On the Explore page, you will see a menu item called "Index", and the settings are under the "config" sub-menu. Just like above, if you need to reset the folder size for that, you can right-click on "Index" and run the reconfig commands like before.

 

Validate Thresholds - MongoDB

In addition to NWDB, NetWitness also stores Endpoint scan results (primarily, what you see in Navigate --> Hosts) in mongoDB on the Endpoint Log Hybrid in the /var/netwitness/mongo folder. NetWitness does not display the folder sizes in the Endpoint Server service's Explore page as it does for those services above. Instead, it just looks at the amount of storage in the /var/netwitness/mongo folder, or, if that isn't separately partitioned, in the /var/netwitness folder. Then it compares the current usage to the value in the "rollover-after" setting here:

Your system may not use this setting if your Data Retention policies (found at Admin --> Services --> Endpoint Server --> Config --> Data Retention Scheduler tab) don't already roll over data before the folder hits 80%. You should also be aware of the settings under endpoint/data-store-thresholds:

If the storage in the corresponding folder (/var/netwitness/mongo if it's partitioned separately, and /var/netwitness if it's not) crosses these thresholds, you will eventually receive Health & Wellness alerts that correspond to those thresholds.

 

Minimum Available Space - The Key to Reliability

The other setting you may have noticed in the previous screenshots that we ignored were the <database_name>.free.space.min settings. A given database can grow past the maximum size we've setup above with no issues, but capture/aggregation will stop if there is less free space than what is specified in the free.space.min setting for the corresponding service. Just like the folder size above is set as 95% of the total volume size, the free.space.min is set to 0.865% of the total size, by default. In both cases, the default setting can be replaced manually with whatever you would like to enter. For most large VMs, the default is fine. However, for smaller hosts capturing small amounts of data, this default may be a bit high and can be adjusted.

 

Please note: the indexes do not have a similar free.space.min setting, and capture/aggregation will continue to run, even if the index volumes are essentially full.

 

For Mongo, you should also be aware of the settings under Admin --> Services --> Endpoint Server --> Explore --> endpoint/data-store-thresholds:

If the storage in the corresponding folder (/var/netwitness/mongo if it's partitioned; /var/netwitness if it's not) crosses the warning-percent level, <this will happen>. If it crosses the fatal-percent level, <this will happen>.

 

Monitoring Part 1: Folder Sizes

As I mentioned in the Overview, for small hosts (roughly <1TB of total storage), I recommend monitoring your volumes to make sure that they don't fill up. To do this, I modified a script I found here to monitor file system usage:


It pulls back every folder other than temp and boot folders, and if any are at 90% or higher, it will generate a syslog, sent to the IP designated by the -n switch (10.10.10.10 in the image above). I've attached that script below as checkVolumeSizes.sh. (Remember, use chmod to make it executable!) If you run chrontab -e from an SSH terminal, the RSA NetWitness Platform's underlying Centos OS will open vi and allow you to set a schedule to run the script. I imagine most of you reading this are familiar with crontab syntax, but if you're not, or if you want to design something overly tricky, this site takes all the work out of it for you: https://crontab.guru/.

 

The messages generated will look like this:

You can ingest that into any system that can ingest syslog messages and alert on it as you see fit. Seeing as RSA NetWitness Platform *IS* a SIEM, it seemed only right to go ahead and monitor that using the RSA NetWitness Platform . The first step involved in that is properly parsing the message, so I built a parser for that using the NetWitness Log Parser Tool (download here: https://community.rsa.com/docs/DOC-94172, learn how to use it here: RSA ESI Beta 3 - YouTube and Parser Development When No Message ID Exists - YouTube). It took maybe 5 minutes.

 

But there aren't any out-of-the-box keys meant to store the size of logical volumes, and I wanted to include that in the e-mail I send to myself, so I added a meta key to the RSA NetWitness Platform for that. If you use my parser you *MUST* create a custom meta key in your system in order for the parser to work properly. Add the custom meta key to the table-map-custom.xml file on the Log Decoder where you are directing these messages.


You can find that attached as table-map-custom.txt. I didn't want to call it table-map-custom.xml because it needs to be added to the existing file, not pasted over the existing file in its entirety.

 

Now, download nwdiskalert.envision, navigate to Admin --> Log Decoder --> Config, click the Parsers tab, and upload that file. After uploading, if you want to make sure the Log Decoder reloaded its parsers, you can switch from Config to Explore:



Once the page loads, expand the "decoder" menu, right-click on "parsers", and choose "Properties".




In the Properties pane, select "reload" from the drop-down menu and then click Send. Now the parsers have been reloaded and you're all set to ingest these messages!

 

Monitoring Part 2: ESA Correlation Rules

I built three ESA rules to monitor my file system at home, one each for medium, high, and critical severity alerts. Here is what I classify as each:

  • Medium Severity:
    • Goal:
      • Monitor folders that shouldn't ever fill up when they reach high levels of utilization, but won't cause any service issues. 
    • Rules:
      • Any of the following folders are at least 90% but no more than 94% disk usage:
        • /home
        • /var/log
        • /var/netwitness
        • /var/netwitness/concentrator
        • /var/netwitness/decoder
        • /var/netwitness/logdecoder
  • High Severity:
    • Goals:
      • Monitor folders that shouldn't ever fill up when they reach extremely high levels of utilization, but won't cause any service issues
      • Monitor folders that could cause service interruption once they pass 95% (which is where many of them will sit most of the time) but haven't yet reached a point where service interruption will occur
      • Monitor the mongodb folder if it reaches concerning levels
    • Rules:
      • Any of the following folders are at least 95% but no more than 97% disk usage:
        • /home
        • /var/log
        • /var/netwitness
        • /var/netwitness/concentrator
        • /var/netwitness/decoder
        • /var/netwitness/logdecoder
        • /var/netwitness/concentrator/index
        • /var/netwitness/decoder/index
      • Any of the following folders are at 96% or 97%:
        • /var/netwitness/concentrator/sessiondb
        • /var/netwitness/concentrator/metadb
        • /var/netwitness/decoder/sessiondb
        • /var/netwitness/decoder/metadb
        • /var/netwitness/decoder/packetdb
        • /var/netwitness/logdecoder/sessiondb
        • /var/netwitness/logdecoder/metadb
        • /var/netwitness/logdecoder/packetdb
    • The /var/netwitness/mongo folder is at least 90% and no more than 94%
  • Critical Severity:
    • Goals:
      • Monitor folders that shouldn't ever fill up when they reach critical levels of utilization
      • Monitor folders that could cause service interruption once they pass 97% and will soon - or are currently - causing service interruption
      • Monitor the mongodb folder if it reaches its "fatal-percent" setting
    • Rules:
      • Any of the folders in the High Severity list are at 98% or above
      • The /var/netwitness/mongo folder is at 95% or above

 

You can find those attached as nwDiskMonitoringESARules_<severity>_Basic.txt. You might ask yourself, "Why did he call them "Basic"? Well, that's because I actually built more detailed rules in my lab to monitor for the free size returned from the event logs. It's absolutely overkill, and it causes the rules to look like this:

Do you really want to do that to yourself? You really shouldn't, but if you insist, feel free to reach out to me and I'll send you those rules as well.

 

Monitoring Part 3: Generating Notifications

When these rules detect something, of course you'll want to generate an e-mail to notify you of their current state. I use a single notification template for all three ESA Rules. I put my notification template in the attached file nwDiskMonitoringNotificationTemplate.txt. The template breaks down like this:

  • Lines 1 - 20: Builds a banner at the top of the e-mail that is yellow for medium alerts, orange for high, and red for critical
  • Line 25: Prints the time the event was generated
  • Line 27: Prints the IP of the RSA NetWitness Platform host that generated the event log
  • Line 29: Prints the folder that the alert is related to
  • Line 31: Prints the % utilization of the folder
  • Line 33: Prints the amount of free space, in MB, left in that folder
  • Line 35: Generates a hyperlink to the raw event log in the RSA NetWitness Platform; make sure you edit both the <NW_URL_or_IP> and the device ID (mine is 6)

(Have questions about any other items in this notification template? Check out my other relevant blog post here: Building the Notifications of Your Dreams in the RSA NetWitness Platform.)

 

Once you've updated those items, place it under Admin --> System --> Global Notifications --> Template (tab), and make sure you select that template when adding your ESA Rules. You can also build an Incident Rule in the RSA NetWitness Platform if you want to generate incidents for these alerts. Here is mine, for reference:

 

Summary

I can't emphasize enough that the Virtual Host Installation Guide has very comprehensive instructions for setting up a virtual RSA NetWitness Platform host, and you should make sure you follow those instructions. However, following some of the additional steps included in this guide can give you peace of mind that your RSA NetWitness Platform environment is running smoothly and collecting your critical security forensic information.

 

Future note: I plan to build some Event Source Monitoring rules to make sure that my hosts are still sending logs. For example, the packetdb folder on your Decoders and Log Decoders should reach 95% eventually and then roll off data, while your Concentrators should reach 95% on their metadb folder. Those should continue to generate logs once they hit 90% utilization at every interval you specified in the cron job. If I ever get the free time to create those, I'll update this post with that information. If someone wants to build that on their own, be my guest!!

Introducing RSA NetWitness Platform's support for AWS VPC Traffic Mirroring!

 

By partnering with AWS and integrating with their AWS VPC Traffic Mirroring, customers are able to access to the right virtual traffic and network metadata from AWS environments. The AWS VPC Traffic Mirroring allows users to capture and inspect network traffic to analyze packets without using any third-party packet forwarding agents. The solution provides insight and access to network traffic across VPC infrastructure. 

 

Packets can now be captured, retained, analyzed and stored in the AWS cloud bringing additional visibility and security with the RSA NetWitness Platform.  With this agent-less packet capture capability, we’re able to provide analysts the context they need to understand the threats they’re investigating.  Combining network visibility with other sources such as Logs, Endpoint and Netflow we’re able to provide a single view to the analyst!

 

RSA NetWitness Platform enables customers to obtain the visibility needed to secure critical infrastructure, and empowers any analyst to identify, understand, and mitigate advanced threats.   RSA’s NetWitness Platform's integration with AWS enables customers to close the visibility gap created by workloads in the cloud.  This solution provides flexible AWS deployment options which allow NetWitness components to be deployed either in a Full Stack (all cloud) or Hybrid (on premise & cloud) configurations.

 

Hybrid Deployment

RSA NetWitness - AWS VPC Traffic Mirroring

 

For technical implementation details, see our AWS Deployment Guide

It often happens to me that while I am testing new alerts and incident aggregation rules, I find that the aggregation condition(s) I chose in my Incident Rule are not what I want.  While I could re-create the raw alerts from scratch, I wanted an easier method to tell the Respond engine to re-apply its aggregation rule policies on the alerts that already exist in the database.

 

To be clear, the Respond engine is always attempting to apply all active and valid Incident Rules against un-aggregated and un-affiliated alerts in the database -- that is, any alert that has not been previously aggregated into any incident can be automatically aggregated into an incident if an incident rule with matching conditions is changed/created.  But for previously aggregated alerts whose incidents have been deleted (leaving the alerts un-aggregated but previously-affiliated), the Respond engine will not attempt to re-aggregate them.

 

So my goal, then, was to get the Respond engine to include these previously-affiliated alerts in its aggregation attempts.  To achieve this, the alerts simply needed to be updated to remove their previously-affiliated status.  And to make it easy to change dozens or even hundreds of alerts at once, I wrote a simple shell script (attached to this blog and pasted below) to do it all for me.

 

#!/bin/bash
#
#grab the deploy_admin password
DEPLOY_PW=$(security-cli-client --get-config-prop --prop-hierarchy nw.security-client --prop-name platform.deployment.password --quiet)

#set a desired time range to query for alerts
#examples: "24 hours ago" or "14 days ago" or "4 weeks ago"
timeRange=$(date +%s%N -d "30 days ago" | cut -b1-13)

#identify primaryESA host
primaryESA=$(echo -e "use orchestration-server\ndb.host.find({installedServices:\"ESAPrimary\"},{hostname:1})" | mongo admin -u deploy_admin -p $DEPLOY_PW --quiet | grep -Po "hostname.*\"" | sed -e "s/hostname.\{5\}\|\"//g")

#change status on all alerts that were part of a deleted incident
#within the timerange from "REMOVED_FROM_INCIDENT" to "NORMALIZED"
echo -e "use respond-server\ndb.alert.update({\$and:[{status:\"REMOVED_FROM_INCIDENT\"},{\"originalHeaders.timestamp\":{\$gte:$timeRange}}]},{\$set:{status:\"NORMALIZED\"}},{multi:true})" | mongo admin -u deploy_admin -p $DEPLOY_PW --host $primaryESA --quiet

 

A couple notes on the script:

  • I used one extremely generic parameter (timestamp within last 30 days) to limit the database query and update operation (line 15)
    • you should feel free to modify the timeRange (line 8) to suit your needs
    • you should also feel free to (carefully) modify the database query to focus on specific alerts in your environment
      • for example, given the following raw alert:

 

...you could change line 15 and add:

echo -e "use respond-server\ndb.alert.update({\$and:[{status:\"REMOVED_FROM_INCIDENT\"},{\"originalHeaders.timestamp\":{\$gte:$timeRange}},{\"originalAlert.moduleName\":\"Alert with source and destination IP values\"}]},{\$set:{status:\"NORMALIZED\"}},{multi:true})" | mongo admin -u deploy_admin -p $DEPLOY_PW --host $primaryESA --quiet

 

...or:

echo -e "use respond-server\ndb.alert.update({\$and:[{status:\"REMOVED_FROM_INCIDENT\"},{\"originalHeaders.timestamp\":{\$gte:$timeRange}},{\"originalAlert.events.ip_src\":\"192.168.20.20\"}]},{\$set:{status:\"NORMALIZED\"}},{multi:true})" | mongo admin -u deploy_admin -p $DEPLOY_PW --host $primaryESA --quiet

 

...or:

echo -e "use respond-server\ndb.alert.update({\$and:[{status:\"REMOVED_FROM_INCIDENT\"},{\"originalHeaders.timestamp\":{\$gte:$timeRange}},{\"originalAlert.moduleName\":\"Alert with source and destination IP values\"},{\"originalAlert.events.ip_src\":\"192.168.20.20\"}]},{\$set:{status:\"NORMALIZED\"}},{multi:true})" | mongo admin -u deploy_admin -p $DEPLOY_PW --host $primaryESA --quiet

  • a successful run of the script will produce output like this, showing you how many alerts in the database were modified (3, in this case):

 

Of course, I recommend testing this (and most everything else) in a pre-prod or test NetWitness environment, if you have one.  And should you have any questions about what might be a good and/or valid database query, the Link community is always on hand to help (please have screenshots and/or specifics about your alerts ready...its hard to help without knowing details...  ).

An administrator uploads custom YARA content to the RSA NetWitness Platform per instructions in the documentation. Turns out they want to change or delete it, but the only options in the user interface are to disable or enable. The naming of the YARA custom files will be different, reflecting names given during upload.

 

Can anything be done?

 

The answer is yes. The steps below explain how to manage custom YARA content via the command-line.

 

  1. Connect to the malware appliance via SSH and change to the YARA directory.
[root@malwareserver yara]# cd /var/netwitness/malware-analytics-server/spectrum/yara

 

  1. Find the custom files you want to delete.
    Rules are merged into a single file. It is unknown if you can modify that file to remove a single rule.
[root@malwareserver yara]# ll
total 492
drwxr-xr-x. 2 netwitness netwitness 6 Aug 20 15:23 error
drwxr-xr-x. 2 netwitness netwitness 4096 Aug 29 14:02 processed
-rw-r--r--. 1 netwitness netwitness 587 Jul 15 16:49 rsa_mw_pdf_artifacts.yara
-rw-r--r--. 1 netwitness netwitness 76289 Jul 15 16:49 rsa_mw_pe_artifacts.yara
-rw-r--r--. 1 netwitness netwitness 96334 Jul 15 16:49 rsa_mw_pe_packers.yara
drwxr-xr-x. 2 netwitness netwitness 6 Aug 20 16:03 watch
-rw-r--r--. 1 netwitness netwitness 317666 Aug 20 16:05 custom_merged_static_rules.yar

 

  1. Remove the file(s) or move it/them to a different directory.
[root@malwareserver yara]# rm -i custom_merged_static_rules.yar

 

  1. Change directory to the YARA processed folder, and remove (or move) the processed files.
[root@malwareserver yara]# cd /var/netwitness/malware-analytics-server/spectrum/yara/processed [root@malwareserver yara]# rm -i custom_merged.yar

 

  1. Restart the Malware service.
systemctl restart rsa-nw-malware-analytics-server

 

After performing these steps, you can verify the remove in the RSA NetWitness Platform UI under Services[name of malware server]ConfigIndicators of CompromiseYARA.

One of the changes introduced in 11.x (11.0, specifically) was the removal of the macros.ftl reference in notification templates.  These templates enable customized notifications (primarily syslog and email) using freemarker syntax. The 10.x templates relied on macros (which are basically just functions, but using freemarker terminology) to build out and populate both the OOTB and (most likely) custom notifications.

 

If you upgraded from 10.x to 11.x and you had any custom notifications, there's a very good chance you noticed that these notifications failed, and if you dug into logs you'd have probably found an error like this:

The good news is there's a very easy fix for this, and it does not require re-writing any of your 10.x notifications.  The contents of macros.ftl file that was previously used in 10.x simply need to be copy/pasted into your existing notification templates, replacing the <#include "macros.ftl"/> line, and they'll continue to work the same as they did in your 10.x environment (props to Eduardo Carbonell for the actual testing and verification of this solution).

 

Example:

 

...becomes:

 

I have attached a copy of the macros.ftl file to this blog, or if you prefer you can find the same on any 11.x ESA host in the "/var/netwitness/esa/freemarker" directory.

G Suite (formerly known as Google Business Suite or Google Apps for Business) is now supported for log collection using the RSA NetWitness Platform.  Collection is achieved via the G Suite Reports API (v1) and is enabled in RSA NetWitness via the plugin framework.

 

 

The G Suite API schema provides several types of events which can be monitored.  Below is the list of event types currently supported by this plugin:

 

  • access_transparency – The G Suite Access Transparency activity reports return information about different types of Access Transparency activity events.
  • admin – The Admin console application's activity reports return account information about different types of administrator activity events.
  • calendar – The G Suite Calendar application's activity reports return information about various Calendar activity events.
  • drive – The Google Drive application's activity reports return information about various Google Drive activity events. The Drive activity report is only available for G Suite Business customers.
  • groups – The Google Groups application's activity reports return information about various Groups activity events.
  • groups_enterprise – The Enterprise Groups activity reports return information about various Enterprise group activity events.
  • login – The G Suite Login application's activity reports return account information about different types of Login activity events.
  • mobile – The G Suite Mobile Audit activity report return information about different types of Mobile Audit activity events.
  • rules – The G Suite Rules activity report return information about different types of Rules activity events.
  • token – The G Suite Token application's activity reports return account information about different types of Token activity events.
  • user_accounts – The G Suite User Accounts application's activity reports return account information about different types of User Accounts activity events.

 

Suggested Use Cases

 

G Suite Admin Report:

 

  1. Top 5 Admin Actions: Depicts the top 5 actions by Admin
  2. Admin activity: Activities performed by admins
  3. App Token Actions: Displays details on app token actions in a pie chart
  4. Users Created and Deleted: Displays users created and deleted as a table chart including details on the user’s email, admin action, and admin email.
  5. Groups - Users Added or Removed: Displays information on Groups, with users added or removed as a table chart including details on the user email, admin action, group email, and admin email.

 

G Suite Activity Report:

 

  1. Activity by IP Address: Shows a table of actions w.r.t IPs
  2. Login State Count: A pie chart that depicts the login states by count
  3. Logins from Multiple IPs: Shows logins from multiple IP addresses by user on a pie chart
  4. Most Active IPs: Shows a table with the most active IP addresses based on the number of events performed by that IP address
  5. Top 10 Apps by Count: Shows the top ten apps by count on a column graph
  6. Login Failures by User: Shows the login failures by user on a pie chart

 

Downloads and Documentation

 

Configuration Guide: Google G Suite 
Collector Package on RSA Live: Google Business Suite Log Collector Configuration
Parser on RSA Live: CEF (device.type='gsuite')

Overview

Sending a notification based on a critical or time-sensitive event seen in your environment is table stakes functionality for any detection platform. Alerting someone in a timely manner is important, but building a custom e-mail that includes relevant, concise information that an analyst can use to determine the appropriate response is just as important. As they work to juggle their daily priorities, they need to know whether an alert requires immediate attention or whether it's something they can filter as a false positive as time permits.

 

The RSA NetWitness Platform uses Apache FreeMarker template engine to build its notifications, be they e-mail, syslog, or SNMP. For the purposes of this post, I'm going to focus on e-mail notifications as the concepts apply to all notifications, and e-mail is the most complex of the options.

 

Available Data

The first step is finding out what information you can include in your notification. All of that data can be seen in the Raw Alert section of an Alert in the Respond UI. That Raw Alert is formatted in JSON, and anything in there can be placed into a notification. To find that Raw Alert data, you can go to one of two places.

 

Location #1:

 

Location #2:

 

Example #1: Basic Email

Let's start with a basic example. I want to send an e-mail that includes the name, severity, and time of the Alert, as well as a link to the raw event (network or log) that generated the alert. Here is a snippet of the data from my Raw Alert (the full alert, with addresses changed to protect the innocent, is attached as raw_alert.json):

 

Under Admin --> System --> Global Notifications, on the Template tab, I add a new template. Give it a name, choose the template type (we're going to select Event Stream Analysis for these), and then paste in the below code (also under example_1.html):

 

Assuming a severity of 9, that gives an e-mail formatted like this (using Gmail):

 

Rows 1 - 20 give us a color-coded banner which highlights the severity of the incident. In rows 3 - 6, you can see that we're making a logical check for the severity to determine the background color of the banner. Row 22 (we'll come back to row 21) prints the rule name. Row 23 gives us the time and includes the field, the input format, and the output format. You can even take epoch time and adjust it for your local time zone, but that's another post. Row 25 builds a hyperlink to the raw event that generated the Alert. Keep in mind that by default, notifications will separate large numbers with commas, which is why row 21 is necessary. Without row 21, the notification link (which I highlighted in the e-mail screenshot) would include commas in the sessionid within the URL, which would obviously not work when clicked. Also, you will need to update two portions of the URL specific to your environment:

 

The [URL_or_IP] is self-explanatory. The [Device_ID] is different for every environment and for every service. If you login to the RSA NetWitness Platform and navigate to the Investigate --> Navigate page and load values, the Device ID will be in the URL string in your browser, and it will correspond to the data source you've selected. In this example, my Broker has a Device ID of 6.

 

Above, we used https://[URL_or_IP]/investigation/[Device_ID]/reconstruction/[Session_ID]/AUTO. This loads the "Default Session View" that each individual user defined in their Profile --> Preferences --> Investigation settings, which by default is "Best Reconstruction" view for network sessions and the "Raw Log" view for log events. Should you prefer to jump directly to other views, you can use these formats:

  • Meta View: https://[URL_or_IP]/investigation/[Device_ID]/reconstruction/[Session_ID]/DETAILS
  • Text View: https://[URL_or_IP]/investigation/[Device_ID]/reconstruction/[Session_ID]/TEXT
  • Hex View: https://[URL_or_IP]/investigation/[Device_ID]/reconstruction/[Session_ID]/HEX
  • Packets View: https://[URL_or_IP]/investigation/[Device_ID]/reconstruction/[Session_ID]/PACKETS
  • Web View: https://[URL_or_IP]/investigation/[Device_ID]/reconstruction/[Session_ID]/WEB
  • Files View: https://[URL_or_IP]/investigation/[Device_ID]/reconstruction/[Session_ID]/FILES

 

Great! Now we have a notification.

 

Example #2: Multiple Values

But what if we have an array of values like analysis_service here:

 

In order to print those multiple values out, we need do some formatting with a FreeMarker macro. I'm pasting the following onto the bottom of my notification:

 

Lines 1 - 11 iterate through any meta value that has more than one value and separate them with a comma. Lines 13 - 22 print out Service Analysis with a comma-separated list of values. First, there is a logical test to see if there are any events in the first place. This was taken from the Default SMTP Template (Admin --> System --> Global Notifications --> Templates tab), and can be used to print out every meta key and all of their values. In my case, I altered it (or, well, Joshua Randall did and I stole borrowed it) to only apply to Service Analysis by adding a logical test (lines 16 and 19) and then only printing out that one meta key. Here is what that looks like:

 

 

If you would like to print out more than one key, you can add elseif statements like this:

 

Testing Your Syntax

So what if you want to use some FreeMarker concepts, but you want to see if they'll work before putting them into the RSA NetWitness Platform? Luckily, there is a tester put out by Apache here - https://try.freemarker.apache.org/.

 

In order to use it on your data, just copy that Raw Alert section from an Alert and paste it into the Data model box shown above. Then paste your FreeMarker code into the Template box and click Evaluate. Keep this in mind: this will not work the same as an RSA NetWitness Platform notification would. If I took the Raw Alert I used for my examples above along with the template I was using, I would not see the output I actually get from the RSA NetWitness Platform. This should ONLY be used to test some basic syntax concepts. For example, printing out UNIX Epoch Time in various formats, adjusted for different time zones, is something this helped me do.

 

Summary

These concepts - along with some basic HTML formatting - give you the tools to build just about any notification you would want. I also recommend taking a peek at the Default SMTP Template I referenced above to use as a starting point for more advanced formatting. If you do some other interesting things or need help getting a notification to work, please post that in the comments below.

One of the most powerful features to make its way into RSA NetWitness Platform version 11.3 is also one of the most subtle in the interface.  11.3 now saves analysts one more step during incident response by integrating rich UEBA, Endpoint, Log, and Full Packet reconstruction directly into the incident panel.  This view is essentially the same as if you were looking at events directly in the Event Analysis part of the UI, or the Users (UEBA) part of the UI, just consolidated into the incident panel.  Prior to this improvement,the only way to view the raw event details was to open the event and click on "Investigate Original Event", pivoting into a new query window.  This option may still be appropriate for some, and still exists, but for those needing the fastest route possible to validating detection and event details, this feature is for you.

 

To use the new feature, for any individual event of interest that has been aggregated or added into an incident you'll see a small hyperlink attached to each event on the left hand side, labeled with one of: "Network", "Endpoint", "Log", "User Entity Behavior Analytics".  These labels correspond to the source of the event, and upon click will slide in the appropriate reconstruction view.

 

User Entity and Behavior Analytics (UEBA) view:

Network packet reconstruction view:

Endpoint reconstruction view:

Log reconstruction view:

 

Happy responding!

Starting in version 11.3, the RSA NetWitness Platform introduced the ability to analyze endpoint data captured by the RSA NetWitness Endpoint Agent (both the free "Insights" version and the full version). For more information on what RSA NetWitness Endpoint is all about, please start with the RSA NetWitness Endpoint Quick Start Guide for 11.3.

 

One of the helpful new features of the endpoint agent is the ability to not only focus the analyst on the "Hosts" context of their environment, but also the ability to gain full visibility into process behaviors and relationships whenever suspicious behaviors have been detected by the RSA NetWitness platform, or when investigating alerts from others.

 

The various pivot points bring an analyst into Process Analysis in the context of a specific process, including it's parent and child process(es) and based on the current analysis timeline which is adjustable if needed.

 

Example Process Analysis view, drilling into all related events recorded by the NW Endpoint Agent

 

Example Process Analysis view, focused on process properties (powershell.exe) collected by the NW Endpoint Agent

 

The feature is simple to use when RSA NetWitness Endpoint agent data exists, and is accessible from a number of locations in the UI depending on where the analyst is in their workflow:

 

Investigate > Hosts > Details (if endpoint alerts exist):

Investigate > Hosts > Processes (regardless of alert/risk score): 

 

Investigate > Event Analysis:

 

Respond > Incident > Event List (card must be expanded):

 

Respond > Incident > Embedded Event Analysis (reconstruction view):

 

Happy Hunting!

Unfortunately sometimes sensitive data can find its way where it is not wanted. It should not, but it happens. Perhaps your IT Person decided connecting the high side network to the low side was a good idea. Maybe someone accidentally uploaded the wrong PCAP (packet capture) to the system. However it happened, there are options to remove that data. If a large amount of data needs to be purged, probably want to start with the storage component (e.g. SAN) to see what capabilities are available. In terms of RSA NetWitness Platform software, one option is to utilize the wipe utility that allows the administrator to strategically overwrite events.

 

  1. The first step is to find the data in question. This can be done via a query either in the RSA NetWitness Investigate user interface, the REST API interface, or the NwConsole. If use the first option will require additional steps to clear user interface cache on the admin server. This is an example of an event found using the Investigate user interface. The PCAP used in this example has one event and was tagged by name during import to make it easier to query.



  2. After you execute the query make note of the session ID (sid) and remote ID (rid) that can be seen here using a custom column group. They are both in the above view as well, but have to scroll down the list of meta to find the remote id. 



  3. Starting with the concentrator, use the wipe command against those session IDs to overwrite them with a pattern.
    • There are multiple options to the wipe command.
      • session - <uint64> The session id whose packets will be wiped
      • payloadOnly - <bool, optional> If true (default), will only overwrite the packet payload
      • pattern - <string, optional> The pattern to use, by default it uses all zeros
      • metaList - <string, optional> Comma separated list of meta to wipe, default (empty) is all meta
      • source - <string, optional, {enum-any:m|p}> The types of data to wipe, meta and/or packets, default is just packets
    • Note that if you use a string as your pattern it will not overwrite any meta values that are not a string type. Therefore best to keep the pattern as a numerical value.
    • Initially go to the concentrator that was found to have those session IDs (sids) and use the wipe command to overwrite the session meta data on disk.



  4. Rinse and repeat this on the upstream service (e.g. decoder, log decoder) in the path of the query. This time use the remote session IDs (rids) to overwrite the raw sessions on disk.



  5. To ensure that the indexed meta values that were stored on the Concentrator are removed, rebuild the index. This can take a long time but is necessary because the wipe command does not remove any data from the Concentrator index. Refer to the Core Database Tuning Guide for instructions.
  6. Now that you have overwritten the data on the decoder, where it was ingested, and the concentrator, where meta related to it was created, you're done right? Well it depends on how you discovered the data in the first place. If you know for sure no one found the data by way of the RSA NetWitness Platform user interface you should be done. If the user interface was used or you just want to be on the safe side continue to the next step. Otherwise might still see the raw event data being rendered from cache like below.



    • If the Investigate > Event Analysis was used to find the data the cache for the event reconstruction should be cleared by restarting the Investigate service.



    • If the Investigate > Events was used to find the data the event reconstruction cache should be cleared by removing the contents of the service folders on the admin server as shown below.



    • The cache for the concentrator and the decoder can also be cleared by executing the delCache command in Admin > Services > sdk > properties for each as shown below.



    • After clearing the cache attempting to view the same session that was wiped you will see the event is unavailable for viewing.

 

To gain further knowledge on protecting the data stored within your RSA NetWitness system take a look at the Data Privacy Management Guide.

(Authored by Steve Schlarman, Portfolio Strategist, RSA)

It was Mark’s big shot.  He finally had a meeting with Sharon, the CIO.  Her schedule was so busy it was legendary and for her to spend time with a risk analyst was a clear indicator she recognized the new challenges facing their company.  Although he only had 15 minutes, Mark was prepared - notepad at the ready, brimming with nervous energy.   After some brief chit-chat he got down to business – ready to drill into a conversation about their company’s biggest obstacles; the most impactful concerns; the top of mind issues; the coup de grace that could spell disaster for the organization.  He took a deep breath and went to his big money question… ‘So, what keeps you up at night? What are you worried about?’ 

 

Sharon beamed.  She spun around to her white board and spewed a litany of projects fueling their company’s digital transformation – an IoT project, the SalesForce.com implementation, a massive VMWare migration and their hybrid cloud, the new employee work-at-home program, the impending customer mobile portal…

While that question got Sharon started, let’s think about this a bit differently.

 

With all the benefits the new digital world offers, there are a host of risks that must be managed.   The major areas of risk remain the ‘usual suspects’ such as security, compliance, resiliency, inherited risks from third parties and operational risk. However, digital business amplifies uncertainty for organizations today.  For example:

  • Digital business, by its very nature, increases the threat of cyber incidents and risks around your intellectual property and customer data.
  • The expanded connectivity and expectations of the ‘always on’ business stresses the importance of resiliency.
  • Business has evolved into an ecosystem of internal and external services and processes leading to a complex web of ‘inherited’ risks.
  • The disappearing perimeter and digital workforce is challenging how organizations engage their customers and employees.

 

Factors such as these are why digital initiatives are forcing organizations to rethink and increasingly integrate their risk and security strategies. 

The objective for today’s risk professional is not just about defending against the bad.  Just like Mark discussing the parade of initiatives with Sharon that clearly impact their company’s future, you must be ready to help usher in a new age of digital operations.  Merely riding the buzzword wave - IoT, social media, big data analytics, augmented reality… - is not enough. 

 

You must look at opportunities to enable innovation in your business while building trust with your customers and throughout your enterprise.  Your business must be comfortable with embracing risk and aggressively pursuing market opportunities offered by new technology.  To do that, risk associated with the use of emerging or disruptive technology in transforming traditional business processes needs to be identified and assessed in the context of fueling innovation.   You also must keep focus on the negative side of risk.  Your business today demands an open, yet controlled, blend of traditional and emerging business tactics.  You must help manage the ongoing risk as these transformed business operations are absorbed into the organization fully, i.e. the new model becomes the normal model of doing business.

Risk is, by definition, uncertainty.  Everyone is concerned about uncertainty in today’s world.  However, if we go back to the simple equation (risk = likelihood * impact), risk should be something we can dissect, understand, and maybe even calculate.   While you are helping your organization embrace the advantages (positive risk) of technologies like IoT, data analytics, machine learning and other emerging digital enablers, the volatile, hyperconnected nature of digital business amplifies the negative side of risk.  It is anxiety about the unknown that leads us into that executive conversation, but it shouldn’t lead to worry.

Worry is about fear.  Your executives shouldn’t be afraid in today’s world.   They should have informed concerns.  And you – as the security or risk person in the room – should be feeding insights to raise their visibility of the likelihood of events and diminish their distress on the negative impacts.  Risk is part of riding the waves of business opportunities.

Risk is not something you should WORRY about…  it is something you should ACT on.

***********

 

To learn more about digital risk management, click on our new Solutions Banners located in the right-hand column of each RSA product page: Third Party RiskCloud TransformationDynamic Workforce, and Cyber Attack Risk.

 

Rui Ataide

Domain Fronting Malware

Posted by Rui Ataide Employee Jun 19, 2019

Customers frequently ask me about malware that uses domain fronting and how to detect it. Simply put, domain fronting is when malware or an application pretends to be going to one domain but instead is going somewhere completely different. (Mitre ATT&CK - T1172)

 

The goal of domain fronting is to have the analysts believe that the connection is being a made to a safe site while the true destination is in fact somewhere completely different.

 

Let’s look at a piece of malware that uses this method. This is a PowerShell Empire sample:

 

 

In the configuration information of this file, we see a URL that will be requested, which is also Base64 encoded. The URL decodes to https://ajax.microsoft.com:443 as seen below:

 

 

So, this script will initiate a connection to ajax.microsoft.com:443, and appear to request /login/process.php. However, because the Host: header is pointing to content-tracker.*******.net, the request will actually go to https://content-tracker.*********.net/login/process.php instead.

 

You may be thinking that all you have to do in order to detect examples of domain fronting is to look for discrepancies between the requested URL and the domain/IP in the Host: header. However, there are  some complexities to deal with. Most of the time the initial connection is SSL encrypted, so you are limited to artifacts related to SSL traffic, unless you have SSL inspection technology in place. Another consideration is if whether a proxy is involved in this connection or not.

 

In order to describe what the analyst would see if SSL inspection technology is in place, let us use a Man-In-The-Middle proxy to inspect this traffic in its clear-text form. SSL Inspection technologies are extremely useful for this and other scenarios where malware communicates over SSL, and it is something that we highly recommend that you deploy in your organization.

 

Let us introduce two terms here to clarify the elements of this technique as we describe how to hunt for it:

            Fronting Domain (ajax.microsoft.com)

            Fronted Domain (content-tracker.*********.net)

 

Here we see the Fronting Domain request, which is also the only thing you would see if you were only relying on proxy logs. The Domain Fronting domain (ajax.microsoft.com in this case) is also what the proxy would use for URL Filtering checks. The proxy logs would not actually see the “Fronted Domain”. For all intents and purposes this would be a legitimate request to a Microsoft site.

 

 

 

However, the response is anything but what you would expect from the site. Namely, instead of some HTTP content the site returns an encoded blob of data that decodes into more PowerShell code.

 

How do I know, it was more PowerShell code, that was easy, I simply replaced the follow-up execution with the output to a file as seen below: 

 

 

Then opening the resulting stage2.ps1 file, you can see to contains additional PowerShell code that is highly obfuscated.

 

 

Let us go back one step and discuss another key aspect regarding Domain Fronting. Namely, the SSL certificates used during this communication. The SSL certificates are legitimate Microsoft signed certificates, since the initial connection is indeed to ajax.microsoft.com. This certicate is tied to many Microsoft domains and Microsoft CDN Domains.

 

 

We could try to de-obfuscate the stage2.ps1 PowerShell script but there really is no need, since by looking at the subsequent request of the malware on the proxy we can get an idea of what it does. Its initial check-in posting back victim information again in an encrypted binary blob of data.

 

 

Additionally, this particular strain of malware also seems to do a legitimate call to the ajax.microsoft.com site as shown below. While not at all relevant for domain fronting, it is important for the analysts to be aware as to why they might see both legitimate and malicious requests mixed together. The analyst will notice that the "Host:" header will match the requested domain in legitimate requests. 

 

 

 

The response from the legitimate site is also completely different and starts a redirect chain that we will show below:

 

 

And finally, the legitimate page for Microsoft Ajax Content Delivery Network.

 

 

Now that we have described in detail the sequence of requests, let us see how this all looks from the Netwitness Packet perspective. There are two cases, one where there is no proxy and one where there is one.

 

Traffic Analysis – SSL Only

 

Let’s start with the traffic without a proxy. I have isolated only the relevant events in a separate collection to facilitate the analysis. I will also point out how some of these indicators can be spotted in larger traffic patterns.

 

In the example below, the indicators are separated in two sessions: a DNS request and an SSL connection. You can see that the DNS request is for one domain name, while the SSL session displays what is referred to as the SNI, which does not match the DNS request.

 

 

For the legitimate traffic, the DNS request and the SSL SNI value both match. These are both extracted into the alias.host key.

 

 

So, how can you detect this type of behavior? It is not easy, especially on high volume environments. However, a starting point is to look for are alias.host values that only show one of the service types (DNS or SSL), but not both. Legitimate traffic will likely have both as shown below:

 

 

You should not expect these values to be balanced or equal as DNS is often cached, but you should expect to see both types of service. Some environments at times do not capture DNS due to volume, but to be successful it is critical to have both.

 

For the malicious traffic each domain will only have one type of traffic (i.e. DNS or SSL). This detection criteria is not  an exact “science” as you could easily have only DNS for all sorts of other types of traffic that are not domain fronting. The Fronting Domain will have the DNS traffic, while the Fronted Domain will have the SSL sessions.

 

 

 

Since the traffic is split between sessions on the packet side, we would need to use an ESA rule to detect this type of activity.

Traffic Analysis – Proxied Requests

 

For explicit proxied traffic things are slightly easier, as all the traffic is contained in a single session. We see the "raw" payload of one such session below. It can seem confusing at first, but Netwitness identifies this traffic as HTTP. This is correct since this part of the traffic is indeed HTTP.

 

 

Since we have all the pieces in one session here, the detection is easier. But how can we do it for high data volumes. In this case the HTTP session will have two different hostnames. While this is at times common for pure HTTP traffic due to socket re-use, it is uncommon for HTTPS/SSL traffic as the standards advise against it for privacy/security purposes, among other reasons.

 

 

This shows a possible solution to detect this type of traffic with a simple App rule that could identify traffic HTTP with 2 unique alias.host values and the presence of a certificate.

 

In summary, domain fronting is a technique used by attackers/red teams with the intent of either circumventing network policies (URL filtering), or hiding in plain sight, as the analysts are more likely to see/notice the legitimate domains than the malicious ones and assume this activity as safe/legitimate. However, this type of activity still has a certain footprint that we have described. Hopefully the information provided here will help you all improve your defenses against this technique.

 

Thank you,

 

Rui

If you need to achieve HA through load balancing and failover for VLCs on AWS you can use the built-in AWS load balancer. I have tested this scenario so I am going to share the outcome here.

 

Before starting I need to state that VLCs failover/balancing  is not an RSA officially supported functionality. Furthermore this can only work with "push" collections such as syslog, snmp, etc. It does not work with "pull" collections such us Windows, Checkpoint, ODBC, etc. (at least not that I am aware of and I have personally never tested it).

 

That being said, let's get started.

 

As you may be aware, in AWS EC2 you have separate geographic areas called Regions (I am using US East - N.Virgina here) and within regions you have different isolated locations called Availability Zones.

 

 

We are going to leverage this concept and we will place two VLCs into two different Availability Zones. If one VLC fails we will have the VLC in the other Availability Zone to take over.

 

The following diagram helps understanding the scenario (for better clarity I omitted the data flow from the VLCs to the Log Decoder/s):

 

Assuming you have already deployed the two VLC instances, the next step to do is creating two different subnets and associate two different Availability Zones to each of them .

 

  • From the AWS Virtual Private Cloud (VPC) menu go to Subnets and start creating the two subnets:

 

 

  • Next we need to create a Target Group (from the EC2 menu) which will be used to route requests to our registered targets (the VLCs):

     

 

  • Finally we need to create the load balancer itself. For this specific test I have used a Network Load Balancer but I think an Application Load Balancer would work too. I selected an internal balancer. I chose syslog on TCP port 514 so I created a listener for that. Actually, the AWS load balancer does not support UDP so I was forced to use TCP, however I would have used syslog over TCP anyway as it is more robust and reliable and large syslog messages can be transferred (especially if it is a production environment). I also select the appropriate VPC and the Availability Zones (and subnets) accordingly.  

     

 

In the advanced health check settings I chose to use port 5671 (by default the balancer would have used the same as the listener, 514). The reason of using 5671 is because the whole log collection mechanism works with rabbitmq which uses this port. In fact the only scenario 514 would not work is when the VLC instance is down or if we stop the syslog collection. I think rabbitmq is more prone to failures and may fail in more scenarios, such as queues filling up because the decoder is not consuming the logs, full partitions, network issues, etc. 

 

 

  • Once the load balancer configuration is finished you will see something similar:

 

 

           We need to take note of the DNS A Record as this is what our event sources will use to send syslog traffic to.

 

  • Now to configure an event source to send syslog logs to the load balancer you just need to point the event source to the load balancer DNS A Record. As an example, for a Red Hat Linux machine you should edit the /etc/rsyslog.conf file as follow:

 

  

 

         We are using @@ because is TCP, for UDP it's just one @.

 

         Then we need to restart the rsyslog service as follow:

 

            --> service rsyslog restart (Red Hat 6)

            --> systemctl restart rsyslog (Red Hat 7)

 

  • To perform a more accurate and controlled test and demonstration, I am installing a tool on the same event source and I will push some rhlinux logs to the load balancer and see what happens. The tool is an RSA proprietary one and is called NwLogPlayer (more details here How To Replay Logs in RSA NetWitness ). It can be installed via Yum if you have enabled the RSA Netwitness repo:

 

   

 

      I also prepared a rhlinux sample logs file with 14000 events and I am going to inject these to the load balancer and       see what happens. Initially my Log Decoder LogStats page is empty:

 

  

 

     Then I start with the first push of the 14000 events:

 

 

     Now I can see the first 14000 events went to VLC2 (172.24.185.126)

 

      At my second push I can see the whole chuck going to VC1 (172.24.185.105)

 

      At the third push the logs went again to VLC2

 

     At the fourth push the logs went to VLC1

 

     At the fifth push, I sent 28000 events (almost simultaneously)  and they get divided to both VLCs

 

     This demonstrates that the load has been balanced equally between the two VLCs.

 

     Now I stop VLC1 (I actually stopped the rabbitmq-service on VLC1) and I push other 14000 logs:

 

     and again

 

     On both instances above VLC2 received the two chunks of 14000 logs since VLC1 was down. We can safely say            that Failover is working fine!

Note: This configuration is not officially supported by RSA customer support. 

I have recently been posting a number of blogs regarding the usage of the RSA NeWitness Platform to detect attackers within your environment. As the list of the blogs grow, it is becoming increasingly difficult to navigate through them easily. In order to combat this, this blog post will contain references to all other blog posts in the Profiling Attackers Series, and will be updated when new posts are made.

 

 

 

 

 

 

 

Special thanks to Rui Ataide for his support and guidance for these posts.

Introduction

Cobalt Strike is a threat emulation tool used by red teams and advanced persistent threats for gaining and maintaining a foothold on networks. This blog post will cover the detection of Cobalt Strike based off a piece of malware identified from Virus Total:

 

NOTE: The malware sample was downloaded and executed in a malware VM under analysts constant supervision as this was/is live malware.

The Detection in NetWitness Packets

NetWitness Packets pulls apart characteristics of the traffic it sees. It does this via a number of Lua parsers that reside on the Packet Decoder itself. Some of the Lua parsers have option files associated with them that parse out additional metadata for analysis. One of these is the HTTP Lua parser, which has an associated HTTP Lua options file, you can view this by navigating to Admin  Services ⮞ Decoder ⮞ Config ⮞ Files - and selecting HTTP_lua_options.lua from the drop down. The option we are interested in for this blog post is the headerCatalog() - making this return true will register the HTTP Headers in the request and response under the meta keys:

  • http.request
  • http.response

 

And the associated values for the headers will be registered under:

  • req.uniq
  • resp.uniq

 

NOTE: This feature is not available in the default options file due to potential performance considerations it may have on the Decoder. This feature is experimental and may be deprecated at any time, so please use this feature with caution, and monitor the health of all components if enabling. Also, please look into the customHeader() function prior to enabling this, as that is a less intensive substitute that could fit your use cases.

 

There are a variety of options that can be enabled here. For more details, it is suggested to read the Hunting Guide - https://community.rsa.com/docs/DOC-62341.

 

These keys will need to be indexed on the Concentrator, and the following addition to the index-concentrator-custom.xml file is suggested:

<key description="HTTP Request Header" format="Text" level="IndexValues" name="http.request" defaultAction="Closed" valueMax="5000" />
<key description="HTTP Response Header" format="Text" level="IndexValues" name="http.response" defaultAction="Closed" valueMax="5000" />
<key description="Unique HTTP Request Header" level="IndexKeys" name="req.uniq" format="Text" defaultAction="Closed"/>
<key description="Unique HTTP Response Header" level="IndexKeys" name="resp.uniq" format="Text" defaultAction="Closed"/>

 

 

The purpose for this, amongst others, is that the trial version of Cobalt Strike has a distinctive HTTP Header that we, as analysts, would like to see: https://blog.cobaltstrike.com/2015/10/14/the-cobalt-strike-trials-evil-bit/. This HTTP header is X-Malware - and with our new option enabled, this header is easy to spot:

NOTE: While this is one use case to demonstrate the value of extracting the HTTP Headers, this metadata proves incredibly valueable across the board, as looking for uncommon headers can help lead analysts to uncover and track malicious activity. Another example where this was useful can be seen in one of the previous posts regarding POSH C2, whereby an application rule was created to look for the incorrectly supplied cachecontrol HTTP response header: https://community.rsa.com/community/products/netwitness/blog/2019/03/04/command-and-control-poshc2

 

Pivoting off this header and opening the Event Analysis view, we can see a HTTP GET request for KHSw, which was direct to IP over port 666 and had a low header count with no referrer - this should stand out as suspicious even without the initial indicator we used for analysis:

 

If we had decided to look for traffic using the Service Analysis key, which pulls apart the characteristics of the traffic, we would have been able to pivot of off these metadata values to whittle down our traffic to this as well:

 

Looking into the response for the GET request, we can see the X-Malware header we pivoted off of, and the stager being downloaded. Also, take notice of the EICAR test string in the X-Malware as well, this is indicative of a trial version of Cobalt Strike as well:

 

NetWitness Packets also has a parser to detect this string, and will populate the metadata, eicar test string, under the Session Analysis meta key (if the Eicar Lua parser is pushed from RSA Live) - this could be another great pivot point to detect this type of traffic:

 

Further looking into the Cobalt Strike traffic, we can start to uncover more details surrounding its behaviour. Upon analysis, we can see that there are multiple HTTP GET requests with no error (i.e. 200), and a content-length of zero, which stands out as suspicious behaviour - as well as this, there is a cookie that looks like a Base64 encoded string (equals at the end for padding) with no name/value pairs, cookies normally consist of name/value pairs, these two observations make the cookie anomalous:

 

Based off of this behaviour, we can start to think about how to build content to detect this type of behaviour. Heading back to our HTTP Lua options file on the Decoder, we can see another option named, customHeaders() - this allows us to extract the values of HTTP headers in a field of our choosing. This means we can choose to extract the cookie into a meta key named cookie, and content-length into a key named http.respsize - this allows us to map a specific HTTP header value to a key so we can create some content based off of the behaviours we previously observed:

 

After making the above change, we need to add the following keys to our index-concentrator-custom.xml file as well - these are set to the index level of, keys, as the values that can be returned are unbounded and we don't want to bloat the index:

<key description="Cookie" format="Text" level="IndexKeys" name="cookie" defaultAction="Closed"  />
<key description="HTTP Response Size" format="Text" level="IndexKeys" name="http.respsize" defaultAction="Closed" />

 

Now we can work on creating our application rules. Firstly, we wanted to alert on the suspicious GET requests we were seeing:

service = 80 && action = 'get' && error !exists && http.respsize = '0' && content='application/octet-stream'

And for the anomalous cookie, we can use the following logic. This will look for no name/value pairs being present and the use of equals signs at the end of the string which can indicate padding for Base64 encoded strings:

service = 80 && cookie regex '^[^=]+=*$' && content='application/octet-stream'

These will be two separate application rules that will be pushed to the Decoders:

 

Now we can start to track the activity of Cobalt Strike easily in the Investigate view. This could also potentially alert the analyst to other infected hosts in their environment. This is why it is important to analyse the malicious traffic and create content to track:

 

Conclusion

Cobalt Strike is a very malleable tool. This means that the indicators we have used here will not detect all instances of Cobalt Strike, with that being said, this is known common Cobalt Strike behaviour. This blog post was intended to showcase how the usage of the HTTP Lua options file can be imperative in identifying anomalous traffic in your environment whilst using real-world Live malware. The extraction of the HTTP headers, whilst a trivial piece of information, can be vital in detecting advanced tools used by attackers. This coupled with the extraction of the values themselves, can help your analysts to create more advanced higher fidelity content.

Filter Blog

By date: By tag: