AnsweredAssumed Answered

User Agent to Device/OS/Application

Question asked by Alexey Fedorov on Mar 16, 2016
Latest reply on Mar 22, 2016 by David Waugh

Like • Show 1 Like1
Comment • 36

Hello,

Somobody has solution how I can transformate data from user.agent metakey (Log and Packet) to some like Device, OS, Application via Feed or LUA Parser? I found some FlexParser in RSA Live, but it is not I want. I found some site like Complete List of iOS User-Agent Strings | Enterprise iOS with DB of iPhone/iPad/IPod, but data is old and I can't create suitable Feed.

I found interesting progect GitHub - piwik/device-detector: The Universal Device Detection library will parse any User Agent and detect the browser,… and his web version Device Detector Demo

It is what I want, but how transformate it in LUA Parser or Feed?

David Waugh

Mar 17, 2016 8:06 AM

Correct Answer

Okay I have a working solution.

- I created a PHP Page page on a Webserver based on the DeviceDetector project (Install the DeviceDetector project on your own PhP enabled webserver then use copy the file hello.php to the same directory where DeviceDetector.php resides)

- I created a shellscript that sits on my PhP Enabled webserver that queries my broker for UserAgents on Service 80 and then creates a feed based on the values returned.

-Edit the line in the UserAgentInfo.sh to replace user 'admin:netwitness' 'http://192.168.123.249:50103' with an account and broker that is able to get values. I used admin netwitness because I was a bit lazy and because this is just a test system!

The script creates a file called /var/www/html/useragentfeed.csv that can then be read by Security Analytics as a feed. Note in the feed definition file make sure that the callback key you are using is case-insensitive. The attached DeviceDetector.xml file contains a feed definition example.

The solution could be tweaked for your own environment but as a proof of concept I'm happy with it =)

Here is a Demo of the hello.php webpage. It basically takes the useragent in the Query and then outputs the findings.

The final output will be the Feed file which I have attached.

I added the following meta keys to my concentrators to use the feed:

See the reply in context

1 person had this question

Outcomes

36 Replies

David Waugh Mar 16, 2016 5:27 AM
Mark Correct
Correct Answer
Hi Alex, I'll take a look. First thoughts are a "Right Click Menu Item" to link to the site which would be quite simple.
Getting it as a feed looks to be more tricky as there is not just an file that you can download.
Like • Show 0 Likes0
Actions
- Alexey Fedorov @ David Waugh on Mar 16, 2016 5:41 AM
  Mark Correct
  Correct Answer
  I agree with you, for investigation process is simple way, but many customers (top managers) want to have those data on reports and dashboards. From MS IIS user agent strings looks like Apple-iPad2C2/1304.15 or Microsoft+Office/15.0+(Windows+NT+6.1;+Microsoft+Outlook+15.0.4787;+Pro).
  If Feed can be use to contains data it is will be simple...
  Like • Show 0 Likes0
  Actions
David Waugh Mar 16, 2016 5:46 AM
Mark Correct
Correct Answer
Okay for a Right Click Menu Action. This will work on Client Meta Key.

Go to Administration ->System -> Context Menu Actions and Click on + to create a New Context Menu action with the following.

{
    "displayName": "DeviceDetector",
    "cssClasses": [
        "client"
    ],
    "description": "",
    "type": "UAP.common.contextmenu.actions.URLContextAction",
    "version": "1",
    "modules": [
        "investigation"
    ],
    "local": "false",
    "groupName": "externalLookupGroup",
    "urlFormat": "http://devicedetector.net//index.php?ua={0}",
    "disabled": "",
    "id": "DeviceDetectorSearch",
    "moduleClasses": [
        "UAP.investigation.navigate.view.NavigationPanel",
        "UAP.investigation.events.view.EventGrid"
    ],
    "openInNewTab": "true",
    "order": "21"
}

Now when you have a client that you wish to investigate - right click

1 person found this helpful
Like • Show 1 Like1
Actions
- Alexey Fedorov @ David Waugh on Mar 16, 2016 5:59 AM
  Mark Correct
  Correct Answer
  I made a little be changes (add user-agent metakey and remove extra "/" in url):
  
  {
      "displayName": "DeviceDetector",
      "cssClasses": [
          "client",
          "user-agent"
      ],
      "description": "",
      "type": "UAP.common.contextmenu.actions.URLContextAction",
      "version": "1",
      "modules": [
          "investigation"
      ],
      "local": "false",
      "groupName": "externalLookupGroup",
      "urlFormat": "http://devicedetector.net/index.php?ua={0}",
      "disabled": "",
      "id": "DeviceDetectorSearch",
      "moduleClasses": [
          "UAP.investigation.navigate.view.NavigationPanel",
          "UAP.investigation.events.view.EventGrid"
      ],
      "openInNewTab": "true",
      "order": "21"
  }
  Like • Show 0 Likes0
  Actions
David Waugh Mar 16, 2016 5:48 AM
Mark Correct
Correct Answer
For the feed I'll need to dig a bit deeper....
Like • Show 0 Likes0
Actions
- Alexey Fedorov @ David Waugh on Mar 16, 2016 6:03 AM
  Mark Correct
  Correct Answer
  I thinks it should be a parser. Look at How DeviceDetector works (About ). They use YML files with regex (GitHub). Maybe you can connect this to your LUA parser...
  Like • Show 0 Likes0
  Actions
David Waugh Mar 16, 2016 9:05 AM
Mark Correct
Correct Answer
I'm afraid my knowledge of PHP is not that great.
Like • Show 0 Likes0
Actions
- Alexey Fedorov @ David Waugh on Mar 16, 2016 9:24 AM
  Mark Correct
  Correct Answer
  Maybe you can try write you own parser who will parse string like (from metakey client and user.agent):
  
  Mozilla/5.0 (Linux; Android 4.4.2; Nexus 5 Build/KOT49H) AppleWebKit/537.36 (KHTML,
  like Gecko) Chrome/32.0.1700.99 Mobile Safari/537.36
  
  And write Android 4.4.2 is metakey OS, Chrome/32.0.1700.99 in metakey browser and Nexux 5 in some else metakey?
  
  But string can have different, for example:
  
  Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
  Microsoft+Office/15.0+(Windows+NT+6.1;+Microsoft+Outlook+15.0.4787;+Pro)
  Apple-iPad2C2/1304.15
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Alexey Fedorov on Mar 16, 2016 1:24 PM
    Mark Correct
    Correct Answer
    Have been having a think about this. I think the best way to do this would be to
    
    - Export All User Agents from Security Analytics
    - Send each UserAgent to the devicedetector website
    -Get the results back and put into a CSV file to use as a feed.
    Like • Show 0 Likes0
    Actions
    - Alexey Fedorov @ David Waugh on Mar 16, 2016 1:35 PM
      Mark Correct
      Correct Answer
      Maybe you are right, but I think it is not unique solution. I hope find solution situable for all customer...
      Like • Show 0 Likes0
      Actions
    - Alexey Fedorov @ David Waugh on Mar 16, 2016 3:19 PM
      Mark Correct
      Correct Answer
      I guess I found solution situable for all customer. We can use Application Rule and regex. In the source code of Device Detector has strings like:
      
      regex: '(?:CPU OS|iPh(?:one)?[ _]OS|iOS)[ _/](\d+(?:[_\.]\d+)*)'
      name: 'iOS'
      version: '$1'
      
      I prepare set of Application Rule for most common OS, Device and Application and share for all. I need few days...
      Like • Show 0 Likes0
      Actions
      - David Waugh @ Alexey Fedorov on Mar 16, 2016 5:40 PM
        Mark Correct
        Correct Answer
        Hi that will be a lot of work. You will basically be duplicating the work of the DeviceDetector project.
        
        So far I have:
        - Written a PHP Web Page that I can submit my own User Agents to
        - Written a Small Script that will submit the Useragent String to the WebPage and output it in feed friendly manner.
        
        As an example:
        
        ./UserAgentInfo.sh "Mozilla/5.0 (phone; CPU iPhone OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1"
        
        returns:
        
        Mozilla/5.0 (phone; CPU iPhone OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1^browser^Mobile Safari^MF^9.0^^iOS^IOS^9.2.1^^AP^iPhone
        
        My feed seprater will be ^
        The column Names are:
        Useragent String^Client type^Client Name^ Client Short Name^Client Version^Client Platfrom^OS Name^OS Short Name^OS Version^OS Platform^Brand^Model
        
        My next plan is to schedule a job to download all the uncategorised Client Usernames from a broker and then feed each line into the script to create a feed file.
        I'll keep you posted!
        Like • Show 0 Likes0
        Actions
        Alexey Fedorov @ David Waugh on Mar 17, 2016 4:31 AM
        Mark Correct
        Correct Answer
        This is not dedup work of DeviceDetector project. I copy-past from their format to format application rule. Now I have finished move OS (oss.yml file from project) to Application Rule and attach result of my work. I ran into a problem with version number and not correct detect Windows NT and GNU/Linux OS. I can't put version number into rule name for some rules. I aslo can't stop filtered Windows NT if user agent string contains both Windows X and Windows NT values.
        Maybe you devide you work and will go by modules? On the first stage move oss.yml to LUA parser (conver or include) and then go to browsers and etc.? It will be very cool, you can make many very good LUA parsers situable for any customers. At this time in RSA Live not present this content.
        I also need LUA parser that can replace "+" to space on the user.agent metakey for correct work with user agent string from MS IIS.
        Attachments
        OSDetectionPackets.nwr.zip2.6 KB
        OSDetectionLogs.nwr.zip2.6 KB
        Like • Show 0 Likes0
        Actions
David Waugh Mar 17, 2016 8:06 AM
Unmark Correct
Correct Answer
Okay I have a working solution.

- I created a PHP Page page on a Webserver based on the DeviceDetector project (Install the DeviceDetector project on your own PhP enabled webserver then use copy the file hello.php to the same directory where DeviceDetector.php resides)
- I created a shellscript that sits on my PhP Enabled webserver that queries my broker for UserAgents on Service 80 and then creates a feed based on the values returned.
-Edit the line in the UserAgentInfo.sh to replace user 'admin:netwitness' 'http://192.168.123.249:50103' with an account and broker that is able to get values. I used admin netwitness because I was a bit lazy and because this is just a test system!

The script creates a file called /var/www/html/useragentfeed.csv that can then be read by Security Analytics as a feed. Note in the feed definition file make sure that the callback key you are using is case-insensitive. The attached DeviceDetector.xml file contains a feed definition example.

The solution could be tweaked for your own environment but as a proof of concept I'm happy with it =)

Here is a Demo of the hello.php webpage. It basically takes the useragent in the Query and then outputs the findings.
The final output will be the Feed file which I have attached.

I added the following meta keys to my concentrators to use the feed:


<key description="ClientInfo Type" format="Text" level="IndexValues" name="clientinfo.type" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Name" format="Text" level="IndexValues" name="clientinfo.name" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Short Name" format="Text" level="IndexValues" name="clientinfo.sname" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Version" format="Text" level="IndexValues" name="clientinfo.ver" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Platform" format="Text" level="IndexValues" name="clientinfo.plat" defaultAction="Open" valueMax="500000"/>
<key description="OS Type" format="Text" level="IndexValues" name="os.type" defaultAction="Open" valueMax="500000"/>
<key description="OS Name" format="Text" level="IndexValues" name="os.name" defaultAction="Open" valueMax="500000"/>
<key description="OS Short Name" format="Text" level="IndexValues" name="os.sname" defaultAction="Open" valueMax="500000"/>
<key description="OS Version" format="Text" level="IndexValues" name="os.ver" defaultAction="Open" valueMax="500000"/>
<key description="OS Platform" format="Text" level="IndexValues" name="os.platform" defaultAction="Open" valueMax="500000"/>
<key description="Brand" format="Text" level="IndexValues" name="brand" defaultAction="Open" valueMax="500000"/>
<key description="Model" format="Text" level="IndexValues" name="model" defaultAction="Open" valueMax="500000"/>
Attachments
- useragentfeed.csv.zip2.7 KB
- DeviceDetector.xml.zip505 bytes
- UserAgentInfo.sh1,007 bytes
- hello.php.zip1.0 KB
2 people found this helpful
Like • Show 1 Like1
Actions
- Alexey Fedorov @ David Waugh on Mar 17, 2016 7:26 AM
  Mark Correct
  Correct Answer
  I will check you solution.
  
  Can I ask you about LUA Parser what will replace all "+" to " " (space) on metakey? From MS IIS user agent string come like "Mozilla/5.0+(Windows+NT+6.2;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/47.0.2526.106+Safari/537.36". I need transformate it to "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" for correct determinate.
  
  And second question. I found yaml to CSV online converter YAML To CSV Converter . If put to this convertor file oss.yml and convert it - you get result like:
  "Tizen[ /]?(\d+[\.\d]+)?","Tizen","$1"
  "Sailfish|Jolla","Sailfish OS",""
  "(?:Ali)?YunOS[ /]?(\d+[\.\d]+)?","YunOS","$1"
  "Windows Phone (?:OS)?[ ]?(\d+[\.\d]+)","Windows Phone","$1"
  "XBLWP7|Windows Phone","Windows Phone",""
  
  First field - regex, Second Filed - OS name, Third field - OS version
  Can you wrap this array to LUA perser and merge (options) second and third fields? This parser will very simple support in future if you can check regex from first filed to metakey and write result (merged second and third fileds) to another metakey(s), for example OS (version). What do you think about it? This structura can be unique for some other parsers.
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Alexey Fedorov on Mar 17, 2016 8:09 AM
    Mark Correct
    Correct Answer
    I'll take a look at the first question. =)
    Like • Show 1 Like1
    Actions
    - David Waugh @ David Waugh on Mar 17, 2016 12:27 PM
      Mark Correct
      Correct Answer
      Here is the LUA Parser that will replace + in IIS User Agents with Spaces.
      Attachments
      RepairIISUA.lua.zip1.2 KB
      1 person found this helpful
      Like • Show 1 Like1
      Actions
      - Alexey Fedorov @ David Waugh on Mar 17, 2016 1:03 PM
        Mark Correct
        Correct Answer
        The parser works greak! Thanks you so much!
        I delete nw.logInfo strings... and change envisionName="useragent" to envisionName="user_agent" in the new table-map-custom.xml string.
        Like • Show 0 Likes0
        Actions
- Alexey Fedorov @ David Waugh on Mar 17, 2016 12:09 PM
  Mark Correct
  Correct Answer
  David, can I put DeviceDetector project on some RSA Security Analytics component? For example on SA Server? I don't have any other Web Servers...
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Alexey Fedorov on Mar 17, 2016 12:34 PM
    Mark Correct
    Correct Answer
    Hi Alexey I wouldnt recommend it and there isnt a web server on the SA Server than runs a PHP webserver either as far as I know.
    
    The alternative would be to run the PHP script from the command line but I didnt have much luck with that. My PHP skills are a little limited!
    Like • Show 0 Likes0
    Actions
    - Alexey Fedorov @ David Waugh on Mar 17, 2016 1:11 PM
      Mark Correct
      Correct Answer
      Hello David,
      
      PHP installed on the RSA Security Analytics by default? You solution is great, but if need additional Virtual Machine or Physical Server with PHP Web Server it can be problem. For example for me it is not applicable at this time. Solution should works on the current instance.
      Like • Show 0 Likes0
      Actions
      - David Waugh @ Alexey Fedorov on Mar 17, 2016 1:21 PM
        Mark Correct
        Correct Answer
        Okay I'll try and figure out how to run the php from the command line....
        Like • Show 0 Likes0
        Actions
David Waugh Mar 17, 2016 2:22 PM
Mark Correct
Correct Answer
Success - here is a version that does not need a Webserver but you do need a server with PHP on it. This would need to be different from an Security Analytics Server as none of them have PHP installed on them and it is not available within our repository.

There is the wrapper script (./UserAgentInfoScript.sh) and the php script (script.php) it calls.

To run just run ./UserAgentInfoScript.sh (after changing the line to point to your broker)

I've attached a sample feed too and XML file to load the feed.
Attachments
- DeviceDetector.xml.zip505 bytes
- useragentfeed.csv.zip1.3 KB
- script.php.zip1.0 KB
- UserAgentInfoScript.sh1.2 KB
1 person found this helpful
Like • Show 1 Like1
Actions
- Alexey Fedorov @ David Waugh on Mar 18, 2016 3:48 AM
  Mark Correct
  Correct Answer
  I will try install PHP on SA Server. I not have any other VM for this.
  How I can filter out some user agent strings from sdk queue? For example I have many string like:
  
  *bajbaaaagwbaaaabaaqpty3q9x9agtjzrlamh1dpu2jq5hnaaawjaiaajaa=
  
  I would like filter out any user agent string if they not begin from [A-Za-Z] or [0-9].
  Your sdk queue receive unique user agent string from which time slot? How often I should start this shell script via cron?
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Alexey Fedorov on Mar 18, 2016 5:23 AM
    Mark Correct
    Correct Answer
    Hello
    
    The command
    http://192.168.123.249:50103/sdk?msg=values&fieldName=client&size=20000
    
    gets all the values that are in the current index slice, so depending on how often your slices roll over will depend on how often your need to query. Hourly should be more than enough.
    
    I would check how the values of
    
    "bajbaaaagwbaaaabaaqpty3q9x9agtjzrlamh1dpu2jq5hnaaawjaiaajaa=" are getting into your user agent meta key, and then improve parser to drop these. If you cant clean up the parser and they are valid then you could amend the LUA parser to not write them. These sorts of strings are a hazard to Security Analytics as you can quickly end up exhausting all the value available in the particular meta key that is being written to.
    
    I need to make a few changes to my script. Currently I just delete the old feed and then reprocess all the user agent strings again. What I will do is keep adding to the feed if no strings are received.
    Like • Show 0 Likes0
    Actions
- Alexey Fedorov @ David Waugh on Mar 21, 2016 4:11 AM
  Mark Correct
  Correct Answer
  Hello David,
  
  I found issue. PHP script doesn't detect apple user agent strings like "apple-iphone8c2/1304.15", meanwhile Device Detector Demo detect it!
  Like • Show 0 Likes0
  Actions
  - David Waugh @ Alexey Fedorov on Mar 22, 2016 10:13 AM
    Mark Correct
    Correct Answer
    Hi Alex, Thanks for letting me know. Looks like a little gremlin! Looking into it now.
    Like • Show 0 Likes0
    Actions
    - Alexey Fedorov @ David Waugh on Mar 22, 2016 10:22 AM
      Mark Correct
      Correct Answer
      Hello David,
      I just disable $UsefulInfo into script.php and all ok. Now I use output:
      
      {
      echo $argv[1];
      echo "^";
      if(!empty($clientInfo["name"])) {echo $clientInfo["name"];} else {echo "";}
      if(!empty($clientInfo["version"])) {echo " ",$clientInfo["version"];} else {echo "";}
      echo "^";
      if(!empty($osInfo["name"])) {echo $osInfo["name"];} else {echo "";}
      if(!empty($osInfo["version"])) {echo " ",$osInfo["version"];} else {echo "";}
      echo "^";
      echo $device;
      echo "^";
      echo $brand;
      if (!empty($model)) {echo " ",$model;} else {echo "";}
      echo "\r\n";
      }
      Like • Show 0 Likes0
      Actions
      - David Waugh @ Alexey Fedorov on Mar 22, 2016 10:26 AM
        Mark Correct
        Correct Answer
        I think your code is much better!
        Like • Show 0 Likes0
        Actions
Alexey Fedorov Mar 18, 2016 5:33 AM
Mark Correct
Correct Answer
Now I faced with issue to start PHP scrip.
I install PHP 5.3.3 to SA Server from CentOS Base repo, but can't run script. I receive this error:
PHP Warning: require_once(vendor/autoload.php): failed to open stream: No such file or directory in /opt/device-detector-master/script.php on line 2
PHP Fatal error: require_once(): Failed opening required 'vendor/autoload.php' (include_path='.:/usr/share/pear:/usr/share/php') in /opt/device-detector-master/script.php on line 2
Like • Show 0 Likes0
Actions
- David Waugh @ Alexey Fedorov on Mar 18, 2016 6:08 AM
  Mark Correct
  Correct Answer
  I would make sure that you follow the steps for the install of devicedetector and just check you can get that running first. Note I cant really support putting PHP on an SA Appliance....
  Like • Show 0 Likes0
  Actions
  - Alexey Fedorov @ David Waugh on Mar 18, 2016 6:20 AM
    Mark Correct
    Correct Answer
    Maybe we can use Device Detector Demo how external PHP server? I can't install PHP (need version >= 5.3.6 to DeviceDecoder) on the SA server. In CentOS-Base repo only 5.3.3.
    Like • Show 0 Likes0
    Actions
    - Alexey Fedorov @ Alexey Fedorov on Mar 18, 2016 7:23 AM
      Mark Correct
      Correct Answer
      I found solution how to install and update PHP and DeviceDetector on SA Server:
      
      1. Install PHP on the SA Server (yum install php php-dom --enablerepo=base).
      2. Install the EPEL repository configuration package (yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm)
      3. Install the Remi repository configuration package (yum install http://rpms.remirepo.net/enterprise/remi-release-6.rpm)
      4. Update php by command "yum --enablerepo=remi update php-\*"
      5. Copy device-detector-master folder to /opt folder on the SA Server
      6. Copy UserAgentInfoScript.sh to /opt/device-detector-maser folder
      7. chmod +x UserAgentInfoScript.sh
      8. Copy script.php to /opt/device-detector-maser folder
      9. chmod +x script.php
      10. Go to /opt/device-detector-maser folder on the SA Server and run "curl -sS https://getcomposer.org/installer | php"
      11. Install Composer "php composer.phar install".
      12. Disable or remove 3th party repo from /etc/yum.repos.d.
      
      How I can merge OS name and OS version in one filed? I done this:
      
      $UsefulInfo =(is_array($clientInfo) and is_array($osInfo)) and
      (array_key_exists("name",$clientInfo) or
      array_key_exists("version",$clientInfo) or
      array_key_exists("name",$osInfo) or
      array_key_exists("version",$osInfo) or
      $brand or $model);
      
      {
      echo $argv[1];
      echo "^";
      if(array_key_exists("name",$clientInfo)) {echo $clientInfo["name"];}
      echo " ";
      if(array_key_exists("version",$clientInfo)) {echo $clientInfo["version"];}
      echo "^";
      if(array_key_exists("name",$osInfo)) {echo $osInfo["name"];}
      echo " ";
      if(array_key_exists("version",$osInfo)) {echo $osInfo["version"];}
      echo "^";
      echo $brand;
      echo "^";
      echo $model;
      echo "\r\n";
      }
      
      But if version exist - all ok, if version dosn't exist - I have extra space (not good). And I have warnigs in console (after modification your script.php) like:
      
      PHP Warning: array_key_exists() expects parameter 2 to be array, null given in /opt/device-detector-master/script.php on line 49
      Like • Show 0 Likes0
      Actions
Alexey Fedorov Mar 18, 2016 1:24 PM
Mark Correct
Correct Answer
I solve all issue what I wrote into previous email.
Attachments
- script.php.zip1.0 KB
Like • Show 0 Likes0
Actions
David Waugh Mar 18, 2016 2:20 PM
Mark Correct
Correct Answer
Thats good news.
For "fun" I downloaded a large list of useragents from Massive list of user agents for User Agent Switcher by Chris Pederik http://forums.chrispederick.com/categories/user-age…

I extracted all the user agents with:
cat useragentswitcher.xml |grep "useragent description" |cut -f3 -d "=" | cut -f2 -d "\"" |grep -v -e '^$' >useragent-strings.txt

And then used the following script to send them to an internal web server so that the traffic was captured by security analytics.

spoof-useragent.sh
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
curl -A "$line" http://192.168.200.30/virustest
done < useragent-strings.txt

Here is a picture of the results. The feed file is also attached to this post.

I've also attached my latest cron job file and script.php.
The new cron job file only sends the string to be processed if the client has not been seen before.
Attachments
- UserAgentInfoScript2.sh1.7 KB
- script.php.zip1.0 KB
- useragentfeed.csv.zip13.7 KB
1 person found this helpful
Like • Show 2 Likes2
Actions
David Waugh Mar 22, 2016 11:55 AM
Mark Correct
Correct Answer
Here are the latest files.

The scripht.php now includes botinfo.
Thank to Alex for pointing out the errors in the original script.
useragentfeed.csv - example feed output
script.php - DeviceDetector script takes Useragent as a string as the first parameter and output a feed friendly line
UserAgentInfoScript2.sh - put in /etc/cron.hourly to generate an updated feed.
DeviceDetector.xml - feed definition file

The additional keys that I used in my index-concentrator-custom.xml file were:


<key description="ClientInfo Type" format="Text" level="IndexValues" name="clientinfo.type" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Name" format="Text" level="IndexValues" name="clientinfo.name" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Short Name" format="Text" level="IndexValues" name="clientinfo.sname" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Version" format="Text" level="IndexValues" name="clientinfo.ver" defaultAction="Open" valueMax="500000"/>
<key description="ClientInfo Platform" format="Text" level="IndexValues" name="clientinfo.plat" defaultAction="Open" valueMax="500000"/>
<key description="OS Type" format="Text" level="IndexValues" name="os.type" defaultAction="Open" valueMax="500000"/>
<key description="OS Name" format="Text" level="IndexValues" name="os.name" defaultAction="Open" valueMax="500000"/>
<key description="OS Short Name" format="Text" level="IndexValues" name="os.sname" defaultAction="Open" valueMax="500000"/>
<key description="OS Version" format="Text" level="IndexValues" name="os.ver" defaultAction="Open" valueMax="500000"/>
<key description="OS Platform" format="Text" level="IndexValues" name="os.platform" defaultAction="Open" valueMax="500000"/>
<key description="Brand" format="Text" level="IndexValues" name="brand" defaultAction="Open" valueMax="500000"/>
<key description="Model" format="Text" level="IndexValues" name="model" defaultAction="Open" valueMax="500000"/>
<key description="BotInfo" format="Text" level="IndexValues" name="bot.info" defaultAction="Open" valueMax="500000"/>
Attachments
- DeviceDetector.xml.zip507 bytes
- UserAgentInfoScript2.sh1.7 KB
- script.php.zip1.0 KB
- useragentfeed.csv.zip16.0 KB
1 person found this helpful
Like • Show 1 Like1
Actions

Incoming Links

Creating a Feed to Automatically Perform a Reverse Lookup of IP Addresses to Hostnames

User Agent to Device/OS/Application

Outcomes

Attachments

Attachments

Attachments

Attachments

Attachments

Attachments

Attachments

Related Content

Recommended Content

Incoming Links