David Waugh

Writing a Parser to Detect SPF Fields in Email Messages

Blog Post created by David Waugh Employee on Apr 27, 2016

I had a request from a customer to parse out some messages from a mail conversation.


Basically the email contains the following headers:


Received-SPF: pass (infra1.csuk.eu.rsa.net: is whitelisted) receiver=infra1.csuk.eu.rsa.net; client-ip=; helo=ECAT.waugh.local; envelope-from=david.waugh@waugh.local; x-software=spfmilter 2.001 http://www.acme.com/software/spfmilter/ with libspf2-1.2.10;
Received: from ECAT.waugh.local ([])
  by infra1.csuk.eu.rsa.net (8.13.8/8.13.8) with ESMTP id u3RCCOxW024832
  for <david.waugh2@rsa.com>; Wed, 27 Apr 2016 12:12:24 GMT
x-metascan-quarantine-id: e3c4973b-d836-4f37-a47d-62271c21a5cc
Received: from UKXXWAUGHDL1C ([]) by ECAT.waugh.local with ESMTP ; Wed, 27 Apr 2016 13:12:23 +0100
From: "10.5 Test" <david.waugh@infra1.esc.ai.pri>
To: <david.waugh2@rsa.com>
Subject: RE: This is a test through my mail system
Date: Wed, 27 Apr 2016 13:12:23 +0100
Message-ID: <0bd101d1a07e$0e479230$2ad6b690$@waugh.local>
MIME-Version: 1.0
Content-Type: multipart/alternative;
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AdGget+xlnYyEWhiQrysrAxYJ7qXxgAApK1AAAAdpuAAAAfMIA==
Content-Language: en-gb
X-Virus-Scanned: clamav-milter devel-clamav-0.98-dmgxar-126-gfde6749 at infra1
X-Virus-Status: Clean


The header of interest here is the one called Received-SPF:


I created a parser based on Detecting Sinkholed Domains With The X-Factor Parser


On my Packet Decoder I created a parser called SPF.parser in /etc/netwitness/ng/parsers

containing the following:


<?xml version="1.0" encoding="utf-8"?>

<parsers xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="parsers.xsd">

      <parser name="SPF Factor" desc="This extracts the SPF string from a Header">


                                               <token name="tXfactor" value="Received-SPF:" options="linestart" />

                                                <number name="vPosition" scope="stream" />

                                                <string name="vXfactor" scope="stream" />

                                                <meta name="meta" key="xfactor" format="Text" />




                                <match name="tXfactor">

                                                <find name="vPosition" value="&#x0d;&#x0a;" length="512">

                                                 <read name="vXfactor" length="$vPosition">

                                                  <register name="meta" value="$vXfactor"/>







Whenever the Received-SPF header was seen, then the rest of the header was put into the xfactor metakey.




If your SPF fields are from a different mail provider, then you could adjust the parser accordingly.

For example if your messages had the following header:

Authentication-Results: mx.messagelabs.com; spf=pass


Then the line in the parser could be changed from:

<token name="tXfactor" value="Received-SPF:" options="linestart" />


<token name="tXfactor" value="Authentication-Results: mx.messagelabs.com; spf=" options="linestart" />


If you wanted to put the result into a different metakey (for example result) then change

<meta name="meta" key="xfactor" format="Text" />


<meta name="meta" key="result" format="Text" />