AnsweredAssumed Answered

Email log parsing - subject encoded (RFC 2047)

Question asked by Uq1lws3RT39tp8reFD8y8NuuHaHt7KaBBIJDVDjgnPY= on Jun 7, 2016
Latest reply on Jun 9, 2016 by Uq1lws3RT39tp8reFD8y8NuuHaHt7KaBBIJDVDjgnPY=

Like • Show 0 Likes0
Comment • 7

Hello,

I have a cisco ironport esa wich provide the following kind of log:

Jun 06 14:00:07 mail_logs4: Info: MID 62990685 Subject '=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?='

Does anyone have developed a (lua) parser to decode the encoded subject to human readable format?

(online decoder Online MIME Headers Decoder (RFC 2047))

David Waugh

Jun 7, 2016 8:03 AM

Correct Answer

Okay I have a working one!

Here is the code attached below.

See the reply in context

No one else had this question

Outcomes

7 Replies

David Waugh

Jun 7, 2016 6:20 AM

Hi that is an interesting issue. Can you give me a full line of a log so that I can test it in my lab please. No promises but lets see...

Actions

Uq1lws3RT39tp8reFD8y8NuuHaHt7KaBBIJDVDjgnPY= @ David Waugh on Jun 7, 2016 7:03 AM

Thanks David Waugh!

It's already the full line (datetime / mailid / subject):

Jun 06 14:00:07 mail_logs4: Info: MID 62990685 Subject '=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?='

This issue/feature has recently been reported to Cisco Bug: CSCun16129 - Translate subjects from base64 to human readable format for export

Moreover:

Sometimes, there is no '\r\n\t' but just '\r\n '
Subject is not always encoded
Encoding varies (utf-8, ISO, etc)

Actions

David Waugh

@ Uq1lws3RT39tp8reFD8y8NuuHaHt7KaBBIJDVDjgnPY= on Jun 7, 2016 7:28 AM

Great. making some progress here.

I use:

Lua: demo

And have the following code as a basis:

Paste in the top box:

function mimedecode(mystring)
t={}
for k,v in string.gmatch(mystring,"=%?ISO%-8859%-%d%?B%?(.-)%?=") do
 t[k] =v
 print("Variable")
 print(dec(k))
end
end


-- Lua 5.1+ base64 v3.0 (c) 2009 by Alex Kloss <alexthkloss@web.de>
-- licensed under the terms of the LGPL2


-- character table string
local b='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'


-- encoding
function enc(data)
    return ((data:gsub('.', function(x) 
        local r,b='',x:byte()
        for i=8,1,-1 do r=r..(b%2^i-b%2^(i-1)>0 and '1' or '0') end
        return r;
    end)..'0000'):gsub('%d%d%d?%d?%d?%d?', function(x)
        if (#x < 6) then return '' end
        local c=0
        for i=1,6 do c=c+(x:sub(i,i)=='1' and 2^(6-i) or 0) end
        return b:sub(c+1,c+1)
    end)..({ '', '==', '=' })[#data%3+1])
end


-- decoding
function dec(data)
    data = string.gsub(data, '[^'..b..'=]', '')
    return (data:gsub('.', function(x)
        if (x == '=') then return '' end
        local r,f='',(b:find(x)-1)
        for i=6,1,-1 do r=r..(f%2^i-f%2^(i-1)>0 and '1' or '0') end
        return r;
    end):gsub('%d%d%d?%d?%d?%d?%d?%d?', function(x)
        if (#x ~= 8) then return '' end
        local c=0
        for i=1,8 do c=c+(x:sub(i,i)=='1' and 2^(8-i) or 0) end
        return string.char(c)
    end))
end

mimedecode("=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?= =?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=")

Now click on run and you get the results:

Variable

If you can read this yo

Variable

u understand the example.

Its unfinished but getting somewhere....

Actions

David Waugh

@ David Waugh on Jun 7, 2016 8:03 AM

Okay I have a working one!

Here is the code attached below.

Attachments

MimeISO.lua.zip687 bytes

1 person found this helpful

Actions

Uq1lws3RT39tp8reFD8y8NuuHaHt7KaBBIJDVDjgnPY= @ David Waugh on Jun 7, 2016 10:45 AM

Thanks David Waugh

I just check the script parserbook/example parsers/mal.lua and I found the function:

function mailParser:rfc2047(encodedString)
    if not encodedString then
        return
    end
    local decodedString, charset = encodedString, nil
    -- look for encoding, e.g.:
    --     =?windows-1256?B?VklQIFRv...=?
    if string.byte(encodedString, 1) == 61 and string.byte(encodedString, 2) == 63 then
        local charsetEnd = string.find(encodedString, "?", 3)
        if charsetEnd then
            if string.byte(encodedString, charsetEnd + 2) == 63 then
                local encodingChar = string.byte(encodedString, charsetEnd + 1)
                if encodingChar == 66 or encodingChar == 81 or encodingChar == 98 or encodingChar == 113 then
                    local stringBegin = charsetEnd + 3
                    local stringEnd = string.find(encodedString, "?=", stringBegin + 1)
                    if stringEnd then
                        stringEnd = stringEnd - 1
                        decodedString = string.sub(encodedString, stringBegin, stringEnd)
                        decodedString = string.gsub(decodedString, "_", "\032")
                        if encodingChar == 66 or encodingChar == 98 then
                            decodedString = nw.base64Decode(decodedString)
                        else
                            decodedString = nwll.decodeQuotedPrintable(decodedString)
                        end
                        charset = string.sub(encodedString, 3, charsetEnd - 1)
                    end
                end
            end
        end
    end
    return decodedString, charset
end

I have to check/adapt the script

Actions

William Motley

Jun 7, 2016 12:33 PM

Actually, MAIL_lua has been substantially rewritten from what's in the book and I haven't found the time to update the book yet. Here's how it looks now:

local function rfc2047(encodedString)
    if not encodedString then
        return
    end
    local decodedString, charset, encoding
    -- look for encoding, e.g.,  =?windows-1256?B?VklQIFRv...?=
    charset, encoding, decodedString = string.match(encodedString, "^%s-=%?([^?]+)%?([BbQq])%?([^?]+)%?=")
    if charset and encoding and decodedString then
        charset = string.match(charset, "^([^*]+)%*") or charset
        decodedString = string.gsub(decodedString, "_", " ")
        if encoding == "B" or encoding == "b" then
            decodedString = nw.base64Decode(decodedString) or decodedString
        elseif encoding == "Q" or encoding == "q" then
            decodedString = nwll.decodeQuotedPrintable(decodedString) or decodedString
        end
    end
    decodedString = decodedString or encodedString
    return decodedString, charset
end

1 person found this helpful

Actions

Uq1lws3RT39tp8reFD8y8NuuHaHt7KaBBIJDVDjgnPY= @ William Motley on Jun 9, 2016 3:23 AM
Mark Correct
Correct Answer
It works! Thanks David Waugh and William Motley
Like • Show 0 Likes0
Actions

Email log parsing - subject encoded (RFC 2047)

Outcomes

Attachments

Related Content

Recommended Content