Top Level Domain (TLD) Lua Parser for Logs

[EricPartington]

The TLD parser has been updated to now deploy on Log Decoders.  

The parser looks for the following keys from log devices to parse out the same information as packets:

• Alias.host
• Host.src
• Host.dst
• Domain.dst
• Domain.src
• FQDN

Which writes out information into:

* alert.id - mapped to risk meta
* analysis.service - hostname characteristics
* cctld - (nonstandard) (optional) country-code top level domain, e.g., www.amazon.co.uk -> co.uk
* sld - (nonstandard) (optional) second level domain, e.g. www.amazon.co.uk -> amazon
* tld - top level domain, e.g. www.amazon.com -> com

When searching for Lua and Log in the RSA Live deployment screen you will see the following:

And linked dependancies:

So this is a really simple method of getting nwll.lua deployed to a log decoder if your custom parser requires that library (PaloAlto URL.raw parser for instance).

[RichardvandenBe]

Thanks a lot for this very useful info. I also found the documentation at Packet Parsers 

We observed the following:

1. tld of mta1._domainkey.example.de -> example.de
   
2. sld of mta1._domainkey.example.de -> _domainkey

This is pretty strange as it pollutes the tld and sld tables.

[WilliamMotley1]

That's intentional.  The behavior in the presence of a ccTLD is,

Four or more total tags

• last tag -> cctld
• 2nd to last and last tag combined -> tld
• 3rd to last tag -> sld

Three or fewer total tags

• last tag -> cctld
• last tag -> tld (also)
• 2nd to last tag -> sld

Examples:

    www.amazon.co.uk

        cctld:  uk

        tld:     co.uk

        sld:    amazon

    www.amazon.de

        cctld:  de

        tld:      de

        sld:     amazon

The parser does not account for policy differences between registries concerning 3rd-level domains, and instead uses the same "rules" consistently for all hosts.  The only record of those policy differences in the "Public Suffix List".  Incorporating those policy differences would necessitate the analyst also being familiar with the Public Suffix List and all the policy differences between registries.  Whereas the behavior of the parser is consistent for all hostnames.