Eric Partington

Filtering F5 UDP Syslog Health Checks

Blog Post created by Eric Partington Employee on Nov 25, 2016

If you happen to have F5 LTM providing balancing or HA in front of your VLC for syslog messages then you may have enabled a monitor on the LTM to check for the VLC syslog service being reachable. 

To do that you might have followed this guide to enable a UDP monitor that also requires an ICMP check to verify if the UDP 514 port is reachable.

 

 

https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip9_3config/BIG_IP_LTM_9_3_Config_Guide-13-1.html

Notice the default string in this example is "default send string"

UDP Monitor Default LTM

These health checks are not valid syslog messages and have no priority flag set (and are 0 payload length).  At volume these messages cause problems with RabbitMq and should be dropped at the VLC to prevent as much of the noise getting to the decoders as possible,

To filter the messages you can use the Filter option available on the VLC under the syslog collection and implement it for both UDP and TCP syslog.

On the log decoders you can grep /var/log/messages to find these 0 length messages and the VLC that they came from to filter.

 

 

If you run tcpdump on the VLC looking for UDP or TCP messages from the 0 length sources you might see this (if the Monitor is configured with defaults).  Notice the default.send.string value which correlates to the default F5 LTM config.

 

 

 

Now we need to define a filter for syslog to filter these messages from the syslog pipeline

VLC > Config > Event Sources > Syslog > Filter

Define a new Filter and then define a new rule

we will use the raw key as the 0 length messages don't have proper formatted message to extract the source IP from (lc.srcid)

Now add or update the syslog collection with the filter

If you want to view stats on the drops performed by the filter you can switch to the explore view of the VLC

VLC > Explore >

logcollection > syslog > stats > eventsources

 

the total_filtered_events count will increase when filtered items are found (this counter is reset when the service is restarted)

If you want to see the debug logging values of what the VLC parses from the messages you can enable debug and event_filter_debug from the explore menu.  For my testing (temporary) i enabled debug and set the event_filter_debug to 15

that drops debug messages into the VLC > Logs section under [DEBUG] and will show you this information about the match or no match values

 

 

 

20161122T162858^P^@^@^@SyslogCollection~Q^@^@^@[syslog-udp.udp514] [processing] [Receiver WorkUnit] [processing] Unidentified content from 10.10.10.10 received on receiver: 'default send string'~A^A^@^@^A^@^@^@^O^@^@^@20161122T162858^Z^@^@^@SyslogCollection(TraceLog)H^A^@^@[syslog-udp.udp514] [processing] [Receiver WorkUnit] [processing] Content received on receiver 10.10.10.10 does not conform to Syslog standards. Valid Syslog format is "<PRI> MESSAGE".                                   Probably raw syslog message is not starting with "<"PRIVAL">" field: 'default send string'. Please rectify the issue at syslog event source.~T^B^@^@^A^@^@^@^O^@^@^@20161122T162858^Z^@^@^@SyslogCollection(TraceLog)[^B^@^@[syslog-udp.udp514] [processing] [Receiver WorkUnit] [processing]  [EventFilter-Accept] syslog.NOFILTER (not filtering-test hits) 1479831814567

    Rule: "no match ident only"

    #1  [raw]  [Contains]  [(ignoreCase)default send string]

        Match=Yes

        Matched=default send string

        Result=Drop

Event:

    collection_meta:

"lc.lpid" : "syslog.syslog-udp"

"lc.cid" : "vlcid"

"lc.msgtype" : "0"

"lc.ctype" : "syslog"

"lc.wuid" : "17562157925649023279"

"lc.esname" : "udp514"

"lc.estype" : "syslog-udp"

"lc.wusn" : "93719"

    content_meta:

    raw_message: default send string

 

Using this debug message you can determine what values are extracted by the VLC to make activities/filters more accurate.  In this case the lc.srcid value has no IP address so we are unable to drop based on that value, requiring the RAW value to be used.

Outcomes