We have several appliances that report tcp.srcport is overflown. We are familiar with this concept for meta keys that have limited index sizes like payload or filename, but tcp.srcport in defined correctly in index-concentrator.xml as:
<key description="TCP Source Port" name="tcp.srcport" format="UInt16" level="IndexValues" valueMax="65536">
Inspecting the index indeed shows one of the index slices has 65536 values, so that is all of the TCP ports 0-65535. What we fail to understand is how a new tcp.srcport value could be different and thus overflow this index. By definition a TCP port is a 16 bit unsigned field, so 0-65535 are the only possible values that could ever exist, right?
{
"msgPack": {
"keys": {
"tcp.srcport": [
{
"keyIndex": {
"accesses": 0,
"key": "tcp.srcport",
"packets": 28867258,
"pages": 28,
"pathname": "/var/netwitness/concentrator/index/managed-values-1/tcp.srcport.nwindex",
"session1": 59920088,
"session2": 61918168,
"sessions": 1772474,
"size": 27724295402,
"summaries": 28,
"summary1": 6866213,
"summary2": 8140168,
"values": 64835
}
},
{
"keyIndex": {
"accesses": 0,
"key": "tcp.srcport",
"packets": 856792452,
"pages": 728,
"pathname": "/var/netwitness/concentrator/index/managed-values-0/tcp.srcport.nwindex",
"session1": 1,
"session2": 59920086,
"sessions": 56358097,
"size": 817822448980,
"summaries": 728,
"summary1": 60,
"summary2": 6864324,
"values": 65536
}
}
]
},
"meta1": 1,
"meta2": 4550001864,
"session1": 1,
"session2": 61918168,
"size": 1047923718704,
"totalKeys": 1,
"totalMemory": 2084005232,
"totalValues": 130371
}
}
Actually, the issue relates to HOW the alert was coded into our system. The overflown meta trigger happens when the number of unique values "=" the valueMax setting, instead of triggering on ">" valueMax setting (or valueMax + 1). This is causing several keys to trigger false positive alerts. Since the valueMax setting is really meant to keep indexes from growing too large, for values that we have no control over (think alias.host, ip.src, ip.dst, etc.) if they get too many unique values, finite keys like the port keys really should not have a valueMax set at all, since they have an physical upper limit. Though on the other hand, having EVERY port value seen in a time period would indicate a scanner or other non-normal traffic indication.
you can either set the valueMax to 65537 for the port keys, or remove the valueMax entry entirely and the alerts will no longer fire, since you know the maximum number of values will not exceed 65537 (unless some other custom feed/parser/application rule accidentally writes to that meta key.
I have opened an internal engineering ticket to address this issue.