AnsweredAssumed Answered

tcp.srcport Index/Meta(s) Overflown

Question asked by Richard van den Berg on Dec 13, 2019
Latest reply on Dec 13, 2019 by Richard van den Berg

Like • Show 0 Likes0
Comment • 8

We have several appliances that report tcp.srcport is overflown. We are familiar with this concept for meta keys that have limited index sizes like payload or filename, but tcp.srcport in defined correctly in index-concentrator.xml as:

<key description="TCP Source Port" name="tcp.srcport" format="UInt16" level="IndexValues" valueMax="65536">

Inspecting the index indeed shows one of the index slices has 65536 values, so that is all of the TCP ports 0-65535. What we fail to understand is how a new tcp.srcport value could be different and thus overflow this index. By definition a TCP port is a 16 bit unsigned field, so 0-65535 are the only possible values that could ever exist, right?

{
  "msgPack": {
    "keys": {
      "tcp.srcport": [
        {
          "keyIndex": {
            "accesses": 0,
            "key": "tcp.srcport",
            "packets": 28867258,
            "pages": 28,
            "pathname": "/var/netwitness/concentrator/index/managed-values-1/tcp.srcport.nwindex",
            "session1": 59920088,
            "session2": 61918168,
            "sessions": 1772474,
            "size": 27724295402,
            "summaries": 28,
            "summary1": 6866213,
            "summary2": 8140168,
            "values": 64835
          }
        },
        {
          "keyIndex": {
            "accesses": 0,
            "key": "tcp.srcport",
            "packets": 856792452,
            "pages": 728,
            "pathname": "/var/netwitness/concentrator/index/managed-values-0/tcp.srcport.nwindex",
            "session1": 1,
            "session2": 59920086,
            "sessions": 56358097,
            "size": 817822448980,
            "summaries": 728,
            "summary1": 60,
            "summary2": 6864324,
            "values": 65536
          }
        }
      ]
    },
    "meta1": 1,
    "meta2": 4550001864,
    "session1": 1,
    "session2": 61918168,
    "size": 1047923718704,
    "totalKeys": 1,
    "totalMemory": 2084005232,
    "totalValues": 130371
  }
}

John Snider

Dec 13, 2019 8:57 AM

Correct Answer

Actually, the issue relates to HOW the alert was coded into our system. The overflown meta trigger happens when the number of unique values "=" the valueMax setting, instead of triggering on ">" valueMax setting (or valueMax + 1). This is causing several keys to trigger false positive alerts. Since the valueMax setting is really meant to keep indexes from growing too large, for values that we have no control over (think alias.host, ip.src, ip.dst, etc.) if they get too many unique values, finite keys like the port keys really should not have a valueMax set at all, since they have an physical upper limit. Though on the other hand, having EVERY port value seen in a time period would indicate a scanner or other non-normal traffic indication.

you can either set the valueMax to 65537 for the port keys, or remove the valueMax entry entirely and the alerts will no longer fire, since you know the maximum number of values will not exceed 65537 (unless some other custom feed/parser/application rule accidentally writes to that meta key.

I have opened an internal engineering ticket to address this issue.

See the reply in context

No one else had this question

Outcomes

Kevin Arunski

Dec 13, 2019 8:34 AM

You probably have some parser (or other content) generating tcp.srcport as something other than UInt16, for example as Text values. It would likely be something non-RSA.

Actions

Richard van den Berg @ Kevin Arunski on Dec 13, 2019 9:53 AM

The few custom parsers we have do not set tcp.srcport, but that would definitely be something to check if they did.

Actions

Kevin Arunski

@ Richard van den Berg on Dec 13, 2019 10:08 AM

I see. I was looking at the total values summary (130371) rather than the per slice number (65536).

Actions

John Snider

Dec 13, 2019 8:57 AM

I have opened an internal engineering ticket to address this issue.

Actions

Richard van den Berg @ John Snider on Dec 13, 2019 9:56 AM

Thanks for confirming this is probably a false positive. I tried to explain this to support but didn't get past 1st line. I've increased the valueMax to 65537 for now to see if that helps.

Actions

Kevin Arunski

@ John Snider on Dec 13, 2019 10:16 AM

The warning isn't written to trigger on 'unique values > valueMax' because that condition is never allowed to occur. We can't put more values than valueMax in the index. So if it was written that way, it would never fire.

The valueMax on the port values is important to prevent against the case of parsers somehow writing in the wrong format.

Therefore a valueMax of 65537 is appropriate. However, I do wonder how they got the full 65536 Uint16 values (0-65535) into tcp.srcport. I don't think you should see a TCP source port of 0 go across the wire.

Actions

Richard van den Berg @ Kevin Arunski on Dec 13, 2019 10:59 AM

That was my first thought as well. I have verified that a tcp.srcport of 0 is indeed in the index. This occurs when the data is fragmented (ip.frag_offset > 0) and the first fragment is not seen. I suspect a flaky span port to be to blame for that. This causes the first 16 bits after the IP destination address to be interpreted by NW as the tcp.srcport (which is probably wrong looking at RFC791).

Actions

Aaron Martin

Dec 13, 2019 9:05 AM

Both Kevin and John are correct. For other keys, particular text ones which aren't bound by a real narrow data type size, it could be a parser or other piece of meta generation that could result in an overflow accidentally occurring. Given we all understand what the max possible values for a tcp port number, we know that is unlikely the case, yet we must also be aware that the alarm triggers when the bucket is completely full; not actually when the bucket actually overflows.

Actions

8 Replies

Kevin Arunski Dec 13, 2019 8:34 AM
Mark Correct
Correct Answer
You probably have some parser (or other content) generating tcp.srcport as something other than UInt16, for example as Text values. It would likely be something non-RSA.
Like • Show 0 Likes0
Actions
- Richard van den Berg @ Kevin Arunski on Dec 13, 2019 9:53 AM
  Mark Correct
  Correct Answer
  The few custom parsers we have do not set tcp.srcport, but that would definitely be something to check if they did.
  Like • Show 0 Likes0
  Actions
  - Kevin Arunski @ Richard van den Berg on Dec 13, 2019 10:08 AM
    Mark Correct
    Correct Answer
    I see. I was looking at the total values summary (130371) rather than the per slice number (65536).
    Like • Show 0 Likes0
    Actions
John Snider Dec 13, 2019 8:57 AM
Unmark Correct
Correct Answer
Actually, the issue relates to HOW the alert was coded into our system. The overflown meta trigger happens when the number of unique values "=" the valueMax setting, instead of triggering on ">" valueMax setting (or valueMax + 1). This is causing several keys to trigger false positive alerts. Since the valueMax setting is really meant to keep indexes from growing too large, for values that we have no control over (think alias.host, ip.src, ip.dst, etc.) if they get too many unique values, finite keys like the port keys really should not have a valueMax set at all, since they have an physical upper limit. Though on the other hand, having EVERY port value seen in a time period would indicate a scanner or other non-normal traffic indication.

you can either set the valueMax to 65537 for the port keys, or remove the valueMax entry entirely and the alerts will no longer fire, since you know the maximum number of values will not exceed 65537 (unless some other custom feed/parser/application rule accidentally writes to that meta key.

I have opened an internal engineering ticket to address this issue.
Like • Show 0 Likes0
Actions
- Richard van den Berg @ John Snider on Dec 13, 2019 9:56 AM
  Mark Correct
  Correct Answer
  Thanks for confirming this is probably a false positive. I tried to explain this to support but didn't get past 1st line. I've increased the valueMax to 65537 for now to see if that helps.
  Like • Show 0 Likes0
  Actions
- Kevin Arunski @ John Snider on Dec 13, 2019 10:16 AM
  Mark Correct
  Correct Answer
  The warning isn't written to trigger on 'unique values > valueMax' because that condition is never allowed to occur. We can't put more values than valueMax in the index. So if it was written that way, it would never fire.
  
  The valueMax on the port values is important to prevent against the case of parsers somehow writing in the wrong format.
  
  Therefore a valueMax of 65537 is appropriate. However, I do wonder how they got the full 65536 Uint16 values (0-65535) into tcp.srcport. I don't think you should see a TCP source port of 0 go across the wire.
  Like • Show 0 Likes0
  Actions
  - Richard van den Berg @ Kevin Arunski on Dec 13, 2019 10:59 AM
    Mark Correct
    Correct Answer
    That was my first thought as well. I have verified that a tcp.srcport of 0 is indeed in the index. This occurs when the data is fragmented (ip.frag_offset > 0) and the first fragment is not seen. I suspect a flaky span port to be to blame for that. This causes the first 16 bits after the IP destination address to be interpreted by NW as the tcp.srcport (which is probably wrong looking at RFC791).
    Like • Show 0 Likes0
    Actions
Aaron Martin Dec 13, 2019 9:05 AM
Mark Correct
Correct Answer
Both Kevin and John are correct. For other keys, particular text ones which aren't bound by a real narrow data type size, it could be a parser or other piece of meta generation that could result in an overflow accidentally occurring. Given we all understand what the max possible values for a tcp port number, we know that is unlikely the case, yet we must also be aware that the alarm triggers when the bucket is completely full; not actually when the bucket actually overflows.
Like • Show 0 Likes0
Actions