Maintaining Table Map and Service Index files

Discussion created by vrPbZkjA7FsIRBKKxruMHsl0sWGv9zxSzgTBwX74lr4= on Mar 31, 2016
Latest reply on Sep 16, 2016 by John Kisner

Like • Show 2 Likes2
Comment • 6

Hi,

I was wondering if anyone could explain me what's the order for service restarting (Log Decoder, Log Collector, Concentrator, Broker¿?)

And what other files (index-concentrator‐custom.xml or index-logdecoder‐custom.xml) should be modify for this case scenarios:

1) Adding new metadata in table-map-custom.xml or index-logdecoder‐custom.xml or index-concentrator‐custom.xml files

2) If I create a Custom Feed, which file shoud I need to update in order to show metadata within Investigation/Navigate screen?

3) If I deploy an EnVision UDS, which file shoud I need to update in order to show metadata within Investigation/Navigate screen?

4) Finally. What's the difference between index-logdecoder‐custom.xml and index-decoder‐custom.xml ?

Frankly I don't get when to update an xml file (...and which one) and when to restart the services and in what order.

Actually, I'm restarting all services at once and if I add some metadata in, for example, index-logdecoder‐custom.xml I also add it to index-concentrator‐custom.xml and viceversa.

Thank you all in advance for your responses.

Best regards.

Outcomes

6 Replies

Sravan Koneti Sep 13, 2016 2:18 AM
Hi Leonardo,

table-map-custom.xml- File used for defining meta key for parsing the meta values. This file exists in Logdecoder.
index-logdecoder‐custom.xml- File used for indexing the meta values for showing up in the investigation page against logdecoder. But logdecoder focuses only on parsing. But not designed for indexing. So, defining values in this file is not recommended.
index-concentrator‐custom.xml- File used for indexing the meta values for showing up in the investigation page against Concentrator.

index-logdecoder‐custom.xml (located in Logdecoder) and index-decoder‐custom.xml (located in Packetdecoder)

How to edit all these files explained in below KB.
000017493 - Enable Parsed Meta Keys That Do Not Currently Show In Investigation

Cheers,
Sravan
2 people found this helpful
Like • Show 1 Like1
Actions
- vrPbZkjA7FsIRBKKxruMHsl0sWGv9zxSzgTBwX74lr4= @ Sravan Koneti on Sep 13, 2016 8:40 AM
  Dear Sravan,
  Till this moment no one could explain me as clear as your reply.
  Thank you so much!
  
  Best regards,
  Leo.
  Like • Show 0 Likes0
  Actions
  - Sravan Koneti @ vrPbZkjA7FsIRBKKxruMHsl0sWGv9zxSzgTBwX74lr4= on Sep 13, 2016 8:48 AM
    Hi Leo,
    
    I am glad to hear that my comments are useful. Hope you can "Mark as helpful" By clicking reply->Actions navigation tab.
    
    Cheers,
    Sravan
    Like • Show 0 Likes0
    Actions
- Gabriel Python @ Sravan Koneti on Sep 13, 2016 9:06 AM
  And what about the ARCHIVER ?
  Kind regards
  Like • Show 0 Likes0
  Actions
  - Sravan Koneti @ Gabriel Python on Sep 13, 2016 9:20 AM
    Hi Gabriel,
    
    Archiver will have index-archiver-custom.xml. But, the meta that is going to be defined in index-archiver-custom.xml should exists in Meta Include list. Otherwise, that meta can't be indexed.
    
    Please use below document for Meta Include details.
    Step 2. Add Log Decoder as a Data Source to Archiver - RSA Security Analytics Documentation
    
    Cheers,
    Sravan
    Like • Show 0 Likes0
    Actions
John Kisner Sep 16, 2016 7:35 PM
Leonardo,

Let me expand on what Sravan has put forward for you.

table-map-custom.xml- This file is used for log parsers only and is designed to tell the log decoder which log meta to save and which to dump once the parsing is completed. There are two possible options for each line in this file. One is Transient and the other is None. If a meta key is set to Transient then that meta will not be saved to disk at the end of the parsing session but can be used by other parsers, application rules or feeds during the parsing session. This allows you to use a piece of meta to help add context to the session without taking up valuable storage space because you don't need it outside the current parsing session. The other option of None which means that the meta will be written to disk at the end of the parsing session and later possibly indexed on the Concentrator.
You may notice that there is a table-map.xml and a table-map-custom.xml. The custom xml is where user changes should be made. If you need to change one of the keys from Transient to None you would copy the line from the table-map.xml into the table-map-custom.xml and then perform the change. This way if an upgrade is performed any custom changes will not be written over.
Note: This file has nothing to do with creating indexes.

index-logdecoder‐custom.xml- This file should hardly be used under most circumstances. The only time I have seen the requirement for placing custom meta language keys into this file is when you need these custom meta in app rules or feeds.Otherwise this file should remain empty. If you do put anything within this file under no circumstance should you use a level other than IndexNone. If you attempt to set the level to IndexValue or IndexKey it will cause your decoder to eventually fill up the index partition and shutdown the decoder. Only time is indexed on the decoder.
Note: This file is used for indexing on a decoder, but only time should be indexed here.

index-concentrator‐custom.xml- This file is used similarly to the custom file on the decoder except here you can use levels of IndexValue and IndexKey. These levels are what determine the kind of indexes that will be generated for the meta that is housed within that particular language key. Once again as this is the custom file all user created changes should be placed into this file to avoid any changes being overwritten during upgrades.The indexes generated on the concentrator are what are required for your investigations to work correctly. If you have no index for a piece of meta then you will not be able to query that meta during an investigation. So the custom file on the concentrator is the most important one when it comes to investigations.
Note: This file is used for creating indexes on meta.
Important: All index-concentrator-custom.xml files must be identical on ALL concentrators that are attached to the same Broker.

The difference between the custom and non custom file (index-concentrator.xml vs. index-concentrator-custom.xml)
All the services have two files one is the default set of indexes the other is used for any customization of those indexes or adding new indexes. When making changes to existing default values always copy them to the custom file and make the changes there. When the services start up they look for both files. The original file is always read first then the custom file is read. Any thing in the custom file that overlaps something in the default file will be over ridden. Say you had a language key alias.host and it had an index level of IndexKey but you wanted to change that to indexValue. You copy the original line from the default index xml and place it into the custom one. Then you change the level to indexValue and save the file. When the service restarts it reads the first alias.host and sets the level to indexKey, then it reads the custom file and sees alias.host should be indexValue so now alias.host will be indexValue.

On restarting services
The order of service restart is usually not an issue. You can generally restart any core service in any order as they only talk to each other when pulling data. However you must always restart a service when you make an index xml file change as these files are only read at service start time. So for instance say you created a custom language key for a custom feed that you made. You would add the new language key to the index-concentrator-custom.xml, you may have to add it to the index-broker-custom.xml or the index-logdecoder-custom.xml under depending on the circumstances. Once that is done you can restart these services in any order. The server that contains the SA UI is the only one that really has services that should be started in a specific order as the jetty process won't fully start until the other ones are running. As for the core devices (decoders, concentrators, brokers, archivers) it doesn't really matter.

Creating custom feeds
When creating custom feeds, as long as you are using existing meta language keys you do not have to adjust anything. However if you are creating a custom language key you should only need to adjust the index-concentrator-custom.xml. Just remember if you have an archiver and you want the custom language key to be indexed over there you need to add it to the Include Meta area within the Archiver Config.

Deploying an Envision UDS
If I understand the Envision UDS correctly the only file you would need to modify is the table-map-custom.xml, if the meta language key being utilized was not already set to level of None. Remember from above if the level is None on the table-map.xml file that means the meta will be saved for the session. At that point all you would need to do is look at what meta language key the table-map was using, then go to the index-concentrator.xml and see if the language key was in the file and had an index level. If the meta language key wasn't there you would then need to add it to the index-concentrator-custom.xml, add a level an then you would be able to perform an Investigation query against it.

I hope this helps to answer some of your questions about this topic.
3 people found this helpful
Like • Show 3 Likes3
Actions

Maintaining Table Map and Service Index files

Outcomes

Related Content

Recommended Content