After upgrading the NetWitness environment, concentrator services fail to start the aggregation. Failures like the below example are logged in /var/log/messages.
Apr 27 01:00:00 concentrator NwConcentrator: [Aggregation] [failure] Failed to initialize device '10.10.10.11:50004' because Invalid language key name 'custom.sample-key', must start with an alpha, then follow with alphanumeric OR . OR _. Device aggregation is being stopped.
Checking /etc/netwitness/ng/index-concentrator-custom.xml confirms that the problematic meta key is incorrectly formatted. e.g. custom.sample-key
Due to the meta key format verification added to 11.5, if a meta key is defined in an unsupported format, the concentrator/archiver/broker and decoder services is expected to fail to start the aggregation or capture.
In order to resolve the issue, please review the following files and modify all custom meta keys to start with an alpha, and contain alphanumeric, (.), (_) only and has a maximum length of 16 characters. Concentrators - /etc/netwitness/ng/index-concentrator.xml /etc/netwitness/ng/index-concentrator-custom.xml
In addition, update the affected custom keys in custom feeds and parsers if exist.
In case the decoder/logdecoder has unconsumed data which was captured before the upgrade, you may need to modify 'Aggregate Hours' from the downstream concentrator's Config page to skip the sessions which contain the problematic sessions. e.g. If the decoder was upgraded at 5 AM and the current time is 10 AM, 'Aggregate Hours' can be set to 5 to aggregate only the sessions which are captured after the upgrade.