Error message "[crit] wtd51 Unable to create thread due to limit on number of processes" in RSA Web Threat Detection
RSA Product Set: Web Threat Detection RSA Version/Condition: 5.0 and above Platform: Linux O/S Version: Red Hat Enterprise Linux 6.x
First seen with issue involving CProfileUpdater, but can affect any process depending on start up order. As each WTD process starts it will create a predefined number of worker threads and once the user process limit is reached, the following error will start to appear in syslog associated with the troubled process:
[crit] wtd51 Unable to create thread due to limit on number of processes.
The resulting behaviour will differ according to the process and actual thread function that failed to be created. Often the process will run on afterwards and appear healthy but the the missing worker threads can cause symptoms such as background operations to fail or for memory and CPU to increase in the remaining worker threads.
RHEL6 has a user process limit of 1024 (RHEL7=4096) which includes worker threads forked by processes. This limit is far too low for use with a production server and in particular on full WTD implementation.
First off, what does ulimit -u return now? On similar OSs the limit can vary but on rhel6 the default is 1024.
# ulimit -u
Start by identifying which process hit this problem. (in my example below it was CProfileUpdater.)
Enable a second console running:
# tail -f <your live syslog> | grep -i cprof
In the main console:
# /etc/init.d/st-CProfileUpdater-wtd51 restart
We would expect the syslog to show something like this:
Oct 20 21:35:12 wtd51 cprofileupdater: [crit] wtd51 Unable to create thread due to limit on number of processes.
now increase the ulimit to 2048 and restart CProfileUpdater.