000031556 - Error message "[crit] wtd51 Unable to create thread due to limit on number of processes" in RSA Web Threat Detection

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000031556
Applies ToRSA Product Set: Web Threat Detection
RSA Version/Condition: 5.0 and above
Platform: Linux
O/S Version: Red Hat Enterprise Linux 6.x
IssueFirst seen with issue involving CProfileUpdater, but can affect any process depending on start up order.
As each WTD process starts it will create a predefined number of worker threads and once the user process limit is reached, the following error will start to appear in syslog associated with the troubled process:
[crit] wtd51 Unable to create thread due to limit on number of processes.

The resulting behaviour will differ according to the process and actual thread function that failed to be created.
Often the process will run on afterwards and appear healthy but the the missing worker threads can cause symptoms such as background operations to fail or for memory and CPU to increase in the remaining worker threads.
 
 
CauseRHEL6 has a user process limit of 1024 (RHEL7=4096) which includes worker threads forked by processes.
This limit is far too low for use with a production server and in particular on full WTD implementation.

 
Resolution
First off, what does ulimit -u return now?  On similar OSs the limit can vary but on rhel6 the default is 1024.


# ulimit -u


 

Start by identifying which process hit this problem. (in my example below it was CProfileUpdater.)

Enable a second console running: 


# tail -f <your live syslog> | grep -i cprof



In the main console:


# /etc/init.d/st-CProfileUpdater-wtd51 restart


 

We would expect the syslog to show something like this:


Oct 20 21:35:12 wtd51 cprofileupdater[26044]: [crit] wtd51 Unable to create thread due to limit on number of processes.


 

now increase the ulimit to 2048 and restart CProfileUpdater.
# ulimit -u 2048
# /etc/init.d/st-CProfileUpdater-wtd51restart


 

If threads are created without the error and logs similar to the example below are reported then we have a good limit. Otherwise increase further and repeat.



Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.0
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.1
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.2
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.3
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.4
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.5
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.6
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.7
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.8
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.9
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.10
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.11
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.12
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.13
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.14
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Subscribing to topic txn.16.15
Oct 20 21:37:33 wtd51 cprofileupdater[26756]: [info] wtd51 Creating 16 threads
Oct 20 21:37:34 wtd51 cprofileupdater[26756]: [info] wtd51 [listener channel 1.1] STMS connected







 
NotesCAUTION:
The ulimit -u <processlimit> command is not permanent and will reset at reboot.  
To make this change permanent modify the line below in the /etc/security/limits.d/90-nproc.conf file.
"* soft nproc 1024"

For more details on this setting, see the following link:  https://bugzilla.redhat.com/show_bug.cgi?id=432903

Attachments

    Outcomes