IIS Hangs on Restart with Many Application Pools
3 years ago
Originally Published: 2006-11-07
Article Number
000057883
Issue
With a large number of application pools defined in IIS, restarting IIS while under a constant load (e.g., from a load balancer sending periodic keepalives) can cause IIS to hang, eventually timing out all application pools.
Windows Event Viewer for applications shows errors describing application pools failing to respond in time and being marked out of use.
Cause
The IIS agent uses the Win32 API function OutputDebugString( ) to dump configuration and request processing information to the Win32 debugger (visible with the utility DBWin32 shipped with the agent, located in the util directory).  The OutputDebugString( ) function uses a mutex named \BaseNamedObjects\DBWinMutex for synchronizing access to low level logging functions.  Per this article in Microsoft's knowledgebase, threads depending on this mutex may fail to release it under certain conditions; in a ClearTrust agent environment, these conditions appear to be a large number of agent processes dumping their configuration information at startup.  This can occur when a large number of application pools are configured.  Since the agent doesn't start until the first request it receives, a low level but constant load can trigger most or all agents to dump configuration simultaneously, locking up every app pool and hanging IIS.
Resolution

This is a known issue that was addressed in hotfix 4.6.0.130.  Please contact customer support to obtain this hotfix. 

The hotfix addresses the issue by suppressing output to the debugger using OutputDebugString( ) when an environment variable, CT_CONFIG_OUTPUT, is set to 0.  Since IIS typically runs under the NETWORK SERVICE account, the environment variable must be defined as a system-wide environment variable, and the operating system must be restarted in order to make that environment variable available to the NETWORK SERVICE account.


Notes

This setting only applies if log_flags 04 bit is unset.

When the log_flags 04 bit is set (flags=20) the agent will always echo all log messages to standard output including the <CONFIG> level log messages.  This is regardless of any setting for C_CONFIG_OUTPUT.

When log_flags 04 bit is unset (flags=16) the agent suppresses normal log messages from being echoed to standard output except for <CONFIG> level log messages, unless the C_CONFIG_OUTPUT environment variable is also configured and set to a value of zero, in which case all log messages including <CONFIG> level messages are suppressed.


Don't see what you're looking for?