|Issue||With a large number of application pools defined in IIS, restarting IIS while under a constant load (e.g., from a load balancer sending periodic keepalives) can cause IIS to hang, eventually timing out all application pools.|
Windows Event Viewer for applications shows errors describing application pools failing to respond in time and being marked out of use.
|Cause||The IIS agent uses the Win32 API function OutputDebugString( ) to dump configuration and request processing information to the Win32 debugger (visible with the utility DBWin32 shipped with the agent, located in the util directory). The OutputDebugString( ) function uses a mutex named \BaseNamedObjects\DBWinMutex for synchronizing access to low level logging functions. Per this article in Microsoft's knowledgebase, threads depending on this mutex may fail to release it under certain conditions; in a ClearTrust agent environment, these conditions appear to be a large number of agent processes dumping their configuration information at startup. This can occur when a large number of application pools are configured. Since the agent doesn't start until the first request it receives, a low level but constant load can trigger most or all agents to dump configuration simultaneously, locking up every app pool and hanging IIS.|
This is a known issue that was addressed in hotfix 22.214.171.124. Please contact customer support to obtain this hotfix.
The hotfix addresses the issue by suppressing output to the debugger using OutputDebugString( ) when an environment variable, CT_CONFIG_OUTPUT, is set to 0. Since IIS typically runs under the NETWORK SERVICE account, the environment variable must be defined as a system-wide environment variable, and the operating system must be restarted in order to make that environment variable available to the NETWORK SERVICE account.
This setting only applies if log_flags 04 bit is unset.
When the log_flags 04 bit is set (flags=20) the agent will always echo all log messages to standard output including the <CONFIG> level log messages. This is regardless of any setting for C_CONFIG_OUTPUT.
When log_flags 04 bit is unset (flags=16) the agent suppresses normal log messages from being echoed to standard output except for <CONFIG> level log messages, unless the C_CONFIG_OUTPUT environment variable is also configured and set to a value of zero, in which case all log messages including <CONFIG> level messages are suppressed.
|Legacy Article ID||a32401|