Service Provider hangs at throughput of 5 assertions per second
2 years ago
Originally Published: 2011-09-21
Article Number
000048122
Applies To
RSA FIM 4.1
Issue
Service Provider hangs at throughput of 5 assertions per second

####<4/06/2011 08:12:52 AM NZST> <Error> <WebLogicServer> <cls-wlg-sa-s2> <rsaappserver> <[STANDBY] ExecuteThread: '379' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1307131972502> <BEA-000337> <[STUCK] ExecuteThread: '233' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "609" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@3413535[
GET /samlrelyingparty/RP?SAMLart=AAHVcNLblvNJNBwfdpa5Tqt4Slo3j08MMoK94rgrp%2BJ6TyRtvZrc2Cwq&TARGET=https%3A%2F%2Frefagency-saml11.logon.i.govt.nz%2Fref-agency2-ite%2Fref%2FLogon HTTP/1.1
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
User-Agent: JMeter Auth User=PTUserITE0010 GLS=${GLS} Thread=10 SAML=2.0
Connection: Keep-Alive
Proxy-Client-IP: 202.175.142.227
X-Forwarded-For: 202.175.142.227
X-WebLogic-Request-ClusterInfo: true
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: -136368276

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-290 "[STUCK] ExecuteThread: '233' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, blocked, priority=1, DAEMON> {
    -- Blocked trying to get lock: org.jpox.store.rdbms.table.ClassTable@2a51b5[fat lock]
    org.jpox.store.rdbms.table.ClassTable.fetch(ClassTable.java:2133)
    org.jpox.store.StoreManager.fetch(StoreManager.java:745)
    org.jpox.state.StateManagerImpl.loadFieldsInFetchPlan(StateManagerImpl.java:1671)
    org.jpox.state.StateManagerImpl.validate(StateManagerImpl.java:4013)
    org.jpox.AbstractPersistenceManager.getObjectById(AbstractPersistenceManager.java:2521)
    org.jpox.AbstractPersistenceManager.getObjectById(AbstractPersistenceManager.java:2496)
    org.jpox.AbstractPersistenceManager.getObjectById(AbstractPersistenceManager.java:2485)
    ^-- Holding lock: org.jpox.PersistenceManagerImpl@5492d30[thin lock]
    com.rsa.fim.profile.util.ProfileHelper.getCertVerificationConfig(ProfileHelper.java:443)
    com.rsa.fim.profile.util.SAML11ProfileHelper.verifySignature(SAML11ProfileHelper.java:295)
    com.rsa.fim.profile.util.SAML11ProfileHelper.verifyResponse(SAML11ProfileHelper.java:203)
    com.rsa.fim.profile.sso.SAML11SSOService.processRelyingPartyMode(SAML11SSOService.java:459)
    com.rsa.fim.profile.sso.SSOProfileBean.processRelyingPartyMode(SSOProfileBean.java:296)
    com.rsa.fim.profile.common.FIMProfileBean.processRelyingPartyMode(FIMProfileBean.java:83)
    com.rsa.fim.profile.common.FIMProfile_mzkd72_EOImpl.processRelyingPartyMode(FIMProfile_mzkd72_EOImpl.java:1789)
    com.rsa.fim.servlet.sso.SAML11AssertionConsumerServiceServlet.doGet(SAML11AssertionConsumerServiceServlet.java:52)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
    weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
    weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
    weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
    weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    com.rsa.fim.servlet.FIMGenericServletFilter.doFilter(FIMGenericServletFilter.java:28)
    weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
    weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3560)
    weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
    weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2136)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2058)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1394)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}


Resolution
Set a positive 'Trust List Verification Interval' in the TrustStore
Settings so that the threads do not perform read & write operations
frequently (The default value is 0 which means for every request a read
and write operation is performed and each thread sets the 'valid until'
to its current time and the new threads always see this as an expired
interval so they also update the interval to their current time.I guess
in production environment we usually set a positive interval. Besides
this problem is seen only in standalone setup (clustered setup uses in
memory cache rather than updating the db) .