Cannot get a connection, pool error Timeout waiting for idle object

[LakshmananRaman]

We been experiencing sporadic  error " Cannot get a connection, pool error Timeout waiting for idle object.

2021-01-07 17:50:12,618 ERROR [pool-11-thread-3] [] [] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] - <Cannot get a connection, pool error Timeout waiting for idle object>

Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

2021-01-07 17:50:14,122 ERROR [http-apr-10.14.138.64-8443-exec-136] [] [] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] - <Cannot get a connection, pool error Timeout waiting for idle object>

org.springframework.orm.hibernate4.HibernateJdbcException: JDBC exception on Hibernate data access: SQLException for SQL [n/a]; SQL state [null]; error code [0]; Could not open connection; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection

Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

2021-01-07 17:50:19,134 ERROR [http-apr-10.14.138.64-8443-exec-141] [] [] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] - <Cannot get a connection, pool error Timeout waiting for idle object>

Currently we have 12 JVM load balanced. total of 1680 database connections in a given time.  Initially we had 9 JVM and increased 3 more and we still have the problem.  Here is the DB connection string.

                maxActive="140"

                maxWait="25000"

                fastConnectionFailoverEnabled="true"

                driverClassName="oracle.jdbc.OracleDriver"

                maxIdle="40"

                validationQuery="Select 1 from Dual"

[LakshmananRaman]

Issue seems to be a know bug in 7.3P4. If anyone using 7.3P4 you will experience the same issue.

Here is the work around.

With the help of Support engineer Nag Madugula we were able to resolve the issue.

As per the internal notes, this was determined to a bug in drools library version 5.6.0 wherein under certain scenarios the code runs into a race condition causing deadlocks in a multithreaded - high volume environment.
The drools library was upgraded from 5.3.1 to 5.6.0 as part of applying 7.3 patch 4 in your environment – which explains why you only see this issue after the upgrade.

Reported issues with 5.6.0 version of Drools:

https://issues.jboss.org/browse/DROOLS-618

https://issues.jboss.org/browse/DROOLS-660

Drools fixed this deadlock issue in their version 7 library, which was integrated into AA in 7.3 P6 version onwards – please note drools version 7 has dependency on mvel2-2.4.x, which only support Java 8 and above.

Since you are still running on Java 7, you could roll back the drools library from 5.6.0 to 5.3.1 (old drools version used in 7.3 P2) as a workaround – until you can upgrade to Java 8 and AA 7.3 P6 or above.

This shouldn’t impact your environment, as the drools version upgrade from 5.3.1 to 5.6.0 was to fix a specific issue with mvel library and Java 8 version, And since you are not using Java 8 at the moment, rolling back to older version of drools (5.3.1) & the associated libraries should NOT cause any compatibility issues.

Below are the steps to roll back drools and associated libs:

1. Stop the server

2. Clear the Tomcat’s temp directory

3. Remove below jars

drools-compiler-5.6.0.Final.jar,

drools-core-5.6.0.Final.jar,

drools-templates-5.6.0.Final.jar,

mvel2-2.1.8.Final.jar,

knowledge-api-5.6.0.Final.jar,

knowledge-internal-api-5.6.0.Final.jar

From \webapps\AdaptiveAuthentication\WEB-INF\lib

4. Download or replace the below jars from the 7.3 P2 version of AdaptiveAuthentication war file:

drools-compiler-5.3.1.Final.jar,

drools-core-5.3.1.Final.jar,

drools-templates-5.3.1.Final.jar,

knowledge-api-5.3.1.Final.jar,

mvel2-2.1.0.drools4.jar

You could also download the above version jars from https://mvnrepository.com and place them under /webapps/AdaptiveAuthentication/WEB-INF/lib

5. Replace deviceRecovery.drl present in “/rsa/configs” directory with the one from 7.3 P2 version.

6. Restart the server.