on Jun 14, 2016Article Content
| Article Number | 000031227 |
| Applies To | RSA Product Set: Identity Management and Governance RSA Product/Service Type: Enterprise Software Platform: WebSphere |
| Issue | Symptoms: - Some Change request staying in Open state - Workpoint Log file indicates that Workpoint QMonitor Service fail to start on secondary node when using Websphere Clustered The following errors can be found on the wpqmonitor.log file 2015-08-24 12:30:19,197 [Thread-100] ERROR com.workpoint.queue.core.QMonitor - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server. java.rmi.RemoteException: Attempt failed to retrieve authorization options from server. Either the server is not running or the client configuration may not be set up correctly. See the client log file for details. at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) The following errors can be found on the WorkPoint.log file: 2015-08-24 12:30:19,185 [Thread-100] ERROR com.workpoint.client.connect.ClientContextEJB - An error occured while attempting to get a connection to the server. java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source) at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) Caused by: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762) at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513) at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180) at javax.naming.InitialContext.lookup(InitialContext.java:436) at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source) ... 10 more Caused by: org.omg.CORBA.NO_RESPONSE: Request 172 timed out vmcid: IBM minor code: B01 completed: Maybe at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502) at com.ibm.rmi.iiop.Connection.send(Connection.java:2429) at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278) at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484) at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806) at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771) at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803) ... 18 more No relevant errors found on aveksaServer.log. The following errors can be found on the SystemOut.log [8/24/15 12:30:19:191 CDT] 0000005f SystemOut O 2015-08-24 12:30:19,185 [Thread-100] ERROR com.workpoint.client.connect.ClientContextEJB - An error occured while attempting to get a connection to the server. java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source) at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) Caused by: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762) at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513) at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180) at javax.naming.InitialContext.lookup(InitialContext.java:436) at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source) ... 10 more Caused by: org.omg.CORBA.NO_RESPONSE: Request 172 timed out vmcid: IBM minor code: B01 completed: Maybe at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502) at com.ibm.rmi.iiop.Connection.send(Connection.java:2429) at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278) at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484) at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806) at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771) at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803) ... 18 more [8/24/15 12:30:19:197 CDT] 0000005f SystemOut O 2015-08-24 12:30:19,197 [Thread-100] WARN com.workpoint.client.ClientContext - Attempt failed to retrieve authorization options from server. Either the server is not running or the client configuration may not be set up correctly. See the client log file for details. [8/24/15 12:30:19:197 CDT] 0000005f SystemOut O 2015-08-24 12:30:19,197 [Thread-100] ERROR com.workpoint.queue.core.QMonitor - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server. java.rmi.RemoteException: Attempt failed to retrieve authorization options from server. Either the server is not running or the client configuration may not be set up correctly. See the client log file for details. at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) [8/24/15 12:30:19:197 CDT] 00000061 SystemOut O 2015-08-24 12:30:19,186 [Thread-99] ERROR com.workpoint.client.connect.ClientContextEJB - An error occured while attempting to get a connection to the server. java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 176 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source) at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) Caused by: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 176 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762) at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513) at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180) at javax.naming.InitialContext.lookup(InitialContext.java:436) at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source) ... 10 more Caused by: org.omg.CORBA.NO_RESPONSE: Request 176 timed out vmcid: IBM minor code: B01 completed: Maybe at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502) at com.ibm.rmi.iiop.Connection.send(Connection.java:2429) at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278) at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484) at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806) at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771) at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803) ... 18 more [8/24/15 12:30:19:197 CDT] 00000061 SystemOut O 2015-08-24 12:30:19,197 [Thread-99] WARN com.workpoint.client.ClientContext - Attempt failed to retrieve authorization options from server. Either the server is not running or the client configuration may not be set up correctly. See the client log file for details. [8/24/15 12:30:19:198 CDT] 00000061 SystemOut O 2015-08-24 12:30:19,197 [Thread-99] ERROR com.workpoint.queue.core.QMonitor - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server. java.rmi.RemoteException: Attempt failed to retrieve authorization options from server. Either the server is not running or the client configuration may not be set up correctly. See the client log file for details. at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) [8/24/15 12:30:19:200 CDT] 00000060 SystemOut O 2015-08-24 12:30:19,200 [Thread-98] ERROR com.workpoint.client.connect.ClientContextEJB - An error occured while attempting to get a connection to the server. java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 174 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source) at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) Caused by: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 174 timed out vmcid: IBM minor code: B01 completed: Maybe] at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762) at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513) at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166) at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180) at javax.naming.InitialContext.lookup(InitialContext.java:436) at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source) at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source) ... 10 more Caused by: org.omg.CORBA.NO_RESPONSE: Request 174 timed out vmcid: IBM minor code: B01 completed: Maybe at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502) at com.ibm.rmi.iiop.Connection.send(Connection.java:2429) at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248) at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709) at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278) at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484) at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810) at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806) at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771) at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804) at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803) ... 18 more [8/24/15 12:30:19:201 CDT] 00000060 SystemOut O 2015-08-24 12:30:19,201 [Thread-98] WARN com.workpoint.client.ClientContext - Attempt failed to retrieve authorization options from server. Either the server is not running or the client configuration may not be set up correctly. See the client log file for details. [8/24/15 12:30:19:201 CDT] 00000060 SystemOut O 2015-08-24 12:30:19,201 [Thread-98] ERROR com.workpoint.queue.core.QMonitor - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server. java.rmi.RemoteException: Attempt failed to retrieve authorization options from server. Either the server is not running or the client configuration may not be set up correctly. See the client log file for details. at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.client.ClientContext.open(Unknown Source) at com.workpoint.queue.core.QMonitor.F(Unknown Source) at com.workpoint.queue.core.QMonitor.H(Unknown Source) at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source) at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source) at java.util.Timer$TimerImpl.run(Timer.java:296) The following error also can be found on SystemOut.log file [8/24/15 13:39:58:503 CDT] 00000013 XARminst E WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@1c721c72. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709) at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:1088) at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:141) at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:626) at com.ibm.tx.jta.PartnerLogTable.recover(PartnerLogTable.java:389) at com.ibm.tx.jta.RecoveryManager.resync(RecoveryManager.java:1530) at com.ibm.tx.jta.RecoveryManager.performResync(RecoveryManager.java:2265) at com.ibm.ws.tx.jta.RecoveryManager.performResync(RecoveryManager.java:114) at com.ibm.tx.jta.RecoveryManager.run(RecoveryManager.java:2218) at java.lang.Thread.run(Thread.java:761) |
| Cause | See http://www-01.ibm.com/support/docview.wss?uid=swg21288663 that indicates that This condition could be caused by corrupt transaction logs. |
| Resolution | Please refer to the following WebSphere Application Server TechNote for instructions on how to properly handle the correction of the tranlogs http://www-01.ibm.com/support/docview.wss?uid=swg21196663 or contact IBM Support for assistance to delete translog. |