000031227 - Some Change request staying in Open state and Workpoint QMonitor Service fail to start on node2 using Websphere Clustered with RSA VIA L&G

Document created by RSA Customer Support Employee on Jun 14, 2016•Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000031227
Applies ToRSA Product Set: Identity Management and Governance
RSA Product/Service Type: Enterprise Software
Platform: WebSphere
 
IssueSymptoms:
- Some Change request staying in Open state
- Workpoint Log file indicates that Workpoint QMonitor Service fail to start on secondary node when using Websphere Clustered 
The following errors can be found on the wpqmonitor.log file 



2015-08-24 12:30:19,197 [Thread-100] ERROR com.workpoint.queue.core.QMonitor  - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server.

java.rmi.RemoteException: Attempt failed to retrieve authorization options from server.  Either the server is not running or the client configuration may not be set up correctly. See the client log file for details.

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)



The following errors can be found on the WorkPoint.log file: 



2015-08-24 12:30:19,185 [Thread-100] ERROR com.workpoint.client.connect.ClientContextEJB  - An error occured while attempting to get a connection to the server.

java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: 

    javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source)

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)

Caused by: 

javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180)

    at javax.naming.InitialContext.lookup(InitialContext.java:436)

    at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source)

    ... 10 more

Caused by: 

org.omg.CORBA.NO_RESPONSE: Request 172 timed out  vmcid: IBM  minor code: B01 completed: Maybe

    at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502)

    at com.ibm.rmi.iiop.Connection.send(Connection.java:2429)

    at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278)

    at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484)

    at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806)

    at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771)

    at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803)

    ... 18 more



No relevant errors found on aveksaServer.log. 



The following errors can be found on the SystemOut.log

[8/24/15 12:30:19:191 CDT] 0000005f SystemOut     O 2015-08-24 12:30:19,185 [Thread-100] ERROR com.workpoint.client.connect.ClientContextEJB  - An error occured while attempting to get a connection to the server.

java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: 

    javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source)

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)

Caused by: 

javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 172 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180)

    at javax.naming.InitialContext.lookup(InitialContext.java:436)

    at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source)

    ... 10 more

Caused by: 

org.omg.CORBA.NO_RESPONSE: Request 172 timed out  vmcid: IBM  minor code: B01 completed: Maybe

    at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502)

    at com.ibm.rmi.iiop.Connection.send(Connection.java:2429)

    at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278)

    at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484)

    at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806)

    at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771)

    at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803)

    ... 18 more

[8/24/15 12:30:19:197 CDT] 0000005f SystemOut     O 2015-08-24 12:30:19,197 [Thread-100] WARN  com.workpoint.client.ClientContext  - Attempt failed to retrieve authorization options from server.  Either the server is not running or the client configuration may not be set up correctly. See the client log file for details.

[8/24/15 12:30:19:197 CDT] 0000005f SystemOut     O 2015-08-24 12:30:19,197 [Thread-100] ERROR com.workpoint.queue.core.QMonitor  - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server.

java.rmi.RemoteException: Attempt failed to retrieve authorization options from server.  Either the server is not running or the client configuration may not be set up correctly. See the client log file for details.

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)

[8/24/15 12:30:19:197 CDT] 00000061 SystemOut     O 2015-08-24 12:30:19,186 [Thread-99] ERROR com.workpoint.client.connect.ClientContextEJB  - An error occured while attempting to get a connection to the server.

java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: 

    javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 176 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source)

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)

Caused by: 

javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 176 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180)

    at javax.naming.InitialContext.lookup(InitialContext.java:436)

    at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source)

    ... 10 more

Caused by: 

org.omg.CORBA.NO_RESPONSE: Request 176 timed out  vmcid: IBM  minor code: B01 completed: Maybe

    at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502)

    at com.ibm.rmi.iiop.Connection.send(Connection.java:2429)

    at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278)

    at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484)

    at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806)

    at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771)

    at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803)

    ... 18 more

[8/24/15 12:30:19:197 CDT] 00000061 SystemOut     O 2015-08-24 12:30:19,197 [Thread-99] WARN  com.workpoint.client.ClientContext  - Attempt failed to retrieve authorization options from server.  Either the server is not running or the client configuration may not be set up correctly. See the client log file for details.

[8/24/15 12:30:19:198 CDT] 00000061 SystemOut     O 2015-08-24 12:30:19,197 [Thread-99] ERROR com.workpoint.queue.core.QMonitor  - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server.

java.rmi.RemoteException: Attempt failed to retrieve authorization options from server.  Either the server is not running or the client configuration may not be set up correctly. See the client log file for details.

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)

[8/24/15 12:30:19:200 CDT] 00000060 SystemOut     O 2015-08-24 12:30:19,200 [Thread-98] ERROR com.workpoint.client.connect.ClientContextEJB  - An error occured while attempting to get a connection to the server.

java.rmi.RemoteException: NamingException caught attempting to locate the 'ServerConfigPvt_EJB' object. Please verify that the server is running and the client configuration is correct.; nested exception is: 

    javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 174 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.workpoint.client.connect.ClientContextEJB.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.getServerConfigPvt(Unknown Source)

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)

Caused by: 

javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.NO_RESPONSE: Request 174 timed out  vmcid: IBM  minor code: B01 completed: Maybe]

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1853)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1762)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1513)

    at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:645)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:166)

    at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:180)

    at javax.naming.InitialContext.lookup(InitialContext.java:436)

    at com.workpoint.client.connect.ClientContextEJB$_A.A(Unknown Source)

    at com.workpoint.client.connect.ClientContextEJB.B(Unknown Source)

    ... 10 more

Caused by: 

org.omg.CORBA.NO_RESPONSE: Request 174 timed out  vmcid: IBM  minor code: B01 completed: Maybe

    at com.ibm.rmi.iiop.Connection.getCallStream(Connection.java:2502)

    at com.ibm.rmi.iiop.Connection.send(Connection.java:2429)

    at com.ibm.rmi.iiop.ClientRequestImpl.invoke(ClientRequestImpl.java:342)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:427)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1248)

    at com.ibm.rmi.corba.ClientDelegate.invoke(ClientDelegate.java:709)

    at com.ibm.CORBA.iiop.ClientDelegate.invoke(ClientDelegate.java:1278)

    at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:484)

    at com.ibm.WsnOptimizedNaming._NamingContextStub.resolve_complete_info(_NamingContextStub.java:492)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2810)

    at com.ibm.ws.naming.jndicos.CNContextImpl$2.run(CNContextImpl.java:2806)

    at com.ibm.ws.naming.util.CommonHelpers.retry(CommonHelpers.java:771)

    at com.ibm.ws.naming.jndicos.CNContextImpl.cosResolve(CNContextImpl.java:2804)

    at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1803)

    ... 18 more

[8/24/15 12:30:19:201 CDT] 00000060 SystemOut     O 2015-08-24 12:30:19,201 [Thread-98] WARN  com.workpoint.client.ClientContext  - Attempt failed to retrieve authorization options from server.  Either the server is not running or the client configuration may not be set up correctly. See the client log file for details.

[8/24/15 12:30:19:201 CDT] 00000060 SystemOut     O 2015-08-24 12:30:19,201 [Thread-98] ERROR com.workpoint.queue.core.QMonitor  - Attempt failed to create a client context. The monitors will not execute properly without this connection to the server.

java.rmi.RemoteException: Attempt failed to retrieve authorization options from server.  Either the server is not running or the client configuration may not be set up correctly. See the client log file for details.

    at com.workpoint.client.ClientContext.retrieveAuthInfo(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.client.ClientContext.open(Unknown Source)

    at com.workpoint.queue.core.QMonitor.F(Unknown Source)

    at com.workpoint.queue.core.QMonitor.H(Unknown Source)

    at com.workpoint.queue.core.QMonitor.startMonitor(Unknown Source)

    at com.workpoint.queue.WpQMonitors$_A.run(Unknown Source)

    at java.util.Timer$TimerImpl.run(Timer.java:296)





The following error also can be found on SystemOut.log file 



[8/24/15 13:39:58:503 CDT] 00000013 XARminst      E   WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@1c721c72. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException

    at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709)

    at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:1088)

    at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:141)

    at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:626)

    at com.ibm.tx.jta.PartnerLogTable.recover(PartnerLogTable.java:389)

    at com.ibm.tx.jta.RecoveryManager.resync(RecoveryManager.java:1530)

    at com.ibm.tx.jta.RecoveryManager.performResync(RecoveryManager.java:2265)

    at com.ibm.ws.tx.jta.RecoveryManager.performResync(RecoveryManager.java:114)

    at com.ibm.tx.jta.RecoveryManager.run(RecoveryManager.java:2218)

    at java.lang.Thread.run(Thread.java:761)
CauseSee http://www-01.ibm.com/support/docview.wss?uid=swg21288663 that indicates that 
This condition could be caused by corrupt transaction logs. 



 
ResolutionPlease refer to the following WebSphere Application Server TechNote for instructions on how to properly handle the correction of the tranlogs 

http://www-01.ibm.com/support/docview.wss?uid=swg21196663  or contact IBM Support for assistance to delete translog.

 

Attachments

    Outcomes