Application is inaccessible and 'javax.naming.NamingException: Could not dereference object' error when stopping RSA Identity Governance & Lifecycle

[#1]

Article Number

000030707

Applies To

RSA Product Set: RSA Identity Governance & Lifecycle 
RSA Version/Condition: 6.9.1, (potentially 7.x as yet unconfirmed)
 

Issue

RSA Identity Governance & Lifecycle is not accessible and an attempt to stop RSA Identity Governance & Lifecycle results in the following stack trace:
 

$ service aveksa_server stop
Stopping Aveksa Server:
Exception in thread "main" javax.naming.NamingException: Could not dereference object 
[Root exception is javax.naming.NameNotFoundException: invoker not bound]
        at org.jnp.interfaces.NamingContext.resolveLink(NamingContext.java:1215)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:758)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
        at javax.naming.InitialContext.lookup(InitialContext.java:409)
        at org.jboss.Shutdown.main(Shutdown.java:214)
Caused by: javax.naming.NameNotFoundException: invoker not bound 
        at org.jnp.server.NamingServer.getBinding(NamingServer.java:529)
        at org.jnp.server.NamingServer.getBinding(NamingServer.java:537)
        at org.jnp.server.NamingServer.getObject(NamingServer.java:543)
        at org.jnp.server.NamingServer.lookup(NamingServer.java:267)
        at org.jnp.server.NamingServer.lookup(NamingServer.java:270)
        at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
        at java.lang.Thread.run(Thread.java:701)
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)
        at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
        at javax.naming.InitialContext.lookup(InitialContext.java:409)
        at org.jnp.interfaces.NamingContext.resolveLink(NamingContext.java:1209)
        ... 4 more

The aveksaServer.log file ($AVEKSA_HOME/jboss-4.2.2.GA/server/default/deploy/aveksa.ear/aveksa.war/log) has the following port conflict:
 

--- MBeans waiting for other MBeans ---
ObjectName: jboss:service=invoker,type=jrmp
  State: FAILED
  Reason: java.rmi.server.ExportException: Port already in use: 4444; nested exception is:
    java.net.BindException: Address already in use
  I Depend On:
    jboss:service=TransactionManager
  Depends On Me:
    jboss:service=proxyFactory,target=ClientUserTransactionFactory
    jboss:service=proxyFactory,target=ClientUserTransaction
    jboss.jmx:type=adaptor,name=Invoker,protocol=jrmp,service=proxyFactory

Note: This was seen on RSA Identity Governance & Lifecycle 6.9.1 which is End of Product Support (EOPS) but there is no reason to assume this could not happen in current versions. If you suspect you have this issue, please refer to RSA Knowledge Base Article 000030327 -- Artifacts to gather in RSA Identity Governance & Lifecycle to find the location of the aveksaServer.log file for your specific deployment. The aveksaServer.log may also be downloaded from the RSA Identity Governance & Lifecycle user interface (Admin > System > Server Nodes tab > under Logs.)
 

Cause

The error from the aveksaServer.log:

java.rmi.server.ExportException: Port already in use: 4444; nested exception is:     
java.net.BindException: Address already in use

indicates that the last time the server started it could not bind to port 4444 due to a conflict. 

This can be caused by a number of different events, but the specific cause in this case was that a third party script was written to stop RSA Identity Governance & Lifecycle in order to perform backups and the script was not cleanly stopping the application server. This prevented RSA Identity Governance & Lifecycle from starting cleanly as well.
 

Resolution

When designing a custom script to stop the RSA Identity Governance & Lifecycle application in order to perform maintenance tasks such as backups, make sure the script is cleanly stopping the application.

If assistance is needed in creating a custom script to automate shutdown and run administrative tasks, please contact the RSA Professional Services team or RSA Identity Governance & Lifecycle Customer Support who can connect you with the RSA Professional Services team.