000027925 - RSA Access Certification Manager server does not start and shows Initialization failure: Error scheduling database statistics

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000027925
Applies ToRSA Product Set:  Identity Management and Governance (IMG)
RSA Product/Service Type:  Access Certification Manager
RSA Version/Condition:  4.x, 5.x





 
IssueThe Access Certification Manager server is not able to start. Usually it is an ACM WebSphere or WebLogic Application Server that won't start.

A review of the aveksaServer.log, shows errors similar to those shown below:


For WebLogic,the path will vary based on your WebLogic install location, as well as your domain, but should follow a pattern similar to: $WEBLOGIC_HOME/user_projects/domains/aveksaDomain/servers/AdminServer/tmp/_WL_user/aveksa/ldedze/aveksa.war/log/aveksaServer.log


02/10/2012 19:03:51.131 INFO ([STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)')
[Container] [Wed Nov 10 19:03:51 CET 2010] [0] [AveksaServiceContainer] [Register Service] [FAILED] [] [Service Registered SystemService]

 
For WebSphere, the path will vary based on your WebSphere install location, as well as your domain, but should follow this pattern: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/hostnameNode01Cell/aveksa.ear/log/aveksaServer.log
 
02/10/2012 18:52:38.982 ERROR (server.startup : 0) [com.aveksa.server.core.Container] Unable to start service SystemService.
com.aveksa.server.runtime.ServerException: Error scheduling database statistics.
at com.aveksa.server.system.SystemServiceProvider.initDBStatsScheduledTask(SystemServiceProvider.java:2910)
at com.aveksa.server.system.SystemServiceProvider.start(SystemServiceProvider.java:450)
at com.aveksa.server.core.Container.startService(Container.java:141)
at com.aveksa.server.core.Container.initialize(Container.java:105)
at com.aveksa.server.runtime.AveksaSystem.initialize(AveksaSystem.java:210)
at com.aveksa.init.Startup.init(Startup.java:41)
at com.aveksa.init.InitServlet.init(InitServlet.java:48)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:358)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:169)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.initialize(ServletWrapper.java:1809)
at com.ibm.wsspi.webcontainer.extension.WebExtensionProcessor.createServletWrapper(WebExtensionProcessor.java:98)
at com.ibm.ws.webcontainer.webapp.WebApp.initializeTargetMappings(WebApp.java:704)
at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:435)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:304)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:100)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:166)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:731)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:616)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:376)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:668)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1123)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1319)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:610)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:944)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:726)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2048)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:441)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:384)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$300(CompositionUnitMgrImpl.java:112)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:951)
at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:349)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by:
com.aveksa.server.system.SystemServiceException: com.aveksa.server.scheduler.SchedulerServiceException: Task exists with the given name : gatherDatabaseStatistics for the
scheduler:SystemScheduler
at com.aveksa.server.system.SystemServiceProvider.setDBStatsSchedule(SystemServiceProvider.java:2884)
at com.aveksa.server.system.SystemServiceProvider.initDBStatsScheduledTask(SystemServiceProvider.java:2907)
... 32 more
Caused by:
com.aveksa.server.scheduler.SchedulerServiceException: Task exists with the given name : gatherDatabaseStatistics for the scheduler:SystemScheduler
at com.aveksa.server.scheduler.SchedulerServiceProvider.addPersistentScheduledTask(SchedulerServiceProvider.java:453)
at com.aveksa.server.scheduler.SchedulerServiceProvider.addPersistentScheduledTask(SchedulerServiceProvider.java:436)
at com.aveksa.server.system.SystemServiceProvider.setDBStatsSchedule(SystemServiceProvider.java:2879)
... 33 more
02/10/2012 18:52:38.984 FATAL (server.startup : 0) [com.aveksa.server.runtime.AveksaSystem]
****************************************
Initialization has failed!
Unable to start service SystemService. com.aveksa.server.runtime.ServerException: Error scheduling database statistics.
Cause

It is important to identify and note the specific service and code impacted in the stack trace found in the aveksaServer.log which causes the initialization error.  Scheduler services are checked on application server startup, and various server exceptions can be seen.   This specific error is related to the Database Statistics job.  Other errors, such as null pointer exceptions, are referenced in other knowledge base articles.
With regards to the error "Unable to start service SystemService.com.aveksa.server.runtime.ServerException: Error scheduling database statistics," the usual reason for this error, is that there is more than one entry for the gatherDatabaseStatistics job in the Scheduled Tasks table.  This can be confirmed with this sqlplus query, run as avuser:


SELECT count(*) FROM T_SCHEDULED_TASKS WHERE name = 'gatherDatabaseStatistics';

If the count value returns more than (1), then there is more than one row for this task and that is the cause of the error.  At this point in time, it is unclear why there may be more than one row. It has been seen to be caused by an incorrect cluster configuration, errors connecting to a clustered database, or multiple ACM Application Servers pointing to the same database in an non-clustered environment.

ResolutionTo correct this problem, delete the extra row, and restart the server.   An example of how to do this is shown below.  It does not matter which row is deleted.   Note that in the very unusual case of having more than two rows,  all the extra rows must be deleted, so that there is only one row for this specific task.  
     1.  From SQL, run the following:

SQL> col name format a30
SQL> set linesize 80
SQL> SELECT id, name FROM t_scheduled_tasks WHERE name = 'gatherDatabaseStatistics';
ID NAME
---------- ------------------------------
1120 gatherDatabaseStatistics
1148 gatherDatabaseStatistics

     2.  To delete a row, run the following, replacing the ID listed with the ID captured above: 
SQL> DELETE FROM t_scheduled_tasks WHERE id = 1120;

     3.  Confirm the deletion by re-running the count(*) command noted above:
SQL> SELECT count(*) FROM T_SCHEDULED_TASKS WHERE name = 'gatherDatabaseStatistics';
NotesThis specific problem may occur in ACM/IMG version 6.x, however it is unlikely.   
The scheduled task name identified in the SQL query run to diagnose this issue is 'gatherDatabaseStatistics'.  This is not the same task as that named 'Database_Statistics.GATHER_DATABASE_STATISTICS'.

Attachments

    Outcomes