000039054 - Jobs are not getting processed even after restarting the RSA Archer Job Engine Service

Document created by RSA Customer Support Employee on Jul 30, 2020
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000039054
Applies ToRSA Product Set:  Archer
RSA Product/Service Type: Archer Control Panel, Job Engine Manager
RSA Version/Condition: 6.x
Platform (Other): SQL Server
O/S Version: Windows
IssueJobs are stuck in Archer Control Panel > Plugins > Job Engine Manager > Available Jobs and no jobs are running even after restarting the Archer Job Engine Service.
When reviewing the Archer.ArcherTech.JobFramework.Job log files (Archer Control Panel > Plugins > Installations Settings > Logging) , the following errors are seen.

 

<ApplicationData>
  <TraceData>
    <DataItem>
      <TraceRecord Severity="Critical" xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord">
        <TraceIdentifier>Engine</TraceIdentifier>
        <Description>Content save failed for content 2474311 with the following error: System.Transactions.TransactionException: The operation is not valid for the state of the transaction. ---&gt; System.TimeoutException: Transaction Timeout
   --- End of inner exception stack trace ---
   at System.Transactions.TransactionState.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction, Guid promoterType)
   at System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Guid promoterType)
   at System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification)
   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
   at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx)
   at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)
   at System.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal&amp; connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal&amp; connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.Practices.EnterpriseLibrary.Data.Database.GetNewOpenConnection()
   at Microsoft.Practices.EnterpriseLibrary.Data.TransactionScopeConnections.GetConnection(Database db)
   at Microsoft.Practices.EnterpriseLibrary.Data.Database.ExecuteReader(DbCommand command)
   at ArcherTech.Kernel.Providers.Data.ArcherSqlDatabase.&lt;&gt;c__DisplayClass5_0.&lt;ExecuteReader&gt;b__0()
   at ArcherTech.Kernel.Providers.Data.ArcherSqlDatabase.WrapDatabaseCall(DbCommand command, Action methodToWrap)
   at ArcherTech.Kernel.Providers.Data.ArcherSqlDatabase.ExecuteReader(DbCommand command)
   at ArcherTech.Kernel.DataSource.Db.DataSourceBase.PopulateIdPair[TKey,TValue](DbCommand command, MapIdPair`2 mapIdPairDelegate)
   at ArcherTech.Kernel.DataSource.Db.ContentDataSource.Update(IEnumerable`1 contentIds, Nullable`1 mainContentId, Boolean auditRecord)
   at ArcherTech.Kernel.Brokers.UpdateContentRecordTask.Execute(ContentContext contentContext)
   at ArcherTech.Kernel.Brokers.TaskExecutionHelper.ExecuteTasks[T](T contentContext, IList`1 tasks)
   at ArcherTech.Kernel.Brokers.TransactionalTask`1.ExecuteTransaction(T contentContext)
   at ArcherTech.Kernel.Brokers.TransactionalTask`1.Execute(T contentContext)
   at ArcherTech.Kernel.Brokers.TaskExecutionHelper.ExecuteTasks[T](T contentContext, IList`1 tasks)
   at ArcherTech.Kernel.Brokers.AcquireLocksTask`1.Execute(T contentContext)
   at ArcherTech.Kernel.Brokers.TaskExecutionHelper.ExecuteTasks[T](T contentContext, IList`1 tasks)
   at ArcherTech.Kernel.Brokers.IndexedContentIteratorTask`1.ExecuteNonthreadedIterator(T contentContext, IList`1 indexedContentContexts)
   at ArcherTech.Kernel.Brokers.TaskExecutionHelper.ExecuteTasks[T](T contentContext, IList`1 tasks)
   at ArcherTech.Kernel.Brokers.ContentBroker.ExecuteContentSavePlan(ContentContext contentContext, IDictionary`2 indexMap, ContentSavePerformanceData performanceData, BatchContentSaveResult returnResults)
   at ArcherTech.Kernel.Brokers.ContentBroker.Save(IList`1 contentContainers, Nullable`1 subformFieldId, ContentSaveOptions saveOptions)
   at ArcherTech.Kernel.Brokers.ContentBroker.Save(CalculationContent calculationContent)
   at ArcherTech.Kernel.Managers.CalculationContentManager.Save(SessionContext sessionContext, CalculationContent calculationContent)
   at ArcherTech.Services.Providers.DataProviders.ArcherCommonDataProvider.SaveContents(String instanceName, EntityPipeListener`1 contentPipeReader)</Description>
        <AppDomain>ArcherTech.JobFramework.Job.exe</AppDomain>
      </TraceRecord>
    </DataItem>
  </TraceData>
</ApplicationData>
CauseThe tblAsyncInstrumentation table does not have enough space.

 
ResolutionRun the following query against the Archer Instance Database to verify the size of the tblAsyncInstrumentation table:
 

sp_spaceused tblAsyncInstrumentation

 


Sample result:

Name          tblAsyncInstrumentation   
Rows          195631   
Reserved      10232720 KB   
Data          10014584 KB     
Index_size    65424 KB 
Unused        152712 KB

 

  1. Please contact RSA Customer Support and request assistance on cleaning the tblAsyncInstrumentation table. Knowledge Base Article - 000030673 - RSA Archer – Performance Issues – Huge tblAsyncInstrumentation table
  2. Restart the RSA Archer Job Engine Service once the Knowledge Base Article above has been applied.
  3. Verify that the "Discontinue Job Processing" is not checked on the Archer Control Panel > Job Engine Manager > Servers
  4. Verify if the jobs started processing again by simply clicking the Refresh button on the bottom-right corner of the Job Engine Manager window.

Attachments

    Outcomes