000030327 - Artifacts to gather in RSA Identity Governance and Lifecycle

Document created by RSA Customer Support Employee on Sep 19, 2016Last modified by RSA Customer Support Employee on Aug 5, 2017
Version 5Show Document
  • View in full screen mode

Article Content

Article Number000030327
Applies ToRSA Product Set: RSA Identity Governance and Lifecycle
IssueThis article outlines what files and information are required by RSA Identity Governance and Lifecycle product support in order to assist with various types of issues.
TasksFor common reports and file locations, including WildFly locations for 7.x, please see the Notes section of this article.  For example, how to generate an ASR, an AWR report or find the aveksaServer.log file location.
ResolutionFor ALL issues, the aveksaServerInfo.log gives valuable information and should be sought in all cases!  The aveksaServerInfo.log file is in the same directory that the aveksaServer.log, see Notes.
  1. Access Request Issues
  2. AFX Issues
    1. AFX Installation
    2. AFX to ACM Connection
    3. AFX Connector Setup
    4. AFX CR not Auto-Fulfilled
    5. AFX Logging
    6. Customize Capability
    7. Bind App to Connector
    8. Connector Setup
    9. Connector Usage
    10. UI
  3. Certificate Issues:
    1. Unexpected Behavior
    2. Certificate Errors
  4. Data Collection issues:
    1. Collection Failure
    2. Collection Performance
    3. Collection Results
  5. Form Issues
  6. Generic Performance Issues
  7. High CPU Issues
  8. Install/Upgrade Issues
  9. Oracle database issues
  10. Report Issues
    1. Report Performance Issue
    2. Report Does Not Provide Expected Information
  11. Role Issues:
    1. Role Incorrect Behavior
    2. Role Product/System Error
    3. Role Performance Issue
  12. Workflow Issues
    1. Workflow Incorrect Behavior or Error
    2. Workflow Performance Issue
  1. JBoss Clustering Issues



 


  1.  Access Request Issues


Below are the artifacts required to troubleshoot access request issues:

  1. The aveksaServer.log in debug mode accompanied with the times the request was created or, in case of error, when the creation was initiated.
  2. The aveksaServerInfo.log
  3. A screenshot of the Requests/Approvals page showing the problem.  This includes any error messages and processing workflows.
  4. Explicit steps regarding the creation of the request.
  5. The workflow name and metadata associated with this request (Admin > Import/Export).  
    • Click on the Workflow tab.
    • Click the Export (all) button.
    • The WorkPoint.log and GeneralMonitor.log are under .../aveksa.war/log/.
  6. The run time data information from the processing workflow if the issue is with completion.
  7. Latest changes on the system prior to the issue (modification of workflow/form, etc.)
  8. For performance issues:
    • The ASR and AWR reports (see Notes).
    • The Oracle database alert log
    • Also, check for unusable indexes in the database

  9.  AFX Issues


Below are the artifacts required to troubleshoot AFX issues.  NOTE: Log file locations may be different depending on platform.  Please see the Notes section for more information on file locations. 

  1.  AFX Installation: For AFX installation issues, gather the following artifacts in order of importance:
    • The aveksaServerInfo.log 
    • For AFX Server installation issues provide the /tmp/afx-install.log
    • For AFX Server Standard Connectors installation issues, gather the following log: /opt/AFX/logs/esb.AFX-CTRL.log 
    •  For install AFX Workbench and Integration plug-ins for ACM installation issues provide console output/
  2. AFX to ACM ConnectionSuggested/required artifacts for connection issues between AFX and ACM are listed below in order of importance:
    • The aveksaServerInfo.log
    • Screenshot of AFX > Settings.
    • Screenshot of the AFX Connection Test popup
    • The /opt/AFX/mule-ee-3.2.1/logs/mule.AFX-CMD.log 
    • The /tmp/connection-test.log (if debug logging enabled)
    • (optional) /opt/AFX/mule-ee-3.2.1/logs/mule.AFX-CTRL.log
  3.  AFX Connector Setup: Suggested/required artifacts for AFX connector setup issues are listed below in order of importance:
  • aveksaServerInfo.log
  • Error message from UI, if applicable
  • /opt/AFX/esb/logs/esb.AFX-CTRL.log (connector setup/configuration)
  • /opt/AFX/esb/logs/esb.AFX-CMD.log (test requests to connector)
  • /opt/AFX/esb/logs/esb.AFX.log (responses from connector)
  • /opt/AFX/esb/logs/mule_ee.log (core AFX engine)
  1.  AFX CR not Auto-Fulfilled: Suggested/required artifacts for Change Request issues failing to automatically fulfill are listed below in order of importance:
  • aveksaServerInfo.log
  • Is AFX workflow linked to application?
  • Did Change Request (CR) use AFX workflow?
  • Did AFX fail and CR reserved to manual fulfillment?
  • Screenshot of manual fulfillment step from workflow (there will be a reason why this was not done automatically).
  • /opt/AFX/esb/logs/esb.AFX.log.
  • ./aveksa.war/log/aveksaServer.log.
  • ./aveksa.war/log/WorkPoint.log.
  • ./aveksa.war/log/GeneralMonitor.log.
  1. AFX Logging: Suggested/required artifacts for AFX logging issues are listed below in order of importance:
  • aveksaServerInfo.log
  • Install AFX Server: /tmp/afx-install.log
  • Install AFX Server Standard Connectors: /opt/AFX/logs/mule.AFX-CTRL.log
  • From the UI: AFX > Servers > select server > Logs tab.
  1.  Customize Capability: Suggested/required artifacts for AFX connector setup issues are listed below in order of importance:
  • aveksaServerInfo.log
  • (if applicable) Error message from UI
  • /opt/AFX/esb/logs/esb.AFX-CTRL.log (connector setup/configuration)
  • /opt/AFX/esb/logs/esb.AFX-CMD.log (test requests to connector)
  • /opt/AFX/esb/logs/esb.AFX.log (responses from connector)
  • /opt/AFX/esb/logs/mule_ee.log (core AFX engine)
  1. Bind App to Connector: Suggested/required artifacts for Bind App to connector issues are listed below in order of importance:
  • aveksaServerInfo.log
  • /opt/AFX/esb/logs/esb.AFX-CTRL.log
  • /opt/AFX/esb/logs/mule_ee.log
  • Is Connector Activated? Discovered? Enabled?
  1.  Connector Setup: Suggested/required artifacts for connector setup issues are listed below in order of importance:
  • aveksaServerInfo.log
  • Core AFX engine log: /opt/AFX/esb/logs/mule_ee.log
  • Connector setup log: /opt/AFX/esb/logs/esb.AFX-CTRL.log
  1.  Connector Usage: Suggested/required artifacts for connector usage issues are listed below in order of importance:
  • aveksaServerInfo.log
  • Core AFX engine log: /opt/AFX/esb/logs/mule_ee.log
  • Requests sent from AFX to ACM, and from AFX to Connectors: /opt/AFX/esb-ee-3.2.1/logs/esb.AFX-CMD.log 
  • Responses received from Connectors to AFX: /opt/AFX/esb-ee-3.2.1/logs/esb.AFX.log
  1.  UI: See How to turn on debug logging for a RSA Via Lifecycle and Governance Access Fulfillment Express (AFX) Connector 7.0, 6.9.1 and 6.8.1 for information about enabling DEBUG for AFX logs
  • aveksaServerInfo.log
  • Debug information specific to the connector in question: /tmp/[connector_name]_requests.log and /tmp/[connector_name]_responses.log
  • Debug information about the request id, that's returned in response: /tmp/f9eebf86-d82c-11e2-9fa7-dbda2ede0cea.log
  • Debug information about connection test between AFX and ACM: /tmp/connection-test.log

  1. Certificate Issues



    1.  

      Unexpected Behavior.  Suggested/required artifacts for unexpected certificate behavior issues are listed below in order of importance:


       
  • aveksaServerInfo.log.
  • Exact description of action being taken (ie, generate new certificate request, insert signed certificate, etc.) which is not working as expected.
  • Output of this Linux command from both the root and oracle user accounts. Parse thru the commands to ensure correct sequence: 
history | grep keytool

  • The certificates (host and any CA certificates) being used.
  • Read through documentation and certificate information in appendix A of the installation guide to understand process, ensure customer followed correct steps.
  1.  Certificate Errors: Suggested/required artifacts for unexpected certificate behavior issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • Exact description of action being taken (ie, generate new certificate request, insert signed certificate, etc) , which is resulting in an error.
  • Any error messages seen from the command line and/or in the aveksaServer.log.
  • Output of this Linux command from both the root and oracle user accounts: history | grep keytool, as above.
  • Screen shots of the certificate error noted in the browser, if accessing https results in an certificate error.
  • The certificates (host and any CA certificates) being used.
  • Read through documentation and certificate information in appendix A of the installation guide to understand process, ensure customer followed correct steps.

  1.  Data Collection Issues


    1. Collection Failure: Suggested/required artifacts for data collection failure issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • ASR Report.
  • aveksaServer.log (the entire log, for the day that the failure happened, not just a segment).
  • Screenshot of the job information for the failing run, from the Monitor Admin UI screen. This can be obtained from the GUI menu.  Admin >Monitoring > Details for Run<runID#> page,showing the run_ID, Task Results and all Task Progress steps for the collection.
  • Database logs for the failing run and for last successful run. These can be obtained either from the Database Logs for Run link using the Save Data button or outside of the UI, via SQL command, which makes use of the collection job run_ID. This SQL should be run as avuser: 
    SELECT * FROM t_av_job_stats WHERE av_run_id='<run_ID#>';

  • Export of the Admin Errors for the failing run.
  • Latest changes on the system prior to the failure: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
  • Collector Metadata (definition xml, from export).
  1. Collection Performance: Suggested/required artifacts for data collection performance issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • ASR Report.
  • AWR Report for the duration of the problematic run.
  • The Oracle database alert log.
  • aveksaServer.log (the entire log, for the day that the failure happened, not just a segment).
  • Screenshot of the job information for the failing run, from the Monitor Admin UI screen. This can be obtained from Aveksa Admin > Monitoring > Details for Run<runID#> page,showing the run_ID, Task Results and all Task Progress steps for the collection.
  • Database logs for the failing run and for last successful run. These can be obtained either from the Database Logs for Run link using the Save Data button or outside of the UI, via a SQL command, which makes use of the collection job run_ID. This SQL should be run as avuser: 
  • avuser: 
    SELECT * FROM t_av_job_stats WHERE av_run_id='<run_ID#>';

  • Latest changes on the system prior to the run: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
  • Size of data to be collected: number of rows for each query in the definition.
  • Information regarding latest scheduled jobs such as DB Statistics.
  • Explain plan for the longest running queries.
  • For deadlocks and blocking sessions see Notes section below.
  • Collector Metadata (definition xml, from export).
  1. Collection Results: Suggested/required artifacts for data collection results issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • ASR Report.
  • Collector Metadata (definition xml, from export).
  • Screenshot/Export of the job information for the problematic run: can be obtained from Aveksa Admin > Monitoring > Details for Run > Raw Data for: page, showing the problematic entries (All Collected, Rejected, etc.).
  • Database logs for the problematic run: can be obtained either from the Database Logs for Run link using the Save Data button or by, using the run_id, running the following SQL query as avuser:
    SELECT * FROM t_av_job_stats WHERE av_run_id='?'

  • Latest changes on the system prior to the problem: modification of collector, Oracle patches, Java updates, source data changes, etc.
  • Expected behavior vs observed behavior including screenshots and steps.

  1.  Form Issues  


Suggested/required artifacts for Form Issues issues are listed below in order of importance:


  • aveksaServerInfo.log
  • aveksaServer.log in debug mode accompanied with the times the Form was executed and/or the Request was created or in case of an error when the Request was initiated.
  • If the Form error can be reproduced using the Run Form or Debug Form button choosing a Test Application with actual data.
  • Export of the Request Form(s) and Request Button(s): Go to Admin > Import/Export > click Export and select the two items only.
  • Screenshot of the Requests/Approvals page showing the problem with the Form or the Form as executed in the workflow.
  • The Workflow name and metadata associated with this Form.
  • Explicit steps regarding the creation of the request where the Form is failing.
  • Latest changes on the system prior to the issue: modification of form/workflow, etc.
  • Export of the Workflows: Go to Admin > Import/Export > Workflow tab and click the Export (all) button.
  • The Workpoint logs (WorkPoint.log and GeneralMonitor.log) under: .../aveksa.war/log/
  • The run time data information from the processing workflow where the Form fails if the issue is with completion.
  • ASR Report.
  • AWR report If this is a workflow performance issue involving the Form.
  • Oracle database alert log.

  1.  Generic Performance Issues 


  • ASR Report.
  • AWR Report for a one hour time frame where degradation was especially poor.
  • The Oracle database alert log.
  • Check memory (physical memory, SGA/PGA settings, huge pages if SGA > 10GB)
$ free -m
$ cat /proc/meminfo
$ sqlplus / as sysdba
SQL> show parameter SGA;
SQL> show parameter PGA

  • If this is overall UI slowness in all parts of the application, enable acm profiling.

  1.  High CPU Issues


Suggested/required artifacts for High CPU issues are listed below in order of importance:

  • aveksaServerInfo.log
  • ASR Report.
  • AWR Report for the duration of the problematic time period.
  • The Oracle database alert log.
  • aveksaServer.log and all jboss logs (the entire log, for the time period)
  • JVM settings/options: 
jinfo <Process_ID>

  • Result of running top on the system during the period.
  • Heap Dump.
  • Latest changes on the system prior to observing the issue: modification of system settings, Oracle patches, Java updates, increase in use, increase in data sources and applications, etc.
  • Frequency of the issue observer.
  • Duration and peak levels (if the system is on and being used, 40%-50% is not considered high cpu issue): all VMS are designed to utilize the CPU to 100% for short periods.
  • Specific UI/system activities performed during this period.
  • Results of the v$resource_limit (from Oracle as sysdba).
  • Results of running:
vmstat


  1.  Install/Upgrade Issues


Artifacts for Install/Upgrade issues are listed below in order or importance:

  • aveksaServerInfo.log
  • Hardware configuration (OS partitions sizes, DB version and config)
  • Server/type of installation (JBoss/local DB, WebLogic Remote DB, etc.).
  • Check if there enough disk space:
df -kh

  • Output of terminal commands used for installation.
  • The install log: /tmp/aveksa-install.log.
  • The Oracle log: /tmp/aveksa/oracle.log.
  • Screenshot of the UI in case of migration error or post-migration error along with the aveksaServer.log and the migrate.log.
  • Hotfix log file.
  • Contents of the JBoss /home/oracle/jboss-4.2.2.GA/server/default/deploy/aveksa.ear/aveksa.war/log directory.
  • WebLogic: See Notes section below.
  • WebSpere: See Notes section below. 
  1. Oracle database issues
    • The Oracle database alert log (Note: the filename format is: alert_$ORACLE_SID.log) 
    • Any trace and/or Incident trace files, associated with the significant error.
  2. Report Issues
Suggested/required artifacts for Reports issues are listed below in order of importance:

  1. Report Performance Issue: Suggested/required artifacts for Report performance issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • Time period within which the Report ran.
  • AWR for the time period.
  • The Oracle database alert log
  • Time for report sql query to run directly against database.
  1. Report Does Not Provide Expected Information: Suggested/required artifacts for Report performance issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • Export of the Report.
  • Direct output of a query used in Report.
  • Report Result.
  • Specific information that is missing or not expected.

  1. Role Issues


    1.  Role Incorrect Behavior.  Suggested/required artifacts for Roles issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • Written steps to reproduce and screenshots as needed.
  • aveksaServer.log - the entire log, for the day that the failure happened, not just a segment.
  • Screenshot of the Role UI error - if there is one.
  • Export of Roles and Role Sets: Go to Roles > Roles > Actions > Export Roles. This will export both the Role Sets and Roles into a file called AveksaRoleData.xml.
  • ASR Report.
  • Latest changes on the system prior to the failure: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
  1.  Role Product/System Error: Suggested/required artifacts for Roles issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • Written steps to reproduce and screenshots as needed.
  • aveksaServer.log - the entire log, for the day that the failure happened, not just a segment.
  • Screenshot of the Role UI error,if there is one.
  • Export of Roles and Role Sets: Go to Roles > Roles > Actions > Export Roles.  This will export both the Role Sets and Roles into a file called AveksaRoleData.xml.
  • ASR Report.
  • Latest changes on the system prior to the failure: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.

  1. Role Performance Issue


    Suggested/required artifacts for Roles issues are listed below in order of importance:


  • aveksaServerInfo.log.
  • AWR Report for the time period during the Role action was executed and either completed or failed.
  • The Oracle database alert log
  • Written steps to reproduce and screenshots as needed.
  • aveksaServer.log - the entire log, for the day that the failure happened, not just a segment.
  • Screenshot of the Role UI error,  if there is one
  • Export of Roles and Role Sets: Go to Roles > Roles > Actions > Export Roles.  This will export both the Role Sets and Roles into a file called AveksaRoleData.xml.
  • ASR Report.
  • Latest changes on the system prior to the failure: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.

  1. Workflow Issues



  1.  Workflow Incorrect Behavior or Error.  Suggested/required artifacts for workflow failure issues are listed below in order of importance:


  • aveksaServerInfo.log .
  • Written steps to reproduce and screenshots as needed.
  • aveksaServer.log.
  • The workflow name.
  • Screenshot of the Workflow UI error, if there is one.
  • Export of the Workflows: Go to Admin > Import/Export > Workflow tab.  Click the Export (all) button.
  • In JBoss copy and provide: 
/home/oracle/jboss-4.2.2.GA/server/default/deploy/aveksa.ear/aveksa.war/log/WorkPoint.log
/home/oracle/jboss-4.2.2.GA/server/default/deploy/aveksa.ear/aveksa.war/log/GeneralMonitor.log

  • Latest changes on the system prior to the failure: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
  • ASR Report.
  1.  Workflow Performance Issue:Suggested/required artifacts for workflow failure issues are listed below in order of importance:
  • aveksaServerInfo.log.
  • AWR Report for the time period during the workflow was executed and either completed or failed.
  • The Oracle database alert log
  • Written steps to reproduce and screenshots as needed.
  • aveksaServer.log.
  • The workflow name.
  • Screenshot of the Workflow UI error, if there is one.
  • Export of the Workflows: Go to Admin > Import/Export > Workflow tab.  Click the Export (all) button.
  • In JBoss copy and provide: 
/home/oracle/jboss-4.2.2.GA/server/default/deploy/aveksa.ear/aveksa.war/log/WorkPoint.log
/home/oracle/jboss-4.2.2.GA/server/default/deploy/aveksa.ear/aveksa.war/log/GeneralMonitor.log

  • Latest changes on the system prior to the failure: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
  • ASR Report.

  1. JBoss Clustering Issues with RSA Identity Governance and Lifecycle 7.0


Collect these files from all servers that will be part of a cluster, including host controllers and domain controllers: 

  • /etc/init.d/aveksa_cluster
  • /home/oracle/wildfly/domain/configuration/host.xml
  • /home/oracle/wildfly/domain/log  (all files)
  • /home/oracle/wildfly/domain/servers/<server-name>/log    (all files) 
  • /home/oracle/wildfly/domain/servers/<servername>/configuration/aveksa-log4j.properties
  • From the domain controller: /home/oracle/wildfly/domain/configuration/domain.xml
  • From RHEL systems: /etc/sysconfig/iptables
  • From SUSE systems: /etc/sysconfig/SuSEfirewall2
NotesASR (Aveksa Statistics Report): This is an informative RSA Identity Governance and Lifecycle report, which provides pertinent application statistics and configuration information. It can be created by logging in as an admin user or as the AveksaAdmin user and navigating to Admin > System > Diagnostics > Create Statistics Report.
    AWR (Oracle Workload Repository): This is an informative Oracle Workload SQL report that can be created either from the command line or from Oracle OEM (Oracle Enterprise Manager) UI.  These should be created for ANY performance issue. 
    aveksaServer.log:  This log file contains information about all activity associated with the application.  Access to this log can be found by logging in as an admin user or as the AveksaAdmin user and navigating to the following path on any platform:  Admin > System > Server Nodes > aveksaServer.log OR from the command line on a 6.9.1 or older appliance by viewing the following file located at /home/oracle/jboss/server/default/deploy/aveksa.ear/aveksa.war/log/aveksaServer.log.
     


    NOTE: For WebSphere and WebLogic the aveksaServer.log file location is dependent on the installation path.  For example: 

    • For WebSphere, the file may be found in a directory similar to the following, where the specific node name would be different:  /home/oracle/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/vm-support-11Node01Cell/aveksa.ear/aveksa.war/log
    • For WebLogic, the file may be found in a directory similar to: .../../user_projects/domains/aveksaDomain/servers/AdminServer/tmp/_WL_user/aveksa/ldedze/aveksa.war/log/aveksaServer.log.
    • For WildFly (7.x) the file may be found in: /home/oracle/wildfly/standalone/log.
    For more information about file locations, please see article number: 000027894 Accessing the aveksaServer.log file for RSA Identity Lifecycle and Governance
    For help with how to find files in Linux, please see article 000032835 How to find particular files on Linux, Unix or POSIX operating systems.
    Oracle alert_AVDB.log
    To locate the Oracle database alert log, go to the Oracle Base directory, then issue the LINUX find command for the log.
    1. cd $ORACLE_BASE
    2. find -name alert_$ORACLE_SID.log -print
    Any trace files can be located using the same method, except that the specific filename from the Alert log should be used.
    Oracle - Detect blocking queries

    Use the SQL statements below to determine blocking queries (run as sysdba):


    1. Identify what SQL/Oracle User Session is blocked:

    SELECT s.sid, s.serial#, q.sql_text FROM v$session s, v$sql q 
    WHERE sid IN (SELECT sid FROM v$session WHERE STATE IN ('WAITING') 
    AND wait_class != 'Idle' AND event='enq: TX - row lock contention' 
    AND ( q.sql_id = s.sql_id OR q.sql_id = s.prev_sql_id));


    1. Identify what SQL/Oracle User Session is blocking:

    SELECT s1.username || '@' || s1.machine|| ' ( SID=' || s1.sid || ',' || s1.serial# || ' ) is blocking '|| s2.username || '@' || s2.machine || ' ( SID=' || s2.sid || ',' || s2.serial# || ' ) ' 
    AS blocking_status, q1.sql_text AS Blocking_SQL, q2.sql_text AS Blocked_SQL FROM v$lock l1, v$session s1, v$sql q1, v$lock l2, v$session s2, v$sql q2 WHERE s1.sid=l1.sid and s2.sid=l2.sid 
    AND l1.BLOCK=1 and l2.request > 0 AND l1.id1 = l2.id1 AND l2.id2 = l2.id2 AND ( q1.sql_id = s1.sql_id OR q1.sql_id = s1.prev_sql_id) AND ( q2.sql_id = s2.sql_id OR q2.sql_id = s2.prev_sql_id);


    Additional session related queries:
    SELECT * FROM v$fast_start_transactions;

    SELECT * FROM v$locked_object;

    SELECT * FROM v$session WHERE sid IN (SELECT session_id FROM v$locked_object);

    Attachments

      Outcomes