A list of artifacts that should be gathered based on the problem category. This should be used as a guide on what information is required to troubleshoot areas of concern.
This article outlines what files and information are required by RSA Identity Governance & Lifecycle product support in order to assist with various types of issues.
For common reports and file locations, including WildFly locations for 7.x, please see the Notes section of this article. For example, how to generate an ASR, an AWR report or find the aveksaServer.log file location.
A new feature called Log Artifact introduced in RSA Identity Governance & Lifecycle 7.1.1 facilitates collecting logs from the RSA Identity Governance & Lifecycle User Interface. For more details, see the blog New Feature: Log Artifact. Log files listed below in this article with a checkmark (√) are included in the Log Artifact .zip file downloadable from the User Interface.
Below are the artifacts that are required to troubleshoot access request issues:
The aveksaServer.log (√) in debug mode that is accompanied with the times the request was created or, in case of error, when the creation was initiated.
A screenshot of the Requests/Approvals page showing the problem. This includes any error messages and processing workflows.
Explicit steps regarding the creation of the request.
The workflow name and metadata associated with this request (Admin > Import/Export).
Click the Workflow tab.
Click the Export (all) button.
The WorkPoint.log (√) and wpqmonitor.log (√)are under .../home/oracle/wildfly/standalone/log/ or /home/oracle/wildfly/domain/servers/<'servername'>/log.
The run time data information from the processing workflow if the issue is with completion.
Latest changes on the system prior to the issue (modification of workflow/form, etc.)
For performance issues:
ASR (√) including the database performance statistics report (see Notes)
Below are the artifacts that are required to troubleshoot AFX issues. NOTE: Log file locations may be different depending on platform. Please see the Notes section for more information about file locations.
AFX Installation:For AFX installation issues, gather the following artifacts in order of importance:
AFX CR not Auto-Fulfilled: Suggested/required artifacts for Change Request issues failing to automatically fulfill are listed below in order of importance:
Exact description of the action being taken (ie, generate a new certificate request, insert signed certificate, etc.) which is not working as expected.
Output of this Linux command from both the root and oracle user accounts. Parse through the commands to ensure correct sequence:
history | grep keytool
The certificates (host and any CA certificates) being used.
Read through documentation and certificate information in appendix A of the installation guide to understand the process, ensure that customer followed the correct steps.
Certificate Errors: Suggested/required artifacts for unexpected certificate behavior issues are listed below in order of importance:
Exact description of the action being taken (ie, generate a new certificate request, insert signed certificate, and so on), which is resulting in an error.
Any error messages that are seen from the command line and/or in the aveksaServer.log (√).
Output of this Linux command from both the root and oracle user accounts: history | grep keytool, as above.
Screenshots of the certificate error noted in the browser if accessing https results in a certificate error.
The certificates (host and any CA certificates) being used.
Read through documentation and certificate information in appendix A of the installation guide to understand the process, ensure that customer followed the correct steps.
aveksaServer.log (√) (the entire log, for the day that the failure happened, not just a segment).
Screenshots or export of the Run Details page for the problematic run id. Go to Admin -> Monitoring -> click on the Run ID hyperlink. To export the full page of run details to a single file, right-click the page, choose "Save As", and save to a file. This will create an 'htm' file as well as a folder of the same name. Zip the folder and send both the htm file and zipped folder. Note: The htm file can work without the corresponding folder, but it is more readable with the folder and its contents.
Database logs for the failing run and for last successful run. These can be obtained either from the Database Logs for Run link using the Save Data button or outside of the UI, via SQL command, which makes use of the collection job run_ID. This SQL should be run as avuser:
SELECT * FROM t_av_job_stats WHERE av_run_id='<run_ID#>';
Export of the Admin Errors for the failing run
Latest changes on the system prior to the failure: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, so forth.
Collector Metadata (definition xml, from export)
Collection Performance:Suggested/required artifacts for data collection performance issues are listed below in order of importance:
ASR (√) including the database performance statistics report (see Notes)
AWR report (see Notes) for the duration of the problematic run
The Oracle database alert log (√)
aveksaServer.log (√) (the entire log, for the day that the failure happened, not just a segment).
Screenshots or export of the Run Details page for the problematic run id. Go to Admin -> Monitoring -> click on the Run ID hyperlink. To export the full page of run details to a single file, right-click the page, choose "Save As", and save to a file. This will create an 'htm' file as well as a folder of the same name. Zip the folder and send both the htm file and zipped folder. Note: The htm file can work without the corresponding folder, but it is more readable with the folder and its contents.
Database logs for the failing run and for last successful run. These can be obtained either from the Database Logs for Run link using the Save Data button or outside of the UI, via a SQL command, which makes use of the collection job run_ID. This SQL statement should be run as avuser:
SELECT * FROM t_av_job_stats WHERE av_run_id='<run_ID#>';
Latest changes on the system prior to the run: modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
Size of data to be collected: number of rows for each query in the definition
Information regarding latest scheduled jobs such as DB Statistics
Screenshots or export of the Run Details page for the problematic run id. Go to Admin -> Monitoring -> click on the Run ID hyperlink. To export the full page of run details to a single file, right-click the page, choose "Save As", and save to a file. This will create an 'htm' file as well as a folder of the same name. Zip the folder and send both the htm file and zipped folder. Note: The htm file can work without the corresponding folder, but it is more readable with the folder and its contents.
Database logs for the problematic run: can be obtained either from the Database Logs for Run link using the Save Data button or by, using the run_id, running the following SQL query as avuser:
SELECT * FROM t_av_job_stats WHERE av_run_id='?'
Latest changes on the system prior to the problem: Modification of collector, Oracle patches, Java updates, source data changes, etc.
Expected behavior vs observed behavior including screenshots and steps.
aveksaServer.log (√) in debug mode accompanied with the times the Form was run and/or the Request was created or in case of an error when the Request was initiated.
If the Form error can be reproduced using the Run Form or Debug Form button choosing a Test Application with actual data.
Export of the Request Form(s) and Request Button(s): Go to Admin > Import/Export > click Export and select the two items only.
Screenshot of the Requests/Approvals page showing the problem with the Form or the Form as run in the workflow.
The Workflow name and metadata associated with this Form.
Explicit steps regarding the creation of the request where the Form is failing.
Latest changes on the system prior to the issue: Modification of form/workflow, etc.
Export of the Workflows: Go to Admin > Import/Export > Workflow tab and click the Export (all) button
The Workpoint logs (WorkPoint.log (√) and wpqmonitor.log (√)) under: .../aveksa.war/log/
The run time data information from the processing workflow where the Form fails if the issue is with completion..
ASR report (√); include the database performance statistics report (see Notes) if this is a workflow performance issue involving the Form.
AWR report (see Notes) if this is a workflow performance issue involving the Form.
ASR (√) including the database performance statistics report (see Notes)
AWR report (see Notes) for the duration of the problematic time period
The Oracle database alert log
aveksaServer.log (√) and all jboss logs (the entire log, for the time period)
JVM settings/options:
jinfo <process_id>
Result of running top on the system during the period.
Heap dump
Latest changes on the system prior to observing the issue: Modification of system settings, Oracle patches, Java updates, increase in use, increase in data sources and applications, etc.
Frequency of the issue observer.
Duration and peak levels (if the system is on and being used, 40%-50% is not considered high cpu issue): All VMS are designed to utilize the CPU to 100% for short periods.
Specific UI/system activities performed during this period.
Results of the v$resource_limit (from Oracle as sysdba).
Results of running:
vmstat
To monitor system usage stats, use both v$osstat or v$sysmetric_history (from Oracle as sysdba)
Hardware configuration (OS partitions sizes, DB version and config)
Server/type of installation (JBoss/local DB, WebLogic Remote DB, etc.)
Check if there enough disk space:
df -kh
Output of terminal commands used for installation
The install log: /tmp/aveksa-install.log (√)
The patch.log (/home/oracle/wildfly-8.2.0.Final/standalone/log/patch.log or /home/oracle/wildfly/domain/servers/<'servername'>/log), for RSA Identity Governance & Lifecycle 7.0 and above, patching issues
The Oracle log: /tmp/aveksa/oracle.log (√)
Screenshot of the UI in case of migration error or post-migration error along with the aveksaServer.log (√) and the migrate log (/tmp/migrate2149357831832095389.log Note the actual filename for the migration log will use a different number for each installation but will be in the format /tmp/migrate*.log).
Written steps to reproduce and screenshots as needed.
aveksaServer.log (√) - the entire log, for the day that the failure happened, not just a segment.
Screenshot of the Role UI error - if there is one.
Export of Roles and Role Sets via Roles > Roles > Actions > Export Roles. This will export both, the Role Sets and Roles into a file called AveksaRoleData.xml
Latest changes on the system prior to the failure: Modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
Role Product/System Error: Suggested/required artifacts for Roles issues are listed below in order of importance:
Written steps to reproduce and screenshots as needed
aveksaServer.log (√) - the entire log, for the day that the failure happened, not just a segment
Screenshot of the Role UI error, if there is one.
Export of Roles and Role Sets (Roles > Roles > Actions > Export Roles). This will export both, the Role Sets and Roles into a file called AveksaRoleData.xml
Latest changes on the system prior to the failure: Modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
AWR report (see Notes) for the time period during the Role action was run and either completed or failed.
The Oracle database alert log (√)
Written steps to reproduce and screenshots as needed.
aveksaServer.log (√) - the entire log, for the day, that the failure happened, not just a segment.
Screenshot of the Role UI error, if there is one.
Export of Roles and Role Sets: Go to Roles > Roles > Actions > Export Roles. This will export both, the Role Sets and Roles into a file called AveksaRoleData.xml.
ASR (√) including the database performance statistics report (see Notes)
Latest changes on the system prior to the failure: Modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
Latest changes on the system prior to the failure: Modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
Latest changes on the system prior to the failure: Modification of collector, Oracle patches, Java updates, source data changes, network changes, collector driver updates, etc.
ASR (√) including the database performance statistics report (see Notes)
From the domain controller: /home/oracle/wildfly/domain/configuration/domain.xml
From RHEL systems: /etc/sysconfig/iptables
From SUSE systems: /etc/sysconfig/SuSEfirewall2
ASR (Aveksa Statistics Report): This is an informative RSA Identity Governance & Lifecycle report, which provides pertinent application statistics and configuration information. It can be created by logging in as an admin user or as the AveksaAdmin user and going to Admin > System > Diagnostics > Create Statistics Report. Customers using an RSA-provided database for RSA Identity Governance & Lifecycle must generate a database performance statistics report for troubleshooting database performance issues. See the following on how to include database performance statistics report in the ASR: How to generate Oracle database performance statistics in RSA Identity Governance & Lifecycle
AWR (Oracle Workload Repository): This is an informative Oracle Workload SQL report that can be created either from the command line or from Oracle Enterprise Manager (OEM) UI. These should be created for ANY performance issue. Generating an AWR report requires licensing to use the Oracle Diagnostics Pack.
aveksaServer.log: This log file contains information about all activities that are associated with the application. Access to this log can be found by logging in as an admin user or as the AveksaAdmin user and going to the following path on any platform:
Admin > System > Server Nodes > aveksaServer.log, or
From the command line on a 6.9.1 or older appliance by viewing the following file located at /home/oracle/jboss/server/default/deploy/aveksa.ear/aveksa.war/log/aveksaServer.log. 7.0.0+ /home/oracle/wildfly-8.2.0.Final/standalone/log, 7.0.0+ (Cluster) /home/oracle/wildfly-8.2.0.Final/domain
NOTE: For WebSphere and WebLogic the aveksaServer.log file location is dependent on the installation path. For example:
For WebSphere, the file may be found in a directory similar to the following, (where the specific node name would be different) /home/oracle/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/vm-support-11Node01Cell/aveksa.ear/aveksa.war/log. Other files to gather are SystemOut.log and SystemErr.log.
For WebLogic, the file may be found in a directory similar to .../../user_projects/domains/aveksaDomain/servers/AdminServer/tmp/_WL_user/aveksa/ldedze/aveksa.war/log/aveksaServer.log. Other files to gather are stdout.log, stderr.log, and, server.log.
For WildFly (7.x) the file may be found in /home/oracle/wildfly/standalone/log.
Use the SQL statements below to determine blocking queries. These must be run as sysdba:
Identify what SQL/Oracle User Session is blocked:
SELECT s.sid, s.serial#, q.sql_text FROM v$session s, v$sql q WHERE sid IN (SELECT sid FROM v$session WHERE STATE IN ('WAITING') AND wait_class != 'Idle' AND event='enq: TX - row lock contention' AND ( q.sql_id = s.sql_id OR q.sql_id = s.prev_sql_id));
Identify what SQL/Oracle User Session is blocking:
SELECT s1.username || '@' || s1.machine|| ' ( SID=' || s1.sid || ',' || s1.serial# || ' ) is blocking '|| s2.username || '@' || s2.machine || ' ( SID=' || s2.sid || ',' || s2.serial# || ' ) ' AS blocking_status, q1.sql_text AS Blocking_SQL, q2.sql_text AS Blocked_SQL FROM v$lock l1, v$session s1, v$sql q1, v$lock l2, v$session s2, v$sql q2 WHERE s1.sid=l1.sid AND s2.sid=l2.sid AND l1.BLOCK=1 and l2.request > 0 AND l1.id1 = l2.id1 AND l2.id2 = l2.id2 AND ( q1.sql_id = s1.sql_id OR q1.sql_id = s1.prev_sql_id) AND ( q2.sql_id = s2.sql_id OR q2.sql_id = s2.prev_sql_id);
Additional session-related queries:
SELECT * FROM v$fast_start_transactions; SELECT * FROM v$locked_object; SELECT * FROM v$session WHERE sid IN (SELECT session_id FROM v$locked_object);
