What tools are available for diagnosing SOA Suite issues?
There are a variety
of tools available to help you and Support diagnose SOA Suite issues in 11g but it can be confusing as to which tool is appropriate for a particular situation and what their
relationships are. This blog post will introduce the various tools and attempt to clarify what each is for and how they are related. Let's first list the tools we'll be addressing:
RDA: Remote Diagnostic Agent
DFW: Diagnostic Framework
Selective Tracing
DMS: Dynamic Monitoring Service
ODL: Oracle Diagnostic Logging
ADR: Automatic Diagnostics Repository
ADRCI: Automatic Diagnostics Repository Command Interpreter
WLDF: WebLogic Diagnostic Framework
This overview is not mean to be a comprehensive guide on using all
of these tools, however, extensive reference materials are included that will provide many more details on their execution. Another
point to note is that all
of these tools are applicable for Fusion Middleware as a whole but specific products may or may not have implemented features to leverage them.
A couple
of the tools have a WebLogic Scripting Tool or 'WLST' interface. WLST is a command interface for executing pre-built functions and custom scripts against a domain. A detailed
WLST tutorial is beyond the scope
of this post but you can find general information here.
There are more specific resources in the below sections.
In this post when we refer to 'Enterprise Manager' or 'EM' we are referring to Enterprise Manager Fusion Middleware Control.
RDA (Remote Diagnostic Agent)
RDA is a standalone tool that is used to collect both static configuration and dynamic runtime information from the SOA environment. RDA is generally
run manually from the command line against a domain or single server. When opening a new Service Request, including an RDA collection can dramatically decrease the back and forth required to
collect logs and configuration information for Support.
After installing RDA you configure it to use the SOA Suite module as decribed in the referenced resources. The SOA module includes the Oracle WebLogic Server (WLS) module by default in order to include all
of the relevant
information for the environment. In addition to this basic configuration there is also an advanced mode where you can set the number
of thread dumps for the collections, log files, Incidents, etc.
When would you use it?
When creating a Service Request or otherwise working with Oracle resources on an issue, capturing environment snapshots to baseline your configuration or to diagnose an issue on your own.
How is it related to the other tools?
RDA is related to DFW in that it collects the last 10 Incidents from the server by default. In a similar manner, RDA is related to ODL through its collection
of the diagnostic logs and these may
contain information from Selective Tracing sessions.
Examples
of what it currently collects: (for details please see the links in the Resources section)
Diagnostic Logs (ODL)
Diagnostic Framework Incidents (DFW)
SOA MDS Deployment Descriptors
SOA Repository Summary Statistics
Thread Dumps
Complete Domain Configuration
RDA Resources:
Webcast Recording: Using RDA with Oracle SOA Suite 11g
Blog Post: Diagnose SOA Suite 11g Issues Using RDA
Download RDA
How to Collect Analysis Information Using RDA for Oracle SOA Suite 11g
Products [ID 1350313.1]
How to Collect Analysis Information Using RDA for Oracle SOA Suite and
BPEL Process Manager 11g [ID 1352181.1]
Getting Started With Remote Diagnostic Agent: Case Study - Oracle
WebLogic Server (Video) [ID 1262157.1]
top
DFW (Diagnostic Framework)
DFW provides the ability to collect specific information for a particular problem when that problem occurs. DFW is included with your SOA Suite installation and deployed to the domain.
Let's define the components
of DFW.
Diagnostic Dumps: Specific diagnostic collections that are defined at either the 'system' or product level. Examples would be diagnostic logs or thread dumps.
Incident: A collection
of Diagnostic Dumps associated with a particular problem
Log Conditions: An Oracle Diagnostic Logging event that DFW is configured to listen for. If the event is identified then an Incident will be created.
WLDF Watch: The WebLogic Diagnostic Framework or 'WLDF' is not a component
of DFW, however, it can be a source
of DFW Incident creation through the use
of a 'Watch'.
WLDF Notification: A Notification is a component
of WLDF and is the link between the Watch and DFW. You can configure multiple Notification types in WLDF and associate them
with your Watches. 'FMWDFW-notification' is available to you out
of the box to allow for DFW notification
of Watch execution.
Rule: Defines a WLDF Watch or Log Condition for which we want to associate a set
of Diagnostic Dumps. When triggered the specified dumps will be collected and added to the
Incident
Rule Action: Defines the specific Diagnostic Dumps to collect for a particular rule
ADR: Automatic Diagnostics Repository; Defined for every server in a domain. This is where Incidents are stored
Now let's walk through a simple flow:
Oracle Web Services error message OWS-04086 (SOAP Fault) is generated on managed server 1
DFW Log Condition for OWS-04086 evaluates to TRUE
DFW creates a new Incident in the ADR for managed server 1
DFW executes the specified Diagnostic Dumps and adds the output to the Incident
In this case we'll grab the diagnostic log and thread dump. We might also want to collect the WSDL binding information and SOA audit trail
When would you use it?
When you want to automatically collect Diagnostic Dumps at a particular time using a trigger or when you want to manually collect the information. In either case it can be readily uploaded to
Oracle Support through the Service Request.
How is it related to the other tools?
DFW generates Incidents which are collections
of Diagnostic Dumps. One
of the system level Diagonstic Dumps collects the current server diagnostic log which is generated by ODL and can contain
information from Selective Tracing sessions. Incidents are included in RDA collections by default and ADRCI is a tool that is used to package an Incident for upload to Oracle Support. In
addition, both ODL and DMS can be used to trigger Incident creation through DFW.
The conditions and rules for generating Incidents can become quite complicated and the below resources go into more detail. A simpler approach to leveraging at least the Diagnostic Dumps is through
WLST (WebLogic Scripting Tool) where there are commands to do the following:
Create an Incident
Execute a single Diagnostic Dump
Describe a Diagnostic Dump
List the available Diagnostic Dumps
The WLST option offers greater control in what is generated and when. It can be a great help when collecting information for Support. There are overlaps with RDA, however, DFW is geared
towards collecting specific runtime information when an issue occurs while existing Incidents are collected by RDA.
There are 3 WLDF Watches configured by default in a SOA Suite 11g domain: Stuck Threads, Unchecked Exception and Deadlock.
These Watches are enabled by default and will generate Incidents in ADR. They are configured to reset automatically after 30 seconds so they have the potential to create multiple Incidents if these
conditions are consistent. The Incidents generated by these Watches will only contain System level Diagnostic Dumps. These same System level Diagnostic Dumps will be included in any application
scoped Incident as well.
Starting in 11.1.1.6, SOA Suite is including its own set
of application scoped Diagnostic Dumps that can be executed from WLST or through a WLDF Watch or Log Condition. These Diagnostic Dumps can be
added to an Incident such as in the earlier example using the error code OWS-04086.
soa.config: MDS configuration files and deployed-composites.xml
soa.composite: All artifacts related to the deployed composite
soa.wsdl: Summary
of endpoints configured for the composite
soa.edn: EDN configuration summary if applicable
soa.db: Summary DB information for the SOA repository
soa.env: Coherence cluster configuration summary
soa.composite.trail: Partial audit trail information for the running composite
The current release
of RDA has the option to collect the soa.wsdl and soa.composite Diagnostic Dumps. More Diagnostic Dumps for SOA Suite products are planned for future
releases along with enhancements to DFW itself.
DFW Resources:
Webcast Recording: SOA Diagnostics Sessions: Diagnostic Framework
Diagnostic Framework Documentation
DFW WLST Command Reference
Documentation for SOA Diagnostic Dumps in 11.1.1.6
top
Selective Tracing
Selective Tracing is a facility available starting in version 11.1.1.4 that allows you to increase the logging level for specific loggers and for a specific context. What this means is that you have
greater capability to collect needed diagnostic log information in a production environment with reduced overhead. For example, a Selective Tracing session can be executed that only increases
the log level for one composite, only one logger, limited to one server in the cluster and for a preset period
of time. In an environment where dozens
of composites are deployed this can dramatically
reduce the volume and overhead
of the logging without sacrificing relevance.
Selective Tracing can be administered either from Enterprise Manager or through WLST. WLST provides a bit more flexibility in terms
of exactly where the tracing is run.
When would you use it?
When there is an issue in production or another environment that lends itself to filtering by an available context criteria and increasing the log level globally results in too much overhead or
irrelevant information. The information is written to the server diagnostic log and is exportable from Enterprise Manager
How is it related to the other tools?
Selective Tracing output is written to the server diagnostic log. This log can be collected by a system level Diagnostic Dump using DFW or through a default RDA collection. Selective Tracing also
heavily leverages ODL fields to determine what to trace and to tag information that is part
of a particular tracing session.
Available Context Criteria:
Application Name
Client Address
Client Host
Composite Name
User Name
Web Service Name
Web Service Port
Selective Tracing Resources:
Webcast Recording: SOA Diagnostics Session: Using Selective Tracing to Diagnose SOA Suite Issues
How to Use Selective Tracing for SOA [ID 1367174.1]
Selective Tracing WLST Reference
top
DMS (Dynamic Monitoring Service)
DMS exposes runtime information for monitoring. This information can be monitored in two ways:
Through the DMS servlet
As exposed MBeans
The servlet is deployed by default and can be accessed through http://<host>:<port>/dms/Spy (use administrative credentials to access). The landing page
of the servlet shows identical columns
of what
are known as Noun Types. If you select a Noun Type you will see a table in the right frame that shows the attributes (Sensors) for the Noun Type and the available instances. SOA Suite has several exposed Noun
Types that are available for viewing through the Spy servlet. Screenshots
of the Spy servlet are available in the Knowledge Base article
How to Monitor Runtime SOA Performance With the Dynamic Monitoring Service (DMS).
Every Noun instance in the runtime is exposed as an MBean instance. As such they are generally available through an MBean browser and available for monitoring through WLDF. You can configure a
WLDF Watch to monitor a particular attribute and
fire a notification when the threshold is exceeded. A WLDF Watch can use the out
of the box DFW notification type to notify DFW to create an Incident.
When would you use it?
When you want to monitor a metric or set
of metrics either manually or through an automated system. When you want to trigger a WLDF Watch based on a metric exposed through DMS.
How is it related to the other tools?
DMS metrics can be monitored with WLDF Watches which can in turn notify DFW to create an Incident.
DMS Resources:
How to Monitor Runtime SOA Performance With the Dynamic Monitoring Service (DMS) [ID 1368291.1]
How to Reset a SOA 11g DMS Metric
DMS Documentation
top
ODL (Oracle Diagnostic Logging)
ODL is the primary facility for most Fusion Middleware applications to log what they are doing. Whenever you change a logging level through Enterprise Manager it is ultimately exposed through ODL and
written to the server diagnostic log. A notable exception to this is WebLogic Server which uses its own log format / file.
ODL logs entries in a consistent, structured way using predefined fields and name/value pairs. Here's an example
of a SOA Suite entry:
[2012-04-25T12:49:28.083-06:00] [AdminServer] [ERROR] [] [oracle.soa.bpel.engine] [tid: [ACTIVE].ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 0963fdde7e77631c:-31a6431d:136eaa46cda:-8000-00000000000000b4,0] [errid: 41] [WEBSERVICE_PORT.name: BPELProcess2_pt] [APP: soa-infra] [composite_name: TestProject2] [J2EE_MODULE.name: fabric] [WEBSERVICE.name: bpelprocess1_client_ep] [J2EE_APP.name: soa-infra] Error occured while handling a post operation[[
When would you use it?
You'll use ODL almost every time you want to identify and diagnose a problem in the environment. The entries are written to the server diagnostic log.
How is it related to the other tools?
The server diagnostic logs are collected by DFW and RDA. Selective Tracing writes its information to the diagnostic log as well.
Additionally, DFW log conditions are triggered by ODL log events.
ODL Resources:
ODL Documentation
top
ADR (Automatic Diagnostics Repository)
ADR is not a tool in and
of itself but is where DFW stores the Incidents it creates. Every server in the domain has an ADR location which can be found under <SERVER_HOME>/adr. This is referred to the as the ADR 'Base' location.
ADR also has what are known as 'Home' locations. Example:
You have a domain called 'myDomain' and an associated managed server called 'myServer'. Your admin server is called 'AdminServer'.
Your domain home directory is called 'myDomain' and it contains a 'servers' directory.
The 'servers' directory contains a directory for the managed server called 'myServer' and here is where you'll find the 'adr' directory which is the ADR 'Base' location for myServer.
To get to the ADR 'Home' locations we drill through a few levels: diag/ofm/myDomain/
In an 11.1.1.6 SOA Suite domain you will see 2 directories here, 'myServer' and 'soa-infra'. These are the ADR 'Home' locations.
'myServer' is the 'system' ADR home and contains system level Incidents.
'soa-infra' is the name that SOA Suite used to register with DFW and this ADR home contains SOA Suite related Incidents
Each ADR home location contains a series
of directories, one
of which is called 'incident'. This is where your Incidents are stored.
When would you use it?
It's a good idea to check on these locations from time to time to see whether a lot
of Incidents are being generated. They can be cleaned out by deleting the Incident directories or through the ADRCI
tool. If you know that an Incident is
of particular interest for an issue you're working with Oracle you can simply zip it up and provide it.
How does it relate to the other tools?
ADR is obviously very important for DFW since it's where the Incidents are stored. Incidents contain Diagnostic Dumps that may relate to diagnostic logs (ODL) and DMS metrics.
The most recent 10 Incident directories are collected by RDA by default and ADRCI relies on the ADR locations to help manage the contents.
top
ADRCI (Automatic Diagnostics Repository Command Interpreter)
ADRCI is a command line tool for packaging and managing Incidents.
When would you use it?
When purging Incidents from an ADR Home location or when you want to package an Incident along with an offline RDA collection for upload to Oracle Support.
How does it relate to the other tools?
ADRCI contains a tool called the Incident Packaging System or IPS. This is used to package an Incident for upload to Oracle Support through a Service Request. Starting in 11.1.1.6 IPS will attempt
to collect an offline RDA collection and include it with the Incident package. This will only work if Perl is available on the path, otherwise it will give a warning and package only the Incident
files.
ADRCI Resources:
How to Use the Incident Packaging System (IPS) in SOA 11g [ID 1381259.1]
ADRCI Documentation
top
WLDF (WebLogic Diagnostic Framework)
WLDF is functionality available in WebLogic Server since version 9. Starting with FMw 11g a link has been added between WLDF and the pre-existing DFW, the WLDF Watch Notification. Let's take a closer look
at the flow:
There is a need to monitor the performance
of your SOA Suite message processing
A WLDF Watch is created in the WLS console that will trigger if the average message processing time exceeds 2 seconds. This metric is monitored through a DMS MBean instance.
The out
of the box DFW Notification (the Notification is called FMWDFW-notification) is added to the Watch. Under the covers this notification is
of type JMX.
The Watch is triggered when the threshold is exceeded and fires the Notification.
DFW has a listener that picks up the Notification and evaluates it according to its rules, etc
When it comes to automatic Incident creation, WLDF is a key component with capabilities that will grow over time.
When would you use it?
When you want to monitor the WLS server log or an MBean metric for some condition and
fire a notification when the Watch is triggered.
How does it relate to the other tools?
WLDF is used to automatically trigger Incident creation through DFW using the DFW Notification.
WLDF Resources:
How to Monitor Runtime SOA Performance With the Dynamic Monitoring Service (DMS) [ID 1368291.1]
How To Script the Creation
of a SOA WLDF Watch in 11g [ID 1377986.1]
WLDF Documentation
top
