Master Note for Troubleshooting Advanced Queuing and Oracle Streams Propagation Issues (Doc ID 233099.1)

Copyright (c) 2010, Oracle Corporation. All Rights Reserved.

In this Document
  Purpose
  Last Review Date
  Instructions for the Reader
  Troubleshooting Details
     1. Scope and Application
      2. Definitions and Classifications
     3. How to Use This Guide
     4. Basic AQ Propagation Troubleshooting
     5. Additional Troubleshooting Steps for AQ Propagation of User-Enqueued and Dequeued Messages
     6. Additional Troubleshooting Steps for Propagation in an Oracle Streams Environment
     7. Performance Issues
  References

Applies to:

Oracle Server - Enterprise Edition - Version: 8.1.7.0 to 11.2.0.2 - Release: 8.1.7 to 11.2
Information in this document applies to any platform.

Purpose

This document presents a step-by-step methodology for troubleshooting and resolving problems with Advanced Queuing Propagation in both Streams and basic Advanced Queuing environments. It also serves as a master reference for other more specific notes on Oracle Streams Propagation and Advanced Queuing Propagation issues.

Last Review Date

December 20, 2010

Instructions for the Reader

Troubleshooting Details

1. Scope and Application

This note is intended for Database Administrators of Oracle databases where issues are being encountered with propagating messages between advanced queues, whether the queues are used for user-created messaging systems or for Oracle Streams. It contains troubleshooting steps and links to notes for further problem resolution.

It can also be used a template to document a problem when it is necessary to engage Oracle Support Services. Knowing what is NOT happening can frequently speed up the resolution process by focusing solely on the pertinent problem area.

This guide is divided into five parts:

• Section 2: Definitions and Classifications (discusses the different types and features of propagations possible - helpful for understanding the rest of the guide)
• Section 3: How to Use this Guide (to be used as a start part for determining the scope of the problem and what sections to consult)
• Section 4. Basic AQ propagation troubleshooting (applies to both AQ propagation of user enqueued and dequeued messages as well as Oracle Streams propagations)
• Section 5. Additional troubleshooting steps for AQ propagation of user enqueued and dequeued messages
• Section 6. Additional troubleshooting steps for Oracle Streams propagation
• Section 7. Performance issues
  

2. Definitions and Classifications

Given the potential scope of issues that can be encountered with AQ propagation, the first recommended step is to do some basic diagnosis to determine the type of problem that is being encountered.

2.1. What Type of Propagation is Being Used?

2.1.1. Buffered Messaging

For an advanced queue, messages can be maintained on disk (persistent messaging) or in memory (buffered messaging). To determine if a queue is buffered or not, reference the GV_$BUFFERED_QUEUES view. If the queue does not appear in this view, it is persistent.

2.1.2. Propagation mode - queue-to-dblink vs queue-to-queue

As of 10.2, an AQ propagation can also be defined as queue-to-dblink, or queue-to-queue:

• queue-to-dblink: The propagation delivers messages or events from the source queue to all subscribing queues at the destination database identified by the dblink. A single propagation schedule is used to propagate messages to all subscribing queues. Hence any changes made to this schedule will affect message delivery to all the subscribing queues. This mode does not support multiple propagations from the same source queue to the same target database.
• queue-to-queue: Added in 10.2, this propagation mode delivers messages or events from the source queue to a specific destination queue identified on the database link. This allows the user to have fine-grained control on the propagation schedule for message delivery. This new propagation mode also supports transparent failover when propagating to a destination Oracle RAC system. With queue-to-queue propagation, you are no longer required to re-point a database link if the owner instance of the queue fails on Oracle RAC. This mode supports multiple propagations to the same target database if the target queues are different.

The default is queue-to-dblink. To verify if queue-to-queue propagation is being used, in non-Streams environments query DBA_QUEUE_SCHEDULES.DESTINATION - if a remote queue is listed along with the remote database link, then queue-to-queue propagation is being used. For Streams environments, the DBA_PROPAGATION.QUEUE_TO_QUEUE column can be checked.

See the following note for a method to switch between the two modes:

Document 827473.1 How to alter propagation from queue-to-queue to queue-to-dblink

2.1.3. Combined Capture and Apply (CCA) for Streams

In 11g Oracle Streams environments, an optimization called Combined Capture and Apply (CCA) is implemented by default when possible. Although a propagation is configured in this case, Streams does not use it; instead it passes information directly from capture to an apply receiver. To see if CCA is in use:

Also, see the following note:

Document 463820.1 Streams Combined Capture and Apply in 11g

2.2. Queue Table Compatibility

There are three types of queue table compatibility. In more recent databases, queue tables may be present in all three modes of compatibility:

• 8.0 - earliest version, deprecated in 10.2 onwards
• 8.1 - support added for RAC, asynchronous notification, secure queues, queue level access control, rule-based subscribers, separate storage of history information
• 10.0 - if the database is in 10.1-compatible mode, then the default value for queue table compatibility is 10.0

2.3. Single vs Multiple Consumer Queue Tables

If more than one recipient can dequeue a message from a queue, then its queue table is multiple consumer. You can propagate messages from a multiple-consumer queue to a single-consumer queue. Propagation from a single-consumer queue to a multiple-consumer queue is not possible.

3. How to Use This Guide

3.1. Are Messages Being Propagated at All, or is the Propagation Just Slow?

Run the following query on the source database for the propagation (assuming that it is running):

If TOTAL_NUMBER is increasing, then propagation is most likely functioning, although it may be slow. For performance issues, see Section 7.

3.2. Propagation Between Persistent User-Created Queues

See Sections 4 and 5 (and optionally Section 6 if performance is an issue).

3.3. Propagation Between Buffered User-Created Queues

See Sections 4, 5, and 6 (and optionally Section 7 if performance is an issue).

3.4. Propagation between Oracle Streams Queues (without Combined Capture and Apply (CCA) Optimization)

See Sections 4 and 6 (and optionally Section 7 if performance is an issue).

3.5. Propagation between Oracle Streams Queues (with Combined Capture and Apply (CCA) Optimization)

Although an AQ propagation is not used directly in this case, some characteristics of the message transfer are inferred from the propagation parameters used. Some parts of Sections 4 and 6 still apply.

3.6. Messaging Gateway Propagations

This note does not apply to Messaging Gateway propagations.

4. Basic AQ Propagation Troubleshooting

4.1. Double-check Your Code

Make sure that you are consistent in your usage of the database link(s) names, queue names, etc. It may be useful to plot a diagram of which queues are connected via which database links to make sure that the logical structure is correct.

4.2. Verify that Job Queue Processes are Running

4.2.1. Versions 10.2 and Lower - DBA_JOBS Package

For versions 10.2 and lower, a scheduled propagation is managed by DBMS_JOB package. The propagation is performed by job queue process background processes. Therefore we need to verify that there are sufficient processes available for the propagation process. We should have at least 4 job queue processes running and preferably more depending on the number of other jobs running in the database. It should be noted that for AQ specific work, AQ will only ever use half of the job queue processes available.

An issue caused by an inadequate job queue processes parameter setting is described in the following note:

Document 298015.1 Kwqjswproc:Excep After Loop: Assigning To Self

4.2.1.1. Job Queue Processes in Initalization Parameter File

The parameter JOB_QUEUE_PROCESSES in the init.ora/spfile should be > 0. The value can be changed dynamically via

4.2.1.2. Job Queue Processes in Memory

The following command will show how many job queue processes are currently
in use by this instance (this may be different than what is in the init.ora/spfile):

4.2.1.3. OS PIDs Corresponding to Job Queue Processes

Identify the operating system process ids (spids) of job queue processes involved in propagation via

and these SPIDs can be used to check at the operating system level that they exist.

In 8i a job queue process will have a name similar to: ora_snp1_<instance_name>.

In 9i onwards you will see a coordinator process: ora_cjq0_ and multiple slave processes: ora_jnnn_<instance_name>, where nnn is an integer between 1 and 999.

4.2.2. Version 11.1 and Above - Oracle Scheduler

In version 11.1 and above, Oracle Scheduler is used to perform AQ and Streams propagations. Oracle Scheduler automatically tunes the number of slave processes for these jobs based on the load on the computer system, and the JOB_QUEUE_PROCESSES initialization parameter is only used to specify the maximum number of slave processes. Therefore, the JOB_QUEUE_PROCESSES initialization parameter does not need to be set (it defaults to a very high number), unless you want to limit the number of slaves that can be created. If JOB_QUEUE_PROCESSES = 0, no propagation jobs will run.

See the following note for a discussion of Oracle Streams 11g and Oracle Scheduler:

Document 1083608.1 11g Streams and Oracle Scheduler

4.2.2.1. Job Queue Processes in Initalization Parameter File

The parameter JOB_QUEUE_PROCESSES in the init.ora/spfile should be > 0, and preferably be left at its default value. The value can be changed dynamically via

To set the JOB_QUEUE_PROCESSES parameter to its default value, run:

and then bounce the instance.

4.2.2.2. Job Queue Processes in Memory

The following command will show how many job queue processes are currently in use by this instance (this may be different than what is in the init.ora/spfile):

4.2.2.3. OS PIDs Corresponding to Job Queue Processes

Identify the operating system process ids (SPIDs) of job queue processes involved in propagation via

and these SPIDs can be used to check at the operating system level that they exist.

You will see a coordinator process: ora_cjq0_ and multiple slave processes: ora_jnnn_<instance_name>, where nnn is an integer between 1 and 999.

4.3. Check the Alert Log and Any Associated Trace Files

The first place to check for propagation failures is the alert logs at all sites (local and if relevant all remote sites). When a job queue process attempts to execute a schedule and fails it will always write an error stack to the alert log. This error stack will also be written in a job queue process trace file, which will be written to the BACKGROUND_DUMP_DEST location for 10.2 and below, and in the DIAGNOSTIC_DEST location for 11g.

The fact that errors are written to the alert log demonstrates that the schedule is executing. This means that the problem could be with the set up of the schedule.

In this example the ORA-02068 demonstrates that the failure was at the remote site. Further investigation revealed that the remote database was not open, hence the ORA-03114 error.

Starting the database resolved the problem.

Other potential errors that may be written to the alert log can be found in the following notes:

Document 827184.1 AQ Propagation with CLOB data types Fails with ORA-22990 (11.1)
Document 846297.1 AQ Propagation Fails : ORA-00600[kope2upic2954] or Ora-00600[Kghsstream_copyn] (10.2, 11.1)
Document 731292.1 ORA-25215 Reported on Local Propagation When Using Transformation with ANYDATA queue tables (10.2, 11.1, 11.2)
Document 365093.1 ORA-07445 [kwqppay2aqe()+7360] Reported on Propagation of a Transformed Message (10.1, 10.2)
Document 219416.1 Advanced Queuing Propagation Fails with ORA-22922 (9.0)
Document 1203544.1 AQ Propagation Aborted with ORA-600 [ociksin: invalid status] on SYS.DBMS_AQADM_SYS.AQ$_PROPAGATION_PROCEDURE After Upgrade (11.1, 11.2)
Document 1087324.1 ORA-01405 ORA-01422 reported by Advanced Queuing Propagation schedules after RAC reconfiguration (10.2)
Document 1079577.1 Advanced Queuing Propagation Fails With "ORA-22370 incorrect usage of method" (9.2, 10.2, 11.1, 11.2)
Document 332792.1 ORA-04061 error relating to SYS.DBMS_PRVTAQIP reported when setting up Statspack (8.1, 9.0, 9.2, 10.1)
Document 353325.1 ORA-24056: Internal inconsistency for QUEUE <queue_name> and destination <dblink> (8.1, 9.0, 9.2, 10.1, 10.2, 11.1, 11.2)
Document 787367.1 ORA-22275 reported on Propagating Messages with LOB component when propagating between 10.1 and 10.2 (10.1, 10.2)
Document 566622.1 ORA-22275 when propagating >4K AQ$_JMS_TEXT_MESSAGEs from 9.2.0.8 to 10.2.0.1 (9.2, 10.1)
Document 731539.1 ORA-29268: HTTP client error 401 Unauthorized Error when the AQ Servlet attempts to Propagate a message via HTTP (9.0, 9.2, 10.1, 10.2, 11.1)
Document 253131.1 Concurrent Writes May Corrupt LOB Segment When Using Auto Segment Space Management (ORA-1555) (9.2)
Document 118884.1 How to unschedule a propagation schedule stuck in pending state
Document 222992.1 DBMS_AQADM.DISABLE_PROPAGATION_SCHEDULE Returns ORA-24082
Document 282987.1 Propagated Messages marked UNDELIVERABLE after Drop and Recreate Of Remote Queue
Document 1204080.1 AQ Propagation Failing With ORA-25329 After Upgraded From 8i or 9i to 10g or 11g.
Document 1233675.1 AQ Propagation stops after upgrade to 11.2.0.1 ORA-30757

4.3.1. Errors Related to Incorrect Network Configuration

The most common propagation errors result from an incorrect network configuration. The list below contains common errors caused by tnsnames.ora file or database links being configured incorrectly:

4.4. Check the Database Links Exist and are Functioning Correctly

For schedules to remote databases confirm the database link exists via.

Connect as the owner of the link and select across it to verify it works and connects to the database we expect. i.e.

You need to ensure that the userid that scheduled the propagation (using DBMS_AQADM.SCHEDULE_PROPAGATION or DBMS_PROPAGATION_ADM.CREATE_PROPAGATION if using Streams) has access to the database link for the destination.

4.5. Has Propagation Been Correctly Scheduled?

Check that the propagation schedule has been created and that a job queue process has been assigned. Look for the entry in DBA_QUEUE_SCHEDULES and SYS.AQ$_SCHEDULES for your schedule. For 10g and below, check that it has a JOBNO entry in SYS.AQ$_SCHEDULES, and that there is an entry in DBA_JOBS with that JOBNO. For 11g and above, check that the schedule has a JOB_NAME entry in SYS.AQ$_SCHEDULES, and that there is an entry in DBA_SCHEDULER_JOBS with that JOB_NAME. Check the destination is as intended and spelled correctly.

AQ$_LOCAL in the destination column shows that the queue to which we are propagating to is in the same database as the source queue. If the propagation was to a remote (different) database, a database link will be in the DESTINATION column. The entry in the SCHEDULE_DISABLED column, N, means that the schedule is NOT disabled. If Y (yes) appears in this column, propagation is disabled and the schedule will not be executed. If not using Oracle Streams, propagation should resume once you have enabled the schedule by invoking DBMS_AQADM.ENABLE_PROPAGATION_SCHEDULE (for 10.2 Oracle Streams and above, the DBMS_PROPAGATION_ADM.START_PROPAGATION procedure should be used). The PROCESS_NAME is the name of the job queue process currently allocated to execute the schedule. This process is allocated dynamically at execution time. If the PROCESS_NAME column is null (empty) the schedule is not currently executing. You may need to execute this statement a number of times to verify if a process is being allocated. If a process is at some time allocated to the schedule, it is attempting to execute.

In 11g, these dates are expressed in TIMESTAMP WITH TIME ZONE datatypes. If the NEXT_RUN_DATE and NEXT_RUN_TIME columns are null when this statement is executed, the scheduled propagation is currently in progress. If they never change it would suggest that the schedule itself is never executing. If the next scheduled execution is too far away, change the NEXT_TIME parameter of the schedule so that schedules are executed more frequently (assuming that the window is not set to be infinite). Parameters of a schedule can be changed using the DBMS_AQADM.ALTER_PROPAGATION_SCHEDULE call.

In 10g and below, scheduling propagation posts a job in the DBA_JOBS view. The columns are more or less the same as DBA_QUEUE_SCHEDULES so you just need to recognize the job and verify that it exists.

For 11g, scheduling propagation posts a job in DBA_SCHEDULER_JOBS instead:

If no job exists, check DBA_QUEUE_SCHEDULES to make sure that the schedule has not been disabled.

For 10g and below, the job number is dynamic for AQ propagation schedules. The procedure that is executed to expedite a propagation schedule runs, removes itself from DBA_JOBS, and then reposts a new job for the next scheduled propagation. The job number should therefore always increment unless the schedule has been set up to run indefinitely.

4.6. Is the Schedule Executing but Failing to Complete?

Run the following query:

The failures column shows how many times we have attempted to execute the schedule and failed. Oracle will attempt to execute the schedule 16 times after which it will be removed from the DBA_JOBS or DBA_SCHEDULER_JOBS view and the schedule will become disabled. The column DBA_QUEUE_SCHEDULES.SCHEDULE_DISABLED will show 'Y'. For 11g and above, the DBA_SCHEDULER_JOBS.STATE column will show 'BROKEN' for the job corresponding to DBA_QUEUE_SCHEDULES.JOB_NAME. Prior to 10g the back off algorithm for failures was exponential, whereas from 10g onwards it is linear. The propagation will become disabled on the 17th attempt. Only the last execution failure will be reflected in the LAST_ERROR_MSG column. That is, if the schedule fails 5 times for 5 different reasons, only the last set of errors will be recorded in DBA_QUEUE_SCHEDULES.

Any errors need to be resolved to allow propagation to continue. If propagation has also become disabled due to 17 failures, first resolve the reason for the error and then re-enable the schedule using the DBMS_AQADM.ENABLE_PROPAGATION_SCHEDULE procedure, or DBMS_PROPAGATION_ADM.START_PROPAGATION if using 10.2 or above Oracle Streams. As soon as the schedule executes successfully the error message entries will be deleted. Oracle does not keep a history of past failures. However, when using Oracle Streams, the errors will be retained in the DBA_PROPAGATION view even after the schedule resumes successfully. See the following note for instructions on how to clear out the errors from the DBA_PROPAGATION view:

Document 808136.1 How to clear the old errors from DBA_PROPAGATION view?

If a schedule is active and no errors are being reported then the source queue may not have any messages to be propagated.

4.7. Do the Propagation Notification Queue Table and Queue Exist?

Check to see that the propagation notification queue table and queue exist and are enabled for enqueue and dequeue. Propagation makes use of the propagation notification queue for handling propagation run-time events, and the messages in this queue are stored in a SYS-owned queue table. This queue should never be stopped or dropped and the corresponding queue table never be dropped.

10g and below
The propagation notification queue table is of the format SYS.AQ$_PROP_TABLE_n, where 'n' is the RAC instance number, i.e. '1' for a non-RAC environment. This queue and queue table are created implicitly when propagation is first scheduled. If propagation has been scheduled and these objects do not exist, try unscheduling and rescheduling propagation. If they still do not exist contact Oracle Support.

If the AQ$_PROP_NOTIFY_1 queue is not enabled for enqueue or dequeue, it should be so enabled using DBMS_AQADM.START_QUEUE. However, the exception queue AQ$_AQ$_PROP_TABLE_1_E should not be enabled for enqueue or dequeue.

11g and above
The propagation notification queue table is of the format SYS.AQ_PROP_TABLE, and is created when the database is created. If they do not exist, contact Oracle Support.

If the AQ_PROP_NOTIFY queue is not enabled for enqueue or dequeue, it should be so enabled using DBMS_AQADM.START_QUEUE. However, the exception queue AQ$_AQ$_PROP_TABLE_E should not be enabled for enqueue or dequeue.

4.8. Does the Remote Queue Exist and is it Enabled for Enqueueing?

Check that the remote queue the propagation is transferring messages to exists and is enabled for enqueue:

4.9. Do the Target and Source Database Charactersets Differ?

If a message fails to propagate, check the database charactersets of the source and target databases. Investigate whether the same message can propagate between the databases with the same characterset or it is only a particular combination of charactersets which causes a problem.

4.10. Check the Queue Table Type Agreement

Propagation is not possible between queue tables which have types that differ in some respect. One way to determine if this is the case is to run the DBMS_AQADM.VERIFY_QUEUE_TYPES procedure for the two queues that the propagation operates on. If the types do not agree, DBMS_AQADM.VERIFY_QUEUE_TYPES will return '0'.

For AQ propagation between databases which have different NLS_LENGTH_SEMANTICS settings, propagation will not work, unless the queues are Oracle Streams ANYDATA queues.

See the following notes for issues caused by lack of type agreement:

Document 1079577.1 Advanced Queuing Propagation Fails With "ORA-22370: incorrect usage of method"
Document 282987.1 Propagated Messages marked UNDELIVERABLE after Drop and Recreate Of Remote Queue
Document 353754.1 Streams Messaging Propagation Fails between Single and Multi-byte Charactersets when using Chararacter Length Semantics in the ADT

4.11. Enable Propagation Tracing

4.11.1. System Level

This is set it in the init.ora/spfile as follows:

and restart the instance

This event cannot be set dynamically with an alter system command until version 10.2:

To unset the event:

Debugging information will be logged to job queue trace file(s) (jnnn) as propagation takes place. You can check the trace file for errors, and for statements indicating that messages have been sent. For the most part the trace information is understandable. This trace should also be uploaded to Oracle Support if a service request is created.

4.11.2. Attaching to a Specific Process

We can also attach to an existing job queue processes that is running a propagation schedule and trace it individually using the oradebug utility, as follows:

10.2 and below

11g

4.11.3. Further Tracing

The previous tracing steps only trace the job queue process executing the propagation on the source. At times it is useful to trace the propagation receiver process (the session which is enqueueing the messages into the target queue) on the target database which is associated with the job queue process on the source database.

These following queries provide ways of identifying the processes involved in propagation so that you can attach to them via oradebug to generate trace information.

In order to identify the propagation receiver process you need to execute the query as a user with privileges to access the v$ views in both the local and remote databases so the database link must connect as a user with those privileges in the remote database. The <DBLINK> in the queries should be replaced by the appropriate database link.

The queries have two forms due to the differences between operating systems. The value returned by 'Rem Process' is the operating system identifier of the propagation receiver on the remote database. Once identified, this process can be attached to and traced on the remote database using the commands given in Section 4.11.2.

10.2 and below - Windows

10.2 and below - Unix

11g - Windows

11g - Unix

5. Additional Troubleshooting Steps for AQ Propagation of User-Enqueued and Dequeued Messages

5.1. Check the Privileges of All Users Involved

Ensure that the owner of the database link has the necessary privileges on the aq packages.

Note that when queue table is created, a view called QT<nnn>_BUFFER is created in the SYS schema, and the queue table owner is given SELECT privileges on it. The <nnn> corresponds to the object_id of the associated queue table.

It is good practice to configure central AQ administrative user. All admin and processing jobs are created, executed and administered as this user. This configuration is not mandatory however, and the database link can be owned by any existing queue user. If this latter configuration is used, ensure that the connecting user has the necessary privileges on the AQ packages and objects involved.

Privileges for an AQ Administrative user

• Execute on DBMS_AQADM
• Execute on DBMS_AQ
• Granted the AQ_ADMINISTRATOR_ROLE

Privileges for an AQ user

• Execute on DBMS_AQ
• Execute on the message payload
• Enqueue privileges on the remote queue
• Dequeue privileges on the originating queue

Privileges need to be confirmed on both sites when propagation is scheduled to remote destinations. Verify that the user ID used to login to the destination through the database link has been granted privileges to use AQ.

5.2. Verify Queue Payload Types

AQ will not propagate messages from one queue to another if the payload types of the two queues are not verified to be equivalent. An AQ administrator can verify if the source and destination's payload types match by executing the DBMS_AQADM.VERIFY_QUEUE_TYPES procedure. The results of the type checking will be stored in the SYS.AQ$_MESSAGE_TYPES table. This table can be accessed using the object identifier OID of the source queue and the address database link of the destination queue, i.e. [schema.]queue_name[@destination].

Prior to Oracle 9i the payload (message type) had to be the same for all the queue tables involved in propagation. From Oracle9i onwards a transformation can be used so that payloads can be converted from one type to another. The following procedural call made on the source database can verify whether we can propagate between the source and the destination queue tables.

If propagation is possible then the return code value will be 1. If it is 0 then propagation is not possible and further investigation of the types and transformations used by and in conjunction with the queue tables is required. With regard to comparison of the types the following sql can be used to extract the DDL for a specific type with' %' changed appropriately on the source and target. This can then be compared for the source and target.

5.3. Check Message State and Destination

The first step in this process is to identify the queue table associated with the problem source queue. Although you schedule propagation for a specific queue, most of the meta-data associated with that queue is stored in the underlying queue table. The following statement finds the queue table for a given queue (note that this is a multiple-consumer queue table).

For a small amount of messages in a multiple-consumer queue table, the following query can be run:

In this example we see 2 messages ready to be propagated to remote queues and 1 that is not. If the address column is blank, the message is not scheduled for propagation and can only be dequeued from the queue upon which it was enqueued. The MSG_STATE column values are discussed in

Document 102330.1 Advanced Queueing MSG_STATE Values and their Interpretation.

If the address column has a value, the message has been enqueued for propagation to another queue. The first row in the example includes a database link (@M2V102.ES). This demonstrates that the message should be propagated to a queue at a remote database. The third row does not include a database link so will be propagated to a queue that resides on the same database as the source queue. The consumer name is the intended recipient at the target queue.

Note that we are not querying the base queue table directly; rather, we are querying a view that is available on top of every queue table, AQ$<queue_table_name>.

A more realistic query in an environment where the queue table contains thousands of messages is

8.0.3-compatible multiple-consumer queue table and all compatibility single-consumer queue tables

8.1.3 and 10.0-compatible queue tables

For multiple-consumer queue tables, if you did not see the expected CONSUMER_NAME , check the syntax of the enqueue code and verify the recipients are declared correctly. If a recipients list is not used on enqueue, check the subscriber list in the AQ$_<queue_table_name>_S view (note that a single-consumer queue table does not have a subscriber view. This view records all members of the default subscription list which were added using the DBMS_AQADM.ADD_SUBSCRIBER procedure and also those enqueued using a recipient list.

In this example we have 2 subscribers registered with the queue. We have a local subscriber AQUSER1, and a remote subscriber AQUSER2, on the queue INQ, owned by AQADM, at M2V102.ES. Unless overridden with a recipient list during enqueue every message enqueued to this queue will be propagated to INQ at M2V102.ES.

For 8.1 style and above multiple consumer queue tables, you can also check the following information at the target:

For 8.0 style queues, if the queue table supports multiple consumers you can obtain the same information from the history column of the queue table:

A non-NULL TRANSACTION_ID indicates that the message was successfully propagated. Further, the DEQ_TIME indicates the time of propagation, the DEQ_USER indicates the userid used for propagation, and the PROPAGATED_MSGID indicates the message ID of the message that was enqueued at the destination.

6. Additional Troubleshooting Steps for Propagation in an Oracle Streams Environment

6.1. Is the Propagation Enabled?

For a propagation job to propagate messages, the propagation must be enabled. For Streams, a special view called DBA_PROPAGATION exists to convey information about Streams propagations. If messages are not being propagated by a propagation as expected, then the propagation might not be enabled. To query for this:

At times, the propagation job may become "broken" or fail to start after an error has been encountered or after a database restart. If an error is indicated by the above query, an attempt to disable the propagation and then re-enable it can be made. In the examples below, for the propagation named STRMADMIN_PROPAGATE where the queue name is STREAMS_QUEUE owned by STRMADMIN and the destination database link is ORCL2.WORLD, the commands would be:

10.2 and above

If the above does not fix the problem, stop the propagation specifying the force parameter (2nd parameter on stop_propagation) as TRUE:

The statistics for the propagation as well as any old error messages are cleared when the force parameter is set to TRUE. Therefore if the propagation schedule is stopped with FORCE set to TRUE, and upon restart there is still an error message in DBA_PROPAGATION, then the error message is current.

9.2 or 10.1

If the above does not fix the problem, perform an unschedule of propagation and then schedule_propagation:

Typically if the error from the first query in Section 6.1 recurs after restarting the propagation as shown above, further troubleshooting of the error is needed.

6.2. Check Propagation Rule Sets and Transformations

Inspect the configuration of the rules in the rule set that is associated with the propagation process to make sure that they evaluate to TRUE as expected. If not, then the object or schema will not be propagated. Remember that when a negative rule evaluates to TRUE, the specified object or schema will not be propagated. Finally inspect any rule-based transformations that are implemented with propagation to make sure they are changing the data in the intended way.

The following query shows what rule sets are assigned to a propagation:

The next two queries list the propagation rules and their conditions. The first is for the positive rule set, the second is for the negative rule set:

6.3. Determining the Total Number of Messages and Bytes Propagated

As in Section 3.1, determining if messages are flowing can be instructive to see whether the propagation is entirely hung or just slow. If the propagation is not in flow control (see Section 6.5.2), but the statistics are incrementing slowly, there may be a performance issue. For Streams implementations two views are available that can assist with this that can show the number of messages sent by a propagation, as well as the number of acknowledgements being returned from the target site: the V$PROPAGATION_SENDER view at the Source site and the V$PROPAGATION_RECEIVER view at the destination site. It is helpful to query both to determine if messages are being delivered to the target. Look for the statistics to increase.

Source:

Target:

6.4. Check Buffered Subscribers

The V$BUFFERED_SUBSCRIBERS view displays information about subscribers for all buffered queues in the instance. This view can be queried to make sure that the site that the propagation is propagating to is listed as a subscriber address for the site being propagated from:

The SUBSCRIBER_ADDRESS column will not be populated when the propagation is local (between queues on the same database).

6.5. Common Streams Propagation Errors

6.5.1. ORA-02082: A loopback database link must have a connection qualifier.

This error can occur if you use the Streams Setup Wizard in Oracle Enterprise Manager without first configuring the GLOBAL_NAME for your database.

6.5.2. ORA-25307: Enqueue rate too high. Enable flow control

DBA_QUEUE_SCHEDULES will display this informational message for propagation when the automatic flow control (10g feature of Streams) has been invoked.

Similar to Streams capture processes, a Streams propagation process can also go into a state of 'flow control. This is an informative message that indicates flow control has been automatically enabled to reduce the rate at which messages are being enqueued into at target queue.

This typically occurs when the target site is unable to keep up with the rate of messages flowing from the source site. Other than checking that the apply process is running normally on the target site, usually no action is required by the DBA. Propagation and the capture process will be resumed automatically when the target site is able to accept more messages.

The following document contains more information:

Document 302109.1 Streams Propagation Error: ORA-25307 Enqueue rate too high. Enable flow control

See the following document for one potential cause of this situation:

Document 1097115.1 Oracle Streams Apply Reader is in 'Paused' State

6.5.3. ORA-25315 unsupported configuration for propagation of buffered messages

This error typically occurs when the target database is RAC and usually indicates that an attempt was made to propagate buffered messages with the database link pointing to an instance in the destination database which is not the owner instance of the destination queue. To resolve the problem, use queue-to-queue propagation for buffered messages.

6.5.4. ORA-600 [KWQBMCRCPTS101] after dropping / recreating propagation

For cause/fixes refer to:

Document 421237.1 ORA-600 [KWQBMCRCPTS101] reported by a Qmon slave process after dropping a Streams Propagation

6.5.5. Stopping or Dropping a Streams Propagation Hangs

See the following note:

Document 1159787.1 Troubleshooting Streams Propagation When It is Not Functioning and Attempts to Stop It Hang

6.6. Streams Propagation-Related Notes for Common Issues

Document 437838.1 Streams Specific Patches
Document 749181.1 How to Recover Streams After Dropping Propagation
Document 368912.1 Queue to Queue Propagation Schedule encountered ORA-12514 in a RAC environment
Document 564649.1 ORA-02068/ORA-03114/ORA-03113 Errors From Streams Propagation Process - Remote Database is Available and Unschedule/Reschedule Does Not Resolve
Document 553017.1 Stream Propagation Process Errors Ora-4052 Ora-6554 From 11g To 10201
Document 944846.1 Streams Propagation Fails Ora-7445 [kohrsmc]
Document 745601.1 ORA-23603 'STREAMS enqueue aborted due to low SGA' Error from Streams Propagation, and V$STREAMS_CAPTURE.STATE Hanging on 'Enqueuing Message'
Document 333068.1 ORA-23603: Streams Enqueue Aborted Eue To Low SGA
Document 363496.1 Ora-25315 Propagating on RAC Streams
Document 368237.1 Unable to Unschedule Propagation. Streams Queue is Invalid
Document 436332.1 dbms_propagation_adm.stop_propagation hangs
Document 727389.1 Propagation Fails With ORA-12528
Document 730911.1 ORA-4063 Is Reported After Dropping Negative Prop.Ruleset
Document 460471.1 Propagation Blocked by Qmon Process - Streams_queue_table / 'library cache lock' waits
Document 1165583.1 ORA-600 [kwqpuspse0-ack] In Streams Environment
Document 1059029.1 Combined Capture and Apply (CCA) : Capture aborts : ORA-1422 after schedule_propagation
Document 556309.1 Changing Propagation/ queue_to_queue : false -> true does does not work; no LCRs propagated
Document 839568.1 Propagation failing with error: ORA-01536: space quota exceeded for tablespace ''
Document 311021.1 Streams Propagation Process : Ora 12154 After Reboot with Transparent Application Failover TAF configured
Document 359971.1 STREAMS propagation to Primary of physical Standby configuation errors with Ora-01033, Ora-02068
Document 1101616.1 DBMS_PROPAGATION_ADM.DROP_PROPAGATION FAILS WITH ORA-1747

7. Performance Issues

A propagation may seem to be slow if the queries from Sections 3.1 and 6.3 show that the message statistics are not changing quickly.

In Oracle Streams, this more usually is due to a slow apply process at the target rather than a slow propagation. Propagation could be inferred to be slow if the message statistics are changing, and the state of a capture process according to V$STREAMS_CAPTURE.STATE is PAUSED FOR FLOW CONTROL, but an ORA-25307 'Enqueue rate too high. Enable flow control' warning is NOT observed in DBA_QUEUE_SCHEDULES per Section 6.5.2. If this is the case, see the following notes / white papers for suggestions to increase performance:

Document 335516.1 Master Note for Streams Performance Recommendations
Document 730036.1 Overview for Troubleshooting Streams Performance Issues
Document 780733.1 Streams Propagation Tuning with Network Parameters
White Paper: http://www.oracle.com/technetwork/database/features/availability/maa-wp-10gr2-streams-performance-130059.pdf
White Paper: Oracle Streams Configuration Best Practices: Oracle Database 10g Release 10.2, http://www.oracle.com/technetwork/database/features/availability/maa-10gr2-streams-configuration-132039.pdf, See APPENDIX A: USING STREAMS CONFIGURATIONS OVER A NETWORK

For basic AQ propagation, the network tuning in the aforementioned Appendix A of the white paper 'Oracle Streams Configuration Best Practices: Oracle Database 10g Release 10.2' is applicable.

References

NOTE:102330.1 - Advanced Queueing MSG_STATE Values and their Interpretation
NOTE:102771.1 - Advanced Queueing Propagation using PL/SQL
NOTE:1059029.1 - Combined Capture and Apply (CCA) : Capture aborts : ORA-1422 after schedule_propagation
NOTE:1079577.1 - Advanced Queuing Propagation Fails With "ORA-22370: incorrect usage of method"
NOTE:1083608.1 - 11g Streams and Oracle Scheduler
NOTE:1087324.1 - ORA-01405 ORA-01422 reported by Adavanced Queueing Propagation schedules after RAC reconfiguration
NOTE:1097115.1 - Oracle Streams Apply Reader is in 'Paused' State
NOTE:1101616.1 - DBMS_PROPAGATION_ADM.DROP_PROPAGATION FAILS WITH ORA-1747
NOTE:1159787.1 - Troubleshooting Streams Propagation When It is Not Functioning and Attempts to Stop It Hang
NOTE:1165583.1 - ORA-600 [kwqpuspse0-ack] In Streams Environment
NOTE:118884.1 - How to unschedule a propagation schedule stuck in pending state
NOTE:1203544.1 - AQ PROPAGATION ABORTED WITH ORA-600[OCIKSIN: INVALID STATUS] ON SYS.DBMS_AQADM_SYS.AQ$_PROPAGATION_PROCEDURE AFTER UPGRADE
NOTE:1204080.1 - AQ Propagation Failing With ORA-25329 After Upgraded From 8i or 9i to 10g or 11g.
NOTE:219416.1 - Advanced Queuing Propagation fails with ORA-22922
NOTE:222992.1 - DBMS_AQADM.DISABLE_PROPAGATION_SCHEDULE Returns ORA-24082
NOTE:253131.1 - Concurrent Writes May Corrupt LOB Segment When Using Auto Segment Space Management (ORA-1555)
NOTE:282987.1 - Propagated Messages marked UNDELIVERABLE after Drop and Recreate Of Remote Queue
NOTE:298015.1 - Kwqjswproc:Excep After Loop: Assigning To Self
NOTE:302109.1 - Streams Propagation Error: ORA-25307 Enqueue rate too high. Enable flow control
NOTE:311021.1 - Streams Propagation Process : Ora 12154 After Reboot with Transparent Application Failover TAF configured
NOTE:332792.1 - ORA-04061 error relating to SYS.DBMS_PRVTAQIP reported when setting up Statspack
NOTE:333068.1 - ORA-23603: Streams Enqueue Aborted Eue To Low SGA
NOTE:335516.1 - Master Note for Streams Performance Recommendations
NOTE:353325.1 - ORA-24056: Internal inconsistency for QUEUE and destination
NOTE:353754.1 - Streams Messaging Propagation Fails between Single and Multi-byte Charactersets when using Chararacter Length Semantics in the ADT.
NOTE:359971.1 - STREAMS propagation to Primary of physical Standby configuation errors with Ora-01033, Ora-02068
NOTE:363496.1 - Ora-25315 Propagating on RAC Streams
NOTE:365093.1 - ORA-07445 [kwqppay2aqe()+7360] reported on Propagation of a Transformed Message
NOTE:368237.1 - Unable to Unschedule Propagation. Streams Queue is Invalid
NOTE:368912.1 - Queue to Queue Propagation Schedule encountered ORA-12514 in a RAC environment
NOTE:421237.1 - ORA-600 [KWQBMCRCPTS101] reported by a Qmon slave process after dropping a Streams Propagation
NOTE:436332.1 - dbms_propagation_adm.stop_propagation hangs
NOTE:437838.1 - Streams Specific Patches
NOTE:460471.1 - Propagation Blocked by Qmon Process - Streams_queue_table / 'library cache lock' waits
NOTE:463820.1 - Streams Combined Capture and Apply in 11g
NOTE:553017.1 - Stream Propagation Process Errors Ora-4052 Ora-6554 From 11g To 10201
NOTE:556309.1 - Changing Propagation/ queue_to_queue : false -> true does does not work; no LCRs propagated
NOTE:564649.1 - ORA-02068/ORA-03114/ORA-03113 Errors From Streams Propagation Process - Remote Database is Available and Unschedule/Reschedule Does Not Resolve
NOTE:566622.1 - ORA-22275 when propagating >4K AQ$_JMS_TEXT_MESSAGEs from 9.2.0.8 to 10.2.0.1
NOTE:727389.1 - Propagation Fails With ORA-12528
NOTE:730036.1 - Overview for Troubleshooting Streams Performance Issues
NOTE:730911.1 - ORA-4063 Is Reported After Dropping Negative Prop.Ruleset
NOTE:731292.1 - ORA-25215 Reported On Local Propagation When Using Transformation with ANYDATA queue tables
NOTE:731539.1 - ORA-29268: HTTP client error 401 Unauthorized Error when the AQ Servlet attempts to Propagate a message via HTTP
NOTE:745601.1 - ORA-23603 'STREAMS enqueue aborted due to low SGA' Error from Streams Propagation, and V$STREAMS_CAPTURE.STATE Hanging on 'Enqueuing Message'
NOTE:749181.1 - How to Recover Streams After Dropping Propagation
NOTE:780733.1 - Streams Propagation Tuning with Network Parameters
NOTE:787367.1 - ORA-22275 reported on Propagating Messages with LOB component when propagating between 10.1 and 10.2
NOTE:808136.1 - How to clear the old errors from DBA_PROPAGATION view ?
NOTE:827184.1 - AQ Propagation with CLOB data types Fails with ORA-22990
NOTE:827473.1 - How to alter propagation from queue_to_queue to queue_to_dblink
NOTE:839568.1 - Propagation failing with error: ORA-01536: space quota exceeded for tablespace ''
NOTE:846297.1 - AQ Propagation Fails : ORA-00600[kope2upic2954] or Ora-00600[Kghsstream_copyn]
NOTE:944846.1 - Streams Propagation Fails Ora-7445 [kohrsmc]