Originally posted on: http://geekswithblogs.net/SERivas/archive/2013/11/11/biztalk-routing-failure-on-delivery-notifications.aspx

This is a detailed explanation of a something I posted a few month ago on stackoverflow, concerning a weird behavior (a bug, really…) of the delivery notifications in BizTalk.

Reminder: what are delivery notifications

Mechanism

BizTalk has the ability to automatically publish positive acknowledgments (ACK) when it has succeeded transmitting a message or negative acknowledgments (NACK) in case of a transmission failure. Orchestrations can use delivery notifications to subscribe to those ACKs and NACKs in order to know if a message sent on a one-way send port has been successfully transmitted.

Delivery Notifications can be “activated” in two ways:

• The most common and easy way is to set the Delivery Notification property of a logical send port (in the orchestration designer) to Transmitted:

• Another way is to set the BTS.AckRequired context property of the message to be sent to true:

NOTE: fundamentally, those methods are strictly equivalent since the fact of setting the Delivery Notification to Transmitted on the send port only tells BizTalk the BTS.AckRequired context property has to be set to true on the outgoing message.

Related context properties

ACKs and NACKs have a common set of propoted context properties, which are :

Propriété	Description
AckType	Equals ACK when successful or NACK otherwise
AckID	MessageID of the message concerned by the acknowledgment
AckOwnerID	InstanceID of the instance associated with the acknowledgment
AckSendPortID	ID of the send port
AckSendPortName	Name of the send port
AckOutboundTransportLocation	URI of the send port
AckReceivePortID	ID of the port the message came from
AckReceivePortName	Name of the port the message came from
AckInboundTransportLocation	URI of the port the message came from

Detailed behavior

The way Delivery Notifications are handled by BizTalk is peculiar compared to the standard behavior of the Message Box: if no active subscription exists for the acknowledgment, it is simply discarded. The direct consequence of this is that there can be no routing failure for an acknowledgment, and an acknowledgment cannot be suspended.

Moreover, when a message is sent to a send port where Delivery Notification = Transmitted, a correlation set is initialized and a correlation token is attached to the message (Context property: CorrelationToken). This correlation token will also be attached to the acknowledgment. So when the acknowledgment is issued, it is automatically routed to the source orchestration.

Finally, when a NACK is received by the source orchestration, a DeliveryFailureException is thrown, which can be caught in Catch section.

Context of the problem

Consider this scenario:

• In an orchestration, Delivery Notifications are activated on a One-Way send port
• In case of a transmission failure, the messaging instance is suspended and the orchestration catches an exception (DeliveryFailureException).
• When the exception is caught, the orchestration does some logging and then terminates (thanks to a Terminate shape).

So that leaves only the suspended messaging instance, waiting to be resumed.

Symptoms

Once the problem that caused the transmission failure is solved, the messaging instance is resumed. Considering what was said in the reminder, we would expect the instance to complete, leaving no active or suspended instance.

Nevertheless, the result is that the messaging instance is once more suspended, this time because of a routing failure:

The routing failure report shows that the suspended message has the following attached properties:

Explanation

Those properties clearly indicate that the message being suspended is an acknowledgment (ACK in this case), which was published in the message box and was supended because no subscribers were found.

This makes sense, since the source orchestration was terminated before we resumed the messaging instance. So its subscription to the acknowledgments was no longer active when the ACK was published, which explains the routing failure.

But this behavior is in direct contradiction with what was said earlier: an acknowledgment must be discarded when no subscriber is found and therefore should not be suspended.

Cause

It is indeed an outright bug, which appeared with the SP1 of BizTalk 2006 R2 and was never corrected since then: not in the next 4 CUs, not in BizTalk 2009, not in 2010 and not event in 2013 – though I haven’t tested CU1 and CU2 for this last edition, but I bet there is nothing to be expected from those CUs (on this particular point).

Side effects

This bug can have pretty nasty side effects: this behavior can be propagated to other ports, due to routing mechanisms.

For instance: you have configured the ESB Toolkit and have activated the “Enable routing failure for failed messages”. The result will be that the ESB Exception SQL send port will also try and publish ACKs or NACKs concerning its own messaging instances. In itself, this is already messy, but remember that those acknowledgments will also have the source correlation token attached to them… See how far it goes?

Well, actually there is more: in SQL send ports, transactions will be rolled back because of the routing failure (I guess it also happens with other adapters - like Oracle, but I haven’t tested them). Again, think of what happens when the send port is the ESB Exception send port: your BizTalk box is going mad, but you have no idea since no exception can be written in the exception database!

All of this can be tricky to diagnose, I can tell you that…

Solution

There is no real solution, only a work-around, but it won’t solve all of the problems and side effects. The idea is to create an orchestration which subscribes to all acknowledgments. That is to say:

• The message type of the incoming message will be XmlDocument
• The BTS.AckType property exists
• The logical receive port will use direct binding

By doing so, all acknowledgments will be consumed by an instance of this orchestration, thus avoiding the routing failure. Here is an example of what this orchestration could look like:

In order not to pollute the HAT and the DTA Db (after all, this orchestration is only meant to be a palliative to some faulty internal BizTalk mechanism, so there should be no trace of its execution), all tracking must be deactivated: