BizTalk host throttling – Singleton pattern and High database size

Posted by S.E.R. on Geeks with Blogs See other posts from Geeks with Blogs or by S.E.R.
Published on Sun, 30 Jun 2013 16:15:04 GMT Indexed on 2013/06/30 22:21 UTC
Read the original article Hit count: 617

Originally posted on: http://geekswithblogs.net/SERivas/archive/2013/06/30/biztalk-host-throttling-ndash-singleton-pattern-and-high-database-size.aspx

I have worked for some days around the singleton pattern (for those unfamiliar with it, read this post by Victor Fehlberg) and have come across a few very interesting posts, among which one dealt with performance issues (here, also by Victor Fehlberg). Simply put: if you have an orchestration which implements the singleton pattern, then performances will continuously decrease as the orchestration receives and consumes messages, and that behavior is more obvious when the orchestration never ends (ie : it keeps looping and never terminates or completes).

As I experienced the same kind of problem (actually I was alerted by SCOM, which told me that the host was being throttled because of High database size), I thought it would be a good idea to dig a little bit a see what happens deep inside BizTalk and thus understand the reasons for this behavior.

NOTE: in this article, I will focus on this High database size throttling condition. I will try and work on the other conditions in some not too distant future…

Test conditions

The singleton orchestration

For the purpose of this study, I have created the following orchestration, which is a very basic implementation of a singleton that piles up incoming messages, then does something else when a certain timeout has been reached without receiving another message:

clip_image002

Throttling settings

I have two distinct hosts :

  • one that hosts the receive port (basic FILE port) : Ports_ReceiveHost
  • one that hosts the orchestration : ProcessingHost

In order to emphasize the throttling mechanism, I have modified the throttling settings for each of these hosts are as follows (all other parameters are set to the default value):

  • [Throttling thresholds] Message count in database: 500 (default value : 50000)

Evolution of performance counters when submitting messages

Since we are investigating the High database size throttling condition, here are the performance counter that we should take a look at (all of them are in the BizTalk:Message Agent performance object):

  • Database size
  • High database size
  • Message delivery throttling state
  • Message publishing throttling state
  • Message delivery delay (ms)
  • Message publishing delay (ms)
  • Message delivery throttling state duration
  • Message publishing throttling state duration

(If you are not used to Perfmon, I strongly recommend that you start using it right now: it is a wonderful tool that allows you to open the hood and see what is going on inside BizTalk – and other systems)

Database size

It is quite obvious that we will start by watching the database size and high database size counters, just to see when the first reaches the configured threshold (500) and when the second rings the alarm.

NOTE : During this test I submitted 600 messages, one message at a time every 10ms to see the evolution of the counters we have previously selected.

clip_image004

It might not show very well on this screenshot, but here is what happened:

  • From 15:46:50 to 15:47:50, the database size for the Ports_ReceiveHost host (blue line) kept growing until it reached a maximum of 504.
  • At 15:47:50, the high database size alert fires

At first I was surprised by this result: why is it the database size of the receiving host that keeps growing since it is the processing host that piles up messages? Actually, it makes total sense. This counter measures the size of the database queue that is being filled by the host, not consumed. Therefore, the high database size alert is raised on the host that fills the queue: Ports_ReceiveHost. More information is available on the Public MPWiki page.

Now, looking at the Message publishing throttling state for the receiving host (green line), we can see that a throttling condition has been reached at 15:47:50:

clip_image006

We can also see that the Message publishing delay(ms) (blue line) has begun growing slowly from this point.

All of this explains why performances keep decreasing when a singleton keeps processing new messages: the database size grows and when it has exceeded the Message count in database threshold, the host is throttled and the publishing delay keeps increasing.

Digging further

So, what happens to the database queue then? Is it flushed some day or does it keep growing and growing indefinitely? The real question being: will the host be throttled forever because of this singleton?

To answer this question, I set the Message count in database threshold to 20 (this value is very low in order not to wait for too long, otherwise I certainly would have fallen asleep in front of my screen) and I submitted 30 messages.

The test was started at 18:26. At 18:56 (ie : exactly 30min later) the throttling was stopped and the database size was divided by 2.

30 min later again, the database size had dropped to almost zero:

clip_image008

I guess I’ll have to find some documentation and do some more testing before I sort this out! My guess is that some maintenance job is at work here, though I cannot tell which one

Digging even further

If we take a look at the Message delivery throttling state counter for the processing host, we can see that this host was also throttled during the submission of the 600 documents:

clip_image010

The value for the counter was 1, meaning that Message delivery incoming rate for the host instance exceeds the Message delivery outgoing rate * the specified Rate overdrive factor (percent) value.

We will see this another day… :)

A last word

Let’s end this article with a warning: DO NOT CHANGE THE THROTTLING SETTINGS LIGHTLY! The temptation can be great to just bypass throttling by setting very high values for each parameter (or zero in some cases, which simply disables throttling). Nevertheless, always keep in mind that this mechanism is here for a very good reason: prevent your BizTalk infrastructure from exploding!! So whatever you do with those settings, do a lot of testing and benchmarking!

© Geeks with Blogs or respective owner

Related posts about BizTalk

Related posts about Singleton pattern