How to read from multiple queues in real-world?
- by Leon Cullens
Here's a theoretical question:
When I'm building an application using message queueing, I'm going to need multiple queues support different data types for different purposes. Let's assume I have 20 queues (e.g. one to create new users, one to process new orders, one to edit user settings, etc.).
I'm going to deploy this to Windows Azure using the 'minimum' of 1 web role and 1 worker role.
How does one read from all those 20 queues in a proper way? This is what I had in mind, but I have little or no real-world practical experience with this:
Create a class that spawns 20 threads in the worker role 'main' class. Let each of these threads execute a method to poll a different queue, and let all those threads sleep between each poll (of course with a back-off mechanism that increases the sleep time).
This leads to have 20 threads (or 21?), and 20 queues that are being actively polled, resulting in a lot of wasted messages (each time you poll an empty queue it's being billed as a message).
How do you solve this problem?
