Scenario: We have one instance of WCF windows service which exposes contracts like:

AddNewJob(Job job), GetJobs(JobQuery query) etc.

This service is consumed by 70-100 instances of client which is Windows Form based .NET app. Typically the service has 50-100 inward calls/minute to add or query jobs that are stored in a table on Sql Server.

The same service is also responsible for processing these jobs in real time. It queries database every 5 seconds picks up the queued jobs and starts processing them. A job has 6 states. Queued, Pre-processing, Processing, Post-processing, Completed, Failed, Locked.

Another responsibility on this service is to update all clients on every state change of every job. This means almost 200+ callbacks to clients per second.

Question: This whole implementation is done using WCF Duplex bindings and works perfectly fine on small number of parallel jobs. Problem arises when we scale it up to 1000 jobs at a time. The notifications don't work as expected, it leads to memory overflow etc. Is there any standard framework that can provide a clean infrastructure for handling this scenario??

Apologies for the long explanation!