How can I configure different worker pools using celery?
- by Chris R
I need to deploy a queued execution service with (generally) the following three classes of worker:
A periodic, low-priority job class that takes a long time and can be processed serially; these jobs should only use 0..2 workers in the system at most.
A periodic, deadline-sensitive job class that take a short to medium amount of time (say, topping out at 5 minutes)
An ad-hoc job class, that is higher priority than #1, but can interleave with #2. Any workers from class #2 that are inactive when this type of job comes in should handle it, without ever starving the pool of workers for #2
All three job classes are the same task, the only difference between them is how they're requested; they'll take the same input and generate the same output, but each one has different performance guarantees.
How can I implement this using celery?
