When designing a job queue, what should determine the scope of a job?
- by Stuart Pegg
We've got a job queue system that'll cheerfully process any kind of job given to it. We intend to use it to process jobs that each contain 2 tasks:
Job (Pass information from one server to another)
Fetch task (get the data, slowly)
Send task (send the data, comparatively quickly)
The difficulty we're having is that we don't know whether to break the tasks into separate jobs, or process the job in one go.
Are there any best practices or useful references on this subject? Is there some obvious benefit to a method that we're missing?
So far we can see these benefits for each method:
Split
Job lease length reflects job length: Rather than total of two
Finer granularity on recovery: If we lose outgoing connectivity we can tell them all to retry
The starting state of the second task is saved to job history: Helps with debugging (although similar logging could be added in single task method)
Single
Single job to be scheduled: Less processing overhead
Data not stale on recovery: If the outgoing downtime is quite long, the pending Send jobs could be outdated
