How to divide work to a network of computers?
- by Morpork
Imagine a scenario as follows: Lets say you have a central computer which generates a lot of data. This data must go through some processing, which unfortunately takes longer than to generate. In order for the processing to catch up with real time, we plug in more slave computers.
Further, we must take into account the possibility of slaves dropping out of the network mid-job as well as additional slaves being added. The central computer should ensure that all jobs are finished to its satisfaction, and that jobs dropped by a slave are retasked to another.
The main question is: What approach should I use to achieve this?
But perhaps the following would help me arrive at an answer:
Is there a name or design pattern to what I am trying to do?
What domain of knowledge do I need to achieve the goal of getting these computers to talk to each other? (eg. will a database, which I have some knowledge of, be enough or will this involve sockets, which I have yet to have knowledge of?)
Are there any examples of such a system? The main question is a bit general so it would be good to have a starting point/reference point.
Note I am assuming constraints of c++ and windows so solutions pointing in that direction would be appreciated.
