Open Grid Engine or Akka/Something more fault tolerant?
- by Mike Lyons
My use case is that I have a pipeline of independent, stand alone programs, that I want to execute in a certain order on specific pieces of data that our output from previous pipeline stages.
The pipeline is entirely linear and doesn't do anything in terms of alternate paths through the pipe.
I'm currently using SGE to do this and it works OK, however occasionally a job will overstep it's memory bounds, fail, and all jobs that require that output data will fail. The pipe needs to be restarted in that case, and it seems that whatever is providing the fault tolerance in akka might solve that for me?
