hi.

creating a distributed crawling python app. it consists of a master server, and associated client apps that will run on client servers. the purpose of the client app is to run across a targeted site, to extract specific data. the clients need to go "deep" within the site, behind multiple levels of forms, so each client is specifically geared towards a given site.

each client app looks something like

main:

parse initial url

call function level1 (data1)

function level1 (data)
 parse the url, for data1
 use the required xpath to get the dom elements
 call the next function
 call level2 (data)


function level2 (data2)
 parse the url, for data2
 use the required xpath to get the dom elements
 call the next function
 call level3

function level3 (dat3)
 parse the url, for data3
 use the required xpath to get the dom elements
 call the next function
 call level4

function level4 (data)
 parse the url, for data4
 use the required xpath to get the dom elements

 at the final function.. 
 --all the data output, and eventually returned to the server        
 --at this point the data has elements from each function...

my question: given that the number of calls that is made to the child function by the current function varies, i'm trying to figure out the best approach.

 each function essentialy fetches a page of content, and then parses 
 the page using a number of different XPath expressions, combined 
 with different regex expressions depending on the site/page.

 if i run a client on a single box, as a sequential process, it'll 
 take awhile, but the load on the box is rather small. i've thought 
 of attempting to implement the child functions as threads from the 
 current function, but that could be a nightmare, as well as quickly 
 bring the "box" to its knees!

 i've thought of breaking the app up in a manner that would allow 
 the master to essentially pass packets to the client boxes, in a 
 way to allow each client/function to be run directly from the 
 master. this process requires a bit of rewrite, but it has a number 
 of advantages. a bunch of redundancy, and speed. it would detect if 
 a section of the process was crashing and restart from that point. 
 but not sure if it would be any faster...

i'm writing the parsing scripts in python..

so... any thoughts/comments would be appreciated...

i can get into a great deal more detail, but didn't want to bore anyone!!

thanks!

tom