Threading best practice when using SFTP in C#

Posted by Christian on Stack Overflow See other posts from Stack Overflow or by Christian
Published on 2010-04-13T15:58:52Z Indexed on 2010/04/13 17:33 UTC
Read the original article Hit count: 549

Filed under:
|

Ok,

this is more one of these "conceptual questions", but I hope I got some pointers in the right direction. First the desired scenario:

  • I want to query an SFTP server for directory and file lists
  • I want to upload or download files simulaneously

Both things are pretty easy using a SFTP class provided by Tamir.SharpSsh, but if I only use one thread, it is kind of slow. Especially the recursion into subdirs gets very "UI blocking", because we are talking about 10.000 of directories.

My basic approach is simple, create some kind of "pool" where I keep 10 open SFTP connections. Then query the first worker for a list of dirs. If this list was obtained, send the next free workers (e.g. 1-10, first one is also free again) to get the subdirectory details. As soon as there is a worker free, send him for the subsubdirs. And so on...

I know the ThreadPool, simple Threads and did some Tests. What confuses me a little bit is the following: I basically need...

  • A list of threads I create, say 10
  • Connect all threads to the server
  • If a connection drops, create a new thread / sftp client
  • If there is work to do, take the first free thread and handle the work

I am currently not sure about the implementation details, especially the "work to do" and the "maintain list of threads" parts.

Is it a good idea to:

  • Enclose the work in an object, containing a job description (path) and a callback
  • Send the threads into an infinite loop with 100ms wait to wait for work
  • If SFTP is dead, either revive it, or kill the whole thread and create a new one
  • How to encapsulate this, do I write my own "10ThreadsManager" or are there some out

Ok, so far...

Btw, I could also use PRISM events and commands, but I think the problem is unrelated. Perhaps the EventModel to signal a done processing of a "work package"...

Thanks for any ideas, critic.. Chris

© Stack Overflow or respective owner

Related posts about c#

Related posts about multithreading