Sending a large file over network continuously

Posted by David Parunakian on Stack Overflow See other posts from Stack Overflow or by David Parunakian
Published on 2010-06-07T09:40:03Z Indexed on 2010/06/07 10:02 UTC
Read the original article Hit count: 151

Filed under:

sockets

|

large-files

|

reliability

|

continuous

Hello,

We need to write software that would continuously (i.e. new data is sent as it becomes available) send very large files (several Tb) to several destinations simultaneously. Some destinations have a dedicated fiber connection to the source, while some do not.

Several questions arise:

We plan to use TCP sockets for this task. What failover procedure would you recommend in order to handle network outages and dropped connections?
What should happen upon upload completion: should the server close the socket? If so, then is it a good design decision to have another daemon provide file checksums on another port?
Could you recommend a method to handle corrupted files, aside from downloading them again? Perhaps I could break them into 10Mb chunks and calculate checksums for each chunk separately?

Thanks.

© Stack Overflow or respective owner

Related posts about sockets

Performance implications of Synchronous Sockets vs Asynchronous Sockets

as seen on Stack Overflow - Search for 'Stack Overflow'
We are trying to build an SMTP Server to receive mail notifications from various clients over internet. As each of the communication will be longer and it needs to log everything, doing this Asynchronous way is little challenging as well as by using Socket's Asynchronous methods we are not sure of… >>> More
Multiple INET sockets (mulple IP's too) connected to UNIX sockets

as seen on Server Fault - Search for 'Server Fault'
HOST = same host all the time, accepts multiple connection. I have a dedicated server and I will buy extra IP's. Socket 1 connects to HOST:PORT, from IP-1 Socket 2 connects to HOST:PORT, from IP-1 Socket 3 connects to HOST:PORT, from IP-1 Socket 4 connects to HOST:PORT, from IP-2 Socket 5 connects… >>> More
Multiple INET sockets (mulple IP's too) connected to UNIX sockets

as seen on Stack Overflow - Search for 'Stack Overflow'
HOST = same host all the time, accepts multiple connection. I have a dedicated server and I will buy extra IP's. Socket 1 connects to HOST:PORT, from IP-1 Socket 2 connects to HOST:PORT, from IP-1 Socket 3 connects to HOST:PORT, from IP-1 Socket 4 connects to HOST:PORT, from IP-2 Socket 5 connects… >>> More
Do WebSockets have exclusive access to their sockets?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm curious to know if, after a WebSocket has been established (after having received the proper handshake from a server that supports them), whether or not the TCP socket used by the "WebSocket connection" is used exclusively by the WebSocket, or if the browser may still make regular HTTP requests… >>> More
What Parallel computing APIs make good use of sockets?

as seen on Stack Overflow - Search for 'Stack Overflow'
My program uses sockets, what Parallel computing APIs could I use that would help me without obligating me to go from sockets to anything else? When we are on a cluster with a special, non-socket infrastructure system this API would emulate something like sockets but using that infrastructure (so… >>> More

Related posts about large-files

AS3 Working With Arbitrarily Large Files

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to read a very large file in AS3 and am having problems with the runtime just crashing on me. I'm currently using a FileStream to open the file asynchronously. This does not work(crashes without an Exception) for files bigger than about 300MB. _fileStream = new FileStream(); _fileStream… >>> More
Use php to zip large files

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have a php form that has a bunch of checkboxes that all contain links to files. Once a user clicks on which checkboxes (files) they want, it then zips up the files and forces a download. I got a simple php zip force download to work, but when one of the files is huge or if someone lets… >>> More
Opening first lines of large files in VIM

as seen on Super User - Search for 'Super User'
I run Windows XP and gvim7.2. I want to inspect a large (130 MB) XML file with VIM. Is it possible to simply open the first 1000 lines of the file? >>> More
Uploading Large Files

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I'm using a windows service on my server to receive large files (1 GB) from desktop clients when I use the TCPClient class it creates astream to send the file which bloks my memory, What is the well designe for this problem >>> More
RichFaces rich:insert takes a long time to output large files

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello I'm using a RichFaces <rich:insert like this: <rich:panel header="my head"> <a4j:outputPanel ajaxRendered="true"> <rich:insert src="#{MyBacking.myPath}" highlight="groovy" /> </a4j:outputPanel> </rich:panel> If I have a 60k file to output… >>> More