How to transfer data between two networks efficiently
- by Tono Nam
I would like to transfer files between two places over the internet. Right now I have a VPN and I am able to browse, download and transfer files. So my question is not really how to transfer the files; Instead, I would like to use the most efficient approach because the two places constantly share a lot of data.
The reason why I want to get rid of the VPN is because it is two slow. Having high upload speed is very expensive/impossible in residential places so I would like to use a different approach.
I was thinking about using programs such as http://www.dropbox.com . The problem with Dropbox is that the free version comes with only 2 GB of storage. I think the deals they offer are OK and I might be willing to pay to get that increase in speed. But I am concerned with the speed of transferring data. Dropbox will upload the file to their server then send it from the server to the other location. I would like it to be even faster.
Anyway I was thinking why not create a program myself. This is the algorithm that I was thinking of. Let me know if it sounds too crazy.
(Remember my goal is to transfer files as fast as possible)
Things that I will use in this algorithm:
Server on the internet called S (Has fast download and upload speed. I pay to host a website and some services in there. I want to take advantage of it.)
Client A at location 1
Client B at location 2
So lets say at location 1, 20 large files are created and need to be transferred to location 2.
Client A compresses the files with the highest compression ratio possible.
Client A starts sending data via UDP to client B.
Because I am using UDP I will include the sequence number on each packet.
Have server S help speed up things. For example every time a packet is lost we can use Server S to inform client A that it needs to resend a packet.
Anyways I think this approach will increase the transfer rate. I do not know if it is possible to start sending data while it is being compressed. Or if it is possible to start decompressing data even if we are not done receiving the whole file. Maybe it will be faster to start sending the files right away without compressing. If I knew that I will always be sending large text files then I will obviously use the compression. I need this as a general algorithm.
So I guess my question is could I increase performance by using UDP instead of TCP and by using an extra server to keep track of lost packets? And how should I compress files before sending? Compressing a 1 GB file with the highest compression ratio takes about 1 hour! I would like to take advantage of that time by sending it as it is being compressed.
