Parallel File Copy

Posted by Jon on Server Fault See other posts from Server Fault or by Jon
Published on 2010-06-18T01:08:29Z Indexed on 2010/06/18 1:13 UTC
Read the original article Hit count: 404

Filed under:

linux

|

copying

|

parallel

I have a list of files I need to copy on a Linux system - each file ranges from 10 to 100GB in size.

I only want to copy to the local filesystem. Is there a way to do this in parallel - with multiple processes each responsible for copying a file - in a simple manner?

I can easily write a multithreaded program to do this, but I'm interested in finding out if there's a low level Linux method for doing this.

© Server Fault or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about copying

'pip install carbon' looks like it works, but pip disagrees afterward

as seen on Server Fault - Search for 'Server Fault'
I'm trying to use pip to install the package carbon, a package related to statistics collection. When I run pip install carbon, it looks like everything works. However, pip is unconvinced that the package is actually installed. (This ultimately causes trouble because I'm using Puppet, and have a rule… >>> More
Permission denied when copying on a fileshare in Finder, but copying via command line works

as seen on Server Fault - Search for 'Server Fault'
I'm trying to copy files on a SMB fileshare. When I attempt to copy the files in Finder, I get the following error: The operation can’t be completed because you don’t have permission to access some of the items. Copying via Terminal.app (using a simple cp command) works just fine. Permissions… >>> More
Win 2008 R2 - copying TO disk is very slow, copying FROM is more or less okay

as seen on Server Fault - Search for 'Server Fault'
I have Windows 2008 R2 SP1 with 4 identical SATA disks (Seagate Barracude 7200) in RAID 5 array. It has 4Gb of memory; all recent updates are installed. Problem: when I copy large file from one folder to another, I get about 10MB/s average speed. When I read this file from network share via 1Gbps… >>> More
Cisco Unifed Communication integration for Microsoft Lync crashes on Remote Desktop services 2008 R2!

as seen on Server Fault - Search for 'Server Fault'
Hi everybody i have deployed office communication server 2007 R2 and communicator 2007 R2 and i made integration with Cisco Unified Communication Manager 7.1 in my network, i also uses Remote Desktop Servers 2008 R2 for Thin Client Computers, now that i installed Cisco UC integration client for communicator… >>> More
Cisco Unified Communication integration for Microsoft Lync crashes on Remote Desktop services 2008 R2!

as seen on Server Fault - Search for 'Server Fault'
Hi everybody i have deployed office communication server 2007 R2 and communicator 2007 R2 and i made integration with Cisco Unified Communication Manager 7.1 in my network, i also use Remote Desktop Servers 2008 R2 for Thin Client Computers, now that i installed Cisco UC integration client for communicator… >>> More