How to backup millions of small files?

Posted by grassbl8d on Server Fault See other posts from Server Fault or by grassbl8d
Published on 2011-02-02T11:24:22Z Indexed on 2011/06/24 0:24 UTC
Read the original article Hit count: 449

Filed under:

windows-server-2008

|

backup

What is the best way to backup millions of small files in a very small time period? We have less than 5 hours to backup a file system which contains around 60 million files which are mostly small files.

We have tried several solutions such as richcopy, 7z, rsync and all of them seems to have a hard time. We are looking for the most optimal way...

We are open to putting the file in an archive first or transferring the file to another location via network or hard disk transfer

thanks

© Server Fault or respective owner

Related posts about windows-server-2008

Password policy problem windows server 2008

as seen on Server Fault - Search for 'Server Fault'
Hi, I'm creating a domain and I need to make users that can have an empty password but administrators have to comply with password complexity how to I solve this problem? Thanks in advance >>> More
SAP DCOM Connector on Windows Server 2008

as seen on Stack Overflow - Search for 'Stack Overflow'
Does anybody know, if it is possible to use the "old" SAP DCOM Connector on a Windows Server 2008 ? I want to migrate a old ASP Web Solution with DCOM Connection to SAP from Windows 2000 Server to Windows 2008 Server. When I try to install the DCOM Connector I get the Error Message: "Setup could… >>> More
Windows Server 2008 R2: AD Module for Windows PowerShell

as seen on Internet.com - Search for 'Internet.com'
Windows Server 2008 R2 includes a new Active Directory module for Windows PowerShell, a consolidated group of 76 cmdlets that can perform a host of tasks against Active Directory's various services. >>> More
Windows Server 2008 Directory Services, Group Policy Preferences - More Control Panel Settings

as seen on Internet.com - Search for 'Internet.com'
In Windows Server 2008, Group Policy Preferences makes it possible to reap the chief benefits of an Active Directory environment by simplifying client management. Control Panel Settings nodes in the Group Policy Management Editor makes this possible. >>> More
Windows Server 2008 R2: Introducing the AD Administrative Center

as seen on Internet.com - Search for 'Internet.com'
The Active Directory Administrative Center in Windows Server 2008 R2 is a significant improvement over its predecessor. Although not without limitations, it offers beefed-up management of AD objects, new navigation capabilities, better task-based management options, and improvements to the properties… >>> More

Related posts about backup

backup exec - backup to disk offline

as seen on Server Fault - Search for 'Server Fault'
Hi We are running backup exec 9.1 doing a backup to disk to portable hard disk drives. When we run the backup manually it works fine. But when the backup is setup to run in the evening on a schedule it does not run as the backup to disk folders goes offline and therefore has to be switched back… >>> More
Ideal backup appliance for backup software like Bacula?

as seen on Server Fault - Search for 'Server Fault'
I'm at a small company and we (the IT department of two) manage <100 client computers and a handful of servers. Currently we're using a company's appliance to handle backup; it does a small backup every night and a full backup every weekend, and a guy comes on Wednesday to take an offsite backup… >>> More
Symantec Backup Exec Error on backup

as seen on Server Fault - Search for 'Server Fault'
Recently we have moved some of servers from real servers- into virtual servers. Since then, we are getting errors like the following: Error category : Resource ErrorsError : e000fed1 - A failure occurred querying the Writer status. For additional information regarding this error refer… >>> More
Windows Server Backup - Recover only shows the latest backup

as seen on Server Fault - Search for 'Server Fault'
We're having quite some trouble at work using Windows Server Backup. We have a HyperV server (Win 2008) running 8 virtual web servers, these are running a variety of OS'es: Win 2003, Win 2008 and a lone Debian. Each virtual server has a separate partition on the physical HyperV server, so e.g. E:… >>> More
Failed Backup Job With Backup Exec 12 and AOFO

as seen on Server Fault - Search for 'Server Fault'
I am backing up a Windows 2003 Small Business Server with SP2. We are running Backup Exec 12 with SP4. Recently the backup job started failing on backing up the system state with the following error: V-79-57344-34110 - AOFO: Initialization failure on: "System?State". Advanced Open File Option… >>> More