rdiff-backup is taking longer and longer every time it runs

Posted by Jakobud on Server Fault See other posts from Server Fault or by Jakobud
Published on 2010-03-17T19:46:23Z Indexed on 2010/03/17 19:51 UTC
Read the original article Hit count: 473

Filed under:

I've been running rdiff-backup for the past week or so, every night at 4am.

It started out not taking that long, but its taking longer and longer and longer every time its runs every night.

In some cases there are quite a few new and changed files and in other cases, not so much.

It started out taking < 10 minutes, and after a week its taking over 4.5 hrs to run. Take a look at my session stats below:

StartTime 1268046002.00 (Mon Mar  8 04:00:02 2010)
EndTime 1268046373.50 (Mon Mar  8 04:06:13 2010)
ElapsedTime 371.50 (6 minutes 11.50 seconds)
SourceFiles 213928
SourceFileSize 277271282225 (258 GB)
MirrorFiles 213914
MirrorFileSize 276693097638 (258 GB)
NewFiles 16
NewFileSize 578209911 (551 MB)
DeletedFiles 2
DeletedFileSize 2598 (2.54 KB)
ChangedFiles 27
ChangedSourceSize 5195150 (4.95 MB)
ChangedMirrorSize 5217876 (4.98 MB)
IncrementFiles 0
IncrementFileSize 0 (0 bytes)
TotalDestinationSizeChange 578184587 (551 MB)
Errors 0


StartTime 1268132402.00 (Tue Mar  9 04:00:02 2010)
EndTime 1268134341.29 (Tue Mar  9 04:32:21 2010)
ElapsedTime 1939.29 (32 minutes 19.29 seconds)
SourceFiles 213963
SourceFileSize 307959842562 (287 GB)
MirrorFiles 213928
MirrorFileSize 277271282225 (258 GB)
NewFiles 37
NewFileSize 31265005547 (29.1 GB)
DeletedFiles 2
DeletedFileSize 576511960 (550 MB)
ChangedFiles 25
ChangedSourceSize 5243761 (5.00 MB)
ChangedMirrorSize 5177011 (4.94 MB)
IncrementFiles 65
IncrementFileSize 577266412 (551 MB)
TotalDestinationSizeChange 31265826749 (29.1 GB)
Errors 0


StartTime 1268218802.00 (Wed Mar 10 04:00:02 2010)
EndTime 1268225230.15 (Wed Mar 10 05:47:10 2010)
ElapsedTime 6428.15 (1 hour 47 minutes 8.15 seconds)
SourceFiles 213971
SourceFileSize 307960643843 (287 GB)
MirrorFiles 213963
MirrorFileSize 307959842562 (287 GB)
NewFiles 9
NewFileSize 694087 (678 KB)
DeletedFiles 1
DeletedFileSize 894 (894 bytes)
ChangedFiles 38
ChangedSourceSize 30656167797 (28.6 GB)
ChangedMirrorSize 30656059709 (28.6 GB)
IncrementFiles 48
IncrementFileSize 289278151 (276 MB)
TotalDestinationSizeChange 290079432 (277 MB)
Errors 0


StartTime 1268305202.00 (Thu Mar 11 04:00:02 2010)
EndTime 1268312788.15 (Thu Mar 11 06:06:28 2010)
ElapsedTime 7586.15 (2 hours 6 minutes 26.15 seconds)
SourceFiles 213971
SourceFileSize 307960643779 (287 GB)
MirrorFiles 213971
MirrorFileSize 307960643843 (287 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 15
ChangedSourceSize 30650824127 (28.5 GB)
ChangedMirrorSize 30650824191 (28.5 GB)
IncrementFiles 16
IncrementFileSize 689437042 (657 MB)
TotalDestinationSizeChange 689436978 (657 MB)
Errors 0


StartTime 1268391601.00 (Fri Mar 12 04:00:01 2010)
EndTime 1268400145.85 (Fri Mar 12 06:22:25 2010)
ElapsedTime 8544.85 (2 hours 22 minutes 24.85 seconds)
SourceFiles 213974
SourceFileSize 307960784445 (287 GB)
MirrorFiles 213971
MirrorFileSize 307960643779 (287 GB)
NewFiles 5
NewFileSize 269587 (263 KB)
DeletedFiles 2
DeletedFileSize 128921 (126 KB)
ChangedFiles 16
ChangedSourceSize 30650815948 (28.5 GB)
ChangedMirrorSize 30650815948 (28.5 GB)
IncrementFiles 26
IncrementFileSize 818279963 (780 MB)
TotalDestinationSizeChange 818420629 (781 MB)
Errors 0


StartTime 1268478002.00 (Sat Mar 13 04:00:02 2010)
EndTime 1268488740.33 (Sat Mar 13 06:59:00 2010)
ElapsedTime 10738.33 (2 hours 58 minutes 58.33 seconds)
SourceFiles 213974
SourceFileSize 307960784238 (287 GB)
MirrorFiles 213974
MirrorFileSize 307960784445 (287 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 16
ChangedSourceSize 30650823920 (28.5 GB)
ChangedMirrorSize 30650824127 (28.5 GB)
IncrementFiles 19
IncrementFileSize 1041846015 (994 MB)
TotalDestinationSizeChange 1041845808 (994 MB)
Errors 0


StartTime 1268560803.00 (Sun Mar 14 04:00:03 2010)
EndTime 1268573194.82 (Sun Mar 14 07:26:34 2010)
ElapsedTime 12391.82 (3 hours 26 minutes 31.82 seconds)
SourceFiles 213974
SourceFileSize 307960784238 (287 GB)
MirrorFiles 213974
MirrorFileSize 307960784238 (287 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 14
ChangedSourceSize 30650815948 (28.5 GB)
ChangedMirrorSize 30650815948 (28.5 GB)
IncrementFiles 15
IncrementFileSize 1116911810 (1.04 GB)
TotalDestinationSizeChange 1116911810 (1.04 GB)
Errors 0


StartTime 1268647203.00 (Mon Mar 15 04:00:03 2010)
EndTime 1268662364.88 (Mon Mar 15 08:12:44 2010)
ElapsedTime 15161.88 (4 hours 12 minutes 41.88 seconds)
SourceFiles 214010
SourceFileSize 307963430178 (287 GB)
MirrorFiles 213974
MirrorFileSize 307960784238 (287 GB)
NewFiles 37
NewFileSize 2684172 (2.56 MB)
DeletedFiles 1
DeletedFileSize 5348 (5.22 KB)
ChangedFiles 32
ChangedSourceSize 30656134913 (28.6 GB)
ChangedMirrorSize 30656167797 (28.6 GB)
IncrementFiles 71
IncrementFileSize 1316460362 (1.23 GB)
TotalDestinationSizeChange 1319106302 (1.23 GB)
Errors 0


StartTime 1268733603.00 (Tue Mar 16 04:00:03 2010)
EndTime 1268750396.76 (Tue Mar 16 08:39:56 2010)
ElapsedTime 16793.76 (4 hours 39 minutes 53.76 seconds)
SourceFiles 214010
SourceFileSize 307963430156 (287 GB)
MirrorFiles 214010
MirrorFileSize 307963430178 (287 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 15
ChangedSourceSize 30650823898 (28.5 GB)
ChangedMirrorSize 30650823920 (28.5 GB)
IncrementFiles 16
IncrementFileSize 936032413 (893 MB)
TotalDestinationSizeChange 936032391 (893 MB)
Errors 0


StartTime 1268820002.00 (Wed Mar 17 04:00:02 2010)
EndTime 1268834619.90 (Wed Mar 17 08:03:39 2010)
ElapsedTime 14617.90 (4 hours 3 minutes 37.90 seconds)
SourceFiles 214010
SourceFileSize 307963430156 (287 GB)
MirrorFiles 214010
MirrorFileSize 307963430156 (287 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 14
ChangedSourceSize 30650815948 (28.5 GB)
ChangedMirrorSize 30650815948 (28.5 GB)
IncrementFiles 15
IncrementFileSize 1289272860 (1.20 GB)
TotalDestinationSizeChange 1289272860 (1.20 GB)
Errors 0

Is this common behavior? It's only about 300gigs of data total. One important thing to note though, rdiff-backup is backup up stuff over to a USB2 external drive. Maybe thats why its so slow?

© Server Fault or respective owner

Related posts about rdiff-backup