Very slow write performance on Debian 6.0 (AMD64) with DMCRYPT/LVM/RAID1
- by jdelic
I'm seeing very strange performance characteristics on one of my servers. This server is running a simple two-disk software-RAID1 setup with LVM spanning /dev/md0. One of the logical volumes /dev/vg0/secure is encrypted using dmcrypt with LUKS and mounted with the sync and noatimes flag. Writing to that volume is incredibly slow at 1.8 MB/s and the CPU usage stays near 0%. There are 8 crpyto/1-8 processes running (it's a Intel Quadcore CPU).
I hope that someone on serverfault has seen this before :-(.
uname -a
2.6.32-5-xen-amd64 #1 SMP Tue Mar 8 00:01:30 UTC 2011 x86_64 GNU/Linux
Interestingly, when I read from the device I get good performance numbers:
reading without encryption:
$ dd if=/dev/vg0/secure of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes (6.6 GB) copied, 68.8951 s, 95.1 MB/s
reading with encryption:
$ dd if=/dev/mapper/secure of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes (6.6 GB) copied, 69.7116 s, 94.0 MB/s
However, when I try to write to the device:
$ dd if=/dev/zero of=./test bs=64k
8809+0 records in
8809+0 records out
577306624 bytes (577 MB) copied, 321.861 s, 1.8 MB/s
Also, when I read I see CPU usage, when I write, the CPU stays at almost 0% usage. Here is output of cryptsetup luksDump:
LUKS header information for /dev/vg0/secure
Version: 1
Cipher name: aes
Cipher mode: cbc-essiv:sha256
Hash spec: sha1
Payload offset: 2056
MK bits: 256
MK digest: dd 62 b9 a5 bf 6c ec 23 36 22 92 4c 39 f8 d6 5d c1 3a b7 37
MK salt: cc 2e b3 d9 fb e3 86 a1 bb ab eb 9d 65 df b3 dd
d9 6b f4 49 de 8f 85 7d 3b 1c 90 83 5d b2 87 e2
MK iterations: 44500
UUID: a7c9af61-d9f0-4d3f-b422-dddf16250c33
Key Slot 0: ENABLED
Iterations: 178282
Salt: 60 24 cb be 5c 51 9f b4 85 64 3d f8 07 22 54 d4
1a 5f 4c bc 4b 82 76 48 d8 a2 d2 6a ee 13 d7 5d
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
