Very poor read performance compared to write performance on md(raid1) / crypt(luks) / lvm
- by Android5360
I'm experiencing very poor read performance over raid1/crypt/lvm. In the same time, write speeds are about 2x+ faster on the same setup. On another raid1 setup on the same machine I get normal read speeds (maybe because I'm not using cryptsetup).
OS related disks: sda + sdb. I have raid1 configuration with two disks, both are in place. I'm using LVM over the RAID. No encryption. Both disks are WD Green, 5400 rpm.
IO test results on this raid1:
dd if=/dev/zero of=/tmp/output.img3 bs=8k count=256k conv=fsync
- 2147483648 bytes (2.1 GB) copied, 22.3392 s, 96.1 MB/s
sync
echo 3 > /proc/sys/vm/drop_caches
dd if=/tmp/output.img3 of=/dev/null bs=8k
- 2147483648 bytes (2.1 GB) copied, 15.9 s, 135 MB/s
And here is the problematic setup (on the same machine). Currently I have only one sdc (WD Green, 5400rpm) configured in software raid1 + crypt (luks, serpent-xts-plain) + lvm. Tomorrow I will attach another disk (sdd) to complete this two-disk raid1 setup.
IO tests results on this raid1:
dd if=/dev/zero of=output.img3 bs=8k count=256k conv=fsync
2147483648 bytes (2.1 GB) copied, 17.7235 s, 121 MB/s
sync
echo 3 > /proc/sys/vm/drop_caches
dd if=output.img3 of=/dev/null bs=8k
2147483648 bytes (2.1 GB) copied, 36.2454 s, 59.2 MB/s
We can see that the read performance is very very bad (59MB/s compared to 135MB/s when using no encryption). Nothing is using the disks during benchmark. I can confirm this because I checked with iostat and dstat.
Details on the hardware:
disks: all are WD green, 5400rpm, 64mb cache.
cpu: FX-8350 at stock speed
ram: 4x4GB at 1066Mhz.
Details on the software:
OS: Debian Wheezy 7, amd64
mdadm: v3.2.5 - 18th May 2012
LVM version: 2.02.95(2) (2012-03-06)
LVM Library version: 1.02.74 (2012-03-06)
LVM Driver version: 4.22.0
cryptsetup: 1.4.3
Here is how I configured the slow raid1+crypt+lvm setup:
parted /dev/sdc
mklabel gpt
type: ext4
start: 2048s
end: -1
Now the raid, crypt and the lvm configuration:
mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdc
cryptsetup --cipher serpent-xts-plain luksFormat /dev/md1
cryptsetup luksOpen /dev/md1 md1_crypt
vgcreate vg_sql /dev/mapper/md1_crypt
lvcreate -l 100%VG vg_sql -n lv_sql
mkfs.ext4 /dev/mapper/vg_sql-lv-sql
mount /dev/mapper/vg_sql-lv_sql /sql
So guys, can you help me identify the reason and fix it? It has to be something with the cryptsetup as there is no such read slowdown on the other setup (sda+sdb) where no encryption is present. But I have no idea what to do.
Thanks!
