Why is domU faster than dom0 on IO?

Posted by Paco on Server Fault See other posts from Server Fault or by Paco
Published on 2013-06-11T10:27:27Z Indexed on 2013/07/01 16:23 UTC
Read the original article Hit count: 219

Filed under:
|
|
|
|

I have installed debian 7 on a physical machine. This is the configuration of the machine:

  • 3 hard drives using RAID 5
  • Strip element size: 1M
  • Read policy: Adaptive read ahead
  • Write policy: Write Through
  • /boot 200 MB ext2
  • / 15 GB ext3
  • SWAP 10GB
  • LVM rest (~500GB) emphasized text I installed postgresql, created a big database (over 1GB). I have an SQL request that takes a lot of time to run (a SELECT statement, so it only reads data from the database). This request takes approximately 5.5 seconds to run. Then, I installed XEN, created a domU, with another debian distro. On this OS, I also installed postgresql, with the same database. The same SQL request takes only 2.5 seconds to run.

I checked the kernel on both dom0 and domU. uname-a returns "Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2+deb7u2 x86_64 GNU/Linux" on both systems. I checked the kernel parameters, which are approximately the same. For those that are relevant, I changed their values to make them match on both systems using sysctl. I saw no changes (the requests still take the same amount of time).

After this, I checked the file systems. I used ext3 on domU. Still no changes.

I installed hdparm, and ran hdparm -Tt on both systems, on all my partitions on both systems, and I get similar results.

Now, I am stuck, I don't know what is different, and what could be the cause of such a big difference.

Additional Info:

  • Debian runs on a Dell server PowerEdge 2950
  • postgresql: 9.1.9 (both dom0 and domU)
  • xen-linux-system: 3.2.0
  • xen-hypervisor: 4.1

Thanks

EDIT:

As Krzysztof Ksiezyk suggested, it might be due to some file caching system. I ran the dd command to test both the read and write speed.

Here is domU:

root@test1:~# dd if=/dev/zero of=/root/dd count=5MB bs=1MB
^C2020+0 records in
2020+0 records out
2020000000 bytes (2.0 GB) copied, 18.8289 s, 107 MB/s

root@test1:~# dd if=/root/dd of=/dev/null count=5MB bs=1MB
2020+0 records in
2020+0 records out
2020000000 bytes (2.0 GB) copied, 15.0549 s, 134 MB/s

And here is dom0:

root@debian:~# dd if=/dev/zero of=/root/dd count=5MB bs=1MB
^C1693+0 records in
1693+0 records out
1693000000 bytes (1.7 GB) copied, 8.87281 s, 191 MB/s

root@debian:~# dd if=/root/dd of=/dev/null count=5MB bs=1MB
1693+0 records in
1693+0 records out
1693000000 bytes (1.7 GB) copied, 0.501509 s, 3.4 GB/s

What can be the cause of this caching system? And how can we "fix" it? Can we apply it to dom0?

EDIT 2:

I switched my virtual disk type. To do so I followed this article.

I did a dd if=/dev/vg0/test1-disk of=/mnt/test1-disk.img bs=16M Then in /etc/xen/test1.cfg, I changed the disk parameter to use file: instead of phy: it should have removed the file caching, but I still get the same numbers (domU being much faster for Postgres)

© Server Fault or respective owner

Related posts about debian

Related posts about raid