Linux RAID-0 performance doesn't scale up over 1 GB/s

Posted by wazoox on Server Fault See other posts from Server Fault or by wazoox
Published on 2010-12-17T15:23:34Z Indexed on 2010/12/23 13:55 UTC
Read the original article Hit count: 400

Filed under:
|
|

I have trouble getting the max throughput out of my setup. The hardware is as follow :

  • dual Quad-Core AMD Opteron(tm) Processor 2376
  • 16 GB DDR2 ECC RAM
  • dual Adaptec 52245 RAID controllers
  • 48 1 TB SATA drives set up as 2 RAID-6 arrays (256KB stripe) + spares.

Software :

  • Plain vanilla 2.6.32.25 kernel, compiled for AMD-64, optimized for NUMA; Debian Lenny userland.
  • benchmarks run : disktest, bonnie++, dd, etc. All give the same results. No discrepancy here.
  • io scheduler used : noop. Yeah, no trick here.

Up until now I basically assumed that striping (RAID 0) several physical devices should augment performance roughly linearly. However this is not the case here :

  • each RAID array achieves about 780 MB/s write, sustained, and 1 GB/s read, sustained.
  • writing to both RAID arrays simultaneously with two different processes gives 750 + 750 MB/s, and reading from both gives 1 + 1 GB/s.
  • however when I stripe both arrays together, using either mdadm or lvm, the performance is about 850 MB/s writing and 1.4 GB/s reading. at least 30% less than expected!
  • running two parallel writer or reader processes against the striped arrays doesn't enhance the figures, in fact it degrades performance even further.

So what's happening here? Basically I ruled out bus or memory contention, because when I run dd on both drives simultaneously, aggregate write speed actually reach 1.5 GB/s and reading speed tops 2 GB/s.

So it's not the PCIe bus. I suppose it's not the RAM. It's not the filesystem, because I get exactly the same numbers benchmarking against the raw device or using XFS. And I also get exactly the same performance using either LVM striping and md striping.

What's wrong? What's preventing a process from going up to the max possible throughput? Is Linux striping defective? What other tests could I run?

© Server Fault or respective owner

Related posts about linux

Related posts about raid