I have trouble getting the max throughput out of my setup. The hardware is as follow :
dual Quad-Core AMD Opteron(tm) Processor 2376
16 GB DDR2 ECC RAM
dual Adaptec 52245 RAID controllers
48 1 TB SATA drives set up as 2 RAID-6 arrays (256KB stripe) + spares.
Software :
Plain vanilla 2.6.32.25 kernel, compiled for AMD-64, optimized for NUMA; Debian Lenny userland.
benchmarks run : disktest, bonnie++, dd, etc. All give the same results. No discrepancy here.
io scheduler used : noop. Yeah, no trick here.
Up until now I basically assumed that striping (RAID 0) several physical devices should augment performance roughly linearly. However this is not the case here :
each RAID array achieves about 780 MB/s write, sustained, and 1 GB/s read, sustained.
writing to both RAID arrays simultaneously with two different processes gives 750 + 750 MB/s, and reading from both gives 1 + 1 GB/s.
however when I stripe both arrays together, using either mdadm or lvm, the performance is about 850 MB/s writing and 1.4 GB/s reading. at least 30% less than expected!
running two parallel writer or reader processes against the striped arrays doesn't enhance the figures, in fact it degrades performance even further.
So what's happening here? Basically I ruled out bus or memory contention, because when I run dd on both drives simultaneously, aggregate write speed actually reach 1.5 GB/s and reading speed tops 2 GB/s.
So it's not the PCIe bus. I suppose it's not the RAM. It's not the filesystem, because I get exactly the same numbers benchmarking against the raw device or using XFS. And I also get exactly the same performance using either LVM striping and md striping.
What's wrong? What's preventing a process from going up to the max possible throughput? Is Linux striping defective? What other tests could I run?
