I'm going mad over figuring out what the problem could be with one of our storage boxes.
With a simple fio script I'm testing random writes using bs=1M and direct=1. The SSD is a Samsung 840pro attached to an LSI HBA (3Gbit/s ports).
This is the result I'm getting under FreeBSD 9.1:
WRITE: io=13169MB, aggrb=224743KB/s, minb=224743KB/s, maxb=224743KB/s, mint=60002msec, maxt=60002msec
This is regardless of sync being set to 0 or 1.
On linux, this is the result with sync=0:
WRITE: io=14828MB, aggrb=253060KB/s, minb=253060KB/s, maxb=253060KB/s, mint=60001msec, maxt=60001msec
and with sync=1:
WRITE: io=6360.0MB, aggrb=108542KB/s, minb=108542KB/s, maxb=108542KB/s, mint=60001msec, maxt=60001msec
My understanding is that since I'm operating on the raw block device, O_SYNC should not make any difference - there's no filesystem, any barrier, anything between the writes and the drive itself. Especially with O_DIRECT|O_SYNC set.
Any ideas?
For reference, here's the fio script I'm testing with:
[global]
bs=1M
ioengine=sync
iodepth=4
size=16g
direct=1
runtime=60
filename=/dev/sdh
sync=1
[rand-write]
rw=randwrite
stonewall
