I'm going mad over figuring out what the problem could be with one of our storage boxes. With a simple fio script I'm testing random writes using bs=1M and direct=1. The SSD is a Samsung 840pro attached to an LSI HBA (3Gbit/s ports).

This is the result I'm getting under FreeBSD 9.1:

WRITE: io=13169MB, aggrb=224743KB/s, minb=224743KB/s, maxb=224743KB/s, mint=60002msec,   maxt=60002msec

This is regardless of sync being set to 0 or 1.

On linux, this is the result with sync=0:

WRITE: io=14828MB, aggrb=253060KB/s, minb=253060KB/s, maxb=253060KB/s, mint=60001msec, maxt=60001msec

and with sync=1:

WRITE: io=6360.0MB, aggrb=108542KB/s, minb=108542KB/s, maxb=108542KB/s, mint=60001msec, maxt=60001msec

My understanding is that since I'm operating on the raw block device, O_SYNC should not make any difference - there's no filesystem, any barrier, anything between the writes and the drive itself. Especially with O_DIRECT|O_SYNC set.

Any ideas?

For reference, here's the fio script I'm testing with:

[global]
bs=1M
ioengine=sync
iodepth=4
size=16g
direct=1
runtime=60
filename=/dev/sdh
sync=1

[rand-write]
rw=randwrite
stonewall