High I/O RAID on UFS2

[James Shank]

Hi FreeBSD Hackers,

I am running FreeBSD 11.1-RELEASE-p15 with a UFS2 filesystem block size 64K, with a RAID 6 of 16 SSD drives on a MRSAS compatible RAID card stripe size of 256K (and yes, I know about the misalignment) and 384GB memory on 48 real cores with hyper threading enabled.  I invite critiques of this configuration, but am more keen on getting help making it run efficiently.

I am running 11.1 because 11.2-RELEASE repeatably panics after failing somewhere after allocbuf() tries to allocate a page (sorry, I don’t still have the output from this state).

I am doing very read heavy operations, via multithreaded readers.  My writes tend to be small, but come in every 15 seconds or so.  I’d estimate read to write ratio of about 100:1 or more.

What I see happening is pagedaemon coming alive and bufferspacedaemon doing the same to clear the corresponding resources.  bufferspacedaemon runs nearly all the time at 100%.  I haven’t yet had much success with tuning things to keep enough resources free to allow my read rates to be sustained.

My read rates can hit and exceed 6GBps when everything is humming along, but only for short periods of time, then they drop off to much lower rates after the kernel starts shuffling resources.  For long periods of time, my best sustainable rate seems to be 3.5GBps, and I can’t seem to get the full read rate for sustained periods.

I have tried several things, but I believe now that I may be missing something that the more experienced gurus will find obvious.

Any pointers on where I should look next?

Thanks!

James