2 bonnies can stop disk activity permanently

[Bruce Evans]

On Mon, 9 Oct 2006, Fluffles.net wrote:

> I'm the "veronica" Arne mentioned in the freebsd-fs mailinglist.
> Regarding the effectiveness of a higher blocksize, these are my findings:
>
> areca RAID5 (8x da, 128KB stripe, default newfs, NCQ enabled)
>              -------Sequential Output-------- ---Sequential Input--
> --Random--
>              -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
> --Seeks---
> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
> /sec %CPU
> ARC8xR5  8480 119973 91.3 247178 58.6 67862 17.5 90426 86.9 172490 24.0
> 120.7  0.5
>
> areca RAID5 (8x da, 128KB stripe, 64KB blocksize newfs, NCQ enabled)
>              -------Sequential Output-------- ---Sequential Input--
> --Random--
>              -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
> --Seeks---
> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
> /sec %CPU
> ARC8xR5  8480 128920 97.8 265920 58.9 116787 31.0 103261 97.8 392970
> 53.8 119.8  0.6
>
> As you can see, the block read increased from ~172MB/s to ~392MB/s,
> quite significant increase. Also the reqrite speed increased from
> ~67MB/s to ~116MB/s.
>
> Ofcourse these tests are on a brand clean filesystem, which might not
> tally with real-life crowded filesystems. But at least there is much
> ...

This is a bit surprising.  FreeBSD is supposed to cluster the i/o so
that (especially for large files on new file systems) almost all i/o
is done in blocks of size 64K or 128K.

I suspect the problems are that the 64K-block i/o is usually perfectly
misaligned unless the fs itself has 64K-blocks and the fs's partition
starts on a 64K-block boundary, and that some hardware or firmware
(mainly RAIDs) want the blocks to be aligned.  I'm not very familiar
with RAIDs but think it would take a fairly advanced/expensive one to
reblock all the i/at so that the alignment doesn't matter.  It would
take more advanced/complicated clustering code or better buffering code
than FreeBSD has to do the reblocking at the clustering or buffering
level.  Perhaps even 64K-blocks are too small with your RAID's stripe
size of 128K.

Bruce