Kind of slow IO under pressure?

[Ivan Voras]

I'm running bonnie++ on UFS hosted on a FC SAN, and during the "rewrite" 
phase of the benchmark (bonnie++ is deliberately trying to trash disk 
caches), the situation from "top" is like this:

    21 root      76    -     0K    16K qsleep  1   2:48 44.38% bufdaemon
  1274 root      76    0 11296K  1716K getblk  7   3:35 41.06% bonnie++
     3 root      -8    -     0K    16K -       2   1:44 21.78% g_up

With disk rates of about 40 MB/s in either direction and around 300 IOPS.

Together these three processes take up a whole CPU core and bonnie++ is 
single-threaded, which sort of suggests that this supposedly IO 
benchmark is actually CPU-bound on this 8-core 2.4 GHz system.

Transaction sizes are curiously not-quite-64KiB:

        tty             da0              da1              da2 
   cpu
  tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy 
in id
    0   380 15.93  28  0.44  63.75 1242 77.30   1.93   0  0.00   0  0  9 
  0 91
    0  1510 16.00   3  0.05  63.86 1148 71.56   0.00   0  0.00   0  0 10 
  0 90
    0  1063 16.00   2  0.03  63.96 1093 68.24   0.00   0  0.00   0  0 14 
  0 86
    0   357  0.00   0  0.00  63.95 1024 63.95   0.00   0  0.00   0  0 15 
  0 85

Interestingly, binding (with cpuset) all three processes to the same CPU 
makes things very strange, with bufdaemon climbing to 90% CPU usage and 
bandwidth dropping to ~~ 5 MB/s (but not completely stalling). Relaxing 
bufdeamon binds to two CPUs makes things normal again.

On the "read" phase the situation is:

  1274 root      61    0 11296K  1704K getblk  0   7:05 26.56% bonnie++
    18 root      44    -     0K    16K psleep  4   0:09  2.88% pagedaemon
     3 root      -8    -     0K    16K -       0   3:31  1.66% g_up
     4 root      -8    -     0K    16K -       2   0:17  1.66% g_down
    12 root     -40    -     0K   432K WAIT    5   0:11  1.27% {swi2: 
cambio}

i.e. no CPU hogging and the performance is ~~ 140 MB/s. The write phase 
is similar, nothing unusual.