kern/139039: [zfs] zpool scrub makes system unbearably slow

[Nathaniel W Filardo]

On Fri, Sep 25, 2009 at 06:28:56PM +0000, pjd at freebsd.org wrote:
> Synopsis: [zfs] zpool scrub makes system unbearably slow
> 
> State-Changed-From-To: open->feedback
> State-Changed-By: pjd
> State-Changed-When: ptk 25 wrz 2009 18:27:48 UTC
> State-Changed-Why: 
> Could you tell which threads are consuming most CPU time?
> Pasting first few lines from 'top -SH' should be enough.
> 
> 
> Responsible-Changed-From-To: freebsd-fs->pjd
> Responsible-Changed-By: pjd
> Responsible-Changed-When: ptk 25 wrz 2009 18:27:48 UTC
> Responsible-Changed-Why: 
> I'll take this one.
> 
> http://www.freebsd.org/cgi/query-pr.cgi?pr=139039

Thanks for looking at this.

The system here is trying to build OpenLDAP in a jail, but that isn't
frequently in the top.  Typical output is...

hydra# top -jSHP
267 processes: 15 running, 236 sleeping, 16 waiting
CPU 0:  0.3% user,  0.0% nice, 97.4% system,  2.3% interrupt,  0.0% idle
CPU 1: 10.7% user,  0.0% nice, 44.4% system,  1.8% interrupt, 43.0% idle
Mem: 147M Active, 242M Inact, 926M Wired, 4008K Cache, 213M Buf, 672M Free
Swap: 4096M Total, 4096M Free

  PID JID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11   0 root     171 ki31     0K    64K RUN     1 376:41 57.18% {idle: cpu1}
    0   0 root     -16    0     0K  3520K CPU0    0   6:04 13.96% {spa_zio_7}
    0   0 root     -16    0     0K  3520K -       0   5:58 13.33% {spa_zio_0}
    0   0 root     -16    0     0K  3520K -       0   5:58 13.13% {spa_zio_3}
    0   0 root     -16    0     0K  3520K -       0   6:01 13.09% {spa_zio_5}
    0   0 root     -16    0     0K  3520K RUN     0   6:01 13.04% {spa_zio_2}
    0   0 root     -16    0     0K  3520K RUN     0   6:00 13.04% {spa_zio_6}
    0   0 root     -16    0     0K  3520K -       0   5:59 12.65% {spa_zio_1}
    0   0 root     -16    0     0K  3520K -       1   6:00 12.11% {spa_zio_4}
   42   0 root      -8    -     0K   480K spa->s  0   4:50  8.54% {txg_thread_enter}
    4   0 root      -8    -     0K    32K -       0   2:13  1.95% g_down
   12   0 root     -40    -     0K   544K WAIT    0   1:25  0.98% {swi2: cambio}
    0   0 root     -16    0     0K  3520K -       0   0:24  0.20% {spa_zio_7}
    0   0 root     -16    0     0K  3520K -       1   0:23  0.20% {spa_zio_3}
   12   0 root     -64    -     0K   544K RUN     0   0:45  0.15% {vec1860: mpt0}
    0   0 root     -16    0     0K  3520K -       1   0:58  0.10% {spa_zio}
   12   0 root     -32    -     0K   544K WAIT    0   1:58  0.05% {swi4: clock}
   42   0 root      -8    -     0K   480K tx->tx  1   0:31  0.05% {txg_thread_enter}
   11   0 root     171 ki31     0K    64K RUN     0 774:48  0.00% {idle: cpu0}

The only thing that seems odd to me is that CPU1 is sitting essientially idle
(I have never seen CPU0 be idle when the system is scrubbing).  The spa_zio_*
threads do in fact run on CPU1, but seemingly rarely.

--nwf;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20090925/dfabdeb4/attachment.pgp