pw operations slow under zfs load

[Mark Blackman]

> On 4 Aug 2015, at 22:07, javocado <javocado at gmail.com> wrote:
> 
> Hi,
> 
> We have a pretty busy ZFS pool running on an 8.3 AMD system. We are
> noticing that when the pool is busy pw-related operations seem to  take a
> long time to complete:
> 
> # time pw unlock 1000
> 0.007u 0.036s 0:39.72 0.0%      45+1953k 0+113io 0pf+0w
> 
> # time pw lock 1000
> 0.032u 0.022s 1:09.63 0.0%      24+1132k 0+114io 0pf+0w
> 
> Wile the command is running, we note that the process is locked in the D
> state:
> 
> root  85051  0.0  0.0  5832   960   0  D+    1:53PM   0:00.02
> /usr/sbin/pwd_mkdb -u 1000 /etc/master.passwd
> 
> We also note that there is next to 0 disk activity on the boot volume:
> 
> # gstat -f ad
> 
> dT: 1.005s  w: 1.000s  filter: ad
> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>    0      0      0      0    0.0      0      0    0.0    0.0| ad6
>    0      0      0      0    0.0      0      0    0.0    0.0| ad8
>    0      0      0      0    0.0      0      0    0.0    0.0| ad10
>    0      0      0      0    0.0      0      0    0.0    0.0| ad12
> 
> And plenty of free mem:
> 
> Mem: 400M Active, 3391M Inact, 128G Wired, 1935M Cache, 14G Buf, 6055M Free
> 
> So, what's going on here? How does a busy pool with it's own set of drives
> (which operate off an HBA) affect the speed of operations involving the
> boot volume (a UFS-formatted SSD connected to the mobo)?

Maybe some ktrace operations are in order? I remember, about 15 years ago, discovering that that ‘-s’ option can make a big difference.

     -s cachesize
           Specify in megabytes the size of the memory cache used by the hash-
           ing library.  On systems with a large user base, a small cache size
           can lead to prohibitively long database file rebuild times.  As a
           rough guide, the memory usage of pwd_mkdb in megabytes will be a
           little bit more than twice the figure specified here.  The default
           is 2 megabytes.

- Mark