zfs listing and CPU

[Eugene M. Zheganin]

Hi,

On 12.08.2017 20:50, Paul Kraus wrote:
>> On Aug 11, 2017, at 2:28 AM, Eugene M. Zheganin <emz at norma.perm.ru> wrote:
>>
>> Why does the zfs listing eat so much of the CPU ?
>> 47114 root           1  20    0 40432K  3840K db->db  4   0:05 26.84% zfs
>> 47099 root           1  20    0 40432K  3840K zio->i 17   0:05 26.83% zfs
>> 47106 root           1  20    0 40432K  3840K db->db 21   0:05 26.81% zfs
>> 47150 root           1  20    0 40432K  3428K db->db 13   0:03 26.31% zfs
>> 47141 root           1  20    0 40432K  3428K zio->i 28   0:03 26.31% zfs
>> 47135 root           1  20    0 40432K  3312K g_wait  9   0:03 25.51% zfs
>> This is from winter 2017 11-STABLE (r310734), one of the 'zfs'es is cloning, and all the others are 'zfs list -t all'. I have like 25 gigs of free RAM, do I have any chance of speeding this up using may be some caching or some sysctl tuning ? We are using a simple ZFS web API that may issue concurrent or sequential listing requests, so as you can see they sometimes do stack.
> How many snapshots do you have ? I have only seen this behavior with LOTS (not hundreds, but thousands) of snapshots.
[root at san1:~]# zfs list -t snapshot | wc -l
       88
> What does your `iostat -x 1` look like ? I expect that you are probably saturating your drives with random I/O.

Well, it's really long, and the disks are busy with random i/o indeed, 
but byst only for 20-30%. As about iostat - it's really long, because I 
have hundreds (not thousands) of zvols, and they do show up in iostat 
-x. But nothing unusual besides that.


Thanks.

Eugene.