zfs meta data slowness

[mike tancsa]

On 7/22/2020 1:29 AM, Eugene Grosbein wrote:
> 22.07.2020 2:37, mike tancsa wrote:
>
>>> Something else special about the setup.
>>> output of "top -b"
>>>
>> ports are right now being built in a VM, but the problem (zrepl hanging)
>> and zfs list -t snapshots taking forever happens regardless
>>
>>   PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU
>> COMMAND
>>  4439 root         12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
>> 98783 root          2  21    0    16M  5136K hdr->b   4   0:01   1.95% zfs
>> 76489 root         21  23    0   738M    54M uwait    1   2:18   0.88% zrepl
>> 98784 root          1  21    0    13M  3832K piperd   3   0:01   0.59% zfs
>> 99563 root          1  20    0    13M  4136K zio->i   4   0:00   0.39% zfs
> You need top -SHPI

last pid: 66981;  load averages:  3.44,  3.25,  3.34  up 1+22:49:34   
08:58:06
2056 threads:  12 running, 1988 sleeping, 3 zombie, 53 waiting
CPU 0:  7.7% user,  0.1% nice, 15.0% system,  0.7% interrupt, 76.5% idle
CPU 1:  9.9% user,  0.1% nice, 16.6% system,  0.1% interrupt, 73.2% idle
CPU 2: 10.0% user,  0.1% nice, 17.5% system,  0.4% interrupt, 71.9% idle
CPU 3: 10.3% user,  0.1% nice, 21.2% system,  0.1% interrupt, 68.2% idle
CPU 4:  9.7% user,  0.1% nice, 15.6% system,  0.4% interrupt, 74.3% idle
CPU 5: 10.2% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.3% idle
CPU 6:  9.7% user,  0.1% nice, 16.6% system,  0.5% interrupt, 73.1% idle
CPU 7: 10.1% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.4% idle
Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free
ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other
     20G Compressed, 29G Uncompressed, 1.43:1 Ratio
Swap: 20G Total, 20G Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 4439 root        123   20  6167M  5856M CPU2     2  20.2H 100.00%
bhyve{vcpu 0}
 4439 root        122   20  6167M  5856M CPU3     3  20.2H  99.17%
bhyve{vcpu 1}
   11 root        155 ki31     0B   128K CPU0     0  35.9H  65.38%
idle{idle: cpu0}
   11 root        155 ki31     0B   128K CPU4     4  34.8H  63.38%
idle{idle: cpu4}
66956 root         90    0    61M    54M CPU5     5   0:09  62.70% zfs
   11 root        155 ki31     0B   128K CPU6     6  34.3H  58.06%
idle{idle: cpu6}
   11 root        155 ki31     0B   128K RUN      2  33.7H  54.49%
idle{idle: cpu2}
   11 root        155 ki31     0B   128K RUN      1  34.3H  53.76%
idle{idle: cpu1}
   11 root        155 ki31     0B   128K RUN      3  32.0H  53.47%
idle{idle: cpu3}
   11 root        155 ki31     0B   128K CPU7     7  32.0H  50.68%
idle{idle: cpu7}
   11 root        155 ki31     0B   128K RUN      5  32.0H  48.29%
idle{idle: cpu5}
   56 root        -12    -     0B  5168K -        1   5:49   9.67%
zpool-zbackup2{zio_write_issue_3}
   56 root        -12    -     0B  5168K -        3   5:48   9.57%
zpool-zbackup2{zio_write_issue_5}
   56 root        -12    -     0B  5168K -        4   5:48   9.47%
zpool-zbackup2{zio_write_issue_4}
   56 root        -12    -     0B  5168K -        7   5:48   9.47%
zpool-zbackup2{zio_write_issue_2}
   56 root        -12    -     0B  5168K -        0   5:49   9.38%
zpool-zbackup2{zio_write_issue_1}
   56 root        -12    -     0B  5168K -        1   5:48   9.38%
zpool-zbackup2{zio_write_issue_0}
63392 root         23    0   729M    32M uwait    0   0:01   4.20%
zrepl{zrepl}