zfs meta data slowness

Eugene Grosbein eugen at grosbein.net
Wed Jul 22 13:27:19 UTC 2020


22.07.2020 20:02, mike tancsa wrote:
> 
> On 7/22/2020 1:29 AM, Eugene Grosbein wrote:
>> 22.07.2020 2:37, mike tancsa wrote:
>>
>>>> Something else special about the setup.
>>>> output of "top -b"
>>>>
>>> ports are right now being built in a VM, but the problem (zrepl hanging)
>>> and zfs list -t snapshots taking forever happens regardless
>>>
>>>   PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU
>>> COMMAND
>>>  4439 root         12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
>>> 98783 root          2  21    0    16M  5136K hdr->b   4   0:01   1.95% zfs
>>> 76489 root         21  23    0   738M    54M uwait    1   2:18   0.88% zrepl
>>> 98784 root          1  21    0    13M  3832K piperd   3   0:01   0.59% zfs
>>> 99563 root          1  20    0    13M  4136K zio->i   4   0:00   0.39% zfs
>> You need top -SHPI
> 
> last pid: 66981;  load averages:  3.44,  3.25,  3.34  up 1+22:49:34   
> 08:58:06
> 2056 threads:  12 running, 1988 sleeping, 3 zombie, 53 waiting
> CPU 0:  7.7% user,  0.1% nice, 15.0% system,  0.7% interrupt, 76.5% idle
> CPU 1:  9.9% user,  0.1% nice, 16.6% system,  0.1% interrupt, 73.2% idle
> CPU 2: 10.0% user,  0.1% nice, 17.5% system,  0.4% interrupt, 71.9% idle
> CPU 3: 10.3% user,  0.1% nice, 21.2% system,  0.1% interrupt, 68.2% idle
> CPU 4:  9.7% user,  0.1% nice, 15.6% system,  0.4% interrupt, 74.3% idle
> CPU 5: 10.2% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.3% idle
> CPU 6:  9.7% user,  0.1% nice, 16.6% system,  0.5% interrupt, 73.1% idle
> CPU 7: 10.1% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.4% idle
> Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free
> ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other
>      20G Compressed, 29G Uncompressed, 1.43:1 Ratio
> Swap: 20G Total, 20G Free
> 
>   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
>  4439 root        123   20  6167M  5856M CPU2     2  20.2H 100.00%
> bhyve{vcpu 0}
>  4439 root        122   20  6167M  5856M CPU3     3  20.2H  99.17%
> bhyve{vcpu 1}
>    11 root        155 ki31     0B   128K CPU0     0  35.9H  65.38%
> idle{idle: cpu0}
>    11 root        155 ki31     0B   128K CPU4     4  34.8H  63.38%
> idle{idle: cpu4}
> 66956 root         90    0    61M    54M CPU5     5   0:09  62.70% zfs
>    11 root        155 ki31     0B   128K CPU6     6  34.3H  58.06%
> idle{idle: cpu6}
>    11 root        155 ki31     0B   128K RUN      2  33.7H  54.49%
> idle{idle: cpu2}
>    11 root        155 ki31     0B   128K RUN      1  34.3H  53.76%
> idle{idle: cpu1}
>    11 root        155 ki31     0B   128K RUN      3  32.0H  53.47%
> idle{idle: cpu3}
>    11 root        155 ki31     0B   128K CPU7     7  32.0H  50.68%
> idle{idle: cpu7}
>    11 root        155 ki31     0B   128K RUN      5  32.0H  48.29%
> idle{idle: cpu5}
>    56 root        -12    -     0B  5168K -        1   5:49   9.67%
> zpool-zbackup2{zio_write_issue_3}
>    56 root        -12    -     0B  5168K -        3   5:48   9.57%
> zpool-zbackup2{zio_write_issue_5}
>    56 root        -12    -     0B  5168K -        4   5:48   9.47%
> zpool-zbackup2{zio_write_issue_4}
>    56 root        -12    -     0B  5168K -        7   5:48   9.47%
> zpool-zbackup2{zio_write_issue_2}
>    56 root        -12    -     0B  5168K -        0   5:49   9.38%
> zpool-zbackup2{zio_write_issue_1}
>    56 root        -12    -     0B  5168K -        1   5:48   9.38%
> zpool-zbackup2{zio_write_issue_0}
> 63392 root         23    0   729M    32M uwait    0   0:01   4.20%
> zrepl{zrepl}

It's hard to read due to wrapping plus, it is truncated.
Maybe additional flag -d3 or similar will help, combined with dedirection < /dev/null > top.out
so it won't use size of your terminal to wrap/truncate output.

Also, make sure you invoke top while "zfs" command is running.
Also, procstat -kk for pid of "zfs" command would be useful (but may occur pretty long).

I suppose it blocks waiting for some kernel lock and procstat would show details.



More information about the freebsd-stable mailing list