zfs, a directory that used to hold lot of files and listing pause

[Steven Hartland]

In your case there your vdev (ada0) is saturated with writes from postgres.

You should consider more / faster disks.

You might also want to consider enabling lz4 compression on the PG 
volume as its works well in IO bound situations.


On 21/10/2016 01:54, Jonathan Chen wrote:
> On 21 October 2016 at 12:56, Steven Hartland <killing at multiplay.co.uk> wrote:
> [...]
>> When you see the stalling what does gstat -pd and top -SHz show?
> On my dev box:
>
> 1:38pm# uname -a
> FreeBSD irontree 10.3-STABLE FreeBSD 10.3-STABLE #0 r307401: Mon Oct
> 17 10:17:22 NZDT 2016     root at irontree:/usr/obj/usr/src/sys/GENERIC
> amd64
> 1:49pm# gstat -pd
> dT: 1.004s  w: 1.000s
>   L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps
> ms/d   %busy Name
>      0      0      0      0    0.0      0      0    0.0      0      0
>   0.0    0.0| cd0
>     18    618      1    128   41.4    606  52854   17.2      0      0
>   0.0  100.5| ada0
> ^C
> 1:49pm# top -SHz
> last pid: 83284;  load averages:  0.89,  0.68,  0.46
>                                                                   up
> 4+03:11:32  13:49:05
> 565 processes: 9 running, 517 sleeping, 17 zombie, 22 waiting
> CPU:  3.7% user,  0.0% nice,  1.9% system,  0.0% interrupt, 94.3% idle
> Mem: 543M Active, 2153M Inact, 11G Wired, 10M Cache, 2132M Free
> ARC: 7249M Total, 1325M MFU, 4534M MRU, 906M Anon, 223M Header, 261M Other
> Swap: 32G Total, 201M Used, 32G Free
>
>    PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
> 83149 postgres    38    0  2197M   528M zio->i  5   1:13  23.19% postgres
> 83148 jonc        22    0 36028K 13476K select  2   0:11   3.86% pg_restore
>    852 postgres    20    0  2181M  2051M select  5   0:27   0.68% postgres
>      0 root       -15    -     0K  4240K -       6   0:50   0.49%
> kernel{zio_write_issue_}
>      0 root       -15    -     0K  4240K -       6   0:50   0.39%
> kernel{zio_write_issue_}
>      0 root       -15    -     0K  4240K -       6   0:50   0.39%
> kernel{zio_write_issue_}
>      0 root       -15    -     0K  4240K -       7   0:50   0.39%
> kernel{zio_write_issue_}
>      0 root       -15    -     0K  4240K -       7   0:50   0.39%
> kernel{zio_write_issue_}
>      0 root       -15    -     0K  4240K -       7   0:50   0.29%
> kernel{zio_write_issue_}
>      3 root        -8    -     0K   112K zio->i  6   1:50   0.20%
> zfskern{txg_thread_enter}
>     12 root       -88    -     0K   352K WAIT    0   1:07   0.20%
> intr{irq268: ahci0}
>      0 root       -16    -     0K  4240K -       4   0:29   0.20%
> kernel{zio_write_intr_4}
>      0 root       -16    -     0K  4240K -       7   0:29   0.10%
> kernel{zio_write_intr_6}
>      0 root       -16    -     0K  4240K -       0   0:29   0.10%
> kernel{zio_write_intr_1}
>      0 root       -16    -     0K  4240K -       5   0:29   0.10%
> kernel{zio_write_intr_2}
>      0 root       -16    -     0K  4240K -       1   0:29   0.10%
> kernel{zio_write_intr_5}
> ...
>
> Taking another look at the internal dir structure for postgres, I'm
> not too sure whether this is related to the original poster's problem
> though.
>
> Cheers.