ZFS and large directories - caveat report

[Ivan Voras]

On 21 July 2011 21:45, Ivan Voras <ivoras at freebsd.org> wrote:
> On 21 July 2011 21:36, Luiz Otavio O Souza <lists.br at gmail.com> wrote:
>
>> But i guess i need to increase the arc_meta_limit as well:
>>
>> vfs.zfs.arc_meta_limit: 536870912
>> vfs.zfs.arc_meta_used: 579461312
>
> You also have arc_meta_used larger than arc_meta_limit ... but not
> nearly as big a difference as on my system.
>
> Can anyone speculate if raising vfs.zfs.arc_meta_limit would help?

Well, it didn't help me - I raised it above what used to be
arc_meta_used and after the reboot arc_meta_used simply rose again
over arc_meta_limit.

Here's another "symptom": while "find" is running, I do a ls of my
home directory on the same zpool and get delays, always at the same
place (after the postgresql source file):

www:~> ll
total 233
drwxrwxr-x  2 ivoras  ivoras          8 Jun  1  2009 backup/
-rw-rw-r--  1 ivoras  ivoras        593 Nov  7  2007 c1.php
-rw-rw-r--  1 ivoras  ivoras   37682863 Apr 30  2009 cms.tgz
drwxrwxr-x  4 ivoras  ivoras          4 Feb 12  2008 devel/
-rw-r--r--  1 ivoras  ivoras      44372 May 24  2007 etcdirs.tgz
-rw-r--r--  1 root    ivoras     215397 Nov 22  2007 lock_profile.txt
-rw-r--r--  1 ivoras  ivoras      18336 Nov 21  2007 lockmgr.diff
-rw-rw-r--  1 ivoras  ivoras   32590585 Oct 31  2007 melc.sql
-rw-r-----  1 ivoras  ivoras       1712 Oct 15  2008 newreq.pem
-rw-rw-r--  1 root    ivoras    3330572 Apr 30  2009 postgresql-server-8.3.1.tbz
load: 0.38  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 3.25r 0.00u 0.01s 0% 2140k
load: 0.67  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 5.00r 0.00u 0.01s 0% 2140k
load: 0.67  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 5.47r 0.00u 0.01s 0% 2140k
load: 0.67  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 6.34r 0.00u 0.01s 0% 2140k
load: 0.67  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 8.30r 0.00u 0.01s 0% 2140k
load: 0.67  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 8.70r 0.00u 0.01s 0% 2140k
load: 0.67  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 9.17r 0.00u 0.01s 0% 2140k
load: 0.70  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 11.08r 0.00u 0.01s 0% 2140k
load: 0.70  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 11.54r 0.00u 0.01s 0% 2140k
load: 0.70  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 13.00r 0.00u 0.01s 0% 2140k
load: 0.70  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 13.49r 0.00u 0.01s 0% 2140k
load: 0.70  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 14.10r 0.00u 0.01s 0% 2140k
load: 0.64  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 14.62r 0.00u 0.01s 0% 2140k
load: 0.64  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 17.41r 0.00u 0.01s 0% 2140k
load: 0.64  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 19.74r 0.00u 0.01s 0% 2140k
load: 0.75  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 20.53r 0.00u 0.01s 0% 2140k
load: 0.75  cmd: ls 1786 [tx->tx_quiesce_done_cv)] 21.30r 0.00u 0.01s 0% 2140k
lrwxrwxr-x  1 ivoras  ivoras          9 Apr 23  2009 services@ -> /services
-rw-rw-r--  1 ivoras  ivoras  160060550 Oct 31  2007 services.tgz
-rw-rw-r--  1 ivoras  ivoras          0 Nov  1  2007 stress.txt
drwxrwxr-x  5 ivoras  ivoras          8 Mar 10  2010 temp/
-rw-rw-r--  1 ivoras  ivoras     257002 Oct 31  2007 ule.tgz
-rw-r--r--  1 ivoras  ivoras    8965978 May 24  2007 wwwdirs.tgz

The "load.." lines in between are  me hitting Ctrl-T for process info.
Observe almost 20 seconds delay in the middle of "ls"!