deadlock with zfs?

[Thomas Vogt]

Hello

I encounter a deadlock while running a few rsync processes mirroring  
remote data to my local zfs pool. After a few hours my system is  
starting more and more vsftpd sessions without closing any inactive  
ftp sessions. I can't kill any rsync and vsftpd processes with "kill  
-9". Even shutdown -r now does not work.

I got a few hunderts vsftpd processes like this

61346 root          1  57    0  7880K  1692K zfs     1   0:00  0.00%  
vsftpd
61481 root          1  68    0  7880K  1696K zfs     1   0:00  0.00%  
vsftpd
61354 root          1  65    0  7880K  1692K zfs     1   0:00  0.00%  
vsftpd
61480 root          1  68    0  7880K  1696K zfs     0   0:00  0.00%  
vsftpd
61600 root          1  69    0  7880K  1704K zfs     1   0:00  0.00%  
vsftpd
61599 root          1  68    0  7880K  1704K zfs     1   0:00  0.00%  
vsftpd

Right now i'm building a debug kernel. Whats the best way to get  
usefull information from this deadlock? The system itself is not  
crashing and response well to ssh and i also have a serial console.

The system is running  8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008  
(64bit) with zpool version 13. I use zfs only as data pool. The base  
system is running on ufs2:

Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/da0s1a      496M    220M    236M    48%    /
devfs            1.0K    1.0K      0B   100%    /dev
/dev/da0s1g      169G     15G    141G     9%    /disk1
/dev/da0s1f      3.9G     17M    3.5G     0%    /tmp
/dev/da0s1e       29G    3.7G     23G    14%    /usr
/dev/da0s1d       19G    7.1G     11G    40%    /var
pool             853G      0B    853G     0%    /usr/local/data
pool/cvsup       858G    5.7G    853G     1%    /usr/local/data/cvsup
pool/ftp         3.3T    2.5T    853G    75%    /usr/local/data/ftp
pool/portsnap    853G    633M    853G     0%    /usr/local/data/portsnap
pool/www         853G     90M    853G     0%    /usr/local/data/www

loader.conf:
vm.kmem_size="1G"
kern.maxfiles="65536"
kern.maxproc="20480"
net.inet.tcp.tcbhashsize="4096"
net.inet.tcp.hostcache.hashsize="1024"
vfs.zfs.arc_min="64M"
vfs.zfs.arc_max="768M"
vfs.zfs.prefetch_disable="1"

I also tried to set vfs.zfs.zil_disable=1 but the problem still exist.

I know there are few deadlock reports listed in the freebsd wiki.  
Maybe we can trigger the root cause of the problem and fix it :)

Regards
Thomas