ZFS deadlock

[Johan Ström]

On Apr 8, 2008, at 9:37 AM, LI Xin wrote:

> Johan Ström wrote:
>> Hello
>> A box of mine running RELENG_7_0 and ZFS over a couple of disks (6  
>> disks, 3 mirrors) seems to have gotten stuck. From Ctrl-T:
>> load: 0.50  cmd: zsh 40188  
>> [zfs:&buf_hash_table.ht_locks[i].ht_lock] 0.02u 0.04s 0% 3404k
>> load: 0.43  cmd: zsh 40188  
>> [zfs:&buf_hash_table.ht_locks[i].ht_lock] 0.02u 0.04s 0% 3404k
>> load: 0.10  cmd: zsh 40188  
>> [zfs:&buf_hash_table.ht_locks[i].ht_lock] 0.02u 0.04s 0% 3404k
>> load: 0.10  cmd: zsh 40188  
>> [zfs:&buf_hash_table.ht_locks[i].ht_lock] 0.02u 0.04s 0% 3404k
>> load: 0.11  cmd: zsh 40188  
>> [zfs:&buf_hash_table.ht_locks[i].ht_lock] 0.02u 0.04s 0% 3404k
>> Worked for a while then that stopped working too (was over ssh).  
>> When trying a local login i only got
>> load: 0.09  cmd: login 1611 [zfs] 0.00u 0.00s 0% 208k
>> I found one post like this earlier (by Xin LI), but nobody seemed  
>> to have replied...
>> in my current conf, I think my kmem/kmem_max is at 512Mb (not sure  
>> though, since I've edited my file yesterday for next reboot), with  
>> 2G of system RAM.. Normally I'd run kmem(max) 1G (with arcsize of  
>> 512M. currently it is at default), but since I just got back to 2G  
>> total mem after some hardware problems I've been runnig at those  
>> lows (1G total is kindof tight with zfs..)
>> Well, just wanted to report... The box is not totally dead yet, ie  
>> I can still do Ctrl-T on console, but thats it.. I don't really  
>> know what more I can do so.. I don't have KDB/DDB.
>> I'll wait another hour or so before I hard reboot it, unless it  
>> "unlocks" or if anyone have any suggestions.
>
> The key is to increase your kmem and prevent it from being  
> exhausted.  I think more recent OpenSolaris's ZFS code has some  
> improvements but I do not have spare devices at hand to test and  
> debug :(

Yep, never had the problem when I was running with 2G total mem, but  
then one stick (damn consumer crap) failed and I was left with 1G, and  
I started to have random problems. Going to tune kmem back up now when  
I got more mem again, thinking about putting in 4G too..

>
>
> Maybe pjd@ would get a new import at some point?  I have cc'ed him.
>
> Cheers,
> -- 
> Xin LI <delphij at delphij.net>	http://www.delphij.net/
> FreeBSD - The Power to Serve!
>