FreeBSD 9-RELEASE zpool Out of File Descriptors ??

[Outback Dingo]

On Wed, May 15, 2013 at 8:22 AM, Ronald Klop <ronald-freebsd8 at klop.yi.org>wrote:

> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo <outbackdingo at gmail.com>
> wrote:
>
>  So it seems a new deployment we just built with zfsonroot mirror and a
>> 48TB
>> master pool is already out of File Descriptors???
>>
>>   pool: master
>>  state: ONLINE
>> status: The pool is formatted using a legacy on-disk format.  The pool can
>>         still be used, but some features are unavailable.
>> action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
>>         pool will no longer be accessible on software that does not
>> support
>> feature
>>         flags.
>>   scan: none requested
>> config:
>>
>>         NAME                      STATE     READ WRITE CKSUM
>>         master                    ONLINE       0     0     0
>>           raidz3-0                ONLINE       0     0     0
>>             multipath/SATA_LUN01  ONLINE       0     0     0
>>             multipath/SATA_LUN03  ONLINE       0     0     0
>>             multipath/SATA_LUN04  ONLINE       0     0     0
>>             multipath/SATA_LUN05  ONLINE       0     0     0
>>             multipath/SATA_LUN07  ONLINE       0     0     0
>>             multipath/SATA_LUN08  ONLINE       0     0     0
>>             multipath/SATA_LUN09  ONLINE       0     0     0
>>             multipath/SATA_LUN10  ONLINE       0     0     0
>>           raidz3-1                ONLINE       0     0     0
>>             multipath/SATA_LUN11  ONLINE       0     0     0
>>             multipath/SATA_LUN12  ONLINE       0     0     0
>>             multipath/SATA_LUN13  ONLINE       0     0     0
>>             multipath/SATA_LUN14  ONLINE       0     0     0
>>             multipath/SATA_LUN15  ONLINE       0     0     0
>>             multipath/SATA_LUN16  ONLINE       0     0     0
>>             multipath/SATA_LUN17  ONLINE       0     0     0
>>             multipath/SATA_LUN18  ONLINE       0     0     0
>>           raidz3-2                ONLINE       0     0     0
>>             multipath/SATA_LUN19  ONLINE       0     0     0
>>             multipath/SATA_LUN20  ONLINE       0     0     0
>>             multipath/SATA_LUN21  ONLINE       0     0     0
>>             multipath/SATA_LUN22  ONLINE       0     0     0
>>             multipath/SATA_LUN23  ONLINE       0     0     0
>>             multipath/SATA_LUN24  ONLINE       0     0     0
>>             multipath/SATA_LUN26  ONLINE       0     0     0
>>             multipath/SATA_LUN27  ONLINE       0     0     0
>>           raidz3-3                ONLINE       0     0     0
>>             multipath/SATA_LUN29  ONLINE       0     0     0
>>             multipath/SATA_LUN30  ONLINE       0     0     0
>>             multipath/SATA_LUN32  ONLINE       0     0     0
>>             multipath/SATA_LUN33  ONLINE       0     0     0
>>             multipath/SATA_LUN35  ONLINE       0     0     0
>>             multipath/SATA_LUN36  ONLINE       0     0     0
>>             multipath/SATA_LUN37  ONLINE       0     0     0
>>             multipath/SATA_LUN38  ONLINE       0     0     0
>>         logs
>>           multipath/SATA_LUN06    ONLINE       0     0     0
>>         cache
>>           multipath/SATA_LUN02    ONLINE       0     0     0
>>
>> errors: No known data errors
>>   pool: tank
>>  state: ONLINE
>> status: The pool is formatted using a legacy on-disk format.  The pool can
>>         still be used, but some features are unavailable.
>> action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
>>         pool will no longer be accessible on software that does not
>> support
>> feature
>>         flags.
>>   scan: none requested
>> config:
>>
>>         NAME        STATE     READ WRITE CKSUM
>>         tank        ONLINE       0     0     0
>>           mirror-0  ONLINE       0     0     0
>>             da34p3  ONLINE       0     0     0
>>             da35p3  ONLINE       0     0     0
>>
>> errors: No known data errors
>>
>> while compiling a kernel, buildworld worked and installed ok
>>
>> lex -t
>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/**
>> aicasm/aicasm_scan.l
>>
>>> aicasm_scan.c
>>>
>> lex -t  -Pmm
>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/**
>> aicasm/aicasm_macro_scan.l
>>
>>> aicasm_macro_scan.c
>>>
>> rm -f .depend_aicasm
>> mkdep -f .depend_aicasm -a    -I.
>> -I/usr/src/sys/modules/**aic7xxx/aicasm/../../../dev/**aic7xxx/aicasm
>> -std=gnu99
>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/**
>> aicasm/aicasm.c
>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/**
>> aicasm/aicasm_symbol.c
>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c
>> Out of file descriptors
>> *** [.depend_aicasm] Error code 2
>>
>> Stop in /usr/src/sys/modules/aic7xxx/**aicasm.
>> *** [buildkernel] Error code 1
>>
>> Stop in /usr/src.
>> *** [buildkernel] Error code 1
>>
>
>
> Do you know what a file descriptor is?
>
> There is something about it here: http://www.freebsd.org/doc/en/**
> books/handbook/configtuning-**kernel-limits.html<http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html>(search for 'file descriptor')
> Or more info in here: https://www.google.nl/search?**
> q=freebsd+Out+of+file+**descriptors<https://www.google.nl/search?q=freebsd+Out+of+file+descriptors>
>
> It mainly says that you have more files open than your system is
> configured to allow. This is often produced by a bug in a program which
> does not close some files properly.
>
>
Yes I do.... but this is a brand new zfsonroot with barely any data on it
and

sysctl -a | grep kern.openfiles
kern.openfiles: 68
root@:/master/builder # sysctl -a | grep kern.maxfiles
kern.maxfiles: 24600
kern.maxfilesperproc: 11095

the openfiles and maxfiles seems to be plenty, but im out of descriptors, i
used to see this back in the 4.x days when you could format
ufs with larger inodes, but zfs ??? really?



> Ronald.
>