FreeBSD 9-RELEASE zpool Out of File Descriptors ??

Wed May 15 13:08:23 UTC 2013

On Wed, 15 May 2013 14:46:32 +0200, Outback Dingo <outbackdingo at gmail.com>  
wrote:

> On Wed, May 15, 2013 at 8:34 AM, Ronald Klop  
> <ronald-freebsd8 at klop.yi.org>wrote:
>
>> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo  
>> <outbackdingo at gmail.com>
>> wrote:
>>
>>  On Wed, May 15, 2013 at 8:22 AM, Ronald Klop  
>> <ronald-freebsd8 at klop.yi.org
>>> >**wrote:
>>>
>>>  On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo <
>>>> outbackdingo at gmail.com>
>>>> wrote:
>>>>
>>>>  So it seems a new deployment we just built with zfsonroot mirror and  
>>>> a
>>>>
>>>>> 48TB
>>>>> master pool is already out of File Descriptors???
>>>>>
>>>>>   pool: master
>>>>>  state: ONLINE
>>>>> status: The pool is formatted using a legacy on-disk format.  The  
>>>>> pool
>>>>> can
>>>>>         still be used, but some features are unavailable.
>>>>> action: Upgrade the pool using 'zpool upgrade'.  Once this is done,  
>>>>> the
>>>>>         pool will no longer be accessible on software that does not
>>>>> support
>>>>> feature
>>>>>         flags.
>>>>>   scan: none requested
>>>>> config:
>>>>>
>>>>>         NAME                      STATE     READ WRITE CKSUM
>>>>>         master                    ONLINE       0     0     0
>>>>>           raidz3-0                ONLINE       0     0     0
>>>>>             multipath/SATA_LUN01  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN03  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN04  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN05  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN07  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN08  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN09  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN10  ONLINE       0     0     0
>>>>>           raidz3-1                ONLINE       0     0     0
>>>>>             multipath/SATA_LUN11  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN12  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN13  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN14  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN15  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN16  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN17  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN18  ONLINE       0     0     0
>>>>>           raidz3-2                ONLINE       0     0     0
>>>>>             multipath/SATA_LUN19  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN20  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN21  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN22  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN23  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN24  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN26  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN27  ONLINE       0     0     0
>>>>>           raidz3-3                ONLINE       0     0     0
>>>>>             multipath/SATA_LUN29  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN30  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN32  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN33  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN35  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN36  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN37  ONLINE       0     0     0
>>>>>             multipath/SATA_LUN38  ONLINE       0     0     0
>>>>>         logs
>>>>>           multipath/SATA_LUN06    ONLINE       0     0     0
>>>>>         cache
>>>>>           multipath/SATA_LUN02    ONLINE       0     0     0
>>>>>
>>>>> errors: No known data errors
>>>>>   pool: tank
>>>>>  state: ONLINE
>>>>> status: The pool is formatted using a legacy on-disk format.  The  
>>>>> pool
>>>>> can
>>>>>         still be used, but some features are unavailable.
>>>>> action: Upgrade the pool using 'zpool upgrade'.  Once this is done,  
>>>>> the
>>>>>         pool will no longer be accessible on software that does not
>>>>> support
>>>>> feature
>>>>>         flags.
>>>>>   scan: none requested
>>>>> config:
>>>>>
>>>>>         NAME        STATE     READ WRITE CKSUM
>>>>>         tank        ONLINE       0     0     0
>>>>>           mirror-0  ONLINE       0     0     0
>>>>>             da34p3  ONLINE       0     0     0
>>>>>             da35p3  ONLINE       0     0     0
>>>>>
>>>>> errors: No known data errors
>>>>>
>>>>> while compiling a kernel, buildworld worked and installed ok
>>>>>
>>>>> lex -t
>>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>>>>> aicasm/aicasm_scan.l
>>>>>
>>>>>  aicasm_scan.c
>>>>>>
>>>>>>  lex -t  -Pmm
>>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>>>>>
>>>>> aicasm/aicasm_macro_scan.l
>>>>>
>>>>>  aicasm_macro_scan.c
>>>>>>
>>>>>>  rm -f .depend_aicasm
>>>>> mkdep -f .depend_aicasm -a    -I.
>>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/****
>>>>> aic7xxx/aicasm
>>>>> -std=gnu99
>>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>>>>> aicasm/aicasm.c
>>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>>>>>
>>>>> aicasm/aicasm_symbol.c
>>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c
>>>>> Out of file descriptors
>>>>> *** [.depend_aicasm] Error code 2
>>>>>
>>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm.
>>>>>
>>>>> *** [buildkernel] Error code 1
>>>>>
>>>>> Stop in /usr/src.
>>>>> *** [buildkernel] Error code 1
>>>>>
>>>>>
>>>>
>>>> Do you know what a file descriptor is?
>>>>
>>>> There is something about it here:  
>>>> http://www.freebsd.org/doc/en/****<http://www.freebsd.org/doc/en/**>
>>>> books/handbook/configtuning-****kernel-limits.html<http://www.**
>>>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html<http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html>>(search
>>>> for 'file descriptor')
>>>> Or more info in here:  
>>>> https://www.google.nl/search?****<https://www.google.nl/search?**>
>>>> q=freebsd+Out+of+file+****descriptors<https://www.**
>>>> google.nl/search?q=freebsd+**Out+of+file+descriptors<https://www.google.nl/search?q=freebsd+Out+of+file+descriptors>
>>>> >
>>>>
>>>>
>>>> It mainly says that you have more files open than your system is
>>>> configured to allow. This is often produced by a bug in a program  
>>>> which
>>>> does not close some files properly.
>>>>
>>>>
>>>>  Yes I do.... but this is a brand new zfsonroot with barely any data  
>>>> on
>>> it
>>> and
>>>
>>> sysctl -a | grep kern.openfiles
>>> kern.openfiles: 68
>>> root@:/master/builder # sysctl -a | grep kern.maxfiles
>>> kern.maxfiles: 24600
>>> kern.maxfilesperproc: 11095
>>>
>>> the openfiles and maxfiles seems to be plenty, but im out of  
>>> descriptors,
>>> i
>>> used to see this back in the 4.x days when you could format
>>> ufs with larger inodes, but zfs ??? really?
>>>
>>
>>
>> Is this kern.openfiles when idle or during your buildkernel?
>> Did you check 'ulimit -a'?
>> You say installworld worked ok while you are just now doing buildkernel.
>> Are your kernel and world out of sync?
>>
>>
> well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make  
> buildworld,
> then a make installworld, when i went to do a make buildkernel
> thats the error i got, so id say, yes it is "out of sync"

Read /usr/src/UPDATING from the line 'COMMON ITEMS:'. Especially  
interesting is 'To upgrade in-place from stable to current'. The order is  
important.
See also  
http://www.freebsd.org/doc/en/books/handbook/updating-upgrading.html for a  
lot of text about the same.

To fix your current situation. I'm not familiar with that situation but  
can imagine to try to put back a 9-RELEASE world. Do you have a snapshot  
of your current system with world from 9-RELEASE? Or another backup?

Ronald.