FreeBSD 9-RELEASE zpool Out of File Descriptors ??

Wed May 15 16:05:59 UTC 2013

On Wed, May 15, 2013 at 12:03 PM, Fleuriot Damien <ml at my.gd> wrote:

>
> On May 15, 2013, at 5:56 PM, Outback Dingo <outbackdingo at gmail.com> wrote:
>
>
>
>
> On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien <ml at my.gd> wrote:
>
>>
>> On May 15, 2013, at 2:46 PM, Outback Dingo <outbackdingo at gmail.com>
>> wrote:
>>
>> > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop <
>> ronald-freebsd8 at klop.yi.org>wrote:
>> >
>> >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo <
>> outbackdingo at gmail.com>
>> >> wrote:
>> >>
>> >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop <
>> ronald-freebsd8 at klop.yi.org
>> >>>> **wrote:
>> >>>
>> >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo <
>> >>>> outbackdingo at gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> So it seems a new deployment we just built with zfsonroot mirror and
>> a
>> >>>>
>> >>>>> 48TB
>> >>>>> master pool is already out of File Descriptors???
>> >>>>>
>> >>>>>  pool: master
>> >>>>> state: ONLINE
>> >>>>> status: The pool is formatted using a legacy on-disk format.  The
>> pool
>> >>>>> can
>> >>>>>        still be used, but some features are unavailable.
>> >>>>> action: Upgrade the pool using 'zpool upgrade'.  Once this is done,
>> the
>> >>>>>        pool will no longer be accessible on software that does not
>> >>>>> support
>> >>>>> feature
>> >>>>>        flags.
>> >>>>>  scan: none requested
>> >>>>> config:
>> >>>>>
>> >>>>>        NAME                      STATE     READ WRITE CKSUM
>> >>>>>        master                    ONLINE       0     0     0
>> >>>>>          raidz3-0                ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN01  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN03  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN04  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN05  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN07  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN08  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN09  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN10  ONLINE       0     0     0
>> >>>>>          raidz3-1                ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN11  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN12  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN13  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN14  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN15  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN16  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN17  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN18  ONLINE       0     0     0
>> >>>>>          raidz3-2                ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN19  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN20  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN21  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN22  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN23  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN24  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN26  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN27  ONLINE       0     0     0
>> >>>>>          raidz3-3                ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN29  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN30  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN32  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN33  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN35  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN36  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN37  ONLINE       0     0     0
>> >>>>>            multipath/SATA_LUN38  ONLINE       0     0     0
>> >>>>>        logs
>> >>>>>          multipath/SATA_LUN06    ONLINE       0     0     0
>> >>>>>        cache
>> >>>>>          multipath/SATA_LUN02    ONLINE       0     0     0
>> >>>>>
>> >>>>> errors: No known data errors
>> >>>>>  pool: tank
>> >>>>> state: ONLINE
>> >>>>> status: The pool is formatted using a legacy on-disk format.  The
>> pool
>> >>>>> can
>> >>>>>        still be used, but some features are unavailable.
>> >>>>> action: Upgrade the pool using 'zpool upgrade'.  Once this is done,
>> the
>> >>>>>        pool will no longer be accessible on software that does not
>> >>>>> support
>> >>>>> feature
>> >>>>>        flags.
>> >>>>>  scan: none requested
>> >>>>> config:
>> >>>>>
>> >>>>>        NAME        STATE     READ WRITE CKSUM
>> >>>>>        tank        ONLINE       0     0     0
>> >>>>>          mirror-0  ONLINE       0     0     0
>> >>>>>            da34p3  ONLINE       0     0     0
>> >>>>>            da35p3  ONLINE       0     0     0
>> >>>>>
>> >>>>> errors: No known data errors
>> >>>>>
>> >>>>> while compiling a kernel, buildworld worked and installed ok
>> >>>>>
>> >>>>> lex -t
>> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>> >>>>> aicasm/aicasm_scan.l
>> >>>>>
>> >>>>> aicasm_scan.c
>> >>>>>>
>> >>>>>> lex -t  -Pmm
>> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>> >>>>>
>> >>>>> aicasm/aicasm_macro_scan.l
>> >>>>>
>> >>>>> aicasm_macro_scan.c
>> >>>>>>
>> >>>>>> rm -f .depend_aicasm
>> >>>>> mkdep -f .depend_aicasm -a    -I.
>> >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/****
>> >>>>> aic7xxx/aicasm
>> >>>>> -std=gnu99
>> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>> >>>>> aicasm/aicasm.c
>> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
>> >>>>>
>> >>>>> aicasm/aicasm_symbol.c
>> >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c
>> >>>>> Out of file descriptors
>> >>>>> *** [.depend_aicasm] Error code 2
>> >>>>>
>> >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm.
>> >>>>>
>> >>>>> *** [buildkernel] Error code 1
>> >>>>>
>> >>>>> Stop in /usr/src.
>> >>>>> *** [buildkernel] Error code 1
>> >>>>>
>> >>>>>
>> >>>>
>> >>>> Do you know what a file descriptor is?
>> >>>>
>> >>>> There is something about it here: http://www.freebsd.org/doc/en/****
>> <http://www.freebsd.org/doc/en/**>
>> >>>> books/handbook/configtuning-****kernel-limits.html<http://www.**
>> >>>>
>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html<
>> http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html
>> >>(search
>> >>>> for 'file descriptor')
>> >>>> Or more info in here: https://www.google.nl/search?****<
>> https://www.google.nl/search?**>
>> >>>> q=freebsd+Out+of+file+****descriptors<https://www.**
>> >>>> google.nl/search?q=freebsd+**Out+of+file+descriptors<
>> https://www.google.nl/search?q=freebsd+Out+of+file+descriptors>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> It mainly says that you have more files open than your system is
>> >>>> configured to allow. This is often produced by a bug in a program
>> which
>> >>>> does not close some files properly.
>> >>>>
>> >>>>
>> >>>> Yes I do.... but this is a brand new zfsonroot with barely any data
>> on
>> >>> it
>> >>> and
>> >>>
>> >>> sysctl -a | grep kern.openfiles
>> >>> kern.openfiles: 68
>> >>> root@:/master/builder # sysctl -a | grep kern.maxfiles
>> >>> kern.maxfiles: 24600
>> >>> kern.maxfilesperproc: 11095
>> >>>
>> >>> the openfiles and maxfiles seems to be plenty, but im out of
>> descriptors,
>> >>> i
>> >>> used to see this back in the 4.x days when you could format
>> >>> ufs with larger inodes, but zfs ??? really?
>> >>>
>> >>
>> >>
>> >> Is this kern.openfiles when idle or during your buildkernel?
>> >> Did you check 'ulimit -a'?
>> >> You say installworld worked ok while you are just now doing
>> buildkernel.
>> >> Are your kernel and world out of sync?
>> >>
>> >>
>> > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make
>> buildworld,
>> > then a make installworld, when i went to do a make buildkernel
>> > thats the error i got, so id say, yes it is "out of sync"
>>
>>
>> Wow wow wow, hold on a sec here.
>>
>> You're installing your new world before the kernel ?
>>
>>
>> I've had no trouble upgrading several boxes from 8-STABLE to
>> 10.0-CURRENT, following the regular procedure as described at:
>> http://www.freebsd.org/doc/handbook/makeworld.html
>>
>>
>> Note that the correct order is:
>> - buildworld (or kernel-toolchain if you only want to test the kernel and
>> not do the whole world)
>> - buildkernel
>> - installkernel
>> - reboot
>> - mergemaster -p (preferably in single user, I've always run it multiuser
>> w/o problems)
>> - installworld
>> - mergemaster
>> - reboot
>>
>> You don't want to run the new world on the old kernel.
>>
>
> yeah i know better, seems when i went to install the old kernel the build
> had failed, i can get through it from here
>
> it was odd because the CURRENT kernel failed to build with a yyparse
> error, until after i had completed a buildworld
>
> the other node i did is fine..........
>
>
>
> Perhaps their sources weren't at the same revision level ?
>
> Anyway, you wanna reinstall a 9.x world from the ISO now, then update
> sources and redo things in order ;)
>
> As a sidenote, if your root is on UFS, you can use nextboot to
> failsafe-try your new kernel.
>

actually there is no need to revert to 9x but thanks, ill just remount the
pool read write and do a make installkernel now and reboot