nfs v2/v3 and diskless boot problem
Danny Braniss
danny at cs.huji.ac.il
Thu Jan 3 05:25:58 PST 2008
> Danny Braniss wrote:
> >> Danny Braniss wrote:
> >>>> Danny Braniss wrote:
> >>>>> there is an undocumented option:
> >>>>> boot-nfsroot-options
> >>>>> that the diskeless boot can use. I tried
> >>>>> boot-nfsroot-options = "nfsv3"
> >>>>> since the pxeboot does the initial mount via nfsv2, and this has at least
> >>>>> one problem: removing a file from the readonly / will hang the system.
> >>>>>
> >>>>> so, the remount to v3 works in the case that the root is served by a Freebsd
> >>>>> nfs server, but fails if it's NetAPP. The reason is that the v2 filehandle
> >>>>> is 32 bytes, and when switching to V3 it becomes 28bytes - sizeof(fhandle_t).
> >>>>> This is not liked by the NetApp, which correctly gives error 1001: BADHANDLE
> >>>>> :-)
> >>>>>
> >>>>> While I'm trying to come up with a solution, I am wondering if someone
> >>>>> can shed some light:
> >>>>> - is sizeof(fhandle_t) == 28 bytes is mystical, or changing it to
> >>>>> 32 bytes will start WW3?
> >>>> NFSv3 file handles (by spec) can be up to 64bytes.
> >>> true, but in freebsd, look at sys/nfs/nfsproto.h
> >>> #define NFSX_V2FH 32
> >>> #define NFSX_V3FH (sizeof (fhandle_t))
> >>> #define NFSX_V4FH 128
> >>>
> >>> so for v3 it's 28 bytes. (fhandle_t is defined in sys/mount.h)
> >>>
> >>>
> >>>> I'm not 100% sure what is happening, but it sounds like the file handle
> >>>> for the mount point or maybe one of the directories is not getting reset
> >>>> on remount.
> >>>>
> >>>> When do you get the BADHANDLE error? Can you capture a
> >>>> tshark/wireshark/tcpdump of the remount and error?
> >>> I did, and if you look in sys/nfsclient/nfs_vfsops.c, nfs_convert_diskless is responsible
> >>> for chopping off the 4 extra bytes. BTW, I tried to change the bcopy count to NFSX_V2FH/32, and
> >>> it panics the kernel :-(
> >>>
> >>> danny
> >>
> >> oh - looks like this says it all:
> >> http://fxr.googlebit.com/source/sys/nfsclient/nfsdiskless.h?v=8-CURRENT#L51
> >>
> > that's where the boot-nfsroot-options comes from:-)
> > if you notice, the filehandle for v3 is 64 bytes, but
> > only 28 are used.
> >
> > but as I mentioned initially, this ONLY works when the server is FreeBSD, and
> > breaks for other servers, ie NetAPP. AND the initial question stands:
> > what's in a filehandle, or can it be > 28bytes.
>
>
> Yea, FreeBSD is making the assumption that all NFS servers will use the
> same size FH for NFSv3. That is just wrong.
>
carful, I think this is the case only if fsb is the server, it will 'probably'
accept
filehandles of other sizes from other servers.
> The FH is a server created opaque handle that it can create however it
> wishes. Most servers use information like inode, generation, fsid, etc
> to create it, but it's something that you can't necessarily decode.
>
yes, but the FH has information that the server can/must use to figure out
which local filesystem it refers to - remember that v2/v3 are stateless.
> I've created a patch that might fix this, but I'm still testing and QEMU
> (which I use for my testing) keeps making my system either panic or lock
> up, so hopefully I should have something for you to try tonight.
>
> Also - can you tell me the exact 'mount' command you tried to do the
> remount/update?
>
it's only in the diskless boot, where setting
boot-nfsroot-options = "nfsv3"
in /boot/loader.conf will do the remount.
cheers,
danny
More information about the freebsd-hackers
mailing list