From nobody Thu Jun 02 20:25:16 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E1D831B59E47 for ; Thu, 2 Jun 2022 20:25:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LDcvn03kDz4Yh7 for ; Thu, 2 Jun 2022 20:25:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 252KPGmR042421 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 2 Jun 2022 23:25:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 252KPGmo042418; Thu, 2 Jun 2022 23:25:16 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 2 Jun 2022 23:25:16 +0300 From: Konstantin Belousov To: Rick Macklem Cc: John-Mark Gurney , "freebsd-net@FreeBSD.org" Subject: Re: make NFSv3 default now on diskless Message-ID: References: <20220602032456.GY88842@funkthat.com> <20220602183522.GZ88842@funkthat.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.5 X-Spam-Checker-Version: SpamAssassin 3.4.5 (2021-03-20) on tom.home X-Rspamd-Queue-Id: 4LDcvn03kDz4Yh7 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.99 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-0.997]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-net]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-ThisMailContainsUnwantedMimeParts: N On Thu, Jun 02, 2022 at 08:03:05PM +0000, Rick Macklem wrote: > John-Mark Gurney wrote: > > Rick Macklem wrote this message on Thu, Jun 02, 2022 at 14:44 +0000: > > > John-Mark Gurney wrote: > > > > I just booted FreeBSD-current diskless, using NFS root, and I ended > > > > up having issues because by default, NFS root is only v2. > > > > > > > > One of things that happened was disk space available was listed as > > > > -138G, or -144830429K. I assume this is because the server is reporting > > > > TBs instead. > > > Yes. NFSv2 uses 32bit sizes. > > > > Should we make the NFS server clamp various sizes then? instead of reporting > > negative numbers? (Sorry if this is already done on newer versions of > > FreeBSD, my server is a bit old.) > My recollection (from long ago) is that, although the RFC defined them as 32bit signed, > some implementations choose to mid-interpret them as unsigned, so that 4G was supported > instead of 2G. > > Having said that, I agree with you that you should just use NFSv3 now. (Many servers are > dropping NFSv2 support entirely.) > > > > > If I mount via mount_nfs, the sizes are normal/correct because it mounts > > > > v3. > > > I believe most specify "nfsv3" in the "/" mount line of /etc/fstab on the > > > remote root fs. Then, when the system does a "mount -u" to make it > > > read/write it gets toggled to NFSv3. > > > > well, I tried doing a: > > mount -u -o nfsv3 / > It used to work when it was in /etc/fstab, when going from read-only to read-write, > from what I recall. > > > on the system and this didn't work switch over to v3, it was still on > > v2.. > > > > I haven't specified nfsv3 in /etc/fstab, but IMO, this should be the > > default.. > Since NFSv3 is the default, it might not need to be explicit. I can't recall. > > > > > Also, I'm right now booting single user mode, because I'm -mapall to > > a user, and lots of FreeBSD breaks when files aren't uid 0, and there > > doesn't appear to be a way to remap the uid to root (that I have found).. > Yep. It would take effect when going multi-user. > > "-mapall=root", although that is not a recommended security setting. > Why don't you just allow the client to use whatever uid it would normally > use instead of "-mapall"? > > > > > The other issue that I ran into is that NFSv2 can't access >4GB files > > > > (or create them). > > > As above, NFSv2 uses 32bit sizes. > > > > > > > Anyone object to adding BOOTP_NFSV3 to GENERIC? > > > Well, that option only works when used with BOOTP_NFSROOT. > > > The GENERIC configs for amd64, arm64,... use the other way. > > > (Just to make it confusing, there are two different ways an NFS root > > > fs is set up.) > > > See below. > > > > > > > Or maybe making it a > > > > tunable that defaults to set, because it seems a bit crazy to default > > > > to v2 these days. > > > I don't think changing the default to NFSv3 will be a problem. > > > The reason it was NFSv2 was that, > > > for some non-FreeBSD NFS servers, the NFSv3 file handle is different > > > than the NFSv2 one. > > > > > > I added NFSv3 support to stand/libsa/nfs.c about 15years ago, so every > > > system should be running the newer NFS code in the loader and be able > > > to do NFSv3 booting. > > > > > > > This option was added in 432aad0e in 1997 so that the nfs_diskless > > > > structure didn't need to be filled out. Does anything even > > > > populate/fill it out anymore? I saw code in i386/i386/locore.s that > > > > does this, but it doesn't appear anywhere else. > > > Yes. For "options NFS_ROOT" (the other way), the loader uses > > > "stand/libsa/nfs.c" to acquire the remote file system's root file handle > > > and fills it in. (See nfs_setup_diskless() in sys/nfs/nfs_diskless.c.) > > > Looking at it, it appears to enable NFSv3 so long as it finds > > > "boot.nfsroot.nfshandlelen" set. > > > > Well, I'm not seeing that, and this system is booting via > > pxeboot+loader, so maybe something is broken? > If it's an up to date (within last 10+ years then, yes, it sounds like it > might be broken. However, I'm not a loader guy... > > You can set it manually at the loader prompt to "28" and then see if it > boots are is running NFSv3. > > I have no way of testing/debugging this. Maybe you can figure out why the > loader isn't setting "boot.nfsroot.nfshandlelen" to 28? I have did several (> 5) lab setups with NFS booting, and it was always like you described: kernel config: options NFS_ROOT /boot/loader.conf boot.nfsroot.options="nolockd" nfscl_load="YES" (yes, my nfscl.ko is a module) /etc/fstab (an example) 192.168.130.1:/usr/home/netboot/zoo/r-freeb44 / nfs rw,nolockd 0 0 and in dhcpd.conf host r-freeb44.zoo { hardware ethernet 28:80:23:a2:37:bc; fixed-address r-freeb44.zoo; next-server rott.zoo; filename "pxeboot"; # filename "loader.efi"; option root-path "192.168.130.1:/usr/home/netboot/zoo/r-freeb44"; } You need pxeboot (or loader.efi) on the tftp server root (rott.zoo in my case). Above demonstrates the loading over the legacy BIOS/PXE. Recomment pxeboot to loader.efi if you have something not so old so it can pxeboot using UEFI. Result: root@r-freeb44:~ # nfsstat -m 192.168.130.1:/usr/home/netboot/zoo/r-freeb44 on / nfsv3,tcp,resvport,nconnect=1,hard,cto,nolockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 > > > > > There also appears to possibly be a way via mount options, but I can't > > > > see where it's documented to set them. > > > I think you just specify "nfsv3" as a mount option in the root fs > > > line in /etc/fstab on the root fs on the NFS server. > > > > See above, this doesn't appear to work, or doesn't work the way I think > > it should... > Well, maybe I just don't recall correctly. > > > > > > I don't think changing the default to NFSv3 will be a problem. > > > The hassle is testing the various cases, to make sure nothing > > > breaks. I have no diskless setup to do testing and I don't even know > > > when installs/upgrades actually replace the loader? > > > > Well, this diskless was easier to setup than I expected, partly > > because I already had most of the infrastructure together (from > > netbooting another machine). Put pxeboot on a tftp server, configure > > the dhcp server to send the correct options, extract base.txz to a > > directory, export it, and it worked. I assume that I'm getting loader > > from that install since I don't specify it in the dhcp server. > I didn't think pxeboot used dhcp and I don't know how it figures out > where to find a file for booting? > > > As for testing, we have the CI system for that, right? ;p > > > > /me needs to get back to work on the lab. > > > > I guess we'd need to list the configurations that we care about, the > > only ones I can think of, off the top of my head are pxeboot (which I'm > > testing now), and u-boot.. > > For servers, are there any servers that > > are NFSv2 only that are in common use today? > You'd have to look pretty hard. FreeBSD had NFSv3 from day 1, > since it was in the CSRG stuff. > > If you're running an > > ancient server that is NFSv2 only, I think you deserve to have to > > rebuild kernels or something instead of making 99% of the rest of us > > do it.. > If you are running a NFSv2 only server, you are running a computer museum. > (As noted, many servers are dropping NFSv2 support entirely.) > > rick > > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > >