nfsd doesn't register with rpcbind
Ondra Knezour
knezour at weboutsourcing.cz
Mon Jan 11 03:00:10 UTC 2021
Hi all,
I have longstanding issue with our NFS server, going probably from 10.x
release times. Also problematic counterpart (client) has undergone
multiple upgrades, but problem persists.
Quick intro - NFS server on FreeBSD 12.2 (running there probably for
about 3-4 years, hence mentioning 10.x release), couple of clients,
mostly Linux. The problematic one is xen on two servers in pool. Now
running latest xcp-ng 8.2 (opensource fork of the Citrix product after
their licensing changes, based on CentOS 7), also with history of
upgrades over the years.
The problem is I can mount and use NFS shares from any client, even
those xcp-ng servers, but I can't add what xen calls NFS storage
repository, which is basically only fancy name for NFS mount with info
stored in xen internal configuration database. There are three ways to
do it (known to me), xe command line utility, web tool from authors of
the mentioned fork named Xen Orchestra and Windows application, which is
remnant of the Citrix era called XenCenter. All three use (AFAIK) some
Python scripting on the server (NFS client in this case) to do it.
It would be easy to blame this Python part of the problem, but I
noticed, that on the FreeBSD side nfsd does not register service with
the rpcbind, so I think this may be (at least part of) the problem. I
read somewhere, that NFSv4 can work without RPC and in fact, it does for
me, at least partially, but I am not sure what specification says and
which part has to be blamed here.
So my questions are:
1. Why my nfsd doesn't register with rpcbind?
2. Is this registration somewhat optional at least for NFSv4?
3. How can I get some useful debug info? Using -d options in our servers
startup configuration where it is available doesn't produce much.
In the following snippets, 172.22.1.4 or storage-smc is the NFS server
and 172.22.1.7 and 172.22.1.27 are the troublesome clients.
From the server
uname -a
FreeBSD storage-smc.ujezd.net 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC amd64
/etc/rc.conf
hostname="storage-smc.ujezd.net"
cloned_interfaces="vlan1000 vlan1500 vlan2000"
ifconfig_vlan1000="inet 172.22.1.4 netmask 255.255.0.0 vlan 1000 vlandev igb1"
ifconfig_vlan2000="inet 10.128.99.4 netmask 255.255.255.0 vlan 2000 vlandev igb1"
ifconfig_vlan1500="inet 192.168.222.1 netmask 255.255.255.0 vlan 1500 vlandev igb1"
ifconfig_igb1="up"
nfsuserd_flags="-verbose"
defaultrouter="172.22.0.1"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
#nfs_server_flags="-h 172.22.1.4"
mountd_enable="YES"
mountd_flags="-r"
kld_list="aesni ipmi"
smartd_enable="YES"
ntpdate_hosts="10.128.99.95"
ntpdate_enable="YES"
ntpd_sync_on_start="YES"
ntpd_enable="YES"
netdata_enable="YES"
microcode_update_enable="YES"
rpc_lockd_enable="YES" # Run NFS rpc.lockd needed for client/server.
rpc_statd_enable="YES" # Run NFS rpc.statd needed for client/server.
/etc/sysctl.conf
vfs.zfs.arc_max="48000000000"
vfs.nfsd.server_min_nfsvers=4
vfs.nfs.enable_uidtostring=1
vfs.nfsd.tcpcachetimeo: 300
vfs.nfsd.tcphighwater: 100000
/etc/exports
/zdata/email -maproot=root 10.128.99.79
/zdata/nf -maproot=root 172.22.255.249
/zdata/odkladgalery -maproot=root 172.22.1.10
/zdata/ios -maproot=root 172.22.1.14
/zdata/xen-iso-library -maproot=root 172.22.1.27
/zdata/xen-iso-library -maproot=root 172.22.1.7
/zdata/xen-nfs-storage -maproot=root 172.22.1.27
/zdata/xen-nfs-storage -maproot=root 172.22.1.7
/zdata/servers/xenserver -maproot=root 172.22.1.27
/zdata/servers/xenserver -maproot=root 172.22.1.7
/zdata/servers/xenserver -maproot=root 172.22.1.32
/zdata/virt -maproot=root 172.22.1.27
/zdata/virt -maproot=root 172.22.1.7
/zdata/virt -maproot=root 172.22.1.13
/zdata/virt -maproot=root 172.22.1.2
/zdata/virt -maproot=root -sec=sys 172.22.1.11
V4: / -sec=sys -network 172.22.0.0 -mask 255.255.0.0
V4: / -sec=sys -network 10.128.99.0 -mask 255.255.255.0
rpcinfo -s
program version(s) netid(s) service owner
100000 2,3,4 local,udp6,tcp6,udp,tcp rpcbind superuser
100024 1 tcp,udp,tcp6,udp6 status superuser
100021 4,3,1,0 tcp,udp,tcp6,udp6 nlockmgr superuser
100005 3,1 tcp,udp,tcp6,udp6 mountd superuser
And from one of the Linux clients - rpcinfo doesn't show nfsd, but
mounts can be probed and mounted without problem.
showmount -e 172.22.1.4
Export list for 172.22.1.4:
/zdata/virt 172.22.1.11,172.22.1.2,172.22.1.13,172.22.1.7,172.22.1.27
/zdata/xen-nfs-storage 172.22.1.7,172.22.1.27
/zdata/xen-iso-library 172.22.1.7,172.22.1.27
/zdata/odkladgalery 172.22.1.10
/zdata/email 10.128.99.79
/zdata/servers/xenserver 172.22.1.32,172.22.1.7,172.22.1.27
/zdata/ios 172.22.1.14
/zdata/nf 172.22.255.249
rpcinfo -s 172.22.1.4
program version(s) netid(s) service owner
100000 2,3,4 local,udp6,tcp6,udp,tcp portmapper superuser
100024 1 tcp,udp,tcp6,udp6 status superuser
100021 4,3,1,0 tcp,udp,tcp6,udp6 nlockmgr superuser
100005 3,1 tcp,udp,tcp6,udp6 mountd superuser
mount.nfs4 172.22.1.4:/zdata/xen-iso-storage /iso-storage/
mount.nfs4 172.22.1.4:/zdata/xen-iso-library /iso-storage/
mount
172.22.1.4:/zdata/xen-iso-library on /iso-storage type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.22.1.7,local_lock=none,addr=172.22.1.4)
172.22.1.4:/zdata/xen-nfs-storage on /nfs-storage type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.22.1.7,local_lock=none,addr=172.22.1.4)
Digging again deeper, I see in log on the client, that missing nfsd in
rpcinfo -s call is probably main culprit here. From where came "missing
serverpath" error I don't know. Also setting rpcdebug -m [nfs|rpc] -s
all (set all debug flags for those two modules) on client yeld nothing
at all.
Dec 27 22:31:36 xen-2u SM: [25530] _testHost: Testing host/port: storage-smc.ujezd.net,2049
Dec 27 22:31:36 xen-2u SM: [25530] scanning2 (target=storage-smc.ujezd.net)
Dec 27 22:31:36 xen-2u SM: [25530] scanning
Dec 27 22:31:36 xen-2u SM: [25530] ['/usr/sbin/showmount', '--no-headers', '-e', 'storage-smc.ujezd.net']
Dec 27 22:31:36 xen-2u SM: [25530] pread SUCCESS
Dec 27 22:31:36 xen-2u SM: [25530] Raising exception [101, The request is missing the serverpath parameter]
Dec 27 22:31:36 xen-2u SM: [25530] lock: released /var/lock/sm/sr
Dec 27 22:31:36 xen-2u SM: [25530] ***** generic exception: sr_probe: EXCEPTION <class 'SR.SROSError'>, The request is missing the serverpath parameter
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/SRCommand.py", line 110, in run
Dec 27 22:31:36 xen-2u SM: [25530] return self._run_locked(sr)
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Dec 27 22:31:36 xen-2u SM: [25530] rv = self._run(sr, target)
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/SRCommand.py", line 332, in _run
Dec 27 22:31:36 xen-2u SM: [25530] txt = sr.probe()
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/NFSSR", line 164, in probe
Dec 27 22:31:36 xen-2u SM: [25530] self.validate_remotepath(True)
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/NFSSR", line 109, in validate_remotepath
Dec 27 22:31:36 xen-2u SM: [25530] raise xs_errors.XenError('ConfigServerPathMissing')
Dec 27 22:31:36 xen-2u SM: [25530]
Dec 27 22:31:36 xen-2u SM: [25530] ***** NFS VHD: EXCEPTION <class 'SR.SROSError'>, The request is missing the serverpath parameter
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/SRCommand.py", line 378, in run
Dec 27 22:31:36 xen-2u SM: [25530] ret = cmd.run(sr)
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/SRCommand.py", line 110, in run
Dec 27 22:31:36 xen-2u SM: [25530] return self._run_locked(sr)
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Dec 27 22:31:36 xen-2u SM: [25530] rv = self._run(sr, target)
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/SRCommand.py", line 332, in _run
Dec 27 22:31:36 xen-2u SM: [25530] txt = sr.probe()
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/NFSSR", line 164, in probe
Dec 27 22:31:36 xen-2u SM: [25530] self.validate_remotepath(True)
Dec 27 22:31:36 xen-2u SM: [25530] File "/opt/xensource/sm/NFSSR", line 109, in validate_remotepath
Dec 27 22:31:36 xen-2u SM: [25530] raise xs_errors.XenError('ConfigServerPathMissing')
Dec 27 22:31:36 xen-2u SM: [25530]
Dec 27 22:32:02 xen-2u SM: [25713] sr_create {'sr_uuid': '34c61bf5-32d6-a0b7-47e7-65209f69ddb9', 'subtask_of': 'DummyRef:|2360ef68-4b8d-4db8-8943-ec4370f77fa7|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:2669e2f3-300e-4748-916f-5811c337e830', 'session_ref': 'OpaqueRef:8f327b0d-9799-420e-8164-a81353aa99ee', 'device_config': {'location': 'storage-smc.ujezd.net:/zdata/xen-iso-library', 'type': 'nfs_iso', 'SRmaster': 'true', 'nfsversion': '4'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:55ecd645-3bf5-4a06-8aae-79443b019047'}
Dec 27 22:32:02 xen-2u SM: [25713] _testHost: Testing host/port: storage-smc.ujezd.net,2049
Dec 27 22:32:02 xen-2u SM: [25713] ['/usr/sbin/rpcinfo', '-s', 'storage-smc.ujezd.net']
Dec 27 22:32:02 xen-2u SM: [25713] pread SUCCESS
Dec 27 22:32:02 xen-2u SM: [25713] NFS service not ready on server storage-smc.ujezd.net
And this is part of the code I suspect produces that error
RPCINFO_BIN = "/usr/sbin/rpcinfo"
SHOWMOUNT_BIN = "/usr/sbin/showmount"
[...]
def check_server_service(server):
"""Ensure NFS service is up and available on the remote server.
Returns False if fails to detect service after
NFS_SERVICE_RETRY * NFS_SERVICE_WAIT
"""
retries = 0
errlist = [errno.EPERM, errno.EPIPE, errno.EIO]
while True:
try:
services = util.pread([RPCINFO_BIN, "-s", "%s" % server])
services = services.split("\n")
for i in range(len(services)):
if services[i].find("nfs") > 0:
return True
except util.CommandException, inst:
if not int(inst.code) in errlist:
raise
util.SMlog("NFS service not ready on server %s" % server)
retries += 1
if retries >= NFS_SERVICE_RETRY:
break
time.sleep(NFS_SERVICE_WAIT)
return False
[...]
Best regards
Ondra Knezour
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5062 bytes
Desc: Elektronicky podpis S/MIME
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20210111/1ba40ea0/attachment.bin>
More information about the freebsd-net
mailing list