zfs/nfs/proftpd problem

[Rick Macklem]

Daniel Braniss wrote:
> > Daniel Braniss wrote:
> > > after upgrading the 'ftp storage' from 8.3 to 9.1-stable, our ftp
> > > server is stuck.
> > >
> > > the old, (ProFTPD Version 1.3.2) and working till before the
> > > upgrade
> > > is stuck
> > > in nlmrcv:
> > > ...
> > > 10000 1213 992 0 44 0 7340 3692 nlmrcv D ?? 0:08.07 proftpd:
> > > ftp -
> > > crawl-66-249-73-193.googlebot.com: anonymous/googlebot at google.com:
> > > RETR 00690145.JPG (proftpd)
> > > ...
> > >
> > I suspect you know that this is waiting for a reply from some
> > rpc.lockd.
> >
> > > so we upgraded the ftp server too, to 9.1/ProFTPD Version 1.3.4b
> > > and
> > > this one
> > > is stuck in rpccwnd:
> > > 10000 1197 984 0 20 0 32292 4792 rpccwnd D ?? 0:00.01 proftpd: ftp
> > > -
> > > mbpro.cs.huji.ac.il: anonymous/mozilla at example.com: LIST (proftpd)
> > >
> > This one is stuck in the client side of UDP for the krpc, in the
> > primitive congestion control stuff that is there.
> may be it's too primitive?
> 
Yes, but the only alternative is no congestion avoidance at all. The
RPC RTT includes round trip time for the messages plus the delay for
processing the RPC at the server. The latter is highly variable and
depends greatly on what the RPC is and how heavily loaded the server
is. (The pre-krpc NFS client could do a little better, since it "knew"
what the RPC was and could assume "writes" would take a lot longer
than a Getattr. A generic krpc implementation can't know anything
about what the RPC does.)

If you're network fabric needs congestion control to achieve low
loss of packets, then TCP is the way to go. Remember that, if any
packet in a request/response is lost, the entire RPC must be retried
when running over UDP.

> >
> > >
> > > any wise suggestions :-)
> > >
> > Well, maybe not wise, but you may already be aware that NFS etc over
> > UDP and the NLM are two of my favourite things (especially the NLM).
> >
> > Basically, it appears to be having difficulties doing RPCs over UDP,
> > at least for the NLM (rpc.lockd), suggesting some transport related
> > issue.
> >
> > First, make sure rpc.statd and rpc.lockd are running on the NFS
> > server
> > and all clients (or disable use of it via the "nolockd" mount
> > option).
> all are ruuning bot rpc.statd and rpc.lockd
> >
> > You can also do a "netstat -s" and see if there is a non-zero count
> > for "fragments dropped due to timeout" in the IP section. (This
> > happens
> > when your network fabric can't handle the burst of IP fragments
> > generated by a large RPC message over UDP.)
> >
> 
> there are none on the cliet (the ftp server)
> 
> > Things you could try:
> > - If you are using a udp mount for NFS...
> >   - reduce your rsize and wsize (especially if "fragments dropped
> >   due
> >     to timeout" is non-zero)
> >   or
> >   - switch to TCP
> >
> > If you are not using udp mounts, then the NLM (rpc.lockd) is using
> > UDP anyhow. If you don't need multiple NFS clients to see the file
> > locks, add "nolockd" to your mount(s).
> >
> > Beyond that, you'll need to capture packets and look at them in
> > wireshark, to see what is going on.
> >
> the mount is tcp.
> I have been staring at the tcpdump and nothing sticks out, but it's
> been a
> while
> since I looked at rpc traffic.
> 
I'll assume you are looking at it using wireshark (tcpdump doesn't understand
these protocols). You would be looking for repeated RPC request messages without
a corresponding RPC reply from the other end. They would be NLM or NSM protocol
RPCs. (Since you mentioned that your NFS mount was TCP, it must be the NLM and/or
associated NSM stuff that is using UDP, I think.)

> some facts:
> it happens every time, with any ftp command, it gets stuck on either
> nlmrcv
> or rpccwnd, mostly the latter.
> I will try to disable the lock stuff, but isn't it avoiding the issue?
> 
If you mean "avoiding the use of a fundamentally flawed protocol designed
in the 1980s for a handful of locally connected machines that always
remain up with the same fixed hostname/ip address" then, yes, you are trying to
avoid the issue. Further to that, the protocol was never well published, so
implementations "guessed" w.r.t. the semantics for things like "how many
times should the NSM try to ping another machine before assuming the other
machine has crashed and lost the file lock.

One of the main parts of NFSv4 was an effort to fix file
locking. Although it isn't widely adopted yet, it is now a 10year old
protocol. (RFC3530 is dated April 2003, if I recall correctly.)

Since rpc.lockd and rpc.statd haven't changed much in a long time and
are essentially the same in 8.3 as stable/9, I'd suspect that something
else has broken this (assuming it worked fine for 8.3). I would suspect
the network device driver for your hardware and I'd suggest trying
things like disabling checksum offload options, TSO and anything else
you can try via ifconfig. If you happen to have a different kind of
network hardware port, I'd try switching to that as well.

rick

> > Good luck with it, rick
> thanks,
> danny