Suddenly frozen fcntl/stat call on NFS over TCP with MTU 9000
gphoto6 at gmail.com
Mon Sep 15 15:57:03 UTC 2008
Currently I was running a mail server using a netapp filer as backend
>From time to time, the whole system get stuck and lasted for 3-5 minutes.
after that, everything recovers normally. During the "stuck" moment, using
auxw shows 200-300 of mail delivery agent(MDA) processes staying in "D"
The command df certainly does not reponse either.
1. NFS server: NetApp FAS3020
2. NFS client: acting as a smtp/pop3/imap server.
freebsd 7.0-stable (almost 7.1-prelease)
hardware: IBM x3550 server
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xc8000000-0xc9ffffff irq 18 at device 0.0 on pci4
miibus0: <MII bus> on bce0
brgphy0: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
bce1: Ethernet address: 00:1a:64:--:--:--
bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W
(0x04000305); Flags( MFW MSI )
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
inet 192.168.1.166 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseTX <full-duplex>)
After further investigation, I found that the situation is most severe when
nfs over tcp and using mtu 9000. If nfs mount is changed to either
(over udp and mtu 9000) or (over tcp and mtu 1500), things get significantly
improvement. The frequency of "suddenly hang" decreases from every 10-15 min
to several hours.
Another observation is the "freeze" happens more frequently when server
load is high, especially working hours. So I believed it is tightly related
to server load (or nfs load).
I tried to modify the source code of MDA (maildrop) and adding some debug
code to identify the problem. What I found is:
1) MDA processing time always approximate 0 sec or < 1 sec when things work
2) MDA processing time may up to 30 second when system got stuck. If the
incoming email continues to come, later emails may cost up to 200 second
to complete. At this time, using ps auxw shows MDAs were in "D" status.
3) Detail trace shows the processing time spent were waiting around the
fcntl (lock) and stat(fstat) code.
One more thing to note:
I've tried to turn on and off rpc.statd,rpc.lockd, -L mount, even compile
NFSLOCKD in kernel. All were in vain, things still got stuck when using
NFS over TCP with mtu 9000.
We have already lots of mail servers whose hardware were different and OS is
freebsd 6-stable. Softwares were all the same but with prior version. Those
servers didn't show any of the above strange behavior.
Based on all of the above experiment and observation, I guess
there might be something wrong with:
1) NFS or network stack of freebsd 7
2) fcntl/stat over NFS
3) bce driver
Need your help/suggestion to solve the problem!
Thanks very much.
More information about the freebsd-stable