freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available

[gnn at freebsd.org]

At Sat, 24 Jan 2009 16:20:06 +0000,
Rui Paulo wrote:
> 
> 
> On 24 Jan 2009, at 12:54, Yony Yossef wrote:
> 
> > Hi All,
> >
> > I'm facing a temporary network hang on my interfaces following a flood
> > ping/stress udp test.
> >
> > I'm running a netperf UDP test which is giving results but does not  
> > return
> > to the shell.
> > client output:
> >
> > UDP UNIDIRECTIONAL SEND TEST from fe80::202:c9ff:fe02:e1fe%mtnic0
> > (fe80::202:c9ff:fe02:e1fe) port 0 AF_INET6 to  
> > fe80::202:c9ff:fe02:e1f4%mt
> > nic0 (fe80::202:c9ff:fe02:e1f4) port 0 AF_INET6
> > Socket  Message  Elapsed      Messages
> > Size    Size     Time         Okay Errors   Throughput
> > bytes   bytes    secs            #      #   10^6bits/sec
> >
> > 32768    1472   10.02      547428 1694280     643.60
> > 32768           10.02       25089             29.50
> >
> >
> > (HANG)
> >
> > After a minute or two it returns to the shell with the following  
> > message:
> > shutdown_control: no response received  errno 55
> >
> > 20 minutes later (!!) the interface is working again.
> >
> > netstat -m and vmstat -z outputs during the hang time:
> >
> > # netstat -m
> > 25687/6578/32265 mbufs in use (current/cache/total)
> > 17404/2438/19842/65536 mbuf clusters in use (current/cache/total/max)
> > 0/1024 mbuf+clusters out of packet secondary zone in use (current/ 
> > cache)
> > 2071/1369/3440/65536 4k (page size) jumbo clusters in use
> > (current/cache/total/max)
> > 0/0/0/65536 9k jumbo clusters in use (current/cache/total/max)
> > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
> > 49513K/11996K/61510K bytes allocated to network (current/cache/total)
> > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> > 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> > 0/0/0 sfbufs in use (current/peak/max)
> > 0 requests for sfbufs denied
> > 0 requests for sfbufs delayed
> > 0 requests for I/O initiated by sendfile
> > 0 calls to protocol drain routines
> 
> I think there are too many mbufs in use. You're probably facing an  
> mbuf leakage and that causes an interface hang.
> 
If this is a large memory machine try upping the number of clusters
and mbufs.  On 64 bit systems with large memories 1,000,000 mbufs is
not unheard of.

kern.ipc.nmbclusters: 1000000

Also, with UDP you can easily overrun different buffers within the
system.  You might also look at:

netstat -id

and see if the driver is dropping packets, and if so you might up its
send queue.  

Best,
George