freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available

[Barney Cordoba]

--- On Mon, 1/26/09, gnn at freebsd.org <gnn at freebsd.org> wrote:

> From: gnn at freebsd.org <gnn at freebsd.org>
> Subject: Re: freebsd 7.0-RELEASE BUG ping: sendto: No buffer space available
> To: "Rui Paulo" <rpaulo at freebsd.org>
> Cc: "Liran Liss" <liranl at mellanox.co.il>, freebsd-net at freebsd.org, "Yony Yossef" <yonyossef.lists at gmail.com>, johan at nocrew.org, "Amit Krig" <amitk at mellanox.co.il>, "Eitan Shefi" <eitans at mellanox.co.il>
> Date: Monday, January 26, 2009, 12:52 PM
> At Sat, 24 Jan 2009 16:20:06 +0000,
> Rui Paulo wrote:
> > 
> > 
> > On 24 Jan 2009, at 12:54, Yony Yossef wrote:
> > 
> > > Hi All,
> > >
> > > I'm facing a temporary network hang on my
> interfaces following a flood
> > > ping/stress udp test.
> > >
> > > I'm running a netperf UDP test which is
> giving results but does not  
> > > return
> > > to the shell.
> > > client output:
> > >
> > > UDP UNIDIRECTIONAL SEND TEST from
> fe80::202:c9ff:fe02:e1fe%mtnic0
> > > (fe80::202:c9ff:fe02:e1fe) port 0 AF_INET6 to  
> > > fe80::202:c9ff:fe02:e1f4%mt
> > > nic0 (fe80::202:c9ff:fe02:e1f4) port 0 AF_INET6
> > > Socket  Message  Elapsed      Messages
> > > Size    Size     Time         Okay Errors  
> Throughput
> > > bytes   bytes    secs            #      #  
> 10^6bits/sec
> > >
> > > 32768    1472   10.02      547428 1694280    
> 643.60
> > > 32768           10.02       25089            
> 29.50
> > >
> > >
> > > (HANG)
> > >
> > > After a minute or two it returns to the shell
> with the following  
> > > message:
> > > shutdown_control: no response received  errno 55
> > >
> > > 20 minutes later (!!) the interface is working
> again.
> > >
> > > netstat -m and vmstat -z outputs during the hang
> time:
> > >
> > > # netstat -m
> > > 25687/6578/32265 mbufs in use
> (current/cache/total)
> > > 17404/2438/19842/65536 mbuf clusters in use
> (current/cache/total/max)
> > > 0/1024 mbuf+clusters out of packet secondary zone
> in use (current/ 
> > > cache)
> > > 2071/1369/3440/65536 4k (page size) jumbo
> clusters in use
> > > (current/cache/total/max)
> > > 0/0/0/65536 9k jumbo clusters in use
> (current/cache/total/max)
> > > 0/0/0/3200 16k jumbo clusters in use
> (current/cache/total/max)
> > > 49513K/11996K/61510K bytes allocated to network
> (current/cache/total)
> > > 0/0/0 requests for mbufs denied
> (mbufs/clusters/mbuf+clusters)
> > > 0/0/0 requests for jumbo clusters denied
> (4k/9k/16k)
> > > 0/0/0 sfbufs in use (current/peak/max)
> > > 0 requests for sfbufs denied
> > > 0 requests for sfbufs delayed
> > > 0 requests for I/O initiated by sendfile
> > > 0 calls to protocol drain routines
> > 
> > I think there are too many mbufs in use. You're
> probably facing an  
> > mbuf leakage and that causes an interface hang.
> > 
> If this is a large memory machine try upping the number of
> clusters
> and mbufs.  On 64 bit systems with large memories 1,000,000
> mbufs is
> not unheard of.
> 
> kern.ipc.nmbclusters: 1000000
> 
> Also, with UDP you can easily overrun different buffers
> within the
> system.  You might also look at:
> 
> netstat -id
> 
> and see if the driver is dropping packets, and if so you
> might up its
> send queue.  
> 
> Best,
> George

The clusters problem is an artifact of the buffers problem. You tend to
get the "No buffer space" when a send queue gets clogged. Sometimes
its difficult to get unclogged.

For example if you look at some of the ethernet drivers, they will queue
the packet when a send error occurs, hoping to send it when the problem clears. But if you have a problem that doesnt resolve those buffers get
stuck in the queue. The only clean way I've found to free them in that
case is to unload the driver and reload it. 

Barney