increasing 'requests for jumbo clusters denied'

[Andre Oppermann]

On 11.03.2013 08:52, Ihsan Junaidi Ibrahim wrote:
> Hi,
>
> I'm on 9.0-RELEASE-p3 and have had a number of instances where my igb0 network connectivity locked up under heavy load.

This problem is also known on CURRENT and we are under active investigation
on how to solve it properly.

> I've had another em1 device which I used for side-band access which does not have as much load as my primary igb0 device.

Jumbo mbufs are only used if you increase the MTU on your interface.
As long as you stay on the standard ethernet MTU of 1500 no jumbo
mbufs will be used.

> I've read previous cases of this but can't seem to find any fitting solutions.
>
> I've tried increasing mbufs and jumbo mbufs:
> nsv01-kul:ihsan $ sysctl -a | grep nmb
> kern.ipc.nmbjumbo16: 3200
> kern.ipc.nmbjumbo9: 25600
> kern.ipc.nmbjumbop: 12800
> kern.ipc.nmbclusters: 262144
>
> but I'm still receiving a lot of 'requests for jumbo clusters denied' stats on netstat -m output.
> nsv01-kul:ihsan $ netstat -m
> 7347/3063/10410 mbufs in use (current/cache/total)
> 334/2658/2992/262144 mbuf clusters in use (current/cache/total/max)
> 334/1330 mbuf+clusters out of packet secondary zone in use (current/cache)
> 1749/1310/3059/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
> 5116/563/5679/25600 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
> 55544K/16388K/71933K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/25154247/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
>
> The server has not locked up yet after increasing the mbufs (been less than 8 hours since increasing this) but can someone enlighten me what could be the causes for jumbo clusters requests being denied although I've increased the 9k jumbo clusters to 25600?

The problem is not the limits but memory fragmentation.  Jumbo
mbufs larger than 4k (PAGE_SIZE) have to be created with adjacent
and contiguous VM pages in physical and virtual memory.  As soon
as your server becomes busy the physical memory becomes 'fragmented'
in the sense that such jumbo mbuf requests can't be fulfilled anymore
at allocation time.

Most modern high speed network cards can do scatter DMA and could
simply split up a received jumbo frame across as many normal mbuf
clusters as necessary.

For a temporary fix you could forgo using a MTU larger than 4096.

-- 
Andre