problem with 9k jumbo clusters

[Yuriy Tabolin]

Hi All.
I have a server with two Intel 10G NIC. OS FreeBSD 10.1-Release amd64. 
Server works like NFS, samba-server and iSCSI target. Both NICs 
aggregated into lagg device and set MTU 9014 to them. There are some 
tuning sysctl.conf:
kern.maxfiles=6289601
kern.maxfilesperproc=5660640
kern.maxvnodes=3339565
kern.ipc.nmbclusters=12255588
kern.ipc.nmbjumbop=6127794
kern.ipc.nmbufs=78435780
kern.ipc.maxsockbuf=16777216
kern.ipc.maxsockets=6289600
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=65536
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=65536

After some days of working, the errors are appearing:
ix1: Interface stopped DISTRIBUTING, possible flapping
ix0: Interface stopped DISTRIBUTING, possible flapping
ix0: Could not setup receive structures
ix1: Could not setup receive structures

After that errors the NICs stoped working. netstat -m shows:
32881/33854/66735 mbufs in use (current/cache/total)
16370/8198/24568/12255588 mbuf clusters in use (current/cache/total/max)
16370/4807 mbuf+clusters out of packet secondary zone in use (current/cache)
0/873/873/6127794 4k (page size) jumbo clusters in use 
(current/cache/total/max)
16383/21517/37900/1815641 9k jumbo clusters in use (current/cache/total/max)
0/0/0/1021298 16k jumbo clusters in use (current/cache/total/max)
188407K/222004K/410411K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/101414306/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

9k jumbo clusters max is too big, but looks like system cannot allocate 
them. There are huge number of "9k requests for jumbo clusters denied". 
ifconfig ix down/up don't helped, reboot is needed. Thanks for any help!


--
Best regards,
Tabolin Yuriy
System administrator
Speech Technology Center