"RX ring hdr initialization error"

[Pyun YongHyeon]

On Wed, Aug 11, 2010 at 10:19:11AM +0200, Victor Ophof wrote:
> 
> 
>  
> From: pyunyh at gmail.com
> Date: Tue, 10 Aug 2010 14:37:54 -0700
> To: mr4hughz at hotmail.com
> CC: jfv at freebsd.org; freebsd-net at freebsd.org
> Subject: Re: "RX ring hdr initialization error"
> 
> On Tue, Aug 10, 2010 at 12:52:56PM +0200, Victor Ophof wrote:
> > 
> > 
> > 
> > 
> > Hi 
> > 
> >  I've bought a asus M4a78-EM Motherboard.  to build a NAS on, 
> > thinking the onboard Realtek would be sufficant speed 
> > unfortunatly the onboard fives 16/31 mbs at best 
> > 
> > ps later It improved with enabeling "polling" in the kernel (duh) 
> > 
> > so I had a PCI intel GT nic around, what gave intermittent tcp/ip connections in a other machine (ESXi) 
> > unfortunatly this was the same with Freebsd (card issue?) The card is still in the machine
> > even with the Intel supplied BSD driver 
> > 
> > now I bought a PCIe intel CT nic, put it in and the kernel panic with 
> > "RX ring hdr initialization error"
> > so replaced the intel with the freebsd one by doing 
> > intel overwrites the freebsd one /boot/kernel/if_em.ko 
> > # cd /usr/src/sys/modules/em/ && make obj depend all install
> > (was already in the kernel) 
> >  
> > 
> > still panic 
> > anybody got some idea's howto fix ? 
> > 
>  --- reaction pyunyh ---
> I have been using the attached patch for em(4)/igb(4) controllers.
> These drivers explicitly calls panic(9) when memory allocation
> failure happens. I don't think it's good idea to panic the box
> under resource shortage condition as it's common to see this
> situation on heavily loaded servers.
>  
> The patch does not solve the one issue yet. The panic caused by
> RX buffer allocation failure condition which in turn means you're
> allocating a lot of buffers. Reduce number of descriptors if you 
> increased that too high and see whether the issue could be gone.
> ---/reaction pyunyh ---What buffers /descriptors I need to reduce? I have 2gb ram and set the following in /boot/loader.conf vm.kmem_size_max="1024m"

The loader tunables are hw.em.txd and hw.em.rxd. I thought you
increased TX/RX descriptor size to large value(e.g. 4096).

> vm.kmem_size="1024m"
> #vfs.zfs.prefetch_disable=1
> vm.kmem_size="2048M"
> vfs.zfs.arc_min="1024M"
> vfs.zfs.arc_max="1536M"
> vfs.zfs.vdev.min_pending=2
> vfs.zfs.vdev.max_pending=8
> vfs.zfs.txg.timeout=5
> aio_load="YES"
> ahci_load="YES"
>     

I guess zfs consumed a lot of memory such that em(4) was not able
to allocate RX buffers. It seems there is nothing can be done in
this case unless some memory is reclaimed from zfs. I'm not
familiar with zfs internals but others can comment on this.

However the patch should fix the panic under these resource
shortage situation.