mbuf_jumbo_9k & iSCSI failing

[YongHyeon PYUN]

On Mon, Jun 26, 2017 at 03:44:58PM +0200, Julien Cigar wrote:
> On Mon, Jun 26, 2017 at 04:13:33PM +0300, Andrey V. Elsukov wrote:
> > On 25.06.2017 18:32, Ryan Stone wrote:
> > > Having looking at the original email more closely, I see that you showed an
> > > mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf
> > > clusters increase while you are far below the zone's limit means that
> > > you're definitely running into the bug I'm describing, and this bug could
> > > plausibly cause the iSCSI errors that you describe.
> > > 
> > > The issue is that the newer version of the driver tries to allocate a
> > > single buffer to accommodate an MTU-sized packet.  Over time, however,
> > > memory will become fragmented and eventually it can become impossible to
> > > allocate a 9k physically contiguous buffer.  When this happens the driver
> > > is unable to allocate buffers to receive packets and is forced to drop
> > > them.  Presumably, if iSCSI suffers too many packet drops it will terminate
> > > the connection.  The older version of the driver limited itself to
> > > page-sized buffers, so it was immune to issues with memory fragmentation.
> > 
> > I think it is not mlxen specific problem, we have the same symptoms with
> > ixgbe(4) driver too. To avoid the problem we have patches that are
> > disable using of 9k mbufs, and instead only use 4k mbufs.
> 
> I had the same issue on a lightly loaded HP DL20 machine (BCM5720 
> chipsets), 8GB of RAM, running 10.3. Problem usually happens
> within 30 days with 9k jumbo clusters allocation failure.
> 

This looks strange to me.  If I recall correctly bge(4) does not
request physically contiguous 9k jumbo buffers for BCM5720 so it
wouldn't suffer from memory fragmentation. (It uses m_cljget() and
takes advantage of extended RX BDs to handle up to 4 DMA segments).
If your controller is either BCM5714/BCM5715 or BCM5780, it
requires physically contiguous 9k jumbo buffers to handle jumbo
frames though.