Frequent hickups on the networking layer

[Mark Schouten]

Hi,

Yes, it did. I see no mbuf errors anymore, no Ethernet errors. Ctld does not crash anymore, it kept running after lowering the mtu to 1500.

I am using vlans and the weirdest thing when lowering the mtu was that everything went crazy when I only lowered the mtu for the vlan interface. Ctld would not start completely, pings started taking several hundred milliseconds. It just wouldn't work anymore. Only when I lowered all interfaces to 1500, stuff was ok and has been since.

I do see a lot of jumbo page allocations during backups at night, which might be nfsd or ctld, but it's not causing any issues.

I've learned, for now: FreeBSD and jumbo frames is a no-go. 

Hope it helps for you too.


Regards,

-- 
Mark Schouten
Tuxis Internet Engineering
mark at tuxis.nl / 0318 200208

> On 10 May 2015, at 02:17, "Christopher Forgeron" <csforgeron at gmail.com> wrote:
> 
> Mark, did switching to a MTU of 1500 ever help?
> 
> I'm currently reliving a problem with this - I'm down to a MTU of 4000, but I still see jumbo pages being allocated - I believe it's my iSCSI setup (using 4k block size, which means the packet is bigger than 4k), but I'm not sure where it's all coming from yet.
> 
> I'm on 10.1 RELEASE fyi. 
> 
> I'm going to patch my network devs to not use MJUM9BYTES and see if that has an effect.
> 
> For me, the problem all started again once I really started putting storage load on the FreeBSD machines. At times, I'm seeing 7 Gbits on the 10 Gbit adapters. 
> 
> Oh, and there are gremlins in the new ctld / iscsi as well. I'll get into that later, but if a heavily loaded iscsi target goes down, when it reboots, the reconnect storm from all the iscsi machines kernel panics the FreeBSD iscsi target host.  My machine looped through three boot-start-panic loops before I caught it and put it into single-user mode. Starting ctld manually seems to make everything okay. 
> 
> 
> 
> 
>> On Wed, May 6, 2015 at 12:03 PM, Mark Schouten <mark at tuxis.nl> wrote:
>> Hi,
>> 
>> On 04/29/2015 04:06 PM, Garrett Wollman wrote:
>> 
>>> If you're using one of the drivers that has this problem, then yes,
>>> keeping your layer-2 MTU/MRU below 4096 will probably cause it to use
>>> 4k (page-sized) clusters instead, which are perfectly safe.
>>> 
>>> As a side note, at least on the hardware I have to support, Infiniband
>>> is limited to 4k MTU -- so I have one "jumbo" network with 4k frames
>>> (that's bridged to IB) and one with 9k frames (that everything else
>>> uses).
>> 
>> So I was thinking, a customer of mine runs mostly the same setup, and has no issues at all. The only difference, MTU of 1500 vs MTU of 9000.
>> 
>> I also created a graph in munin, graphing the number of mbuf_jumbo requests and failures. I find that when lots of writes occur to the iscsi-layer, the number of failed requests grow, and so so the number of errors on the ethernet interface. See attached images. My customer is also not suffering from crashing ctld-daemons, which crashes every other minute in my setup.
>> 
>> So tonight I'm going to switch to an MTU of 1500, I'll let you know if that helped.
>> 
>> 
>> Regards,
>> 
>> Mark Schouten
>> 
>> 
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>