FreeBSD 10.1-REL - network unaccessible after high traffic

Christopher Forgeron csforgeron at gmail.com
Fri Jun 12 12:37:39 UTC 2015 

rsycn burns memory - I'd say you have a good chance you're running out of
mem before it's replenished.

For vmstat 5 - Don't run it on console. Connect via a second box with ssh,
and run it there - That way it's the last thing on the ssh terminal screen
when the box dies, and you'll have your proof.

On Fri, Jun 12, 2015 at 9:31 AM, Cs <bimmer at field.hu> wrote:

> machine has been restarted before I could check the "vmstat 5" output.
> Yep, it's rsync. Anyway I disabled the backup transfer it'll solve, but I
> can't really accept this for solution.
>
>
> 2015.06.12. 14:29 keltezéssel, Christopher Forgeron írta:
>
>> Well, even at low speed it could drop due to memory from what I've seen.
>>
>> What was the last line from vmstat 5 before it locked up?
>>
>>   I find that the em driver isn't crap, but there is a deeper problem
>> inside
>> of FreeBSD that is being exposed now - For me it's due to faster network
>> connections.
>>
>>   Are you using rsync to move the files?
>>
>> On Fri, Jun 12, 2015 at 7:17 AM, Cs <bimmer at field.hu> wrote:
>>
>>  it seems it's not memory related. Server just died a few minutes ago
>>> during transporting the backup (400GB) around 800Mbps speed..
>>> will disable remote backup, it's a shame that em driver is such a crap.
>>>
>>>
>>> 2015.06.08. 5:01 keltezéssel, Christopher Forgeron írta:
>>>
>>>  You know what helped me:
>>>>
>>>> 'vmstat 5'
>>>>
>>>> Leave that running. If the last thing on the console after a crash/hang
>>>> is
>>>> vmstat showing 8k of memory left, then you're in the same problem-park
>>>> as
>>>> me.
>>>>
>>>> My 10.1 96GiB RAM box is chewing ~8 GiB of RAM in less than 5 seconds,
>>>> and
>>>> then crashing/panicking/hanging.
>>>>
>>>> There's others with this issues if you search for it; a sysctl
>>>> to vm.v_free_min to double or triple that value may help, but first let
>>>> us
>>>> know if that's what is bonking your sever.
>>>>
>>>>
>>>>
>>>> On Sun, Jun 7, 2015 at 11:03 AM, Cs <bimmer at field.hu> wrote:
>>>>
>>>>   ok, just lowered it to 1500 but please also note that it was on 1500
>>>> for
>>>>
>>>>> 2
>>>>> years
>>>>>
>>>>> 2015.06.07. 14:57 keltezéssel, Rick Macklem írta:
>>>>>
>>>>>   Since disabling TSO didn't help, you could try dropping to 1500mtu
>>>>>
>>>>>> on both interfaces. Some people run into problems when 9K jumbo
>>>>>> clusters
>>>>>> fragment the kernel address space used to allocate mbufs.
>>>>>>
>>>>>> Good luck with it, rick
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>
>>>>>>   Hi All,
>>>>>>
>>>>>>> It worked fine for two weeks but I had a network outage 2 days ago
>>>>>>> then
>>>>>>> today. Tried to disable rxcsum and txcsum after the first one, didn't
>>>>>>> help. Don't know what else to do it's a shame that I can't use this
>>>>>>> card
>>>>>>> with fbsd i REALLY don't want to install linux instead but my
>>>>>>> production
>>>>>>> servers outages are not welcomed by the customers..
>>>>>>>
>>>>>>> 2015.05.26. 10:36 keltezéssel, Cs írta:
>>>>>>>
>>>>>>>   Thanks Mark, good idea. I found this thread which is exactly the
>>>>>>>
>>>>>>>> same
>>>>>>>> problem as mine:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/
>>>>>>>>
>>>>>>>> Will see if it helps in a couple weeks.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Csaba
>>>>>>>>
>>>>>>>> 2015.05.26. 10:30 keltezéssel, Mark Schouten írta:
>>>>>>>>
>>>>>>>>   Oh, didn't see your lowest remark. Then, the next thing that comes
>>>>>>>>
>>>>>>>>> past here a few times per week is 'Try disabling TSO'.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
>>>>>>>>> Mark Schouten  | Tuxis Internet Engineering
>>>>>>>>> KvK: 61527076 | http://www.tuxis.nl/
>>>>>>>>> T: 0318 200208 | info at tuxis.nl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>      Van:   Cs <bimmer at field.hu>
>>>>>>>>>      Aan:   Mark Schouten <mark at tuxis.nl>
>>>>>>>>>      Cc:    <freebsd-net at freebsd.org>
>>>>>>>>>      Verzonden:   25-5-2015 11:12
>>>>>>>>>      Onderwerp:   Re: FreeBSD 10.1-REL - network unaccessible after
>>>>>>>>>      high
>>>>>>>>> traffic
>>>>>>>>>
>>>>>>>>> It was on 1500 for ~3 years :)
>>>>>>>>>      Regards,
>>>>>>>>> Csaba
>>>>>>>>>          On May 25, 2015, 10:30, at 10:30, Mark Schouten
>>>>>>>>>          <mark at tuxis.nl>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>   Try lowering your mtu to 1500, that worked miracles for me..
>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Mark Schouten
>>>>>>>>>> Tuxis Internet Engineering
>>>>>>>>>> mark at tuxis.nl / 0318 200208
>>>>>>>>>>
>>>>>>>>>>    On 25 May 2015, at 09:36, "Cs" <bimmer at field.hu> wrote:
>>>>>>>>>>
>>>>>>>>>>       Hi all,
>>>>>>>>>>>      I have two FreeBSd 10.1-RELEASE servers connected to each
>>>>>>>>>>>      other.
>>>>>>>>>>> They
>>>>>>>>>>>
>>>>>>>>>>>   were connected via cross link, but they are connected to a
>>>>>>>>>>> cisco
>>>>>>>>>>>
>>>>>>>>>> switch
>>>>>>>>>> now (the problem was the same with cross link too). When
>>>>>>>>>> transferring
>>>>>>>>>> huge files (50-500GB backup files) via Gigabit (it is important!)
>>>>>>>>>> the
>>>>>>>>>> network randomly dies. The backup runs every day/week and
>>>>>>>>>> sometimes the
>>>>>>>>>> connection is ok for months sometimes it happens twice a week.
>>>>>>>>>> When the
>>>>>>>>>> network dies I can log in to the server via IPMI and use the
>>>>>>>>>> console
>>>>>>>>>> everything is OK, but can't send anything out on the network.
>>>>>>>>>> ifconfig
>>>>>>>>>> em0 down/up doesn't help nor netif restart. The problem never
>>>>>>>>>> occured
>>>>>>>>>> when I used 100Mbit connection between them, but it was 3com NIC
>>>>>>>>>> (xl),
>>>>>>>>>> gigabit adapter is Intel (em0). When I limit the transfer rate
>>>>>>>>>> (rsync
>>>>>>>>>> bandwith limit or ipfw pipe) the problem is much more rare.
>>>>>>>>>>
>>>>>>>>>>       I tried to set these tuning parameters on both servers with
>>>>>>>>>>
>>>>>>>>>>> different
>>>>>>>>>>>
>>>>>>>>>>>   buffer size but nothing helped:
>>>>>>>>>>>
>>>>>>>>>>       # cat /etc/sysctl.conf
>>>>>>>>>>
>>>>>>>>>>> security.bsd.see_other_uids=0
>>>>>>>>>>> net.inet.tcp.recvspace=512000
>>>>>>>>>>> net.route.netisr_maxqlen=2048
>>>>>>>>>>> kern.ipc.nmbclusters=1310720
>>>>>>>>>>> net.inet.tcp.sendbuf_max=16777216
>>>>>>>>>>> net.inet.tcp.recvbuf_max=16777216
>>>>>>>>>>> kern.ipc.soacceptqueue=32768
>>>>>>>>>>>      # cat /boot/loader.conf
>>>>>>>>>>> geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8))
>>>>>>>>>>> ipfw_load="YES"
>>>>>>>>>>> net.inet.ip.fw.default_to_accept=1
>>>>>>>>>>> kern.maxusers=4096
>>>>>>>>>>> accf_data_load="YES"
>>>>>>>>>>>      The duplex settings are identical on both servers.
>>>>>>>>>>>      Server A:
>>>>>>>>>>> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>>>>>>>>>>> mtu
>>>>>>>>>>>
>>>>>>>>>>>   9000
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>             ether 00:25:90:24:52:66
>>>>>>>>>>
>>>>>>>>>>             inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x
>>>>>>>>>>>            nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>>>>>>>>>            media: Ethernet autoselect (1000baseT <full-duplex>)
>>>>>>>>>>>            status: active
>>>>>>>>>>>      Server B:
>>>>>>>>>>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>>>>>>>>>>> mtu
>>>>>>>>>>>
>>>>>>>>>>>   9000
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>             ether 00:30:48:dd:fe:3e
>>>>>>>>>>
>>>>>>>>>>             inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x
>>>>>>>>>>>            nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>>>>>>>>>            media: Ethernet autoselect (1000baseT <full-duplex>)
>>>>>>>>>>>            status: active
>>>>>>>>>>>      Today I tried to set mtu to 9000 but in tcpdump I see that
>>>>>>>>>>>      during
>>>>>>>>>>> scp
>>>>>>>>>>>
>>>>>>>>>>>   it is still 1500:
>>>>>>>>>>>
>>>>>>>>>>         x.x.x.x.222 > x.x.x.x.37612: Flags [.], cksum 0xb6ee
>>>>>>>>>>
>>>>>>>>>>>        (incorrect ->
>>>>>>>>>>>
>>>>>>>>>>>   0xda6f), seq 35749, ack 113701596, win 7986, options
>>>>>>>>>>> [nop,nop,TS
>>>>>>>>>>>
>>>>>>>>>> val
>>>>>>>>>> 3103966325 ecr 853712893], length 0
>>>>>>>>>>
>>>>>>>>>>   09:27:33.912354 IP (tos 0x8, ttl 64, id 1028, offset 0, flags
>>>>>>>>>>
>>>>>>>>>>> [DF],
>>>>>>>>>>>
>>>>>>>>>>>   proto TCP (6), length 1500)
>>>>>>>>>>>
>>>>>>>>>>   09:27:33.912358 IP (tos 0x8, ttl 64, id 1029, offset 0, flags
>>>>>>>>>>
>>>>>>>>>>> [DF],
>>>>>>>>>>>
>>>>>>>>>>>   proto TCP (6), length 1500)
>>>>>>>>>>>
>>>>>>>>>>         Any ideas? Thanks guys!
>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>>>>
>>>>>>>>>>>   "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>>>>
>>>>>>>>>>   _______________________________________________
>>>>>>>>>>
>>>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    _______________________________________________
>>>>>>>>>
>>>>>>>>>  freebsd-net at freebsd.org mailing list
>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>> To unsubscribe, send any mail to
>>>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>
>>>>>>>>   _______________________________________________
>>>>>>>>
>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>
>>>>>>>   _______________________________________________
>>>>>>>
>>>>>> freebsd-net at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>>>
>>>>>   _______________________________________________
>>>>>
>>>> freebsd-net at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>>
>>>>  _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>>>  _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>

More information about the freebsd-net mailing list