FreeBSD 10.1-REL - network unaccessible after high traffic

Christopher Forgeron csforgeron at gmail.com
Fri Jun 12 12:43:19 UTC 2015


Ah, but the 'why' will come later, after we know for sure what the 'what'
is in your problem.

I'm just pointing out the problems that I'm having, as yours sound similar.
Once the box runs out of memory, all sorts of interesting things can
happen. Perhaps that's not your case, but it's quite possible.

Setup a remote terminal, do the copy again, and send in the last few lines
of 'vmstat 5' after it's locked up, perhaps I can help.

On Fri, Jun 12, 2015 at 9:39 AM, Cs <bimmer at field.hu> wrote:

>  but why is that machine runs fine except the network if it's memory
> related? swap didn't increased before the network outage.
>
>
> 2015.06.12. 14:37 keltezéssel, Christopher Forgeron írta:
>
> rsycn burns memory - I'd say you have a good chance you're running out of
> mem before it's replenished.
>
>  For vmstat 5 - Don't run it on console. Connect via a second box with
> ssh, and run it there - That way it's the last thing on the ssh terminal
> screen when the box dies, and you'll have your proof.
>
> On Fri, Jun 12, 2015 at 9:31 AM, Cs <bimmer at field.hu> wrote:
>
>> machine has been restarted before I could check the "vmstat 5" output.
>> Yep, it's rsync. Anyway I disabled the backup transfer it'll solve, but I
>> can't really accept this for solution.
>>
>>
>> 2015.06.12. 14:29 keltezéssel, Christopher Forgeron írta:
>>
>>> Well, even at low speed it could drop due to memory from what I've seen.
>>>
>>> What was the last line from vmstat 5 before it locked up?
>>>
>>>   I find that the em driver isn't crap, but there is a deeper problem
>>> inside
>>> of FreeBSD that is being exposed now - For me it's due to faster network
>>> connections.
>>>
>>>   Are you using rsync to move the files?
>>>
>>> On Fri, Jun 12, 2015 at 7:17 AM, Cs <bimmer at field.hu> wrote:
>>>
>>>  it seems it's not memory related. Server just died a few minutes ago
>>>> during transporting the backup (400GB) around 800Mbps speed..
>>>> will disable remote backup, it's a shame that em driver is such a crap.
>>>>
>>>>
>>>> 2015.06.08. 5:01 keltezéssel, Christopher Forgeron írta:
>>>>
>>>>  You know what helped me:
>>>>>
>>>>> 'vmstat 5'
>>>>>
>>>>> Leave that running. If the last thing on the console after a
>>>>> crash/hang is
>>>>> vmstat showing 8k of memory left, then you're in the same problem-park
>>>>> as
>>>>> me.
>>>>>
>>>>> My 10.1 96GiB RAM box is chewing ~8 GiB of RAM in less than 5 seconds,
>>>>> and
>>>>> then crashing/panicking/hanging.
>>>>>
>>>>> There's others with this issues if you search for it; a sysctl
>>>>> to vm.v_free_min to double or triple that value may help, but first
>>>>> let us
>>>>> know if that's what is bonking your sever.
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jun 7, 2015 at 11:03 AM, Cs <bimmer at field.hu> wrote:
>>>>>
>>>>>   ok, just lowered it to 1500 but please also note that it was on 1500
>>>>> for
>>>>>
>>>>>> 2
>>>>>> years
>>>>>>
>>>>>> 2015.06.07. 14 <2015.06.07.%2014>:57 keltezéssel, Rick Macklem írta:
>>>>>>
>>>>>>   Since disabling TSO didn't help, you could try dropping to 1500mtu
>>>>>>
>>>>>>> on both interfaces. Some people run into problems when 9K jumbo
>>>>>>> clusters
>>>>>>> fragment the kernel address space used to allocate mbufs.
>>>>>>>
>>>>>>> Good luck with it, rick
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>
>>>>>>>   Hi All,
>>>>>>>
>>>>>>>> It worked fine for two weeks but I had a network outage 2 days ago
>>>>>>>> then
>>>>>>>> today. Tried to disable rxcsum and txcsum after the first one,
>>>>>>>> didn't
>>>>>>>> help. Don't know what else to do it's a shame that I can't use this
>>>>>>>> card
>>>>>>>> with fbsd i REALLY don't want to install linux instead but my
>>>>>>>> production
>>>>>>>> servers outages are not welcomed by the customers..
>>>>>>>>
>>>>>>>> 2015.05.26. 10 <2015.05.26.%2010>:36 keltezéssel, Cs írta:
>>>>>>>>
>>>>>>>>   Thanks Mark, good idea. I found this thread which is exactly the
>>>>>>>>
>>>>>>>>> same
>>>>>>>>> problem as mine:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/
>>>>>>>>>
>>>>>>>>> Will see if it helps in a couple weeks.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Csaba
>>>>>>>>>
>>>>>>>>> 2015.05.26. 10 <2015.05.26.%2010>:30 keltezéssel, Mark Schouten
>>>>>>>>> írta:
>>>>>>>>>
>>>>>>>>>   Oh, didn't see your lowest remark. Then, the next thing that
>>>>>>>>> comes
>>>>>>>>>
>>>>>>>>>> past here a few times per week is 'Try disabling TSO'.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
>>>>>>>>>> Mark Schouten  | Tuxis Internet Engineering
>>>>>>>>>> KvK: 61527076 | http://www.tuxis.nl/
>>>>>>>>>> T: 0318 200208 | info at tuxis.nl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>      Van:   Cs <bimmer at field.hu>
>>>>>>>>>>      Aan:   Mark Schouten <mark at tuxis.nl>
>>>>>>>>>>      Cc:    <freebsd-net at freebsd.org>
>>>>>>>>>>      Verzonden:   25-5-2015 11:12
>>>>>>>>>>      Onderwerp:   Re: FreeBSD 10.1-REL - network unaccessible
>>>>>>>>>> after
>>>>>>>>>>      high
>>>>>>>>>> traffic
>>>>>>>>>>
>>>>>>>>>> It was on 1500 for ~3 years :)
>>>>>>>>>>      Regards,
>>>>>>>>>> Csaba
>>>>>>>>>>          On May 25, 2015, 10:30, at 10:30, Mark Schouten
>>>>>>>>>>          <mark at tuxis.nl>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>   Try lowering your mtu to 1500, that worked miracles for me..
>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Mark Schouten
>>>>>>>>>>> Tuxis Internet Engineering
>>>>>>>>>>> mark at tuxis.nl / 0318 200208
>>>>>>>>>>>
>>>>>>>>>>>    On 25 May 2015, at 09:36, "Cs" <bimmer at field.hu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>       Hi all,
>>>>>>>>>>>>      I have two FreeBSd 10.1-RELEASE servers connected to each
>>>>>>>>>>>>      other.
>>>>>>>>>>>> They
>>>>>>>>>>>>
>>>>>>>>>>>>   were connected via cross link, but they are connected to a
>>>>>>>>>>>> cisco
>>>>>>>>>>>>
>>>>>>>>>>> switch
>>>>>>>>>>> now (the problem was the same with cross link too). When
>>>>>>>>>>> transferring
>>>>>>>>>>> huge files (50-500GB backup files) via Gigabit (it is important!)
>>>>>>>>>>> the
>>>>>>>>>>> network randomly dies. The backup runs every day/week and
>>>>>>>>>>> sometimes the
>>>>>>>>>>> connection is ok for months sometimes it happens twice a week.
>>>>>>>>>>> When the
>>>>>>>>>>> network dies I can log in to the server via IPMI and use the
>>>>>>>>>>> console
>>>>>>>>>>> everything is OK, but can't send anything out on the network.
>>>>>>>>>>> ifconfig
>>>>>>>>>>> em0 down/up doesn't help nor netif restart. The problem never
>>>>>>>>>>> occured
>>>>>>>>>>> when I used 100Mbit connection between them, but it was 3com NIC
>>>>>>>>>>> (xl),
>>>>>>>>>>> gigabit adapter is Intel (em0). When I limit the transfer rate
>>>>>>>>>>> (rsync
>>>>>>>>>>> bandwith limit or ipfw pipe) the problem is much more rare.
>>>>>>>>>>>
>>>>>>>>>>>       I tried to set these tuning parameters on both servers with
>>>>>>>>>>>
>>>>>>>>>>>> different
>>>>>>>>>>>>
>>>>>>>>>>>>   buffer size but nothing helped:
>>>>>>>>>>>>
>>>>>>>>>>>       # cat /etc/sysctl.conf
>>>>>>>>>>>
>>>>>>>>>>>> security.bsd.see_other_uids=0
>>>>>>>>>>>> net.inet.tcp.recvspace=512000
>>>>>>>>>>>> net.route.netisr_maxqlen=2048
>>>>>>>>>>>> kern.ipc.nmbclusters=1310720
>>>>>>>>>>>> net.inet.tcp.sendbuf_max=16777216
>>>>>>>>>>>> net.inet.tcp.recvbuf_max=16777216
>>>>>>>>>>>> kern.ipc.soacceptqueue=32768
>>>>>>>>>>>>      # cat /boot/loader.conf
>>>>>>>>>>>> geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8))
>>>>>>>>>>>> ipfw_load="YES"
>>>>>>>>>>>> net.inet.ip.fw.default_to_accept=1
>>>>>>>>>>>> kern.maxusers=4096
>>>>>>>>>>>> accf_data_load="YES"
>>>>>>>>>>>>      The duplex settings are identical on both servers.
>>>>>>>>>>>>      Server A:
>>>>>>>>>>>> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>>>>>>>>>>>> mtu
>>>>>>>>>>>>
>>>>>>>>>>>>   9000
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>             ether 00:25:90:24:52:66
>>>>>>>>>>>
>>>>>>>>>>>             inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x
>>>>>>>>>>>>            nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>>>>>>>>>>            media: Ethernet autoselect (1000baseT <full-duplex>)
>>>>>>>>>>>>            status: active
>>>>>>>>>>>>      Server B:
>>>>>>>>>>>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>>>>>>>>>>>> mtu
>>>>>>>>>>>>
>>>>>>>>>>>>   9000
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>             ether 00:30:48:dd:fe:3e
>>>>>>>>>>>
>>>>>>>>>>>             inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x
>>>>>>>>>>>>            nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>>>>>>>>>>            media: Ethernet autoselect (1000baseT <full-duplex>)
>>>>>>>>>>>>            status: active
>>>>>>>>>>>>      Today I tried to set mtu to 9000 but in tcpdump I see that
>>>>>>>>>>>>      during
>>>>>>>>>>>> scp
>>>>>>>>>>>>
>>>>>>>>>>>>   it is still 1500:
>>>>>>>>>>>>
>>>>>>>>>>>         x.x.x.x.222 > x.x.x.x.37612: Flags [.], cksum 0xb6ee
>>>>>>>>>>>
>>>>>>>>>>>>        (incorrect ->
>>>>>>>>>>>>
>>>>>>>>>>>>   0xda6f), seq 35749, ack 113701596, win 7986, options
>>>>>>>>>>>> [nop,nop,TS
>>>>>>>>>>>>
>>>>>>>>>>> val
>>>>>>>>>>> 3103966325 ecr 853712893], length 0
>>>>>>>>>>>
>>>>>>>>>>>   09:27:33.912354 IP (tos 0x8, ttl 64, id 1028, offset 0, flags
>>>>>>>>>>>
>>>>>>>>>>>> [DF],
>>>>>>>>>>>>
>>>>>>>>>>>>   proto TCP (6), length 1500)
>>>>>>>>>>>>
>>>>>>>>>>>   09:27:33.912358 IP (tos 0x8, ttl 64, id 1029, offset 0, flags
>>>>>>>>>>>
>>>>>>>>>>>> [DF],
>>>>>>>>>>>>
>>>>>>>>>>>>   proto TCP (6), length 1500)
>>>>>>>>>>>>
>>>>>>>>>>>         Any ideas? Thanks guys!
>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>>>>>
>>>>>>>>>>>>   "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>>>>>
>>>>>>>>>>>   _______________________________________________
>>>>>>>>>>>
>>>>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    _______________________________________________
>>>>>>>>>>
>>>>>>>>>>  freebsd-net at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>>
>>>>>>>>>   _______________________________________________
>>>>>>>>>
>>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>> To unsubscribe, send any mail to
>>>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>
>>>>>>>>   _______________________________________________
>>>>>>>>
>>>>>>>  freebsd-net at freebsd.org mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org
>>>>>> "
>>>>>>
>>>>>>   _______________________________________________
>>>>>>
>>>>> freebsd-net at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>>>
>>>>>  _______________________________________________
>>>> freebsd-net at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>>
>>>>  _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>>
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>
>
>


More information about the freebsd-net mailing list