Continuing problems in a bridged VNET setup

[Patrick M. Hausen]

Hi all,

we still experience occasional network outages in production,
yet have not been able to find the root cause.

We run around 50 servers with VNET jails. some of them with
a handful, the busiest ones with 50 or more jails each.

Every now and then the jails are not reachable over the net,
anymore. The server itself is up and running, all jails are
up and running, one can ssh to the server but none of the
jails can communicate over the network.

There seems to be no pattern to the time of occurrance except
that more jails on one system make it "more likely".
Also having more than one bridge, e.g. for private networks
between jails seems to increase the probability.
When a server shows the problem it tends to get into the state
rather frequently, a couple of hours inbetween. Then again
most servers run for weeks without exhibiting the problem.
That's what makes it so hard to reproduce. The last couple of
days one system was failing regularly until we reduced the number
of jails from around 80 to around 50. Now it seems stable again.

I have a test system with lots of jails that I work with gatling
that did not show a single failure so far :-(


Setup:

All jails are iocage jails with VNET interfaces. They are
connected to at least one bridge that starts with the
physical external interface as a member and gets jails'
epair interfaces added as they start up. All jails are managed
by iocage.

ifconfig_igb0="-rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag -vlanhwtso up"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="inet0"
ifconfig_inet0="addm igb0 up"
ifconfig_inet0_ipv6="inet6 <host-address>/64 auto_linklocal"

$ iocage get interfaces vpro0087
vnet0:inet0

$ ifconfig inet0
inet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 90:1b:0e:63:ef:51
	inet6 fe80::921b:eff:fe63:ef51%inet0 prefixlen 64 scopeid 0x4
	inet6 <host-address> prefixlen 64
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: bridge
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 7 priority 128 path cost 2000
	member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 6 priority 128 path cost 2000
	member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 2000000


What we tried:

At first we suspected the bridge to become "wedged" somehow. This was
corroborated by talking to various people at devsummits and EuroBSDCon
with Kristof Provost specifically suggesting that if_bridge was
still under giant lock and there might be a problem here that the lock is
not released under some race condition and then the entire bridge subsystem
would be stalled. That sounds plausible given the random occurrance.

But I think we can rule out that one, because:

- ifconfig up/down does not help
- the host is still communicating fine over the same bridge interface
- tearing down the bridge, kldunload (!) of if_bridge.ko followed by
  a new kldload and reconstructing the members with `ifconfig addm`
  does not help, either
- only a host reboot restores function

Finally I created a not iocage managed jail on the problem host.
Please ignore the `iocage` in the path, I used it to populate the
root directory. But it is not started by iocage at boot time and
the manual config is this:

testjail {
        host.hostname = "testjail";   # hostname
        path = "/iocage/jails/testjail/root";     # root directory
        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";
        vnet; 
	vnet.interface = "epair999b";
        exec.prestart += "ifconfig epair999 create; ifconfig epair999a inet6 2A00:B580:8000:8000::1/64 auto_linklocal";
        exec.poststop += "sleep 2; ifconfig epair999a destroy; sleep 2";
 
        # Standard stuff
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
        exec.consolelog = "/var/log/jail_testjail_console.log";
        mount.devfs;          #mount devfs
        allow.raw_sockets;    #allow ping-pong
        devfs_ruleset="4";    #devfs ruleset for this jail
}

$ cat /iocage/jails/testjail/root/etc/rc.conf
hostname="testjail"

ifconfig_epair999b_ipv6="inet6 2A00:B580:8000:8000::2/64 auto_linklocal"

When I do `service jail onestart testjail` I can then ping6 the jail from
the host and the host from the jail. As you can see the if_bridge is not
involved in this traffic.

When the host is in the wedged state and I start this testjail the same
way, no communication across the epair interface is possible.

To me this seems to indicate that not the bridge but all epair interfaces
stop working at the very same time.


OS is RELENG_11_3, hardware and specifically network adapters vary, we have
igb, ix, ixl, bnxt ...


Does anyone have a suggestion what diagnostic measures could help to pinpoint
the culprit? The random occurrance and the fact that the problem seems to
prefer the production environment only makes this a real pain ...


Thanks and kind regards,
Patrick
-- 
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Kaiserallee 13a
76133 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
info at punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein