From nobody Mon Mar 07 20:40:21 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E6A901A07A7C for ; Mon, 7 Mar 2022 20:40:26 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KC9MG0L7Zz3pX8 for ; Mon, 7 Mar 2022 20:40:26 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by mail-ed1-x536.google.com with SMTP id s10so431794edd.0 for ; Mon, 07 Mar 2022 12:40:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language :from:to:references:in-reply-to:content-transfer-encoding; bh=7tjtvBuK6YYG4kOxX13KGCajhIbOj45i+RaTLFdKVAo=; b=Lvs4Z2gSlB+P++bt+R5FKV1HOHWzZmyqSYl7Bk6JFXxcz8WsHC8D/b5D/49n8INmS+ rq3JvnVYlxoQWoJZn6G96n71cVSCrPrbg2rscpoI5DzJpPvRtCqORn4f/54CHpRXXLPZ Og3hUpsxBv6palxOBHA6QyOOJSRo7Hdszf4VlozmA2V7/bgAZyXlp0MYhYPjIz41PZOD 8DF0TA4+BoBsdoIdfFPY/79WLbe2WHoEYJw4nQyvelsyZqw0W/bKOnWLcAcdVd/xJ4+n Yp7qNDieXZlvK9d40vWtwEZFkQaDTgu92eKZcZ6i5UtUMjZYf58wAP3K9XxxDTZkeOCB mYqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:from:to:references:in-reply-to :content-transfer-encoding; bh=7tjtvBuK6YYG4kOxX13KGCajhIbOj45i+RaTLFdKVAo=; b=GbpXmHsr0cLHT73nU4kL+D76aag9uefqqLtg1osMsNUq2YV596Pvaqcc5NyJHoSePu /XWqFZGFXlY4InTD7H7V6ExDqf3o8X3K7rhwxmbr5L+sp6wy06jZncdp7r6FruT9OCYd wljdj6gM61nl6C3dJyUAJIg6JHUwZ126bFpqpkln0wBWKlQjgjTezoEFWfypx0Ojag37 FjLyd1OipMqH+zI6XfeypLYDJ+i0rORiumRp+J0pjL15FS/qChSsK3Q+qBoBVHdkNUb5 J9tJ5c+apuFQdp2iUeWW1xfX3JBPJBOnmP6GwQp7IvCX/rdvgh4SuKhi+IrBFY8k4vEl tpjQ== X-Gm-Message-State: AOAM530a5X0c9Bm3X3s099Veb8l66bFZDnCXaoCTPBi5KaYOZ3Eldf53 mL7aZ+6gY0vmi48Ac7n6f5f0jKUd+r4= X-Google-Smtp-Source: ABdhPJxXTDFvlzIiauJ11WAmFYs8yjG95bgoLSJUDppK+6BWaslS/bdyMIJ3ogEO+KTB2KfDzmdjbA== X-Received: by 2002:a05:6402:1e8f:b0:3fa:72cb:1733 with SMTP id f15-20020a0564021e8f00b003fa72cb1733mr13120456edf.24.1646685624689; Mon, 07 Mar 2022 12:40:24 -0800 (PST) Received: from [192.168.1.18] (85-147-130-226.cable.dynamic.v4.ziggo.nl. [85.147.130.226]) by smtp.gmail.com with ESMTPSA id k23-20020a1709062a5700b006ccd8fdc300sm5039908eje.180.2022.03.07.12.40.24 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 07 Mar 2022 12:40:24 -0800 (PST) Message-ID: Date: Mon, 7 Mar 2022 21:40:21 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.6.2 Subject: Re: vnet jails loose network connectivity Content-Language: en-US From: Johan Hendriks To: FreeBSD Current References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4KC9MG0L7Zz3pX8 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=Lvs4Z2gS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of johhendriks@gmail.com designates 2a00:1450:4864:20::536 as permitted sender) smtp.mailfrom=johhendriks@gmail.com X-Spamd-Result: default: False [-3.87 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.91)[-0.905]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.962]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::536:from]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 04/03/2022 15:36, Johan Hendriks wrote: > Hello all, i use jails for some testing, but i can not seem to make it > stable. > I use vnet jails with a bridge but when i put some load on it, some > jails loose there network connectivity. > > My setup is as follows, haproxy internal IP 10.233.185.20 using binat > to make it Public accessable. > Then a varnish jail, and two web servers al on the 10.233.185.x range. > > If i give it a little load with hey (hey -h2 -n 10 -c 20 -z 60s > https://wp.test.nl) than within the test the haproxy jail is not > reachable anymore it is not pingable from the host machine, and from > the other jails. restarting the jails solves it, if i leave the system > alone for some time i saw the varnish jail become unresponsive. > > If i do a tcpdump on the epair${name}a interface i do see the packages > from the host machine to the jail but the jail itself is not reachable. > > There is nothing in the logs from the host and the jail itself, i can > ping the jails ip adres from the jail itself. > > > I do not think i have a special setup, but i could be doing something > wrong. > my jail.conf > > # Global settings applied to all jails. > $domain = "test.nl"; > $subdomain = ""; > > exec.start = "/bin/sh /etc/rc"; > exec.stop = "/bin/sh /etc/rc.shutdown"; > exec.clean; > > mount.fstab = "/storage/jails/$name.fstab"; > > exec.system_user  = "root"; > exec.jail_user    = "root"; > mount.devfs; > sysvshm="new"; > sysvsem="new"; > allow.raw_sockets; > allow.set_hostname = 0; > allow.sysvipc; > enforce_statfs = "2"; > devfs_ruleset     = "11"; > > path = "/storage/jails/${name}"; > host.hostname = "${name}${subdomain}.${domain}"; > > # Networking > $uplinkdev        = "vtnet1"; > $epid             = "${ip}"; > $subnet           = "10.233.185."; > $cidr             = "/24"; > $ipv4_addr        = "${subnet}${ip}${cidr}"; > vnet; > vnet.interface    = "vnet0"; > > $epair=epair${ip}; > vnet; > #vnet.interface    = "${epair}b";  # default vnet interface > exec.prestart     = "ifconfig bridge0 > /dev/null 2>&1 || ( ifconfig > bridge0 create up && ifconfig bridge0 addm $uplinkdev )"; > exec.prestart    += "ifconfig ${epair} create up description > jail_${name}   || echo 'Skipped creating epair (exists?)'"; > exec.prestart    += "ifconfig bridge0 addm ${epair}a           || echo > 'Skipped adding bridge member (already member?)'"; > exec.created      = "ifconfig ${epair}b name vnet0"; > exec.start        = "/bin/sh /etc/rc"; > exec.consolelog   = "/var/log/jail/$name.test.nl"; > exec.stop         = "/bin/sh /etc/rc.shutdown"; > exec.poststop     = "ifconfig bridge0 deletem ${epair}a"; > exec.poststop    += "ifconfig ${epair}a destroy"; > > varnish01 { >     $ip = 16; >     mount.fstab = ""; >     path = "/storage/jails/${name}"; > } > > web01 { >     $ip = 18; > } > > web02 { >     $ip = 19; > } > > haproxy { >     $ip = 20; >     mount.fstab = ""; >     path = "/storage/jails/${name}"; > } > > My ifconfig > > bridge0: flags=8843 metric 0 > mtu 1500 >     ether 58:9c:fc:10:ff:82 >     inet 10.233.185.1 netmask 0xffffff00 broadcast 10.233.185.255 >     id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >     maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >     root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >     member: epair20a flags=143 >             ifmaxaddr 0 port 13 priority 128 path cost 2000 >     member: epair19a flags=143 >             ifmaxaddr 0 port 53 priority 128 path cost 2000 >     member: epair18a flags=143 >             ifmaxaddr 0 port 48 priority 128 path cost 2000 >     member: epair16a flags=143 >             ifmaxaddr 0 port 28 priority 128 path cost 2000 >     groups: bridge >     nd6 options=9 > epair16a: flags=8963 > metric 0 mtu 1500 >     description: jail_varnish01 >     options=8 >     ether 02:76:32:8e:0e:0a >     groups: epair >     media: Ethernet 10Gbase-T (10Gbase-T ) >     status: active >     nd6 options=29 > epair18a: flags=8963 > metric 0 mtu 1500 >     description: jail_web01 >     options=8 >     ether 02:6d:be:b8:36:0a >     groups: epair >     media: Ethernet 10Gbase-T (10Gbase-T ) >     status: active >     nd6 options=29 > epair19a: flags=8963 > metric 0 mtu 1500 >     description: jail_web02 >     options=8 >     ether 02:54:fd:77:9a:0a >     groups: epair >     media: Ethernet 10Gbase-T (10Gbase-T ) >     status: active >     nd6 options=29 > epair20a: flags=8963 > metric 0 mtu 1500 >     description: jail_haproxy >     options=8 >     ether 02:f8:58:06:78:0a >     groups: epair >     media: Ethernet 10Gbase-T (10Gbase-T ) >     status: active >     nd6 options=29 > > This is on both 13-STABLE and 14-HEAD. > > For the sake of testing i tried it with FreeBSD 13.0-RELEASE-p7 and this works fine. This is an exact copy of the setup i use on 14-CURRENT and 13-STABLE. (i did a ZFS send and receive of the jails and a copy of the jail.conf. pf.conf and so on) I did run the hey command targeting the 13-0-RELEASE multiple times. hey -h2 -n 10 -c 30 -z 300s https://wp.test.nl Summary:   Total:    300.0045 secs   Slowest:    0.1137 secs   Fastest:    0.0006 secs   Average:    0.0090 secs   Requests/sec:    4627.4504 Response time histogram:   0.001 [1]    |   0.012 [977291]    |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■   0.023 [21236]    |■   0.035 [1125]    |   0.046 [230]    |   0.057 [12]    |   0.068 [18]    |   0.080 [9]    |   0.091 [18]    |   0.102 [30]    |   0.114 [30]    | Latency distribution:   10% in 0.0037 secs   25% in 0.0046 secs   50% in 0.0061 secs   75% in 0.0080 secs   90% in 0.0096 secs   95% in 0.0106 secs   99% in 0.0133 secs Details (average, fastest, slowest):   DNS+dialup:    0.0000 secs, 0.0006 secs, 0.1137 secs   DNS-lookup:    0.0000 secs, 0.0000 secs, 0.0028 secs   req write:    0.0001 secs, 0.0000 secs, 0.1126 secs   resp wait:    0.0192 secs, 0.0000 secs, 214.9645 secs   resp read:    0.0018 secs, 0.0002 secs, 0.1076 secs Status code distribution:   [200]    1000000 responses All is fine on the 13.0-RELEASE-p7 also with a higher concurrency, however if i do it against the 14-CURRENT or the 13-STABLE, even a run of 60 seconds kills the network connectivity of the jail. (haproxy in my case) regards, Johan