From nobody Tue Mar 15 17:48:34 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 35BA71A1C4AF for ; Tue, 15 Mar 2022 17:48:57 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_CHACHA20_POLY1305_SHA256 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KJ19g5Ctpz53Ds; Tue, 15 Mar 2022 17:48:55 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id 1f02269e; Tue, 15 Mar 2022 17:48:51 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 15db6250 (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Tue, 15 Mar 2022 17:48:47 +0000 (UTC) Date: Tue, 15 Mar 2022 18:48:34 +0100 From: Michael Gmelin To: Kristof Provost Cc: Michael Gmelin , "Bjoern A. Zeeb" , Johan Hendriks , "Patrick M. Hausen" , freeBSD-net Subject: Re: epair and vnet jail loose connection. Message-ID: <20220315184834.20d0def5.grembo@freebsd.org> In-Reply-To: <2131DA64-EB0F-4908-9B6C-50175311D941@FreeBSD.org> References: <797A280E-5DF2-4276-BB72-E4E1053A19FA@lists.zabbadoz.net> <6086BA6D-3D54-4851-B636-3B32FACB35E9@freebsd.org> <3B5E2D6F-5444-4448-B7C3-704E294368C3@lists.zabbadoz.net> <20220314144451.35f803a9.grembo@freebsd.org> <20220315010230.6083dd72.grembo@freebsd.org> <2131DA64-EB0F-4908-9B6C-50175311D941@FreeBSD.org> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4KJ19g5Ctpz53Ds X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=softfail (mx1.freebsd.org: 213.239.217.29 is neither permitted nor denied by domain of grembo@freebsd.org) smtp.mailfrom=grembo@freebsd.org X-Spamd-Result: default: False [0.94 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEFALL_USER(0.00)[grembo]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_MEDIUM(-0.52)[-0.519]; RCVD_TLS_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCPT_COUNT_FIVE(0.00)[6]; NEURAL_HAM_LONG(-0.99)[-0.988]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MID_CONTAINS_FROM(1.00)[]; NEURAL_SPAM_SHORT(0.04)[0.044]; MLMMJ_DEST(0.00)[freebsd-net]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; FREEMAIL_CC(0.00)[freebsd.org,lists.zabbadoz.net,gmail.com,punkt.de]; SUSPICIOUS_RECIPS(1.50)[] X-ThisMailContainsUnwantedMimeParts: N On Tue, 15 Mar 2022 10:30:41 -0600 Kristof Provost wrote: > On 14 Mar 2022, at 18:02, Michael Gmelin wrote: > > On Mon, 14 Mar 2022 09:09:49 -0600 > > Kristof Provost wrote: > > =20 > >> On 14 Mar 2022, at 7:44, Michael Gmelin wrote: =20 > >>> On Sun, 13 Mar 2022 17:53:44 +0000 > >>> "Bjoern A. Zeeb" wrote: > >>> =20 > >>>> On 13 Mar 2022, at 17:45, Michael Gmelin wrote: > >>>> =20 > >>>>>> On 13. Mar 2022, at 18:16, Bjoern A. Zeeb > >>>>>> wrote: > >>>>>> > >>>>>> =EF=BB=BFOn 13 Mar 2022, at 16:33, Michael Gmelin wrote: =20 > >>>>>>> It's important to point out that this only happens with =20 > >>>>>>> kern.ncpu>1. With kern.ncpu=3D=3D1 nothing gets stuck. =20 > >>>>>>> > >>>>>>> This perfectly fits into the picture, since, as pointed out by > >>>>>>> Johan, > >>>>>>> the first commit that is affected[0] is about multicore > >>>>>>> support. =20 > >>>>>> > >>>>>> Ignore my ignorance, what is the default of net.isr.maxthreads > >>>>>> and net.isr.bindthreads (in stable/13) these days? > >>>>>> =20 > >>>>> > >>>>> My tests were on CURRENT and I=E2=80=99m afk, but according to > >>>>> cgit[0][1], max is 1 and bind is 0. > >>>>> > >>>>> Would it make sense to repeat the test with max=3D-1? =20 > >>>> > >>>> I=E2=80=99d say yes, I=E2=80=99d also bind, but that=E2=80=99s just = me. > >>>> > >>>> I would almost assume Kristof running with -1 by default (but he > >>>> can chime in on that). =20 > >>> > >>> I tried various configuration permutations, all with ncpu=3D2: > >>> > >>> - 14.0-CURRENT #0 main-n253697-f1d450ddee6 > >>> - 13.1-BETA1 #0 releng/13.1-n249974-ad329796bdb > >>> - net.isr.maxthreads: -1 (which results in 2 threads), 1, 2 > >>> - net.isr.bindthreads: -1, 0, 1, 2 > >>> - net.isr.dispatch: direct, deferred > >>> > >>> All resulting in the same behavior (hang after a few seconds). > >>> They all > >>> work ok when running on a single core instance (threads=3D1 in this > >>> case). > >>> > >>> I also ran the same test on 13.0-RELEASE-p7 for > >>> comparison (unsurprisingly, it's ok). > >>> > >>> I placed the script to reproduce the issue on freefall for your > >>> convenience, so running it is as simple as: > >>> > >>> fetch https://people.freebsd.org/~grembo/hang_epair.sh > >>> # inspect content > >>> sh hang_epair.sh > >>> > >>> or, if you feel lucky > >>> > >>> fetch -o - https://people.freebsd.org/~grembo/hang_epair.sh | > >>> sh=20 > >> With that script I can also reproduce the problem. > >> > >> I=E2=80=99ve experimented with this hack: > >> > >> diff --git a/sys/net/if_epair.c b/sys/net/if_epair.c > >> index c39434b31b9f..1e6bb07ccc4e 100644 > >> --- a/sys/net/if_epair.c > >> +++ b/sys/net/if_epair.c > >> @@ -415,7 +415,10 @@ epair_ioctl(struct ifnet *ifp, u_long > >> cmd, caddr_t data) > >> > >> case SIOCSIFMEDIA: > >> case SIOCGIFMEDIA: > >> + printf("KP: %s() SIOCGIFMEDIA\n", > >> __func__); sc =3D ifp->if_softc; > >> + taskqueue_enqueue(epair_tasks.tq[0], > >> &sc->queues[0].tx_task); > >> + > >> error =3D ifmedia_ioctl(ifp, ifr, &sc->media, > >> cmd); break; > >> > >> That kicks the receive code whenever I `ifconfig epair0a`, and I > >> see a little more traffic every time I do so. > >> That suggests pretty strongly that there=E2=80=99s an issue with how we > >> dispatch work to the handler thread. So presumably there=E2=80=99s a r= ace > >> between epair_menq() and epair_tx_start_deferred(). > >> > >> epair_menq() tries to only enqueue the receive work if there=E2=80=99s > >> nothing in the buf_ring, on the grounds that if there is the > >> previous packet scheduled the work. Clearly there=E2=80=99s an issue t= here. > >> > >> I=E2=80=99ll try to dig into that in the next few days. > >> =20 > > > > Hi Kristof, > > > > This sounds plausible. I spent a few hours getting familiar with the > > epair code and came up with a patch that seems to fix the issue at > > hand (both with and without RSS). I'm not certain that it is a good > > solution, especially in terms of performance, but I wanted to share > > it with you anyway, maybe it helps: > > https://people.freebsd.org/~grembo/epair.patch > > =20 > That seems to be working, and at first glance doesn=E2=80=99t look like i= t=E2=80=99d > hurt performance too badly. >=20 > Can you write up a commit message and post it on phabricator? >=20 Please see https://reviews.freebsd.org/D34569 Best Michael --=20 Michael Gmelin