From nobody Mon Mar 14 15:09:49 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B9FC01A0EF2E for ; Mon, 14 Mar 2022 15:09:55 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KHKhg4rXFz3pnr; Mon, 14 Mar 2022 15:09:55 +0000 (UTC) (envelope-from kp@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647270595; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RfJLrB5Cnq/MLrQJe7IEdhfdYWA1psqu1lQaIrZ9CYQ=; b=SGE5lMtEKtBH7abtdQHgM8YRHzI+A2t14dhUmWru+siwBFGKOoMdydDtN75ZLilJn7wMKo n1Or759SfAQykz8A8ZDCuedeNQD82U1Jw/lkiTae6mbMftFBc7I8HVoIUXAKPGgVS5IhMe atXfskonLGs1dlBQLKfUpOKZUDM/tE+dJet3mf/UGxz18Z1OOy6Q/hy+aHg6K+WEqGC1kS br/ED6BFaijqEujJRkGhC46SrTKHb7kWXrdZVXjyl0poAUWVtUIgz+6LinLr6eZOnYFfpg 8pXJNJMrsTjjGqT94O6kaWkq0Dw1TGPoMzGQJR3SpRmzGuxcEsRVYtxeLg1x8w== Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "R3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 58B752765; Mon, 14 Mar 2022 15:09:55 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 9422E1C923; Mon, 14 Mar 2022 16:09:52 +0100 (CET) From: Kristof Provost To: Michael Gmelin Cc: "Bjoern A. Zeeb" , Johan Hendriks , "Patrick M. Hausen" , freeBSD-net Subject: Re: epair and vnet jail loose connection. Date: Mon, 14 Mar 2022 09:09:49 -0600 X-Mailer: MailMate (1.14r5852) Message-ID: In-Reply-To: <20220314144451.35f803a9.grembo@freebsd.org> References: <797A280E-5DF2-4276-BB72-E4E1053A19FA@lists.zabbadoz.net> <6086BA6D-3D54-4851-B636-3B32FACB35E9@freebsd.org> <3B5E2D6F-5444-4448-B7C3-704E294368C3@lists.zabbadoz.net> <20220314144451.35f803a9.grembo@freebsd.org> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_MailMate_3AB37302-0AB7-4470-92AA-E00D57EB46B0_=" Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1647270595; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RfJLrB5Cnq/MLrQJe7IEdhfdYWA1psqu1lQaIrZ9CYQ=; b=SZyUMbl2P7+gsjBZJGpKSWoTHOVA0b9i7OTAy2YT2cnJGclKZ17nm6a/EQIAi6wBi8D3sS i6gHcNWvzJL8OTrhNsxqq/xYygo+kXNQWmFG61snn0SIsQ04M7T1umPKXM9JEyn/Zb2J+u NJ6kfotqIfYR9KDsZcCR6Fto97YlT5RdTpvUbdQt+LWMI5FQi0OKLa/bzNlqm0I0rEoyP4 js/G1oidwDbfVdPaJZkpF2A56l9gQrGGYB3naXx/5uU6U1OfxbtQ/txPQN35xBTxwVA2fa FJjnVswFIvBnIDFy18+BeS238IqOZYBQdUbI37LrPS3X2pPdKsMsCV9aaZd/iQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1647270595; a=rsa-sha256; cv=none; b=rWiWMewUR47kC4Lw8kQ38HBaaD7hcGsbIoOc+6UDl4G4vcqrCDTJJ5ET9vusZmKk1jeNFA Yb17Jd0LwKoqdALx1Vki0/TTGL/6DYd1OsmhEvpLRSRCFAZStXOQ8OfZXPBSnJ343fzNeW QMhP/rRT8MDrgowtGnSYxsMBpyohwxPHLnOQ5zqGuMV6ZdCtUGOhsZ9JJRq3WrIdmKacWP 2AeAm8xWo9L3jg9eOAaj/utgAw2l3ToEJL8NHcvRC4V4XISvnZNYDZgvxxOojB8wD6l8sp mdhzkbR2p9do83uNOFeB7Ib9N+vpX0N9yd+RetaVhRuXzePEC5bT/MlCjGZyMg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N --=_MailMate_3AB37302-0AB7-4470-92AA-E00D57EB46B0_= Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 14 Mar 2022, at 7:44, Michael Gmelin wrote: > On Sun, 13 Mar 2022 17:53:44 +0000 > "Bjoern A. Zeeb" wrote: > >> On 13 Mar 2022, at 17:45, Michael Gmelin wrote: >> >>>> On 13. Mar 2022, at 18:16, Bjoern A. Zeeb >>>> wrote: >>>> >>>> On 13 Mar 2022, at 16:33, Michael Gmelin wrote: >>>>> It's important to point out that this only happens with >>>>> kern.ncpu>1. With kern.ncpu==1 nothing gets stuck. >>>>> >>>>> This perfectly fits into the picture, since, as pointed out by >>>>> Johan, >>>>> the first commit that is affected[0] is about multicore support. >>>> >>>> Ignore my ignorance, what is the default of net.isr.maxthreads and >>>> net.isr.bindthreads (in stable/13) these days? >>>> >>> >>> My tests were on CURRENT and I’m afk, but according to cgit[0][1], >>> max is 1 and bind is 0. >>> >>> Would it make sense to repeat the test with max=-1? >> >> I’d say yes, I’d also bind, but that’s just me. >> >> I would almost assume Kristof running with -1 by default (but he can >> chime in on that). > > I tried various configuration permutations, all with ncpu=2: > > - 14.0-CURRENT #0 main-n253697-f1d450ddee6 > - 13.1-BETA1 #0 releng/13.1-n249974-ad329796bdb > - net.isr.maxthreads: -1 (which results in 2 threads), 1, 2 > - net.isr.bindthreads: -1, 0, 1, 2 > - net.isr.dispatch: direct, deferred > > All resulting in the same behavior (hang after a few seconds). They > all > work ok when running on a single core instance (threads=1 in this > case). > > I also ran the same test on 13.0-RELEASE-p7 for > comparison (unsurprisingly, it's ok). > > I placed the script to reproduce the issue on freefall for your > convenience, so running it is as simple as: > > fetch https://people.freebsd.org/~grembo/hang_epair.sh > # inspect content > sh hang_epair.sh > > or, if you feel lucky > > fetch -o - https://people.freebsd.org/~grembo/hang_epair.sh | sh > With that script I can also reproduce the problem. I’ve experimented with this hack: diff --git a/sys/net/if_epair.c b/sys/net/if_epair.c index c39434b31b9f..1e6bb07ccc4e 100644 --- a/sys/net/if_epair.c +++ b/sys/net/if_epair.c @@ -415,7 +415,10 @@ epair_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data) case SIOCSIFMEDIA: case SIOCGIFMEDIA: + printf("KP: %s() SIOCGIFMEDIA\n", __func__); sc = ifp->if_softc; + taskqueue_enqueue(epair_tasks.tq[0], &sc->queues[0].tx_task); + error = ifmedia_ioctl(ifp, ifr, &sc->media, cmd); break; That kicks the receive code whenever I `ifconfig epair0a`, and I see a little more traffic every time I do so. That suggests pretty strongly that there’s an issue with how we dispatch work to the handler thread. So presumably there’s a race between epair_menq() and epair_tx_start_deferred(). epair_menq() tries to only enqueue the receive work if there’s nothing in the buf_ring, on the grounds that if there is the previous packet scheduled the work. Clearly there’s an issue there. I’ll try to dig into that in the next few days. Kristof --=_MailMate_3AB37302-0AB7-4470-92AA-E00D57EB46B0_= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On 14 Mar 2022, at 7:44, Michael Gmelin wrote:

On Sun, 13 Mar 2022 17:53:44 +0000
"Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> wrote:

On 13 Mar 2022, at 17:45, Michael Gmelin wrote:

On 13. M= ar 2022, at 18:16, Bjoern A. Zeeb
<bzeeb-lists@lists.zabbadoz.net> wrote:

=EF=BB=BFOn 13 Mar 2022, at 16:33, Michael Gmelin wrote:<= /p>

It's important to point out that this only happens with
kern.ncpu>1. With kern.ncpu=3D=3D1 nothing gets stuck.

This perfectly fits into the picture, since, as pointed o= ut by
Johan,
the first commit that is affected[0] is about multicore support.

Ignore my ignorance, what is the default of = net.isr.maxthreads and
net.isr.bindthreads (in stable/13) these days?

My tests were on CURRENT and I=E2=80=99m afk= , but according to cgit[0][1],
max is 1 and bind is 0.

Would it make sense to repeat the test with max=3D-1?

=

I=E2=80=99d say yes, I=E2=80=99d also bind, = but that=E2=80=99s just me.

I would almost assume Kristof running with -1 by default = (but he can
chime in on that).

I tried various configuration permutations, = all with ncpu=3D2:

- 14.0-CURRENT #0 main-n253697-f1d450ddee6
- 13.1-BETA1 #0 releng/13.1-n249974-ad329796bdb
- net.isr.maxthreads: -1 (which results in 2 threads), 1, 2
- net.isr.bindthreads: -1, 0, 1, 2
- net.isr.dispatch: direct, deferred

All resulting in the same behavior (hang after a few seco= nds). They all
work ok when running on a single core instance (threads=3D1 in this case)= =2E

I also ran the same test on 13.0-RELEASE-p7 for
comparison (unsurprisingly, it's ok).

I placed the script to reproduce the issue on freefall fo= r your
convenience, so running it is as simple as:

fetch https://people.freebsd.org/~grembo/hang_epair.sh
# inspect content
sh hang_epair.sh

or, if you feel lucky

fetch -o - https://people.freebsd.org/~grembo/hang_epair.sh |= sh


With that script I can also reproduce the problem.

I=E2=80=99ve experimented with this hack:

diff --git a/sys/net/if_epair.c b/sys/net/if_epair.c
index c39434b31b9f..1e6bb07ccc4e 100644
--- a/sys/net/if_epair.c
+++ b/sys/net/if_epair.c
@@ -415,7 +415,10 @@ epair_ioctl(struct ifnet *ifp, u_long cmd, caddr_t d=
ata)

        case SIOCSIFMEDIA:
        case SIOCGIFMEDIA:
+               printf("KP: %s() SIOCGIFMEDIA\n", __func__);
                sc =3D ifp->if_softc;
+               taskqueue_enqueue(epair_tasks.tq[0], &sc->queues[0=
].tx_task);
+
                error =3D ifmedia_ioctl(ifp, ifr, &sc->media, cmd)=
;
                break;

That kicks the receive code whenever I ifconfig epa= ir0a, and I see a little more traffic every time I do so.
That suggests pretty strongly that there=E2=80=99s an issue with how we d= ispatch work to the handler thread. So presumably there=E2=80=99s a race = between epair_menq() and epair_tx_start_deferred().

epair_menq() tries to only enqueue the receive work if th= ere=E2=80=99s nothing in the buf_ring, on the grounds that if there is th= e previous packet scheduled the work. Clearly there=E2=80=99s an issue th= ere.

I=E2=80=99ll try to dig into that in the next few days.

Kristof

--=_MailMate_3AB37302-0AB7-4470-92AA-E00D57EB46B0_=--