From nobody Thu Sep 12 17:25:32 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4PV14Lw1z5WKKY for ; Thu, 12 Sep 2024 17:25:45 +0000 (UTC) (envelope-from pprocacci@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4PV12k4Rz4VYF for ; Thu, 12 Sep 2024 17:25:45 +0000 (UTC) (envelope-from pprocacci@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a7a843bef98so155338966b.2 for ; Thu, 12 Sep 2024 10:25:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726161944; x=1726766744; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=JFG+z53YCIxCIvkf/zwz+x9EQipLaVqBcDVBO1vi2rM=; b=ZL+qyTTMKaIf0/gVX4eaqAWI6X+x9hBLdeFXn3ejkbtliIuSK7EZHieAaCQUcPTP7B /ldDhDYKvqZKw4XwdG2r9scutYZYorhncLEN8Ff+H/la7TQjvNplqwg7DBmQFuVQniaL qpBe1UoYuFBntXgDPpTNxoo5kaF5UjFeOiYGR1mIIamOvM+DRaD1uieqVA/4DNPtF/ma bpCoPJyJEwmytTqNVPW/oopkfZVvGBxQ1qhQSdMaYFwhkxXj4/ZxLW8x7uiT3ubnVcsx QNIPa1mlZ69R3vblJlxI3m0tDKBQaHMq6i/zATrUHOD5GsBsgtG4dOlUuX8f/q3T8DZA J8yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726161944; x=1726766744; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JFG+z53YCIxCIvkf/zwz+x9EQipLaVqBcDVBO1vi2rM=; b=jq+Vvd73/V3rX73vfduVFes7QajWrpMO9kIZzDsNAi9AfvTQrfOozUv8OObVo9Vx8B UYtkb3a4qvZTSHXwjpPRstNU7lxuDHbHJTM+gKlTe5tsUh25dGfF6WkMAKqdYP79QGor 9RfUs7D76F4tttfj7fbit1/7VZ+o6mzwu84RKCUKrZ/jzGMzj9h2ygWgVfD088bS/QhW bdvO7AX2mJRq5OM6DJc9OJ71CSFOHe4d06Xf9b1GedI3LdWpwRMP6qkSrHt+DPAAFIoQ r6RbirlLPUgeA0DXY/4bvTp/XnPBIHO0P5Qp0a3JwPVJfZL+gZNr//E5unfmGYYNI/FS 1NDw== X-Gm-Message-State: AOJu0Yy673/EbuN+DH4tOsTA2Fyu6in+B3GKAQaV1tWkrT+7fVW25tD1 hrRzGtKdXR5HdfQOVJiq+peTWvzdxJBuuaI89AkhRukEO13xw7Z+zg0KiMjruz3qpgrVTNWG2rh Z9v2CE3Aqj5f+/gaKUqxlmGNUTw== X-Google-Smtp-Source: AGHT+IFMis4d78birh5oc8xNAGocDSKqOAu2o3wC+sibPggjsoMDbblW+X6Xj3N4/+81kiqVg47KgSFXabgG2ylf22E= X-Received: by 2002:a17:906:c14a:b0:a86:7514:e649 with SMTP id a640c23a62f3a-a902960cff6mr358745966b.52.1726161943109; Thu, 12 Sep 2024 10:25:43 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> In-Reply-To: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> From: Paul Procacci Date: Thu, 12 Sep 2024 13:25:32 -0400 Message-ID: Subject: Re: Performance issues with vnet jails + epair + bridge To: Sad Clouds Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="00000000000071abcf0621ef652e" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X4PV12k4Rz4VYF --00000000000071abcf0621ef652e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Sep 12, 2024 at 1:16=E2=80=AFPM Sad Clouds wrote: > Hi, I'm using FreeBSD-14.1 and on this particular system I only have a > single physical network interface, so I followed instructions for > networking vnet jails via epair and bridge, e.g. > > devel > { > vnet; > vnet.interface =3D "e0b_devel"; > exec.prestart +=3D "/jails/jib addm devel genet0"; > exec.poststop +=3D "/jails/jib destroy devel"; > } > > The issue is bulk TCP performance throughput between this jail and the > host is quite poor, with one CPU spinning 100% in kernel and others > sitting mostly idle. > > It seems there is some lock contention somewhere, but I'm not sure if > this is around vnet, epair or bridge subsystems. Are there > other alternatives for vnet jails? Can anyone recommend specific > deployment scenarios? I've seen references to netgraph which could be > used with jails. Does it have better performance and scalability and > could replace epair and bridge combination? > > Thanks. > > You need to define `poor'. You need to show `top -SH` while the `problem' occurs. My guess is packets are getting shuttled between a global taskqueue thread. This is the default, or at least I'm not aware of this default being changed. You can try enabling `options RSS' in your kernel as this would introduce a taskqueue worker thread per cpu. ~Paul --=20 __________________ :(){ :|:& };: --00000000000071abcf0621ef652e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Sep 12, 2024 at 1:1= 6=E2=80=AFPM Sad Clouds <= cryintothebluesky@gmail.com> wrote:
Hi, I'm using FreeBSD-14.1 and on this parti= cular system I only have a
single physical network interface, so I followed instructions for
networking vnet jails via epair and bridge, e.g.

devel
{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vnet;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 vnet.interface =3D "e0b_devel";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 exec.prestart +=3D "/jails/jib addm devel = genet0";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 exec.poststop +=3D "/jails/jib destroy dev= el";
}

The issue is bulk TCP performance throughput between this jail and the
host is quite poor, with one CPU spinning 100% in kernel and others
sitting mostly idle.

It seems there is some lock contention somewhere, but I'm not sure if this is around vnet, epair or bridge subsystems. Are there
other alternatives for vnet jails? Can anyone recommend specific
deployment scenarios? I've seen references to netgraph which could be used with jails. Does it have better performance and scalability and
could replace epair and bridge combination?

Thanks.



You need to define `poor'.
= You need to show `top -SH` while the `problem' occurs.

My guess is packets are getting shuttled between a global tas= kqueue thread.
This is the default, or at least I'm not aware = of this default being changed.
You can try enabling `options R= SS' in your kernel as this would introduce a taskqueue worker thread pe= r cpu.

~Paul

--
__________________

:(){ :|:& };:
<= /div>
--00000000000071abcf0621ef652e--