From nobody Thu Sep 12 18:50:20 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4RMh1nprz5WW6s for ; Thu, 12 Sep 2024 18:50:24 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4RMg2plDz4j5N for ; Thu, 12 Sep 2024 18:50:23 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-42cafda818aso12236945e9.2 for ; Thu, 12 Sep 2024 11:50:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726167022; x=1726771822; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=FuwxJ8qEQQDa/4S2t8feeNNxf7SWqs2c8D4i7VlN5bk=; b=EA8PfWCY9igcuZoHut3/Et0IRgGkbi/RGjGAE6nHP6SDUAqEBCJ/rv67exBwV8xLrt YoZFDKEqhsKAg6H+7GvAEP26PuUNmCTL4SyucJhpHTUzHZUV+/9Qvw+SPyQRKiigpwLS 6YsulEvyUfex64a0QnEtZFUIwL61rWggYWFZlprg6tY99Is5tvLjTZNiLXc09+UqjxYx df7YSzP5GeaojHAUkFfs3HXxHDxSeJu/xWNsenmjQGUq5OnDfalO6SUOreOv7agR+S8s skHJ3v9xLIaFX3tmOmRSYYHulj/RMQH5eWGD2t9Ylb0ednOealjQM+xI/aDedUbwQAiD +Ucw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726167022; x=1726771822; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FuwxJ8qEQQDa/4S2t8feeNNxf7SWqs2c8D4i7VlN5bk=; b=qTaZtrQx6GWhoBeR8feNyO7160uV1f0nb1M7sG9KfJSoE+DdDw1Cj5rDG3V7SZPFuV M1bNCPAOUTj+c4Yqev9YPEYTaLZlG2CSwLvDYvvAicXEVy4XXO1xWWAF0Tl/r40XuEwH e+3DcYfpLGhUaXtYFcg+AhOkqTLjtloGysklPsqkVo9Sv9P5Zzcmw93R8OlTbDtx3HA8 TrcFatwC4SryYQpRvHUtw/MeD6mvCtLWEiVa0OmUDhyDujG5TpjcFUOFcUCHj4PEtRb3 QVFmkYV5b2qWd9p/m0DEZkyFuUynS3JDZ3ZMhmvIYbzd7oNoPCucylvPz/ZR+60nI2XN qenA== X-Gm-Message-State: AOJu0YyqT1kLLkmcvu/+ebIBDfsdZ0PHboULflDy2H1IG8FLsotfxSEA J0lTcfDdDJpIu+vwyJsP489Xe0qV38qwhalK2Chd2GdkSNJ94mJk X-Google-Smtp-Source: AGHT+IF/K3tPFVFJvOMOuWm+C7vvYSoCQJZrAF2kOQ6om3VhAHx3SkcOpw68+0rBER21xhtGikv4AQ== X-Received: by 2002:a05:600c:4f03:b0:42c:b97a:5f7d with SMTP id 5b1f17b1804b1-42cdb509fc7mr34791985e9.7.1726167021265; Thu, 12 Sep 2024 11:50:21 -0700 (PDT) Received: from z600.home.lan (118.129.159.143.dyn.plus.net. [143.159.129.118]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42d9b16c9d7sm644245e9.30.2024.09.12.11.50.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Sep 2024 11:50:20 -0700 (PDT) Date: Thu, 12 Sep 2024 19:50:20 +0100 From: Sad Clouds To: Paul Procacci Cc: freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240912195020.e1bac64a72cc68dee2302238@gmail.com> In-Reply-To: References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4X4RMg2plDz4j5N On Thu, 12 Sep 2024 13:25:32 -0400 Paul Procacci wrote: > You need to define `poor'. > You need to show `top -SH` while the `problem' occurs. > > My guess is packets are getting shuttled between a global taskqueue thread. > This is the default, or at least I'm not aware of this default being > changed. > You can try enabling `options RSS' in your kernel as this would introduce a > taskqueue worker thread per cpu. Hello, a bit more info. I have a tcp benchmark which performs simultaneous send and receive between sever and client processes. This is running on a 4-core Raspberry Pi 4 and using a total of 8 threads: 4 threads for server process and 4 threads for client process. Each thread opens a single TCP connection. # ifconfig genet0 genet0: flags=1008943 metric 0 mtu 1500 options=280009 ether dc:a6:32:31:71:32 inet 192.168.1.18 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (1000baseT ) status: active nd6 options=29 Running TCP test via localhost keeps all CPUs busy and gives combined throughput of around 900 MiB/sec. Running TCP test between the host and jail gives combined throughput of around 128 MiB/sec. Disabling RXCSUM and re-running the test gives the same 128 MiB/sec. Running "top -SH" shows this: last pid: 4542; load averages: 0.68, 0.42, 0.27 up 0+09:39:19 19:19:03 154 threads: 7 running, 126 sleeping, 21 waiting CPU 0: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle CPU 1: 0.0% user, 0.0% nice, 15.6% system, 0.0% interrupt, 84.4% idle CPU 2: 0.8% user, 0.0% nice, 42.2% system, 0.0% interrupt, 57.0% idle CPU 3: 0.8% user, 0.0% nice, 23.4% system, 0.0% interrupt, 75.8% idle Mem: 12M Active, 46M Inact, 274M Wired, 162M Buf, 3500M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -64 - 0B 608K CPU0 0 4:11 100.00% kernel{epair_task} 11 root 187 ki31 0B 64K RUN 1 575:35 81.58% idle{idle: cpu1} 11 root 187 ki31 0B 64K CPU3 3 574:58 74.14% idle{idle: cpu3} 11 root 187 ki31 0B 64K RUN 2 576:04 60.94% idle{idle: cpu2} 4542 root 4 0 29M 3408K select 2 0:01 11.95% tcp_bench{tcp_bench} 4542 root 28 0 29M 3408K select 2 0:00 11.79% tcp_bench{tcp_bench} 4542 root 32 0 29M 3408K select 3 0:01 10.85% tcp_bench{tcp_bench} 4538 root 26 0 29M 3624K select 3 0:01 10.62% tcp_bench{tcp_bench} 4538 root 21 0 29M 3624K CPU1 1 0:00 10.07% tcp_bench{tcp_bench} 4538 root 26 0 29M 3624K select 1 0:01 10.05% tcp_bench{tcp_bench} 4538 root 24 0 29M 3624K select 1 0:01 8.91% tcp_bench{tcp_bench} 4542 root 31 0 29M 3408K select 1 0:01 8.78% tcp_bench{tcp_bench} 4537 root 20 0 14M 3748K CPU2 2 0:00 0.24% top I will try adding "options RSS" to GENERIC config file and then build new kernel.