From nobody Fri Oct 21 18:45:37 2022 X-Original-To: freebsd-performance@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MvD1b2Nbnz4gT3c for ; Fri, 21 Oct 2022 18:45:39 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MvD1Z42Jzz3LhQ for ; Fri, 21 Oct 2022 18:45:38 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 29LIjaqH048702 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 21 Oct 2022 14:45:37 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:f808:cefc:42f3:2221] ([IPv6:2607:f3e0:0:4:f808:cefc:42f3:2221]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 29LIja8p028076 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 21 Oct 2022 14:45:36 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <39ca9375-e742-618e-5020-dda5fa24ac0a@sentex.net> Date: Fri, 21 Oct 2022 14:45:37 -0400 List-Id: Performance/tuning List-Archive: https://lists.freebsd.org/archives/freebsd-performance List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.3.3 Subject: Re: Chelsio Forwarding performance and RELENG_13 vs RELENG_12 Content-Language: en-US To: Navdeep Parhar , Freebsd performance References: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> <92cdf4b8-2209-ec44-8151-a59b9e8f1504@gmail.com> <8166abfe-a796-2cf0-ade2-de08df8eecd2@gmail.com> From: mike tancsa In-Reply-To: <8166abfe-a796-2cf0-ade2-de08df8eecd2@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4MvD1Z42Jzz3LhQ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-performance@freebsd.org]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[mike]; RCPT_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 10/21/2022 2:30 PM, Navdeep Parhar wrote: > On 10/21/22 11:15 AM, mike tancsa wrote: >> On 10/21/2022 2:13 PM, Navdeep Parhar wrote: >>> >>>>> hw.cxgbe.ntxq=4 >>>>> hw.cxgbe.nrxq=4 >>>>> >>>> Thanks Navdeep! >>>> >>>> Unfortunately, still the odd dropped packet :( >>> >>> Can you try increasing the size of the queues? >>> >>> hw.cxgbe.qsize_txq=2048 >>> hw.cxgbe.qsize_rxq=2048 >>> >>> The stats show that you are using MTU 1500.  If you were using MTU >>> 9000 I'd also have suggested setting largest_rx_cluster to 4K. >>> >>> hw.cxgbe.largest_rx_cluster=4096 >>> >> Thanks again, just 1500 MTU.  Shall I keep nt and nrxq at 4 still ? > > Yes, I think 4 queues are enough for 10G. > Sadly, no luck. Still about the same rate of overflows :( dev.cxl.0.stats.rx_trunc3: 0 dev.cxl.0.stats.rx_trunc2: 0 dev.cxl.0.stats.rx_trunc1: 0 dev.cxl.0.stats.rx_trunc0: 5 dev.cxl.0.stats.rx_ovflow3: 0 dev.cxl.0.stats.rx_ovflow2: 0 dev.cxl.0.stats.rx_ovflow1: 0 dev.cxl.0.stats.rx_ovflow0: 61 dev.cxl.0.stats.rx_ppp7: 0 dev.cxl.0.stats.rx_ppp6: 0 dev.cxl.0.stats.rx_ppp5: 0 dev.cxl.0.stats.rx_ppp4: 0 dev.cxl.0.stats.rx_ppp3: 0 dev.cxl.0.stats.rx_ppp2: 0 dev.cxl.0.stats.rx_ppp1: 0 dev.cxl.0.stats.rx_ppp0: 0 dev.cxl.0.stats.rx_pause: 0 dev.cxl.0.stats.rx_frames_1519_max: 0 dev.cxl.0.stats.rx_frames_1024_1518: 25966401 dev.cxl.0.stats.rx_frames_512_1023: 1569927 dev.cxl.0.stats.rx_frames_256_511: 1856460 dev.cxl.0.stats.rx_frames_128_255: 2718106 dev.cxl.0.stats.rx_frames_65_127: 4548626 dev.cxl.0.stats.rx_frames_64: 3044300 dev.cxl.0.stats.rx_runt: 0 dev.cxl.0.stats.rx_symbol_err: 0 dev.cxl.0.stats.rx_len_err: 0 dev.cxl.0.stats.rx_fcs_err: 0 dev.cxl.0.stats.rx_jabber: 0 dev.cxl.0.stats.rx_too_long: 0 dev.cxl.0.stats.rx_ucast_frames: 39618919 dev.cxl.0.stats.rx_mcast_frames: 36716 dev.cxl.0.stats.rx_bcast_frames: 48219 dev.cxl.0.stats.rx_frames: 39703866 dev.cxl.0.stats.rx_octets: 39678756369 dev.cxl.0.stats.tx_ppp7: 0