From nobody Mon Feb 26 00:56:37 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tjhys16Rhz5Bjm2 for ; Mon, 26 Feb 2024 00:56:53 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-ua1-x935.google.com (mail-ua1-x935.google.com [IPv6:2607:f8b0:4864:20::935]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tjhyq4FWrz4svq for ; Mon, 26 Feb 2024 00:56:51 +0000 (UTC) (envelope-from nonesuch@longcount.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=longcount.org header.s=google header.b=mAvuhsxM; dmarc=none; spf=pass (mx1.freebsd.org: domain of nonesuch@longcount.org designates 2607:f8b0:4864:20::935 as permitted sender) smtp.mailfrom=nonesuch@longcount.org Received: by mail-ua1-x935.google.com with SMTP id a1e0cc1a2514c-7ce3c7566e0so888755241.1 for ; Sun, 25 Feb 2024 16:56:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; t=1708909010; x=1709513810; darn=freebsd.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=GPWHLO6w8MEevlPfFaIdseAbW5WNpmPMvboroD2ChOA=; b=mAvuhsxMQBKHFyCL1IPUdwFKCsqTNlYSfAwAjMEoGyet1+KSAQIYSkGwbhBMTAI17Q ti/K01GRGsDC73L52wjzLlEFDE+DK/uFyFvVDF2RfIiCcmGTMes8L9ZceHl0TpeqzhOU UQbdxQZrMozblplR8FJvgPxVcUB5Gxhwn0/xToWV10SwdG2h5KntP1PQ7uJyeb6xUsnm B2baDMz10LjYvKH94dPa5u+qTswluc8ZyCgWhSPLLBsL0s2kG00Lr/m+DCS4jNq71SNi LHk814zCyVC+VSX4akJBnutIm67q1gDC9YiDWzvoy0ujdPRxQyQSr4TCwq29oUtNTzrJ lwRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708909010; x=1709513810; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GPWHLO6w8MEevlPfFaIdseAbW5WNpmPMvboroD2ChOA=; b=ZF8dCBez4fAaJbxYDhz/1aNl1ZX6JdWIF34pH1f3hcjbaPzQ0s0nMikTL5H/PQ47V6 Bn4OGkzuiDHdNNXkKoc9nfN8tAQf7UFg5jBhEPtyONeyL4x6TXzmXM3WaV8aJxWDjZVG 58kxWgmbAG9uP6JmLICLJ8rvvP+7RZHNLcL4uhDTcX1bN5+tZ7bO72IW6DpWATd2ljMi szCyfDr2iY414aobpUH0VnkOAs46gPPQooV/1sIvQIDJRW9qBiAVpuByYW3MU2/LL5aD Mv29DAGWjN/la4gPRQ7yIvlT0jakOhjIDLVU4uMWWXai83eE8JXJ63XhRIjTHHRhSU9V 2QRA== X-Gm-Message-State: AOJu0YxTy9t8JtQjluyCZwoBBk3OEbtL/lT2qf6m+p8iRLALNCsO2PUd LOEx9PiA29pEeyD+ta8Fi9EyloxYPkklO6k0fvSDB5pickCDALhMYRZkH9gaU/Oti2UScaNVIUf +DTuprp+WCJyNUbTJtiZCaKIPWWhmJH53KjLRqIoq5hr/1vW7F1w= X-Google-Smtp-Source: AGHT+IG8UcMbXRRxFF7lPEWxXVESys9via6twcEKAupqZqXt5yb2tk5hBTBX5284MydKl2Kfa4f2z89TcnbXPDkWuVs= X-Received: by 2002:a1f:e481:0:b0:4c8:16fc:611 with SMTP id b123-20020a1fe481000000b004c816fc0611mr1843951vkh.4.1708909009741; Sun, 25 Feb 2024 16:56:49 -0800 (PST) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <034cc6ea-26d8-4520-879a-672459832407@fsfe.org> <9066A50F-26DC-4314-B79E-66120A2B5A2F@freebsd.org> In-Reply-To: From: Mark Saad Date: Sun, 25 Feb 2024 19:56:37 -0500 Message-ID: Subject: Re: NFS performance with 10GBase-T To: "freebsd-net@freebsd.org" Content-Type: multipart/alternative; boundary="0000000000007ac81606123e6250" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[longcount.org:s=google]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[longcount.org]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; TO_DN_EQ_ADDR_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::935:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DKIM_TRACE(0.00)[longcount.org:+] X-Rspamd-Queue-Id: 4Tjhyq4FWrz4svq --0000000000007ac81606123e6250 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable H On Sun, Feb 25, 2024 at 6:51=E2=80=AFPM Rick Macklem wrote: > On Sun, Feb 25, 2024 at 1:21=E2=80=AFAM wrote: > > > > CAUTION: This email originated from outside of the University of Guelph= . > Do not click links or open attachments unless you recognize the sender an= d > know the content is safe. If in doubt, forward suspicious emails to > IThelp@uoguelph.ca. > > > > > > > On Feb 25, 2024, at 01:18, Hannes Hauswedell > wrote: > > > > > > Hi everyone, > > > > > > I am coming here from > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D2771971160 > > I guess this should read: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277197 > Btw, what Hannes reported in the PR was that performance was > about the same for Linux and FreeBSD NFS clients when the link > was using a 1500byte ethernet frames. > However, Linux performs much better with 9K jumbo frames > whereas FreeBSD performance does not improve for 9K jumbo > frames. (Some mount options I suggested did help somewhat > for FreeBSD. Basically increasing rsize/wsize did help, but he > still sees performance below what Linux gets when 9K jumbo frames > are used. (I did note the potential problem of mbuf cluster pool > fragmentation when 9K jumbo frames are used, although I did not > intent to imply that this issue is involved, just that it is a known > deficiency that "might" be a factor.) > > rick > > > > Best regards > > Michael > > > > > > TL;DR: > > > > > > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) adapto= r > each. > > > * I am trying to improve the NFS throughput. > > > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS. > > > * Increasing rsize and wsize to 1 MiB increases throughput to 395 MiB= /s > > > * But a Linux client achieves 560-600 MiB/s with any rsize. > > > * The mtu is set to 9000 but this doesn't make a difference for the > FreeBSD client (it does make a difference for Linux). > > > > > > I assume < 400 MiB/s is not the expected performance? Do you have any > advice on debugging this? > > > > > > Thank you for your help, > > > Hannes > > > > > > > > > > > Hannes In the dmesg posted I see that you have a epair loaded . Are you trying to do NFS inside of a Jail ? Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames a long time ago ? I know in isilon land their FreeBSD can do this, but I can't say I have any idea how it's being done. They do have some kernel auto-tune magic as well to find optimal send and receive buffers. Maybe what we are seeing is Linux having better ergonomics on buffers out of the box ? Hannes Can you post your boot.conf and sysctl.conf settings. --=20 mark saad | nonesuch@longcount.org --0000000000007ac81606123e6250 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
H

On Sun, Feb 25, 2024 at 6:51=E2=80=AFPM Rick Mac= klem <rick.macklem@gmail.com> wrote:
On= Sun, Feb 25, 2024 at 1:21=E2=80=AFAM <tuexen@freebsd.org> wrote:
>
> CAUTION: This email originated from outside of the University of Guelp= h. Do not click links or open attachments unless you recognize the sender a= nd know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.=
>
>
> > On Feb 25, 2024, at 01:18, Hannes Hauswedell <h2+lists2024@fsfe.org> = wrote:
> >
> > Hi everyone,
> >
> > I am coming here from
> > https://bugs.freebsd.org/bug= zilla/show_bug.cgi?id=3D2771971160
> I guess this should read:
> https://bugs.freebsd.org/bugzilla/sho= w_bug.cgi?id=3D277197
Btw, what Hannes reported in the PR was that performance was
about the same for Linux and FreeBSD NFS clients when the link
was using a 1500byte ethernet frames.
However, Linux performs much better with 9K jumbo frames
whereas FreeBSD performance does not improve for 9K jumbo
frames. (Some mount options I suggested did help somewhat
for FreeBSD. Basically increasing rsize/wsize did help, but he
still sees performance below what Linux gets when 9K jumbo frames
are used. (I did note the potential problem of mbuf cluster pool
fragmentation when 9K jumbo frames are used, although I did not
intent to imply that this issue is involved, just that it is a known
deficiency that "might" be a factor.)

rick
>
> Best regards
> Michael
> >
> > TL;DR:
> >
> > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) ad= aptor each.
> > * I am trying to improve the NFS throughput.
> > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS.
> > * Increasing rsize and wsize to 1 MiB increases throughput to 395= MiB/s
> > * But a Linux client achieves 560-600 MiB/s with any rsize.
> > * The mtu is set to 9000 but this doesn't make a difference f= or the FreeBSD client (it does make a difference for Linux).
> >
> > I assume < 400 MiB/s is not the expected performance? Do you h= ave any advice on debugging this?
> >
> > Thank you for your help,
> > Hannes
> >
>
>
>

=C2=A0Hannes
=C2=A0=C2=A0 In the dmesg po= sted I see that you have a epair loaded . Are you trying to do NFS inside o= f a Jail ?

Rick, Didn't someone from Isil= on or Dell/EMC post about the 9K frames a long time ago ?=C2=A0 I know in i= silon land
their FreeBSD can do this, but I can't say I have = any idea how it's being done. They do have some kernel auto-tune magic = as well
to find optimal send and receive buffers. Maybe what we a= re seeing is Linux having better ergonomics on buffers out of the box ?

Hannes
=C2=A0 Can you post your boot.conf a= nd sysctl.conf settings.
--
mark saad | nonesuch@longcoun= t.org
--0000000000007ac81606123e6250--