From nobody Mon Feb 26 03:16:52 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tjm4f14mpz5CCjK for ; Mon, 26 Feb 2024 03:17:06 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tjm4d5Mnpz46Sj for ; Mon, 26 Feb 2024 03:17:05 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-299354e5f01so1419908a91.1 for ; Sun, 25 Feb 2024 19:17:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708917424; x=1709522224; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ljaivC3/0fVOJJa45G2UNJD8J/oENdi486mhlQaqzHw=; b=SQNbDZWqRUA+Dr6jjAxU3MToXqeAEx2nqre7XsM/IuKN6gPq11nN51RvCy1ZJViLMz 5zleANF/pfzt5rB4hvyO3jwZ1jMfX/tiyUwaQ260a8BWV6q4rOSpfs7XHdqhjw4Wyi9v 5hUgG32PFrki1TlbUa/lqwjsBAxh/7Eu1+JovPgK0Uz/fak61Bs5tH6GhUueIM6dyzTT M7D6cXFMoE6rVQ7Yxi/Kc+vzMBkCgrUGJ1QNJSMOVGodRf95j2ezXouliG0EQbpDyf03 Uj5ZPJFXy5fNzRpM3I1bfqp7m7yf893v+jJ8vRenqn6TzTjQPTnMsdbQsFqZadlaVgyS gjUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708917424; x=1709522224; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ljaivC3/0fVOJJa45G2UNJD8J/oENdi486mhlQaqzHw=; b=XGs/XD8dp3OzzNvPKYfeef1pf5v5ujiYVNaIzecCGnv9LnS909dhNpR8PaLg08S6st +ZUQ/uPO8R7j7Hh+viOyK9DcajGUTY5ymSiu1F4p1elkf1/cjbxbtq1rRFNn1ezQVR3r D5S6Rb3eGSTv5/41cXW2FKW1LhLhrHS0zkAdKUfRNJSPBX2IXvYKjxDoG97MzBSDQBqL tYqv5enHCbVnnemyh5/I3zJJWejw7vyD2Xjsi1pYYT9xu5AqoFWRpV5fuJAM6NhMx91S utuEaagxCyN1XSclFYyWdFaGzfIh0YEAlkz5PBN+XPtkinecIxJdB1fjyuQGS3oB5lG0 rdgQ== X-Gm-Message-State: AOJu0YxUhtipZ31f7tP14ZwteU0HFC7QcD0URRmUrABKM9N4wMQMIuh+ /ZzZdUYkHy86IcUL2YeQuKBFn84rOt4vDQOTC5yqiHDzcoDtvLy5wmIQzqKbqe0psApYeqUkb5j hVqEtLExrsKt1Kd7VnJtv0U0kAg== X-Google-Smtp-Source: AGHT+IFpH2yz7+DOwhGKzChxqBd85at2Wghou9oCC4U3yWSfUFnqKNlU6tn1ryd1RWnpksYjjCzaqlqrZLOTq2Ktg0Q= X-Received: by 2002:a17:90a:1789:b0:29a:9dd1:d45b with SMTP id q9-20020a17090a178900b0029a9dd1d45bmr6308267pja.3.1708917423778; Sun, 25 Feb 2024 19:17:03 -0800 (PST) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <034cc6ea-26d8-4520-879a-672459832407@fsfe.org> <9066A50F-26DC-4314-B79E-66120A2B5A2F@freebsd.org> In-Reply-To: From: Rick Macklem Date: Sun, 25 Feb 2024 19:16:52 -0800 Message-ID: Subject: Re: NFS performance with 10GBase-T To: Mark Saad Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4Tjm4d5Mnpz46Sj On Sun, Feb 25, 2024 at 4:57=E2=80=AFPM Mark Saad = wrote: > > H > > On Sun, Feb 25, 2024 at 6:51=E2=80=AFPM Rick Macklem wrote: >> >> On Sun, Feb 25, 2024 at 1:21=E2=80=AFAM wrote: >> > >> > CAUTION: This email originated from outside of the University of Guelp= h. Do not click links or open attachments unless you recognize the sender a= nd know the content is safe. If in doubt, forward suspicious emails to IThe= lp@uoguelph.ca. >> > >> > >> > > On Feb 25, 2024, at 01:18, Hannes Hauswedell = wrote: >> > > >> > > Hi everyone, >> > > >> > > I am coming here from >> > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D2771971160 >> > I guess this should read: >> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277197 >> Btw, what Hannes reported in the PR was that performance was >> about the same for Linux and FreeBSD NFS clients when the link >> was using a 1500byte ethernet frames. >> However, Linux performs much better with 9K jumbo frames >> whereas FreeBSD performance does not improve for 9K jumbo >> frames. (Some mount options I suggested did help somewhat >> for FreeBSD. Basically increasing rsize/wsize did help, but he >> still sees performance below what Linux gets when 9K jumbo frames >> are used. (I did note the potential problem of mbuf cluster pool >> fragmentation when 9K jumbo frames are used, although I did not >> intent to imply that this issue is involved, just that it is a known >> deficiency that "might" be a factor.) >> >> rick >> > >> > Best regards >> > Michael >> > > >> > > TL;DR: >> > > >> > > * I have a FreeBSD14 Server and Client with an Intel X540 (ix) adapt= or each. >> > > * I am trying to improve the NFS throughput. >> > > * I get 1160 MiB/s via nc, but only ~200 MiB/s via NFS. >> > > * Increasing rsize and wsize to 1 MiB increases throughput to 395 Mi= B/s >> > > * But a Linux client achieves 560-600 MiB/s with any rsize. >> > > * The mtu is set to 9000 but this doesn't make a difference for the = FreeBSD client (it does make a difference for Linux). >> > > >> > > I assume < 400 MiB/s is not the expected performance? Do you have an= y advice on debugging this? >> > > >> > > Thank you for your help, >> > > Hannes >> > > >> > >> > >> > >> > Hannes > In the dmesg posted I see that you have a epair loaded . Are you tryin= g to do NFS inside of a Jail ? > > Rick, Didn't someone from Isilon or Dell/EMC post about the 9K frames a l= ong time ago ? I know in isilon land > their FreeBSD can do this, but I can't say I have any idea how it's being= done. They do have some kernel auto-tune magic as well > to find optimal send and receive buffers. Maybe what we are seeing is Lin= ux having better ergonomics on buffers out of the box ? > Oops, my bad. Yes, I just took a quick look and it appears that jumbo mbuf clusters are now allocated from separate uma_zones, so the fragmentation problem sho= uld not exist. (I hadn't heard about this for quite a while, but hadn't noticed a commit fixing it I'll wade through the commit log to see when it got changed tomor= row.) So, I think the above comment should be ignored, unless others need to corr= ect it further. Sorry about this, rick ps: I have no idea how Linux does network buffers. > Hannes > Can you post your boot.conf and sysctl.conf settings. > -- > mark saad | nonesuch@longcount.org