From nobody Tue Jul 06 22:49:09 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B7B8D11DA3CF for ; Tue, 6 Jul 2021 22:49:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660059.outbound.protection.outlook.com [40.107.66.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GKHmS3lTwz3RJ0; Tue, 6 Jul 2021 22:49:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HzrrmAU7JIh9ukaMtjZ+0IK2KHBiUY3LgzZxkyz7gN5U72XBbB8U/lmkjY7bwfcSRYuXH+jzc+WBzjBfyzKUQeHCBiZhO1kKVVtinDAC16rJkRsqYw0sbS+J4PLBLNxaPTUg8OetJ5pdQnYO4fq/4ED/JtFwQjp1amWOHo05mq1boCzbva3qZylMxrs0Hldt/QUD0ZI21Ef5nt3VwhRIWsMDrjsGa0w54XXMlmzKvufcyTuwCgf5Wcn08xIkkKJxls3O6rp4HXb3FFbfSzlGK2XPt59ObIK2CtVJP8ynmdorwnZyH3qN7bK5Uz/K6McjQHSGXk2gQJm7bsWMehnPIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Hj4EOA0fQZh2UG1RpvoSmiqlczOZMJXFfZtHzMdPA4U=; b=KltWpWpyP7o/Q5EFPyxHHCLYnRmtKiSGcLY2pcZdf2DFw1DkbJDRRfl5f6O5uEHAaUlVB+vr03WF7O6OtB7p4pYuofd38iTKwYe/BWsJhLr6kSrTM/VZFzGkOf4VPNIPqbGU5IF7ZOBIBpD04tVV4KqVrVaXgdvvQz83ZWLDk9X4Ls/KVpXdcawHrJGwq5N+Gk6jmp1cV4eR7CFaTm/Zwdwo/Mj1tkmEf1D+aM/RQFuQ36Hot84VIlIDNSkREYqBPImIromA8+s3f1R4IDDbnpTGwWVlQn2CJpiZ7pJ3TmAxRnhhFK4A8fdMPYUK+lUM5iLX+wT1VK7FiZH8cdhsdA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Hj4EOA0fQZh2UG1RpvoSmiqlczOZMJXFfZtHzMdPA4U=; b=Rt2JJr31kCkb9u4L0iszqq2Qsx1evKbWvCjyT1ZLjbu3z8M31/X0TAQeoyFgbkdduIcBSv8zkSbKEDZ/o7ERvjRCkABJv73HtTcYOz35G2gOPDRUA5o2SyZQh/J2n7PKVb4EcP7KhxmY5JGZqe2xEdFd0aa8p0x/jWTH02/wuocHQ5quwRIhdzJGbBvFFV9fZ0Au7rQ7W/g2wWS4FTqODzKNbFLpK5Lzx+/Rla+JhlFZvBL15Gi8huAnw9sbWcdeVLH6RDjDty0eaNWlBM7gbMvJtCQeKpZ6VuB1h9AIyShHpYP+a9FbwvfblPsEjSMR+CL2dbF7Ddz02+X3dRAr8A== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB4357.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:6::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.19; Tue, 6 Jul 2021 22:49:10 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf%4]) with mapi id 15.20.4287.033; Tue, 6 Jul 2021 22:49:09 +0000 From: Rick Macklem To: Adam Stylinski CC: "rscheff@freebsd.org" , "freebsd-fs@freebsd.org" Subject: Re: Issues with NFS RPC Thread-Topic: Issues with NFS RPC Thread-Index: AQHXcm21NJMKcd9NW0iP6xnIn7lYjqs2Ar5IgAADWwCAAAAt6YAAK9CAgABYpNU= Date: Tue, 6 Jul 2021 22:49:09 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a371218c-3421-4f77-25f8-08d940d044b5 x-ms-traffictypediagnostic: YQXPR01MB4357: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8273; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: /uL0QND9rXK8StENL8qkchVdwAGkTM5yzJK0OEbkC4EHneYVjDKDMjatHnTUZIKEqtwWvTLdG7cMIGg/n7s1/brY+bRE6PKxwzLXGieEWq7Q3s4WPXaWLbB+VIotBGtm5eStuPvwpqFG2j3HRwj/ruQImQ/EV9cxuth47B3CkwUleItdY+5osVH0EfURp5Xkh+Becp1bSYuq7aUC7+KDOSKgn465SKsBrKwSGrtPtVkBFqCLe/Fne07JH8OCyqGhfy54hQdjrX2qNZSjpxtSmOzLxxLq9fZkdexxGQO6DmYfwtTKDHYUJDK8enPSbtJkiYewq2mYC9YV9J5eZFN5kI+BCTJnPepmNwJ/QU7B2/JArRmLDZSBexnqFoHWPfRjVLgWyiyDJMAYFxC64oa6yqGHqkxtLhdvlYYy/I2lm6OeTdSKX0Yu5cuRAPvLIY6Pfn9hmkbBrEXNxooUOA+MCGvd+qihQ6IyassHDnXVfFCn/YrF0JrLiyav769/59A7ivEPv4MXx00Ifn97pJzPA0/URGDO421aKT3BeWrmy6TkMFj+R10EuTH921jnVoj5hhDfBalIxbxXIrot4eas7oOb2ho8i1bcyHmT3AVUdim+zfBxSemwd9PcirKruGGM37Mt9Fwmbu5d354IDyC5xw== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(136003)(376002)(366004)(396003)(346002)(39860400002)(6916009)(45080400002)(8936002)(8676002)(76116006)(122000001)(66476007)(4326008)(66556008)(66946007)(64756008)(71200400001)(478600001)(186003)(7696005)(9686003)(66446008)(55016002)(53546011)(38100700002)(6506007)(54906003)(52536014)(316002)(5660300002)(786003)(33656002)(3480700007)(2906002)(86362001)(83380400001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?maj0GUsmNZX9nmrEDuIpi23HJ+HaGprwRxh5uZFHbgOFJfRjiy7CKygmks?= =?iso-8859-1?Q?qxNBc4b0V/QvJtgP4jx3Yz35cVnwZFGAQwytEmUTWCC8Thxl3qXOizj9xU?= =?iso-8859-1?Q?MTet9/W2gWHx3f3kDtnI+j1/hQyguNqlumtEiiapuOkS41xpRm3ENdLyqo?= =?iso-8859-1?Q?MEaAAJ7EnprxJMHWzPVe21SCXzisRDgnwIEoY/AwLPAdnXWyLD5KI7mcNl?= =?iso-8859-1?Q?SH9DjArWQGVcYKrOHIzNQ28nTUn8iwcuraH4wSSCHD2TEiYYa2ZFsMOH8s?= =?iso-8859-1?Q?5w8TrA93UqwyLY9jgqpA/te/oTjqZpYCam171QAi1VqJ0vDzG0mw+12Epp?= =?iso-8859-1?Q?B9hSeyCuFn2Nb6Zq1G/CF0qN1KgRhZVWmfhdWuV9+xGb2MmUYKYU4if6XR?= =?iso-8859-1?Q?9pHa8JDZSTUzr6KPrNK7ja1ZJJdFrImiSm71EXcmAtZNsflHcinli5Qa6x?= =?iso-8859-1?Q?Z5L/4UVCqXJLgUA1YC52+xvwDkKbG43vxm14jRMjqt66YjrajUaME3149N?= =?iso-8859-1?Q?8ROFHbcvX4SN6FRourpYEhjomWdKAtk4nTfoOApm3Wu5pbT1+0O7pIR4jG?= =?iso-8859-1?Q?To6eEo7NtFutp/4o/T8d/hqNbhNLnUYoQNTPLa7llkexIkmNMd27RDxm1K?= =?iso-8859-1?Q?8XeXOYzG+u9NDC2ruXIO0TIcwEGeFAHm9+IvG8r5M8iE7Za6Gsn0XxDrRs?= =?iso-8859-1?Q?aI4duX9yWUEOui0NxhWMQNwxAfCaP6Ro5jluVvk3Gx6B45unKAQHWJcvXl?= =?iso-8859-1?Q?ZwurjeS31nC8zY0GG7ZgImHkpf736ORdNAX1SDjV8qqPu36jhHoqdxiSHv?= =?iso-8859-1?Q?CrRJ2OE4TpXgS+KNOIMqbxUTMKw1zsr0/EqZb9r9U8XGs/bm11XbXWqh5n?= =?iso-8859-1?Q?gQo5nrjKec0CG3L4TP7R5FL8QUfLbywPziBQaRbkwYq1oYXBmVkM6QLP85?= =?iso-8859-1?Q?t+5f+kGR9ArIxA5p0mZ2WPYhEPqfQ/qyfwcrsHsybEBKRKvEZ5ifsNjdCQ?= =?iso-8859-1?Q?OLUIqKPq5DBrC3KbdVPQHgMesZ2h9q3i5o071VbyYivrpqs2Kju9agRhTP?= =?iso-8859-1?Q?JGnjitIorwxz8W5DsMT53zEdUszEhXxQO9O8pRenZCZXNjKAcd1FPvCmZm?= =?iso-8859-1?Q?LgYDgwG0wqgG58Tu8f7lvSS4Pv2gdYf3RyRr9tcD4gH0nudRXHgh7b4Iz9?= =?iso-8859-1?Q?/C9xKcIRdZ8sQfNoxTXk+HvIr5zo9eLFgvBicJaxqUC001HFj84De9FvbA?= =?iso-8859-1?Q?2QxP4+mook3EGI3y1bX4rHqdsJAv90fRBZ03iNp/hMLSAeO+dfgJHmKoPc?= =?iso-8859-1?Q?TazHNEVvV8etFw6R+ovipE24kt54dB46T0lPqhQVGrRmviWOOBrD3nAME9?= =?iso-8859-1?Q?HRJu1Bl81EsNARc/LiuMZr19KG7Pv29SSAUSAqVIKXhf0ngicEUZw=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: a371218c-3421-4f77-25f8-08d940d044b5 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Jul 2021 22:49:09.8744 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: OzB4ZeZBYx9un5Ih9Q3O8DeRYHmTSSSF8P6GRdBBPmZGxjyPPwQ+f+ucPJrktOgcb+Da6mqyjHGTBvUXhFl5yg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB4357 X-Rspamd-Queue-Id: 4GKHmS3lTwz3RJ0 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N Adam Stylinski wrote:=0A= >Well, I put in a plea in that PR for something to trickle into RELENG, but= , if not possible, does that patch you posted >in there for -STABLE apply c= leanly to the RELENG branch? It looks like it calls some helper functions = but from what I >can tell they are fully present in the 13.0 releng branch = (CC'ing the list and rscheff on this as well).=0A= Just to clarify...the patch in the PR reverts r367492 (the commit that caus= ed the breakage). It should=0A= apply cleanly to a releng13.0 kernel.=0A= =0A= It is not the patch that fixes the problem that is in stable/13. That one i= s on phabricator in D29690. I do=0A= not know if this patch can be safely applied to a releng13.0 kernel. Presum= ably rscheff@ will know if it does.=0A= =0A= >I used gmail for all of this correspondence so hopefully the mutt or other= mail users don't lose their mind with all >the top posting. Sorry in adva= nce for the poor ML etiquette.=0A= Personally, I don't care and, personally, sometimes it makes sense, if the = reply is not addressing individual=0A= comments within a post. Not to mention, Outlook is what I use and top posti= ng is certainly easy (and expected)=0A= by it.=0A= =0A= You should do "netstat -a" on the server the next time you see the hang.=0A= The connection will be in ESTABLISHED state with the Recv-Q value non-zero = and increasing=0A= over time, if this is the cause of your problem.=0A= =0A= rick=0A= =0A= =0A= On Tue, Jul 6, 2021 at 10:59 AM Rick Macklem > wrote:=0A= Adam Stylinski wrote:=0A= >Yes, I'm using 13.0-RELEASE, with the latest security updates from freebsd= -update. If this is that issue, I'm glad it's >known and fixed, but I'd re= ally hoped it'd get backported to -RELEASE :(.=0A= At this time, you need to build a kernel from patched sources. You can eith= er use stable/13 kernel=0A= sources (which, of course, pulls in other changes) or apply the patch that = reverts r367492 that is=0A= an attachment on the PR to releng13.0 kernel sources.=0A= =0A= You could ask rscheff@freebsd.org (who is the a= uthor of r367492 and the patch that is in stable/13=0A= and is believed to have fixed this problem if he has considered asking re@f= reebsd.org w.r.t. doing=0A= the patch as an errata for releng13.0.=0A= =0A= >Other info I forgot to include: the firmware on both adapters is the same = and the latest for the ConnectX-3 series. >The ring size, channels, offloa= d, and all other settings are the driver defaults. This issue is not limit= ed to just this >workload, it occurs on other files as well.=0A= If you look at r367492, it changed the timing and lock semantics related to= TCP socket upcalls.=0A= As such, it probably doesn't matter what network interface, etc, that you a= re using.=0A= --> Although I would not recommend it, you could use NFSv3 over UDP, since = this is a TCP specific issue.=0A= =0A= rick=0A= =0A=