From nobody Mon Feb 26 18:25:26 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tk8Dr2QtVz5Bvpc for ; Mon, 26 Feb 2024 18:25:32 +0000 (UTC) (envelope-from alex@hotelwifi.com) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on20701.outbound.protection.outlook.com [IPv6:2a01:111:f403:2405::701]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tk8Dq26Qqz4nk6 for ; Mon, 26 Feb 2024 18:25:31 +0000 (UTC) (envelope-from alex@hotelwifi.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=HotelInternetServices.onmicrosoft.com header.s=selector1-HotelInternetServices-onmicrosoft-com header.b=jyx0CcCP; dmarc=pass (policy=none) header.from=hotelwifi.com; spf=pass (mx1.freebsd.org: domain of alex@hotelwifi.com designates 2a01:111:f403:2405::701 as permitted sender) smtp.mailfrom=alex@hotelwifi.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ahuhwEaqC2ELHvmTRTfi+CP3fNj+LzY6h8avqjFvHM9tkjeoku6mpTyvEqVCipGTl2KVg/DFrSlPWINPUoT+zQh/ELzHGhEYu3wXvPR4OxEoRynNrhVv+Zm+MkEhXuThHAlj752U8s7/sxmICsmVXz44/mk8M1psOADf8D1K15fVjBlbKDNMFSGG0koPIsu3B5RLfTJcQw4mTb9f30EvKYY7xuFpuXmAHShQeg0KHiBXdtysfbYGQ4ddA9IWEcC90uasd4bvorKr0MmoYGj3uTBGJrM01HgSL3ysVGd3C77BQUQMNCTQNLY4DJCVIA9jFDB9mktBym6uM9l0CsPnXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=y4eViX9+nwCgC4TqSnaMKqgfUueqXaiW3bB0eGL1dU4=; b=VfAqOtfM4oYYsmKZKHsNxE3WmEhdoMjk9o+Kohhco8xvSTtx/jfQ7t9Hn9fGUTL0WiPPmrwSBgbsepz0xn4qISQVvkIMlBRl1Xky2qPyuwBXRpzGRGXNdrRwvhcRROPitLcvNnlfoTVvC/RrPeH+qHx3SSCiAtwTbyyf7isf7IG2gVqblpn+6slc7GLcq5j7KDgdlUKk0UktYwqerSluooYjfNDOy4kN2RFkYdwDMSbSR8InfNu9KhV+gA4TrHu+y1gTtLxKv9YlX+fLyzaMRpKF274F3OClvv8x+9Q7IcRsAjE44GW7tbRRTARU2pvwx2pOBt1BdagfIPodLp3vzg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hotelwifi.com; dmarc=pass action=none header.from=hotelwifi.com; dkim=pass header.d=hotelwifi.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=HotelInternetServices.onmicrosoft.com; s=selector1-HotelInternetServices-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y4eViX9+nwCgC4TqSnaMKqgfUueqXaiW3bB0eGL1dU4=; b=jyx0CcCPlFyEj0DjAANzXGyNxIsNFQ17DGq42XF1J3fE9kPRk+3nUVTkpdRezWurfZQUo4szrjYXEgrFjAxJ30fXeF2RYyzjEz7ufklIrCISE8JjhnFpVgW4A0RnEEjI8LkQk03rom4N+kuY4IcthPxmDokPsxAQe0/N/MhqQtM= Received: from BY3PR13MB4929.namprd13.prod.outlook.com (2603:10b6:a03:361::22) by PH7PR13MB5892.namprd13.prod.outlook.com (2603:10b6:510:15b::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.34; Mon, 26 Feb 2024 18:25:26 +0000 Received: from BY3PR13MB4929.namprd13.prod.outlook.com ([fe80::f2c9:5c95:6aae:f1ae]) by BY3PR13MB4929.namprd13.prod.outlook.com ([fe80::f2c9:5c95:6aae:f1ae%4]) with mapi id 15.20.7316.035; Mon, 26 Feb 2024 18:25:26 +0000 From: Alex Shalima To: "freebsd-net@freebsd.org" Subject: X710 stalled TX Queue and loss of networking Thread-Topic: X710 stalled TX Queue and loss of networking Thread-Index: Adpo4QO30tiNh452THKcfhpXYYPjXw== Date: Mon, 26 Feb 2024 18:25:26 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: BY3PR13MB4929:EE_|PH7PR13MB5892:EE_ x-ms-office365-filtering-correlation-id: 45133775-82e1-4613-611e-08dc36f84dbf x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ZPEwJLJLR7hBlJ6JItObGmeYjnx0AY4Df2JXWpvbv+XUyf5XJ88sb+B6NBYDWRD0LI+4yPwpe++cJYEiG/C3nr/gY4uvlVsYeL2tKrQknAgpWKmmpN1b5Kk1cEziBIcxP/IaMF34/D6aRQ5b8ka5OAMrC7+HcAG4m5fs17V7bfRxsbIljHdgNw21vcayqOebXO4uCQIwmCAtT/c0H9MHwNLvDQ0UCCSX+V00/agQqmlVZHBUcZ0cgpMxjFrgW36tsu+LDPUTMIzplwJd1FPLGTlILxXSVUvtbZtGpNAovzKgbKkgZEwUGci9cDFDy11Gf+zy8YQE+iO99C7ygW9EPkfrD0F4XtfuRtFd9EHjUVmcXOpbBnEEdEfviQeXlye7XpH+IGrGAh4TG/rVY1YHGJ48iopLv2HUSTsr8MJ/bfcjkA2SIR/k47rF0KGERRJ6hdME9BXl8WPj0rIHGieAsBs+i5FLcsZFs2ATyPoDQlAd4gTHJE6CJiaEUTE9CssmpxcjnbS+dSh6USnhoSIhb5PcZPQ71BKWcKBWHPy8tkTxWTMFFdeN7E4tfkGiJCqhW3DsW4fGZFqpFcBK82dbi0Q1NtFcAw0+vNNX/6AB+5v6e9YZeqqGJ3KcCeDK/Q285S49kzHf4/ZidLKbEvvrDdCkroZdyJJReclureVPTB31mpTTz8+E7F+JHntaU2BNQXncCoiT8CQKuSPNl2am5XV9xrDyDFUwDBvQ6hAsKY4= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY3PR13MB4929.namprd13.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(38070700009);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?0I2wFs+HThcsSplxh5tw6kK2IfJXRWXAt/tmaZ4x4cfrscCYFBU2L0oNpA9l?= =?us-ascii?Q?+sdsdhR3csWl20JdR/o01wb5Hc36uOQUGhisB/yl04POlvD2BHiy160MSNeW?= =?us-ascii?Q?8ZG92ZeEWQ4tIchoIQY+0UueI2uzLMCMVCS8lJOmB4XZ+PfqJ6GzMkjlxTBG?= =?us-ascii?Q?MUd0NCTROApp1L5Hj7zSXl1p6epcClRnfmp2dVQhUwUWpFsDJ8eNeEIeS0lq?= =?us-ascii?Q?kDdOBY4qOVADz8N/jG2ywSXT6aqMQeIx3bIo19XMszGc4edPrT+nQMYF/4e3?= =?us-ascii?Q?LegKEO/TW+/VZek8sTzZYGHkgK2Pw5eqSkMtzBAIowfjccUdyFVvUTVOMi+Q?= =?us-ascii?Q?j+5ccu+VG2/UXrWG+Pwj60oeBB3/YpTzXp6rguXQ/QzHdy9aDqwlyo75onbM?= =?us-ascii?Q?ggaWN6hty4Z2CYQcn/WFYuCF2ZwaZQsrlKDwSzAChRyvtyW06TS/3lcQswkE?= =?us-ascii?Q?DajIAxU1F9L/8xoFNnI+PWqprLGiYcos1j3P5n+Yl2J+n6ussj8+g5qEIPin?= =?us-ascii?Q?JrbuC7/sAMEOrukMVpJGegxzOZVDO1g7mQCeQiwRmTtFXxvCXFkg+Dr217S9?= =?us-ascii?Q?H454k1+9NMO/7iIEjP9zlyoDC6bqrPhRPrCafoF3rps0iYDv6JCWy1g+2hSd?= =?us-ascii?Q?oe5papwimY5ejmEv2GLQTXA2tgADl432rDgxgMr5I6zI8LEx5x07QpLqwDm8?= =?us-ascii?Q?IKGX/CPuCS7NIgi3ALvUdzguW8/nR8qYkfoS//8SkUWzRq/oZdULIUnQceTI?= =?us-ascii?Q?ua8YM7Jn7W3tfRGhJ5PN2qtx7m/toN26URrOLQB+lvohhOMVjaPqrYiDF9xI?= =?us-ascii?Q?hyePsSaSjEOdt9jXGVenjFE4C0I0rfQ+COuADGp1+90GHTUPZ2N82juWGYUP?= =?us-ascii?Q?pvllapQeQyBd7j7+s57sOZJrzJZ+UxDanPOYbNYdBEN6xsCJiZ+tUTHuJlEo?= =?us-ascii?Q?Ff8vEzfNGvHUZ4UmR3l9wAtRX8pnzc2TjQDTUWa/tCuKQPJ0hRqiUjZKbJ1T?= =?us-ascii?Q?7dBTEY8nzQkjnJNl2YvhE6a+s3/Z0OCApZ48D012mblD4ZhLI5fO5KaAdIo4?= =?us-ascii?Q?qs3g8+3p3HlQvjxasVlFpOBoJ+WpOAOKyCgf7GbR1NQHk2x/A58RtQGqykyh?= =?us-ascii?Q?xNlyCUozeBbDPUeGLy1jEK91o0AI2yOam8PQQZTcwEyrgGX5tG8A0z3PU2Uv?= =?us-ascii?Q?uVId3ZCwXSJUQOFAyhUGOSdxkwCLbwDtxyKg0+gindwmuEQiAU3BtuX9h+wL?= =?us-ascii?Q?uoAzicBt8V4LApejOibSK+cobxRqFCSdaEU1gUlDZB6XmOANfUb2BcOIHdsk?= =?us-ascii?Q?cWDOYEEZ9IYOWs6SiD6V7QqymzKO9dw9B084eDT84Q5WTFDuI/7umfOZ4vDt?= =?us-ascii?Q?Pu5JL3fyEwFQCD1lGL4fNbeQUCnm4i8C6/r1iLGg+Oy+SaRxnPAF+XzrhcLu?= =?us-ascii?Q?HBXcZRWGSR+/DJMgQDqHSPtTPr/bB7lIAiH3KBqO9CbzSzs+Nh+8HL4BpwsJ?= =?us-ascii?Q?rpoNI/8fIAT+lNJC3neIKM1aROW/OrMSq0EyxQSgFGDN+5qFOBoprUoEjunI?= =?us-ascii?Q?aKnpR4+i7Yv/RdAyQaU=3D?= Content-Type: multipart/alternative; boundary="_000_BY3PR13MB4929243A0160A0B6206530C3CE5A2BY3PR13MB4929namp_" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: hotelwifi.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY3PR13MB4929.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45133775-82e1-4613-611e-08dc36f84dbf X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Feb 2024 18:25:26.3533 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4f7c46a4-73cd-4ede-b226-abd17a692a0e X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Lt7BEso3JJxXOF8XfkoEg4ksJE3LrOgDafNJLLGO+N7bobyX75eZZFkGP5UD3bWXmFJ6xP2kU6eWQn+FEMWxQg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR13MB5892 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-5.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[hotelwifi.com,none]; R_DKIM_ALLOW(-0.20)[HotelInternetServices.onmicrosoft.com:s=selector1-HotelInternetServices-onmicrosoft-com]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f403::/49]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[alex]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a01:111:f403:2405::701:from]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[HotelInternetServices.onmicrosoft.com:+] X-Rspamd-Queue-Id: 4Tk8Dq26Qqz4nk6 --_000_BY3PR13MB4929243A0160A0B6206530C3CE5A2BY3PR13MB4929namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, DATA We are running FreeBSD 13.2-RELEASE-p9 #25 on top of several Dell R650 (Exa= mple Service Tag: 8FKQRY3). The system is running bhyve for other FreeBSD V= irtual Machines. All these servers have X710-DA4 Fiber Network cards (4 port of SFP+). dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k dev.ixl.0.fw_version: fw 9.840.76614 api 1.15 nvm 9.40 etid 8000e9b5 oem 22= .5632.7 Some servers have an additional X710-DA2 (same card but with 2 ports) for e= xtra fiber ports. ISSUE Periodically, the networking will stop working on individual interfaces. Du= ring packet capture we can see that the networking card is receiving traffi= c, but no traffic is being set out. During further investigation we found t= hat ixl interface TX queue is getting into STALLED mode. [user@server ~]$ sysctl dev.ixl | grep ring_state dev.ixl.5.iflib.txq0.ring_state: pidx_head: 0751 pidx_tail: 0751 cidx: 0751= state: IDLE dev.ixl.4.iflib.txq0.ring_state: pidx_head: 1254 pidx_tail: 1254 cidx: 1254= state: IDLE dev.ixl.3.iflib.txq0.ring_state: pidx_head: 1193 pidx_tail: 1193 cidx: 1195= state: STALLED dev.ixl.2.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000= state: IDLE dev.ixl.1.iflib.txq0.ring_state: pidx_head: 1393 pidx_tail: 1393 cidx: 1395= state: STALLED dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0181 pidx_tail: 0181 cidx: 0183= state: STALLE RESOLUTIONS TRIED * Factory resetting the system (not a permanent fix, issue comes back) * Recreating Netowrking interfaces invluding VLANs (not a permanent fix= , issue comes back) * Updating the driver with Dell iDRAC to the latest official QUESTION Is there anything else we can try to get this permanently resolved? Best Regards, Alex --_000_BY3PR13MB4929243A0160A0B6206530C3CE5A2BY3PR13MB4929namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

DATA

We are running FreeBSD 13.2-RELEASE-p9 #25 on top of= several Dell R650 (Example Service Tag: 8FKQRY3). The system is running bh= yve for other FreeBSD Virtual Machines.

 

All these servers have X710-DA4 Fiber Network cards = (4 port of SFP+).

dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 f= or 10GbE SFP+ - 2.3.3-k

dev.ixl.0.fw_version: fw 9.840.76614 api 1.15 nvm 9.= 40 etid 8000e9b5 oem 22.5632.7

 

Some servers have an additional X710-DA2 (same card = but with 2 ports) for extra fiber ports.

 

 

ISSUE

Periodically, the networking will stop working on in= dividual interfaces. During packet capture we can see that the networking c= ard is receiving traffic, but no traffic is being set out. During further i= nvestigation we found that ixl interface TX queue is getting into STALLED mode.

 

[user@server ~]$ sysctl dev.ixl | grep ring_state dev.ixl.5.iflib.txq0.ring_state: pidx_head: 0751 pidx_tail: 0751 cidx: 0751= state: IDLE
dev.ixl.4.iflib.txq0.ring_state: pidx_head: 1254 pidx_tail: 1254 cidx: 1254= state: IDLE
dev.ixl.3.iflib.txq0.ring_state: pidx_head: 1193 pidx_tail: 1193 cidx: 1195= state: STALLED
dev.ixl.2.iflib.txq0.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000= state: IDLE
dev.ixl.1.iflib.txq0.ring_state: pidx_head: 1393 pidx_tail: 1393 cidx: 1395= state: STALLED
dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0181 pidx_tail: 0181 cidx: 0183= state: STALLE

 

 

RESOLUTIONS TRIED

  • Factory resetting the system (not a permanent fix, issue comes back)<= o:p>
  • Recreating Netowrking interfaces invluding VLANs (not = a permanent fix, issue comes back)
  • Updating the driver= with Dell iDRAC to the latest official

 

 

QUESTION

Is there anything else we can try to get this perman= ently resolved?

 

 

Best Regards,

Alex

--_000_BY3PR13MB4929243A0160A0B6206530C3CE5A2BY3PR13MB4929namp_--