From nobody Fri Jul 09 09:16:59 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4E482124133A for ; Fri, 9 Jul 2021 09:17:05 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from APC01-HK2-obe.outbound.protection.outlook.com (mail-eopbgr1300121.outbound.protection.outlook.com [40.107.130.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GLnc014B3z4Rwy; Fri, 9 Jul 2021 09:17:03 +0000 (UTC) (envelope-from weh@microsoft.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EMg/P7V1H+blLqvQ8PWRymzqn0EzyzfAVBhrHHogejfHmbWHKJmN0RRaoznOJ+dcnOwlaOWKfTI0CCoNOs9jGSuWkk5rX+oaNYri46zUqbSaFotVhg57u0x/kImR7RvBc1a5bo/l4zhsKuWH8qu2QDUUav1sYJmYNYPz4IUc178w/5xAfijSFnTw4JV9jWZnmabY1SzqM/zO2tNR5XdUhJdYKYAcl8T0c6K5MpgKkT4RbDahneyu/3QxPQrubDPrdVO1Pe22PykqSd1NR4mkwF/Duqb0Doemc5kBKm1URyoEDM7ge9C2UmrDs3IrCPQY+/KP2EA6CwFWYT0Poo/3Ww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rBGwM/gkARAUmrJFls2ziTKSjjam3YmWP1zQRllZxIU=; b=nN35PXoyCjlij79+/nLaRnb4X/DXr8in3Oa8QDirERKnBLsHZHbkXqJGe/yd5gvTtOCtFuM7ZyAZpM/ZWfyj3nrU/Fy4ReltwRL4I3s4OtEWMKmrhypxLt1js4NbZo/E9jsTT0vqHkYUegahbMaXWyezF06dgVj6osIr4zc6OVWTLGeVsOQd7TBRTgqgHkZIKQlqGg/KXyNxS8h5SUKl9lCuuY7iJqGcMpveeCQUwxiQ0sR0BIBa0reSNVScT/dr1ZLjBvXJV0bIWaP9//X5Pkl8swQ9F3L4Zqr3SEuBd/zaEYzEpOaspnpLKY8a2Rx5OrP/b8te2JSV8NyLLEaTEg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rBGwM/gkARAUmrJFls2ziTKSjjam3YmWP1zQRllZxIU=; b=dY5zsuyN7qy4jIiMrQFst5s0gcMNTi4Q1n5lFEvoPNMYUbn5KSTG5Pj7oc/iU433InAWz2tRNf3RTAECb4fD/ZrZVCUyQBwpzcN2JMlIc6LE22Qd2vDBDTt7WkswqAkQ6WwsBx1Ylyimr3sXeoIYnowMF2DLxMx23Xt+lLOSLRo= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by SI2P153MB0522.APCP153.PROD.OUTLOOK.COM (2603:1096:4:12b::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.7; Fri, 9 Jul 2021 09:16:59 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::958:db78:a743:2057]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::958:db78:a743:2057%9]) with mapi id 15.20.4331.014; Fri, 9 Jul 2021 09:16:59 +0000 To: "freebsd-net@FreeBSD.org" CC: "imp@bsdimp.com" , Li-Wen Hsu Subject: Send path scaling problem Thread-Topic: Send path scaling problem Thread-Index: Add0mRhesvtMFPMmRgOO0bJFRPNJMA== Date: Fri, 9 Jul 2021 09:16:59 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=53492b47-1c04-4ab0-aade-9ff59d3b51e1;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-07-09T08:03:51Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7e3eae4e-bd2b-4c88-667e-08d942ba4e3b x-ms-traffictypediagnostic: SI2P153MB0522: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 0iJY8yZvuLfkxK8N9bLY2lcOv3nrLAWwzddyQxqviDjW98rkNZe6APpccU1TN0WY02ItwIsF24tVbbH9QT1v1NDhSmHMai8DXSXcYkSqmNuLZPsa9Vltx2x44n2ASPR5vHetG1xEY2RSWiTLPK5a4GWmT5jJPvJ1V9E7anyZvtnQoOj7qU+wanq0c9Q/ibWstffS56jHCqFOB2ABZyww48v7SWLoNuZv6FWfWjH5DZArU+5nBr/G8roCGFpY+uf8dG4nnrqZGXbQ/SVgKiCl3/7edC8G6ux298bLtXkY+1IcBA/si+TGJc9ZcyGPLc1yK5EY+NCC4KS38g1ziwR/yn8WLwDcu7JvOKy7ECr5enh7MTval0ls8nKlbZSJPF6tgGe/7THdgFqO8yjrMJRfUQiD4oZ2u18SOzIYG+pcw5j7oFZj5RXRSv/3tDjSBi5y/A66RBTQqKjDr3F/5jP2p+zXxUmo+gSgXaeLnq1lUOq5TZJekmVaCoiQFJUFCj3rxcDqE95P6f9rhFyaVbPbbxwE4+XuMaMDxajEq0uPfQxlY2KAtIrPNylk9QN2sh27MmJjam1At7p1ETsKFTX6/F+sZvAIK96l88qqjcfcOcXFeIKE4vp1Puv4n+v0hIMoawogUL8R1LhcjAT5x61O9g== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(4636009)(366004)(4326008)(478600001)(66476007)(3480700007)(8990500004)(82950400001)(6506007)(38100700002)(122000001)(7696005)(83380400001)(8676002)(186003)(64756008)(8936002)(86362001)(33656002)(26005)(2906002)(71200400001)(76116006)(52536014)(66446008)(66946007)(9686003)(5660300002)(316002)(10290500003)(54906003)(6916009)(66556008)(82960400001)(55016002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?lx7eUGnTmJIYu+QeEUz8xVBgiV8R/CnTnjnwYow9xP/lWf9IhPtv7p+azkO3?= =?us-ascii?Q?nHSvluySx86uHMDftCr/JEWU38AqFupW3a0+8mChNmNvvr4E0AUUAw3U19nI?= =?us-ascii?Q?4wvrI+YY1cN0urkMM7sAuytGF2YKyzdFXiJmn6sLgRKZGBl/ko8P1TCxi+BN?= =?us-ascii?Q?O7PcJPaHwUAHYfRtD1nywF8oskYORfHLkeKRnPNGGtBeLFXQ+4WszU1GkBgo?= =?us-ascii?Q?hA3bdYTfemRSXYhFB77BcsdKCc0sBYryz023yINfVTXeQbuQhU4FBI6y6cTG?= =?us-ascii?Q?KB3L1/k+ycC0EdmWh6FtekUvF3ZKTOBsf0uijBsh8iPO/HBdBDeGD0e7g04Z?= =?us-ascii?Q?o54hd9wlt8Bep91h6tzzEh2SIfqorJwSrbDC73Y9UArqDxPBADmasJ14oQXz?= =?us-ascii?Q?OhZk+YQPlZLdrLzkTqSRsPVG0W1r/HsuWQ2qpaxZFE1lqAAUoHN05K4WUUhW?= =?us-ascii?Q?m4ery3mnLYNHRKlyqOxyS1VHRHsyGpyzAHP83KAmGLSMGfkFaKjy3owi8Vpk?= =?us-ascii?Q?4dCiFOI7NqPkL8goF68aZmgKWRdmvLLeAsjhr48trXVvH7EivrYgpTOV3csU?= =?us-ascii?Q?FjjgHfx2pbM3yyJC+3ICo7n8iijsEYKuVYgekg9GqSBUdlfE3xNveS1IKcB2?= =?us-ascii?Q?I0pgxijS/sf3DwSV7dNn2FzpS4/faW/fqpLkvJZ2VEuq/np7JeQICAk0aZO3?= =?us-ascii?Q?1K0OJt8q9RqevARa8Z2T+KHpaygepezNryQDjJKWGWQHFog6xUtbS4pItGXz?= =?us-ascii?Q?TjlYCUOdiGidRfn+agk8eWBp+46mGypGX0Vh2TKOk2Ywnmjj9yzIafg3q4sE?= =?us-ascii?Q?UVg6O+2cfgXPFmdcLY/n9fjA7NkVwT6mIcqHEhTF8QkjCyr4vsv0qdb3BSK5?= =?us-ascii?Q?bWgYmQ8LivwUpUhDya/HswUo5yOVsouTJTJg5DLaENIqBkTl61SbAJOV2G4J?= =?us-ascii?Q?KVB/+lGbIR/Ix5aCCg3TfT5LTJdJ8wA7O6dvyUblkjf7Qbd0Qlg5+ifzzmtS?= =?us-ascii?Q?dYMjnwikQqjJHcjTB62MiR1hOVXE+4Vu8R04mfX9zf61WeJWmL/WXGhFztCn?= =?us-ascii?Q?KqDmfKGMoGxFOjX5VmtStKLeJqRnG1DnY38h+D9iNvbtb5dw71X8POgotDn0?= =?us-ascii?Q?Ab2uBj7vZDYVJ4Jp5KnUsjlIDRGSPPvZBBdayA5bUIj1FupyLH0AdKXShlp/?= =?us-ascii?Q?+jZ63kP62CGQP/MTejuH7pCX9NchivyKcd7a/9fIKje5JuxNmiU+dAzEuSxg?= =?us-ascii?Q?yeMUZMHmkGB0tmbmqNA8LhzZ24IpA52OLGlvKr77auqs6zg2+Z4X+pGfYiD2?= =?us-ascii?Q?cYc=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 7e3eae4e-bd2b-4c88-667e-08d942ba4e3b X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jul 2021 09:16:59.0544 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: AizIbnEHfhFw0cqoHfac8pKhq9q0teHdPJp1JWbty1hpogYqE4QqUhHIxb6RNhWmbhdl35EhFxdB4lbDGhIzgQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SI2P153MB0522 X-Rspamd-Queue-Id: 4GLnc014B3z4Rwy X-Spamd-Bar: ---------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=microsoft.com header.s=selector2 header.b=dY5zsuyN; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=reject) header.from=microsoft.com; spf=pass (mx1.freebsd.org: domain of weh@microsoft.com designates 40.107.130.121 as permitted sender) smtp.mailfrom=weh@microsoft.com X-Spamd-Result: default: False [-10.10 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[microsoft.com:+]; DMARC_POLICY_ALLOW(-0.50)[microsoft.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_LOW(-0.10)[40.107.130.121:from]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.130.121:from]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[microsoft.com:s=selector2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DWL_DNSWL_MED(-2.00)[microsoft.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.130.121:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; WHITELIST_SPF_DKIM(-3.00)[microsoft.com:d:+,microsoft.com:s:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.130.121:from]; MAILMAN_DEST(0.00)[freebsd-net] Reply-To: weh@microsoft.com From: Wei Hu via freebsd-net X-Original-From: Wei Hu X-ThisMailContainsUnwantedMimeParts: N Hello, I am working a driver for a new SRIOV nic on FreeBSD vms running on Hyper-V= . The driver coding has largely completed. The performance test shows some = scaling problems on the send path which I am seeking some advices. The nic is 100Gbps. I am running iperf2 as client generating tcp traffic fr= om the a 15-vcpu FreeBSD guest. So it has 15 tx and rx queues respectively.= When just using 1 iperf send stream (-P1), it hits over 30Gbps which is qu= ite good. With 2 send streams(-P2) in iperf2, it reaches 43Gbps. The more s= treams I use, the less obvious the scaling I can observe. The best performa= nce is around 65Gbps with 6 send streams (-P6). After that, there seem sno = much scaling I can see with more send streams, though the VM still has more= vcpu and tx queues available.=20 I can see a few things when doing the test, which I appreciate if anyone ca= n provide more insight.=20 1. In those cases with higher number of send streams (>6), I can see more l= ikely some send streams terminated with Broken Pipe error before then full = test time ends. For example, in a test with 10 send streams for 30 seconds,= there could be one to four streams terminated in just a few seconds with B= roken Pipe errors. The same test on Linux guest with same test server, I ha= ve never seen such problem. 2. The driver selects the tx queue based on mbuf's m_pkthdr.flowid field. I= can see each stream get different 4-byte flowid values. However, it is ver= y likely multiple flowids still collide to same tx queue, if we just use al= gorithm like "flowid % number_of_tx_queues" to get the tx queue. Any sugges= tions on how to avoid such case? 3. The tx ring size is 256. I allocate 1024 buf rings for each tx queue to = queue up the send requests. I have seen under heavy tx load the tx queue ha= s to be stopped till more completions are done, however, I have never seen = any drbr queue errors. Does this number look good or need further optimizat= ion? 3. On the tx completion path, a task thread is scheduled for each tx queue= when completion interrupt is received. This thread is not bind to any cpu,= so it can run on any cpu. Is it useful to bind it to a specific cpu? I did= try this but see little difference. Any other ideas are also very welcome. Thanks, Wei