From nobody Fri Feb 02 09:21:46 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TR9Jb2Mjqz58MNV; Fri, 2 Feb 2024 09:21:51 +0000 (UTC) (envelope-from rscheff@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TR9Jb1sCTz4mGT; Fri, 2 Feb 2024 09:21:51 +0000 (UTC) (envelope-from rscheff@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706865711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=mOvaPzYeToomQJOU+lrH7efIQCYNOF7zA2xnSoEvWaI=; b=VR8mq/XW0roKUy0nIca2JSO4Yohp1vvMeCoiulRNKOZhA3fZfj+sadun88VzvZFSVy+Vpj dd3XgtXuYc1WkuVOk8cTerIKdqJArj0JSVLZ/nUuX8ErlyBqM6dOrjoXd3HVLwGVK8AE0k pe0zRjn/9d0QoXjcFKuLR7tbYnUiH5W1F2jvGSAnrAWj4iEqdSWUcwR1+Z19ZGUk476ZvD 9/gu79DlGt+XBMjLDf8mSAP7y+xw6zwKHLnvr8uVWdjcvN01erAM62ZPR14sQGHldNJvtf NlWAqXINJo/kLr4rx/GuAFcDMQe51sXcyaVoRy4w/w6BSzmSEYX4G/PrVHQ6dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706865711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=mOvaPzYeToomQJOU+lrH7efIQCYNOF7zA2xnSoEvWaI=; b=mO7fFfkTcB8PldRS1vsIdumZGSwBY7jtLc7HLl5labbxBGlJ4Fa85liE9v1kkXFfvO51Rq DGW+fzhYf3fFEHEH32Fz1Ujp91NVaP8zxEhpMkwzObDkWnWXVW4CnR35HhssGhskcZf4v6 kjA6rzqn0/9MFM6alYnBmNuoT/Yr6TLlAYHiH+K2IJ89ReyK5XbO5s1dliJUpZ9IdRdveV NWynk4dRIESOe44f0Z6jLF5SNVo/QDSQqgKanyKo6Qeji6aiir4Q1J5chdEnPI4jZcWf2b /woOdLfp9TVwS3XETwxdkgH/fMtPsHbOFB96g9InNfyfxT2ffF4kWXixQ954dQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706865711; a=rsa-sha256; cv=none; b=UG0VuFqZNJnUt6I6C83ddZAjetfbtHyHwL7dw5jXSs491KPkaQeZ6IWTyVS7g2FFKKTLwp 9MaKpybrcquCDlR0VC0LbwasCzlgnKE8xYD83VscgAqK+r94lVOiZMYA097ZRPZX5eqpr3 9RunNUJR4B5oiP0Lmvuk87+dE12Gx+t2No5pHIo4jTJys13ylMl0hrYILopSZRN/pgbzrS nzfDbwfA3EFBw3cwFOQIvulBRVcIRSBcY0mm5LS8erU/jNy2mwFv6fS7IHwtB9Y+D6+Jv6 u9V4aV1gNLS7YGeVru6thQzKLfPyoR1VT0Avmmv84wQg5BvCldmM/9wi4fnUsA== Received: from [192.168.233.122] (unknown [185.236.167.136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: rscheff/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TR9JZ3h3RzXQM; Fri, 2 Feb 2024 09:21:50 +0000 (UTC) (envelope-from rscheff@freebsd.org) Message-ID: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Date: Fri, 2 Feb 2024 10:21:46 +0100 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: "freebsd-net@FreeBSD.org" , FreeBSD Transport , rmacklem@freebsd.org, gallatin@FreeBSD.org, kp@FreeBSD.org From: "Scheffenegger, Richard" Subject: Increasing TCP TSO size support Autocrypt: addr=rscheff@freebsd.org; keydata= xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iziPF0FzrN K1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9yZz7CmgQTFgoAQhYh BDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGABQsJCAcCAyICAQYVCgkICwIE FgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/nvht8kExJ31M+3qpjOqdVypMp+/Ojvh5Z lsk96QEA5HCBkteJcrohwRA7llZvLH3m25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdV AQUBAQdA1Dim8ZWpXRS5i9hb3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2 S7eZrINEWrPNtHEXvliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bA EPqOH+JCIND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8= Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------cFlXxpdNPSvFDnJ2Q7jJvaOb" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------cFlXxpdNPSvFDnJ2Q7jJvaOb Content-Type: multipart/mixed; boundary="------------imiNeDBqBOWYIblw01WKAtFz"; protected-headers="v1" From: "Scheffenegger, Richard" To: "freebsd-net@FreeBSD.org" , FreeBSD Transport , rmacklem@freebsd.org, gallatin@FreeBSD.org, kp@FreeBSD.org Message-ID: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Subject: Increasing TCP TSO size support --------------imiNeDBqBOWYIblw01WKAtFz Content-Type: multipart/mixed; boundary="------------f8bM96TDLFPthBSFcPuWJDK5" --------------f8bM96TDLFPthBSFcPuWJDK5 Content-Type: multipart/alternative; boundary="------------g30wiijxOrV2m8hMPa7cSCrN" --------------g30wiijxOrV2m8hMPa7cSCrN Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 DQpIaSwNCg0KV2UgaGF2ZSBydW4gYSB0ZXN0IGZvciBhIFJQQyB3b3JrbG9hZCB3aXRoIDFN QiBJTyBzaXplcywgYW5kIGNvbGxlY3RlZCANCnRoZSB0Y3BfZGVmYXVsdF9vdXRwdXQoKSBs ZW4oZ3RoKSBkdXJpbmcgdGhlIGZpcnN0IHBhc3MgaW4gdGhlIG91dHB1dCBsb29wLg0KDQpJ biBzdWNoIGEgc2NlbmFyaW8sIHdoZXJlIHRoZSBhcHBsaWNhdGlvbiBmcmVxdWVudGx5IGlu dHJvZHVjZXMgc21hbGwgDQpwYXVzZXMgKHNpbmNlIHRoZSBuZXh0IGxhcmdlIElPIGlzIG9u bHkgc2VudCBhZnRlciB0aGUgY29ycmVzcG9uZGluZyANCnJlcXVlc3QgZnJvbSB0aGUgY2xp ZW50IGhhcyBiZWVuIHJlY2VpdmVkIGFuZCBwcm9jZXNzZWQpIGJldHdlZW4gc2VuZGluZyAN CmFkZGl0aW9uYWwgZGF0YSwgdGhlIGN1cnJlbnQgVFNPIGxpbWl0IG9mIDY0a0IgVFNPIG1h eGltdW0gKDQ1KjE0NDggaW4gDQplZmZlY3QpIHJlcXVpcmVzIG11bHRpcGxlIHBhc3NlcyBp biB0aGUgb3V0cHV0IHJvdXRpbmUgdG8gc2VuZCBhbGwgdGhlIA0KYWxsb3dhYmxlIChjd25k IGxpbWl0ZWQpIGRhdGEuDQoNCkknbGwgdHJ5IHRvIGdldCBhIGRhdGEgY29sbGVjdGlvbiB3 aXRoIGJldHRlciBncmFudWxhcml5IGFib3ZlIDkwIDAwMCANCmJ5dGVzIC0gYnV0IGV2ZW4g aGVyZSB0aGUgYXZlcmFnZSBzdHJvbmdseSBpbmRpY2F0ZXMgdGhhdCBhIG1ham9yaXR5IG9m IA0KdHJhbnNtaXNzaW9uIG9wcG9ydHVuaXRpZXMgYXJlIGluIHRoZSA1MTIga0IgYXJlYSAt IHByb2JhYmx5IGFsc28gaGF2aW5nIA0KdG8gZG8gd2l0aCBMUk8gYW5kIEFDSyB0aGlubmlu ZyBlZmZlY3RzIGJ5IHRoZSBjbGllbnQuDQoNCldpdGggb3RoZXIgd29yZHMsIHRoZSB0Y3Ag b3V0cHV0IGhhcyB0byBydW4gYWJvdXQgOSB0aW1lcyB3aXRoIFRTTywgdG8gDQp0cmFuc21p dCBhbGwgZWxlZ2libGUgZGF0YSAtIGluY3JlYXNpbmcgdGhlIEZyZWVCU0Qgc3VwcG9ydGVk IG1heGltdW0gDQpUU08gc2l6ZSB0byB3aGF0IGN1cnJlbnQgaGFyZHdhcmUgY291bGQgaGFu ZGxlICgyNTZrQi4uMU1CKSB3b3VsZCByZWR1Y2UgDQp0aGUgQ1BVIGJ1cmRlbiBoZXJlLg0K DQoNCklzIGluY3JlYXNpbmcgdGhlIHNvZndhcmUgc3VwcG9ydGVkIFRTTyBzaXplIHRvIGFs bG93IGZvciB3aGF0IHRoZSBOSUNzIA0KY291bGQgbm93YWRheXMgZG8gc29tZXRoaW5nIGFu eW9uZSBhcGFydCBmcm9tIHVzIHdvdWxkIGJlIGludGVyZXN0ZWQgaW4gDQooaW4gcGFydGlj dWxhciwgdGhvc2Ugd2hvIHdvcmsgd2l0aCB0aGUgZHJpdmVycyk/DQoNCg0KQmVzdCByZWdh cmRzLA0KDQogwqAgUmljaGFyZA0KDQoNCg0KDQp0c28gc2l6ZSAodHJhbnNtaXNzaW9ucyA8 IDE0NDggd291bGQgbm90IGJlIGFjY291bnRlZCBoZXJlIGF0IGFsbCkNCg0KIMKgwqDCoCDC oMKgwqAgwqDCoMKgIMKgwqDCoCDCoMKgwqAgIyBjb3VudA0KDQo8MTAwMCAJMA0KPDIwMDAg CTIzDQo8MzAwMCAJMTExDQo8NDAwMCAJNDANCjw1MDAwIAkzMA0KPDcwMDAgCTE0DQo8ODAw MCAJMTM0DQo8OTAwMCAJNDQyDQo8MTAwMDAgCTkzOTYNCjwyMDAwMCAJNDYyMjcNCjwzMDAw MCAJMjU2NDYNCjw0MDAwMCAJMzMwNjANCjw2MDAwMCAJMjMxNjINCjw3MDAwMCAJMjQzNjgN Cjw4MDAwMCAJMTk3NzINCjw5MDAwMCAJNDAxMDENCiA+PTkwMDAwIAk3NTM4NDE2OQ0KQXZl cmFnZTogCTU3ODg0NC40NA0KDQo= --------------g30wiijxOrV2m8hMPa7cSCrN Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


Hi,

We have run a test for a RPC workload with 1MB IO sizes, and collected the tcp_default_output() len(gth) during the first pass in the output loop.

In such a scenario, where the application frequently introduces small pauses (since the next large IO is only sent after the corresponding request from the client has been received and processed) between sending additional data, the current TSO limit of 64kB TSO maximum (45*1448 in effect) requires multiple passes in the output routine to send all the allowable (cwnd limited) data.

I'll try to get a data collection with better granulariy above 90 000 bytes - but even here the average strongly indicates that a majority of transmission opportunities are in the 512 kB area - probably also having to do with LRO and ACK thinning effects by the client.

With other words, the tcp output has to run about 9 times with TSO, to transmit all elegible data - increasing the FreeBSD supported maximum TSO size to what current hardware could handle (256kB..1MB) would reduce the CPU burden here.


Is increasing the sofware supported TSO size to allow for what the NICs could nowadays do something anyone apart from us would be interested in (in particular, those who work with the drivers)?

=


Best regards,

=C2=A0 Richard




tso size (transmissions < 1448 would not be accounted here at all)

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 # count

<1000 0
<2000 23
<3000 111
<4000 40
<5000 30
<7000 14
<8000 134
<9000 442
<10000 9396
<20000 46227
<30000 25646
<40000 33060
<60000 23162
<70000 24368
<80000 19772
<90000 40101
>=3D90000 75384169
Average: 578844.44
--------------g30wiijxOrV2m8hMPa7cSCrN-- --------------f8bM96TDLFPthBSFcPuWJDK5 Content-Type: application/pgp-keys; name="OpenPGP_0x17BE5899E0B1439B.asc" Content-Disposition: attachment; filename="OpenPGP_0x17BE5899E0B1439B.asc" Content-Description: OpenPGP public key Content-Transfer-Encoding: quoted-printable -----BEGIN PGP PUBLIC KEY BLOCK----- xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iz iPF0FzrNK1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9y Zz7CmgQTFgoAQhYhBDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGA BQsJCAcCAyICAQYVCgkICwIEFgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/n vht8kExJ31M+3qpjOqdVypMp+/Ojvh5Zlsk96QEA5HCBkteJcrohwRA7llZvLH3m 25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdVAQUBAQdA1Dim8ZWpXRS5i9hb 3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2S7eZrINEWrPNtHEX vliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bAEPqOH+JC IND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8=3D =3DBwxS -----END PGP PUBLIC KEY BLOCK----- --------------f8bM96TDLFPthBSFcPuWJDK5-- --------------imiNeDBqBOWYIblw01WKAtFz-- --------------cFlXxpdNPSvFDnJ2Q7jJvaOb Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQQ2S7eZrINEWrPNtHEXvliZ4LFDmwUCZby0KgUDAAAAAAAKCRAXvliZ4LFDm73n AQC00+GZ7IDDzaPlORXmsNmMrrKgmFRT/rH6+MIpnbvA8AEA+130h9m/ksyrLjAOFWgXqG68ByH7 +gaZN4mhV5Q13Q0= =QnaZ -----END PGP SIGNATURE----- --------------cFlXxpdNPSvFDnJ2Q7jJvaOb--