From nobody Thu Nov 11 11:31:17 2021 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A13E81859A60; Thu, 11 Nov 2021 11:31:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hqfg94BHYz3CyB; Thu, 11 Nov 2021 11:31:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6F553305F; Thu, 11 Nov 2021 11:31:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1ABBVHNi017373; Thu, 11 Nov 2021 11:31:17 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1ABBVH6s017371; Thu, 11 Nov 2021 11:31:17 GMT (envelope-from git) Date: Thu, 11 Nov 2021 11:31:17 GMT Message-Id: <202111111131.1ABBVH6s017371@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Randall Stewart Subject: git: b8d60729deef - main - tcp: Congestion control cleanup. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: rrs X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b8d60729deefa0bd13e6a395fcab4928e6e10445 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by rrs: URL: https://cgit.FreeBSD.org/src/commit/?id=b8d60729deefa0bd13e6a395fcab4928e6e10445 commit b8d60729deefa0bd13e6a395fcab4928e6e10445 Author: Randall Stewart AuthorDate: 2021-11-11 11:28:18 +0000 Commit: Randall Stewart CommitDate: 2021-11-11 11:28:18 +0000 tcp: Congestion control cleanup. NOTE: HEADS UP read the note below if your kernel config is not including GENERIC!! This patch does a bit of cleanup on TCP congestion control modules. There were some rather interesting surprises that one could get i.e. where you use a socket option to change from one CC (say cc_cubic) to another CC (say cc_vegas) and you could in theory get a memory failure and end up on cc_newreno. This is not what one would expect. The new code fixes this by requiring a cc_data_sz() function so we can malloc with M_WAITOK and pass in to the init function preallocated memory. The CC init is expected in this case *not* to fail but if it does and a module does break the "no fail with memory given" contract we do fall back to the CC that was in place at the time. This also fixes up a set of common newreno utilities that can be shared amongst other CC modules instead of the other CC modules reaching into newreno and executing what they think is a "common and understood" function. Lets put these functions in cc.c and that way we have a common place that is easily findable by future developers or bug fixers. This also allows newreno to evolve and grow support for its features i.e. ABE and HYSTART++ without having to dance through hoops for other CC modules, instead both newreno and the other modules just call into the common functions if they desire that behavior or roll there own if that makes more sense. Note: This commit changes the kernel configuration!! If you are not using GENERIC in some form you must add a CC module option (one of CC_NEWRENO, CC_VEGAS, CC_CUBIC, CC_CDG, CC_CHD, CC_DCTCP, CC_HTCP, CC_HD). You can have more than one defined as well if you desire. Note that if you create a kernel configuration that does not define a congestion control module and includes INET or INET6 the kernel compile will break. Also you need to define a default, generic adds 'options CC_DEFAULT=\"newreno\" but you can specify any string that represents the name of the CC module (same names that show up in the CC module list under net.inet.tcp.cc). If you fail to add the options CC_DEFAULT in your kernel configuration the kernel build will also break. Reviewed by: Michael Tuexen Sponsored by: Netflix Inc. RELNOTES:YES Differential Revision: https://reviews.freebsd.org/D32693 --- UPDATING | 13 ++ share/man/man4/cc_newreno.4 | 52 +++++++ share/man/man4/mod_cc.4 | 50 +++++++ share/man/man9/mod_cc.9 | 75 +++++++++- sys/amd64/conf/GENERIC | 2 + sys/arm/conf/std.armv6 | 2 + sys/arm/conf/std.armv7 | 2 + sys/arm64/conf/std.arm64 | 2 + sys/conf/NOTES | 21 ++- sys/conf/files | 16 +- sys/conf/options | 9 ++ sys/i386/conf/GENERIC | 2 + sys/modules/cc/Makefile | 3 +- sys/modules/cc/cc_newreno/Makefile | 7 + sys/netinet/cc/cc.c | 299 ++++++++++++++++++++++++++++++++----- sys/netinet/cc/cc.h | 34 ++++- sys/netinet/cc/cc_cdg.c | 46 +++--- sys/netinet/cc/cc_chd.c | 41 +++-- sys/netinet/cc/cc_cubic.c | 38 +++-- sys/netinet/cc/cc_dctcp.c | 47 +++--- sys/netinet/cc/cc_hd.c | 24 +-- sys/netinet/cc/cc_htcp.c | 41 +++-- sys/netinet/cc/cc_newreno.c | 141 +++++------------ sys/netinet/cc/cc_vegas.c | 47 +++--- sys/netinet/tcp_subr.c | 79 ++++++---- sys/netinet/tcp_usrreq.c | 151 ++++++++++++++----- sys/powerpc/conf/GENERIC | 2 + sys/riscv/conf/GENERIC | 2 + 28 files changed, 922 insertions(+), 326 deletions(-) diff --git a/UPDATING b/UPDATING index f013826393fb..cde26f7ea030 100644 --- a/UPDATING +++ b/UPDATING @@ -27,6 +27,19 @@ NOTE TO PEOPLE WHO THINK THAT FreeBSD 14.x IS SLOW: world, or to merely disable the most expensive debugging functionality at runtime, run "ln -s 'abort:false,junk:false' /etc/malloc.conf".) +20211110: + Commit xxxxxx changed the TCP congestion control framework so + that any of the included congestion control modules could be + the single module built into the kernel. Previously newreno + was automatically built in through direct reference. Has of + this commit you are required to declare at least one congestion + control module (e.g. 'options CC_NEWRENO') and to also delcare a + default using the CC_DEFAULT option (e.g. options CC_DEFAULT="newreno\"). + The GENERIC configuation includes CC_NEWRENO and defines newreno + as the default. If no congestion control option is built into the + kernel and you are including networking, the kernel compile will + fail. Also if no default is declared the kernel compile will fail. + 20211106: Commit f0c9847a6c47 changed the arguments for VOP_ALLOCATE. The NFS modules must be rebuilt from sources and any out diff --git a/share/man/man4/cc_newreno.4 b/share/man/man4/cc_newreno.4 index 0ac59beeec48..1c4f4a39cb36 100644 --- a/share/man/man4/cc_newreno.4 +++ b/share/man/man4/cc_newreno.4 @@ -75,7 +75,33 @@ the congestion window in response to an ECN congestion signal when .Va net.inet.tcp.cc.abe=1 per: cwnd = (cwnd * CC_NEWRENO_BETA_ECN) / 100. Default is 80. +.It Va CC_NEWRENO_ENABLE_HYSTART +will enable or disable the application of Hystart++. +The current implementation allows the values 0, 1, 2 and 3. +A value of 0 (the default) disables the use of Hystart++. +Setting the value to 1 enables Hystart++. +Setting the value to 2 enables Hystart++ but also will cause, on exit from Hystart++'s CSS, to +set the cwnd to the value of where the increase in RTT first began as +well as setting ssthresh to the flight at send when we exit CSS. +Setting a value of 3 will keep the setting of the cwnd the same as 2, but will cause ssthresh +to be set to the average value between the lowest fas rtt (the value cwnd is +set to) and the fas value at exit of CSS. +.PP +Note that currently the only way to enable +hystart++ is to enable it via socket option. +When enabling it a value of 1 will enable precise internet-draft behavior +(subject to any MIB variable settings), other setting (2 and 3) are experimental. .El +.PP +Note that hystart++ requires the TCP stack be able to call to the congestion +controller with both the +.Va newround +function as well as the +.Va rttsample +function. +Currently the only TCP stacks that provide this feedback to the +congestion controller is rack. +.Pp .Sh MIB Variables The algorithm exposes these variables in the .Va net.inet.tcp.cc.newreno @@ -94,6 +120,32 @@ the congestion window in response to an ECN congestion signal when .Va net.inet.tcp.cc.abe=1 per: cwnd = (cwnd * beta_ecn) / 100. Default is 80. +.It Va hystartplusplus.bblogs +This boolean controls if black box logging will be done for hystart++ events. If set +to zero (the default) no logging is performed. +If set to one then black box logs will be generated on all hystart++ events. +.It Va hystartplusplus.css_rounds +This value controls the number of rounds that CSS runs for. +The default value matches the current internet-draft of 5. +.It Va hystartplusplus.css_growth_div +This value controls the divisor applied to slowstart during CSS. +The default value matches the current internet-draft of 4. +.It Va hystartplusplus.n_rttsamples +This value controls how many rtt samples must be collected in each round for +hystart++ to be active. +The default value matches the current internet-draft of 8. +.It Va hystartplusplus.maxrtt_thresh +This value controls the maximum rtt variance clamp when considering if CSS is needed. +The default value matches the current internet-draft of 16000 (in microseconds). +For further explanation please see the internet-draft. +.It Va hystartplusplus.minrtt_thresh +This value controls the minimum rtt variance clamp when considering if CSS is needed. +The default value matches the current internet-draft of 4000 (in microseconds). +For further explanation please see the internet-draft. +.It Va hystartplusplus.lowcwnd +This value controls what is the lowest congestion window that the tcp +stack must be at before hystart++ engages. +The default value matches the current internet-draft of 16. .El .Sh SEE ALSO .Xr cc_cdg 4 , diff --git a/share/man/man4/mod_cc.4 b/share/man/man4/mod_cc.4 index 273d898cf5e2..84edf0b2a8ed 100644 --- a/share/man/man4/mod_cc.4 +++ b/share/man/man4/mod_cc.4 @@ -67,6 +67,16 @@ socket option (see for details). Callers must pass a pointer to an algorithm specific data, and specify its size. +.Pp +Unloading a congestion control module will fail if it is used as a +default by any Vnet. +When unloading a module, the Vnet default is +used to switch a connection to an alternate congestion control. +Note that the new congestion control module may fail to initialize its +internal memory, if so it will fail the module unload. +If this occurs often times retrying the unload will succeed since the temporary +memory shortage as the new CC module malloc's memory, that prevented the +switch is often transient. .Sh MIB Variables The framework exposes the following variables in the .Va net.inet.tcp.cc @@ -93,6 +103,44 @@ support for ABE and for configuration details. If non-zero, apply standard beta instead of ABE-beta during ECN-signalled congestion recovery episodes if loss also needs to be repaired. .El +.Pp +Each congestion control module may also expose other MIB variables +to control their behaviour. +.Sh Kernel Configuration +.Pp +All of the available congestion control modules may also be loaded +via kernel configutation options. +A kernel configuration is required to have at least one congestion control +algorithm built into it via kernel option and a system default specified. +Compilation of the kernel will fail if these two conditions are not met. +.Sh Kernel Configuration Options +The framework exposes the following kernel configuration options. +.Bl -tag -width ".Va CC_NEWRENO" +.It Va CC_NEWRENO +This directive loads the newreno congestion control algorithm and is included +in GENERIC by default. +.It Va CC_CUBIC +This directive loads the cubic congestion control algorithm. +.It Va CC_VEGAS +This directive loads the vegas congestion control algorithm, note that +this algorithm also requires the TCP_HHOOK option as well. +.It Va CC_CDG +This directive loads the cdg congestion control algorithm, note that +this algorithm also requires the TCP_HHOOK option as well. +.It Va CC_DCTCP +This directive loads the dctcp congestion control algorithm. +.It Va CC_HD +This directive loads the hd congestion control algorithm, note that +this algorithm also requires the TCP_HHOOK option as well. +.It Va CC_CHD +This directive loads the chd congestion control algorithm, note that +this algorithm also requires the TCP_HHOOK option as well. +.It Va CC_HTCP +This directive loads the htcp congestion control algorithm. +.It Va CC_DEFAULT +This directive specifies the string that represents the name of the system default algorithm, the GENERIC kernel +defaults this to newreno. +.El .Sh SEE ALSO .Xr cc_cdg 4 , .Xr cc_chd 4 , @@ -103,6 +151,8 @@ congestion recovery episodes if loss also needs to be repaired. .Xr cc_newreno 4 , .Xr cc_vegas 4 , .Xr tcp 4 , +.Xr config 5 , +.Xr config 8 , .Xr mod_cc 9 .Sh ACKNOWLEDGEMENTS Development and testing of this software were made possible in part by grants diff --git a/share/man/man9/mod_cc.9 b/share/man/man9/mod_cc.9 index 8be6fb2381bd..6bfb1eceed56 100644 --- a/share/man/man9/mod_cc.9 +++ b/share/man/man9/mod_cc.9 @@ -68,7 +68,8 @@ struct cc_algo { char name[TCP_CA_NAME_MAX]; int (*mod_init) (void); int (*mod_destroy) (void); - int (*cb_init) (struct cc_var *ccv); + size_t (*cc_data_sz)(void); + int (*cb_init) (struct cc_var *ccv, void *ptr); void (*cb_destroy) (struct cc_var *ccv); void (*conn_init) (struct cc_var *ccv); void (*ack_received) (struct cc_var *ccv, uint16_t type); @@ -76,6 +77,8 @@ struct cc_algo { void (*post_recovery) (struct cc_var *ccv); void (*after_idle) (struct cc_var *ccv); int (*ctl_output)(struct cc_var *, struct sockopt *, void *); + void (*rttsample)(struct cc_var *, uint32_t, uint32_t, uint32_t); + void (*newround)(struct cc_var *, uint32_t); }; .Ed .Pp @@ -104,6 +107,17 @@ being removed from the kernel. The return value is currently ignored. .Pp The +.Va cc_data_sz +function is called by the socket option code to get the size of +data that the +.Va cb_init +function needs. +The socket option code then preallocates the modules memory so that the +.Va cb_init +function will not fail (the socket option code uses M_WAITOK with +no locks held to do this). +.Pp +The .Va cb_init function is called when a TCP control block .Vt struct tcpcb @@ -114,6 +128,9 @@ Returning a non-zero value from .Va cb_init will cause the connection set up to be aborted, terminating the connection as a result. +Note that the ptr argument passed to the function should be checked to +see if it is non-NULL, if so it is preallocated memory that the cb_init function +must use instead of calling malloc itself. .Pp The .Va cb_destroy @@ -182,6 +199,30 @@ pointer forwarded unmodified from the TCP control, and a pointer to algorithm specific argument. .Pp The +.Va rttsample +function is called to pass round trip time information to the +congestion controller. +The additional arguments to the function include the microsecond RTT +that is being noted, the number of times that the data being +acknowledged was retransmitted as well as the flightsize at send. +For transports that do not track flightsize at send, this variable +will be the current cwnd at the time of the call. +.Pp +The +.Va newround +function is called each time a new round trip time begins. +The montonically increasing round number is also passed to the +congestion controller as well. +This can be used for various purposes by the congestion controller (e.g Hystart++). +.Pp +Note that currently not all TCP stacks call the +.Va rttsample +and +.Va newround +function so dependancy on these functions is also +dependant upon which TCP stack is in use. +.Pp +The .Fn DECLARE_CC_MODULE macro provides a convenient wrapper around the .Xr DECLARE_MODULE 9 @@ -203,8 +244,23 @@ modules must instantiate a .Vt struct cc_algo , but are only required to set the name field, and optionally any of the function pointers. +Note that if a module defines the +.Va cb_init +function it also must define a +.Va cc_data_sz +function. +This is because when switching from one congestion control +module to another the socket option code will preallocate memory for the +.Va cb_init +function. If no memory is allocated by the modules +.Va cb_init +then the +.Va cc_data_sz +function should return 0. +.Pp The stack will skip calling any function pointer which is NULL, so there is no -requirement to implement any of the function pointers. +requirement to implement any of the function pointers (with the exception of +the cb_init <-> cc_data_sz dependancy noted above). Using the C99 designated initialiser feature to set fields is encouraged. .Pp Each function pointer which deals with congestion control state is passed a @@ -222,6 +278,8 @@ struct cc_var { struct tcpcb *tcp; struct sctp_nets *sctp; } ccvc; + uint16_t nsegs; + uint8_t labc; }; .Ed .Pp @@ -305,6 +363,19 @@ and is set when the connection's ability to send data is currently constrained by the value of the congestion window. Algorithms should use the absence of this flag being set to avoid accumulating a large difference between the congestion window and send window. +.Pp +The +.Va nsegs +variable is used to pass in how much compression was done by the local +LRO system. +So for example if LRO pushed three in-order acknowledgements into +one acknowledgement the variable would be set to three. +.Pp +The +.Va labc +variable is used in conjunction with the CCF_USE_LOCAL_ABC flag +to override what labc variable the congestion controller will use +for this particular acknowledgement. .Sh SEE ALSO .Xr cc_cdg 4 , .Xr cc_chd 4 , diff --git a/sys/amd64/conf/GENERIC b/sys/amd64/conf/GENERIC index f47b995beb2c..eb2b0cbd12e9 100644 --- a/sys/amd64/conf/GENERIC +++ b/sys/amd64/conf/GENERIC @@ -30,6 +30,8 @@ options PREEMPTION # Enable kernel thread preemption options VIMAGE # Subsystem virtualization, e.g. VNET options INET # InterNETworking options INET6 # IPv6 communications protocols +options CC_NEWRENO # include newreno congestion control +options CC_DEFAULT=\"newreno\" # define our default CC module it should be compiled in. options IPSEC_SUPPORT # Allow kldload of ipsec and tcpmd5 options ROUTE_MPATH # Multipath routing support options FIB_ALGO # Modular fib lookups diff --git a/sys/arm/conf/std.armv6 b/sys/arm/conf/std.armv6 index 5f7fb1ebb221..cdfb755eb9c2 100644 --- a/sys/arm/conf/std.armv6 +++ b/sys/arm/conf/std.armv6 @@ -8,6 +8,8 @@ options PREEMPTION # Enable kernel thread preemption options VIMAGE # Subsystem virtualization, e.g. VNET options INET # InterNETworking options INET6 # IPv6 communications protocols +options CC_NEWRENO # include newreno congestion control +options CC_DEFAULT=\"newreno\" # define our default CC module it should be compiled in. options TCP_HHOOK # hhook(9) framework for TCP device crypto # core crypto support options IPSEC_SUPPORT # Allow kldload of ipsec and tcpmd5 diff --git a/sys/arm/conf/std.armv7 b/sys/arm/conf/std.armv7 index fda291df289a..7512bbaa2939 100644 --- a/sys/arm/conf/std.armv7 +++ b/sys/arm/conf/std.armv7 @@ -8,6 +8,8 @@ options PREEMPTION # Enable kernel thread preemption options VIMAGE # Subsystem virtualization, e.g. VNET options INET # InterNETworking options INET6 # IPv6 communications protocols +options CC_NEWRENO # include newreno congestion control +options CC_DEFAULT=\"newreno\" # define our default CC module it should be compiled in. options TCP_HHOOK # hhook(9) framework for TCP device crypto # core crypto support options IPSEC_SUPPORT # Allow kldload of ipsec and tcpmd5 diff --git a/sys/arm64/conf/std.arm64 b/sys/arm64/conf/std.arm64 index 5c26f19b4299..599fd2aebeff 100644 --- a/sys/arm64/conf/std.arm64 +++ b/sys/arm64/conf/std.arm64 @@ -11,6 +11,8 @@ options PREEMPTION # Enable kernel thread preemption options VIMAGE # Subsystem virtualization, e.g. VNET options INET # InterNETworking options INET6 # IPv6 communications protocols +options CC_NEWRENO # include newreno congestion control +options CC_DEFAULT=\"newreno\" # define our default CC module it should be compiled in. options IPSEC_SUPPORT # Allow kldload of ipsec and tcpmd5 options ROUTE_MPATH # Multipath routing support options FIB_ALGO # Modular fib lookups diff --git a/sys/conf/NOTES b/sys/conf/NOTES index 35662db3ef27..a1f466a7a8b5 100644 --- a/sys/conf/NOTES +++ b/sys/conf/NOTES @@ -646,7 +646,26 @@ options HWPMC_HOOKS # Other necessary kernel hooks # options INET #Internet communications protocols options INET6 #IPv6 communications protocols - +# +# Note if you include INET/INET6 or both options +# You *must* define at least one of the congestion control +# options or the compile will fail. Generic defines +# options CC_NEWRENO. You also will need to specify +# a default or the compile of your kernel will fail +# as well. The string in default is the name of the +# cc module as it would appear in the sysctl for +# setting the default. Generic defines newreno +# as shown below. +# +options CC_CDG +options CC_CHD +options CC_CUBIC +options CC_DCTCP +options CC_HD +options CC_HTCP +options CC_NEWRENO +options CC_VEGAS +options CC_DEFAULT=\"newreno\" options RATELIMIT # TX rate limiting support options ROUTETABLES=2 # allocated fibs up to 65536. default is 1. diff --git a/sys/conf/files b/sys/conf/files index e1ee0e9fc9ef..30f98817e290 100644 --- a/sys/conf/files +++ b/sys/conf/files @@ -4351,8 +4351,20 @@ netinet/ip_options.c optional inet netinet/ip_output.c optional inet netinet/ip_reass.c optional inet netinet/raw_ip.c optional inet | inet6 -netinet/cc/cc.c optional inet | inet6 -netinet/cc/cc_newreno.c optional inet | inet6 +netinet/cc/cc.c optional cc_newreno inet | cc_vegas inet | \ + cc_htcp inet | cc_hd inet | cc_dctcp inet | cc_cubic inet | \ + cc_chd inet | cc_cdg inet | cc_newreno inet6 | cc_vegas inet6 | \ + cc_htcp inet6 | cc_hd inet6 |cc_dctcp inet6 | cc_cubic inet6 | \ + cc_chd inet6 | cc_cdg inet6 +netinet/cc/cc_cdg.c optional inet cc_cdg tcp_hhook +netinet/cc/cc_chd.c optional inet cc_chd tcp_hhook +netinet/cc/cc_cubic.c optional inet cc_cubic | inet6 cc_cubic +netinet/cc/cc_dctcp.c optional inet cc_dctcp | inet6 cc_dctcp +netinet/cc/cc_hd.c optional inet cc_hd tcp_hhook +netinet/cc/cc_htcp.c optional inet cc_htcp | inet6 cc_htcp +netinet/cc/cc_newreno.c optional inet cc_newreno | inet6 cc_newreno +netinet/cc/cc_vegas.c optional inet cc_vegas tcp_hhook +netinet/khelp/h_ertt.c optional inet tcp_hhook netinet/sctp_asconf.c optional inet sctp | inet6 sctp netinet/sctp_auth.c optional inet sctp | inet6 sctp netinet/sctp_bsd_addr.c optional inet sctp | inet6 sctp diff --git a/sys/conf/options b/sys/conf/options index 2d99dc8c67db..a103ab85c1b9 100644 --- a/sys/conf/options +++ b/sys/conf/options @@ -81,6 +81,15 @@ BOOTVERBOSE opt_global.h CALLOUT_PROFILING CAPABILITIES opt_capsicum.h CAPABILITY_MODE opt_capsicum.h +CC_CDG opt_global.h +CC_CHD opt_global.h +CC_CUBIC opt_global.h +CC_DEFAULT opt_cc.h +CC_DCTCP opt_global.h +CC_HD opt_global.h +CC_HTCP opt_global.h +CC_NEWRENO opt_global.h +CC_VEGAS opt_global.h COMPAT_43 opt_global.h COMPAT_43TTY opt_global.h COMPAT_FREEBSD4 opt_global.h diff --git a/sys/i386/conf/GENERIC b/sys/i386/conf/GENERIC index 5447c452c4f7..3b5556675555 100644 --- a/sys/i386/conf/GENERIC +++ b/sys/i386/conf/GENERIC @@ -31,6 +31,8 @@ options PREEMPTION # Enable kernel thread preemption options VIMAGE # Subsystem virtualization, e.g. VNET options INET # InterNETworking options INET6 # IPv6 communications protocols +options CC_NEWRENO # include newreno congestion control +options CC_DEFAULT=\"newreno\" # define our default CC module it should be compiled in. options IPSEC_SUPPORT # Allow kldload of ipsec and tcpmd5 options ROUTE_MPATH # Multipath routing support options TCP_HHOOK # hhook(9) framework for TCP diff --git a/sys/modules/cc/Makefile b/sys/modules/cc/Makefile index ec89d89e8c80..3f7110024722 100644 --- a/sys/modules/cc/Makefile +++ b/sys/modules/cc/Makefile @@ -1,6 +1,7 @@ # $FreeBSD$ -SUBDIR= cc_cubic \ +SUBDIR= cc_newreno \ + cc_cubic \ cc_dctcp \ cc_htcp diff --git a/sys/modules/cc/cc_newreno/Makefile b/sys/modules/cc/cc_newreno/Makefile new file mode 100644 index 000000000000..8d9b82ad2a9e --- /dev/null +++ b/sys/modules/cc/cc_newreno/Makefile @@ -0,0 +1,7 @@ +# $FreeBSD$ + +.PATH: ${SRCTOP}/sys/netinet/cc +KMOD= cc_newreno +SRCS= cc_newreno.c + +.include diff --git a/sys/netinet/cc/cc.c b/sys/netinet/cc/cc.c index 0d60b64dbe32..0a61aff37c96 100644 --- a/sys/netinet/cc/cc.c +++ b/sys/netinet/cc/cc.c @@ -50,7 +50,7 @@ #include __FBSDID("$FreeBSD$"); - +#include #include #include #include @@ -70,11 +70,15 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include +#include +#include #include - #include +MALLOC_DEFINE(M_CC_MEM, "CC Mem", "Congestion Control State memory"); + /* * List of available cc algorithms on the current system. First element * is used as the system default CC algorithm. @@ -84,7 +88,10 @@ struct cc_head cc_list = STAILQ_HEAD_INITIALIZER(cc_list); /* Protects the cc_list TAILQ. */ struct rwlock cc_list_lock; -VNET_DEFINE(struct cc_algo *, default_cc_ptr) = &newreno_cc_algo; +VNET_DEFINE(struct cc_algo *, default_cc_ptr) = NULL; + +VNET_DEFINE(uint32_t, newreno_beta) = 50; +#define V_newreno_beta VNET(newreno_beta) /* * Sysctl handler to show and change the default CC algorithm. @@ -98,7 +105,10 @@ cc_default_algo(SYSCTL_HANDLER_ARGS) /* Get the current default: */ CC_LIST_RLOCK(); - strlcpy(default_cc, CC_DEFAULT()->name, sizeof(default_cc)); + if (CC_DEFAULT_ALGO() != NULL) + strlcpy(default_cc, CC_DEFAULT_ALGO()->name, sizeof(default_cc)); + else + memset(default_cc, 0, TCP_CA_NAME_MAX); CC_LIST_RUNLOCK(); error = sysctl_handle_string(oidp, default_cc, sizeof(default_cc), req); @@ -108,7 +118,6 @@ cc_default_algo(SYSCTL_HANDLER_ARGS) goto done; error = ESRCH; - /* Find algo with specified name and set it to default. */ CC_LIST_RLOCK(); STAILQ_FOREACH(funcs, &cc_list, entries) { @@ -141,7 +150,9 @@ cc_list_available(SYSCTL_HANDLER_ARGS) nalgos++; } CC_LIST_RUNLOCK(); - + if (nalgos == 0) { + return (ENOENT); + } s = sbuf_new(NULL, NULL, nalgos * TCP_CA_NAME_MAX, SBUF_FIXEDLEN); if (s == NULL) @@ -176,12 +187,13 @@ cc_list_available(SYSCTL_HANDLER_ARGS) } /* - * Reset the default CC algo to NewReno for any netstack which is using the algo - * that is about to go away as its default. + * Return the number of times a proposed removal_cc is + * being used as the default. */ -static void -cc_checkreset_default(struct cc_algo *remove_cc) +static int +cc_check_default(struct cc_algo *remove_cc) { + int cnt = 0; VNET_ITERATOR_DECL(vnet_iter); CC_LIST_LOCK_ASSERT(); @@ -189,12 +201,16 @@ cc_checkreset_default(struct cc_algo *remove_cc) VNET_LIST_RLOCK_NOSLEEP(); VNET_FOREACH(vnet_iter) { CURVNET_SET(vnet_iter); - if (strncmp(CC_DEFAULT()->name, remove_cc->name, - TCP_CA_NAME_MAX) == 0) - V_default_cc_ptr = &newreno_cc_algo; + if ((CC_DEFAULT_ALGO() != NULL) && + strncmp(CC_DEFAULT_ALGO()->name, + remove_cc->name, + TCP_CA_NAME_MAX) == 0) { + cnt++; + } CURVNET_RESTORE(); } VNET_LIST_RUNLOCK_NOSLEEP(); + return (cnt); } /* @@ -218,31 +234,36 @@ cc_deregister_algo(struct cc_algo *remove_cc) err = ENOENT; - /* Never allow newreno to be deregistered. */ - if (&newreno_cc_algo == remove_cc) - return (EPERM); - /* Remove algo from cc_list so that new connections can't use it. */ CC_LIST_WLOCK(); STAILQ_FOREACH_SAFE(funcs, &cc_list, entries, tmpfuncs) { if (funcs == remove_cc) { - cc_checkreset_default(remove_cc); - STAILQ_REMOVE(&cc_list, funcs, cc_algo, entries); - err = 0; + if (cc_check_default(remove_cc)) { + err = EBUSY; + break; + } + /* Add a temp flag to stop new adds to it */ + funcs->flags |= CC_MODULE_BEING_REMOVED; + break; + } + } + CC_LIST_WUNLOCK(); + err = tcp_ccalgounload(remove_cc); + /* + * Now back through and we either remove the temp flag + * or pull the registration. + */ + CC_LIST_WLOCK(); + STAILQ_FOREACH_SAFE(funcs, &cc_list, entries, tmpfuncs) { + if (funcs == remove_cc) { + if (err == 0) + STAILQ_REMOVE(&cc_list, funcs, cc_algo, entries); + else + funcs->flags &= ~CC_MODULE_BEING_REMOVED; break; } } CC_LIST_WUNLOCK(); - - if (!err) - /* - * XXXLAS: - * - We may need to handle non-zero return values in future. - * - If we add CC framework support for protocols other than - * TCP, we may want a more generic way to handle this step. - */ - tcp_ccalgounload(remove_cc); - return (err); } @@ -263,19 +284,218 @@ cc_register_algo(struct cc_algo *add_cc) */ CC_LIST_WLOCK(); STAILQ_FOREACH(funcs, &cc_list, entries) { - if (funcs == add_cc || strncmp(funcs->name, add_cc->name, - TCP_CA_NAME_MAX) == 0) + if (funcs == add_cc || + strncmp(funcs->name, add_cc->name, + TCP_CA_NAME_MAX) == 0) { err = EEXIST; + break; + } } - - if (!err) + /* + * The first loaded congestion control module will become + * the default until we find the "CC_DEFAULT" defined in + * the config (if we do). + */ + if (!err) { STAILQ_INSERT_TAIL(&cc_list, add_cc, entries); - + if (strcmp(add_cc->name, CC_DEFAULT) == 0) { + V_default_cc_ptr = add_cc; + } else if (V_default_cc_ptr == NULL) { + V_default_cc_ptr = add_cc; + } + } CC_LIST_WUNLOCK(); return (err); } +/* + * Perform any necessary tasks before we exit congestion recovery. + */ +void +newreno_cc_post_recovery(struct cc_var *ccv) +{ + int pipe; + + if (IN_FASTRECOVERY(CCV(ccv, t_flags))) { + /* + * Fast recovery will conclude after returning from this + * function. Window inflation should have left us with + * approximately snd_ssthresh outstanding data. But in case we + * would be inclined to send a burst, better to do it via the + * slow start mechanism. + * + * XXXLAS: Find a way to do this without needing curack + */ + if (V_tcp_do_newsack) + pipe = tcp_compute_pipe(ccv->ccvc.tcp); + else + pipe = CCV(ccv, snd_max) - ccv->curack; + if (pipe < CCV(ccv, snd_ssthresh)) + /* + * Ensure that cwnd does not collapse to 1 MSS under + * adverse conditons. Implements RFC6582 + */ + CCV(ccv, snd_cwnd) = max(pipe, CCV(ccv, t_maxseg)) + + CCV(ccv, t_maxseg); + else + CCV(ccv, snd_cwnd) = CCV(ccv, snd_ssthresh); + } +} + +void +newreno_cc_after_idle(struct cc_var *ccv) +{ + uint32_t rw; + /* + * If we've been idle for more than one retransmit timeout the old + * congestion window is no longer current and we have to reduce it to + * the restart window before we can transmit again. + * + * The restart window is the initial window or the last CWND, whichever + * is smaller. + * + * This is done to prevent us from flooding the path with a full CWND at + * wirespeed, overloading router and switch buffers along the way. + * + * See RFC5681 Section 4.1. "Restarting Idle Connections". + * + * In addition, per RFC2861 Section 2, the ssthresh is set to the + * maximum of the former ssthresh or 3/4 of the old cwnd, to + * not exit slow-start prematurely. + */ + rw = tcp_compute_initwnd(tcp_maxseg(ccv->ccvc.tcp)); + + CCV(ccv, snd_ssthresh) = max(CCV(ccv, snd_ssthresh), + CCV(ccv, snd_cwnd)-(CCV(ccv, snd_cwnd)>>2)); + + CCV(ccv, snd_cwnd) = min(rw, CCV(ccv, snd_cwnd)); +} + +/* + * Perform any necessary tasks before we enter congestion recovery. + */ +void +newreno_cc_cong_signal(struct cc_var *ccv, uint32_t type) +{ + uint32_t cwin, factor; + u_int mss; + + cwin = CCV(ccv, snd_cwnd); + mss = tcp_fixed_maxseg(ccv->ccvc.tcp); + /* + * Other TCP congestion controls use newreno_cong_signal(), but + * with their own private cc_data. Make sure the cc_data is used + * correctly. + */ + factor = V_newreno_beta; + + /* Catch algos which mistakenly leak private signal types. */ + KASSERT((type & CC_SIGPRIVMASK) == 0, + ("%s: congestion signal type 0x%08x is private\n", __func__, type)); + + cwin = max(((uint64_t)cwin * (uint64_t)factor) / (100ULL * (uint64_t)mss), + 2) * mss; + + switch (type) { + case CC_NDUPACK: + if (!IN_FASTRECOVERY(CCV(ccv, t_flags))) { + if (!IN_CONGRECOVERY(CCV(ccv, t_flags))) + CCV(ccv, snd_ssthresh) = cwin; + ENTER_RECOVERY(CCV(ccv, t_flags)); + } + break; + case CC_ECN: + if (!IN_CONGRECOVERY(CCV(ccv, t_flags))) { + CCV(ccv, snd_ssthresh) = cwin; + CCV(ccv, snd_cwnd) = cwin; + ENTER_CONGRECOVERY(CCV(ccv, t_flags)); + } + break; + case CC_RTO: + CCV(ccv, snd_ssthresh) = max(min(CCV(ccv, snd_wnd), + CCV(ccv, snd_cwnd)) / 2 / mss, + 2) * mss; + CCV(ccv, snd_cwnd) = mss; + break; + } +} + +void +newreno_cc_ack_received(struct cc_var *ccv, uint16_t type) +{ + if (type == CC_ACK && !IN_RECOVERY(CCV(ccv, t_flags)) && + (ccv->flags & CCF_CWND_LIMITED)) { + u_int cw = CCV(ccv, snd_cwnd); + u_int incr = CCV(ccv, t_maxseg); + + /* + * Regular in-order ACK, open the congestion window. + * Method depends on which congestion control state we're + * in (slow start or cong avoid) and if ABC (RFC 3465) is + * enabled. + * + * slow start: cwnd <= ssthresh + * cong avoid: cwnd > ssthresh + * + * slow start and ABC (RFC 3465): + * Grow cwnd exponentially by the amount of data + * ACKed capping the max increment per ACK to + * (abc_l_var * maxseg) bytes. + * + * slow start without ABC (RFC 5681): + * Grow cwnd exponentially by maxseg per ACK. + * + * cong avoid and ABC (RFC 3465): + * Grow cwnd linearly by maxseg per RTT for each + * cwnd worth of ACKed data. + * + * cong avoid without ABC (RFC 5681): + * Grow cwnd linearly by approximately maxseg per RTT using + * maxseg^2 / cwnd per ACK as the increment. + * If cwnd > maxseg^2, fix the cwnd increment at 1 byte to + * avoid capping cwnd. + */ + if (cw > CCV(ccv, snd_ssthresh)) { + if (V_tcp_do_rfc3465) { + if (ccv->flags & CCF_ABC_SENTAWND) + ccv->flags &= ~CCF_ABC_SENTAWND; + else + incr = 0; + } else + incr = max((incr * incr / cw), 1); + } else if (V_tcp_do_rfc3465) { + /* + * In slow-start with ABC enabled and no RTO in sight? + * (Must not use abc_l_var > 1 if slow starting after + * an RTO. On RTO, snd_nxt = snd_una, so the + * snd_nxt == snd_max check is sufficient to + * handle this). + * + * XXXLAS: Find a way to signal SS after RTO that + * doesn't rely on tcpcb vars. + */ + uint16_t abc_val; + + if (ccv->flags & CCF_USE_LOCAL_ABC) + abc_val = ccv->labc; + else + abc_val = V_tcp_abc_l_var; + if (CCV(ccv, snd_nxt) == CCV(ccv, snd_max)) + incr = min(ccv->bytes_this_ack, + ccv->nsegs * abc_val * + CCV(ccv, t_maxseg)); + else + incr = min(ccv->bytes_this_ack, CCV(ccv, t_maxseg)); + + } + /* ABC is on by default, so incr equals 0 frequently. */ + if (incr > 0) + CCV(ccv, snd_cwnd) = min(cw + incr, + TCP_MAXWIN << CCV(ccv, snd_scale)); + } +} + /* * Handles kld related events. Returns 0 on success, non-zero on failure. */ @@ -290,6 +510,15 @@ cc_modevent(module_t mod, int event_type, void *data) switch(event_type) { case MOD_LOAD: + if ((algo->cc_data_sz == NULL) && (algo->cb_init != NULL)) { + /* + * A module must have a cc_data_sz function + * even if it has no data it should return 0. + */ + printf("Module Load Fails, it lacks a cc_data_sz() function but has a cb_init()!\n"); + err = EINVAL; + break; + } if (algo->mod_init != NULL) err = algo->mod_init(); if (!err) diff --git a/sys/netinet/cc/cc.h b/sys/netinet/cc/cc.h index e5fc328cc0c2..4282f0585bbd 100644 --- a/sys/netinet/cc/cc.h +++ b/sys/netinet/cc/cc.h @@ -53,10 +53,11 @@ #ifdef _KERNEL +MALLOC_DECLARE(M_CC_MEM); + /* Global CC vars. */ extern STAILQ_HEAD(cc_head, cc_algo) cc_list; extern const int tcprexmtthresh; -extern struct cc_algo newreno_cc_algo; /* Per-netstack bits. */ VNET_DECLARE(struct cc_algo *, default_cc_ptr); @@ -139,8 +140,19 @@ struct cc_algo { /* Cleanup global module state on kldunload. */ int (*mod_destroy)(void); - /* Init CC state for a new control block. */ - int (*cb_init)(struct cc_var *ccv); + /* Return the size of the void pointer the CC needs for state */ + size_t (*cc_data_sz)(void); + + /* + * Init CC state for a new control block. The CC + * module may be passed a NULL ptr indicating that *** 1429 LINES SKIPPED ***