Patch to set TCP_NOPUSH on libfetch HTTP connections
Kelly Yancey
kelly at nttmcl.com
Fri Feb 11 13:34:25 PST 2005
I recently noticed that libfetch often splits HTTP GET requests across
multiple packets as a result of calling writev() for each line of the
header. A simple request ends up on the wire as:
13:19:42.647461 216.69.71.45.1390 > 216.69.64.149.80: S 1287054177:1287054177(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 87600634 0> (DF)
0x0000 4500 003c a6c1 4000 4006 5bad d845 472d E..<.. at .@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e361 0000 0000 .E at ..n.PL..a....
0x0020 a002 e000 49b1 0000 0204 05b4 0103 0300 ....I...........
0x0030 0101 080a 0538 adfa 0000 0000 .....8......
13:19:42.653735 216.69.64.149.80 > 216.69.71.45.1390: S 2666070983:2666070983(0) ack 1287054178 win 17376 <mss 1460,nop,wscale 0,nop,nop,timestamp 747421059 87600634> (DF)
0x0000 4500 003c 7647 4000 3c06 9027 d845 4095 E..<vG at .<..'.E at .
0x0010 d845 472d 0050 056e 9ee9 03c7 4cb6 e362 .EG-.P.n....L..b
0x0020 a012 43e0 5900 0000 0204 05b4 0103 0300 ..C.Y...........
0x0030 0101 080a 2c8c bd83 0538 adfa ....,....8..
13:19:42.653800 216.69.71.45.1390 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87600635 747421059> (DF)
0x0000 4500 0034 a6c2 4000 4006 5bb4 d845 472d E..4.. at .@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e362 9ee9 03c8 .E at ..n.PL..b....
0x0020 8010 e240 e662 0000 0101 080a 0538 adfb ... at .b.......8..
0x0030 2c8c bd83 ,...
13:19:42.654342 216.69.71.45.1390 > 216.69.64.149.80: P 1:33(32) ack 1 win 57920 <nop,nop,timestamp 87600635 747421059> (DF)
0x0000 4500 0054 a6c3 4000 4006 5b93 d845 472d E..T.. at .@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e362 9ee9 03c8 .E at ..n.PL..b....
0x0020 8018 e240 e42b 0000 0101 080a 0538 adfb ... at .+.......8..
0x0030 2c8c bd83 4745 5420 2f6e 6f6e 6578 6973 ,...GET./nonexis
0x0040 7465 6e74 2e68 746d 6c20 4854 5450 2f31 tent.html.HTTP/1
0x0050 2e31 0d0a .1..
13:19:42.851075 216.69.64.149.80 > 216.69.71.45.1390: . ack 33 win 17376 <nop,nop,timestamp 747421059 87600635> (DF)
0x0000 4500 0034 640f 4000 3c06 a267 d845 4095 E..4d. at .<..g.E at .
0x0010 d845 472d 0050 056e 9ee9 03c8 4cb6 e382 .EG-.P.n....L...
0x0020 8010 43e0 84a3 0000 0101 080a 2c8c bd83 ..C.........,...
0x0030 0538 adfb .8..
13:19:42.851127 216.69.71.45.1390 > 216.69.64.149.80: P 33:108(75) ack 1 win 57920 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 007f a6c4 4000 4006 5b67 d845 472d E..... at .@.[g.EG-
0x0010 d845 4095 056e 0050 4cb6 e382 9ee9 03c8 .E at ..n.PL.......
0x0020 8018 e240 51b2 0000 0101 080a 0538 ae0f ... at Q........8..
0x0030 2c8c bd83 486f 7374 3a20 7777 772e 6e74 ,...Host:.www.nt
0x0040 746d 636c 2e63 6f6d 0d0a 5573 6572 2d41 tmcl.com..User-A
0x0050 6765 6e74 3a20 6665 7463 6820 6c69 6266 gent:.fetch.libf
0x0060 6574 6368 2f32 2e30 0d0a 436f 6e6e 6563 etch/2.0..Connec
0x0070 7469 6f6e 3a20 636c 6f73 650d 0a0d 0a tion:.close....
13:19:42.857163 216.69.64.149.80 > 216.69.71.45.1390: F 514:514(0) ack 108 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
0x0000 4500 0034 323a 4000 3c06 d43c d845 4095 E..42:@.<..<.E at .
0x0010 d845 472d 0050 056e 9ee9 05c9 4cb6 e3cd .EG-.P.n....L...
0x0020 8011 43e0 8242 0000 0101 080a 2c8c bd83 ..C..B......,...
0x0030 0538 ae0f .8..
13:19:42.857180 216.69.71.45.1390 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 0034 a6c5 4000 4006 5bb1 d845 472d E..4.. at .@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e3cd 9ee9 03c8 .E at ..n.PL.......
0x0020 8010 e240 e5e3 0000 0101 080a 0538 ae0f ... at .........8..
0x0030 2c8c bd83 ,...
13:19:42.859754 216.69.64.149.80 > 216.69.71.45.1390: P 1:514(513) ack 108 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
[ snip file contents ]
13:19:42.859773 216.69.71.45.1390 > 216.69.64.149.80: . ack 515 win 57407 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 0034 a6c6 4000 4006 5bb0 d845 472d E..4.. at .@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e3cd 9ee9 05ca .E at ..n.PL.......
0x0020 8010 e03f e5e2 0000 0101 080a 0538 ae0f ...?.........8..
0x0030 2c8c bd83 ,...
13:19:42.860070 216.69.71.45.1390 > 216.69.64.149.80: F 108:108(0) ack 515 win 57920 <nop,nop,timestamp 87600655 747421059> (DF)
0x0000 4500 0034 a6c7 4000 4006 5baf d845 472d E..4.. at .@.[..EG-
0x0010 d845 4095 056e 0050 4cb6 e3cd 9ee9 05ca .E at ..n.PL.......
0x0020 8011 e240 e3e0 0000 0101 080a 0538 ae0f ... at .........8..
0x0030 2c8c bd83 ,...
13:19:42.865699 216.69.64.149.80 > 216.69.71.45.1390: . ack 109 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
0x0000 4500 0034 55a4 4000 3c06 b0d2 d845 4095 E..4U. at .<....E at .
0x0010 d845 472d 0050 056e 9ee9 05ca 4cb6 e3ce .EG-.P.n....L...
0x0020 8010 43e0 8241 0000 0101 080a 2c8c bd83 ..C..A......,...
0x0030 0538 ae0f .8..
13:19:42.871415 216.69.64.149.80 > 216.69.71.45.1390: . ack 109 win 17376 <nop,nop,timestamp 747421059 87600655> (DF)
0x0000 4500 0034 069a 4000 3c06 ffdc d845 4095 E..4.. at .<....E at .
0x0010 d845 472d 0050 056e 9ee9 05ca 4cb6 e3ce .EG-.P.n....L...
0x0020 8010 43e0 8241 0000 0101 080a 2c8c bd83 ..C..A......,...
0x0030 0538 ae0f .8..
13:19:42.871449 216.69.71.45.1390 > 216.69.64.149.80: R 1287054286:1287054286(0) win 0
0x0000 4500 0028 a6c8 0000 4006 9bba d845 472d E..(.... at ....EG-
0x0010 d845 4095 056e 0050 4cb6 e3ce 0000 0000 .E at ..n.PL.......
0x0020 5004 0000 4150 0000 P...AP..
The attached patch sets the TCP_NOPUSH option on the socket and uses
shutdown(conn->sd, SHUT_WR) at the end of the HTTP request in order to
force the entire HTTP request to be coelesced into a minimum number of
packets. With the attached patch applied, the same request shown above
appears on the wire as:
13:17:10.659049 216.69.71.45.2218 > 216.69.64.149.80: S 2067322044:2067322044(0) win 57344 <mss 1460,nop,wscale 0,nop,nop,timestamp 87585436 0> (DF)
0x0000 4500 003c 9c27 4000 4006 6647 d845 472d E..<.'@. at .fG.EG-
0x0010 d845 4095 08aa 0050 7b38 d4bc 0000 0000 .E at ....P{8......
0x0020 a002 e000 61f6 0000 0204 05b4 0103 0300 ....a...........
0x0030 0101 080a 0538 729c 0000 0000 .....8r.....
13:17:10.663461 216.69.64.149.80 > 216.69.71.45.2218: S 3505347452:3505347452(0) ack 2067322045 win 17376 <mss 1460,nop,wscale 0,nop,nop,timestamp 747420755 87585436> (DF)
0x0000 4500 003c da68 4000 3c06 2c06 d845 4095 E..<.h at .<.,..E at .
0x0010 d845 472d 0050 08aa d0ef 5b7c 7b38 d4bd .EG-.P....[|{8..
0x0020 a012 43e0 e8b9 0000 0204 05b4 0103 0300 ..C.............
0x0030 0101 080a 2c8c bc53 0538 729c ....,..S.8r.
13:17:10.663510 216.69.71.45.2218 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87585436 747420755> (DF)
0x0000 4500 0034 9c28 4000 4006 664e d845 472d E..4.(@. at .fN.EG-
0x0010 d845 4095 08aa 0050 7b38 d4bd d0ef 5b7d .E at ....P{8....[}
0x0020 8010 e240 761d 0000 0101 080a 0538 729c ... at v........8r.
0x0030 2c8c bc53 ,..S
13:17:10.664197 216.69.71.45.2218 > 216.69.64.149.80: FP 1:108(107) ack 1 win 57920 <nop,nop,timestamp 87585436 747420755> (DF)
0x0000 4500 009f 9c29 4000 4006 65e2 d845 472d E....)@. at .e..EG-
0x0010 d845 4095 08aa 0050 7b38 d4bd d0ef 5b7d .E at ....P{8....[}
0x0020 8019 e240 df70 0000 0101 080a 0538 729c ... at .p.......8r.
0x0030 2c8c bc53 4745 5420 2f6e 6f6e 6578 6973 ,..SGET./nonexis
0x0040 7465 6e74 2e68 746d 6c20 4854 5450 2f31 tent.html.HTTP/1
0x0050 2e31 0d0a 486f 7374 3a20 7777 772e 6e74 .1..Host:.www.nt
0x0060 746d 636c 2e63 6f6d 0d0a 5573 6572 2d41 tmcl.com..User-A
0x0070 6765 6e74 3a20 6665 7463 6820 6c69 6266 gent:.fetch.libf
0x0080 6574 6368 2f32 2e30 0d0a 436f 6e6e 6563 etch/2.0..Connec
0x0090 7469 6f6e 3a20 636c 6f73 650d 0a0d 0a tion:.close....
13:17:10.669275 216.69.64.149.80 > 216.69.71.45.2218: . ack 109 win 17269 <nop,nop,timestamp 747420755 87585436> (DF)
0x0000 4500 0034 8371 4000 3c06 8305 d845 4095 E..4.q at .<....E at .
0x0010 d845 472d 0050 08aa d0ef 5b7d 7b38 d529 .EG-.P....[}{8.)
0x0020 8010 4375 147d 0000 0101 080a 2c8c bc53 ..Cu.}......,..S
0x0030 0538 729c .8r.
13:17:10.670352 216.69.64.149.80 > 216.69.71.45.2218: F 514:514(0) ack 109 win 17376 <nop,nop,timestamp 747420755 87585436> (DF)
0x0000 4500 0034 ebbd 4000 3c06 1ab9 d845 4095 E..4.. at .<....E at .
0x0010 d845 472d 0050 08aa d0ef 5d7e 7b38 d529 .EG-.P....]~{8.)
0x0020 8011 43e0 1210 0000 0101 080a 2c8c bc53 ..C.........,..S
0x0030 0538 729c .8r.
13:17:10.670378 216.69.71.45.2218 > 216.69.64.149.80: . ack 1 win 57920 <nop,nop,timestamp 87585437 747420755> (DF)
0x0000 4500 0034 9c2a 4000 4006 664c d845 472d E..4.*@. at .fL.EG-
0x0010 d845 4095 08aa 0050 7b38 d529 d0ef 5b7d .E at ....P{8.)..[}
0x0020 8010 e240 75b0 0000 0101 080a 0538 729d ... at u........8r.
0x0030 2c8c bc53 ,..S
13:17:10.672885 216.69.64.149.80 > 216.69.71.45.2218: P 1:514(513) ack 109 win 17376 <nop,nop,timestamp 747420755 87585436> (DF)
[ snip file contents ]
13:17:10.672906 216.69.71.45.2218 > 216.69.64.149.80: . ack 515 win 57407 <nop,nop,timestamp 87585437 747420755> (DF)
0x0000 4500 0034 9c2b 4000 4006 664b d845 472d E..4.+ at .@.fK.EG-
0x0010 d845 4095 08aa 0050 7b38 d529 d0ef 5d7f .E at ....P{8.)..].
0x0020 8010 e03f 75af 0000 0101 080a 0538 729d ...?u........8r.
0x0030 2c8c bc53
Thus reducing the number of packets on the wire from 14 to 9. Obviously
for larger transfers, the difference gets lost in the noise. Nonetheless,
unless someone spots some undesireable side-effect that may be caused
by the change, I'll commit the attached patch in a few days.
Thanks,
Kelly
--
Kelly Yancey -- kbyanc@{posi.net,FreeBSD.org} -- kelly at nttmcl.com
-------------- next part --------------
Index: http.c
===================================================================
RCS file: /home/ncvs/src/lib/libfetch/http.c,v
retrieving revision 1.74
diff -u -p -r1.74 http.c
--- http.c 21 Sep 2004 18:35:21 -0000 1.74
+++ http.c 11 Feb 2005 21:30:41 -0000
@@ -76,6 +76,9 @@ __FBSDID("$FreeBSD$");
#include <time.h>
#include <unistd.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+
#include "fetch.h"
#include "common.h"
#include "httperr.h"
@@ -670,7 +673,7 @@ _http_connect(struct url *URL, struct ur
{
conn_t *conn;
int verbose;
- int af;
+ int af, val;
#ifdef INET6
af = AF_UNSPEC;
@@ -705,6 +708,10 @@ _http_connect(struct url *URL, struct ur
_fetch_syserr();
return (NULL);
}
+
+ val = 1;
+ setsockopt(conn->sd, IPPROTO_TCP, TCP_NOPUSH, &val, sizeof(val));
+
return (conn);
}
@@ -906,6 +913,7 @@ _http_request(struct url *URL, const cha
_http_cmd(conn, "Range: bytes=%lld-", (long long)url->offset);
_http_cmd(conn, "Connection: close");
_http_cmd(conn, "");
+ shutdown(conn->sd, SHUT_WR);
/* get reply */
switch (_http_get_reply(conn)) {
More information about the freebsd-net
mailing list