Very weird network behaviour with 4.7-RELEASE-p10 (large)

Giovanni P. Tirloni gpt at tirloni.org
Sun Jun 1 00:12:58 PDT 2003 

Hi,

 I have been doing some tests using hping2 and TCP SYN
 pings targeting local and remote hosts from two FreeBSD
 4.7-RELEASE-p10 and one Linux 2.4.18 host.
 
 The three machines have the same hardware configuration
 and have been running for 6 months now. The average load
 isnt too high (usually 0.01 to 0.15) on the FreeBSD machines.

 Here is the output from hping2 (excuse me the line wrap),
 
 root:~# hping -S -p 80 -c 6 www.freebsd.org
 HPING www.freebsd.org (fxp0 216.136.204.117): S set, 40 headers + 0 data bytes
 len=46 ip=216.136.204.117 ttl=50 DF id=20997 sport=80 flags=SA seq=0 win=57344 rtt=212.6 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=21140 sport=80 flags=SA seq=1 win=57344 rtt=212.3 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=21301 sport=80 flags=SA seq=2 win=57344 rtt=212.3 ms
 DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21494 sport=80 flags=SA seq=0 win=57344 rtt=3207.3 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=21498 sport=80 flags=SA seq=3 win=57344 rtt=212.3 ms
 DUP! len=46 ip=216.136.204.117 ttl=50 DF id=21650 sport=80 flags=SA seq=1 win=57344 rtt=3209.4 ms
 
 --- www.freebsd.org hping statistic ---
 5 packets tramitted, 6 packets received, -20% packet loss
 round-trip min/avg/max = 212.3/1211.0/3209.4 ms
 
 And tcpdump confirms this and shows more packets after hping2
 was stopped,
 
 root at srv0-cta:~# tcpdump -n host 216.136.204.117 and port 80
 tcpdump: listening on fxp0
 03:55:07.989538 200.203.183.32.1288 > 216.136.204.117.80: S 852569665:852569665(0) win 512
 03:55:08.202061 216.136.204.117.80 > 200.203.183.32.1288: S 3771704683:3771704683(0) ack 85256966603:55:11.229621 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 <mss 1460> (DF)
 03:55:12.027408 200.203.183.32.1292 > 216.136.204.117.80: S 1532350261:1532350261(0) win 512
 03:55:12.206721 216.136.204.117.80 > 200.203.183.32.1289: S 3305368856:3305368856(0) ack 1218353109 win 57344 <mss 1460> (DF)
 03:55:12.239377 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)
 03:55:13.216677 216.136.204.117.80 > 200.203.183.32.1290: S 2652913389:2652913389(0) ack 422129860 win 57344 <mss 1460> (DF)
 03:55:14.226573 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 <mss 1460> (DF)
 03:55:15.236648 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)
 03:55:17.196452 216.136.204.117.80 > 200.203.183.32.1288: S 3771704683:3771704683(0) ack 852569666 win 57344 <mss 1460> (DF)
 03:55:18.206412 216.136.204.117.80 > 200.203.183.32.1289: S 3305368856:3305368856(0) ack 1218353109 win 57344 <mss 1460> (DF)
 03:55:19.216406 216.136.204.117.80 > 200.203.183.32.1290: S 2652913389:2652913389(0) ack 422129860 win 57344 <mss 1460> (DF)
 03:55:20.226331 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 <mss 1460> (DF)
 03:55:21.236273 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)
 03:55:29.195990 216.136.204.117.80 > 200.203.183.32.1288: S 3771704683:3771704683(0) ack 852569666 win 57344 <mss 1460> (DF)
 03:55:30.205987 216.136.204.117.80 > 200.203.183.32.1289: S 3305368856:3305368856(0) ack 1218353109 win 57344 <mss 1460> (DF)
 03:55:31.216041 216.136.204.117.80 > 200.203.183.32.1290: S 2652913389:2652913389(0) ack 422129860 win 57344 <mss 1460> (DF)
 03:55:32.225922 216.136.204.117.80 > 200.203.183.32.1291: S 2387314094:2387314094(0) ack 2741695 win 57344 <mss 1460> (DF)
 03:55:33.235869 216.136.204.117.80 > 200.203.183.32.1292: S 3378516967:3378516967(0) ack 1532350262 win 57344 <mss 1460> (DF)


 Another different output from hping (strange rtt's),
 
 root at srv0-cta:~# hping -S -p 80 -c 8 www.freebsd.org
 HPING www.freebsd.org (fxp0 216.136.204.117): S set, 40 headers + 0 data bytes
 len=46 ip=216.136.204.117 ttl=50 DF id=38289 sport=80 flags=SA seq=0 win=57344 rtt=212.5 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=38291 sport=80 flags=SA seq=0 win=57344 rtt=0.0 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=38371 sport=80 flags=SA seq=1 win=57344 rtt=211.9 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=38372 sport=80 flags=SA seq=0 win=57344 rtt=0.0 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=38404 sport=80 flags=SA seq=2 win=57344 rtt=211.9 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=38405 sport=80 flags=SA seq=0 win=57344 rtt=0.0 ms
 DUP! len=46 ip=216.136.204.117 ttl=50 DF id=38687 sport=80 flags=SA seq=0 win=57344 rtt=3207.3 ms
 len=46 ip=216.136.204.117 ttl=50 DF id=38702 sport=80 flags=SA seq=3 win=57344 rtt=212.0 ms
 
 --- www.freebsd.org hping statistic ---
 4 packets tramitted, 8 packets received, -100% packet loss
 round-trip min/avg/max = 211.9/811.1/3207.3 ms
 
 And the tcpdump output,
 
 04:00:57.419466 216.136.204.117.80 > 200.203.183.32.2381: S 1076154568:1076154568(0) ack 1439276851 win 57344 <mss 1460> (DF)
 04:00:58.201940 200.203.183.32.1995 > 216.136.204.117.80: S 264430648:264430648(0) win 512
 04:00:58.414397 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
 04:00:58.429518 216.136.204.117.80 > 200.203.183.32.2382: S 3830791941:3830791941(0) ack 451178042 win 57344 <mss 1460> (DF)
 04:00:59.206642 200.203.183.32.1996 > 216.136.204.117.80: S 1522574199:1522574199(0) win 512
 04:00:59.418526 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
 04:00:59.439330 216.136.204.117.80 > 200.203.183.32.2383: S 1160193874:1160193874(0) ack 1028452909 win 57344 <mss 1460> (DF)
 04:01:00.216649 200.203.183.32.1997 > 216.136.204.117.80: S 442375938:442375938(0) win 512
 04:01:00.428557 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
 04:01:00.449324 216.136.204.117.80 > 200.203.183.32.2384: S 3552821723:3552821723(0) ack 1069527292 win 57344 <mss 1460> (DF)
 04:01:01.226660 200.203.183.32.1998 > 216.136.204.117.80: S 240094281:240094281(0) win 512
 04:01:01.409234 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
 04:01:01.438601 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
 04:01:01.529193 216.136.204.117.80 > 200.203.183.32.2065: S 4017739092:4017739092(0) ack 1616120023 win 57344 <mss 1460> (DF)
 04:01:02.409213 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
 04:01:02.529165 216.136.204.117.80 > 200.203.183.32.2066: S 754183561:754183561(0) ack 1231654702 win 57344 <mss 1460> (DF)
 04:01:03.419286 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
 04:01:03.539156 216.136.204.117.80 > 200.203.183.32.2067: S 1059652438:1059652438(0) ack 2093248177 win 57344 <mss 1460> (DF)
 04:01:04.429131 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
 04:01:04.549163 216.136.204.117.80 > 200.203.183.32.2068: S 3136323399:3136323399(0) ack 723495504 win 57344 <mss 1460> (DF)
 04:01:07.409090 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
 04:01:08.408993 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
 04:01:09.418980 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
 04:01:10.429020 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
 04:01:19.408629 216.136.204.117.80 > 200.203.183.32.1995: S 2379854270:2379854270(0) ack 264430649 win 57344 <mss 1460> (DF)
 04:01:20.408544 216.136.204.117.80 > 200.203.183.32.1996: S 3804689919:3804689919(0) ack 1522574200 win 57344 <mss 1460> (DF)
 04:01:21.418566 216.136.204.117.80 > 200.203.183.32.1997: S 3901380776:3901380776(0) ack 442375939 win 57344 <mss 1460> (DF)
 04:01:22.428427 216.136.204.117.80 > 200.203.183.32.1998: S 4153685029:4153685029(0) ack 240094282 win 57344 <mss 1460> (DF)
 
 Some users have complaint about server timeouts since Monday (but
 I havent experienced the problems they seem to be having). At first 
 I thought it was the switch but I don't see errors from the Linux 
 machine.

 That happens when pinging (with hping -S -p 80),

   bsd1:fxp0 -> bsd2
   bsd2:fxp0 -> bsd1
   bsd1:lo0  -> bsd1:lo0
   bsd2:lo0  -> bsd2:lo0
   bsd1:fxp0 -> remote
   bsd2:fxp0 -> remote
   bsd1:fxp0 -> linux
 
 Any help is welcome and sorry if this email doesn't have all the
 information needed to solve this problem. I'll be happy to provide
 more details and do more tests if needed.
 
 Here is the output from ifconfig fxp0,
 
  fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 200.203.183.32 netmask 0xffffffc0 broadcast 200.203.183.63
        inet 200.203.183.33 netmask 0xffffffff broadcast 200.203.183.33
        inet 200.203.183.37 netmask 0xffffffff broadcast 200.203.183.37
        ether 00:07:e9:ad:2a:ab
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active                                                  

 Someone mentioned cvsup'ing to 4.8-STABLE but I'm afraid of doing
 so because those two boxes are very critical.
 
 --
 Giovanni P. Tirloni
 http://www.tirloni.org

More information about the freebsd-stable mailing list