FreeBSD 8.0 and CDN connection issue

Mon Feb 15 19:00:35 UTC 2010

Hi

We are having a strange problem with FreeBSD 8.0  ( problem is not seen
on 7.X or 6.X ) and its behavior towards what appears to be a problem
with the footprint cdn which hosts sites such as:
http://www.formula1.com
http://www.vw.com
http://www.rca.com

The issue can be seen below:

PF enabled with scrubing:

/etc/pf.conf:
#----
scrub in all
pass in on lo0 all
pass out on lo0 all
pass in on em0 all
pass out on em0 all
#----

telnet to cdn on port 80. 

tcpdump below:
18:09:41.625409 IP freebsd.8.51776 > 209.84.7.126.80: Flags [S], seq
4208441727, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val
161897 ecr 0], length 0
18:09:41.900230 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
3063615393, ack 4208441728, win 5792, options [mss 1460,sackOK,TS val
813444467 ecr 161897,nop,wscale 7], length 0
18:09:41.900236 IP freebsd.8.51776 > 209.84.7.126.80: Flags [.], ack 1,
win 8326, options [nop,nop,TS val 161924 ecr 813444467], length 0
18:09:41.900242 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
3332367005, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900248 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
4174817132, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900254 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
440460550, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900467 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
477325580, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900473 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
769752490, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900479 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
629432722, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900485 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
4152361545, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900491 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
1928751848, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900497 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
3230160684, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900503 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
1491106974, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900509 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
2033022417, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900515 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
1187979504, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900521 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
797713074, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900527 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
3546267649, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900533 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
245712922, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.900539 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
1525656528, ack 4208441728, win 5840, options [mss 1460], length 0
18:09:41.901017 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
2249622145, ack 4208441728, win 5792, options [mss 1460,sackOK,TS val
246213904 ecr 161897,nop,wscale 7], length 0
18:09:46.241996 IP 209.84.7.126.80 > freebsd.8.51776: Flags [S.], seq
2249622145, ack 4208441728, win 5792, options [mss 1460,sackOK,TS val
246214338 ecr 161897,nop,wscale 7], length 0

pf disabled:

telnet 209.84.7.126 80
Trying 209.84.7.126...
telnet: connect to address 209.84.7.126: Connection reset by peer
telnet: Unable to connect to remote host

tcpdump:
18:11:29.122444 IP freebsd.8.41986 > 209.84.7.126.80: Flags [S], seq
2294539745, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val
172648 ecr 0], length 0
18:11:29.395219 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
2724299112, ack 2294539746, win 5792, options [mss 1460,sackOK,TS val
813551987 ecr 172648,nop,wscale 7], length 0
18:11:29.395225 IP freebsd.8.41986 > 209.84.7.126.80: Flags [.], ack 1,
win 8326, options [nop,nop,TS val 172676 ecr 813551987], length 0
18:11:29.395231 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
3789304658, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395237 IP freebsd.8.41986 > 209.84.7.126.80: Flags [.], ack
3229961751, win 8326, options [nop,nop,TS val 172676 ecr 813551987],
length 0
18:11:29.395243 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
3256912235, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395249 IP freebsd.8.41986 > 209.84.7.126.80: Flags [.], ack
3762354174, win 8326, options [nop,nop,TS val 172676 ecr 813551987],
length 0
18:11:29.395255 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
737801599, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395261 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R.], seq 1,
ack 1986497514, win 8326, options [nop,nop,TS val 172676 ecr 813551987],
length 0
18:11:29.395267 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
2722528016, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395273 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395279 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
960716006, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395285 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395291 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
4035042379, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395297 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395303 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
1231177745, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395309 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395315 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
2938041058, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395321 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395327 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
1919167960, ack 2294539746, win 5792, options [mss 1460,sackOK,TS val
245411549 ecr 172648,nop,wscale 7], length 0
18:11:29.395333 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395339 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
3549488364, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395345 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395351 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
3970540065, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395357 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395363 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
2087470875, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395369 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395375 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
2845644336, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395381 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395396 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
1745999935, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395402 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395408 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
2071807029, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395414 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395420 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
1370643748, ack 2294539746, win 5840, options [mss 1460], length 0
18:11:29.395426 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0
18:11:29.395513 IP 209.84.7.126.80 > freebsd.8.41986: Flags [S.], seq
1908254671, ack 2294539746, win 5792, options [mss 1460,sackOK,TS val
245411038 ecr 172648,nop,wscale 7], length 0
18:11:29.395519 IP freebsd.8.41986 > 209.84.7.126.80: Flags [R], seq
2294539746, win 0, length 0

At first read this does appear to be a problem with the remote host.

However this problems appears to only effect FreeBSD 8.0 boxes.

>From Linux we see a similar behavior but linux does not reset the
session and thus the site works:

20:18:29.774362 IP linux.2.6.18.8.39655 > 209.84.7.126.80: S
486001251:486001251(0) win 5840 <mss 1460,sackOK,timestamp 969292099
0,nop,wscale 4>
20:18:29.862571 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
128664214:128664214(0) ack 486001252 win 5792 <mss 1460,sackOK,timestamp
813971316 969292099,nop,wscale 7>
20:18:29.862642 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack 1 win
365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862653 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
586015071:586015071(0) ack 486001252 win 5792 <mss 1460,sackOK,timestamp
245451202 969292099,nop,wscale 7>
20:18:29.862662 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
3837616440 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862666 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
590852225:590852225(0) ack 486001252 win 5792 <mss 1460,sackOK,timestamp
245452997 969292099,nop,wscale 7>
20:18:29.862671 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
3832779286 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862674 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
1998807262:1998807262(0) ack 486001252 win 5840 <mss 1460>
20:18:29.862680 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
2424824249 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862683 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
593249148:593249148(0) ack 486001252 win 5792 <mss 1460,sackOK,timestamp
245460356 969292099,nop,wscale 7>
20:18:29.862688 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
3830382363 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862691 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
1042245924:1042245924(0) ack 486001252 win 5792 <mss
1460,sackOK,timestamp 246266613 969292099,nop,wscale 7>
20:18:29.862696 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
3381385587 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862699 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
603011058:603011058(0) ack 486001252 win 5792 <mss 1460,sackOK,timestamp
245451793 969292099,nop,wscale 7>
20:18:29.862704 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
3820620453 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862707 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
4095345615:4095345615(0) ack 486001252 win 5840 <mss 1460>
20:18:29.862712 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
328285896 win 365 <nop,nop,timestamp 969292187 813971316,nop,nop,sack 1
{0:1}>
20:18:29.862715 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
1518933688:1518933688(0) ack 486001252 win 5840 <mss 1460>
20:18:29.862720 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
2904697823 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862725 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
584671130:584671130(0) ack 486001252 win 5792 <mss 1460,sackOK,timestamp
245453508 969292099,nop,wscale 7>
20:18:29.862731 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
3838960381 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862733 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
2163470686:2163470686(0) ack 486001252 win 5840 <mss 1460>
20:18:29.862738 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
2260160825 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862743 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
1915446676:1915446676(0) ack 486001252 win 5840 <mss 1460>
20:18:29.862748 IP linux.2.6.18.8.39655 > 209.84.7.126.80: . ack
2508184835 win 365 <nop,nop,timestamp 969292187 813971316>
20:18:29.862751 IP 209.84.7.126.80 > linux.2.6.18.8.39655: S
2325962623:2325962623(0) ack 486001252 win 5840 <mss 1460>

I know that 'scrub in all' normalized the traffic, but why do I need to
normalize traffic in 8.0 when 7.x did need this to be done ?

I can't use pf as a solution as this box is currently using ifpw to
redirect the stuff to a transparent proxy.
Having pf enabled results other issues when the box under heavy load (
loss of states )

Is the a sysctl variable that can be enabled to 'behave like 7.X' ?

Thanks
Vikash

Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers at is.co.za and a copy will be emailed to you.