Help with strange web server problem

Jerry Bell jbell at stelesys.com
Mon Feb 13 20:07:40 PST 2006


Looks like it's still an issue, so I'd say the firewall issue is still 
in play.  If there is not a firewall/proxy in place, are there any known 
issues with IPFW (or anything else with FBSD) that could cause this 
behavior?

Jerry Bell wrote:
> Charles - thank you for your excellent investigation!  I'm pretty sure 
> that my colo provider isn't running a firewall (I've asked them not 
> to, anyhow).  I am running IPFW on that box, with the standard "allow 
> tcp from any to any established" followed by the  "allow tcp any to 
> my_ip 80 setup".  I've done that on other servers without it being a 
> problem like this.  I'm going to have the colo double check for router 
> acl's or something like that in the morning.
>
> Since this is such an intermittent problem, I can't yet say that it's 
> fixed, but I ran with the "disks being idled" theory and wrote a small 
> script that creates a file and deletes a file every minute, and since 
> that's been running, I've not seeing the issue repeat - but then this 
> is not a very repeatable problem.
>
> Thanks again for your great assistance.
>
> Jerry
>
>
> Charles Swiger wrote:
>> On Feb 13, 2006, at 3:12 PM, Jerry Bell wrote:
>>> I didn't want to spam the link out, but it's www.musiclodge.com.  I 
>>> will
>>> gather the capture data from working and non working sessions and 
>>> send it
>>> out.
>>
>> Well, I can confirm the behavior you've described.
>>
>> It looks somewhat like a stateful firewall or is in the way and is 
>> generating an RST, even while your webserver tries to generate a 
>> response.  However, once the firewall sees the outbound traffic, it 
>> seems to create a dynamic rule which lets the traffic from subsequent 
>> connections through:
>>
>> 5-pan# tcpdump -tnXs 0 host www.musiclodge.com
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol 
>> decode
>> listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
>> IP 199.103.21.238.50740 > 63.175.100.44.80: S 
>> 2282569549:2282569549(0) win 65535 <mss 1460,nop,wscale 
>> 0,nop,nop,timestamp 1159441862 0>
>>         0x0000:  4510 003c 4653 4000 4006 7328 c767 15ee  
>> E..<FS at .@.s(.g..
>>         0x0010:  3faf 642c c634 0050 880d 3f4d 0000 0000  
>> ?.d,.4.P..?M....
>>         0x0020:  a002 ffff 815f 0000 0204 05b4 0103 0300  
>> ....._..........
>>         0x0030:  0101 080a 451b adc6 0000 0000            ....E.......
>> IP 63.175.100.44.80 > 199.103.21.238.50740: S 
>> 2634350592:2634350592(0) ack 2282569550 win 65535
>>         0x0000:  4500 0028 0000 4000 2506 d49f 3faf 642c  
>> E..(.. at .%...?.d,
>>         0x0010:  c767 15ee 0050 c634 9d05 0000 880d 3f4e  
>> .g...P.4......?N
>>         0x0020:  5012 ffff 03bc 0000 0000 0000 0000 1b60  
>> P..............`
>>         0x0030:  2678                                     &x
>> IP 199.103.21.238.50740 > 63.175.100.44.80: . ack 1 win 65535
>>         0x0000:  4510 0028 4655 4000 4006 733a c767 15ee  
>> E..(FU at .@.s:.g..
>>         0x0010:  3faf 642c c634 0050 880d 3f4e 9d05 0001  
>> ?.d,.4.P..?N....
>>         0x0020:  5010 ffff 03bd 0000                      P.......
>>
>> 3-way handshake is completed here, next traffic should be from my 
>> machine making the "GET /", request, but instead your machine sends 
>> another ACK:
>>
>> IP 63.175.100.44.80 > 199.103.21.238.50740: S 
>> 2238145710:2238145710(0) ack 2282569550 win 65535 <mss 
>> 1460,nop,wscale 1,nop,nop,timestamp 1453026167 1159441862>
>>         0x0000:  4500 003c 57fa 4000 3206 6f91 3faf 642c  
>> E..<W. at .2.o.?.d,
>>         0x0010:  c767 15ee 0050 c634 8567 64ae 880d 3f4e  
>> .g...P.4.gd...?N
>>         0x0020:  a012 ffff 9cdb 0000 0204 05b4 0103 0301  
>> ................
>>         0x0030:  0101 080a 569b 6b77 451b adc6 9345 1153  
>> ....V.kwE....E.S
>>
>> Interesting that the previous ack had no TCP options set, whereas 
>> this one does include a timestamp in response.
>>
>> IP 199.103.21.238.50740 > 63.175.100.44.80: . ack 396204883 win 65535 
>> <nop,nop,timestamp 1159441863 1453026167>
>>         0x0000:  4510 0034 4656 4000 4006 732d c767 15ee  
>> E..4FV at .@.s-.g..
>>         0x0010:  3faf 642c c634 0050 880d 3f4e 9d05 0001  
>> ?.d,.4.P..?N....
>>         0x0020:  8010 ffff 8157 0000 0101 080a 451b adc7  
>> .....W......E...
>>         0x0030:  569b 6b77                                V.kw
>>
>> Where did sequence # 396204883 come from?  And your side follows up 
>> with a pair of connection resets, and a normal ACK packet, too.
>>
>> IP 63.175.100.44.80 > 199.103.21.238.50740: R 
>> 2634350593:2634350593(0) win 0
>>         0x0000:  4500 0028 b6f6 4000 3206 10a9 3faf 642c  
>> E..(.. at .2...?.d,
>>         0x0010:  c767 15ee 0050 c634 9d05 0001 0000 0000  
>> .g...P.4........
>>         0x0020:  5004 0000 cb24 0000 0000 0000 0000 f3fa  
>> P....$..........
>>         0x0030:  5489                                     T.
>> IP 63.175.100.44.80 > 199.103.21.238.50740: R 
>> 2634350593:2634350593(0) win 0
>>         0x0000:  4500 0028 4bfc 4000 3206 7ba3 3faf 642c  
>> E..(K. at .2.{.?.d,
>>         0x0010:  c767 15ee 0050 c634 9d05 0001 0000 0000  
>> .g...P.4........
>>         0x0020:  5004 0000 cb24 0000 0000 0000 0000 abb8  
>> P....$..........
>>         0x0030:  c9be                                     ..
>> IP 63.175.100.44.80 > 199.103.21.238.50740: S 
>> 2238145710:2238145710(0) ack 2282569550 win 65535 <mss 
>> 1460,nop,wscale 1,nop,nop,timestamp 1453026467 1159441862>
>>         0x0000:  4500 003c 3a9d 4000 3206 8cee 3faf 642c  
>> E..<:. at .2...?.d,
>>         0x0010:  c767 15ee 0050 c634 8567 64ae 880d 3f4e  
>> .g...P.4.gd...?N
>>         0x0020:  a012 ffff 9baf 0000 0204 05b4 0103 0301  
>> ................
>>         0x0030:  0101 080a 569b 6ca3 451b adc6 bdd6 d7c9  
>> ....V.l.E.......
>>
>> ...and my side closes, too.  Something is badly confused.
>>
>> IP 199.103.21.238.50740 > 63.175.100.44.80: R 
>> 2282569550:2282569550(0) win 0
>>         0x0000:  4500 0028 465a 4000 4006 7345 c767 15ee  
>> E..(FZ at .@.sE.g..
>>         0x0010:  3faf 642c c634 0050 880d 3f4e 0000 0000  
>> ?.d,.4.P..?N....
>>         0x0020:  5004 0000 a0cf 0000                      P.......
>>
>>     -------------------
>>
>> When I repeat the connection attempt a few seconds later:
>>
>> IP 199.103.21.238.50743 > 63.175.100.44.80: S 262625798:262625798(0) 
>> win 65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 1159442517 0>
>>         0x0000:  4510 003c 46c8 4000 4006 72b3 c767 15ee  
>> E..<F. at .@.r..g..
>>         0x0010:  3faf 642c c637 0050 0fa7 5a06 0000 0000  
>> ?.d,.7.P..Z.....
>>         0x0020:  a002 ffff 815f 0000 0204 05b4 0103 0300  
>> ....._..........
>>         0x0030:  0101 080a 451b b055 0000 0000            ....E..U....
>> IP 63.175.100.44.80 > 199.103.21.238.50743: S 362624500:362624500(0) 
>> ack 262625799 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 
>> 1453058903 1159442517>
>>         0x0000:  4500 003c e034 4000 3206 e756 3faf 642c  
>> E..<.4 at .2..V?.d,
>>         0x0010:  c767 15ee 0050 c637 159d 35f4 0fa7 5a07  
>> .g...P.7..5...Z.
>>         0x0020:  a012 ffff 169b 0000 0204 05b4 0103 0301  
>> ................
>>         0x0030:  0101 080a 569b eb57 451b b055 55d6 9ceb  
>> ....V..WE..UU...
>> IP 199.103.21.238.50743 > 63.175.100.44.80: . ack 1 win 65535 
>> <nop,nop,timestamp 1159442517 1453058903>
>>         0x0000:  4510 0034 46c9 4000 4006 72ba c767 15ee  
>> E..4F. at .@.r..g..
>>         0x0010:  3faf 642c c637 0050 0fa7 5a07 159d 35f5  
>> ?.d,.7.P..Z...5.
>>         0x0020:  8010 ffff 8157 0000 0101 080a 451b b055  
>> .....W......E..U
>>         0x0030:  569b eb57                                V..W
>>
>> 3-way handshake finishes OK, this time your initial ACK is OK and 
>> includes a timestamp back.
>>
>> IP 199.103.21.238.50743 > 63.175.100.44.80: P 1:17(16) ack 1 win 
>> 65535 <nop,nop,timestamp 1159442525 1453058903>
>>         0x0000:  4510 0044 46cd 4000 4006 72a6 c767 15ee  
>> E..DF. at .@.r..g..
>>         0x0010:  3faf 642c c637 0050 0fa7 5a07 159d 35f5  
>> ?.d,.7.P..Z...5.
>>         0x0020:  8018 ffff 8167 0000 0101 080a 451b b05d  
>> .....g......E..]
>>         0x0030:  569b eb57 4745 5420 2f20 4854 5450 2f31  
>> V..WGET./.HTTP/1
>>         0x0040:  2e30 0d0a                                .0..
>>
>> Here was my GET, which you then ACK....
>>
>> IP 63.175.100.44.80 > 199.103.21.238.50743: . ack 17 win 33304 
>> <nop,nop,timestamp 1453059309 1159442525>
>>         0x0000:  4500 0034 052a 4000 3206 c269 3faf 642c  
>> E..4.*@.2..i?.d,
>>         0x0010:  c767 15ee 0050 c637 159d 35f5 0fa7 5a17  
>> .g...P.7..5...Z.
>>         0x0020:  8010 8218 be99 0000 0101 080a 569b eced  
>> ............V...
>>         0x0030:  451b b05d db4e 4827                      E..].NH'
>> IP 199.103.21.238.50743 > 63.175.100.44.80: P 17:19(2) ack 1 win 
>> 65535 <nop,nop,timestamp 1159442526 1453059309>
>>         0x0000:  4510 0036 46ce 4000 4006 72b3 c767 15ee  
>> E..6F. at .@.r..g..
>>         0x0010:  3faf 642c c637 0050 0fa7 5a17 159d 35f5  
>> ?.d,.7.P..Z...5.
>>         0x0020:  8018 ffff 8159 0000 0101 080a 451b b05e  
>> .....Y......E..^
>>         0x0030:  569b eced 0d0a                           V.....
>>
>> ...followed by a correct data response.
>>
>> IP 63.175.100.44.80 > 199.103.21.238.50743: . 1:1449(1448) ack 19 win 
>> 33304 <nop,nop,timestamp 1453059329 1159442526>
>>         0x0000:  4500 05dc c865 4000 3206 f985 3faf 642c  
>> E....e at .2...?.d,
>>         0x0010:  c767 15ee 0050 c637 159d 35f5 0fa7 5a19  
>> .g...P.7..5...Z.
>>         0x0020:  8010 8218 2e5f 0000 0101 080a 569b ed01  
>> ....._......V...
>>         0x0030:  451b b05e 4854 5450 2f31 2e31 2032 3030  
>> E..^HTTP/1.1.200
>>         0x0040:  204f 4b0d 0a44 6174 653a 204d 6f6e 2c20  
>> .OK..Date:.Mon,.
>> [ ... ]
>>
>> Check your firewall, or see whether your ISP or whoever has put a 
>> HTTP reverse proxy in place which is breaking these connections.
>>
>> ---Chuck
>>
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to 
> "freebsd-questions-unsubscribe at freebsd.org"


More information about the freebsd-questions mailing list