IPSec tcp session stalling

Sat Oct 22 16:40:58 PDT 2005

To try something else: Could you guys try to disable SACK on the machines 
involved?  I haven't looked at the dumps as of yet, but that's one simple 
test that might help to identify the problem.

sysctl net.inet.tcp.sack.enable=0

On Sunday 23 October 2005 02:23, Volker wrote:
> Michael,
>
> I not that sure if I'm right in checking what you suggested but when
> trying to do ping hostB from hostA with oversized packets through the
> IPSec tunnel by:
>
> # ping -c 10 -s 12000 10.128.6.1
>
> I'm getting replies easily.
>
> While doing that and tcpdump'ing the gif interface, I'm seeing the
> fragmented packets coming in properly.
>
> If that's a reliable check for MTU than the problem should not be MTU
> related. Is there any other way to check MTU problems by using `ping'?
>
> Thanks,
>
> Volker
>
> On 2005-10-22 20:16, Michael VInce wrote:
> > Try sending different sized pings or other packet size control utils to
> > really make sure its not MTU related.
> > Maybe there is an upstream router thats blocking ICMP fragment packets,
> > have you ever seen them? try forcing the creation of some.
> >
> > Mike
> >
> > Volker wrote:
> >> Still having the same problem with an IPSec tunnel between FreeBSD 5.4R
> >> hosts.
> >>
> >> Problem description:
> >> scp session tries to transfer a large file through an IPSec tunnel. The
> >> file is being transmitted but scp says 'stalled' after 56K (49152 bytes
> >> file size). The IPSec tunnel itself is still up even after the scp
> >> abort. Other tcp sessions break, too when sending too much traffic
> >> through the tunnel.
> >>
> >> I've taken a closer look to it and tried to get something useful out of
> >> the tcpdump but I'm unable to see any errors or I'm misinterpreting
> >> something.
> >>
> >> The connection looks like:
> >>
> >> extIP: A.B.C.D
> >> extIP: E.F.G.H
> >> host A ------------------ (internet) ------------------ host B
> >> tunnelIP: 10.128.1.6                                           tunnelIP:
> >> 10.128.6.1
> >>
> >> host A just has an external interface (em1) connected to a leased line
> >> with a fixed IP address (IP-addr A.B.C.D).
> >> host B has an S-DSL connection at xl0, PPPoE at ng0 (IP-addr. E.F.G.H).
> >>
> >> Both hosts are using gif for the IPSec tunnel.
> >>
> >> The routing tables (netstat -rnWf inet) are looking good and IMHO the
> >> MTU is fine.
> >>
> >> host A:
> >> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >>        options=b<RXCSUM,TXCSUM,VLAN_MTU>
> >>        inet A.B.C.D netmask 0xfffffff8 broadcast A.B.C.z
> >>        ether 00:c0:9f:46:ec:c7
> >>        media: Ethernet autoselect (100baseTX <full-duplex>)
> >>        status: active
> >> gif6: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
> >>        tunnel inet A.B.C.D --> E.F.G.H
> >>        inet 10.128.1.6 --> 10.128.6.1 netmask 0xffffffff
> >>        inet6 fe80::2c0:9fff:fe46:ecc6%gif6 prefixlen 64 scopeid 0x4
> >>
> >> Routing tables (shortened)
> >> Destination        Gateway            Flags    Refs      Use    Mtu
> >> Netif Expire
> >> default            A.B.C.x      UGS         2   516686   1500      em1
> >> 10.128.1.6         127.0.0.1          UH          0       14
> >> 16384      lo0
> >> 10.128.6.1         10.128.1.6         UH          0     6017
> >> 1280     gif6
> >> 127.0.0.1          127.0.0.1          UH          0    31633
> >> 16384      lo0
> >> A.B.C.x/29   link#2             UC          0        0   1500      em1
> >> A.B.C.D      00:c0:9f:46:ec:c7  UHLW        0      112   1500      lo0
> >>
> >> On host B the interfaces and routing tables are looking like:
> >> xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >>        options=8<VLAN_MTU>
> >>        inet 0.0.0.0 netmask 0xff000000 broadcast 0.255.255.255
> >>        inet6 fe80::260:8ff:fe6c:e73c%xl0 prefixlen 64 scopeid 0x1
> >>        ether 00:60:08:6c:e7:3c
> >>        media: Ethernet 10baseT/UTP (10baseT/UTP <half-duplex>)
> >>        status: active
> >> gif1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
> >>        tunnel inet E.F.G.H --> A.B.C.D
> >>        inet6 fe80::260:8ff:fe6c:e73c%gif1 prefixlen 64 scopeid 0x4
> >>        inet 10.128.6.1 --> 10.128.1.6 netmask 0xffffffff
> >> ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1456
> >>        inet E.F.G.H --> 217.5.98.186 netmask 0xffffffff
> >>
> >> Routing tables (shortened)
> >> Destination        Gateway            Flags    Refs      Use    Mtu
> >> Netif Expire
> >> 0                  link#1             UC          0        0   1500
> >> xl0 =>
> >> default            217.5.98.186       UGS         1    38474
> >> 1456      ng0
> >> 10.128.1.6         10.128.6.1         UH          4     2196
> >> 1280     gif1
> >> 127.0.0.1          127.0.0.1          UH          0    80424
> >> 16384      lo0
> >> 217.5.98.186       E.F.G.H       UH          1        0   1456      ng0
> >> E.F.G.H       lo0                UHS         0        0  16384      lo0
> >>
> >> While trying to fetch a file by scp on host A (receiver) from host B
> >> (sender), I captured the following tcpdump on host B:
> >>
> >> tcpdump -netttvvi gif1:
> >>> 000023 AF 2 1280: IP (tos 0x8, ttl  64, id 13202, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 43864:45092(1228) ack 1330 win 33156 <nop,nop,timestamp 481770567
> >>> 565002838>
> >>> 000207 AF 2 1280: IP (tos 0x8, ttl  64, id 52187, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 45092:46320(1228) ack 1330 win 33156 <nop,nop,timestamp 481770567
> >>> 565002838>
> >>> 000220 AF 2 1280: IP (tos 0x8, ttl  64, id 33774, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 46320:47548(1228) ack 1330 win 33156 <nop,nop,timestamp 481770568
> >>> 565002838>
> >>> 003524 AF 2 52: IP (tos 0x8, ttl  64, id 42063, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 38952 win 33156 <nop,nop,timestamp 565002844
> >>> 481770524> 000024 AF 2 1280: IP (tos 0x8, ttl  64, id 48541, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 47548:48776(1228) ack 1330 win 33156 <nop,nop,timestamp 481770571
> >>> 565002844>
> >>> 011203 AF 2 52: IP (tos 0x8, ttl  64, id 60517, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 41408 win 32542 <nop,nop,timestamp 565002855
> >>> 481770530> 000058 AF 2 1280: IP (tos 0x8, ttl  64, id 15798, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 48776:50004(1228) ack 1330 win 33156 <nop,nop,timestamp 481770582
> >>> 565002855>
> >>> 000246 AF 2 1280: IP (tos 0x8, ttl  64, id 31721, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 50004:51232(1228) ack 1330 win 33156 <nop,nop,timestamp 481770583
> >>> 565002855>
> >>> 005147 AF 2 52: IP (tos 0x8, ttl  64, id 22347, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 42636 win 33156 <nop,nop,timestamp 565002861
> >>> 481770542> 000024 AF 2 1280: IP (tos 0x8, ttl  64, id 61057, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 51232:52460(1228) ack 1330 win 33156 <nop,nop,timestamp 481770588
> >>> 565002861>
> >>> 020769 AF 2 52: IP (tos 0x8, ttl  64, id 27692, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 45092 win 32542 <nop,nop,timestamp 565002881
> >>> 481770547> 000027 AF 2 1280: IP (tos 0x8, ttl  64, id 64167, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 52460:53688(1228) ack 1330 win 33156 <nop,nop,timestamp 481770609
> >>> 565002881>
> >>> 000209 AF 2 1280: IP (tos 0x8, ttl  64, id 45457, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 53688:54916(1228) ack 1330 win 33156 <nop,nop,timestamp 481770609
> >>> 565002881>
> >>> 005260 AF 2 52: IP (tos 0x8, ttl  64, id 53832, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 46320 win 33156 <nop,nop,timestamp 565002887
> >>> 481770567> 000024 AF 2 1280: IP (tos 0x8, ttl  64, id 3515, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 54916:56144(1228) ack 1330 win 33156 <nop,nop,timestamp 481770614
> >>> 565002887>
> >>> 011020 AF 2 52: IP (tos 0x8, ttl  64, id 11608, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 48776 win 32542 <nop,nop,timestamp 565002898
> >>> 481770568> 000026 AF 2 1280: IP (tos 0x8, ttl  64, id 5848, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 56144:57372(1228) ack 1330 win 33156 <nop,nop,timestamp 481770625
> >>> 565002898>
> >>> 000211 AF 2 1280: IP (tos 0x8, ttl  64, id 39892, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 57372:58600(1228) ack 1330 win 33156 <nop,nop,timestamp 481770625
> >>> 565002898>
> >>> 005641 AF 2 52: IP (tos 0x8, ttl  64, id 7943, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 50004 win 33156 <nop,nop,timestamp 565002904
> >>> 481770582> 000024 AF 2 1280: IP (tos 0x8, ttl  64, id 8678, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 58600:59828(1228) ack 1330 win 33156 <nop,nop,timestamp 481770631
> >>> 565002904>
> >>> 011072 AF 2 52: IP (tos 0x8, ttl  64, id 38257, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 52460 win 32542 <nop,nop,timestamp 565002915
> >>> 481770583> 000025 AF 2 1280: IP (tos 0x8, ttl  64, id 12255, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 59828:61056(1228) ack 1330 win 33156 <nop,nop,timestamp 481770642
> >>> 565002915>
> >>> 000209 AF 2 1280: IP (tos 0x8, ttl  64, id 46257, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 61056:62284(1228) ack 1330 win 33156 <nop,nop,timestamp 481770642
> >>> 565002915>
> >>> 000222 AF 2 1280: IP (tos 0x8, ttl  64, id 4093, offset 0, flags
> >>> [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 62284:63512(1228) ack 1330 win 33156 <nop,nop,timestamp 481770643
> >>> 565002915>
> >>> 007065 AF 2 52: IP (tos 0x8, ttl  64, id 18720, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 53688 win 33156 <nop,nop,timestamp 565002922
> >>> 481770609> 000025 AF 2 1280: IP (tos 0x8, ttl  64, id 38378, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 63512:64740(1228) ack 1330 win 33156 <nop,nop,timestamp 481770650
> >>> 565002922>
> >>> 011034 AF 2 52: IP (tos 0x8, ttl  64, id 18718, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 56144 win 32542 <nop,nop,timestamp 565002934
> >>> 481770609> 000024 AF 2 1280: IP (tos 0x8, ttl  64, id 8148, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: .
> >>> 64740:65968(1228) ack 1330 win 33156 <nop,nop,timestamp 481770661
> >>> 565002934>
> >>> 005991 AF 2 52: IP (tos 0x8, ttl  64, id 62285, offset 0, flags
> >>> [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok]
> >>> 1330:1330(0) ack 57372 win 33156 <nop,nop,timestamp 565002939
> >>> 481770625> 010726 AF 2 52: IP (tos 0x8, ttl  64, id 1549, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum
> >>> ok] 1330:1330(0) ack 59828 win 32542 <nop,nop,timestamp 565002950
> >>> 481770625> 005670 AF 2 52: IP (tos 0x8, ttl  64, id 61504, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum
> >>> ok] 1330:1330(0) ack 61056 win 33156 <nop,nop,timestamp 565002956
> >>> 481770642> 011260 AF 2 52: IP (tos 0x8, ttl  64, id 32633, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum
> >>> ok] 1330:1330(0) ack 63512 win 32542 <nop,nop,timestamp 565002967
> >>> 481770642> 005510 AF 2 52: IP (tos 0x8, ttl  64, id 54614, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum
> >>> ok] 1330:1330(0) ack 64740 win 33156 <nop,nop,timestamp 565002973
> >>> 481770650> 104909 AF 2 52: IP (tos 0x8, ttl  64, id 50471, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum
> >>> ok] 1330:1330(0) ack 65968 win 33156 <nop,nop,timestamp 565003078
> >>> 481770661>
> >>
> >> tcpdump -netttvvi ng0 host A.B.C.D:
> >>> 000227 AF 2 1352: IP (tos 0x8, ttl  64, id 25895, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10b)
> >>> 011042 AF 2 128: IP (tos 0x8, ttl  61, id 5786, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf0)
> >>> 000226 AF 2 1352: IP (tos 0x8, ttl  64, id 36701, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10c)
> >>> 000216 AF 2 1352: IP (tos 0x8, ttl  64, id 8789, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10d)
> >>> 004853 AF 2 128: IP (tos 0x8, ttl  61, id 17128, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf1)
> >>> 000227 AF 2 1352: IP (tos 0x8, ttl  64, id 34888, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10e)
> >>> 018747 AF 2 128: IP (tos 0x8, ttl  61, id 14828, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf2)
> >>> 000248 AF 2 1352: IP (tos 0x8, ttl  64, id 34356, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10f)
> >>> 000223 AF 2 1352: IP (tos 0x8, ttl  64, id 34151, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x110)
> >>> 005030 AF 2 128: IP (tos 0x8, ttl  61, id 45476, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf3)
> >>> 000228 AF 2 1352: IP (tos 0x8, ttl  64, id 39765, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x111)
> >>> 011247 AF 2 128: IP (tos 0x8, ttl  61, id 63692, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf4)
> >>> 000226 AF 2 1352: IP (tos 0x8, ttl  64, id 29240, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x112)
> >>> 000222 AF 2 1352: IP (tos 0x8, ttl  64, id 43306, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x113)
> >>> 005663 AF 2 128: IP (tos 0x8, ttl  61, id 32980, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf5)
> >>> 000228 AF 2 1352: IP (tos 0x8, ttl  64, id 56920, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x114)
> >>> 010190 AF 2 128: IP (tos 0x8, ttl  61, id 3206, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf6)
> >>> 000227 AF 2 1352: IP (tos 0x8, ttl  64, id 4655, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x115)
> >>> 000215 AF 2 1352: IP (tos 0x8, ttl  64, id 62740, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x116)
> >>> 000203 AF 2 1352: IP (tos 0x8, ttl  64, id 35642, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x117)
> >>> 006875 AF 2 128: IP (tos 0x8, ttl  61, id 37801, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf7)
> >>> 000234 AF 2 1352: IP (tos 0x8, ttl  64, id 41803, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x118)
> >>> 010651 AF 2 128: IP (tos 0x8, ttl  61, id 54256, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf8)
> >>> 000235 AF 2 1352: IP (tos 0x8, ttl  64, id 30732, offset 0, flags
> >>> [none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x119)
> >>> 007913 AF 2 128: IP (tos 0x8, ttl  61, id 7647, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf9)
> >>> 011166 AF 2 128: IP (tos 0x8, ttl  61, id 58037, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfa)
> >>> 005483 AF 2 128: IP (tos 0x8, ttl  61, id 65275, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfb)
> >>> 011250 AF 2 128: IP (tos 0x8, ttl  61, id 47289, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfc)
> >>> 005505 AF 2 128: IP (tos 0x8, ttl  61, id 203, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfd)
> >>> 104747 AF 2 128: IP (tos 0x8, ttl  61, id 45263, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfe)
> >>> 8. 338674 AF 2 128: IP (tos 0x8, ttl  61, id 36351, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xff)
> >>> 319992 AF 2 128: IP (tos 0x8, ttl  61, id 18085, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x100)
> >>> 441837 AF 2 128: IP (tos 0x8, ttl  61, id 58323, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x101)
> >>> 684077 AF 2 128: IP (tos 0x8, ttl  61, id 35487, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x102)
> >>> 1. 167602 AF 2 128: IP (tos 0x8, ttl  61, id 34442, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x103)
> >>> 2. 136032 AF 2 128: IP (tos 0x8, ttl  61, id 8345, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x104)
> >>> 2. 984665 AF 2 128: IP (tos 0x8, ttl  61, id 35456, offset 0, flags
> >>> [none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x105)
> >>>
> >>>
> >>>
> >>>
> >>> From what I'm seeing host B just stops sending without any reason. At
> >>
> >> least I don't see any fragmented packets. The only thing I've seen is
> >> some packets doesn't get ack'ed by the receiver.
> >>
> >> These packets never get ack'ed:
> >> 46320:47548(1228)
> >> 50004:51232(1228)
> >> 53688:54916(1228)
> >> 57372:58600(1228)
> >> 61056:62284(1228)
> >>
> >> On host A I dumped the following:
> >>
> >> tcpdump -netttvvi gif6
> >>
> >>> 1129985378.941282 AF 2 52: IP (tos 0x8, ttl  64, id 41637, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 45092 win 32542 <nop,nop,timestamp 574090240
> >>> 490857876>
> >>> 1129985378.952628 AF 2 1280: IP (tos 0x8, ttl  64, id 14004, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 45092:46320(1228) ack 1330 win 33156 <nop,nop,timestamp 490857901
> >>> 574090210>
> >>> 1129985378.952657 AF 2 52: IP (tos 0x8, ttl  64, id 23243, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 46320 win 33156 <nop,nop,timestamp 574090251
> >>> 490857901>
> >>> 1129985378.958250 AF 2 1280: IP (tos 0x8, ttl  64, id 4306, offset 0,
> >>> flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 46320:47548(1228) ack 1330 win 33156 <nop,nop,timestamp 490857901
> >>> 574090210>
> >>> 1129985378.971118 AF 2 1280: IP (tos 0x8, ttl  64, id 33534, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 47548:48776(1228) ack 1330 win 33156 <nop,nop,timestamp 490857920
> >>> 574090229>
> >>> 1129985378.971137 AF 2 52: IP (tos 0x8, ttl  64, id 60095, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 48776 win 32542 <nop,nop,timestamp 574090270
> >>> 490857901>
> >>> 1129985378.982488 AF 2 1280: IP (tos 0x8, ttl  64, id 11459, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 48776:50004(1228) ack 1330 win 33156 <nop,nop,timestamp 490857931
> >>> 574090240>
> >>> 1129985378.982516 AF 2 52: IP (tos 0x8, ttl  64, id 33184, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 50004 win 33156 <nop,nop,timestamp 574090281
> >>> 490857931>
> >>> 1129985378.987989 AF 2 1280: IP (tos 0x8, ttl  64, id 54180, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 50004:51232(1228) ack 1330 win 33156 <nop,nop,timestamp 490857931
> >>> 574090240>
> >>> 1129985378.994231 AF 2 1280: IP (tos 0x8, ttl  64, id 24535, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 51232:52460(1228) ack 1330 win 33156 <nop,nop,timestamp 490857942
> >>> 574090251>
> >>> 1129985378.994250 AF 2 52: IP (tos 0x8, ttl  64, id 30647, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 52460 win 32542 <nop,nop,timestamp 574090293
> >>> 490857931>
> >>> 1129985379.012101 AF 2 1280: IP (tos 0x8, ttl  64, id 61397, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 52460:53688(1228) ack 1330 win 33156 <nop,nop,timestamp 490857960
> >>> 574090270>
> >>> 1129985379.012132 AF 2 52: IP (tos 0x8, ttl  64, id 60550, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 53688 win 33156 <nop,nop,timestamp 574090311
> >>> 490857960>
> >>> 1129985379.017754 AF 2 1280: IP (tos 0x8, ttl  64, id 28408, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 53688:54916(1228) ack 1330 win 33156 <nop,nop,timestamp 490857961
> >>> 574090270>
> >>> 1129985379.023720 AF 2 1280: IP (tos 0x8, ttl  64, id 27558, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 54916:56144(1228) ack 1330 win 33156 <nop,nop,timestamp 490857972
> >>> 574090281>
> >>> 1129985379.023741 AF 2 52: IP (tos 0x8, ttl  64, id 21502, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 56144 win 32542 <nop,nop,timestamp 574090322
> >>> 490857961>
> >>> 1129985379.035333 AF 2 1280: IP (tos 0x8, ttl  64, id 18885, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 56144:57372(1228) ack 1330 win 33156 <nop,nop,timestamp 490857984
> >>> 574090293>
> >>> 1129985379.035362 AF 2 52: IP (tos 0x8, ttl  64, id 59875, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 57372 win 33156 <nop,nop,timestamp 574090334
> >>> 490857984>
> >>> 1129985379.040830 AF 2 1280: IP (tos 0x8, ttl  64, id 37252, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 57372:58600(1228) ack 1330 win 33156 <nop,nop,timestamp 490857984
> >>> 574090293>
> >>> 1129985379.046576 AF 2 1280: IP (tos 0x8, ttl  64, id 18349, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 58600:59828(1228) ack 1330 win 33156 <nop,nop,timestamp 490857984
> >>> 574090293>
> >>> 1129985379.046595 AF 2 52: IP (tos 0x8, ttl  64, id 43697, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 59828 win 32542 <nop,nop,timestamp 574090345
> >>> 490857984>
> >>> 1129985379.064961 AF 2 1280: IP (tos 0x8, ttl  64, id 38300, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 59828:61056(1228) ack 1330 win 33156 <nop,nop,timestamp 490858013
> >>> 574090322>
> >>> 1129985379.064993 AF 2 52: IP (tos 0x8, ttl  64, id 47539, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 61056 win 33156 <nop,nop,timestamp 574090364
> >>> 490858013>
> >>> 1129985379.070688 AF 2 1280: IP (tos 0x8, ttl  64, id 30345, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 61056:62284(1228) ack 1330 win 33156 <nop,nop,timestamp 490858013
> >>> 574090322>
> >>> 1129985379.076184 AF 2 1280: IP (tos 0x8, ttl  64, id 37536, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 62284:63512(1228) ack 1330 win 33156 <nop,nop,timestamp 490858014
> >>> 574090322>
> >>> 1129985379.076202 AF 2 52: IP (tos 0x8, ttl  64, id 34201, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 63512 win 32542 <nop,nop,timestamp 574090375
> >>> 490858013>
> >>> 1129985379.081680 AF 2 1280: IP (tos 0x8, ttl  64, id 20637, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 63512:64740(1228) ack 1330 win 33156 <nop,nop,timestamp 490858025
> >>> 574090334>
> >>> 1129985379.081709 AF 2 52: IP (tos 0x8, ttl  64, id 59866, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 64740 win 33156 <nop,nop,timestamp 574090380
> >>> 490858025>
> >>> 1129985379.087678 AF 2 1280: IP (tos 0x8, ttl  64, id 35213, offset
> >>> 0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: .
> >>> 64740:65968(1228) ack 1330 win 33156 <nop,nop,timestamp 490858036
> >>> 574090345>
> >>> 1129985379.186906 AF 2 52: IP (tos 0x8, ttl  64, id 2465, offset 0,
> >>> flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp
> >>> sum ok] 1330:1330(0) ack 65968 win 33156 <nop,nop,timestamp 574090486
> >>> 490858036>
> >>
> >> tcpdump -netttvvi em1 host E.F.G.H
> >>
> >>> 1129985379.064825 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype
> >>> IPv4 (0x0800), length 1366: IP (tos 0x8, ttl  61, id 45003, offset 0,
> >>> flags [none], length: 1352) E.F.G.H > A.B.C.D:
> >>> ESP(spi=0x0e0dffaa,seq=0x3e)
> >>> 1129985379.065024 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype
> >>> IPv4 (0x0800), length 142: IP (tos 0x8, ttl  64, id 1195, offset 0,
> >>> flags [none], length: 128) A.B.C.D > E.F.G.H:
> >>> ESP(spi=0x029a41b4,seq=0x2f)
> >>> 1129985379.070572 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype
> >>> IPv4 (0x0800), length 1366: IP (tos 0x8, ttl  61, id 36820, offset 0,
> >>> flags [none], length: 1352) E.F.G.H > A.B.C.D:
> >>> ESP(spi=0x0e0dffaa,seq=0x3f)
> >>> 1129985379.076069 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype
> >>> IPv4 (0x0800), length 1366: IP (tos 0x8, ttl  61, id 44971, offset 0,
> >>> flags [none], length: 1352) E.F.G.H > A.B.C.D:
> >>> ESP(spi=0x0e0dffaa,seq=0x40)
> >>> 1129985379.076233 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype
> >>> IPv4 (0x0800), length 142: IP (tos 0x8, ttl  64, id 56964, offset 0,
> >>> flags [none], length: 128) A.B.C.D > E.F.G.H:
> >>> ESP(spi=0x029a41b4,seq=0x30)
> >>> 1129985379.081565 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype
> >>> IPv4 (0x0800), length 1366: IP (tos 0x8, ttl  61, id 24742, offset 0,
> >>> flags [none], length: 1352) E.F.G.H > A.B.C.D:
> >>> ESP(spi=0x0e0dffaa,seq=0x41)
> >>> 1129985379.081741 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype
> >>> IPv4 (0x0800), length 142: IP (tos 0x8, ttl  64, id 9390, offset 0,
> >>> flags [none], length: 128) A.B.C.D > E.F.G.H:
> >>> ESP(spi=0x029a41b4,seq=0x31)
> >>> 1129985379.087562 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype
> >>> IPv4 (0x0800), length 1366: IP (tos 0x8, ttl  61, id 48065, offset 0,
> >>> flags [none], length: 1352) E.F.G.H > A.B.C.D:
> >>> ESP(spi=0x0e0dffaa,seq=0x42)
> >>> 1129985379.186945 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype
> >>> IPv4 (0x0800), length 142: IP (tos 0x8, ttl  64, id 36315, offset 0,
> >>> flags [none], length: 128) A.B.C.D > E.F.G.H:
> >>> ESP(spi=0x029a41b4,seq=0x32)
> >>
> >> If I'm not misleaded, this also doesn't show any errors except the
> >> missing ack's. host B just stops sending. If there's an ack missing,
> >> doesn't have the sending host to just repeat the un-ack'ed packet?
> >>
> >> The IPSec tunnel does not die. Even shortly after the (scp) transfer
> >> stalls the tunnel itself is still usable (for small amounts of data). To
> >> make it more worse, when disabling pf at the senders side, the transfer
> >> works. I've tripple checked pflog for denied packets on both sides but
> >> pf didn't filter any packets out.
> >>
> >> When disabling the IPSec rules using `setkey -F; setkey -FP' on the
> >> tunnel for a moment, the scp transfer does not stall. So it's not a gif
> >> issue.
> >>
> >> It doesn't seem to be an MTU issue (pf has also the rule 'scrub in/out
> >> all no-df'), but what kind of issue is that?? Has anybody ever
> >> experienced similar things? Or am I misinterpreting the tcpdump output?
> >>
> >>
> >> Any help and hint is appreciated! Without an error message I'm lost.
> >>
> >> Volker
> >>
> >> _______________________________________________
> >> freebsd-net at freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> >> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"

-- 
/"\  Best regards,                      | mlaier at freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier at EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20051023/040bd5d5/attachment.bin