Re: Issue with packets routing/forwarding
- In reply to: kaycee gb : "Re: Issue with packets routing/forwarding"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 10 Oct 2021 11:28:47 UTC
Hello, I'm back to this case. It makes me crazy as I don't understand why it does not works. I'll try to describe the problem in a different manner. My topology: I have FreeBSD 11.4 host configured as a router/fw/nat with different jails set and multiple FIBs. . ___Host_______________. | | |ifaces: lo0, vsw0, ... | | _______ | 192.168.1.1 | | jail1 | | em0 |------->|adsl modem/router| | | jail2 | | | | jailn | | ue0 |------->| 4g modem/router | | |_______| | 192.168.42.129 | _____ | |____| em1 |____________| | LAN -----| I have a jail available from local lan and configured to attach to vsw0. Jail's conf (relevant part): > ... > interface = "vsw0"; > ip4.addr += "vsw0|192.168.1.92/32"; > ... > exec.fib = 0; >... vsw0 configuration: ># ifconfig vsw0 >vsw0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > inet6 fe80::1%vsw0 prefixlen 64 scopeid 0xc > inet 192.168.1.92 netmask 0xffffffff > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> > fib: 0 > groups: lo My pf configuration is like this (top lines for translation and packet filtering sections): >#### fib 1 >no nat log on $VSW from $proxout >nat log on ue0 tagged P100 -> ($phone_if ) >nat log on em0 tagged P100 -> $lan_ip >#### fib 1: end > >#### fib 0 >nat log on em0 from $proxout tag PROXOUTNAT -> $lan_ip >nat log on $phone_if from $proxout tag PROXOUTNAT -> ( $phone_if ) >#### fib 0: end ... >#### fib 1 > pass out log quick on $VSW proto tcp from $proxout to port {80, 443} >user 100 tag P100 no state pass in log quick on $VSW tagged P100 rtable 0 > > pass out quick on em0 tagged P100 rtable 0 > pass out quick on ue0 tagged P100 rtable 0 >#### fib 1: end > >#### fib 0 > pass out quick on em0 tagged PROXOUTNAT > pass out quick on ue0 tagged PROXOUTNAT >#### fib 0: end > >block log quick from 109.0.64.169 >block log quick to 109.0.64.169 So, in a setup where everything is configured to use em0/192.168.1.1 as default route, it's a working setup > 2021-10-10 12:06:04.419095 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.55998 > 109.0.64.169.80: Flags [S], seq 817941716, win 65535, > options [mss 1460,nop,wscale 6,sackOK ,TS val 567386157 ecr 0], length 0 > 2021-10-10 12:06:04.745475 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.59942 > 109.0.64.169.80: Flags [S], seq 178756216, win 65535, > options [mss 1460,nop,wscale 6,sackOK ,TS val 1682297331 ecr 0], length 0 > 2021-10-10 12:06:04.841617 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.58758 > 52.84.228.93.443: Flags [S], seq 3657906762, win 65535, > options [mss 1460,nop,wscale 6,sack OK,TS val 498309471 ecr 0], length 0 > 2021-10-10 12:06:06.758062 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.54067 > 69.16.175.42.80: Flags [S], seq 1689219730, win 65535, > options [mss 1460,nop,wscale 6,sackO K,TS val 3560654970 ecr 0], length 0 > 2021-10-10 12:06:07.499880 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.52905 > 93.20.64.1.80: Flags [S], seq 2950807804, win 65535, > options [mss 1460,nop,wscale 6,sackOK, TS val 1714325115 ecr 0], length 0 > 2021-10-10 12:06:07.500191 rule 3/0(match) [uid 100]: nat out on em0: > 192.168.1.50.54440 > 93.20.64.1.80: Flags [S], seq 2349377925, win 65535, > options [mss 1460,nop,wscale 6,sackOK, TS val 176674686 ecr 0], length 0 Then, I want to simulate an adsl failure, so I route wanted traffic via ue0/4g. > # A ; route add 109.0.64.169 $defr && route add 69.16.175.42 $defr && route > add 69.16.175.10 $defr && route add 93.20.64.1 $defr > add host 109.0.64.169: gateway 192.168.42.129 fib 0 > add host 69.16.175.42: gateway 192.168.42.129 fib 0 > add host 69.16.175.10: gateway 192.168.42.129 fib 0 > add host 93.20.64.1: gateway 192.168.42.129 fib 0 Here, my setup is working also correctly: > 2021-10-10 12:19:03.752618 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.61024 > 109.0.64.169.80: Flags [S], seq 3895230720, win 65535, > options [mss 1460,nop,wscale 6,sack OK,TS val 434432976 ecr 0], length 0 > 2021-10-10 12:19:07.217710 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.63790 > 109.0.64.169.80: Flags [S], seq 1094822977, win 65535, > options [mss 1460,nop,wscale 6,sack OK,TS val 3161827496 ecr 0], length 0 > 2021-10-10 12:19:07.219786 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.64426 > 69.16.175.10.80: Flags [S], seq 3835135791, win 65535, > options [mss 1460,nop,wscale 6,sack OK,TS val 2326859600 ecr 0], length 0 > 2021-10-10 12:19:07.221208 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.62982 > 109.0.64.169.80: Flags [S], seq 2534996192, win 65535, > options [mss 1460,nop,wscale 6,sack OK,TS val 3218952486 ecr 0], length 0 > 2021-10-10 12:19:07.765953 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.55202 > 93.20.64.1.80: Flags [S], seq 2614725609, win 65535, > options [mss 1460,nop,wscale 6,sackOK ,TS val 370574760 ecr 0], length 0 > 2021-10-10 12:19:08.057355 rule 4/0(match) [uid 100]: nat out on ue0: > 192.168.42.93.58890 > 93.20.64.1.80: Flags [S], seq 3356530101, win 65535, > options [mss 1460,nop,wscale 6,sackOK ,TS val 1210627653 ecr 0], length 0 Then, I want to isolate this service and make it usable only when pf is enabled. So, I stop this jail, change his configuration to: > ... > exec.fib = 1; >... Change vsw0 configuration to: > # ifconfig vsw0 fib 1 And start the jail again. At this point my routes via ue0 are still active, and the setup is working. > 2021-10-10 12:32:35.790102 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17138 > 109.0.64.169.80: Flags [S], seq 3903052933, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0 > 2021-10-10 12:32:35.790155 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17138 > 109.0.64.169.80: Flags [S], seq 3903052933, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0 > 2021-10-10 12:32:35.790211 rule 1/0(match): nat out on ue0: > 192.168.42.93.56331 > 109.0.64.169.80: Flags [S], seq 3903052933, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0 > 2021-10-10 12:32:40.997496 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17141 > 109.0.64.169.80: Flags [S], seq 3383832228, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length 0 > 2021-10-10 12:32:40.997555 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17141 > 109.0.64.169.80: Flags [S], seq 3383832228, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length 0 > 2021-10-10 12:32:40.997619 rule 1/0(match): nat out on ue0: > 192.168.42.93.57246 > 109.0.64.169.80: Flags [S], seq 3383832228, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length 0 > 2021-10-10 12:32:41.002776 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17142 > 109.0.64.169.80: Flags [S], seq 1628607117, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length 0 > 2021-10-10 12:32:41.002835 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17142 > 109.0.64.169.80: Flags [S], seq 1628607117, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length 0 > 2021-10-10 12:32:41.002911 rule 1/0(match): nat out on ue0: > 192.168.42.93.60146 > 109.0.64.169.80: Flags [S], seq 1628607117, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length 0 > 2021-10-10 12:32:41.003179 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17143 > 69.16.175.10.80: Flags [S], seq 3985646764, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length 0 > 2021-10-10 12:32:41.003224 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17143 > 69.16.175.10.80: Flags [S], seq 3985646764, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length 0 > 2021-10-10 12:32:41.003276 rule 1/0(match): nat out on ue0: > 192.168.42.93.60290 > 69.16.175.10.80: Flags [S], seq 3985646764, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length 0 > 2021-10-10 12:32:44.224200 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17149 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0 > 2021-10-10 12:32:44.224257 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17149 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0 > 2021-10-10 12:32:44.224319 rule 1/0(match): nat out on ue0: > 192.168.42.93.57313 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0 > 2021-10-10 12:32:44.297253 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17150 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0 > 2021-10-10 12:32:44.297313 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17150 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0 > 2021-10-10 12:32:44.297368 rule 1/0(match): nat out on ue0: > 192.168.42.93.64517 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0 > 2021-10-10 12:32:44.900273 rule 1/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.17151 > 54.148.181.117.443: Flags [S], seq 1363702430, win > 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], length > 0 > 2021-10-10 12:32:44.900326 rule 2/0(match): pass in on vsw0: > 192.168.1.92.17151 > 54.148.181.117.443: Flags [S], seq 1363702430, win > 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], length > 0 > 2021-10-10 12:32:44.900385 rule 2/0(match): nat out on em0: > 192.168.1.50.51385 > 54.148.181.117.443: Flags [S], seq 1363702430, win > 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], length > 0 The problem begins when I want to stop adsl failure simulation and delete the routes I added before. > # D ; route del 109.0.64.169 ; route del 69.16.175.42 ; route del 69.16.175.10 > ; route del 93.20.64.1 > del host 109.0.64.169 fib 0 > del host 69.16.175.42 fib 0 > del host 69.16.175.10 fib 0 > del host 93.20.64.1 fib 0 From there, the service have no connectivity to the outside anymore. What I see on screen is that it tries to connect until it timeouts. > 2021-10-10 13:04:18.446492 rule 0/0(match) [uid 100]: pass out on vsw0: > 192.168.1.92.53781 > 109.0.64.169.80: Flags [S], seq 2140928614, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length 0 > 2021-10-10 13:04:18.446543 rule 2/0(match): pass in on vsw0: > 192.168.1.92.53781 > 109.0.64.169.80: Flags [S], seq 2140928614, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length 0 > 2021-10-10 13:04:18.446604 rule 2/0(match): nat out on em0: > 192.168.1.50.51537 > 109.0.64.169.80: Flags [S], seq 2140928614, win 65535, > options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length 0 And tcpdump show that he tries again and again without success. > 13:04:18.446626 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:21.451833 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:21.460066 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:21.460221 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:21.460635 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:21.967652 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:22.778014 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:22.982230 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:22.982328 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:24.197788 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:24.982527 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:24.982620 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:26.862180 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:28.982095 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:28.982191 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:31.948459 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:36.982090 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:36.982207 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:40.728729 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:45.415470 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:45.423316 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0 > 13:04:58.085578 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449 > 13:04:58.099480 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 1448 > 13:04:58.099594 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 > 13:04:58.100358 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 1168 > 13:04:58.100391 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0 As we can see, traffic from the address I want to join comes back to host and it seems it disapears somewhere. The strange part is that routing via 4g/ue0 is working and 4g/ue0 is similar to adsl/em0 in physical terms. What could I check else to see what's happening ? Is there a better place maybe to ask questions like this ? Are there other informations needed ? Thanks, K.