Re: Issue with packets routing/forwarding
- In reply to: kaycee gb : "Re: Issue with packets routing/forwarding"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 10 Oct 2021 11:28:47 UTC
Hello,
I'm back to this case. It makes me crazy as I don't understand why it does not
works.
I'll try to describe the problem in a different manner.
My topology:
I have FreeBSD 11.4 host configured as a router/fw/nat with different jails set
and multiple FIBs.
. ___Host_______________.
| |
|ifaces: lo0, vsw0, ... |
| _______ | 192.168.1.1
| | jail1 | | em0 |------->|adsl modem/router|
| | jail2 | |
| | jailn | | ue0 |------->| 4g modem/router |
| |_______| | 192.168.42.129
| _____ |
|____| em1 |____________|
|
LAN -----|
I have a jail available from local lan and configured to attach to vsw0.
Jail's conf (relevant part):
> ...
> interface = "vsw0";
> ip4.addr += "vsw0|192.168.1.92/32";
> ...
> exec.fib = 0;
>...
vsw0 configuration:
># ifconfig vsw0
>vsw0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 fe80::1%vsw0 prefixlen 64 scopeid 0xc
> inet 192.168.1.92 netmask 0xffffffff
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> fib: 0
> groups: lo
My pf configuration is like this (top lines for translation and packet
filtering sections):
>#### fib 1
>no nat log on $VSW from $proxout
>nat log on ue0 tagged P100 -> ($phone_if )
>nat log on em0 tagged P100 -> $lan_ip
>#### fib 1: end
>
>#### fib 0
>nat log on em0 from $proxout tag PROXOUTNAT -> $lan_ip
>nat log on $phone_if from $proxout tag PROXOUTNAT -> ( $phone_if )
>#### fib 0: end
...
>#### fib 1
> pass out log quick on $VSW proto tcp from $proxout to port {80, 443}
>user 100 tag P100 no state pass in log quick on $VSW tagged P100 rtable 0
>
> pass out quick on em0 tagged P100 rtable 0
> pass out quick on ue0 tagged P100 rtable 0
>#### fib 1: end
>
>#### fib 0
> pass out quick on em0 tagged PROXOUTNAT
> pass out quick on ue0 tagged PROXOUTNAT
>#### fib 0: end
>
>block log quick from 109.0.64.169
>block log quick to 109.0.64.169
So, in a setup where everything is configured to use em0/192.168.1.1 as default
route, it's a working setup
> 2021-10-10 12:06:04.419095 rule 3/0(match) [uid 100]: nat out on em0:
> 192.168.1.50.55998 > 109.0.64.169.80: Flags [S], seq 817941716, win 65535,
> options [mss 1460,nop,wscale 6,sackOK ,TS val 567386157 ecr 0], length 0
> 2021-10-10 12:06:04.745475 rule 3/0(match) [uid 100]: nat out on em0:
> 192.168.1.50.59942 > 109.0.64.169.80: Flags [S], seq 178756216, win 65535,
> options [mss 1460,nop,wscale 6,sackOK ,TS val 1682297331 ecr 0], length 0
> 2021-10-10 12:06:04.841617 rule 3/0(match) [uid 100]: nat out on em0:
> 192.168.1.50.58758 > 52.84.228.93.443: Flags [S], seq 3657906762, win 65535,
> options [mss 1460,nop,wscale 6,sack OK,TS val 498309471 ecr 0], length 0
> 2021-10-10 12:06:06.758062 rule 3/0(match) [uid 100]: nat out on em0:
> 192.168.1.50.54067 > 69.16.175.42.80: Flags [S], seq 1689219730, win 65535,
> options [mss 1460,nop,wscale 6,sackO K,TS val 3560654970 ecr 0], length 0
> 2021-10-10 12:06:07.499880 rule 3/0(match) [uid 100]: nat out on em0:
> 192.168.1.50.52905 > 93.20.64.1.80: Flags [S], seq 2950807804, win 65535,
> options [mss 1460,nop,wscale 6,sackOK, TS val 1714325115 ecr 0], length 0
> 2021-10-10 12:06:07.500191 rule 3/0(match) [uid 100]: nat out on em0:
> 192.168.1.50.54440 > 93.20.64.1.80: Flags [S], seq 2349377925, win 65535,
> options [mss 1460,nop,wscale 6,sackOK, TS val 176674686 ecr 0], length 0
Then, I want to simulate an adsl failure, so I route wanted traffic via ue0/4g.
> # A ; route add 109.0.64.169 $defr && route add 69.16.175.42 $defr && route
> add 69.16.175.10 $defr && route add 93.20.64.1 $defr
> add host 109.0.64.169: gateway 192.168.42.129 fib 0
> add host 69.16.175.42: gateway 192.168.42.129 fib 0
> add host 69.16.175.10: gateway 192.168.42.129 fib 0
> add host 93.20.64.1: gateway 192.168.42.129 fib 0
Here, my setup is working also correctly:
> 2021-10-10 12:19:03.752618 rule 4/0(match) [uid 100]: nat out on ue0:
> 192.168.42.93.61024 > 109.0.64.169.80: Flags [S], seq 3895230720, win 65535,
> options [mss 1460,nop,wscale 6,sack OK,TS val 434432976 ecr 0], length 0
> 2021-10-10 12:19:07.217710 rule 4/0(match) [uid 100]: nat out on ue0:
> 192.168.42.93.63790 > 109.0.64.169.80: Flags [S], seq 1094822977, win 65535,
> options [mss 1460,nop,wscale 6,sack OK,TS val 3161827496 ecr 0], length 0
> 2021-10-10 12:19:07.219786 rule 4/0(match) [uid 100]: nat out on ue0:
> 192.168.42.93.64426 > 69.16.175.10.80: Flags [S], seq 3835135791, win 65535,
> options [mss 1460,nop,wscale 6,sack OK,TS val 2326859600 ecr 0], length 0
> 2021-10-10 12:19:07.221208 rule 4/0(match) [uid 100]: nat out on ue0:
> 192.168.42.93.62982 > 109.0.64.169.80: Flags [S], seq 2534996192, win 65535,
> options [mss 1460,nop,wscale 6,sack OK,TS val 3218952486 ecr 0], length 0
> 2021-10-10 12:19:07.765953 rule 4/0(match) [uid 100]: nat out on ue0:
> 192.168.42.93.55202 > 93.20.64.1.80: Flags [S], seq 2614725609, win 65535,
> options [mss 1460,nop,wscale 6,sackOK ,TS val 370574760 ecr 0], length 0
> 2021-10-10 12:19:08.057355 rule 4/0(match) [uid 100]: nat out on ue0:
> 192.168.42.93.58890 > 93.20.64.1.80: Flags [S], seq 3356530101, win 65535,
> options [mss 1460,nop,wscale 6,sackOK ,TS val 1210627653 ecr 0], length 0
Then, I want to isolate this service and make it usable only when pf is
enabled. So, I stop this jail, change his configuration to:
> ...
> exec.fib = 1;
>...
Change vsw0 configuration to:
> # ifconfig vsw0 fib 1
And start the jail again.
At this point my routes via ue0 are still active, and the setup is working.
> 2021-10-10 12:32:35.790102 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.17138 > 109.0.64.169.80: Flags [S], seq 3903052933, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0
> 2021-10-10 12:32:35.790155 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.17138 > 109.0.64.169.80: Flags [S], seq 3903052933, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0
> 2021-10-10 12:32:35.790211 rule 1/0(match): nat out on ue0:
> 192.168.42.93.56331 > 109.0.64.169.80: Flags [S], seq 3903052933, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 134177506 ecr 0], length 0
> 2021-10-10 12:32:40.997496 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.17141 > 109.0.64.169.80: Flags [S], seq 3383832228, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length 0
> 2021-10-10 12:32:40.997555 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.17141 > 109.0.64.169.80: Flags [S], seq 3383832228, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length 0
> 2021-10-10 12:32:40.997619 rule 1/0(match): nat out on ue0:
> 192.168.42.93.57246 > 109.0.64.169.80: Flags [S], seq 3383832228, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 1005814631 ecr 0], length 0
> 2021-10-10 12:32:41.002776 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.17142 > 109.0.64.169.80: Flags [S], seq 1628607117, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length 0
> 2021-10-10 12:32:41.002835 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.17142 > 109.0.64.169.80: Flags [S], seq 1628607117, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length 0
> 2021-10-10 12:32:41.002911 rule 1/0(match): nat out on ue0:
> 192.168.42.93.60146 > 109.0.64.169.80: Flags [S], seq 1628607117, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 2533761821 ecr 0], length 0
> 2021-10-10 12:32:41.003179 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.17143 > 69.16.175.10.80: Flags [S], seq 3985646764, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length 0
> 2021-10-10 12:32:41.003224 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.17143 > 69.16.175.10.80: Flags [S], seq 3985646764, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length 0
> 2021-10-10 12:32:41.003276 rule 1/0(match): nat out on ue0:
> 192.168.42.93.60290 > 69.16.175.10.80: Flags [S], seq 3985646764, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 2836463753 ecr 0], length 0
> 2021-10-10 12:32:44.224200 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.17149 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0
> 2021-10-10 12:32:44.224257 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.17149 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0
> 2021-10-10 12:32:44.224319 rule 1/0(match): nat out on ue0:
> 192.168.42.93.57313 > 93.20.64.1.80: Flags [S], seq 712910303, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 752785833 ecr 0], length 0
> 2021-10-10 12:32:44.297253 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.17150 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0
> 2021-10-10 12:32:44.297313 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.17150 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0
> 2021-10-10 12:32:44.297368 rule 1/0(match): nat out on ue0:
> 192.168.42.93.64517 > 93.20.64.1.80: Flags [S], seq 3451366591, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 923442688 ecr 0], length 0
> 2021-10-10 12:32:44.900273 rule 1/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.17151 > 54.148.181.117.443: Flags [S], seq 1363702430, win
> 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], length
> 0
> 2021-10-10 12:32:44.900326 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.17151 > 54.148.181.117.443: Flags [S], seq 1363702430, win
> 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], length
> 0
> 2021-10-10 12:32:44.900385 rule 2/0(match): nat out on em0:
> 192.168.1.50.51385 > 54.148.181.117.443: Flags [S], seq 1363702430, win
> 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 541777222 ecr 0], length
> 0
The problem begins when I want to stop adsl failure simulation and delete the
routes I added before.
> # D ; route del 109.0.64.169 ; route del 69.16.175.42 ; route del 69.16.175.10
> ; route del 93.20.64.1
> del host 109.0.64.169 fib 0
> del host 69.16.175.42 fib 0
> del host 69.16.175.10 fib 0
> del host 93.20.64.1 fib 0
From there, the service have no connectivity to the outside anymore. What I see
on screen is that it tries to connect until it timeouts.
> 2021-10-10 13:04:18.446492 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.53781 > 109.0.64.169.80: Flags [S], seq 2140928614, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length 0
> 2021-10-10 13:04:18.446543 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.53781 > 109.0.64.169.80: Flags [S], seq 2140928614, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length 0
> 2021-10-10 13:04:18.446604 rule 2/0(match): nat out on em0:
> 192.168.1.50.51537 > 109.0.64.169.80: Flags [S], seq 2140928614, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 4171138223 ecr 0], length 0
And tcpdump show that he tries again and again without success.
> 13:04:18.446626 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:21.451833 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:21.460066 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0
> 13:04:21.460221 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:21.460635 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:21.967652 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:22.778014 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:22.982230 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0
> 13:04:22.982328 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:24.197788 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:24.982527 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0
> 13:04:24.982620 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:26.862180 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:28.982095 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0
> 13:04:28.982191 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:31.948459 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:36.982090 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0
> 13:04:36.982207 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:40.728729 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:45.415470 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:45.423316 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 0
> 13:04:58.085578 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 449
> 13:04:58.099480 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 1448
> 13:04:58.099594 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
> 13:04:58.100358 IP 109.0.64.169.80 > 192.168.1.50.51537: tcp 1168
> 13:04:58.100391 IP 192.168.1.50.51537 > 109.0.64.169.80: tcp 0
As we can see, traffic from the address I want to join comes back to host and
it seems it disapears somewhere.
The strange part is that routing via 4g/ue0 is working and 4g/ue0 is similar to
adsl/em0 in physical terms.
What could I check else to see what's happening ? Is there a better place maybe
to ask questions like this ? Are there other informations needed ?
Thanks,
K.