Issue with packets routing/forwarding

Hello,

Cross posting because I am not sure where I am wrong here.

I have a setup with some jails configured to use a dedicated virtual interface
and with alternate routing tables/fibs. This is running on FreeBSD 11.4 amd64.

The host has dual wan configuration. One adsl line via a router and one 4g via
android phone. The problem I am facing is that I can use only one wan at a time
depending on the combo routing/pf I use. 

I will go with the configuratin I would prefer to make it work. 

Here some config parts
> # cat /etc/rc.local
> ...
> /sbin/ifconfig lo100 create >/dev/null
> /sbin/ifconfig lo100 name vsw0 >/dev/null
> /sbin/ifconfig vsw0 fib 1 >/dev/null
> /sbin/route add default -iface vsw0 -fib 1 >/dev/null
> ...

One of my jail's config relevant part
> ...
>        interface = "vsw0";
>        ip4.addr += "vsw0|192.168.1.92/32";
> ...
>        exec.fib = 1;
>...

At the top of my pf.conf file, I have these lines 
> ...
> no nat  on $VSW from $proxout
> nat     on $phone_if tag PROXOUT tagged PROXOUTNAT -> ( $phone_if )
> nat     on $lan_if tag PROXOUT tagged PROXOUTNAT -> $lan_ip
> 
> pass out log quick on $VSW \
>   proto tcp from $proxout to port {80, 443} user 100 tag PROXOUT100 no state
> pass in log quick on $VSW tagged PROXOUT100 tag PROXOUTNAT rtable 0
> pass out log quick on $phone_if tagged PROXOUT rtable 0
> pass out log quick on $lan_if tagged PROXOUT rtable 0
> 
> block log quick from 109.0.64.169
> block log quick to 109.0.64.169


In this configuration, I can only use the secondary/backup (4G) line from the
jail. This is done via routing some IPs via ue0 interface. When I delete these
routes or change the routing to specifically use the adsl line (so same as
default route), traffic is not coming back to the process but I can see
something coming back to the host. 

> 19:45:46.210775 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0
> 19:45:49.209728 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0
> 19:45:49.216661 IP 109.0.64.169.80 > 192.168.1.50.57922: tcp 0
> 19:45:49.216816 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0
> 19:45:49.217280 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:49.452641 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:49.716200 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:50.036820 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:50.216261 IP 109.0.64.169.80 > 192.168.1.50.57922: tcp 0
> 19:45:50.216355 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0
> 19:45:50.476754 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:51.156785 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:52.216646 IP 109.0.64.169.80 > 192.168.1.50.57922: tcp 0
> 19:45:52.216725 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0
> 19:45:52.316836 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:54.457517 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:55.964243 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0
> 19:45:55.971939 IP 109.0.64.169.80 > 192.168.1.50.57922: tcp 0
> 19:45:58.498646 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 449
> 19:45:58.511970 IP 109.0.64.169.80 > 192.168.1.50.57922: tcp 1448
> 19:45:58.512087 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0
> 19:45:58.512878 IP 109.0.64.169.80 > 192.168.1.50.57922: tcp 1168
> 19:45:58.512916 IP 192.168.1.50.57922 > 109.0.64.169.80: tcp 0

Log from pf:
> 2021-09-09 19:45:46.210601 rule 0/0(match) [uid 100]: pass out on vsw0:
> 192.168.1.92.13153 > 109.0.64.169.80: Flags [S], seq 3268104299, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 1426589561 ecr 0], length 0
> 2021-09-09 19:45:46.210670 rule 2/0(match): pass in on vsw0:
> 192.168.1.92.13153 > 109.0.64.169.80: Flags [S], seq 3268104299, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 1426589561 ecr 0], length 0
> 2021-09-09 19:45:46.210746 rule 4/0(match): pass out on em0:
> 192.168.1.50.57922 > 109.0.64.169.80: Flags [S], seq 3268104299, win 65535,
> options [mss 16344,nop,wscale 6,sackOK,TS val 1426589561 ecr 0], length 0

In addition, traffic from my lan is ok and from host too. Both are using the
adsl line.

As I said I have another configuration example where this is the opposite.
Traffic from adsl line is ok but from 4g timeouts. I don't think that's a
problem with the jail as swapping configuration doesn't touch jail's conf. 

I don't really know where to look further. I try for some days now to make
understand what happens. 
Maybe someone have an idea. 

Thanks,
K.