Re: iocage, vnet jail does not go outside

From: Michael Gmelin <freebsd_at_grem.de>
Date: Fri, 23 Jul 2021 19:51:42 +0200
On Fri, 23 Jul 2021 18:36:25 +0200
Jacques Foucry <jacques+freebsd_at_foucry.net> wrote:

> Hello friends,
> 
> I'm turing crazy.
> 
> I made a new jail ,on my hosted system using iocage.
> 
> Here is the config.json file:
> 
> more config.json
> {
>     "allow_mount": 1,
>     "allow_mount_devfs": 1,
>     "allow_mount_nullfs": 1,
>     "allow_mount_procfs": 1,
>     "allow_mount_tmpfs": 1,
>     "allow_mount_zfs": 1,
>     "allow_raw_sockets": 1,
>     "allow_socket_af": 1,
>     "allow_sysvipc": 1,
>     "bpf": 1,
>     "cloned_release": "13.0-RELEASE",
>     "defaultrouter": "10.0.10.1",
>     "defaultrouter6": "auto",
>     "dhcp": 0,
>     "host_hostname": "examplejail",
>     "host_hostuuid": "examplejail",
>     "ip4_addr": "vnet0|10.0.10.23/24",
>     "ip6_addr": "vnet0|2a01:4f9:4a:1fd8::23",
>     "jail_zfs_dataset": "iocage/jails/examplejail/data",
>     "last_started": "2021-07-23 15:11:28",
>     "nat": 0,
>     "release": "13.0-RELEASE-p3",
>     "vnet": 1,
>     "vnet0_mac": "b42e999c5bca b42e999c5bcb",
>     "vnet_default_interface": "auto"
> }
> 
> The jail's ifconfig:
> 
> ifconfig
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 	inet6 ::1 prefixlen 128
> 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> 	inet 127.0.0.1 netmask 0xff000000
> 	groups: lo
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> pflog0: flags=0<> metric 0 mtu 33160
> 	groups: pflog
> epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 1500 options=8<VLAN_MTU>
> 	ether b4:2e:99:9c:5b:cb
> 	hwaddr 02:ae:46:07:62:0b
> 	inet 10.0.10.23 netmask 0xffffff00 broadcast 10.0.10.255
> 	inet6 2a01:4f9:4a:1fd8::23 prefixlen 64
> 	inet6 fe80::b62e:99ff:fe9c:5bcb%epair0b prefixlen 64 scopeid
> 0x3 groups: epair
> 	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> 	status: active
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> 
> The jail's netstat:
> 
> netstat -rn
> Routing tables
> 
> Internet:
> Destination        Gateway            Flags     Netif Expire
> default            10.0.10.1          UGS     epair0b
> 10.0.10.0/24       link#3             U       epair0b
> 10.0.10.23         link#3             UHS         lo0
> 127.0.0.1          link#1             UH          lo0
> 
> Internet6:
> Destination                       Gateway                       Flags
>     Netif Expire ::/96                             ::1
>            UGRS        lo0 default
> fe80::1%epair0b               UGS     epair0b ::1
>           link#1                        UHS         lo0
> ::ffff:0.0.0.0/96                 ::1                           UGRS
>       lo0 2a01:4f9:4a:1fd8::/64             link#3
>     U       epair0b 2a01:4f9:4a:1fd8::23              link#3
>               UHS         lo0 fe80::/10                         ::1
>                         UGRS        lo0 fe80::%lo0/64
>     link#1                        U           lo0 fe80::1%lo0
>               link#1                        UHS         lo0
> fe80::%epair0b/64                 link#3                        U
>   epair0b fe80::b62e:99ff:fe9c:5bcb%epair0b link#3
>     UHS         lo0 ff02::/16
> 
> On the host, the ifconfig (note thereis a lot of old fashion jails):
> 
> ifconfig
> em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
> ether b4:2e:99:6a:80:9d inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
> 	inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
> 	inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
> 	inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
> 	media: Ethernet autoselect (1000baseT <full-duplex>)
> 	status: active
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 	inet6 ::1 prefixlen 128
> 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> 	inet 127.0.0.1 netmask 0xff000000
> 	inet 127.0.12.1 netmask 0xff000000
> 	inet 127.0.1.5 netmask 0xffffffff
> 	inet 127.0.1.11 netmask 0xffffffff
> 	inet 127.0.1.12 netmask 0xffffffff
> 	inet 127.0.1.15 netmask 0xffffffff
> 	inet 127.0.1.16 netmask 0xffffffff
> 	inet 127.0.1.18 netmask 0xffffffff
> 	inet 127.0.1.19 netmask 0xffffffff
> 	inet 127.0.1.21 netmask 0xffffffff
> 	inet 127.0.1.22 netmask 0xffffffff
> 	inet 127.0.1.25 netmask 0xffffffff
> 	inet 127.0.1.14 netmask 0xffffffff
> 	inet 127.0.1.29 netmask 0xffffffff
> 	inet 127.0.1.17 netmask 0xffffffff
> 	groups: lo
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 	inet 192.168.12.1 netmask 0xffffff00
> 	inet 192.168.12.5 netmask 0xffffffff
> 	inet 192.168.12.11 netmask 0xffffff00
> 	inet 192.168.12.12 netmask 0xffffff00
> 	inet 192.168.12.15 netmask 0xffffff00
> 	inet 192.168.12.16 netmask 0xffffff00
> 	inet 192.168.12.18 netmask 0xffffff00
> 	inet 192.168.12.19 netmask 0xffffff00
> 	inet 192.168.12.21 netmask 0xffffff00
> 	inet 192.168.12.22 netmask 0xffffff00
> 	inet 192.168.12.25 netmask 0xffffff00
> 	inet 192.168.12.14 netmask 0xffffff00
> 	inet 192.168.12.29 netmask 0xffffff00
> 	inet 192.168.12.17 netmask 0xffffff00
> 	groups: lo
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> pflog0: flags=100<PROMISC> metric 0 mtu 33160
> 	groups: pflog
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 1500 description: jails-bridge
> 	ether 58:9c:fc:10:ed:66
> 	inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255
> 	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> 	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> 	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> 	member: vnet0.655
> flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6
> priority 128 path cost 2000 member: em0
> flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1
> priority 128 path cost 20000 groups: bridge
> 	nd6 options=9<PERFORMNUD,IFDISABLED>
> vnet0.655: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500 description: associated with jail: examplejail as
> nic: epair0b options=8<VLAN_MTU>
> 	ether b4:2e:99:9c:5b:ca
> 	hwaddr 02:ae:46:07:62:0a
> 	groups: epair
> 	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> 	status: active
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> 
> And host's netstat (again with many old fashion jail):
> 
> netstat -rn
> Routing tables
> 
> Internet:
> Destination        Gateway            Flags     Netif Expire
> default            95.217.83.193      UGS         em0
> 10.0.10.0/24       link#5             U       bridge0
> 10.0.10.1          link#5             UHS         lo0
> 95.217.83.192/26   link#1             U           em0
> 95.217.83.231      link#1             UHS         lo0
> 127.0.0.1          link#2             UH          lo0
> 127.0.1.5          link#2             UH          lo0
> 127.0.1.11         link#2             UH          lo0
> 127.0.1.12         link#2             UH          lo0
> 127.0.1.14         link#2             UH          lo0
> 127.0.1.15         link#2             UH          lo0
> 127.0.1.16         link#2             UH          lo0
> 127.0.1.17         link#2             UH          lo0
> 127.0.1.18         link#2             UH          lo0
> 127.0.1.19         link#2             UH          lo0
> 127.0.1.21         link#2             UH          lo0
> 127.0.1.22         link#2             UH          lo0
> 127.0.1.25         link#2             UH          lo0
> 127.0.1.29         link#2             UH          lo0
> 127.0.12.1         link#2             UH          lo0
> 192.168.12.1       link#3             UH          lo1
> 192.168.12.5       link#3             UH          lo1
> 192.168.12.11      link#3             UH          lo1
> 192.168.12.12      link#3             UH          lo1
> 192.168.12.14      link#3             UH          lo1
> 192.168.12.15      link#3             UH          lo1
> 192.168.12.16      link#3             UH          lo1
> 192.168.12.17      link#3             UH          lo1
> 192.168.12.18      link#3             UH          lo1
> 192.168.12.19      link#3             UH          lo1
> 192.168.12.21      link#3             UH          lo1
> 192.168.12.22      link#3             UH          lo1
> 192.168.12.25      link#3             UH          lo1
> 192.168.12.29      link#3             UH          lo1
> 
> Internet6:
> Destination                       Gateway                       Flags
>     Netif Expire ::/96                             ::1
>            UGRS        lo0 default
> fe80::1%em0                   UGS         em0 ::1
>           link#2                        UHS         lo0
> ::ffff:0.0.0.0/96                 ::1                           UGRS
>       lo0 2a01:4f9:4a:1fd8::/64             link#1
>     U           em0 2a01:4f9:4a:1fd8::2               link#1
>               UHS         lo0 2a01:4f9:4a:1fd8::5
> link#1                        UHS         lo0 2a01:4f9:4a:1fd8::11
>           link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::12              link#1                        UHS
>       lo0 2a01:4f9:4a:1fd8::14              link#1
>     UHS         lo0 2a01:4f9:4a:1fd8::15              link#1
>               UHS         lo0 2a01:4f9:4a:1fd8::16
> link#1                        UHS         lo0 2a01:4f9:4a:1fd8::17
>           link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::18              link#1                        UHS
>       lo0 2a01:4f9:4a:1fd8::19              link#1
>     UHS         lo0 2a01:4f9:4a:1fd8::21              link#1
>               UHS         lo0 2a01:4f9:4a:1fd8::22
> link#1                        UHS         lo0 2a01:4f9:4a:1fd8::25
>           link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::29              link#1                        UHS
>       lo0 fe80::/10                         ::1
>     UGRS        lo0 fe80::%em0/64                     link#1
>               U           em0 fe80::b62e:99ff:fe6a:809d%em0
> link#1                        UHS         lo0 fe80::%lo0/64
>           link#2                        U           lo0 fe80::1%lo0
>                     link#2                        UHS         lo0
> ff02::/16                         ::1                           UGRS
>       lo0
> 
> The bridge0 had the em0 and vnet0:655 interfaces.
> 
> From the jail in can ping oustside world:
> 
> ping google.ca
> PING6(56=40+8+8 bytes) 2a01:4f9:4a:1fd8::23 -->
> 2a00:1450:400f:803::2003 16 bytes from 2a00:1450:400f:803::2003,
> icmp_seq=0 hlim=118 time=7.927 ms 16 bytes from
> 2a00:1450:400f:803::2003, icmp_seq=1 hlim=118 time=7.800 ms 16 bytes
> from 2a00:1450:400f:803::2003, icmp_seq=2 hlim=118 time=7.798 ms ^C
> --- google.ca ping6 statistics ---
> 3 packets transmitted, 3 packets received, 0.0% packet loss
> round-trip min/avg/max/std-dev = 7.798/7.842/7.927/0.061 ms
> 
> The problem is, I cannot ssh to an external computer (for example, my
> nextcloud hosted at home):
> 
> ssh -vvv nextcloud.foucry.net -p2250
> OpenSSH_7.9p1, OpenSSL 1.1.1k-freebsd  25 Mar 2021
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug2: resolving "nextcloud.foucry.net" port 2250
> debug2: ssh_connect_direct
> debug1: Connecting to nextcloud.foucry.net
> [2a01:e0a:434:44e0:ff:60ff:feba:b582] port 2250. debug1: connect to
> address 2a01:e0a:434:44e0:ff:60ff:feba:b582 port 2250: Operation
> timed out debug1: Connecting to nextcloud.foucry.net [82.65.174.130]
> port 2250. debug1: connect to address 82.65.174.130 port 2250:
> Operation timed out ssh: connect to host nextcloud.foucry.net port
> 2250: Operation timed out
> 
> What's look strange (for me) is the traceroute (using ipv4):
> 
> traceroute nextcloud.foucry.net
> traceroute to nextcloud.foucry.net (82.65.174.130), 64 hops max, 40
> byte packets 1  10.0.10.1 (10.0.10.1)  0.086 ms  0.051 ms  0.037 ms
>  2  static.193.83.217.95.clients.your-server.de (95.217.83.193)
> 0.451 ms  0.571 ms  0.392 ms 3  core32.hel1.hetzner.com
> (213.239.252.97)  11.621 ms core31.hel1.hetzner.com (213.239.252.93)
> 1.812 ms core32.hel1.hetzner.com (213.239.252.97)  2.793 ms
>  4  core9.fra.hetzner.com (213.239.224.166)  21.295 ms
>     core8.fra.hetzner.com (213.239.224.149)  20.730 ms
>     core9.fra.hetzner.com (213.239.224.170)  20.333 ms
>  5  core4.fra.hetzner.com (213.239.245.85)  28.499 ms
>     core4.fra.hetzner.com (213.239.224.177)  20.507 ms  22.850 ms
>  6  * * *
>  7  * * *
>  8  * * *
>  9  * * *
> 10  * * *
> 11  * * *
> 12  * * *
> 13  *^C
> 
> 
> Look's like something wrong on the way, but I could connect on the
> same host form any other jails.
> 
> 
> There is for me a mysterious behaviiors that I can't understand. 
> 
> Any help will be appreciate.
> 
> Thanks for reading me, and the time your spend on my problem.

You need to enable some sort of NAT at your end, e.g. using pf. Traffic
is leaving your host on a private IP.

-m

-- 
Michael Gmelin
Received on Fri Jul 23 2021 - 17:51:42 UTC

Original text of this message