iocage, vnet jail does not go outside

From: Jacques Foucry <jacques+freebsd_at_foucry.net>
Date: Fri, 23 Jul 2021 18:36:25 +0200
Hello friends,

I'm turing crazy.

I made a new jail ,on my hosted system using iocage.

Here is the config.json file:

more config.json
{
    "allow_mount": 1,
    "allow_mount_devfs": 1,
    "allow_mount_nullfs": 1,
    "allow_mount_procfs": 1,
    "allow_mount_tmpfs": 1,
    "allow_mount_zfs": 1,
    "allow_raw_sockets": 1,
    "allow_socket_af": 1,
    "allow_sysvipc": 1,
    "bpf": 1,
    "cloned_release": "13.0-RELEASE",
    "defaultrouter": "10.0.10.1",
    "defaultrouter6": "auto",
    "dhcp": 0,
    "host_hostname": "examplejail",
    "host_hostuuid": "examplejail",
    "ip4_addr": "vnet0|10.0.10.23/24",
    "ip6_addr": "vnet0|2a01:4f9:4a:1fd8::23",
    "jail_zfs_dataset": "iocage/jails/examplejail/data",
    "last_started": "2021-07-23 15:11:28",
    "nat": 0,
    "release": "13.0-RELEASE-p3",
    "vnet": 1,
    "vnet0_mac": "b42e999c5bca b42e999c5bcb",
    "vnet_default_interface": "auto"
}

The jail's ifconfig:

ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
	groups: pflog
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether b4:2e:99:9c:5b:cb
	hwaddr 02:ae:46:07:62:0b
	inet 10.0.10.23 netmask 0xffffff00 broadcast 10.0.10.255
	inet6 2a01:4f9:4a:1fd8::23 prefixlen 64
	inet6 fe80::b62e:99ff:fe9c:5bcb%epair0b prefixlen 64 scopeid 0x3
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The jail's netstat:

netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.0.10.1          UGS     epair0b
10.0.10.0/24       link#3             U       epair0b
10.0.10.23         link#3             UHS         lo0
127.0.0.1          link#1             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           fe80::1%epair0b               UGS     epair0b
::1                               link#1                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2a01:4f9:4a:1fd8::/64             link#3                        U       epair0b
2a01:4f9:4a:1fd8::23              link#3                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
fe80::%epair0b/64                 link#3                        U       epair0b
fe80::b62e:99ff:fe9c:5bcb%epair0b link#3                        UHS         lo0
ff02::/16

On the host, the ifconfig (note thereis a lot of old fashion jails):

ifconfig
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
	ether b4:2e:99:6a:80:9d
	inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
	inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
	inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
	inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
	inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	inet 127.0.12.1 netmask 0xff000000
	inet 127.0.1.5 netmask 0xffffffff
	inet 127.0.1.11 netmask 0xffffffff
	inet 127.0.1.12 netmask 0xffffffff
	inet 127.0.1.15 netmask 0xffffffff
	inet 127.0.1.16 netmask 0xffffffff
	inet 127.0.1.18 netmask 0xffffffff
	inet 127.0.1.19 netmask 0xffffffff
	inet 127.0.1.21 netmask 0xffffffff
	inet 127.0.1.22 netmask 0xffffffff
	inet 127.0.1.25 netmask 0xffffffff
	inet 127.0.1.14 netmask 0xffffffff
	inet 127.0.1.29 netmask 0xffffffff
	inet 127.0.1.17 netmask 0xffffffff
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 192.168.12.1 netmask 0xffffff00
	inet 192.168.12.5 netmask 0xffffffff
	inet 192.168.12.11 netmask 0xffffff00
	inet 192.168.12.12 netmask 0xffffff00
	inet 192.168.12.15 netmask 0xffffff00
	inet 192.168.12.16 netmask 0xffffff00
	inet 192.168.12.18 netmask 0xffffff00
	inet 192.168.12.19 netmask 0xffffff00
	inet 192.168.12.21 netmask 0xffffff00
	inet 192.168.12.22 netmask 0xffffff00
	inet 192.168.12.25 netmask 0xffffff00
	inet 192.168.12.14 netmask 0xffffff00
	inet 192.168.12.29 netmask 0xffffff00
	inet 192.168.12.17 netmask 0xffffff00
	groups: lo
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: jails-bridge
	ether 58:9c:fc:10:ed:66
	inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0.655 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 6 priority 128 path cost 2000
	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 20000
	groups: bridge
	nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.655: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: examplejail as nic: epair0b
	options=8<VLAN_MTU>
	ether b4:2e:99:9c:5b:ca
	hwaddr 02:ae:46:07:62:0a
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

And host's netstat (again with many old fashion jail):

netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            95.217.83.193      UGS         em0
10.0.10.0/24       link#5             U       bridge0
10.0.10.1          link#5             UHS         lo0
95.217.83.192/26   link#1             U           em0
95.217.83.231      link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0
127.0.1.5          link#2             UH          lo0
127.0.1.11         link#2             UH          lo0
127.0.1.12         link#2             UH          lo0
127.0.1.14         link#2             UH          lo0
127.0.1.15         link#2             UH          lo0
127.0.1.16         link#2             UH          lo0
127.0.1.17         link#2             UH          lo0
127.0.1.18         link#2             UH          lo0
127.0.1.19         link#2             UH          lo0
127.0.1.21         link#2             UH          lo0
127.0.1.22         link#2             UH          lo0
127.0.1.25         link#2             UH          lo0
127.0.1.29         link#2             UH          lo0
127.0.12.1         link#2             UH          lo0
192.168.12.1       link#3             UH          lo1
192.168.12.5       link#3             UH          lo1
192.168.12.11      link#3             UH          lo1
192.168.12.12      link#3             UH          lo1
192.168.12.14      link#3             UH          lo1
192.168.12.15      link#3             UH          lo1
192.168.12.16      link#3             UH          lo1
192.168.12.17      link#3             UH          lo1
192.168.12.18      link#3             UH          lo1
192.168.12.19      link#3             UH          lo1
192.168.12.21      link#3             UH          lo1
192.168.12.22      link#3             UH          lo1
192.168.12.25      link#3             UH          lo1
192.168.12.29      link#3             UH          lo1

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           fe80::1%em0                   UGS         em0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2a01:4f9:4a:1fd8::/64             link#1                        U           em0
2a01:4f9:4a:1fd8::2               link#1                        UHS         lo0
2a01:4f9:4a:1fd8::5               link#1                        UHS         lo0
2a01:4f9:4a:1fd8::11              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::12              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::14              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::15              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::16              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::17              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::18              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::19              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::21              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::22              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::25              link#1                        UHS         lo0
2a01:4f9:4a:1fd8::29              link#1                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em0/64                     link#1                        U           em0
fe80::b62e:99ff:fe6a:809d%em0     link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

The bridge0 had the em0 and vnet0:655 interfaces.

From the jail in can ping oustside world:

ping google.ca
PING6(56=40+8+8 bytes) 2a01:4f9:4a:1fd8::23 --> 2a00:1450:400f:803::2003
16 bytes from 2a00:1450:400f:803::2003, icmp_seq=0 hlim=118 time=7.927 ms
16 bytes from 2a00:1450:400f:803::2003, icmp_seq=1 hlim=118 time=7.800 ms
16 bytes from 2a00:1450:400f:803::2003, icmp_seq=2 hlim=118 time=7.798 ms
^C
--- google.ca ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 7.798/7.842/7.927/0.061 ms

The problem is, I cannot ssh to an external computer (for example, my
nextcloud hosted at home):

ssh -vvv nextcloud.foucry.net -p2250
OpenSSH_7.9p1, OpenSSL 1.1.1k-freebsd  25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "nextcloud.foucry.net" port 2250
debug2: ssh_connect_direct
debug1: Connecting to nextcloud.foucry.net [2a01:e0a:434:44e0:ff:60ff:feba:b582] port 2250.
debug1: connect to address 2a01:e0a:434:44e0:ff:60ff:feba:b582 port 2250: Operation timed out
debug1: Connecting to nextcloud.foucry.net [82.65.174.130] port 2250.
debug1: connect to address 82.65.174.130 port 2250: Operation timed out
ssh: connect to host nextcloud.foucry.net port 2250: Operation timed out

What's look strange (for me) is the traceroute (using ipv4):

traceroute nextcloud.foucry.net
traceroute to nextcloud.foucry.net (82.65.174.130), 64 hops max, 40 byte packets
 1  10.0.10.1 (10.0.10.1)  0.086 ms  0.051 ms  0.037 ms
 2  static.193.83.217.95.clients.your-server.de (95.217.83.193)  0.451 ms  0.571 ms  0.392 ms
 3  core32.hel1.hetzner.com (213.239.252.97)  11.621 ms
    core31.hel1.hetzner.com (213.239.252.93)  1.812 ms
    core32.hel1.hetzner.com (213.239.252.97)  2.793 ms
 4  core9.fra.hetzner.com (213.239.224.166)  21.295 ms
    core8.fra.hetzner.com (213.239.224.149)  20.730 ms
    core9.fra.hetzner.com (213.239.224.170)  20.333 ms
 5  core4.fra.hetzner.com (213.239.245.85)  28.499 ms
    core4.fra.hetzner.com (213.239.224.177)  20.507 ms  22.850 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  *^C


Look's like something wrong on the way, but I could connect on the same host
form any other jails.


There is for me a mysterious behaviiors that I can't understand. 

Any help will be appreciate.

Thanks for reading me, and the time your spend on my problem.
-- 
Jacques Foucry

Received on Fri Jul 23 2021 - 16:36:25 UTC

Original text of this message