Re: iocage, vnet jail does not go outside

From: infoomatic <infoomatic_at_gmx.at>
Date: Fri, 23 Jul 2021 23:06:41 +0200
iocage autoatically creates a bridge with your physical interface and
the vnet interface. Imho this is wrong behaviour so I quit using iocage,
however, there is a workaround, for more info see [1]

Regards,

Robert


[1] https://github.com/iocage/iocage/issues/521


On 23.07.21 18:36, Jacques Foucry wrote:
> Hello friends,
>
> I'm turing crazy.
>
> I made a new jail ,on my hosted system using iocage.
>
> Here is the config.json file:
>
> more config.json
> {
>     "allow_mount": 1,
>     "allow_mount_devfs": 1,
>     "allow_mount_nullfs": 1,
>     "allow_mount_procfs": 1,
>     "allow_mount_tmpfs": 1,
>     "allow_mount_zfs": 1,
>     "allow_raw_sockets": 1,
>     "allow_socket_af": 1,
>     "allow_sysvipc": 1,
>     "bpf": 1,
>     "cloned_release": "13.0-RELEASE",
>     "defaultrouter": "10.0.10.1",
>     "defaultrouter6": "auto",
>     "dhcp": 0,
>     "host_hostname": "examplejail",
>     "host_hostuuid": "examplejail",
>     "ip4_addr": "vnet0|10.0.10.23/24",
>     "ip6_addr": "vnet0|2a01:4f9:4a:1fd8::23",
>     "jail_zfs_dataset": "iocage/jails/examplejail/data",
>     "last_started": "2021-07-23 15:11:28",
>     "nat": 0,
>     "release": "13.0-RELEASE-p3",
>     "vnet": 1,
>     "vnet0_mac": "b42e999c5bca b42e999c5bcb",
>     "vnet_default_interface": "auto"
> }
>
> The jail's ifconfig:
>
> ifconfig
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 	inet6 ::1 prefixlen 128
> 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> 	inet 127.0.0.1 netmask 0xff000000
> 	groups: lo
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> pflog0: flags=0<> metric 0 mtu 33160
> 	groups: pflog
> epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	options=8<VLAN_MTU>
> 	ether b4:2e:99:9c:5b:cb
> 	hwaddr 02:ae:46:07:62:0b
> 	inet 10.0.10.23 netmask 0xffffff00 broadcast 10.0.10.255
> 	inet6 2a01:4f9:4a:1fd8::23 prefixlen 64
> 	inet6 fe80::b62e:99ff:fe9c:5bcb%epair0b prefixlen 64 scopeid 0x3
> 	groups: epair
> 	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> 	status: active
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>
> The jail's netstat:
>
> netstat -rn
> Routing tables
>
> Internet:
> Destination        Gateway            Flags     Netif Expire
> default            10.0.10.1          UGS     epair0b
> 10.0.10.0/24       link#3             U       epair0b
> 10.0.10.23         link#3             UHS         lo0
> 127.0.0.1          link#1             UH          lo0
>
> Internet6:
> Destination                       Gateway                       Flags     Netif Expire
> ::/96                             ::1                           UGRS        lo0
> default                           fe80::1%epair0b               UGS     epair0b
> ::1                               link#1                        UHS         lo0
> ::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
> 2a01:4f9:4a:1fd8::/64             link#3                        U       epair0b
> 2a01:4f9:4a:1fd8::23              link#3                        UHS         lo0
> fe80::/10                         ::1                           UGRS        lo0
> fe80::%lo0/64                     link#1                        U           lo0
> fe80::1%lo0                       link#1                        UHS         lo0
> fe80::%epair0b/64                 link#3                        U       epair0b
> fe80::b62e:99ff:fe9c:5bcb%epair0b link#3                        UHS         lo0
> ff02::/16
>
> On the host, the ifconfig (note thereis a lot of old fashion jails):
>
> ifconfig
> em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
> 	ether b4:2e:99:6a:80:9d
> 	inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
> 	inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
> 	inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
> 	inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
> 	inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
> 	media: Ethernet autoselect (1000baseT <full-duplex>)
> 	status: active
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 	inet6 ::1 prefixlen 128
> 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> 	inet 127.0.0.1 netmask 0xff000000
> 	inet 127.0.12.1 netmask 0xff000000
> 	inet 127.0.1.5 netmask 0xffffffff
> 	inet 127.0.1.11 netmask 0xffffffff
> 	inet 127.0.1.12 netmask 0xffffffff
> 	inet 127.0.1.15 netmask 0xffffffff
> 	inet 127.0.1.16 netmask 0xffffffff
> 	inet 127.0.1.18 netmask 0xffffffff
> 	inet 127.0.1.19 netmask 0xffffffff
> 	inet 127.0.1.21 netmask 0xffffffff
> 	inet 127.0.1.22 netmask 0xffffffff
> 	inet 127.0.1.25 netmask 0xffffffff
> 	inet 127.0.1.14 netmask 0xffffffff
> 	inet 127.0.1.29 netmask 0xffffffff
> 	inet 127.0.1.17 netmask 0xffffffff
> 	groups: lo
> 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> 	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> 	inet 192.168.12.1 netmask 0xffffff00
> 	inet 192.168.12.5 netmask 0xffffffff
> 	inet 192.168.12.11 netmask 0xffffff00
> 	inet 192.168.12.12 netmask 0xffffff00
> 	inet 192.168.12.15 netmask 0xffffff00
> 	inet 192.168.12.16 netmask 0xffffff00
> 	inet 192.168.12.18 netmask 0xffffff00
> 	inet 192.168.12.19 netmask 0xffffff00
> 	inet 192.168.12.21 netmask 0xffffff00
> 	inet 192.168.12.22 netmask 0xffffff00
> 	inet 192.168.12.25 netmask 0xffffff00
> 	inet 192.168.12.14 netmask 0xffffff00
> 	inet 192.168.12.29 netmask 0xffffff00
> 	inet 192.168.12.17 netmask 0xffffff00
> 	groups: lo
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> pflog0: flags=100<PROMISC> metric 0 mtu 33160
> 	groups: pflog
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	description: jails-bridge
> 	ether 58:9c:fc:10:ed:66
> 	inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255
> 	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> 	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> 	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> 	member: vnet0.655 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> 	        ifmaxaddr 0 port 6 priority 128 path cost 2000
> 	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> 	        ifmaxaddr 0 port 1 priority 128 path cost 20000
> 	groups: bridge
> 	nd6 options=9<PERFORMNUD,IFDISABLED>
> vnet0.655: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 	description: associated with jail: examplejail as nic: epair0b
> 	options=8<VLAN_MTU>
> 	ether b4:2e:99:9c:5b:ca
> 	hwaddr 02:ae:46:07:62:0a
> 	groups: epair
> 	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> 	status: active
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> And host's netstat (again with many old fashion jail):
>
> netstat -rn
> Routing tables
>
> Internet:
> Destination        Gateway            Flags     Netif Expire
> default            95.217.83.193      UGS         em0
> 10.0.10.0/24       link#5             U       bridge0
> 10.0.10.1          link#5             UHS         lo0
> 95.217.83.192/26   link#1             U           em0
> 95.217.83.231      link#1             UHS         lo0
> 127.0.0.1          link#2             UH          lo0
> 127.0.1.5          link#2             UH          lo0
> 127.0.1.11         link#2             UH          lo0
> 127.0.1.12         link#2             UH          lo0
> 127.0.1.14         link#2             UH          lo0
> 127.0.1.15         link#2             UH          lo0
> 127.0.1.16         link#2             UH          lo0
> 127.0.1.17         link#2             UH          lo0
> 127.0.1.18         link#2             UH          lo0
> 127.0.1.19         link#2             UH          lo0
> 127.0.1.21         link#2             UH          lo0
> 127.0.1.22         link#2             UH          lo0
> 127.0.1.25         link#2             UH          lo0
> 127.0.1.29         link#2             UH          lo0
> 127.0.12.1         link#2             UH          lo0
> 192.168.12.1       link#3             UH          lo1
> 192.168.12.5       link#3             UH          lo1
> 192.168.12.11      link#3             UH          lo1
> 192.168.12.12      link#3             UH          lo1
> 192.168.12.14      link#3             UH          lo1
> 192.168.12.15      link#3             UH          lo1
> 192.168.12.16      link#3             UH          lo1
> 192.168.12.17      link#3             UH          lo1
> 192.168.12.18      link#3             UH          lo1
> 192.168.12.19      link#3             UH          lo1
> 192.168.12.21      link#3             UH          lo1
> 192.168.12.22      link#3             UH          lo1
> 192.168.12.25      link#3             UH          lo1
> 192.168.12.29      link#3             UH          lo1
>
> Internet6:
> Destination                       Gateway                       Flags     Netif Expire
> ::/96                             ::1                           UGRS        lo0
> default                           fe80::1%em0                   UGS         em0
> ::1                               link#2                        UHS         lo0
> ::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
> 2a01:4f9:4a:1fd8::/64             link#1                        U           em0
> 2a01:4f9:4a:1fd8::2               link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::5               link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::11              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::12              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::14              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::15              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::16              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::17              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::18              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::19              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::21              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::22              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::25              link#1                        UHS         lo0
> 2a01:4f9:4a:1fd8::29              link#1                        UHS         lo0
> fe80::/10                         ::1                           UGRS        lo0
> fe80::%em0/64                     link#1                        U           em0
> fe80::b62e:99ff:fe6a:809d%em0     link#1                        UHS         lo0
> fe80::%lo0/64                     link#2                        U           lo0
> fe80::1%lo0                       link#2                        UHS         lo0
> ff02::/16                         ::1                           UGRS        lo0
>
> The bridge0 had the em0 and vnet0:655 interfaces.
>
> From the jail in can ping oustside world:
>
> ping google.ca
> PING6(56=40+8+8 bytes) 2a01:4f9:4a:1fd8::23 --> 2a00:1450:400f:803::2003
> 16 bytes from 2a00:1450:400f:803::2003, icmp_seq=0 hlim=118 time=7.927 ms
> 16 bytes from 2a00:1450:400f:803::2003, icmp_seq=1 hlim=118 time=7.800 ms
> 16 bytes from 2a00:1450:400f:803::2003, icmp_seq=2 hlim=118 time=7.798 ms
> ^C
> --- google.ca ping6 statistics ---
> 3 packets transmitted, 3 packets received, 0.0% packet loss
> round-trip min/avg/max/std-dev = 7.798/7.842/7.927/0.061 ms
>
> The problem is, I cannot ssh to an external computer (for example, my
> nextcloud hosted at home):
>
> ssh -vvv nextcloud.foucry.net -p2250
> OpenSSH_7.9p1, OpenSSL 1.1.1k-freebsd  25 Mar 2021
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug2: resolving "nextcloud.foucry.net" port 2250
> debug2: ssh_connect_direct
> debug1: Connecting to nextcloud.foucry.net [2a01:e0a:434:44e0:ff:60ff:feba:b582] port 2250.
> debug1: connect to address 2a01:e0a:434:44e0:ff:60ff:feba:b582 port 2250: Operation timed out
> debug1: Connecting to nextcloud.foucry.net [82.65.174.130] port 2250.
> debug1: connect to address 82.65.174.130 port 2250: Operation timed out
> ssh: connect to host nextcloud.foucry.net port 2250: Operation timed out
>
> What's look strange (for me) is the traceroute (using ipv4):
>
> traceroute nextcloud.foucry.net
> traceroute to nextcloud.foucry.net (82.65.174.130), 64 hops max, 40 byte packets
>  1  10.0.10.1 (10.0.10.1)  0.086 ms  0.051 ms  0.037 ms
>  2  static.193.83.217.95.clients.your-server.de (95.217.83.193)  0.451 ms  0.571 ms  0.392 ms
>  3  core32.hel1.hetzner.com (213.239.252.97)  11.621 ms
>     core31.hel1.hetzner.com (213.239.252.93)  1.812 ms
>     core32.hel1.hetzner.com (213.239.252.97)  2.793 ms
>  4  core9.fra.hetzner.com (213.239.224.166)  21.295 ms
>     core8.fra.hetzner.com (213.239.224.149)  20.730 ms
>     core9.fra.hetzner.com (213.239.224.170)  20.333 ms
>  5  core4.fra.hetzner.com (213.239.245.85)  28.499 ms
>     core4.fra.hetzner.com (213.239.224.177)  20.507 ms  22.850 ms
>  6  * * *
>  7  * * *
>  8  * * *
>  9  * * *
> 10  * * *
> 11  * * *
> 12  * * *
> 13  *^C
>
>
> Look's like something wrong on the way, but I could connect on the same host
> form any other jails.
>
>
> There is for me a mysterious behaviiors that I can't understand.
>
> Any help will be appreciate.
>
> Thanks for reading me, and the time your spend on my problem.
Received on Fri Jul 23 2021 - 21:06:41 UTC

Original text of this message