bhyve networking
Paul Esson
paul.esson at redstor.com
Thu Apr 26 12:40:43 UTC 2018
Hi Folks,
Apologies for the lack of detail on my first post. To recap, I am attempting to set-up a guest using vm-bhyve. I have a Dell PER730xd server with quad-port INTEL 350 NIC. The first two ports have been configured on a) a management LAN for the host and b) an application LAN for the guests.
FreeBSD 11.1-RELEASE-p9
Dell PowerEdge R730xd - INTEL i350 NICs
NIC-1 igb0 24:6E:96:B4:61:CC VLAN92 ge-6/0/11 (Host)
NIC-2 igb1 24:6E:96:B4:61:CD VLAN101 ge-6/0/18 (Guests) - not a trunk
Both interfaces are active as viewed from the host, but I have only assigned an ipv4 address to igb0 for management of the host
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cc
hwaddr 24:6e:96:b4:61:cc
inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cd
hwaddr 24:6e:96:b4:61:cd
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
If I assign a temporary address to igb1 I can then ping other computers on the guests subnet - I've had to hide the address as the network is restricted.
# ifconfig igb1 inet xx.xxx.xxx.xx/25 up
# ping xx.xxx.xxx.xx
PING xx.xxx.xxx.xx (xx.xxx.xxx.xx): 56 data bytes
64 bytes from xx.xxx.xxx.xx: icmp_seq=0 ttl=64 time=0.145 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=1 ttl=64 time=0.080 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=2 ttl=64 time=0.078 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=3 ttl=64 time=0.077 ms
64 bytes from xx.xxx.xxx.xx: icmp_seq=4 ttl=64 time=0.076 ms
I then used the "vm" command to create a virtual switch and add interface igb1 to it. This automatically created the bridge interface.
root at dc1-olbp-hn-01:~ # vm switch create public
root at dc1-olbp-hn-01:~ # vm switch add public igb1
root at dc1-olbp-hn-01:~ # vm switch info public
------------------------
Virtual Switch: public
------------------------
type: auto
ident: bridge0
vlan: -
nat: -
physical-ports: igb1
bytes-in: 0 (0.000B)
bytes-out: 0 (0.000B)
Finally, I created a guest VM and gave its NIC the same ipv4 address details I used previously to test igb1 from the host. This automatically created the tap interface.
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cc
hwaddr 24:6e:96:b4:61:cc
inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 24:6e:96:b4:61:cd
hwaddr 24:6e:96:b4:61:cd
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vm-public
ether 02:ee:ce:b0:6a:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 20000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vmnet-testvm-0-public
options=80000<LINKSTATE>
ether 00:bd:dd:51:0a:00
hwaddr 00:bd:dd:51:0a:00
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: tap
Opened by PID 1791
>From the guest VM I can see that the interface vtnet0 is up and has the relevant ipv4 address information. However, I cannot communicate with any other computer on the guest subnet or beyond.
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:08:4a:20
hwaddr 58:9c:fc:08:4a:20
inet xx.xxx.xxx.xx netmask 0xffffff80 broadcast xx.xxx.xxx.xx
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
The guest configuration file has the following network details
network0_type="virtio-net"
network0_switch="public"
>From the vm-bhyve.log I see the following
Apr 26 07:59:23: initialising
Apr 26 07:59:23: [loader: bhyveload]
Apr 26 07:59:23: [uefi: no]
Apr 26 07:59:23: [cpu: 1]
Apr 26 07:59:23: [memory: 256M]
Apr 26 07:59:23: [hostbridge: standard]
Apr 26 07:59:23: [com ports: com1]
Apr 26 07:59:23: [uuid: 417cfb63-491f-11e8-949b-246e96b461cc]
Apr 26 07:59:23: [utctime: no]
Apr 26 07:59:23: [debug mode: no]
Apr 26 07:59:23: [primary disk: disk0]
Apr 26 07:59:23: [primary disk dev: sparse-zvol]
Apr 26 07:59:23: generated static mac 58:9c:fc:08:4a:20 (based on 'testvm:0:1524725963:0')
Apr 26 07:59:23: initialising network device tap0
Apr 26 07:59:23: adding tap0 -> bridge0 (public)
Apr 26 07:59:23: booting
Should I have to supply ipv4 details anywhere other than the guest's own vtnet0 interface? If I re-configure the switch to remove the igb1 interface and add igb0 instead, then change the guest ipv4 address details to the management network (172.16.92.0/24), I can connect to other computers on that subnet and beyond.
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:08:4a:20
hwaddr 58:9c:fc:08:4a:20
inet 172.16.92.21 netmask 0xffffff80 broadcast 172.16.92.127
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
$ ping 172.16.92.11
PING 172.16.92.11 (172.16.92.11): 56 data bytes
64 bytes from 172.16.92.11: icmp_seq=0 ttl=64 time=0.416 ms
64 bytes from 172.16.92.11: icmp_seq=1 ttl=64 time=0.371 ms
64 bytes from 172.16.92.11: icmp_seq=2 ttl=64 time=0.369 ms
--- 172.16.92.11 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.369/0.385/0.416/0.022 ms
$ telnet bbc.co.uk 443
Trying 151.101.192.81...
Connected to bbc.co.uk.
Escape character is '^]'.
Regards,
Paul Esson | Redstor Limited
t +44 (0)118 951 5235 | m +44 (0)776 690 6514
e paul.esson at redstor.com
www.redstor.com
-----Original Message-----
From: Rodney W. Grimes <freebsd-rwg at pdx.rh.CN85.dnsmgr.net>
Sent: 25 April 2018 22:31
To: Paul Esson <paul.esson at redstor.com>
Cc: freebsd-virtualization at freebsd.org
Subject: Re: bhyve networking
> Hi Rod,
> Can you share a command line for that?
Its not just a single command, but you want these in /etc/rc.conf of the GUEST:
network_interfaces="lo0"
cloned_interfaces="vlan48"
ifconfig_lo0=" inet 127.0.0.1 netmask 0xff000000"
ifconfig_vtnet0=" up"
ifconfig_vlan48=" inet 192.168.48.38 netmask 0xffffff00 vlan 48 vlandev vtnet0"
That may be your issue... is your vtnet0 "up" in the guest.
It would help a whole lot to share more of the info about your system, from commands, not from "vm-bhyve" settings.
Like
ifconfig -a
on the host and the guest would be a starting point.
> I also tried presenting an access port from my switch on a specific VLAN - not trimmed.
Trimmed? You mean you set the switch port to untagged mode, and had the switch tag/untag the packets to a specific vlan.
Be sure you also set the default incoming tag at the switch if you did this, some switches do not follow the vlan setting.
> Would I still have to tag the interface on the guest in that scenario?
No. If I understand what I think you meant by trimmed.
> Regards,
>
>
> Paul Esson | Redstor Limited
> t +44 (0)118 951 5235
> m +44 (0)776 690 6514
> e paul.esson at redstor.com
>
> ________________________________
> From: Rodney W. Grimes <freebsd-rwg at pdx.rh.CN85.dnsmgr.net>
> Sent: Wednesday, April 25, 2018 9:33:57 PM
> To: Paul Esson
> Cc: Harry Schmalzbauer; freebsd-virtualization at freebsd.org
> Subject: Re: bhyve networking
>
> [ Charset windows-1252 unsupported, converting... ]
> > Hi Harry,
> > I?m simply using the ?vm? utility as in
> >
> > vm switch create public
> > vm switch add public igb1
> >
> > That must make underlying calls to if config or equivalent as the bridge and tap interfaces are created automatically.
> >
> > The vm template file has these relevant parameters
> >
> > network0_type=?virtio-net?
> > network0_switch=?public?
> >
> > I?ve done nothing to the igb1 interface other than connect it to a physical switch on the appropriate VLAN.
>
> How have you configured your vtnet devices inside the guest? If your
> pass a "trunked" ethernet device to a guest the guest is going to need to run vlan decapuslation. I do this here, and it works fine.
>
> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> ether 58:9c:fc:0e:8b:ec
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T <full-duplex>
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet 127.0.0.1 netmask 0xff000000
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
> vlan48: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> ether 58:9c:fc:0e:8b:ec
> inet 192.168.48.38 netmask 0xffffff00 broadcast 192.168.48.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T <full-duplex>
> status: active
> vlan: 48 vlanpcp: 0 parent interface: vtnet0
> groups: vlan
>
> ...
> >
> > Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime):
> > > Hi Folks,
> > >
> > > I'm struggling with networking when using vm-bhyve on FreeBSD
> > > 11.1-RELEASE. I have two NICs and have configured the first (igb0)
> > > on a management network and want to use the second (igb1) for VMs.
> > > However, I can't get any VM to communicate through the virtual
> > > switch if I have igb1 added to it. If I take the NIC out of the
> > > switch and configure an ipv4 address on it I can reach other hosts
> > > on the relevant subnet so I believe the poet set-up is valid. If
> > > I replace igb1 in the switch with igb0, I can then configure VMs
> > > on my management network and they have network connectivity. Can
> > > I only use
> >
> > Hello,
> >
> > a example of the command you used was nice.
> > I guess you're using if_bridge(4) ? the example would clarify.
> > But there's ng_bridge(4) and vale(4) also, and others are using
> > OpenVSwitch...
> >
> > > an interface that has an IP address configured at the host level before adding it to the switch? I've used other
> >
> > No.
> > But the interface has to be in promisc mode. And some offloading
> > functions must be disabled, but in case of if_bridge(4), it's done
> > automagically (and reverted if you remove the interface again).
> >
> > Hth,
> >
> > -harry
> > _______________________________________________
> > freebsd-virtualization at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
> >
> >
>
> --
> Rod Grimes rgrimes at freebsd.org
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
>
--
Rod Grimes rgrimes at freebsd.org
Paul Esson | Redstor Limited
t +44 (0)118 951 5235 | m +44 (0)776 690 6514
e paul.esson at redstor.com
www.redstor.com
-----Original Message-----
From: Harry Schmalzbauer <freebsd at omnilan.de>
Sent: 26 April 2018 08:39
To: Paul Esson <paul.esson at redstor.com>
Cc: freebsd-virtualization at freebsd.org
Subject: Re: bhyve networking
Bezüglich Paul Esson's Nachricht vom 25.04.2018 23:15 (localtime):
> Hi Rod,
> Can you share a command line for that? I also tried presenting an
> access port from my switch on a specific VLAN - not trimmed. Would I
> still have to tag the interface on the guest in that scenario?
Hmm, I lost the overview - I'm not familar with 'vm'.
To filter a specific id (tag/untag frames) inside the guest:
'ifconfig vlan[N] create vlandev vtnet0 vlan nnnn'
'ifconfig vlan[N] create vlandev vtnet0 vlan nnnm'
At boot time by rc(8):
vlans_vtnet0="vtnet_dmz vtnet_dmz2"
create_args_vtnet_dmz="vlan nnnn"
create_args_vtnet_dmz2="vlan nnnm"
[To optionally also rename the vlan interfaces after manually creating cloned vlan interfaces, which is what the rc.conf(5) example does:
ifconfig rename vlan0 vtnet_dmz; ifconfig rename vlan0 vtnet_dmz2; ]
Hth,
-harry
More information about the freebsd-virtualization
mailing list