bhyve networking
Rodney W. Grimes
freebsd-rwg at pdx.rh.CN85.dnsmgr.net
Thu Apr 26 14:31:26 UTC 2018
[ Charset ISO-8859-1 unsupported, converting... ]
> Hi Folks,
>
> Apologies for the lack of detail on my first post. To recap, I am attempting to set-up a guest using vm-bhyve. I have a Dell PER730xd server with quad-port INTEL 350 NIC. The first two ports have been configured on a) a management LAN for the host and b) an application LAN for the guests.
>
> FreeBSD 11.1-RELEASE-p9
> Dell PowerEdge R730xd - INTEL i350 NICs
>
> NIC-1 igb0 24:6E:96:B4:61:CC VLAN92 ge-6/0/11 (Host)
> NIC-2 igb1 24:6E:96:B4:61:CD VLAN101 ge-6/0/18 (Guests) - not a trunk
>
> Both interfaces are active as viewed from the host, but I have only assigned an ipv4 address to igb0 for management of the host
>
> igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
> ether 24:6e:96:b4:61:cc
> hwaddr 24:6e:96:b4:61:cc
> inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
>
> igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
^^ MIssing UP, interface is down
> ether 24:6e:96:b4:61:cd
> hwaddr 24:6e:96:b4:61:cd
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
>
> If I assign a temporary address to igb1 I can then ping other computers on the guests subnet - I've had to hide the address as the network is restricted.
>
> # ifconfig igb1 inet xx.xxx.xxx.xx/25 up
> # ping xx.xxx.xxx.xx
> PING xx.xxx.xxx.xx (xx.xxx.xxx.xx): 56 data bytes
> 64 bytes from xx.xxx.xxx.xx: icmp_seq=0 ttl=64 time=0.145 ms
> 64 bytes from xx.xxx.xxx.xx: icmp_seq=1 ttl=64 time=0.080 ms
> 64 bytes from xx.xxx.xxx.xx: icmp_seq=2 ttl=64 time=0.078 ms
> 64 bytes from xx.xxx.xxx.xx: icmp_seq=3 ttl=64 time=0.077 ms
> 64 bytes from xx.xxx.xxx.xx: icmp_seq=4 ttl=64 time=0.076 ms
>
> I then used the "vm" command to create a virtual switch and add interface igb1 to it. This automatically created the bridge interface.
>
> root at dc1-olbp-hn-01:~ # vm switch create public
> root at dc1-olbp-hn-01:~ # vm switch add public igb1
> root at dc1-olbp-hn-01:~ # vm switch info public
> ------------------------
> Virtual Switch: public
> ------------------------
> type: auto
> ident: bridge0
> vlan: -
> nat: -
> physical-ports: igb1
> bytes-in: 0 (0.000B)
> bytes-out: 0 (0.000B)
>
> Finally, I created a guest VM and gave its NIC the same ipv4 address details I used previously to test igb1 from the host. This automatically created the tap interface.
>
> igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
> ether 24:6e:96:b4:61:cc
> hwaddr 24:6e:96:b4:61:cc
> inet 172.16.92.20 netmask 0xffffff00 broadcast 172.16.92.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
>
> igb1: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
^^ mising up, interface is down
> ether 24:6e:96:b4:61:cd
> hwaddr 24:6e:96:b4:61:cd
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
>
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> description: vm-public
> ether 02:ee:ce:b0:6a:00
> nd6 options=1<PERFORMNUD>
> groups: bridge
> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> ifmaxaddr 0 port 7 priority 128 path cost 2000000
> member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> ifmaxaddr 0 port 2 priority 128 path cost 20000
>
> tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> description: vmnet-testvm-0-public
> options=80000<LINKSTATE>
> ether 00:bd:dd:51:0a:00
> hwaddr 00:bd:dd:51:0a:00
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> groups: tap
> Opened by PID 1791
>
> >From the guest VM I can see that the interface vtnet0 is up and has the relevant ipv4 address information. However, I cannot communicate with any other computer on the guest subnet or beyond.
Guest may be up, but the host interface is in state down.
> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> ether 58:9c:fc:08:4a:20
> hwaddr 58:9c:fc:08:4a:20
> inet xx.xxx.xxx.xx netmask 0xffffff80 broadcast xx.xxx.xxx.xx
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T <full-duplex>
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
>
> The guest configuration file has the following network details
> network0_type="virtio-net"
> network0_switch="public"
>
> >From the vm-bhyve.log I see the following
> Apr 26 07:59:23: initialising
> Apr 26 07:59:23: [loader: bhyveload]
> Apr 26 07:59:23: [uefi: no]
> Apr 26 07:59:23: [cpu: 1]
> Apr 26 07:59:23: [memory: 256M]
> Apr 26 07:59:23: [hostbridge: standard]
> Apr 26 07:59:23: [com ports: com1]
> Apr 26 07:59:23: [uuid: 417cfb63-491f-11e8-949b-246e96b461cc]
> Apr 26 07:59:23: [utctime: no]
> Apr 26 07:59:23: [debug mode: no]
> Apr 26 07:59:23: [primary disk: disk0]
> Apr 26 07:59:23: [primary disk dev: sparse-zvol]
> Apr 26 07:59:23: generated static mac 58:9c:fc:08:4a:20 (based on 'testvm:0:1524725963:0')
> Apr 26 07:59:23: initialising network device tap0
> Apr 26 07:59:23: adding tap0 -> bridge0 (public)
> Apr 26 07:59:23: booting
>
> Should I have to supply ipv4 details anywhere other than the guest's own vtnet0 interface? If I re-configure the switch to remove the igb1 interface and add igb0 instead, then change the guest ipv4 address details to the management network (172.16.92.0/24), I can connect to other computers on that subnet and beyond.
>
> vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> ether 58:9c:fc:08:4a:20
> hwaddr 58:9c:fc:08:4a:20
> inet 172.16.92.21 netmask 0xffffff80 broadcast 172.16.92.127
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T <full-duplex>
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> groups: lo
> $ ping 172.16.92.11
> PING 172.16.92.11 (172.16.92.11): 56 data bytes
> 64 bytes from 172.16.92.11: icmp_seq=0 ttl=64 time=0.416 ms
> 64 bytes from 172.16.92.11: icmp_seq=1 ttl=64 time=0.371 ms
> 64 bytes from 172.16.92.11: icmp_seq=2 ttl=64 time=0.369 ms
> --- 172.16.92.11 ping statistics ---
> 3 packets transmitted, 3 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 0.369/0.385/0.416/0.022 ms
>
> $ telnet bbc.co.uk 443
> Trying 151.101.192.81...
> Connected to bbc.co.uk.
> Escape character is '^]'.
>
>
> Regards,
>
>
> Paul Esson | Redstor Limited
> t +44 (0)118 951 5235 | m +44 (0)776 690 6514
> e paul.esson at redstor.com
> www.redstor.com
>
>
>
>
>
> -----Original Message-----
> From: Rodney W. Grimes <freebsd-rwg at pdx.rh.CN85.dnsmgr.net>
> Sent: 25 April 2018 22:31
> To: Paul Esson <paul.esson at redstor.com>
> Cc: freebsd-virtualization at freebsd.org
> Subject: Re: bhyve networking
>
> > Hi Rod,
> > Can you share a command line for that?
> Its not just a single command, but you want these in /etc/rc.conf of the GUEST:
> network_interfaces="lo0"
> cloned_interfaces="vlan48"
> ifconfig_lo0=" inet 127.0.0.1 netmask 0xff000000"
> ifconfig_vtnet0=" up"
> ifconfig_vlan48=" inet 192.168.48.38 netmask 0xffffff00 vlan 48 vlandev vtnet0"
>
> That may be your issue... is your vtnet0 "up" in the guest.
Or better yet, is your igb1 interface "up" on the host?
Add
ifconfig_igb1="up"
to your hosts /etc/rc.conf file.
And type:
ifconfig igb1 up
and your probelm should resolve.
> It would help a whole lot to share more of the info about your system, from commands, not from "vm-bhyve" settings.
> Like
> ifconfig -a
> on the host and the guest would be a starting point.
>
> > I also tried presenting an access port from my switch on a specific VLAN - not trimmed.
> Trimmed? You mean you set the switch port to untagged mode, and had the switch tag/untag the packets to a specific vlan.
> Be sure you also set the default incoming tag at the switch if you did this, some switches do not follow the vlan setting.
>
> > Would I still have to tag the interface on the guest in that scenario?
> No. If I understand what I think you meant by trimmed.
>
> > Regards,
> >
> >
> > Paul Esson | Redstor Limited
> > t +44 (0)118 951 5235
> > m +44 (0)776 690 6514
> > e paul.esson at redstor.com
> >
> > ________________________________
> > From: Rodney W. Grimes <freebsd-rwg at pdx.rh.CN85.dnsmgr.net>
> > Sent: Wednesday, April 25, 2018 9:33:57 PM
> > To: Paul Esson
> > Cc: Harry Schmalzbauer; freebsd-virtualization at freebsd.org
> > Subject: Re: bhyve networking
> >
> > [ Charset windows-1252 unsupported, converting... ]
> > > Hi Harry,
> > > I?m simply using the ?vm? utility as in
> > >
> > > vm switch create public
> > > vm switch add public igb1
> > >
> > > That must make underlying calls to if config or equivalent as the bridge and tap interfaces are created automatically.
> > >
> > > The vm template file has these relevant parameters
> > >
> > > network0_type=?virtio-net?
> > > network0_switch=?public?
> > >
> > > I?ve done nothing to the igb1 interface other than connect it to a physical switch on the appropriate VLAN.
> >
> > How have you configured your vtnet devices inside the guest? If your
> > pass a "trunked" ethernet device to a guest the guest is going to need to run vlan decapuslation. I do this here, and it works fine.
> >
> > vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
> > ether 58:9c:fc:0e:8b:ec
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > media: Ethernet 10Gbase-T <full-duplex>
> > status: active
> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> > options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> > inet 127.0.0.1 netmask 0xff000000
> > inet6 ::1 prefixlen 128
> > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> > nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> > groups: lo
> > vlan48: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > ether 58:9c:fc:0e:8b:ec
> > inet 192.168.48.38 netmask 0xffffff00 broadcast 192.168.48.255
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > media: Ethernet 10Gbase-T <full-duplex>
> > status: active
> > vlan: 48 vlanpcp: 0 parent interface: vtnet0
> > groups: vlan
> >
> > ...
> > >
> > > Bez?glich Paul Esson's Nachricht vom 25.04.2018 20:44 (localtime):
> > > > Hi Folks,
> > > >
> > > > I'm struggling with networking when using vm-bhyve on FreeBSD
> > > > 11.1-RELEASE. I have two NICs and have configured the first (igb0)
> > > > on a management network and want to use the second (igb1) for VMs.
> > > > However, I can't get any VM to communicate through the virtual
> > > > switch if I have igb1 added to it. If I take the NIC out of the
> > > > switch and configure an ipv4 address on it I can reach other hosts
> > > > on the relevant subnet so I believe the poet set-up is valid. If
> > > > I replace igb1 in the switch with igb0, I can then configure VMs
> > > > on my management network and they have network connectivity. Can
> > > > I only use
> > >
> > > Hello,
> > >
> > > a example of the command you used was nice.
> > > I guess you're using if_bridge(4) ? the example would clarify.
> > > But there's ng_bridge(4) and vale(4) also, and others are using
> > > OpenVSwitch...
> > >
> > > > an interface that has an IP address configured at the host level before adding it to the switch? I've used other
> > >
> > > No.
> > > But the interface has to be in promisc mode. And some offloading
> > > functions must be disabled, but in case of if_bridge(4), it's done
> > > automagically (and reverted if you remove the interface again).
> > >
> > > Hth,
> > >
> > > -harry
> > > _______________________________________________
> > > freebsd-virtualization at freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
> > >
> > >
> >
> > --
> > Rod Grimes rgrimes at freebsd.org
> > _______________________________________________
> > freebsd-virtualization at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
> >
>
> --
> Rod Grimes rgrimes at freebsd.org
>
> Paul Esson??|??Redstor Limited
> t??+44 (0)118 951 5235??|???m??+44 (0)776 690 6514
> e??paul.esson at redstor.com
> www.redstor.com
>
>
>
>
>
> -----Original Message-----
> From: Harry Schmalzbauer <freebsd at omnilan.de>
> Sent: 26 April 2018 08:39
> To: Paul Esson <paul.esson at redstor.com>
> Cc: freebsd-virtualization at freebsd.org
> Subject: Re: bhyve networking
>
> Bez?glich Paul Esson's Nachricht vom 25.04.2018 23:15 (localtime):
> > Hi Rod,
> > Can you share a command line for that? I also tried presenting an
> > access port from my switch on a specific VLAN - not trimmed. Would I
> > still have to tag the interface on the guest in that scenario?
>
> Hmm, I lost the overview - I'm not familar with 'vm'.
> To filter a specific id (tag/untag frames) inside the guest:
> 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnn'
> 'ifconfig vlan[N] create vlandev vtnet0 vlan nnnm'
>
> At boot time by rc(8):
> vlans_vtnet0="vtnet_dmz vtnet_dmz2"
> create_args_vtnet_dmz="vlan nnnn"
> create_args_vtnet_dmz2="vlan nnnm"
>
> [To optionally also rename the vlan interfaces after manually creating cloned vlan interfaces, which is what the rc.conf(5) example does:
> ifconfig rename vlan0 vtnet_dmz; ifconfig rename vlan0 vtnet_dmz2; ]
>
> Hth,
>
> -harry
>
>
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
>
>
--
Rod Grimes rgrimes at freebsd.org
More information about the freebsd-virtualization
mailing list