bhyve NAT issue

[Pete Wright]

Hello,
I have a baremetal server hosted by Vultr that I would like to host 
several bhyve instances on.  I have been given one public ipv4 address, 
so my goal is to run the bhyve instances on a private subnet 
(172.16.0.0/24) then use pf to NAT this subnet and do port forwarding 
to.  I am having an issues though getting the VM instances network to work.

Here is how the hybervisor is configured:
uname:
FreeBSD cojo 11.2-STABLE FreeBSD 11.2-STABLE #1 r337947: Fri Aug 17 
03:22:33 PDT 2018 
pete at cojo:/usr/obj/usr/home/pete/src/freebsd-stable/sys/GENERIC amd64

rc.conf:
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="inet 172.16.0.1 netmask 255.255.255.0 addm tap0 up"

gateway_enable="YES"
ipv6_gateway_enable="YES"
pf_enable="YES"


pf.conf:
ext_if=ix0

scrub in all

nat on $ext_if inet from 172.16.0.0/24 to any -> ($ext_if)

pass in all
pass out all


and my script to invoke the VM:
/usr/sbin/bhyve -AHP -s 31:0,lpc \
   -s 2:0,virtio-net,tap0 \
   -s 3:0,virtio-blk,/vms/freebsd0 \
   -s 29,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait \
   -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
   -c 2 -m 1024M \
   test0


I am able to bring up the FreeBSD VM and can attach to it via tightvnc, 
so that's great.  I have configured the VM to have an IPv4 address of 
172.16.0.20/24 with a gateway of 172.16.0.1.  This is where I get stuck 
- when I attempt to ping the gateway from the VM I get "ping: sendto 
host down" errors.  Then I run "arp -an" to see what's up from the VM 
and see the following:

? (172.16.0.1) at (incomplete) on vtnet0 expired [ethernet]

when I run tcpdump against the bridge0 interface on the hypervisor while 
ping is run I see the following output:
15:56:48.995284 ARP, Request who-has 172.16.0.1 tell 172.16.0.20, length 46
15:56:48.995292 ARP, Reply 172.16.0.1 is-at 02:46:2f:56:ab:00 (oui 
Unknown), length 28

And I can verify that that is the correct MAC addr of the bridge0 interface:
$ ifconfig bridge0|grep ether
     ether 02:46:2f:56:ab:00

yet, tcpdump on the vtnet0 interface from the VM does not show the 
responses to the ARP requests:
ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28
ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28
ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28


I suspect I'm missing something trivial here, so any input would be 
appreciated :)

Oh, one last bit - here are my sysctl settings:
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
kern.randompid=1
security.bsd.stack_guard_page=1
vfs.zfs.min_auto_ashift=12

net.link.tap.up_on_open=1
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

Cheers,
-pete

-- 
Pete Wright
pete at nomadlogic.org
@nomadlogicLA