Restricting IP ranges for guests over tap devices

[Jason Tubnor]

On Sun, 2 Aug 2020 at 00:51, Mark Raynsford via freebsd-virtualization <
freebsd-virtualization at freebsd.org> wrote:

> Hello!
>
> Let's say I have a machine running a few dozen bhyve guests. Each bhyve
> guest gets its own tap device, and all of the tap devices are connected
> to a bridge.
>
> Everything works fine. I can write pf rules that control access between
> each guest, and between each guest and the world. I can't directly
> observe the IP addresses that the guests have assigned to the tap
> devices I gave them, but if I know the addresses beforehand, I can for
> example write pf rules that say things like:
>
>   block log all
>   pass in on tap23 proto tcp \
>     from any to $guest_23_ip port ssh modulate state
>
> That then means that even if the guest is compromised and tries to bind
> a server to another address, the pf rules won't allow anyone else to
> actually connect to it.
>
> The good thing about this is also the bad thing about this; I have to
> write specific rules that say "only allow access to this specific IP
> via this specific tap device". Over dozens of guests, that can multiply
> to hundreds of laboriously maintained rules.
>
> Is there some more general way I can supply a mapping between tap
> devices and allowed addresses? Remember that pf can't see the guest
> addresses on the host sides of the tap devices, so I can't use the
> (device) syntax to expand to "the address of a NIC called 'device'".
>
>
>
Treat the tap interface as a bridge and only define the destination port.
That way you are able to protect the guest from the host without knowing
the guest IP address.

I'd do it a bit differently though.  I'd treat the bridge that everything
is tapped into as being a hostile environment.  As such, each guest would
protect itself as if you had a VPN on the public internet, using the guests
built-in firewall.

Another way is isolating each guest or bunch of guests on private VLANs and
then protect these subnets on the host.

Cheers,

Jason.