misc/188261: FreeBSD DomU PVHVM guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host.

[Karl Pielorz]

>Number:         188261
>Category:       misc
>Synopsis:       FreeBSD DomU PVHVM guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host.
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Apr 04 15:30:00 UTC 2014
>Closed-Date:
>Last-Modified:
>Originator:     Karl Pielorz
>Release:        9.2-STABLE / 10.0-RELEASE
>Organization:
>Environment:
FreeBSD host 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789 : Thu Jan 16 22:34:59 UTC 2014    root at snap.freebsd.org/usr/obj/usr/src/sys/GENERIC amd64
>Description:
When running FreeBSD under Xen as a DomU guest - a PVHVM based FreeBSD machine cannot route traffic for any other PV based DomU guests on the same Xen Dom0.

>How-To-Repeat:
Install XenServer 6.2.

Install FreeBSD 9.2 / 10.0 as a DomU guest, using the PVHVM (so you end up with a NIC called 'xn0' etc.)

Set this first machine up with (for example) 'gateway_enable="YES"' etc. and configure it to route or NAT traffic to the Internet.

Install another DomU guest (e.g. FreeBSD again, or Windows) on the same XenServer.

Make the default gateway of the 2nd DomU the IP of the first DomU.

Even though the fist DomU machine can fetch data/route traffic to/from "The Internet" - the second DomU machine cannot use it as a gateway. Pings will work, TCP sessions will initially 'connect' but cannot exchange any traffic.

If you replace the 'router' DomU machine with say a Linux box (or Windows box) it works as expected. Only FreeBSD in PVHVM mode does not work as the gateway.
>Fix:

To fix the problem either:

 - Replace the DomU router machine with a Linux guest (not ideal!)

 - Drop the DomU router machine into HVM mode (i.e. xn0 etc. get replaced by rl0 et'al)

 - Drop the other DomU guests from PV/PVHVM mode down to HVM mode (this also appears to fix the problem!)

 - Move the DomU router machine to a different XenServer, even if it's in the same pool (problem only happens if the DomU router machine, and the DomU guest trying to use it as a gateway are on the same physical Xen Dom0 host).

None of these solutions are ideal - it's basically precluding you from running a 'gateway' machine on XenServer unless it's either cited on it's own pool - or not efficient (i.e. HVM mode only) - which in turn makes it non-agile.

>Release-Note:
>Audit-Trail:
>Unformatted: