vnet NAT'd jails extremely slow, connection dies

[Kristof Provost]

On 2019-02-05 18:47:23 (+0100), Michael Grimm <trashcan at ellael.org> wrote:
> Farhan Khan <khanzf at gmail.com> wrote:
> > On Mon, Feb 4, 2019 at 2:29 PM Farhan Khan <khanzf at gmail.com> wrote:
> 
> >> I have a jail NAT'd to a base system, but the connection is extremely
> >> slow and frequently disconnects drops, whereas the base is fine has
> >> perfectly fine connectivity.
> >> 
> >> My configuration is as follows:
> >> vtnet0: Has routeable IPv4 address and 172.16.0.1/16
> >> Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16.
> >> The base and jail can ping each other.
> >> bridge0: contains vtnet0 and epair4a.
> >> 
> >> I have gateway_enable="YES"
> >> My pf.conf is as follows:
> >> nat pass from 172.16.0.0/16 to any -> (vtnet0)
> >> 
> >> When I try to run clamav, the connectivity stalls after a few minutes
> >> and eventually disconnects. I ran tcpdump on the bridge and saw a lot
> >> of HTTP seq and ack packets but no actual data. I am not using IPv6
> >> yet.
> > 
> > Just to provide more context to my previous email, outside of the jail
> > I can download the FreeBSD ISO installer image at 3 MBps. Within the
> > jail it drops to 12KBps.
> 
> This sounds familiar to me ;-)
> 
> Please have a look at https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html
> Solution in https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049484.html
> 
> I ended up with the following additions to /boot/loader.conf (and a subsequent reboot):
> 
> 	# needs to become turned off (LRO) in order to restore tcp performance within VNET jails:
> 	hw.vtnet.lro_disable="1"   
> 	hw.vtnet.tso_disable="1"
> 
Farhan has also solved his issue by turning off lro/tso. (We talked on
IRC).

I've not seen this issue myself, but I'm interested in a couple of
points to hopefully pinpoint and maybe even fix the problem.

These are questions for anyone who's running pf on top of a hypervisor
and has vnet or other jails, and has seen slowdowns.

 * What hypervisor are you running?
 * Does the problem affect only the jails, or also the host system?
 * Does it only happen with NAT, or with routed packets as well?

If anyone is affected and not using pf that'd be interesting information
as well.

Regards,
Kristof