vnet NAT'd jails extremely slow, connection dies

[Farhan Khan]

On Tue, Feb 5, 2019 at 12:58 PM Kristof Provost <kp at freebsd.org> wrote:
>
> On 2019-02-05 18:47:23 (+0100), Michael Grimm <trashcan at ellael.org> wrote:
> > Farhan Khan <khanzf at gmail.com> wrote:
> > > On Mon, Feb 4, 2019 at 2:29 PM Farhan Khan <khanzf at gmail.com> wrote:
> >
> > >> I have a jail NAT'd to a base system, but the connection is extremely
> > >> slow and frequently disconnects drops, whereas the base is fine has
> > >> perfectly fine connectivity.
> > >>
> > >> My configuration is as follows:
> > >> vtnet0: Has routeable IPv4 address and 172.16.0.1/16
> > >> Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16.
> > >> The base and jail can ping each other.
> > >> bridge0: contains vtnet0 and epair4a.
> > >>
> > >> I have gateway_enable="YES"
> > >> My pf.conf is as follows:
> > >> nat pass from 172.16.0.0/16 to any -> (vtnet0)
> > >>
> > >> When I try to run clamav, the connectivity stalls after a few minutes
> > >> and eventually disconnects. I ran tcpdump on the bridge and saw a lot
> > >> of HTTP seq and ack packets but no actual data. I am not using IPv6
> > >> yet.
> > >
> > > Just to provide more context to my previous email, outside of the jail
> > > I can download the FreeBSD ISO installer image at 3 MBps. Within the
> > > jail it drops to 12KBps.
> >
> > This sounds familiar to me ;-)
> >
> > Please have a look at https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html
> > Solution in https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049484.html
> >
> > I ended up with the following additions to /boot/loader.conf (and a subsequent reboot):
> >
> >       # needs to become turned off (LRO) in order to restore tcp performance within VNET jails:
> >       hw.vtnet.lro_disable="1"
> >       hw.vtnet.tso_disable="1"
> >
> Farhan has also solved his issue by turning off lro/tso. (We talked on
> IRC).
>
> I've not seen this issue myself, but I'm interested in a couple of
> points to hopefully pinpoint and maybe even fix the problem.
>
> These are questions for anyone who's running pf on top of a hypervisor
> and has vnet or other jails, and has seen slowdowns.
>
>  * What hypervisor are you running?
>  * Does the problem affect only the jails, or also the host system?
>  * Does it only happen with NAT, or with routed packets as well?
>
> If anyone is affected and not using pf that'd be interesting information
> as well.
>
> Regards,
> Kristof

Michael, thank you very much. This appears to do the trick, as Kristof
also directed me.

A. This was on a Vultr instance. Per they are using KVM, per a support ticket.
B. Just the Jail, not the post
C. I will have to get back to you on that, as I do not have a publicly
routeable IP to test on at the moment.