Question About TCP Reassembly Inside VImages

[Jorge Sanchez]

Hola Jason,

I also observed a similar phenomenon on my system's vimages. I have several thousands dropped packets due to "insufficient memory" (the counter you mention in netstat -m for me is also incremented in the net.inet.tcp.reass.overflows read-only sysctl variable) and I routinely have TCP connections dropped within vimages because of it. I think that the TCP packet reassembly queue length is essentially zero once options VIMAGE is enabled... which would explain my problems when trying to contact hosts that are on flaky links or are situated very far away hop-wise.

I think there is something very wrong with the TCP reassembly when options VIMAGE is enabled. Did you try increasing the net.inet.reass.maxqlen or net.inet.reass.maxsegments sysctls? I was able to increase maxqlen but maxsegments must be set in loader.conf and this value is not inherited into any vimages I create after booting! :-(

If you come up with a fix, I would appreciate it too since this prevents me from performing realistic TCP testing within the virtualization framework.

Adios!
Jorge


From: jason.fines at gmail.com
To: 
	  freebsd-virtualization at freebsd.org
Subject: Re: Question About TCP Reassembly Inside VImages
Date: Sat, 22 Nov 2008 08:52:16 -0500
Thanks Julian,

I suspect you are correct as nmbclusters is a system wide sysctl variable
set at boot time and although V_tcp_reass_maxseg is set per vimage it is the
result of a constant operation done on nmbclusters (nmbclusters / 16).

What I've described is what I suspect is the root of my problem.  The
manifestation of this problem is that TCP packets passing through my
vimage(s) are not reassembled when they are out of order and I get an
exceptionally high value reported by netstat -m stating that packets were
dropped due to "insufficient memory".  Posts I've found on the net point to
the reassembly queue length, which in the vimages is zero for some reason.

Perhaps this additional information will help clarify my exact problem.

Thanks,
Jason

On Sat, Nov 22, 2008 at 5:12 AM, Julian Elischer <julian at elischer.org>wrote:

> Jason Fines wrote:
>
>> Hello all,
>>
>> I've got a question about setting the sysctl variable
>> net.inet.tcp.reass.maxsegments to a non-zero value inside my vimages.  I'm
>> currently running the FreeBSD 7 with the VIMAGE package available at
>> http://imunes.tel.fer.hr/virtnet/vimage_7-20081015.tgz.
>>
>> My problem is with TCP reassembly support inside of the vimages, namely
>> with
>> the tcp.reass.maxsegments sysctl variable.  I've tracked down where in the
>> code the variable is set to line 122 in tcp_reass_init() of
>> netinet/tcp_reass.c: "V_tcp_reass_maxseg = nmbclusters / 16;".  The line
>> clearly reads that maxsegments should be set to "nmbclusters /16", in the
>> main OS (not in any vimage) the value is correctly set to 1/16 of what my
>> nmbclusters sysctl variable is set to.  However, inside all my vimages
>> nmbclusters is set correctly, while reass.maxsegments is incorrectly set
>> to
>> zero!!!
>>
>
> V_tcp_reass_maxseg is a macro that hides the fact that
> tcp_reass_maxseg is a PER Vimage variable.
>
> Part of the patch
> is to make some sysctls be per-vimage. I do not know exactly
> about that one.. I suspect it is actually a read-only
> whole-system value, and not per vimage.
>
>
>
>
>
>> Is it possible that nmbclusters when read on line 122 of
>> netinet/tcp_reass.c
>> is zero?  Has anyone else experienced this problem?  Is TCP reassembly not
>> supported/tested inside vimages?
>>
>> Any help in this area would be greatly appreciated.
>>
>> Thanks,
>> Jason
>>
>> P.S. This technology is phenomenal, and thanks to everyone who is involved
>> developing it.
>> _______________________________________________
>> freebsd-virtualization at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>> To unsubscribe, send any mail to "
>> freebsd-virtualization-unsubscribe at freebsd.org"
>>


_________________________________________________________________