BBR patches?
Randall Stewart
rrs at netflix.com
Thu Sep 19 18:05:22 UTC 2019
Ahh.. Look in your directory.
You did not create the sub’dirs or put rack.c or bbr.c into the right places
When you load the patch (which was updated Sept 17th and should be different slightly)
you will end up dropping a Makefile and bbr.c and tcp_bbr.h into the local dir since
for what every reason patch cannot figure out how to place it.
You must move
bbr.c/tcp_bbr.h to -> sys/netinet/tcp_stacks
You must create
sys/modules/tcp/bbr
and
place the make file under there (just like rack’s).
Do that and I think you will finally have it.. not sure why patch is not able to
put the new files in the right places :0
R
> On Sep 19, 2019, at 10:42 AM, vm finance <vm.finance2 at gmail.com> wrote:
>
> Hello Randall,
>
> I just retried it with headvm config that you had shared but still see error while doing
> "make cleandepend"
>
> Any pointers highly appreciated.
>
> Thanks
>
> root at osboxes:/usr/src/sys/amd65/compile/headvm # make cleandepend
> [...snip...]
>
> ===> sysvipc/sysvshm (obj)
> /usr/src/sys/amd64/compile/headvm/modules/usr/src/sys/modules/sysvipc/sysvshm created for /usr/src/sys/modules/sysvipc/sysvshm
> ===> tcp (obj)
> ===> tcp/bbr (obj)
> cd: /usr/src/sys/modules/tcp/bbr: No such file or directory
> *** Error code 2
>
> Stop.
> make[2]: stopped in /usr/src/sys/modules/tcp
> *** Error code 1
>
> Stop.
> make[1]: stopped in /usr/src/sys/modules
> *** Error code 1
>
> Stop.
> make: stopped in /usr/src/sys/amd64/compile/headvm
> root at osboxes:/usr/src/sys/amd64/compile/headvm #
>
>
> On Wed, Sep 18, 2019 at 11:08 PM vm finance <vm.finance2 at gmail.com> wrote:
> Hi Randall,
>
> I applied the patch available at:
> https://reviews.freebsd.org/D21582?id=62213
> NOTE: This raw patch seems to be exactly the same as the one I downloaded on Sep10...hope I'm not missing anything here...
>
> I clicked on "download raw diff" and saved it to bbr_patch.diff file on my local machine
> Step 1: Apply PATCH:
> cd /usr/src/sys
> patch < bbr_patch.diff
> Patch was applied successfully - no errors seen there.
>
> Step 2: Update kernel config:
> cp /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/MYKERNEL
> Added three options to MYKERNEL (per ur email - I have also attached the compile MYKERNEL file)
> cd /usr/src/sys/amd64/compile/MYKERNEL
> make cleandepend
> =>This fails due to missing "bbr" under /usr/src/sys/modules/tcp/
> [pls find the attached log file for exact error]
>
> make depend
> => This also fails due to missing "/usr/src/sys/modules/tcp/bbr"
>
> STEP 3:
> cd /usr/src/
> make buildkernel KERNCONF=MYKERNEL
> This also fails due to prior errors in Step 2.
>
> Please find MYKERNEL config file + error logs.
>
> Many many thanks for taking time to help me through this!
>
> PS: I did a make buildkernel without any KERNCONF=MYKERNEL, and that successfully finished.
>
>
>
> On Thu, Sep 19, 2019 at 12:19 PM Randall Stewart <rrs at netflix.com> wrote:
> You can look in the config I sent.. but here is what
> I have added to enable BBR and Rack to be built
> *****
> makeoptions WITH_EXTRA_TCP_STACKS=1
> options TCPHPTS
> options RATELIMIT
> ******
>
> So you should
> 1) Apply the current patch in phabricator
> 2) edit your config and add the above three lines
> 3) go to the src dir and type
> make buildkernel KERNCONF=myconf
>
> That should build and get you a kernel that will include bbr and rack. You will
> need to, after it boots up type
>
> kldload tcp_bbr
>
> or
>
> kldload tcp_rack
>
> To get the modules loaded
>
> R
>
>
>
> > On Sep 18, 2019, at 2:31 PM, vm finance <vm.finance2 at gmail.com> wrote:
> >
> > Hi Randall, Michael,
> >
> > Build is done successfully. I would appreciate if you could share config changes needed for BBR.
> >
> > My svn revision is: 352483
> >
> > =======================================================
> > awk -f /usr/src/sys/conf/kmod_syms.awk zlib.ko.full export_syms | xargs -J% objcopy % zlib.ko.full
> > objcopy --only-keep-debug zlib.ko.full zlib.ko.debug
> > objcopy --strip-debug --add-gnu-debuglink=zlib.ko.debug zlib.ko.full zlib.ko
> > --------------------------------------------------------------
> > >>> Kernel build for GENERIC completed on Wed Sep 18 21:08:31 UTC 2019
> > --------------------------------------------------------------
> > >>> Kernel(s) GENERIC built in 1972 seconds, ncpu: 4
> > =======================================================
> >
> > Thank you
> > Vishal.
> >
> >
> > On Wed, Sep 18, 2019 at 3:34 PM vm finance <vm.finance2 at gmail.com> wrote:
> > I'm using amd64. I'd get back as soon as base build is complete.
> >
> > Thanks!
> >
> > On Wed, Sep 18, 2019 at 7:09 AM Randall Stewart <rrs at netflix.com> wrote:
> > To get bbr running you will need to change your kernel config.
> >
> > Are you building i386 or amd64?
> >
> > After you have successfully did
> >
> > 1) buildworld
> > 2) buildkernel
> > 3) installkernel
> >
> > (you can look in UPDATING for instructions .. though the file is long :D)
> >
> > successfully let me know.. and then I will give you the tweaks you need to add
> > to the kerneconf.
> >
> > It won’t take as long to build because at that point you can add in the
> > NO_CLEAN NO_CLEANDIR options as well since you will have built everything
> > the first time
> >
> > R
> >
> > > On Sep 18, 2019, at 7:06 AM, vm finance <vm.finance2 at gmail.com> wrote:
> > >
> > > BTW, if you think I should be making any changes in configs, please do let me know.
> > > My goal is to build a freebsd image with BBR patches on x86 VM.
> > > Nothing fancy.
> > >
> > > thanks!
> > >
> > > On Wed, Sep 18, 2019 at 7:03 AM vm finance <vm.finance2 at gmail.com> wrote:
> > > Thanks Randall, Michael,
> > >
> > > I did "svn svn://svn.freebsd.org/base/head /usr/src"
> > > and now doing "make buildworld buildkernel"
> > > [I didn't change anything in configs - just whatever are the defaults]
> > >
> > > I would update as soon as its done.
> > >
> > > Thanks again!
> > >
> > > On Wed, Sep 18, 2019 at 6:53 AM Randall Stewart <rrs at netflix.com> wrote:
> > > One other note.. I notice his kernel conf he sent does not have
> > > the right things to get BBR even to attempt to build.
> > >
> > > I would suggest using that config for the first steps.. then he must add
> > > the additional tcp stacks and the hpts in order to get bbr/rack and any other
> > > extra stack….
> > >
> > > But I would suggest to build with his current config first and once he has
> > > that in place and building
> > >
> > > a) apply the patch
> > > b) add the extra kernel options
> > >
> > > R
> > >
> > > > On Sep 18, 2019, at 6:50 AM, Randall Stewart <rrs at netflix.com> wrote:
> > > >
> > > > Thats great idea Michael.
> > > >
> > > > From the look fo the build log I was sent, his blow-up has nothing to do
> > > > with the patches.
> > > >
> > > > He should probably
> > > >
> > > > 1) Check out a fresh version of head.
> > > > 2) Follow the instructions in UPDATING to get a clean build.
> > > > — make buildworld
> > > > — make buildkernel KERNCONF=his-conf
> > > > — make installkernel KERNCONF=his-conf
> > > >
> > > > ** do the reboot and merge master **
> > > >
> > > > Then after that he could
> > > >
> > > > 3) Apply the current patch in the review
> > > > 4) do build that
> > > > — make buildkernel KERNCONF=his_conf -DNO_CLEAN -DNO_CLEANDIR
> > > > — make install kernel KERNCONF=his_conf
> > > >
> > > > And he should have it all working :)
> > > >
> > > > I have done multiple builds with the latest head from yesterday both with and without
> > > > the extra stacks and hpts.. and I can’t get a build failure.. this would verify he can
> > > > at least build head..
> > > >
> > > > R
> > > >
> > > >> On Sep 18, 2019, at 12:10 AM, Michael Tuexen <michael.tuexen at lurchi.franken.de> wrote:
> > > >>
> > > >>> On 18. Sep 2019, at 08:19, vm finance <vm.finance2 at gmail.com> wrote:
> > > >>>
> > > >>> correcting a typo:
> > > >>>
> > > >>> svn co svn://svn.freebsd.org/base/head /usr/src
> > > >>> current revision: 352434
> > > >> I suggest to build/install head first without any patches. After that has worked,
> > > >> apply the patches you are interested in. That way it is easy to separate generic
> > > >> build issues and issue specific to a patch.
> > > >>
> > > >> Best regards
> > > >> Michael
> > > >>>
> > > >>> Thank you!
> > > >>>
> > > >>> On Tue, Sep 17, 2019 at 10:11 PM vm finance <vm.finance2 at gmail.com> wrote:
> > > >>>
> > > >>>> Actually I am on head already as mentioned previously. Pulled it using
> > > >>>> yesterday:
> > > >>>>
> > > >>>> svn co svn://svn.freebsd.org/base/head /use/src
> > > >>>>
> > > >>>>
> > > >>>> If you could pls let me know the new patch, I can try that.
> > > >>>>
> > > >>>> Thanks
> > > >>>>
> > > >>>> Sent from my iPhone
> > > >>>>
> > > >>>> On 18-Sep-2019, at 8:56 AM, Randall Stewart <rrs at netflix.com> wrote:
> > > >>>>
> > > >>>> There have been several patches pre-this one that provide
> > > >>>> the infrastructure to support BBR.
> > > >>>>
> > > >>>> Release 12.0 will *not* have these patches and will *not* compile it.
> > > >>>>
> > > >>>> I have no intention at this point in doing a MFC of this work.. so if you
> > > >>>> want
> > > >>>> to run BBR you need to run Head
> > > >>>>
> > > >>>> R
> > > >>>>
> > > >>>> On Sep 17, 2019, at 7:26 PM, vm finance <vm.finance2 at gmail.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>> Hi Randall,
> > > >>>>
> > > >>>>
> > > >>>> Could you please provide a pointer to the latest patch. I had applied the
> > > >>>> one you published Sep-10.
> > > >>>>
> > > >>>>
> > > >>>> Following is what I have done:
> > > >>>>
> > > >>>> 1. Picked FreeBSD 12.0 RELEASE VM from osboxes.org
> > > >>>>
> > > >>>> 2. Got VM up under VMPlayer on x86 laptop
> > > >>>>
> > > >>>> 3. Checked out latest codebase from freebsd repo:
> > > >>>>
> > > >>>> svn co svn://svn.freebsd.org/base/head /usr/src*
> > > >>>>
> > > >>>> cd /usr/src;
> > > >>>>
> > > >>>> make buildworld buildkernel
> > > >>>>
> > > >>>> [this is where make fails when patch is applied]
> > > >>>>
> > > >>>>
> > > >>>> I could try to build with your latest patch - please send me a pointer.
> > > >>>>
> > > >>>>
> > > >>>> Thanks for your help!
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Sep 17, 2019 at 1:33 PM Randall Stewart <rrs at netflix.com> wrote:
> > > >>>>
> > > >>>> Looking at your make file log I can’t really tell what you are doing.
> > > >>>>
> > > >>>>
> > > >>>> Its not the BBR or Rack code that is blowing up…
> > > >>>>
> > > >>>>
> > > >>>> Are you cross compiling?
> > > >>>>
> > > >>>>
> > > >>>> I have done the old fashioned kernel make
> > > >>>>
> > > >>>>
> > > >>>> i.e.
> > > >>>>
> > > >>>>
> > > >>>> cd src/sys/amd64/config
> > > >>>>
> > > >>>> config headvm
> > > >>>>
> > > >>>> cd ../compile/headvm
> > > >>>>
> > > >>>>
> > > >>>> make cleandepend ; make depend; make -j3
> > > >>>>
> > > >>>>
> > > >>>> I have done
> > > >>>>
> > > >>>>
> > > >>>> cd src
> > > >>>>
> > > >>>> make buildkernel KERNCONF=headvm
> > > >>>>
> > > >>>>
> > > >>>> And even
> > > >>>>
> > > >>>>
> > > >>>> make buildkernel KERNCONF=GENERIC
> > > >>>>
> > > >>>>
> > > >>>> to make sure that the build works without bbr.
> > > >>>>
> > > >>>>
> > > >>>> I attach my headvm config..
> > > >>>>
> > > >>>>
> > > >>>> What exactly are you building and how? kernel config too please?
> > > >>>>
> > > >>>>
> > > >>>> R
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Sep 17, 2019, at 1:11 PM, Randall Stewart <rrs at netflix.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>> looking
> > > >>>>
> > > >>>>
> > > >>>> I was at 352408.. let me update and try it
> > > >>>>
> > > >>>>
> > > >>>> R
> > > >>>>
> > > >>>>
> > > >>>> On Sep 17, 2019, at 1:10 PM, Randall Stewart <rrs at netflix.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>> Hmm
> > > >>>>
> > > >>>>
> > > >>>> Did you get the patch I updated too this am?
> > > >>>>
> > > >>>>
> > > >>>> I have built it both with and without the bbr stack and had no issue..
> > > >>>> there was
> > > >>>>
> > > >>>> an issue with KTLS before the update though.
> > > >>>>
> > > >>>>
> > > >>>> I don’t recognize what you have below there though…
> > > >>>>
> > > >>>>
> > > >>>> R
> > > >>>>
> > > >>>>
> > > >>>> On Sep 17, 2019, at 11:47 AM, vm finance <vm.finance2 at gmail.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>> Got it - thank you!
> > > >>>>
> > > >>>>
> > > >>>> btw, I tried to build the patch but its giving an error. Following is SVN
> > > >>>> info + make error mesg.
> > > >>>>
> > > >>>>
> > > >>>> Please let me know what am I missing here?
> > > >>>>
> > > >>>>
> > > >>>> Thanks!
> > > >>>>
> > > >>>>
> > > >>>> #svnlite revision
> > > >>>>
> > > >>>> Path: .
> > > >>>>
> > > >>>> Working Copy Root Path: /usr/src
> > > >>>>
> > > >>>> URL: svn://svn.freebsd.org/base/head
> > > >>>>
> > > >>>> Relative URL: ^/head
> > > >>>>
> > > >>>> Repository Root: svn://svn.freebsd.org/base
> > > >>>>
> > > >>>> Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
> > > >>>>
> > > >>>> Revision: 352436
> > > >>>>
> > > >>>> Node Kind: directory
> > > >>>>
> > > >>>> Schedule: normal
> > > >>>>
> > > >>>> Last Changed Author: jah
> > > >>>>
> > > >>>> Last Changed Rev: 352434
> > > >>>>
> > > >>>> Last Changed Date: 2019-09-17 03:39:31 +0000 (Tue, 17 Sep 2019)
> > > >>>>
> > > >>>>
> > > >>>> root at osboxes:/usr/src #
> > > >>>>
> > > >>>>
> > > >>>> -------------------------------------------- snip
> > > >>>> ----------------------------
> > > >>>>
> > > >>>>
> > > >>>> ad_elf64_obj.llo reloc_elf64.llo disk.llo part.llo vdisk.llo dev_net.llo
> > > >>>> bcache.llo interp_simple.llo zfs_cmd.llo
> > > >>>>
> > > >>>> rm -f .depend .depend.* GPATH GRTAGS GSYMS GTAGS
> > > >>>>
> > > >>>> ===> sys (cleandir)
> > > >>>>
> > > >>>> rm -f export_syms machine x86 tcp_bbr.ko tcp_bbr.kld bbr.o sack_filter.o
> > > >>>> rack_bbr_common.o opt_inet.h opt_inet6.h opt_ipsec.h opt_tcpdebug.h
> > > >>>> opt_kern_tls.h
> > > >>>>
> > > >>>> rm: x86: is a directory
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make[3]: stopped in /usr/src/sys
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make[2]: stopped in /usr/src
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make[1]: stopped in /usr/src
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make: stopped in /usr/src
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Sep 17, 2019 at 6:41 PM vm finance <vm.finance2 at gmail.com> wrote:
> > > >>>>
> > > >>>> Got it - thank you!
> > > >>>>
> > > >>>>
> > > >>>> btw, I tried to build the patch but its giving an error. Following is SVN
> > > >>>> info + make error mesg.
> > > >>>>
> > > >>>> I have also attached the entire build log...snippet is below
> > > >>>>
> > > >>>>
> > > >>>> Please let me know what am I missing here?
> > > >>>>
> > > >>>>
> > > >>>> Thanks!
> > > >>>>
> > > >>>>
> > > >>>> #svnlite revision
> > > >>>>
> > > >>>> Path: .
> > > >>>>
> > > >>>> Working Copy Root Path: /usr/src
> > > >>>>
> > > >>>> URL: svn://svn.freebsd.org/base/head
> > > >>>>
> > > >>>> Relative URL: ^/head
> > > >>>>
> > > >>>> Repository Root: svn://svn.freebsd.org/base
> > > >>>>
> > > >>>> Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
> > > >>>>
> > > >>>> Revision: 352436
> > > >>>>
> > > >>>> Node Kind: directory
> > > >>>>
> > > >>>> Schedule: normal
> > > >>>>
> > > >>>> Last Changed Author: jah
> > > >>>>
> > > >>>> Last Changed Rev: 352434
> > > >>>>
> > > >>>> Last Changed Date: 2019-09-17 03:39:31 +0000 (Tue, 17 Sep 2019)
> > > >>>>
> > > >>>>
> > > >>>> root at osboxes:/usr/src #
> > > >>>>
> > > >>>>
> > > >>>> -------------------------------------------- snip
> > > >>>> ----------------------------
> > > >>>>
> > > >>>>
> > > >>>> ad_elf64_obj.llo reloc_elf64.llo disk.llo part.llo vdisk.llo dev_net.llo
> > > >>>> bcache.llo interp_simple.llo zfs_cmd.llo
> > > >>>>
> > > >>>> rm -f .depend .depend.* GPATH GRTAGS GSYMS GTAGS
> > > >>>>
> > > >>>> ===> sys (cleandir)
> > > >>>>
> > > >>>> rm -f export_syms machine x86 tcp_bbr.ko tcp_bbr.kld bbr.o sack_filter.o
> > > >>>> rack_bbr_common.o opt_inet.h opt_inet6.h opt_ipsec.h opt_tcpdebug.h
> > > >>>> opt_kern_tls.h
> > > >>>>
> > > >>>> rm: x86: is a directory
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make[3]: stopped in /usr/src/sys
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make[2]: stopped in /usr/src
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make[1]: stopped in /usr/src
> > > >>>>
> > > >>>> *** Error code 1
> > > >>>>
> > > >>>>
> > > >>>> Stop.
> > > >>>>
> > > >>>> make: stopped in /usr/src
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Sep 17, 2019 at 6:27 PM Randall Stewart <rrs at netflix.com> wrote:
> > > >>>>
> > > >>>> Pacing is provided by tcp_hpts.c. The current linux patches do not have
> > > >>>>
> > > >>>> to have fq.. they built an alternate means of doing pacing into bbr.
> > > >>>>
> > > >>>>
> > > >>>> In either case our testing has shown that our pacing is more accurate than
> > > >>>>
> > > >>>> either fq or the internal pacer :)
> > > >>>>
> > > >>>>
> > > >>>> R
> > > >>>>
> > > >>>>
> > > >>>> On Sep 17, 2019, at 11:05 AM, vm finance <vm.finance2 at gmail.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>> Thanks Randall.
> > > >>>>
> > > >>>> I was able to apply the patch - now rebuilding the kernel. Would update on
> > > >>>> how it goes.
> > > >>>>
> > > >>>>
> > > >>>> BTW, is there any description on how lack of tc_fq under FreeBSD is
> > > >>>> compensated here?
> > > >>>>
> > > >>>> The original BBR patches on Linux show that as a must-have? Is that
> > > >>>> functionality implemented via tcp_ratelimit.[ch]?
> > > >>>>
> > > >>>>
> > > >>>> Any pointers to existing presentations/discussions highly appreciated?
> > > >>>>
> > > >>>>
> > > >>>> Thanks a lot.
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Sep 17, 2019 at 5:39 AM Randall Stewart <rrs at netflix.com> wrote:
> > > >>>>
> > > >>>> You should be able to compile it against the current head. I re-doing that
> > > >>>> now (had an
> > > >>>>
> > > >>>> issue with my machine and had to roll it back to a backup).
> > > >>>>
> > > >>>>
> > > >>>> When I put the patch up on Sept 10th it complied with and without BBR on
> > > >>>> whatever
> > > >>>>
> > > >>>> was that rev..
> > > >>>>
> > > >>>>
> > > >>>> Looking in the commit logs that would have been around 352191
> > > >>>>
> > > >>>>
> > > >>>> Hope that helps
> > > >>>>
> > > >>>>
> > > >>>> R
> > > >>>>
> > > >>>>
> > > >>>> On Sep 17, 2019, at 3:32 AM, vm finance <vm.finance2 at gmail.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>> Hi Randall,
> > > >>>>
> > > >>>>
> > > >>>> Thanks for releasing BBR patch:
> > > >>>>
> > > >>>> https://reviews.freebsd.org/D21582#change-xcAWBif3E9Jq
> > > >>>>
> > > >>>>
> > > >>>> Could you please let me know what SVN/GIT label tag this is based on? I
> > > >>>> would like to patch and experiment with it. I couldn't find this info in
> > > >>>> the released patch.
> > > >>>>
> > > >>>>
> > > >>>> Thanks a lot!
> > > >>>>
> > > >>>>
> > > >>>> On Tue, Sep 10, 2019 at 10:26 AM Ryan Stone <rysto32 at gmail.com> wrote:
> > > >>>>
> > > >>>> rrs@ has just posted the BBR patch to phabricator:
> > > >>>>
> > > >>>>
> > > >>>> https://reviews.freebsd.org/D21582
> > > >>>>
> > > >>>>
> > > >>>> ------
> > > >>>>
> > > >>>> Randall Stewart
> > > >>>>
> > > >>>> rrs at netflix.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ------
> > > >>>>
> > > >>>> Randall Stewart
> > > >>>>
> > > >>>> rrs at netflix.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ------
> > > >>>>
> > > >>>> Randall Stewart
> > > >>>>
> > > >>>> rrs at netflix.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ------
> > > >>>>
> > > >>>> Randall Stewart
> > > >>>>
> > > >>>> rrs at netflix.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ------
> > > >>>>
> > > >>>> Randall Stewart
> > > >>>>
> > > >>>> rrs at netflix.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ------
> > > >>>> Randall Stewart
> > > >>>> rrs at netflix.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>> _______________________________________________
> > > >>> freebsd-net at freebsd.org mailing list
> > > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > > >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> > > >
> > > > ------
> > > > Randall Stewart
> > > > rrs at netflix.com
> > > >
> > > >
> > > >
> > >
> > > ------
> > > Randall Stewart
> > > rrs at netflix.com
> > >
> > >
> > >
> >
> > ------
> > Randall Stewart
> > rrs at netflix.com
> >
> >
> >
>
> ------
> Randall Stewart
> rrs at netflix.com
>
>
>
------
Randall Stewart
rrs at netflix.com
More information about the freebsd-net
mailing list