serious networking (em) performance (ggate and NFS) problem

[Emanuel Strobl]

Am Freitag, 19. November 2004 13:56 schrieb Robert Watson:
> On Fri, 19 Nov 2004, Emanuel Strobl wrote:
> > Am Donnerstag, 18. November 2004 13:27 schrieb Robert Watson:
> > > On Wed, 17 Nov 2004, Emanuel Strobl wrote:
> > > > I really love 5.3 in many ways but here're some unbelievable transfer
[...]
> Well, the claim that if_em doesn't benefit from polling is inaccurate in
> the general case, but quite accurate in the specific case.  In a box with
> multiple NIC's, using polling can make quite a big difference, not just by
> mitigating interrupt load, but also by helping to prioritize and manage
> the load, preventing live lock.  As I indicated in my earlier e-mail,

I understand, thanks for the explanation

> It looks like the netperf TCP test is getting just under 27MB/s, or
> 214Mb/s.  That does seem on the low side for the PCI bus, but it's also

Nut sure if I understand that sentence correctly, does it mean the "slow" 
400MHz PII is causing this limit? (low side for the PCI bus?)

> instructive to look at the netperf UDP_STREAM results, which indicate that
> the box believes it is transmitting 417Mb/s but only 67Mb/s are being
> received or processed fast enough by netserver on the remote box.  This
> means you've achieved a send rate to the card of about 54Mb/s.  Note that
> you can actually do the math on cycles/packet or cycles/byte here -- with
> TCP_STREAM, it looks like some combination of recipient CPU and latency
> overhead is the limiting factor, with netserver running at 94% busy.

Hmm, I can't puzzle a picture out of this. 

>
> Could you try using geom gate to export a malloc-backed md device, and see
> what performance you see there?  This would eliminate the storage round

It's a pleasure:

test2:~#15: dd if=/dev/zero of=/mdgate/testfile bs=16k count=6000
6000+0 records in
6000+0 records out
98304000 bytes transferred in 5.944915 secs (16535812 bytes/sec)
test2:~#17: dd if=/mdgate/testfile of=/dev/null bs=16k
6000+0 records in
6000+0 records out
98304000 bytes transferred in 5.664384 secs (17354755 bytes/sec)

This time it's no difference between disk and memory filesystem, but on 
another machine with a ich2 chipset and a 3ware controller (my current 
productive system which I try to replace with this project) there was a big 
difference. Attached is the corresponding message.

Thanks,

-Harry

> trip and guarantee the source is in memory, eliminating some possible
> sources of synchronous operation (which would increase latency, reducing
> throughput).  Looking at CPU consumption here would also be helpful, as it
> would allow us to reason about where the CPU is going.
>
> > I was aware of that and because of lacking a GbE switch anyway I decided
> > to use a simple cable ;)
>
> Yes, this is my favorite configuration :-).
>
> > > (5) Next, I'd measure CPU consumption on the end box -- in particular,
> > > use top -S and systat -vmstat 1 to compare the idle condition of the
> > > system and the system under load.
> >
> > I additionally added these values to the netperf results.
>
> Thanks for your very complete and careful testing and reporting :-).
>
> Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
> robert at fledge.watson.org      Principal Research Scientist, McAfee Research
>
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
-------------- next part --------------
An embedded message was scrubbed...
From: Emanuel Strobl <Emanuel.Strobl at gmx.net>
Subject: Re: asymmetric NFS transfer rates
Date: Mon, 8 Nov 2004 04:29:11 +0100
Size: 4557
Url: http://lists.freebsd.org/pipermail/freebsd-current/attachments/20041119/ab85f6a0/attachment.mht
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-current/attachments/20041119/ab85f6a0/attachment.bin