Much improved sendfile(2) kernel implementation
gallatin at cs.duke.edu
Fri Sep 22 18:47:39 PDT 2006
Andre Oppermann writes:
> Andrew Gallatin wrote:
> > Between TSO and your sendfile changes, things are looking up!
> > Here are some Myri10GbE 1500 byte results from a 1.8GHz UP
> > FreeBSD/amd64 machine (AMD Athlon(tm) 64 Processor 3000+) sending to a
> > 2.0GHz SMP Linux/x86_64 machine (AMD Athlon(tm) 64 X2 Dual Core Processor
> > 3800+) running 26.17.7smp and our 1.1.0 Myri10GE driver (with LRO).
> > I used a linux receiver because LRO is the only way to receive
> > standard frames at line rate (without a TOE).
> > These tests are all for sendfile of a 10MB file in /var/tmp:
> > % netperf242 -Hrome-my -tTCP_SENDFILE -F /var//tmp/zot -T,1 -c -C -- -s393216
> You should use -m5M as well. netperf is kinda dumb and does only
I will try this.
> > The -T,1 is required to force the netserver to use a different core
> > than the interrupt handler is bound to on the linux machine. BTW,
> > it would be really nice if FreeBSD supported CPU affinity for processes
> > and interrupt handlers..
> I have a gross version of that in my tree. The kernel itself supports
> it but it's not yet exposed to userland for manual intervention.
> Be a bit careful with the CPU usage figures. The numbers netperf reports
> differ quite a bit from those reported by time(1) on the high side. And
> there are some differences in the approach how FreeBSD and Linux do their
> statistical measurements of user and system time. This doesn't change the
Netperf essentially subtracts idle time from all other time. The
numbers I see reported from netperf have historically matched what
vmstat says. BTW, I use netperf from svn. I don't use the
> throughput number though. But see the -m5M option. New sendfile is really
> optimized to chew on a large file (larger than the socket buffer size) as
> it normally happens in reality.
> > For comparision, if I reboot the sender into RHEL (Linux 2.6.9-11.EL x86_64):
> > 87380 65536 65536 10.01 9333.00 28.98 75.23 0.254 1.321
> > The above results are the median result for 5 runs at each setting.
> How large is the variance between the runs?
Much bigger for FreeBSD than for Linux. I think I had one 9.1 and 4
9.3s for linux. I had a 6.2 a 6.5 and 3 around 6.9 for FreeBSD,
More information about the freebsd-net