From freebsd at sopwith.solgatos.com Thu Oct 1 04:11:52 2009 From: freebsd at sopwith.solgatos.com (Dieter) Date: Thu Oct 1 04:11:59 2009 Subject: A specific example of a disk i/o problem In-Reply-To: Your message of "Wed, 30 Sep 2009 08:30:34 +0200." <3bbf2fe10909292330t753bcad1r69ae67d7e898ee35@mail.gmail.com> Message-ID: <200909301955.TAA20656@sopwith.solgatos.com> > >> > My question is why is FreeBSD's disk i/o performance so bad? > > Here is a specific demo of one disk i/o problem I'm seeing. Should be > > easy to reproduce? > > > > http://lists.freebsd.org/pipermail/freebsd-performance/2008-July/003533.html > > > > This was over a year ago, so add 7.1 to the list of versions with the problem. > > I believe that the > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 1148109, size: 4096 > > messages I'm getting are the same problem. A user process is hogging > > the bottleneck (disk buffer cache?) and the swapper/pager is getting starved. > > Sorry, do you have a PR/describing e-mail with this issue? Can you be > a bit more precise? I have not submitted a PR for this particular problem. (yet) The hardware seems to work fine. A single process can access a disk at full speed, over 100 MB/s for recent 7200 rpm SATA drives. Same for the nforce4-ultra (chipset), JMB363 (PCIe card), or SiI 3132 (PCIe card) controllers. Same for Hitachi, Seagate, Samsung, or WD drives. CPU bound processes play well together. The problem is when I run a disk i/o bound process like cat, dd, etc. The i/o bound process sucks up some resource and other processes get starved for disk i/o not just for milliseconds, but for seconds, even minutes. The example in .../2008-July/003533.html uses a single disk, but the problem also occurs across disks and across controllers. Coming up with a demo using multiple disks that would be easy for someone else to duplicate is more difficult, which is why the demo uses a single disk. It happens with both reading and writing. I don't think it has anything to do with the filesystem (FFS with softdeps). It doesn't matter which process starts first. Given the behaviour, the bottleneck must be something that is common to all the disks, such as the disk cache. The BSD kernel has changed significantly since I took the internals class, so my understanding of the internals is somewhat obsolete. But my best guess is that the bottleneck is some kernel disk cache or disk job queue that the i/o bound job fills up and keeps filled up, and other processes rarely get a chance to get their i/o requests in. Nice, even idprio, has little if any effect. On the machines that Unix grew up on (PDP11, VAX) the CPU was nearly always the scarce resource, so the scheduler doesn't penalize a process for using lots of i/o. This is a serious problem on current hardware. There is no way to keep one process's i/o from interferring with another process. > The problem reported in the earlier post, however, is interesting and > worths more analysis. Can anyone reproduce it? > More speficially, would you be interested in reproducing and playing a > bit with some diagnostic tool/configurations I can point you at? I would welcome info on diagnosing/config/tuning/etc. From wmoran at collaborativefusion.com Thu Oct 1 13:10:45 2009 From: wmoran at collaborativefusion.com (Bill Moran) Date: Thu Oct 1 13:10:51 2009 Subject: A specific example of a disk i/o problem In-Reply-To: <200909301955.TAA20656@sopwith.solgatos.com> References: <3bbf2fe10909292330t753bcad1r69ae67d7e898ee35@mail.gmail.com> <200909301955.TAA20656@sopwith.solgatos.com> Message-ID: <20091001091043.477f4b9b.wmoran@collaborativefusion.com> In response to Dieter : > > >> > My question is why is FreeBSD's disk i/o performance so bad? > > > > Here is a specific demo of one disk i/o problem I'm seeing. Should be > > > easy to reproduce? > > > > > > http://lists.freebsd.org/pipermail/freebsd-performance/2008-July/003533.html FYI, I thought I'd play around with this some in an attempt to add some useful information to the investigation. I can not reproduce the problem at all. I created a 9G file, did the cat as described in the above URL, and the man request completed in roughly the same time it did without the cat running. Just to mix it up a bit, I tried running ls -R on a large directory tree while the cat was running as well, and performance did not seem to be significantly impacted there, either. I ran the tests on my work machine, which is a Dell Optiplex 960 running FreeBSD 7.2-RELEASE-p3 i386. Some relevant dmesg stuff: atapci0: port 0xfe80-0xfe87,0xfe90-0xfe93,0xfea0-0xfea7,0xfeb0-0xfeb3,0xfef0-0xfeff irq 18 at device 3.2 on pci0 atapci1: port 0xfe00-0xfe07,0xfe10-0xfe13,0xfe20-0xfe27,0xfe30-0xfe33,0xfec0-0xfedf mem 0xff970000-0xff9707ff irq 18 at device 31.2 on pci0 ad14: 476940MB at ata7-master SATA150 -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ From freebsd at sopwith.solgatos.com Fri Oct 2 05:11:54 2009 From: freebsd at sopwith.solgatos.com (Dieter) Date: Fri Oct 2 05:15:04 2009 Subject: A specific example of a disk i/o problem In-Reply-To: Your message of "Thu, 01 Oct 2009 09:10:43 EDT." <20091001091043.477f4b9b.wmoran@collaborativefusion.com> Message-ID: <200910020507.FAA13295@sopwith.solgatos.com> > > > >> > My question is why is FreeBSD's disk i/o performance so bad? > > > > > > Here is a specific demo of one disk i/o problem I'm seeing. Should be > > > > easy to reproduce? > > > > > > > > http://lists.freebsd.org/pipermail/freebsd-performance/2008-July/003533.html > > FYI, I thought I'd play around with this some in an attempt to add some > useful information to the investigation. > > I can not reproduce the problem at all. I created a 9G file, did the cat > as described in the above URL, and the man request completed in roughly > the same time it did without the cat running. How large is main memory on this machine? e.g. is 9 GB large enough to flush everything else out of the disk cache before running the man command again? I haven't studied the new unified memory thing, but if we assume worst case, reading at 50 M/s would take 40 seconds to flush 2 GiB. BTW there is nothing magic about a 9 GB file, just that it is large enough to flush the 2 GiB of main memory on my machine and takes long enough to read to notice a difference in how long it takes to run man. Updated demo, just to make sure: # big_file is larger than main memory, on same disk as man (/usr) time man de # get baseline time for man command without competing i/o cat big_file > /dev/null # flush man command and data from memory cat big_file > /dev/null & # generate i/o, attempt to use up bottleneck time man de # see how much longer man takes with competing i/o From mahlerrd at yahoo.com Fri Oct 2 17:29:51 2009 From: mahlerrd at yahoo.com (Richard Mahlerwein) Date: Fri Oct 2 17:29:57 2009 Subject: A specific example of a disk i/o problem Message-ID: <160343.45338.qm@web51006.mail.re2.yahoo.com> >From: Dieter >Subject: Re: A specific example of a disk i/o problem >To: freebsd-performance@freebsd.org >Date: Friday, October 2, 2009, 1:07 AM > >Updated demo, just to make sure: > ># big_file is larger than main memory, on same disk as man (/usr) >time man de??? ??? >cat big_file > /dev/null??? >cat big_file > /dev/null &? >time man de Trying it on something that surely will have i/o issues: PIII 500 w/ 60 GB PATA drive and 256 MB RAM.? Blank minimal install + man pages right now.? I'm trying with a "file /etc/" command instead of trying to reliably time man... It seems the file command always hits the disk.? This should also make any sort of "flushing" unecessary, shouldn't it? Anyway, I tried it with the man command but couldn't discern any difference at all with it, so I'm repeating with "file" which at least gives SOME variance. liebnitz# uname -a FreeBSD xyz 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May? 1 08:49:13 UTC 2009? ???root@walker.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC? i386 liebnitz# time dd if=/dev/zero of=/usr/bigfile count=4096 bs=1M 4096+0 records in 4096+0 records out 4294967296 bytes transferred in 363.700967 secs (11809062 bytes/sec) 0.055u 83.971s 6:03.78 23.0%? ? 25+1071k 128+34391io 0pf+0w liebnitz# time file /etc/ [run 5 times] 4.052u 0.047s 0:04.14 98.7%? ??15+1085k 0+0io 0pf+0w 4.068u 0.031s 0:04.13 99.0%????16+1095k 0+0io 0pf+0w 4.052u 0.046s 0:04.13 99.0%????16+1101k 0+0io 0pf+0w 4.044u 0.054s 0:04.13 99.0%????16+1092k 0+0io 0pf+0w 4.052u 0.046s 0:04.13 99.0%????15+1091k 0+0io 0pf+0w Ok, so that looks pretty repeatable.? Now... liebnitz# cat /usr/bigfile > /dev/null & [1] 37923 liebnitz# time file /etc/* [again run 5 times] 4.146u 0.038s 0:06.80 61.3%????15+1103k 87+0io 8pf+0w 4.128u 0.054s 0:06.73 61.9%????15+1095k 93+0io 6pf+0w 4.127u 0.056s 0:06.65 62.7% 14+1058k 88+0io 4pf+0w 4.111u 0.072s 0:06.84 61.1% 15+1049k 91+0io 6pf+0w 4.115u 0.066s 0:06.86 60.7% 15+1022k 91+0io 6pf+0w BTW, after the cat is done... liebnitz# time file /etc/* 4.051u 0.047s 0:04.13 99.0% 15+1078k 0+0io 0pf+0w I see no unexpected issues, only a minor slowdown, and that could be processer contention, IMO. (Sorry for duplicate, I was rewriting with the change to "file" instead of "man" and the two year old decided to press some keys and sent the message.) From mahlerrd at yahoo.com Fri Oct 2 17:44:07 2009 From: mahlerrd at yahoo.com (Richard Mahlerwein) Date: Fri Oct 2 17:44:13 2009 Subject: A specific example of a disk i/o problem Message-ID: <969908.40235.qm@web51006.mail.re2.yahoo.com> >From: Dieter >Subject: Re: A specific example of a disk i/o problem >To: freebsd-performance@freebsd.org >Date: Friday, October 2, 2009, 1:07 AM > >Updated demo, just to make sure: > ># big_file is larger than main memory, on same disk as man (/usr) >time man de??? ??? # get baseline time for man command without >competing i/o >cat big_file > /dev/null???# flush man command and data from memory >cat big_file > /dev/null &?# generate i/o, attempt to use up bottleneck >time man de? ??? # see how much longer man takes with competing >i/o Trying it on something that surely will have i/o issues: PIII 500 w/ 60 GB PATA drive and 128 MB RAM. Blank minimal install + man pages right now. I'm trying with a "file /etc/" command instead of trying to reliably time man... It seems the file command always hits the disk. This should also make any sort of "flushing" unecessary, shouldn't it? liebnitz# uname -a FreeBSD xyz 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May 1 08:49:13 UTC 2009 root@walker.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 liebnitz# time dd if=/dev/zero of=/usr/bigfile count=4096 bs=1M 4096+0 records in 4096+0 records out 4294967296 bytes transferred in 363.700967 secs (11809062 bytes/sec) 0.055u 83.971s 6:03.78 23.0% 25+1071k 128+34391io 0pf+0w liebnitz# time file /etc/ [run 5 times] 4.052u 0.047s 0:04.14 98.7% 15+1085k 0+0io 0pf+0w 4.068u 0.031s 0:04.13 99.0% 16+1095k 0+0io 0pf+0w 4.052u 0.046s 0:04.13 99.0% 16+1101k 0+0io 0pf+0w 4.044u 0.054s 0:04.13 99.0% 16+1092k 0+0io 0pf+0w 4.052u 0.046s 0:04.13 99.0% 15+1091k 0+0io 0pf+0w Ok, so that looks pretty repeatable. Now... liebnitz# cat /usr/bigfile > /dev/null & [1] 37923 liebnitz# time file /etc/* [again run 5 times] 4.146u 0.038s 0:06.80 61.3% 15+1103k 87+0io 8pf+0w 4.128u 0.054s 0:06.73 61.9% 15+1095k 93+0io 6pf+0w Flush as example. liebnitz# cat /usr/bigfile > /dev/null liebnitz# cat /usr/bigfile > /dev/null & [1] 37890 From lists at stringsutils.com Fri Oct 2 18:26:25 2009 From: lists at stringsutils.com (Francisco Reyes) Date: Fri Oct 2 18:26:31 2009 Subject: Performance evaluation of PostgreSQL's historic releases References: <3bbf2fe10909290811p6ac49f7fxaf7e7d4bf631da4c@mail.gmail.com> Message-ID: Gy?rgy Vilmos writes: > BTW, even on one thread, Linux (2.6.31) performs much better, better means > here 790 TPS vs. 580... What test did you use to meassure TPS? That is a pretty big difference. From vilmos.gyorgy at gmail.com Sat Oct 3 18:23:48 2009 From: vilmos.gyorgy at gmail.com (=?ISO-8859-1?Q?Gy=F6rgy_Vilmos?=) Date: Sat Oct 3 19:56:11 2009 Subject: Performance evaluation of PostgreSQL's historic releases In-Reply-To: References: <3bbf2fe10909290811p6ac49f7fxaf7e7d4bf631da4c@mail.gmail.com> Message-ID: 2009/10/2 Francisco Reyes > Gy?rgy Vilmos writes: > > BTW, even on one thread, Linux (2.6.31) performs much better, better means >> here 790 TPS vs. 580... >> > > What test did you use to meassure TPS? > That is a pretty big difference. > > sysbench -- http://suckit.blog.hu/ From serenity at exscape.org Mon Oct 5 09:02:53 2009 From: serenity at exscape.org (Thomas Backman) Date: Mon Oct 5 09:02:59 2009 Subject: A specific example of a disk i/o problem Message-ID: (Note: I hope this reply shows up correctly. I subscribed to the mailing list after the fact and had to "forge" the subject line.) Hey everyone, I'm having serious trouble with the same thing, and just found this thread while looking for the correct place to post. Looks like I found it. (I wrote most of the post before finding the thread, so some of it will seem a bit odd.) I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old 80GB 7200rpm disk. My problem is that I get completely unacceptable latency on console IO (both via SSH and serial console) when the system is performing disk IO. The worst case I've noticed yet was when I tried copying a core dump from a lzjb compressed ZFS file system to a gzip-9 compressed one, to compare the file size/compression ratio. screen(1) took at LEAST ten seconds - probably a bit more - I'm not exaggerating here - to switch to another window, and an "ls" in an empty directory also about 5-10 seconds. Doing some silly CPU load with two instances of "yes >/dev/null" (on a single core, remember) doesn't change anything, the system remains very responsive. "cat /dev/zero > /uncompressed_fs/..." however produces the extreme slowdown. (On a gzip-1 FS it doesn't, since the file ends up extremely small - a kilobyte or so - even after a while, thus performing minimal IO). I'm thinking about switching to FreeBSD on my beefier "production" system (dual-core amd64, 4GB RAM, 4x1TB disks, compared to this one, single-core, 2GB RAM, 80GB disk), but unless I feel assured this won't happen there as well, I'm not so sure anymore. I can do any kind of heavy IO/compilation/whatever on that box, currently running Linux, and it's always unnoticable. In this case it's impossible *not* to notice that your key input is lagging behind 5-10 seconds... I thought multiple times that the box must have panicked. I do realize that the hardware isn't the best, especially the disks, but this is far worse than it should be! Here's some of the testing done in this thread (or at least something like it): [root@chaos ~]# time file /etc/* >/dev/null real 0m1.725s user 0m0.993s sys 0m0.021s [root@chaos ~]# time file /etc/* >/dev/null real 0m1.008s user 0m0.990s sys 0m0.015s [root@chaos ~]# time file /etc/* >/dev/null real 0m1.008s user 0m0.967s sys 0m0.038s [root@chaos ~]# time file /etc/* >/dev/null real 0m1.015s user 0m0.998s sys 0m0.008s So, pretty much exactly 1 second every time once the cache is warmed up. Now, let's try it 10 seconds after starting heavy disk writing... [root@chaos ~]# cat /dev/zero > /DELETE_ME & (wait for 10 seconds) [root@chaos ~]# time file /etc/* >/dev/null real 0m13.217s user 0m1.004s sys 0m0.023s [root@chaos ~]# time file /etc/* >/dev/null real 0m24.012s user 0m1.020s sys 0m0.008s ... the numbers speak for themselves. FWIW I tried the same on the Linux box: "file" took 0.8 seconds the first time, 0.125s subsequent times. While running cat, 0.13-0.21 seconds (0.21 being the first, subsequent runs took 0.13s). The system remained completely responsive, which cannot be said for the FreeBSD one! Any advice? Info below - please ask if you need more! Regards, Thomas ----------------------------------------------------------------------------- Basic info: [root@chaos ~]# uname -a FreeBSD chaos.exscape.org 8.0-RC1 FreeBSD 8.0-RC1 #1 r197613M: Tue Sep 29 15:04:44 CEST 2009 root@chaos.exscape.org:/usr/obj/usr/src/sys/ DTRACE amd64 (KDB/DDB enabled, WITNESS/INVARIANTS *disabled*) [root@chaos ~]# mount tank/root on / (zfs, local, noatime) devfs on /dev (devfs, local, multilabel) /dev/ad0s1a on /bootdir (ufs, local, soft-updates) tank/export on /export (zfs, NFS exported, local, noatime) tank/tmp on /tmp (zfs, local, noatime) tank/usr on /usr (zfs, local, noatime) tank/usr/obj on /usr/obj (zfs, local, noatime) tank/usr/ports on /usr/ports (zfs, local, noatime) tank/usr/ports/distfiles on /usr/ports/distfiles (zfs, local, noatime) tank/usr/src on /usr/src (zfs, local, noatime) tank/usr/src_r196905 on /usr/src_r196905 (zfs, local, noatime, read- only) tank/var on /var (zfs, local, noatime) tank/var/crash on /var/crash (zfs, local, noatime) tank/var/log on /var/log (zfs, local, noatime) tank/var/tmp on /var/tmp (zfs, local, noatime) dmesg: ----------------------------------------------------------------------------- Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC1 #1 r197613M: Tue Sep 29 15:04:44 CEST 2009 root@chaos.exscape.org:/usr/obj/usr/src/sys/DTRACE Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3200+ (2009.27-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x10ff0 Stepping = 0 Features = 0x78bfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> AMD Features=0xe2500800 AMD Features2=0x1 real memory = 2147483648 (2048 MB) avail memory = 2052362240 (1957 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 7fef0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.0 (no driver attached) isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xfe02f000-0xfe02ffff irq 21 at device 2.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ehci0: mem 0xfe02e000-0xfe02e0ff irq 22 at device 2.1 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfb00-0xfb0f at device 6.0 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xf600-0xf60f mem 0xfe02b000-0xfe02bfff irq 23 at device 7.0 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] atapci2: port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xf100-0xf10f mem 0xfe02a000-0xfe02afff irq 21 at device 8.0 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] pcib1: at device 9.0 on pci0 pci1: on pcib1 vgapci0: mem 0xfcff8000-0xfcffbfff, 0xfd000000-0xfd7fffff,0xfc000000-0xfc7fffff irq 17 at device 7.0 on pci1 xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xdf00-0xdf7f mem 0xfcfff000-0xfcfff07f irq 18 at device 9.0 on pci1 miibus0: on xl0 xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus0 xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl0: Ethernet address: 00:50:da:44:c0:4a xl0: [ITHREAD] nfe0: port 0xf000-0xf007 mem 0xfe029000-0xfe029fff irq 22 at device 10.0 on pci0 miibus1: on nfe0 e1000phy0: PHY 1 on miibus1 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto nfe0: Ethernet address: 00:13:d3:a2:aa:0f nfe0: [FILTER] pcib2: at device 11.0 on pci0 pci2: on pcib2 pcib3: at device 12.0 on pci0 pci3: on pcib3 pcib4: at device 13.0 on pci0 pci4: on pcib4 pcib5: at device 14.0 on pci0 pci5: on pcib5 amdtemp0: on hostb3 acpi_tz0: on acpi0 atrtc0: port 0x70-0x73 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] cpu0: on acpi0 powernow0: on cpu0 device_attach: powernow0 attach returned 6 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff, 0xcc000-0xcc7ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ZFS NOTICE: system has less than 4GB and prefetch enable is not set... disabling. ZFS filesystem version 13 ZFS storage pool version 13 Timecounter "TSC" frequency 2009269513 Hz quality 800 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 ad0: 76318MB at ata0-master UDMA100 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ad2: 9768MB at ata1-master UDMA100 GEOM: ad2s1: geometry does not match label (255h,63s != 16h,63s). Root mount waiting for: usbus1 usbus0 uhub0: 10 ports with 10 removable, self powered Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 uhub1: 10 ports with 10 removable, self powered Trying to mount root from zfs:tank/root netsmb_dev: loaded ----------------------------------------------------------------------------- The 80GB disk is used for everything except swap (aka. dump device) for which the 10GB disk is used, exclusively. loader.conf has nothing of value (just loading a few modules and a vfs.root.mountfrom, plus serial console settings). From vilmos.gyorgy at gmail.com Mon Oct 5 10:45:53 2009 From: vilmos.gyorgy at gmail.com (=?ISO-8859-1?Q?Gy=F6rgy_Vilmos?=) Date: Mon Oct 5 11:41:49 2009 Subject: MySQL and PostgreSQL benchmarks, FreeBSD 7 vs. 8 Message-ID: Hello, I have two new articles: First, MySQL history, which takes some older (major) versions from MySQL and shows their performance on FreeBSD 8: http://suckit.blog.hu/2009/10/03/mysql_history and an article, which compares PostgreSQL and MySQL performance between FreeBSD 7 and 8: http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade -- http://suckit.blog.hu/ From pieter at degoeje.nl Mon Oct 5 14:48:55 2009 From: pieter at degoeje.nl (Pieter de Goeje) Date: Mon Oct 5 14:49:01 2009 Subject: MySQL and PostgreSQL benchmarks, FreeBSD 7 vs. 8 In-Reply-To: References: Message-ID: <200910051615.00313.pieter@degoeje.nl> On Monday 05 October 2009 12:45:50 Gy?rgy Vilmos wrote: > Hello, > > I have two new articles: > First, MySQL history, which takes some older (major) versions from MySQL > and shows their performance on FreeBSD 8: > http://suckit.blog.hu/2009/10/03/mysql_history > > and an article, which compares PostgreSQL and MySQL performance between > FreeBSD 7 and 8: > http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade Very interesting. Good to see 8 is performing better than 7 :-) Did you see any performance change after implementing the correct cpu topology? -- Pieter de Goeje From freebsd at sopwith.solgatos.com Tue Oct 6 04:15:35 2009 From: freebsd at sopwith.solgatos.com (Dieter) Date: Tue Oct 6 04:15:47 2009 Subject: ZFS Re: A specific example of a disk i/o problem In-Reply-To: Your message of "Mon, 05 Oct 2009 10:45:48 +0200." Message-ID: <200910051605.QAA06722@sopwith.solgatos.com> In message , Thomas Backman writes: > I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old > 80GB 7200rpm disk. > > My problem is that I get completely unacceptable latency on console IO > (both via SSH and serial console) when the system is performing disk > IO. The worst case I've noticed yet was when I tried copying a core > dump from a lzjb compressed ZFS file system to a gzip-9 compressed > one, to compare the file size/compression ratio. screen(1) took at > LEAST ten seconds - probably a bit more - I'm not exaggerating here - > to switch to another window, and an "ls" in an empty directory also > about 5-10 seconds. You might find the "RELENG_7 heavy disk = system crawls" thread interesting: } I didn't actually solve it or do anything. } I just upgraded to RELENG_8. } } Now it's behaving more like FreeBSD should. } I can do sequential reads/writes and still } use kbd/mouse/X11/buildworld and so on. From freebsd at sopwith.solgatos.com Tue Oct 6 04:15:35 2009 From: freebsd at sopwith.solgatos.com (Dieter) Date: Tue Oct 6 04:15:47 2009 Subject: tuning FFS for large files Re: A specific example of a disk i/o problem In-Reply-To: Your message of "Thu, 01 Oct 2009 09:10:43 EDT." <20091001091043.477f4b9b.wmoran@collaborativefusion.com> Message-ID: <200910051755.RAA11047@sopwith.solgatos.com> I found a clue! The problem occurs with my big data partitions, which are newfs-ed with options intended to improve things. Reading a large file from the normal ad4s5b partition only delays other commands slightly, as expected. Reading a large file from the tuned ad4s11 partition yields the delay of minutes for other i/o. # tunefs -p /dev/ad4s5b tunefs: ACLs: (-a) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: gjournal: (-J) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 1024 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: optimization preference: (-o) time tunefs: volume label: (-L) # tunefs -p /dev/ad4s11 tunefs: ACLs: (-a) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: gjournal: (-J) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 57984 tunefs: average file size: (-f) 67108864 tunefs: average number of files in a directory: (-s) 16 tunefs: minimum percentage of free space: (-m) 8% tunefs: optimization preference: (-o) time tunefs: volume label: (-L) Here is the newfs command used for creating large data partitions: newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition Even this isn't tuned the way I wanted to. -g * -h must be less than 4 G due to 32 bit problem (system panics). Note the 32 bit problem is in the filesystem code, I'm running amd64. IIRC there is a PR about this. (I'm assuming the bug hasn't been fixed yet) Result is that I must specify -g and -h smaller than they should be. And they have way more inodes than needed. (IIRC it doesn't actually use -i 67108864) Partition sizes range from ~200 GB to 1.5 TB. A small number of directories, perhaps a dozen. About 1/2 the files are small, a few KB, the other half are large, 500MB-25GB. Goals are to minimize seeking and use space efficiently. Files are written sequentially, read mostly (99.99%) sequentially, no editing. I still think it may have something to do with a common resource such as disk cache, as it can affect other disks, including swap. Hmmm, does swap use the disk cache? But now it appears that FFS tuning is the cause of the bottleneck. Do these tuning parameters increase the number of some data structure needed? Can I crank that up? Do I have to change the FFS tuning? Suggestions? From serenity at exscape.org Tue Oct 6 06:56:56 2009 From: serenity at exscape.org (Thomas Backman) Date: Tue Oct 6 06:57:03 2009 Subject: ZFS Re: A specific example of a disk i/o problem In-Reply-To: <200910051605.QAA06722@sopwith.solgatos.com> References: <200910051605.QAA06722@sopwith.solgatos.com> Message-ID: <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> On Oct 5, 2009, at 6:05 PM, Dieter wrote: > In message , > Thomas Backman writes: > >> I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old >> 80GB 7200rpm disk. >> >> My problem is that I get completely unacceptable latency on console >> IO >> (both via SSH and serial console) when the system is performing disk >> IO. The worst case I've noticed yet was when I tried copying a core >> dump from a lzjb compressed ZFS file system to a gzip-9 compressed >> one, to compare the file size/compression ratio. screen(1) took at >> LEAST ten seconds - probably a bit more - I'm not exaggerating here - >> to switch to another window, and an "ls" in an empty directory also >> about 5-10 seconds. > > You might find the "RELENG_7 heavy disk = system crawls" thread > interesting: > > } I didn't actually solve it or do anything. > } I just upgraded to RELENG_8. > } > } Now it's behaving more like FreeBSD should. > } I can do sequential reads/writes and still > } use kbd/mouse/X11/buildworld and so on. Doesn't really help much I'm afraid, since 8.0-RC1 = RELENG_8 = stable/ 8. Been running current/stable-8 since May. Regards, Thomas From ohartman at zedat.fu-berlin.de Tue Oct 6 09:29:18 2009 From: ohartman at zedat.fu-berlin.de (O. Hartmann) Date: Tue Oct 6 09:29:24 2009 Subject: ZFS Re: A specific example of a disk i/o problem In-Reply-To: <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> References: <200910051605.QAA06722@sopwith.solgatos.com> <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> Message-ID: <4ACB068E.2070004@zedat.fu-berlin.de> Thomas Backman wrote: > On Oct 5, 2009, at 6:05 PM, Dieter wrote: > >> In message , Thomas >> Backman writes: >> >>> I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old >>> 80GB 7200rpm disk. >>> >>> My problem is that I get completely unacceptable latency on console IO >>> (both via SSH and serial console) when the system is performing disk >>> IO. The worst case I've noticed yet was when I tried copying a core >>> dump from a lzjb compressed ZFS file system to a gzip-9 compressed >>> one, to compare the file size/compression ratio. screen(1) took at >>> LEAST ten seconds - probably a bit more - I'm not exaggerating here - >>> to switch to another window, and an "ls" in an empty directory also >>> about 5-10 seconds. >> >> You might find the "RELENG_7 heavy disk = system crawls" thread >> interesting: >> >> } I didn't actually solve it or do anything. >> } I just upgraded to RELENG_8. >> } >> } Now it's behaving more like FreeBSD should. >> } I can do sequential reads/writes and still >> } use kbd/mouse/X11/buildworld and so on. > > Doesn't really help much I'm afraid, since 8.0-RC1 = RELENG_8 = > stable/8. Been running current/stable-8 since May. > > Regards, > Thomas > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" Is this really ZFS-bound? I also saw (and still see!) massive performance impacts when a lot of disk activities, especially when compiling large packages (done on UFS2 disks). Copying data from one ZFS drive to (on different harddrive) another ZFS drive (which is compressed) doesn't impact performance that much. When the box hit this performance issue, console, X11 and other activities tend to be stuck for minutes! This happens on an UP box (Athlon64 3500+, 2 GB ECC RAM, 2x 250 GB WD SATA-II disks and 1 TB WD SATA-II disk (1x 250 GB UFS2 for system, 1x 250 GB ZFS compressed for backup, 1x 1 TB ZFS holding /home). But this impact is also noticable on my lab's machine: Quad core Intel Q6600 CPU at 3,0 GHZ, 8 GB RAM, 1x 320 GB SATA-II UFS2 disk for system, 2x SATA-II 500 ZFS and 1x 750 GB disks ZFS and even on a 8 core, two socket Dell PowerEdge 1950-III box with two XEON CPUs running at 2,5 GHz, 16 GB RAM and two SATA-II harddrives attached to the built in SAS controller. Maybe this is of interest: http://www.phoronix.com/scan.php?page=article&item=freebsd8_ubuntu910&num=1 Watch the threaded I/O impact ... Regards, Oliver From brde at optusnet.com.au Tue Oct 6 09:47:05 2009 From: brde at optusnet.com.au (Bruce Evans) Date: Tue Oct 6 09:47:13 2009 Subject: tuning FFS for large files Re: A specific example of a disk i/o problem In-Reply-To: <200910051755.RAA11047@sopwith.solgatos.com> References: <200910051755.RAA11047@sopwith.solgatos.com> Message-ID: <20091006174121.V25604@delplex.bde.org> On Mon, 5 Oct 2009, Dieter wrote: > I found a clue! The problem occurs with my big data partitions, > which are newfs-ed with options intended to improve things. > > Reading a large file from the normal ad4s5b partition only delays other > commands slightly, as expected. Reading a large file from the tuned > ad4s11 partition yields the delay of minutes for other i/o. > ... > Here is the newfs command used for creating large data partitions: > newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition Any block size above the default (16K) tends to thrash and fragment buffer cache virtual memory. This is obviously a good pessimization with lots of small files, and apparently, as you have found, it is a good pessimization with a few large files too. I think severe fragmentation can easily take several seconds to recover from. The worst case for causing fragmentaion is probably a mixture of small and large files. Some users fear fs consistency bugs with block sizes >= 16K, but I've never seen them cause any bugs ecept performance ones. > Even this isn't tuned the way I wanted to. > -g * -h must be less than 4 G due to 32 bit problem (system panics). The panic is now avoided in some versions of FreeBSD (-8 and -current at least). > Note the 32 bit problem is in the filesystem code, I'm running amd64. > IIRC there is a PR about this. (I'm assuming the bug hasn't been fixed yet) > Result is that I must specify -g and -h smaller than they should be. I bet you can't see any difference (except the panic) from enlarging -g and -h. > And they have way more inodes than needed. (IIRC it doesn't actually > use -i 67108864) It has to have at least 1 inode per cg, and may as well have a full block of them, which gives a fairly large number of inodes especially if the block size is too large (64K), so the -i ratio is limited. Bruce From serenity at exscape.org Tue Oct 6 20:08:06 2009 From: serenity at exscape.org (Thomas Backman) Date: Tue Oct 6 20:08:14 2009 Subject: ZFS Re: A specific example of a disk i/o problem In-Reply-To: <4ACB068E.2070004@zedat.fu-berlin.de> References: <200910051605.QAA06722@sopwith.solgatos.com> <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> <4ACB068E.2070004@zedat.fu-berlin.de> Message-ID: <74A4B916-9090-40B1-958F-8FF772BD94AB@exscape.org> On Oct 6, 2009, at 10:57 AM, O. Hartmann wrote: > Thomas Backman wrote: >> On Oct 5, 2009, at 6:05 PM, Dieter wrote: >>> In message , >>> Thomas Backman writes: >>> >>>> I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an >>>> old >>>> 80GB 7200rpm disk. >>>> >>>> My problem is that I get completely unacceptable latency on >>>> console IO >>>> (both via SSH and serial console) when the system is performing >>>> disk >>>> IO. The worst case I've noticed yet was when I tried copying a core >>>> dump from a lzjb compressed ZFS file system to a gzip-9 compressed >>>> one, to compare the file size/compression ratio. screen(1) took at >>>> LEAST ten seconds - probably a bit more - I'm not exaggerating >>>> here - >>>> to switch to another window, and an "ls" in an empty directory also >>>> about 5-10 seconds. >>> >>> You might find the "RELENG_7 heavy disk = system crawls" thread >>> interesting: >>> >>> } I didn't actually solve it or do anything. >>> } I just upgraded to RELENG_8. >>> } >>> } Now it's behaving more like FreeBSD should. >>> } I can do sequential reads/writes and still >>> } use kbd/mouse/X11/buildworld and so on. >> Doesn't really help much I'm afraid, since 8.0-RC1 = RELENG_8 = >> stable/8. Been running current/stable-8 since May. >> Regards, >> Thomas >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org >> " > > Is this really ZFS-bound? I also saw (and still see!) massive > performance impacts when a lot of disk activities, especially when > compiling large packages (done on UFS2 disks). Copying data from one > ZFS drive to (on different harddrive) another ZFS drive (which is > compressed) doesn't impact performance that much. > > When the box hit this performance issue, console, X11 and other > activities tend to be stuck for minutes! This happens on an UP box > (Athlon64 3500+, 2 GB ECC RAM, 2x 250 GB WD SATA-II disks and 1 TB > WD SATA-II disk (1x 250 GB UFS2 for system, 1x 250 GB ZFS compressed > for backup, 1x 1 TB ZFS holding /home). > > But this impact is also noticable on my lab's machine: Quad core > Intel Q6600 CPU at 3,0 GHZ, 8 GB RAM, 1x 320 GB SATA-II UFS2 disk > for system, 2x SATA-II 500 ZFS and 1x 750 GB disks ZFS and even on a > 8 core, two socket Dell PowerEdge 1950-III box with two XEON CPUs > running at 2,5 GHz, 16 GB RAM and two SATA-II harddrives attached to > the built in SAS controller. > > Maybe this is of interest: > http://www.phoronix.com/scan.php?page=article&item=freebsd8_ubuntu910&num=1 > > Watch the threaded I/O impact ... > > Regards, > Oliver It doesn't appear to be an ZFS issue, no. (Note that I wasn't the one who added it to the subject line :) I just tried it to my one UFS filesystem, and while screen(1) DID remain 100% responsive, everything didn't. vim worked OK most of the time, but other FS ops on the same disk... oh no. I aborted the "file / etc/*" after 57 seconds. Tried it again after stopping the disk IO (cat /dev/zero > /bootdir/filename) - 1.52 seconds. This raises the question: is this some kind of hardware specific issue? If so, what hardware? I can't think of anything my computer would have in common with the ones you've listed! I mean, come on, FreeBSD's IO performance can't be this abysmal for everybody or nobody would use it for serious applications. Something must be wrong for a few of us, but where and why? Regards, Thomas From net.help at m2k.com.tw Sat Oct 10 16:23:24 2009 From: net.help at m2k.com.tw (Ckcheng) Date: Sat Oct 10 16:43:51 2009 Subject: Migrate large amount of small files Message-ID: <1255189612.42461.net.help@m2k.com.tw> Hi all, Currently, I have a directory with over 5M small files (1~32K). Now, I want to transfer this directory to another machine and found that it's extremely slow and painful process. I tried the following method: 1. rsync 2. tar via ssh 3. tar via nc (all take hours and hours to finish) None of them is able to give me a reasonable migration time. So, I'm here for asking help. Any suggestion is extremely welcomed. Thank you. Btw, here is brief information of my server. (both machines are the same) OS: FreeBSD 6.4-Stable 64Bit CPU: 2 x Xeon L5420 2.50GHz RAM: 2 x 2G ECC DDR2-667 (full buffered) DISK: Seagate Barracuda ES 16MB (SATA 300) Network: 1Gbps (Broadcom BCM5708) Filesystem: UFS2 Regards, From lists at stringsutils.com Sun Oct 11 20:50:20 2009 From: lists at stringsutils.com (Francisco Reyes) Date: Sun Oct 11 20:50:27 2009 Subject: Migrate large amount of small files References: <1255189612.42461.net.help@m2k.com.tw> Message-ID: Ckcheng writes: > 1. rsync > 2. tar .. If this is a migration I find that tar to the local machine, copy over, restore, then rsync are likely the best options. In my experience copying lots of small files is going to take a long time, no matter which method you use. >From all the combinations I have tried in the past tar first then rsync worked best for me. From mike at sentex.net Sun Oct 11 21:06:54 2009 From: mike at sentex.net (Mike Tancsa) Date: Sun Oct 11 21:07:00 2009 Subject: Migrate large amount of small files In-Reply-To: <1255189612.42461.net.help@m2k.com.tw> References: <1255189612.42461.net.help@m2k.com.tw> Message-ID: <200910112106.n9BL6p7G074295@lava.sentex.ca> At 11:46 AM 10/10/2009, Ckcheng wrote: >Hi all, >Currently, I have a directory with over 5M small files (1~32K). Now, >I want to transfer this directory to another machine and found >that it's extremely slow and painful process. I tried the following >method: It might help if you mount all -onoatime as well as bump up vfs.ufs.dirhash_maxmem to 4x the default size ---Mike >1. rsync >2. tar via ssh >3. tar via nc >(all take hours and hours to finish) > >None of them is able to give me a reasonable migration time. So, I'm >here for asking help. Any suggestion is extremely welcomed. Thank you. > >Btw, here is brief information of my server. (both machines are the same) > >OS: FreeBSD 6.4-Stable 64Bit >CPU: 2 x Xeon L5420 2.50GHz >RAM: 2 x 2G ECC DDR2-667 (full buffered) >DISK: Seagate Barracuda ES 16MB (SATA 300) >Network: 1Gbps (Broadcom BCM5708) >Filesystem: UFS2 > >Regards, >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike From jguojun at sbcglobal.net Sun Oct 11 23:18:58 2009 From: jguojun at sbcglobal.net (Jin Guojun) Date: Sun Oct 11 23:19:05 2009 Subject: Migrate large amount of small files In-Reply-To: <200910112106.n9BL6p7G074295@lava.sentex.ca> Message-ID: <598544.44269.qm@web82207.mail.mud.yahoo.com> The major factor affects the performance the the number of files in "PER" directory. If I got impression correct, there are 5M files in JUST one directory. If this is true, then it is the problem. The best way to avoid performance penalty is to sort different files into differet named directories. Each directory should contain about 1K files or less. Carefully tuned FS may have better performance for a directory conataining a few thousand files, but no more. If files are created by some state machine automatically, and you do not have a good sorting algorithm for directory naming, use Year-Month-week for auto directory naming. --- On Sun, 10/11/09, Mike Tancsa wrote: > From: Mike Tancsa > Subject: Re: Migrate large amount of small files > To: net.help@m2k.com.tw, "freebsd-performance" > Date: Sunday, October 11, 2009, 9:07 PM > At 11:46 AM 10/10/2009, Ckcheng > wrote: > >Hi all, > >Currently, I have a directory with over 5M small files > (1~32K). Now, > >I want to transfer this directory to another machine > and found > >that it's extremely slow and painful process. I tried > the following > >method: > > It might help if you mount all -onoatime as well as bump up > > vfs.ufs.dirhash_maxmem to 4x the default size > > ---Mike > > >1. rsync > >2. tar via ssh > >3. tar via nc > >(all take hours and hours to finish) > > > >None of them is able to give me a reasonable migration > time. So, I'm > >here for asking help. Any suggestion is extremely > welcomed. Thank you. > > > >Btw, here is brief information of my server. (both > machines are the same) > > > >OS: FreeBSD 6.4-Stable 64Bit > >CPU: 2 x Xeon L5420 2.50GHz > >RAM: 2 x 2G ECC DDR2-667 (full buffered) > >DISK: Seagate Barracuda ES 16MB (SATA 300) > >Network: 1Gbps (Broadcom BCM5708) > >Filesystem: UFS2 > > > >Regards, > >_______________________________________________ > >freebsd-performance@freebsd.org > mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-performance > >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > -------------------------------------------------------------------- > Mike Tancsa, > > tel +1 519 651 3400 > Sentex Communications, > > mike@sentex.net > Providing Internet since 1994 > www.sentex.net > Cambridge, Ontario Canada > > www.sentex.net/mike > > _______________________________________________ > freebsd-performance@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From ohartman at zedat.fu-berlin.de Mon Oct 12 07:44:38 2009 From: ohartman at zedat.fu-berlin.de (O. Hartmann) Date: Mon Oct 12 07:44:50 2009 Subject: Scheduler weirdness In-Reply-To: <20091012044911.GA39479@troutmask.apl.washington.edu> References: <6729ad0409e449f8dbda69ecd8feb618.squirrel@webmail.lerctr.org> <20091012014846.GB38325@troutmask.apl.washington.edu> <20091012023912.GA38822@troutmask.apl.washington.edu> <4AD29937.2040004@mailinglist.ahhyes.net> <20091012043358.GA39364@troutmask.apl.washington.edu> <4AD2B203.8030405@mailinglist.ahhyes.net> <20091012044911.GA39479@troutmask.apl.washington.edu> Message-ID: <4AD2DE71.5060601@zedat.fu-berlin.de> Steve Kargl wrote: > On Mon, Oct 12, 2009 at 03:35:15PM +1100, Alex R wrote: >> Steve Kargl wrote: >>> On Mon, Oct 12, 2009 at 01:49:27PM +1100, Alex R wrote: >>> >>>> Steve Kargl wrote: >>>> >>>>> So, you have 4 cpus and 4 folding-at-home processes and you're >>>>> trying to use the system with other apps? Switch to 4BSD. >>>>> >>>>> >>>>> >>>> I thought SCHED_ULE was meant to be a much better choice under an SMP >>>> environment. Why are you suggesting he rebuild his kernel and use the >>>> legacy scheduler? >>>> >>>> >>> If you have N cpus and N+1 numerical intensitive applications, >>> ULE may have poor performance compared to 4BSD. In OP's case, >>> he has 4 cpus and 4 numerical intensity (?) applications. He, >>> however, also is trying to use the system in some interactive >>> way. >>> >>> >> Ah ok. Is this just an accepted thing by the freebsd dev's or are they >> trying to fix it? >> > > Jeff appears to be extremely busy with other projects. He is aware of > the problem, and I have set up my system to give him access when/if it > is so desired. > > Here's the text of my last set of tests that I sent to him > > OK, I've manage to recreate the problem. User kargl launches a mpi > job on node10 that creates two images on node20. This is command z > in the top(1) info. 30 seconds later, user sgk lauches a mpi process > on node10 that creates 8 images on node20. This is command rivmp in > top(1) info. With 8 available cpus, this is a (slightly) oversubscribed > node. > > For 4BSD, I see > > last pid: 1432; load averages: 8.68, 5.65, 2.82 up 0+01:52:14 17:07:22 > 40 processes: 11 running, 29 sleeping > CPU: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle > Mem: 32M Active, 12M Inact, 203M Wired, 424K Cache, 29M Buf, 31G Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 1428 sgk 1 124 0 81788K 5848K CPU3 6 1:13 78.81% rivmp > 1431 sgk 1 124 0 81788K 5652K RUN 1 1:13 78.52% rivmp > 1415 kargl 1 124 0 78780K 4668K CPU7 1 1:38 78.42% z > 1414 kargl 1 124 0 78780K 4664K CPU0 0 1:37 77.25% z > 1427 sgk 1 124 0 81788K 5852K CPU4 3 1:13 78.42% rivmp > 1432 sgk 1 124 0 81788K 5652K CPU2 4 1:13 78.27% rivmp > 1425 sgk 1 124 0 81788K 6004K CPU5 5 1:12 78.17% rivmp > 1426 sgk 1 124 0 81788K 5832K RUN 6 1:13 78.03% rivmp > 1429 sgk 1 124 0 81788K 5788K CPU6 7 1:12 77.98% rivmp > 1430 sgk 1 124 0 81788K 5764K RUN 2 1:13 77.93% rivmp > > > Notice, the accumulated times appear reasonable. At this point in the > computations, rivmp is doing no communication between processes. z is > the netpipe benchmark and is essentially sending messages between the > two processes over the memory bus. > > > For ULE, I see > > last pid: 1169; load averages: 7.56, 2.61, 1.02 up 0+00:03:15 17:13:01 > 40 processes: 11 running, 29 sleeping > CPU: 100% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle > Mem: 31M Active, 9392K Inact, 197M Wired, 248K Cache, 26M Buf, 31G Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 1168 sgk 1 118 0 81788K 5472K CPU6 6 1:18 100.00% rivmp > 1169 sgk 1 118 0 81788K 5416K CPU7 7 1:18 100.00% rivmp > 1167 sgk 1 118 0 81788K 5496K CPU5 5 1:18 100.00% rivmp > 1166 sgk 1 118 0 81788K 5564K RUN 4 1:18 100.00% rivmp > 1151 kargl 1 118 0 78780K 4464K CPU3 3 1:48 99.27% z > 1152 kargl 1 110 0 78780K 4464K CPU0 0 1:18 62.89% z > 1164 sgk 1 113 0 81788K 5592K CPU1 1 0:55 80.76% rivmp > 1165 sgk 1 110 0 81788K 5544K RUN 0 0:52 62.16% rivmp > 1163 sgk 1 107 0 81788K 5624K RUN 2 0:40 50.68% rivmp > 1162 sgk 1 107 0 81788K 5824K CPU2 2 0:39 50.49% rivmp > > > In the above, processes 1162-1165 are clearly not receiving sufficient time > slices to keep up with the other 4 rivmp images. From watching top at a > 1 second interval, once the 4 rivmp hit 100% CPU, they stayed pinned to > their cpu and stay at 100% CPU. It is also seen that processes 1152, 1165 > and 1162, 1163 are stuck on cpus 0 and 2, respectively. > This isn't only bound to floating-point intense applications, even the operating system itselfs seems to suffer from SCHED_ULE. I saw, see and reported several performance issue under heavy load and for seconds (if not minutes!) 4+ CPU boxes get as stuck as a UP box does. Those sticky sitiuations are painful in cases where the box needs to be accessed via X11. The remaining four FreeBSD 8.0-boxes used for numerical applications in our lab (others switched to Linux a long time ago) all uses SCHED_ULE, as this scheduler was introduced to be the superior scheduler over the legacy 4BSD. Well, I'll give 4BSD a chance again. At the moment, even our 8-core DELL Poweredge box is in production use, but if there is something I can do, menas: benchmarking, I'll give it a try. Regards, Oliver From htyin at huawei.com Thu Oct 15 01:57:46 2009 From: htyin at huawei.com (Hongtao Yin) Date: Thu Oct 15 01:57:53 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance Message-ID: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Hi, I compared TCP performance between FreeBSD and Linux by running test tools Netperf and Iperf with Intel NIC. The kernels are full version and default values are used in the testing except TCP Congestion Control algorithm set to Reno. >From the test results we can see Linux TCP performance in throughput is better than FreeBSD. The worst case (send msg size 128) shows that FreeBSD throughput is only 43% of Linux's. I like to get some feedback if anyone did similar comparison test, or knows any issues with kernels or drivers. Thanks lot. FreeBSD and Linux Sysctl captures are attached for reference. Regards, Hongtao Test Environments: PC: Dell Precision T3400 (same 4 PCs) CPU: Intel Core 2 Duo CPU E4600@2.4Ghz FreeBSD: V7.1 (full version) (TCP CC: newReno) Linux: V2.6.31.1 (full version) (TCP CC: Reno) Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter (Gigabit) chip 82546EB (only one port used for each PC) Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) Iperf: V2.0.4 Netperf: V2.4.4 Setup: ---------- | switch | ---------- ---------------------| | | |-------------------- | | | | | --------| |-------- | | | | | | | | | -------------- -------------- -------------- -------------- | PC1 | | PC2 | | PC3 | | PC4 | | FreeBSD | | FreeBSD | | Linux | | Linux | |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| -------------- -------------- -------------- -------------- ================================ Netperf Test Results ================================ TCP Throughput Test ------------------- PC2/4: #netserver -p 22113 PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 32768 10.34 598.11 Linux: 87380 16384 16384 10.04 779.02 PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m 64/128/256/512/1024/2048/4096 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 64 10.19 417.10 65536 32768 128 10.35 336.63 65536 32768 256 10.36 576.99 65536 32768 512 10.35 569.79 65536 32768 1024 10.35 553.70 65536 32768 2048 10.35 584.20 65536 32768 4096 10.35 602.45 Linux: 87380 16384 64 10.03 778.21 87380 16384 128 10.03 779.72 87380 16384 256 10.04 780.16 87380 16384 512 10.03 776.85 87380 16384 1024 10.04 777.52 87380 16384 2048 10.04 777.83 87380 16384 4096 10.03 780.17 =============================== Iperf Test Results =============================== Bandwidth Test -------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 740 600 Linux: 0.0-10.0 972 815 PC1/3: #iperf -c 192.168.1.20 -d Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.0 402 337 0.0-10.0 404 338 Linux: 0.0-10.0 926 776 0.0-10.0 44.1 36.9 Parallel Test ------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 -P 2 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 370 300 0.0-10.3 370 300 SUM: 0.0-10.3 739 600 Linux: 0.0-10.0 479 402 0.0-10.0 473 396 SUM: 0.0-10.0 952 797 -------------- next part -------------- A non-text attachment was scrubbed... Name: FreeBSD_sysctl Type: application/octet-stream Size: 44503 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20091015/6be13f7f/FreeBSD_sysctl.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: Linux_sysctl Type: application/octet-stream Size: 20177 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20091015/6be13f7f/Linux_sysctl.obj From ohartman at zedat.fu-berlin.de Thu Oct 15 09:34:20 2009 From: ohartman at zedat.fu-berlin.de (O. Hartmann) Date: Thu Oct 15 09:34:26 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: <4AD6ECAB.4050906@zedat.fu-berlin.de> Hongtao Yin wrote: > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test tools > Netperf and Iperf with Intel NIC. > > The kernels are full version and default values are used in the testing > except TCP Congestion Control algorithm set to Reno. > >>From the test results we can see Linux TCP performance in throughput is > better than FreeBSD. The worst case (send msg size 128) shows that FreeBSD > throughput is only 43% of Linux's. > > > > I like to get some feedback if anyone did similar comparison test, or knows > any issues with kernels or drivers. Thanks lot. > > > > FreeBSD and Linux Sysctl captures are attached for reference. > > > > Regards, > > Hongtao > > > > > > > > Test Environments: > > PC: Dell Precision T3400 (same 4 PCs) > > CPU: Intel Core 2 Duo CPU E4600@2.4Ghz > > FreeBSD: V7.1 (full version) (TCP CC: newReno) > > Linux: V2.6.31.1 (full version) (TCP CC: Reno) > > Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter (Gigabit) > chip 82546EB (only one port used for each PC) > > Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) > > Iperf: V2.0.4 > > Netperf: V2.4.4 > > > > Setup: > > > > > > ---------- > > | switch | > > ---------- > > ---------------------| | | |-------------------- > > | | | | > > | --------| |-------- | > > | | | | > > | | | | > > -------------- -------------- -------------- -------------- > > | PC1 | | PC2 | | PC3 | | PC4 | > > | FreeBSD | | FreeBSD | | Linux | | Linux | > > |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| > > -------------- -------------- -------------- -------------- > > > > > > > > > > > > ================================ > > Netperf Test Results > > ================================ > > > > TCP Throughput Test > > ------------------- > > > > PC2/4: #netserver -p 22113 > > PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 32768 10.34 > 598.11 > > Linux: 87380 16384 16384 10.04 > 779.02 > > > > > > PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m > 64/128/256/512/1024/2048/4096 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 64 10.19 > 417.10 > > 65536 32768 128 10.35 > 336.63 > > 65536 32768 256 10.36 > 576.99 > > 65536 32768 512 10.35 > 569.79 > > 65536 32768 1024 10.35 > 553.70 > > 65536 32768 2048 10.35 > 584.20 > > 65536 32768 4096 10.35 > 602.45 > > > > Linux: 87380 16384 64 10.03 > 778.21 > > 87380 16384 128 10.03 > 779.72 > > 87380 16384 256 10.04 > 780.16 > > 87380 16384 512 10.03 > 776.85 > > 87380 16384 1024 10.04 > 777.52 > > 87380 16384 2048 10.04 > 777.83 > > 87380 16384 4096 10.03 > 780.17 > > > > > > > > > > =============================== > > Iperf Test Results > > =============================== > > > > Bandwidth Test > > -------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 740 600 > > Linux: 0.0-10.0 972 815 > > > > > > PC1/3: #iperf -c 192.168.1.20 -d > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.0 402 337 > > 0.0-10.0 404 338 > > > > Linux: 0.0-10.0 926 776 > > 0.0-10.0 44.1 36.9 > > > > > > Parallel Test > > ------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 -P 2 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 370 300 > > 0.0-10.3 370 300 > > SUM: 0.0-10.3 739 600 > > > > Linux: 0.0-10.0 479 402 > > 0.0-10.0 473 396 > > SUM: 0.0-10.0 952 797 > > > > > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" FreeBSD 7.1 is quite old compared to Linux 2.6.31 - I'd like to see at least FreeBSD 7.2 compared, if not 8.0-RC1. Maybe also the most recent FreeBSD 4 should be taken into this test. From leccine at gmail.com Thu Oct 15 12:12:53 2009 From: leccine at gmail.com (=?ISO-8859-1?B?SXN0duFu?=) Date: Thu Oct 15 12:12:59 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: use netpipe TCP pls. And graph it. Thanks in advance, Istvan On Thu, Oct 15, 2009 at 2:42 AM, Hongtao Yin wrote: > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test tools > Netperf and Iperf with Intel NIC. > > The kernels are full version and default values are used in the testing > except TCP Congestion Control algorithm set to Reno. > > >From the test results we can see Linux TCP performance in throughput is > better than FreeBSD. The worst case (send msg size 128) shows that FreeBSD > throughput is only 43% of Linux's. > > > > I like to get some feedback if anyone did similar comparison test, or knows > any issues with kernels or drivers. Thanks lot. > > > > FreeBSD and Linux Sysctl captures are attached for reference. > > > > Regards, > > Hongtao > > > > > > > > Test Environments: > > PC: Dell Precision T3400 (same 4 PCs) > > CPU: Intel Core 2 Duo CPU E4600@2.4Ghz > > FreeBSD: V7.1 (full version) (TCP CC: newReno) > > Linux: V2.6.31.1 (full version) (TCP CC: Reno) > > Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter > (Gigabit) > chip 82546EB (only one port used for each PC) > > Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) > > Iperf: V2.0.4 > > Netperf: V2.4.4 > > > > Setup: > > > > > > ---------- > > | switch | > > ---------- > > ---------------------| | | |-------------------- > > | | | | > > | --------| |-------- | > > | | | | > > | | | | > > -------------- -------------- -------------- -------------- > > | PC1 | | PC2 | | PC3 | | PC4 | > > | FreeBSD | | FreeBSD | | Linux | | Linux | > > |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| > > -------------- -------------- -------------- -------------- > > > > > > > > > > > > ================================ > > Netperf Test Results > > ================================ > > > > TCP Throughput Test > > ------------------- > > > > PC2/4: #netserver -p 22113 > > PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 32768 10.34 > 598.11 > > Linux: 87380 16384 16384 10.04 > 779.02 > > > > > > PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m > 64/128/256/512/1024/2048/4096 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 64 10.19 > 417.10 > > 65536 32768 128 10.35 > 336.63 > > 65536 32768 256 10.36 > 576.99 > > 65536 32768 512 10.35 > 569.79 > > 65536 32768 1024 10.35 > 553.70 > > 65536 32768 2048 10.35 > 584.20 > > 65536 32768 4096 10.35 > 602.45 > > > > Linux: 87380 16384 64 10.03 > 778.21 > > 87380 16384 128 10.03 > 779.72 > > 87380 16384 256 10.04 > 780.16 > > 87380 16384 512 10.03 > 776.85 > > 87380 16384 1024 10.04 > 777.52 > > 87380 16384 2048 10.04 > 777.83 > > 87380 16384 4096 10.03 > 780.17 > > > > > > > > > > =============================== > > Iperf Test Results > > =============================== > > > > Bandwidth Test > > -------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 740 600 > > Linux: 0.0-10.0 972 815 > > > > > > PC1/3: #iperf -c 192.168.1.20 -d > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.0 402 337 > > 0.0-10.0 404 338 > > > > Linux: 0.0-10.0 926 776 > > 0.0-10.0 44.1 36.9 > > > > > > Parallel Test > > ------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 -P 2 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 370 300 > > 0.0-10.3 370 300 > > SUM: 0.0-10.3 739 600 > > > > Linux: 0.0-10.0 479 402 > > 0.0-10.0 473 396 > > SUM: 0.0-10.0 952 797 > > > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > -- the sun shines for all From adrian at freebsd.org Thu Oct 15 21:21:05 2009 From: adrian at freebsd.org (Adrian Chadd) Date: Thu Oct 15 21:21:12 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: 2009/10/15 Hongtao Yin : > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test tools > Netperf and Iperf with Intel NIC. Did you compare syscalls made and time taken? For example, do either/both of them do a lot of gettimeofday() calls? FreeBSD and Linux have (had?) different behaviours and performance with those. I'd suggest digging a bit deeper? :) adrian From sdong at huawei.com Fri Oct 16 01:01:05 2009 From: sdong at huawei.com (Steve Dong) Date: Fri Oct 16 01:01:12 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: <000d01ca4dfa$081c36f0$3322c10a@china.huawei.com> There are two gettimeofday calls, one at the beginning of the test and the other at the end. netperf-2.4.4 is used. I believe gettimeofday is a vsyscall in linux, but a real system call in freebsd. Thanks, Steve -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Adrian Chadd Sent: Thursday, October 15, 2009 1:55 PM To: Hongtao Yin Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance 2009/10/15 Hongtao Yin : > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test > tools Netperf and Iperf with Intel NIC. Did you compare syscalls made and time taken? For example, do either/both of them do a lot of gettimeofday() calls? FreeBSD and Linux have (had?) different behaviours and performance with those. I'd suggest digging a bit deeper? :) adrian _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From htyin at huawei.com Fri Oct 16 03:52:10 2009 From: htyin at huawei.com (Hongtao Yin) Date: Fri Oct 16 03:52:17 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: <4831593800614E6796A45F20BA4B818E@china.huawei.com> Istvan, I ran newer FreeBSD code 8.0 RC1 this time. By using NetPIPE, we collected test data and the result shows that FreeBSD TCP performance is worse than Linux. I got troubles to plot using gnuplot, so here I am attaching the raw data files. The netpipe commands are as follows: PC2: #NPtcp PC1: #NPtcp ¨Ch 192.168.1.20 Regards, Hongtao _____ From: Istv¨¢n [mailto:leccine@gmail.com] Sent: 2009Äê10ÔÂ15ÈÕ 5:13 To: Hongtao Yin Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance use netpipe TCP pls. And graph it. Thanks in advance, Istvan On Thu, Oct 15, 2009 at 2:42 AM, Hongtao Yin wrote: Hi, I compared TCP performance between FreeBSD and Linux by running test tools Netperf and Iperf with Intel NIC. The kernels are full version and default values are used in the testing except TCP Congestion Control algorithm set to Reno. >From the test results we can see Linux TCP performance in throughput is better than FreeBSD. The worst case (send msg size 128) shows that FreeBSD throughput is only 43% of Linux's. I like to get some feedback if anyone did similar comparison test, or knows any issues with kernels or drivers. Thanks lot. FreeBSD and Linux Sysctl captures are attached for reference. Regards, Hongtao Test Environments: PC: Dell Precision T3400 (same 4 PCs) CPU: Intel Core 2 Duo CPU E4600@2.4Ghz FreeBSD: V7.1 (full version) (TCP CC: newReno) Linux: V2.6.31.1 (full version) (TCP CC: Reno) Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter (Gigabit) chip 82546EB (only one port used for each PC) Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) Iperf: V2.0.4 Netperf: V2.4.4 Setup: ---------- | switch | ---------- ---------------------| | | |-------------------- | | | | | --------| |-------- | | | | | | | | | -------------- -------------- -------------- -------------- | PC1 | | PC2 | | PC3 | | PC4 | | FreeBSD | | FreeBSD | | Linux | | Linux | |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| -------------- -------------- -------------- -------------- ================================ Netperf Test Results ================================ TCP Throughput Test ------------------- PC2/4: #netserver -p 22113 PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 32768 10.34 598.11 Linux: 87380 16384 16384 10.04 779.02 PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m 64/128/256/512/1024/2048/4096 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 64 10.19 417.10 65536 32768 128 10.35 336.63 65536 32768 256 10.36 576.99 65536 32768 512 10.35 569.79 65536 32768 1024 10.35 553.70 65536 32768 2048 10.35 584.20 65536 32768 4096 10.35 602.45 Linux: 87380 16384 64 10.03 778.21 87380 16384 128 10.03 779.72 87380 16384 256 10.04 780.16 87380 16384 512 10.03 776.85 87380 16384 1024 10.04 777.52 87380 16384 2048 10.04 777.83 87380 16384 4096 10.03 780.17 =============================== Iperf Test Results =============================== Bandwidth Test -------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 740 600 Linux: 0.0-10.0 972 815 PC1/3: #iperf -c 192.168.1.20 -d Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.0 402 337 0.0-10.0 404 338 Linux: 0.0-10.0 926 776 0.0-10.0 44.1 36.9 Parallel Test ------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 -P 2 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 370 300 0.0-10.3 370 300 SUM: 0.0-10.3 739 600 Linux: 0.0-10.0 479 402 0.0-10.0 473 396 SUM: 0.0-10.0 952 797 _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" -- the sun shines for all -------------- next part -------------- A non-text attachment was scrubbed... Name: linux_netpipe Type: application/octet-stream Size: 8681 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20091016/b448c897/linux_netpipe.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: freebsd_netpipe Type: application/octet-stream Size: 8681 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20091016/b448c897/freebsd_netpipe.obj From sdong at huawei.com Fri Oct 16 05:44:39 2009 From: sdong at huawei.com (Steve Dong) Date: Fri Oct 16 05:44:46 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <4831593800614E6796A45F20BA4B818E@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> Message-ID: <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> Here are graphs from the netpipe test results with 8.0 RC1 Thanks, Steve -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Hongtao Yin Sent: Thursday, October 15, 2009 8:52 PM To: 'Istv¨¢n' Cc: freebsd-performance@freebsd.org Subject: RE: Comparison of FreeBSD/Linux TCP Throughput performance Istvan, I ran newer FreeBSD code 8.0 RC1 this time. By using NetPIPE, we collected test data and the result shows that FreeBSD TCP performance is worse than Linux. I got troubles to plot using gnuplot, so here I am attaching the raw data files. The netpipe commands are as follows: PC2: #NPtcp PC1: #NPtcp ¨Ch 192.168.1.20 Regards, Hongtao _____ From: Istv¨¢n [mailto:leccine@gmail.com] Sent: 2009Äê10ÔÂ15ÈÕ 5:13 To: Hongtao Yin Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance use netpipe TCP pls. And graph it. Thanks in advance, Istvan On Thu, Oct 15, 2009 at 2:42 AM, Hongtao Yin wrote: Hi, I compared TCP performance between FreeBSD and Linux by running test tools Netperf and Iperf with Intel NIC. The kernels are full version and default values are used in the testing except TCP Congestion Control algorithm set to Reno. >From the test results we can see Linux TCP performance in throughput is better than FreeBSD. The worst case (send msg size 128) shows that FreeBSD throughput is only 43% of Linux's. I like to get some feedback if anyone did similar comparison test, or knows any issues with kernels or drivers. Thanks lot. FreeBSD and Linux Sysctl captures are attached for reference. Regards, Hongtao Test Environments: PC: Dell Precision T3400 (same 4 PCs) CPU: Intel Core 2 Duo CPU E4600@2.4Ghz FreeBSD: V7.1 (full version) (TCP CC: newReno) Linux: V2.6.31.1 (full version) (TCP CC: Reno) Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter (Gigabit) chip 82546EB (only one port used for each PC) Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) Iperf: V2.0.4 Netperf: V2.4.4 Setup: ---------- | switch | ---------- ---------------------| | | |-------------------- | | | | | --------| |-------- | | | | | | | | | -------------- -------------- -------------- -------------- | PC1 | | PC2 | | PC3 | | PC4 | | FreeBSD | | FreeBSD | | Linux | | Linux | |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| -------------- -------------- -------------- -------------- ================================ Netperf Test Results ================================ TCP Throughput Test ------------------- PC2/4: #netserver -p 22113 PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 32768 10.34 598.11 Linux: 87380 16384 16384 10.04 779.02 PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m 64/128/256/512/1024/2048/4096 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 64 10.19 417.10 65536 32768 128 10.35 336.63 65536 32768 256 10.36 576.99 65536 32768 512 10.35 569.79 65536 32768 1024 10.35 553.70 65536 32768 2048 10.35 584.20 65536 32768 4096 10.35 602.45 Linux: 87380 16384 64 10.03 778.21 87380 16384 128 10.03 779.72 87380 16384 256 10.04 780.16 87380 16384 512 10.03 776.85 87380 16384 1024 10.04 777.52 87380 16384 2048 10.04 777.83 87380 16384 4096 10.03 780.17 =============================== Iperf Test Results =============================== Bandwidth Test -------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 740 600 Linux: 0.0-10.0 972 815 PC1/3: #iperf -c 192.168.1.20 -d Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.0 402 337 0.0-10.0 404 338 Linux: 0.0-10.0 926 776 0.0-10.0 44.1 36.9 Parallel Test ------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 -P 2 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 370 300 0.0-10.3 370 300 SUM: 0.0-10.3 739 600 Linux: 0.0-10.0 479 402 0.0-10.0 473 396 SUM: 0.0-10.0 952 797 _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" -- the sun shines for all From sdong at huawei.com Fri Oct 16 05:53:21 2009 From: sdong at huawei.com (Steve Dong) Date: Fri Oct 16 05:53:28 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> Message-ID: <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> It looks the jpeg attachments were somehow dropped. Trying again with pdf attachment. Hopefully it works this time. Thanks, Steve -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Steve Dong Sent: Thursday, October 15, 2009 10:44 PM To: 'Hongtao Yin'; 'Istv¨¢n' Cc: freebsd-performance@freebsd.org Subject: RE: Comparison of FreeBSD/Linux TCP Throughput performance Here are graphs from the netpipe test results with 8.0 RC1 Thanks, Steve -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Hongtao Yin Sent: Thursday, October 15, 2009 8:52 PM To: 'Istv¨¢n' Cc: freebsd-performance@freebsd.org Subject: RE: Comparison of FreeBSD/Linux TCP Throughput performance Istvan, I ran newer FreeBSD code 8.0 RC1 this time. By using NetPIPE, we collected test data and the result shows that FreeBSD TCP performance is worse than Linux. I got troubles to plot using gnuplot, so here I am attaching the raw data files. The netpipe commands are as follows: PC2: #NPtcp PC1: #NPtcp ¨Ch 192.168.1.20 Regards, Hongtao _____ From: Istv¨¢n [mailto:leccine@gmail.com] Sent: 2009Äê10ÔÂ15ÈÕ 5:13 To: Hongtao Yin Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance use netpipe TCP pls. And graph it. Thanks in advance, Istvan On Thu, Oct 15, 2009 at 2:42 AM, Hongtao Yin wrote: Hi, I compared TCP performance between FreeBSD and Linux by running test tools Netperf and Iperf with Intel NIC. The kernels are full version and default values are used in the testing except TCP Congestion Control algorithm set to Reno. >From the test results we can see Linux TCP performance in throughput is better than FreeBSD. The worst case (send msg size 128) shows that FreeBSD throughput is only 43% of Linux's. I like to get some feedback if anyone did similar comparison test, or knows any issues with kernels or drivers. Thanks lot. FreeBSD and Linux Sysctl captures are attached for reference. Regards, Hongtao Test Environments: PC: Dell Precision T3400 (same 4 PCs) CPU: Intel Core 2 Duo CPU E4600@2.4Ghz FreeBSD: V7.1 (full version) (TCP CC: newReno) Linux: V2.6.31.1 (full version) (TCP CC: Reno) Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter (Gigabit) chip 82546EB (only one port used for each PC) Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) Iperf: V2.0.4 Netperf: V2.4.4 Setup: ---------- | switch | ---------- ---------------------| | | |-------------------- | | | | | --------| |-------- | | | | | | | | | -------------- -------------- -------------- -------------- | PC1 | | PC2 | | PC3 | | PC4 | | FreeBSD | | FreeBSD | | Linux | | Linux | |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| -------------- -------------- -------------- -------------- ================================ Netperf Test Results ================================ TCP Throughput Test ------------------- PC2/4: #netserver -p 22113 PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 32768 10.34 598.11 Linux: 87380 16384 16384 10.04 779.02 PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m 64/128/256/512/1024/2048/4096 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 64 10.19 417.10 65536 32768 128 10.35 336.63 65536 32768 256 10.36 576.99 65536 32768 512 10.35 569.79 65536 32768 1024 10.35 553.70 65536 32768 2048 10.35 584.20 65536 32768 4096 10.35 602.45 Linux: 87380 16384 64 10.03 778.21 87380 16384 128 10.03 779.72 87380 16384 256 10.04 780.16 87380 16384 512 10.03 776.85 87380 16384 1024 10.04 777.52 87380 16384 2048 10.04 777.83 87380 16384 4096 10.03 780.17 =============================== Iperf Test Results =============================== Bandwidth Test -------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 740 600 Linux: 0.0-10.0 972 815 PC1/3: #iperf -c 192.168.1.20 -d Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.0 402 337 0.0-10.0 404 338 Linux: 0.0-10.0 926 776 0.0-10.0 44.1 36.9 Parallel Test ------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 -P 2 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 370 300 0.0-10.3 370 300 SUM: 0.0-10.3 739 600 Linux: 0.0-10.0 479 402 0.0-10.0 473 396 SUM: 0.0-10.0 952 797 _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" -- the sun shines for all -------------- next part -------------- A non-text attachment was scrubbed... Name: netpipe.pdf Type: application/pdf Size: 40078 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20091016/fd421ba2/netpipe.pdf From dkelly at hiwaay.net Fri Oct 16 07:04:21 2009 From: dkelly at hiwaay.net (David Kelly) Date: Fri Oct 16 07:04:28 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> Message-ID: <5FD087E4-438B-41F0-B6C3-D6C3792F5B4D@hiwaay.net> On Oct 16, 2009, at 12:53 AM, Steve Dong wrote: > It looks the jpeg attachments were somehow dropped. Trying again > with pdf > attachment. Hopefully it works this time. *I* saw two jpeg charts at the very bottom of Steve's top-posted reply. This demonstrates just one of many evils of top-posting: the top-poster doesn't read everything he re-sends. If its not worth reading then why send it again? -------------- next part -------------- A non-text attachment was scrubbed... Name: netpipe.pdf Type: application/pdf Size: 40078 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20091016/5a7c546c/netpipe-0001.pdf -------------- next part -------------- -- David Kelly N4HHE, dkelly@HiWAAY.net ======================================================================== Whom computers would destroy, they must first drive mad. From wmoran at collaborativefusion.com Fri Oct 16 11:53:57 2009 From: wmoran at collaborativefusion.com (Bill Moran) Date: Fri Oct 16 11:54:03 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> Message-ID: <20091016075336.03eb17f2.wmoran@collaborativefusion.com> Steve Dong wrote: > > It looks the jpeg attachments were somehow dropped. Trying again with pdf > attachment. Hopefully it works this time. Really. Don't post attachments to mailing lists. It's just a bad idea, a lot of people will be upset with the bandwidth it consumes. Keep in mind that not everyone on the list is interested in every conversation. -- Bill Moran Collaborative Fusion Inc. wmoran@collaborativefusion.com Phone: 412-422-3463x4023 **************************************************************** IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. **************************************************************** From kometen at gmail.com Fri Oct 16 12:24:43 2009 From: kometen at gmail.com (Claus Guttesen) Date: Fri Oct 16 12:24:50 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <20091016075336.03eb17f2.wmoran@collaborativefusion.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> <20091016075336.03eb17f2.wmoran@collaborativefusion.com> Message-ID: > Really. ?Don't post attachments to mailing lists. ?It's just a bad idea, > a lot of people will be upset with the bandwidth it consumes. ?Keep in > mind that not everyone on the list is interested in every conversation. Disclaimers should go the same way too! :) echo " **************************************************************** IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****************************************************************" | wc 16 129 958 -- regards Claus When lenity and cruelty play for a kingdom, the gentler gamester is the soonest winner. Shakespeare twitter.com/kometen From leccine at gmail.com Fri Oct 16 12:28:39 2009 From: leccine at gmail.com (=?ISO-8859-1?B?SXN0duFu?=) Date: Fri Oct 16 12:28:46 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <4831593800614E6796A45F20BA4B818E@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> Message-ID: I see. It shows that linux default setup is better. Have you seen any FreeBSD performance tuning guides? Regards, Istvan 2009/10/16 Hongtao Yin > Istvan, > > > > I ran newer FreeBSD code 8.0 RC1 this time. By using NetPIPE, we collecte= d > test data and the result shows that FreeBSD TCP performance is worse than > Linux. I got troubles to plot using gnuplot, so here I am attaching the r= aw > data files. > > The netpipe commands are as follows: > > PC2: #NPtcp > > PC1: #NPtcp =A8Ch 192.168.1.20 > > > > Regards, > > Hongtao > > > > > > > ------------------------------ > > *From:* Istv=A8=A2n [mailto:leccine@gmail.com] > *Sent:* 2009=C4=EA10=D4=C215=C8=D5 5:13 > > *To:* Hongtao Yin > *Cc:* freebsd-performance@freebsd.org > *Subject:* Re: Comparison of FreeBSD/Linux TCP Throughput performance > > > > use netpipe TCP pls. > > > > And graph it. > > > > Thanks in advance, > > Istvan > > On Thu, Oct 15, 2009 at 2:42 AM, Hongtao Yin wrote: > > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test tool= s > Netperf and Iperf with Intel NIC. > > The kernels are full version and default values are used in the testing > except TCP Congestion Control algorithm set to Reno. > > >From the test results we can see Linux TCP performance in throughput is > better than FreeBSD. The worst case (send msg size 128) shows that FreeBS= D > throughput is only 43% of Linux's. > > > > I like to get some feedback if anyone did similar comparison test, or kno= ws > any issues with kernels or drivers. Thanks lot. > > > > FreeBSD and Linux Sysctl captures are attached for reference. > > > > Regards, > > Hongtao > > > > > > > > Test Environments: > > PC: Dell Precision T3400 (same 4 PCs) > > CPU: Intel Core 2 Duo CPU E4600@2.4Ghz > > FreeBSD: V7.1 (full version) (TCP CC: newReno) > > Linux: V2.6.31.1 (full version) (TCP CC: Reno) > > Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter > (Gigabit) > chip 82546EB (only one port used for each PC) > > Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) > > Iperf: V2.0.4 > > Netperf: V2.4.4 > > > > Setup: > > > > > > ---------- > > | switch | > > ---------- > > ---------------------| | | |-------------------- > > | | | | > > | --------| |-------- | > > | | | | > > | | | | > > -------------- -------------- -------------- -------------- > > | PC1 | | PC2 | | PC3 | | PC4 | > > | FreeBSD | | FreeBSD | | Linux | | Linux | > > |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| > > -------------- -------------- -------------- -------------- > > > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > Netperf Test Results > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > > TCP Throughput Test > > ------------------- > > > > PC2/4: #netserver -p 22113 > > PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 32768 10.34 > 598.11 > > Linux: 87380 16384 16384 10.04 > 779.02 > > > > > > PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m > 64/128/256/512/1024/2048/4096 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 64 10.19 > 417.10 > > 65536 32768 128 10.35 > 336.63 > > 65536 32768 256 10.36 > 576.99 > > 65536 32768 512 10.35 > 569.79 > > 65536 32768 1024 10.35 > 553.70 > > 65536 32768 2048 10.35 > 584.20 > > 65536 32768 4096 10.35 > 602.45 > > > > Linux: 87380 16384 64 10.03 > 778.21 > > 87380 16384 128 10.03 > 779.72 > > 87380 16384 256 10.04 > 780.16 > > 87380 16384 512 10.03 > 776.85 > > 87380 16384 1024 10.04 > 777.52 > > 87380 16384 2048 10.04 > 777.83 > > 87380 16384 4096 10.03 > 780.17 > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > Iperf Test Results > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > > > Bandwidth Test > > -------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 740 600 > > Linux: 0.0-10.0 972 815 > > > > > > PC1/3: #iperf -c 192.168.1.20 -d > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.0 402 337 > > 0.0-10.0 404 338 > > > > Linux: 0.0-10.0 926 776 > > 0.0-10.0 44.1 36.9 > > > > > > Parallel Test > > ------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 -P 2 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 370 300 > > 0.0-10.3 370 300 > > SUM: 0.0-10.3 739 600 > > > > Linux: 0.0-10.0 479 402 > > 0.0-10.0 473 396 > > SUM: 0.0-10.0 952 797 > > > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > > > > > -- > the sun shines for all > --=20 the sun shines for all From leccine at gmail.com Fri Oct 16 12:29:31 2009 From: leccine at gmail.com (=?ISO-8859-1?B?SXN0duFu?=) Date: Fri Oct 16 12:29:42 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> Message-ID: Thanks! Regards, Istvan 2009/10/16 Steve Dong > Here are graphs from the netpipe test results with 8.0 RC1 > > > Thanks, > Steve > > > > -----Original Message----- > From: owner-freebsd-performance@freebsd.org > [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Hongtao Yin > Sent: Thursday, October 15, 2009 8:52 PM > To: 'Istv=A8=A2n' > Cc: freebsd-performance@freebsd.org > Subject: RE: Comparison of FreeBSD/Linux TCP Throughput performance > > Istvan, > > > > I ran newer FreeBSD code 8.0 RC1 this time. By using NetPIPE, we collecte= d > test data and the result shows that FreeBSD TCP performance is worse than > Linux. I got troubles to plot using gnuplot, so here I am attaching the r= aw > data files. > > The netpipe commands are as follows: > > PC2: #NPtcp > > PC1: #NPtcp =A8Ch 192.168.1.20 > > > > Regards, > > Hongtao > > > > > > > > _____ > > From: Istv=A8=A2n [mailto:leccine@gmail.com] > Sent: 2009=C4=EA10=D4=C215=C8=D5 5:13 > To: Hongtao Yin > Cc: freebsd-performance@freebsd.org > Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance > > > > use netpipe TCP pls. > > > > And graph it. > > > > Thanks in advance, > > Istvan > > On Thu, Oct 15, 2009 at 2:42 AM, Hongtao Yin wrote: > > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test tool= s > Netperf and Iperf with Intel NIC. > > The kernels are full version and default values are used in the testing > except TCP Congestion Control algorithm set to Reno. > > >From the test results we can see Linux TCP performance in throughput is > better than FreeBSD. The worst case (send msg size 128) shows that FreeBS= D > throughput is only 43% of Linux's. > > > > I like to get some feedback if anyone did similar comparison test, or kno= ws > any issues with kernels or drivers. Thanks lot. > > > > FreeBSD and Linux Sysctl captures are attached for reference. > > > > Regards, > > Hongtao > > > > > > > > Test Environments: > > PC: Dell Precision T3400 (same 4 PCs) > > CPU: Intel Core 2 Duo CPU E4600@2.4Ghz > > FreeBSD: V7.1 (full version) (TCP CC: newReno) > > Linux: V2.6.31.1 (full version) (TCP CC: Reno) > > Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter > (Gigabit) > chip 82546EB (only one port used for each PC) > > Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) > > Iperf: V2.0.4 > > Netperf: V2.4.4 > > > > Setup: > > > > > > ---------- > > | switch | > > ---------- > > ---------------------| | | |-------------------- > > | | | | > > | --------| |-------- | > > | | | | > > | | | | > > -------------- -------------- -------------- -------------- > > | PC1 | | PC2 | | PC3 | | PC4 | > > | FreeBSD | | FreeBSD | | Linux | | Linux | > > |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| > > -------------- -------------- -------------- -------------- > > > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > Netperf Test Results > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > > TCP Throughput Test > > ------------------- > > > > PC2/4: #netserver -p 22113 > > PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 32768 10.34 > 598.11 > > Linux: 87380 16384 16384 10.04 > 779.02 > > > > > > PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m > 64/128/256/512/1024/2048/4096 > > > > Recv Send Send Elapsed > Throughput > > Socket Socket Message Time > 10^6 bits/sec > > Size Size Size Sec. > > bytes bytes bytes > > FreeBSD: 65536 32768 64 10.19 > 417.10 > > 65536 32768 128 10.35 > 336.63 > > 65536 32768 256 10.36 > 576.99 > > 65536 32768 512 10.35 > 569.79 > > 65536 32768 1024 10.35 > 553.70 > > 65536 32768 2048 10.35 > 584.20 > > 65536 32768 4096 10.35 > 602.45 > > > > Linux: 87380 16384 64 10.03 > 778.21 > > 87380 16384 128 10.03 > 779.72 > > 87380 16384 256 10.04 > 780.16 > > 87380 16384 512 10.03 > 776.85 > > 87380 16384 1024 10.04 > 777.52 > > 87380 16384 2048 10.04 > 777.83 > > 87380 16384 4096 10.03 > 780.17 > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > Iperf Test Results > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > > > Bandwidth Test > > -------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 740 600 > > Linux: 0.0-10.0 972 815 > > > > > > PC1/3: #iperf -c 192.168.1.20 -d > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.0 402 337 > > 0.0-10.0 404 338 > > > > Linux: 0.0-10.0 926 776 > > 0.0-10.0 44.1 36.9 > > > > > > Parallel Test > > ------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 -P 2 > > > > Interval Transfer Bandwidth > > sec MBytes Mbits/sec > > FreeBSD: 0.0-10.3 370 300 > > 0.0-10.3 370 300 > > SUM: 0.0-10.3 739 600 > > > > Linux: 0.0-10.0 479 402 > > 0.0-10.0 473 396 > > SUM: 0.0-10.0 952 797 > > > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > > > > > -- > the sun shines for all > > --=20 the sun shines for all From lists at stringsutils.com Fri Oct 16 18:57:01 2009 From: lists at stringsutils.com (Francisco Reyes) Date: Fri Oct 16 18:57:07 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> Message-ID: Hongtao Yin writes: > I ran newer FreeBSD code 8.0 RC1 this time. By using NetPIPE, we collected Check man tuning There are a few parameters there worth exploring. For example check the section on net.inet.tcp.sendspace and net.inet.tcp.recvspace From htyin at huawei.com Sat Oct 17 02:08:49 2009 From: htyin at huawei.com (Hongtao Yin) Date: Sat Oct 17 02:08:57 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> Message-ID: This is one of the first things we tried.=20 =20 Does anyone have the tuned parameters?=20 =20 Actually we are looking for info like =20 1. Any bugs in Freebsd driver which have been fixed in linux kernel=20 2. Any TCP features/RFCs implemented in Linux, but not in Freebsd=20 3. Any other discrepancies between Linux and FreeBSD TCP implementation = that could potentially have caused this =20 =20 Thanks. Hongtao =20 =20 _____ =20 From: Istv=A8=A2n [mailto:leccine@gmail.com]=20 Sent: 2009=C4=EA10=D4=C216=C8=D5 5:29 To: Hongtao Yin Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance =20 I see. =20 It shows that linux default setup is better. =20 Have you seen any FreeBSD performance tuning guides? =20 Regards, Istvan 2009/10/16 Hongtao Yin Istvan, =20 I ran newer FreeBSD code 8.0 RC1 this time. By using NetPIPE, we = collected test data and the result shows that FreeBSD TCP performance is worse = than Linux. I got troubles to plot using gnuplot, so here I am attaching the = raw data files. The netpipe commands are as follows: PC2: #NPtcp PC1: #NPtcp =A8Ch 192.168.1.20 =20 Regards, Hongtao =20 =20 =20 _____ =20 From: Istv=A8=A2n [mailto:leccine@gmail.com]=20 Sent: 2009=C4=EA10=D4=C215=C8=D5 5:13 To: Hongtao Yin Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance =20 use netpipe TCP pls. =20 And graph it. =20 Thanks in advance, Istvan On Thu, Oct 15, 2009 at 2:42 AM, Hongtao Yin wrote: Hi, I compared TCP performance between FreeBSD and Linux by running test = tools Netperf and Iperf with Intel NIC. The kernels are full version and default values are used in the testing except TCP Congestion Control algorithm set to Reno. >From the test results we can see Linux TCP performance in throughput is better than FreeBSD. The worst case (send msg size 128) shows that = FreeBSD throughput is only 43% of Linux's. I like to get some feedback if anyone did similar comparison test, or = knows any issues with kernels or drivers. Thanks lot. FreeBSD and Linux Sysctl captures are attached for reference. Regards, Hongtao Test Environments: PC: Dell Precision T3400 (same 4 PCs) CPU: Intel Core 2 Duo CPU E4600@2.4Ghz FreeBSD: V7.1 (full version) (TCP CC: newReno) Linux: V2.6.31.1 (full version) (TCP CC: Reno) Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter = (Gigabit) chip 82546EB (only one port used for each PC) Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) Iperf: V2.0.4 Netperf: V2.4.4 Setup: ---------- | switch | ---------- ---------------------| | | |-------------------- | | | | | --------| |-------- | | | | | | | | | -------------- -------------- -------------- -------------- | PC1 | | PC2 | | PC3 | | PC4 | | FreeBSD | | FreeBSD | | Linux | | Linux | |192.168.1.10| |192.168.1.20| |192.168.1.30| |192.168.1.40| -------------- -------------- -------------- -------------- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D Netperf Test Results =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D TCP Throughput Test ------------------- PC2/4: #netserver -p 22113 PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 32768 10.34 598.11 Linux: 87380 16384 16384 10.04 779.02 PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m 64/128/256/512/1024/2048/4096 Recv Send Send Elapsed Throughput Socket Socket Message Time 10^6 bits/sec Size Size Size Sec. bytes bytes bytes FreeBSD: 65536 32768 64 10.19 417.10 65536 32768 128 10.35 336.63 65536 32768 256 10.36 576.99 65536 32768 512 10.35 569.79 65536 32768 1024 10.35 553.70 65536 32768 2048 10.35 584.20 65536 32768 4096 10.35 602.45 Linux: 87380 16384 64 10.03 778.21 87380 16384 128 10.03 779.72 87380 16384 256 10.04 780.16 87380 16384 512 10.03 776.85 87380 16384 1024 10.04 777.52 87380 16384 2048 10.04 777.83 87380 16384 4096 10.03 780.17 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D Iperf Test Results =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D Bandwidth Test -------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 740 600 Linux: 0.0-10.0 972 815 PC1/3: #iperf -c 192.168.1.20 -d Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.0 402 337 0.0-10.0 404 338 Linux: 0.0-10.0 926 776 0.0-10.0 44.1 36.9 Parallel Test ------------- PC2/4: #iperf -s PC1/3: #iperf -c 192.168.1.20 -P 2 Interval Transfer Bandwidth sec MBytes Mbits/sec FreeBSD: 0.0-10.3 370 300 0.0-10.3 370 300 SUM: 0.0-10.3 739 600 Linux: 0.0-10.0 479 402 0.0-10.0 473 396 SUM: 0.0-10.0 952 797 _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" --=20 the sun shines for all --=20 the sun shines for all From adrian at freebsd.org Sat Oct 17 10:15:40 2009 From: adrian at freebsd.org (Adrian Chadd) Date: Sat Oct 17 10:15:46 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> Message-ID: 2009/10/16 Istv?n : > I see. > It shows that linux default setup is better. .. being completely correct, it shows the linux default setup _for netpipe_ is better on that particular hardware. That identifies a few other variables which may need addressing. :) Adrian From leccine at gmail.com Sat Oct 17 10:44:55 2009 From: leccine at gmail.com (=?ISO-8859-1?B?SXN0duFu?=) Date: Sat Oct 17 10:45:00 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> Message-ID: I guess it is not only for netpipe, it is doing a pretty decent job changing the packet size checking the performance so finally you have an overview about the size, lag, bw I like! :) On Sat, Oct 17, 2009 at 11:15 AM, Adrian Chadd wrote: > 2009/10/16 Istv?n : > > I see. > > It shows that linux default setup is better. > > .. being completely correct, it shows the linux default setup _for > netpipe_ is better on that particular hardware. > > That identifies a few other variables which may need addressing. :) > > > Adrian > -- the sun shines for all From ohartman at mail.zedat.fu-berlin.de Sat Oct 17 14:17:04 2009 From: ohartman at mail.zedat.fu-berlin.de (O. Hartmann) Date: Sat Oct 17 15:05:45 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> Message-ID: <4AD9D1DE.9050504@mail.zedat.fu-berlin.de> Istv?n wrote: > I guess it is not only for netpipe, it is doing a pretty decent job changing > the packet size checking the performance so finally you have an overview > about the size, lag, bw > > I like! :) > > On Sat, Oct 17, 2009 at 11:15 AM, Adrian Chadd wrote: > >> 2009/10/16 Istv?n : >>> I see. >>> It shows that linux default setup is better. >> .. being completely correct, it shows the linux default setup _for >> netpipe_ is better on that particular hardware. >> >> That identifies a few other variables which may need addressing. :) >> >> >> Adrian >> > > > And maybe the wise can send out word of doing well on FreeBSD, so we can choose a better setup with looking forward repeating the test under 'tuned' conditions? I'm willing to perform some tests within the next 4 weeks when our server hardware (Dell PowerEdge 1950-III with two if_bge() NICs and 16GB RAM) changes OS from FreeBSD 8.0 to RedHat Linux. In a time-window of about a week I might be capable of testing FreeBSD 8.0 as it would be that time by the mid of November with a setup of Linux (distro doesn't matter as I can choose). I need to know WHAT, WHERE and HOW. Thanks. oh From sdong at huawei.com Sat Oct 17 15:14:12 2009 From: sdong at huawei.com (Steve Dong) Date: Sat Oct 17 15:14:39 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <20091016075336.03eb17f2.wmoran@collaborativefusion.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> <20091016075336.03eb17f2.wmoran@collaborativefusion.com> Message-ID: <000001ca4f3c$78dc3550$6501a8c0@china.huawei.com> If there's a better/lighter way to show these graphics, I'd like to know. Thanks, Steve -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Bill Moran Sent: Friday, October 16, 2009 4:54 AM To: Steve Dong Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance Steve Dong wrote: > > It looks the jpeg attachments were somehow dropped. Trying again with > pdf attachment. Hopefully it works this time. Really. Don't post attachments to mailing lists. It's just a bad idea, a lot of people will be upset with the bandwidth it consumes. Keep in mind that not everyone on the list is interested in every conversation. -- Bill Moran Collaborative Fusion Inc. wmoran@collaborativefusion.com Phone: 412-422-3463x4023 **************************************************************** IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. **************************************************************** _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From cswiger at mac.com Sat Oct 17 16:58:46 2009 From: cswiger at mac.com (Chuck Swiger) Date: Sat Oct 17 16:58:53 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <000001ca4f3c$78dc3550$6501a8c0@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> <20091016075336.03eb17f2.wmoran@collaborativefusion.com> <000001ca4f3c$78dc3550$6501a8c0@china.huawei.com> Message-ID: <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> Hi, Steve-- On Oct 17, 2009, at 8:14 AM, Steve Dong wrote: > If there's a better/lighter way to show these graphics, I'd like to > know. Sure-- put 'em on a webserver somewhere, and put links to them in your email to this mailing list. If you wanted to do even better than that, set up a simple webpage describing what you are doing in your comparison, have a link to the dmesg/boot output for each platform as a .txt file and a description of any system tweaks & tuning, have a link that points to a description of the test setup (ie, your ASCII diagram of the switch and 4 machines), then your graphs, then the raw data (or links to it, depending). You can then throw in netstat -s output, or NIC driver stats from sysctl, or switch stats, etc-- anything else that adds useful context. There are a fair number of posts in the list archives which describe how to benchmark reliably, and the people who are most likely to be making code changes to FreeBSD also tend to like to know whether you've collected enough data, in a controlled fashion, to have an idea as to whether your measurements are reproducible. I'm not a purist, and I believe you can get useful estimations without rigorous testing, but there are others who make the point that if you haven't provided at least a standard deviation, then you haven't collected enough data-- done enough trials-- to determine whether the results are meaningful. (See /usr/src/tools/tools/ministat/README) Of course, you're not obligated to do any of the above, but if you want the effort you've put in to be more useful, consider these a suggestion. Finally, the next step beyond that would be to try tweaking some things, and see what kind of changes you get from that versus the original performance. It might be the case that making a simple tuning change would have a significant difference in performance; if you can identify that, then FreeBSD or Linux developers can use that information to better tune the OS defaults. Regards, -- -Chuck From gnn at neville-neil.com Sat Oct 17 20:36:42 2009 From: gnn at neville-neil.com (George Neville-Neil) Date: Sun Oct 18 03:30:57 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> <20091016075336.03eb17f2.wmoran@collaborativefusion.com> <000001ca4f3c$78dc3550$6501a8c0@china.huawei.com> <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> Message-ID: Hi, Trying to chime in with a few pointers here. Things to check when doing a TCP benchmark on FreeBSD. In particular make sure to adjust theses: net.inet.tcp.recvbuf_max: 262144 net.inet.tcp.recvbuf_inc: 16384 net.inet.tcp.recvbuf_auto: 1 net.inet.tcp.sendbuf_max: 262144 net.inet.tcp.sendbuf_inc: 8192 net.inet.tcp.sendbuf_auto: 1 Leave the auto on, but increase the max values and you should probably also change the inc (increment) values as well. Make sure that if you increase the buffer sizes you increase your number of mbufs and clusters as well. See kern.ipc.nmbclusters, which is a kernel tunable that can be set in /boot/loader.conf . Make sure that both of the systems you're testing have the same low level hardware support such as TCP Segment Offload (TSO) and TCP Checkusm Offload are turned on. Also you might want to turn this off: net.inet.tcp.inflight.enable: 1 This page http://fasterdata.es.net/TCP-tuning/FreeBSD.html claims that it can harm high speed connections. Those are the basics to start with. A search of "Tuning FreeBSD TCP" turns up some decent pages as well. Best, George From freebsd at sopwith.solgatos.com Sun Oct 18 04:44:44 2009 From: freebsd at sopwith.solgatos.com (Dieter) Date: Sun Oct 18 04:45:22 2009 Subject: tuning FFS for large files Re: A specific example of a disk i/o problem In-Reply-To: Your message of "Tue, 06 Oct 2009 18:03:16 +1100." <20091006174121.V25604@delplex.bde.org> Message-ID: <200910180440.EAA21373@sopwith.solgatos.com> > > I found a clue! The problem occurs with my big data partitions, > > which are newfs-ed with options intended to improve things. > > > > Reading a large file from the normal ad4s5b partition only delays other > > commands slightly, as expected. Reading a large file from the tuned > > ad4s11 partition yields the delay of minutes for other i/o. > > ... > > Here is the newfs command used for creating large data partitions: > > newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition > > Any block size above the default (16K) tends to thrash and fragment buffer > cache virtual memory. This is obviously a good pessimization with lots of > small files, and apparently, as you have found, it is a good pessimization > with a few large files too. I think severe fragmentation can easily take > several seconds to recover from. The worst case for causing fragmentaion > is probably a mixture of small and large files. Is there any way to avoid the "thrash and fragment buffercache virtual memory" problem other than keeping the block size 16K or smaller? > Some users fear fs consistency bugs with block sizes >= 16K, but I've never > seen them cause any bugs ecept performance ones. Yep, many TB of files on filesystems created with above newfs command and no corruption/consistency problems. > > And they have way more inodes than needed. (IIRC it doesn't actually > > use -i 67108864) > > It has to have at least 1 inode per cg, and may as well have a full block > of them, which gives a fairly large number of inodes especially if the > block size is too large (64K), so the -i ratio is limited. I converted a few filesystems to the default. In addition to losing space, fsck time went through the roof. So back to playing with newfs options. For some reason, larger block/frag sizes allow fewer cylinder groups, which reduces the number of inodes more than the larger block size increases it. From my reading of the newfs man page, -c only allows making cylinder groups smaller, not larger, and that appears to be the case in practice. default: newfs -U /dev/ad14s4 /dev/ad14s4: 431252.6MB (883205320 sectors) block size 16384, fragment size 2048 using 2348 cylinder groups of 183.72MB, 11758 blks, 23552 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 417678 0 384263 0% 2 55300092 0% fsck -fp: real 0m37.165s Attempt to reduce number of inodes: newfs -U -i 134217728 -g 134217728 -h 16 -e 261129 /dev/ad14s4 density reduced from 134217728 to 3676160 /dev/ad14s4: 431252.6MB (883205320 sectors) block size 16384, fragment size 2048 using 1923 cylinder groups of 224.38MB, 14360 blks, 64 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 431162 0 396669 0% 2 123068 0% fsck -fp: real 0m32.687s Bigger block size: newfs -U -i 134217728 -g 134217728 -h 16 -e 261129 -b 65536 /dev/ad14s4 increasing fragment size from 2048 to block size / 8 (8192) density reduced from 134217728 to 14860288 /dev/ad14s4: 431252.6MB (883205312 sectors) block size 65536, fragment size 8192 using 119 cylinder groups of 3628.00MB, 58048 blks, 256 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 431230 0 396731 0% 2 30460 0% fsck -fp: real 0m3.144s Bigger block size and bigger frag size: newfs -U -i 134217728 -g 134217728 -h 16 -e 261129 -b 65536 -f 65536 /dev/ad14s4 density reduced from 134217728 to 66846720 /dev/ad14s4: 431252.6MB (883205248 sectors) block size 65536, fragment size 65536 using 27 cylinder groups of 16320.56MB, 261129 blks, 512 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 431245 0 396745 0% 2 13820 0% fsck -fp: real 0m0.369s With -b 65536 -f 65536 I'm finally approaching a reasonable number of inodes (even less would be better). The fsck time varies by a factor of over 100, and results are roughly similar on filesystems with files in them. From adrian at freebsd.org Sun Oct 18 14:55:02 2009 From: adrian at freebsd.org (Adrian Chadd) Date: Sun Oct 18 14:55:09 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: FYI, I installed netperf on my local p4-D test boxes that I use for other testing. 128 byte send/receive buffers on the client side: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 128 128 10.00 426.17 1kbyte send/receive buffers: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 1024 1024 10.00 903.39 8kbyte send/receive buffers: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 8192 8192 10.00 913.71 Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some onboard flavour of the intel e1000 NIC: device = '82573E Intel Corporation 82573E Gigabit Ethernet Controller (Copper)' They are connected via a Cisco 3750G L3 switch. In fact, the traffic is routed, rather than switched. My /etc/sysctl.conf: net.inet.icmp.icmplim=0 net.inet.icmp.icmplim_output=0 net.inet.tcp.msl=3000 net.inet.tcp.sendspace=8192 net.inet.tcp.recvspace=8192 kern.maxfilesperproc=65536 kern.maxfiles=262144 kern.ipc.maxsockets=32768 kern.ipc.somaxconn=1024 kern.ipc.nmbclusters=131072 net.inet.ip.fw.enable=0 kern.ipc.somaxconn=10240 2c, Adrian 2009/10/15 Hongtao Yin : > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test tools > Netperf and Iperf with Intel NIC. > > The kernels are full version and default values are used in the testing > except TCP Congestion Control algorithm set to Reno. > > >From the test results we can see Linux TCP performance in throughput is > better than FreeBSD. The worst case (send msg size 128) shows that FreeBSD > throughput is only 43% of Linux's. > > > > I like to get some feedback if anyone did similar comparison test, or knows > any issues with kernels or drivers. Thanks lot. > > > > FreeBSD and Linux Sysctl captures are attached for reference. > > > > Regards, > > Hongtao > > > > > > > > Test Environments: > > PC: Dell Precision T3400 (same 4 PCs) > > CPU: Intel Core 2 Duo CPU E4600@2.4Ghz > > FreeBSD: V7.1 (full version) (TCP CC: newReno) > > Linux: V2.6.31.1 (full version) (TCP CC: Reno) > > Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter (Gigabit) > chip 82546EB (only one port used for each PC) > > Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) > > Iperf: V2.0.4 > > Netperf: V2.4.4 > > > > Setup: > > > > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?---------- > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?| switch | > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?---------- > > ? ? ? ? ? ? ? ---------------------| | | |-------------------- > > ? ? ? ? ? ? ? | ? ? ? ? ? ? ? ? ? ? ?| | ? ? ? ? ? ? ? ? ? ? | > > ? ? ? ? ? ? ? | ? ? ? ? ? ? ?--------| |-------- ? ? ? ? ? ? | > > ? ? ? ? ? ? ? | ? ? ? ? ? ? ?| ? ? ? ? ? ? ? ? | ? ? ? ? ? ? | > > ? ? ? ? ? ? ? | ? ? ? ? ? ? ?| ? ? ? ? ? ? ? ? | ? ? ? ? ? ? | > > ? ? ? ?-------------- ?-------------- ?-------------- ?-------------- > > ? ? ? ?| ? ? PC1 ? ?| ?| ? ?PC2 ? ? | ?| ? ?PC3 ? ? | ?| ? ?PC4 ? ? | > > ? ? ? ?| ? FreeBSD ?| ?| ? FreeBSD ?| ?| ? Linux ? ?| ?| ? Linux ? ?| > > ? ? ? ?|192.168.1.10| ?|192.168.1.20| ?|192.168.1.30| ?|192.168.1.40| > > ? ? ? ?-------------- ?-------------- ?-------------- ?-------------- > > > > > > > > > > > > ================================ > > ? ? Netperf Test Results > > ================================ > > > > TCP Throughput Test > > ------------------- > > > > PC2/4: #netserver -p 22113 > > PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 > > > > ? ? ? ? ?Recv ? ? ? ? ? ? Send ? ? ? ? ? ?Send ? ? ? ? ? Elapsed > Throughput > > ? ? ? ? ?Socket ? ? ? ? ? Socket ? ? ? ? ?Message ? ? ? ?Time > 10^6 bits/sec > > ? ? ? ? ?Size ? ? ? ? ? ? Size ? ? ? ? ? ?Size ? ? ? ? ? Sec. > > ? ? ? ? ?bytes ? ? ? ? ? ?bytes ? ? ? ? ? bytes > > FreeBSD: ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 32768 ? ? ? ? ?10.34 > 598.11 > > Linux: ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 16384 ? ? ? ? ?10.04 > 779.02 > > > > > > PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m > 64/128/256/512/1024/2048/4096 > > > > ? ? ? ? ?Recv ? ? ? ? ? ? Send ? ? ? ? ? ?Send ? ? ? ? ? Elapsed > Throughput > > ? ? ? ? ?Socket ? ? ? ? ? Socket ? ? ? ? ?Message ? ? ? ?Time > 10^6 bits/sec > > ? ? ? ? ?Size ? ? ? ? ? ? Size ? ? ? ? ? ?Size ? ? ? ? ? Sec. > > ? ? ? ? ?bytes ? ? ? ? ? ?bytes ? ? ? ? ? bytes > > FreeBSD: ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 64 ? ? ? ? ? ? ?10.19 > 417.10 > > ? ? ? ? ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 128 ? ? ? ? ? ? 10.35 > 336.63 > > ? ? ? ? ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 256 ? ? ? ? ? ? 10.36 > 576.99 > > ? ? ? ? ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 512 ? ? ? ? ? ? 10.35 > 569.79 > > ? ? ? ? ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 1024 ? ? ? ? ? ?10.35 > 553.70 > > ? ? ? ? ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 2048 ? ? ? ? ? ?10.35 > 584.20 > > ? ? ? ? ?65536 ? ? ? ? ? ?32768 ? ? ? ? ? 4096 ? ? ? ? ? ?10.35 > 602.45 > > > > Linux: ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 64 ? ? ? ? ? ? ?10.03 > 778.21 > > ? ? ? ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 128 ? ? ? ? ? ? 10.03 > 779.72 > > ? ? ? ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 256 ? ? ? ? ? ? 10.04 > 780.16 > > ? ? ? ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 512 ? ? ? ? ? ? 10.03 > 776.85 > > ? ? ? ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 1024 ? ? ? ? ? ?10.04 > 777.52 > > ? ? ? ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 2048 ? ? ? ? ? ?10.04 > 777.83 > > ? ? ? ? ?87380 ? ? ? ? ? ?16384 ? ? ? ? ? 4096 ? ? ? ? ? ?10.03 > 780.17 > > > > > > > > > > =============================== > > ? ? ?Iperf Test Results > > =============================== > > > > Bandwidth Test > > -------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 > > > > ? ? ? ? ? Interval ? ? ? ?Transfer ? ? ? ?Bandwidth > > ? ? ? ? ? sec ? ? ? ? ? ? MBytes ? ? ? ? ?Mbits/sec > > FreeBSD: ? 0.0-10.3 ? ? ? ?740 ? ? ? ? ? ? 600 > > Linux: ? ? 0.0-10.0 ? ? ? ?972 ? ? ? ? ? ? 815 > > > > > > PC1/3: #iperf -c 192.168.1.20 -d > > > > ? ? ? ? ? Interval ? ? ? ?Transfer ? ? ? ?Bandwidth > > ? ? ? ? ? sec ? ? ? ? ? ? MBytes ? ? ? ? ?Mbits/sec > > FreeBSD: ? 0.0-10.0 ? ? ? ?402 ? ? ? ? ? ? 337 > > ? ? ? ? ? 0.0-10.0 ? ? ? ?404 ? ? ? ? ? ? 338 > > > > Linux: ? ? 0.0-10.0 ? ? ? ?926 ? ? ? ? ? ? 776 > > ? ? ? ? ? 0.0-10.0 ? ? ? ?44.1 ? ? ? ? ? ?36.9 > > > > > > Parallel Test > > ------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 -P 2 > > > > ? ? ? ? ? Interval ? ? ? ?Transfer ? ? ? ?Bandwidth > > ? ? ? ? ? sec ? ? ? ? ? ? MBytes ? ? ? ? ?Mbits/sec > > FreeBSD: ? 0.0-10.3 ? ? ? ?370 ? ? ? ? ? ? 300 > > ? ? ? ? ? 0.0-10.3 ? ? ? ?370 ? ? ? ? ? ? 300 > > ? ? ?SUM: 0.0-10.3 ? ? ? ?739 ? ? ? ? ? ? 600 > > > > Linux: ? ? 0.0-10.0 ? ? ? ?479 ? ? ? ? ? ? 402 > > ? ? ? ? ? 0.0-10.0 ? ? ? ?473 ? ? ? ? ? ? 396 > > ? ? ?SUM: 0.0-10.0 ? ? ? ?952 ? ? ? ? ? ? 797 > > > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From brent at servuhome.net Sun Oct 18 20:35:30 2009 From: brent at servuhome.net (Brent Jones) Date: Sun Oct 18 20:35:37 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: On Sun, Oct 18, 2009 at 7:55 AM, Adrian Chadd wrote: > FYI, I installed netperf on my local p4-D test boxes that I use for > other testing. > > 128 byte send/receive buffers on the client side: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv ? Send ? ?Send > Socket Socket ?Message ?Elapsed > Size ? Size ? ?Size ? ? Time ? ? Throughput > bytes ?bytes ? bytes ? ?secs. ? ?10^6bits/sec > > ?8192 ? ?128 ? ?128 ? ?10.00 ? ? 426.17 > > 1kbyte send/receive buffers: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv ? Send ? ?Send > Socket Socket ?Message ?Elapsed > Size ? Size ? ?Size ? ? Time ? ? Throughput > bytes ?bytes ? bytes ? ?secs. ? ?10^6bits/sec > > ?8192 ? 1024 ? 1024 ? ?10.00 ? ? 903.39 > > 8kbyte send/receive buffers: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv ? Send ? ?Send > Socket Socket ?Message ?Elapsed > Size ? Size ? ?Size ? ? Time ? ? Throughput > bytes ?bytes ? bytes ? ?secs. ? ?10^6bits/sec > > ?8192 ? 8192 ? 8192 ? ?10.00 ? ? 913.71 > > Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some > onboard flavour of the intel e1000 NIC: > > ? ?device ? ? = '82573E Intel Corporation 82573E Gigabit Ethernet > Controller (Copper)' > > They are connected via a Cisco 3750G L3 switch. In fact, the traffic > is routed, rather than switched. > > My /etc/sysctl.conf: > > net.inet.icmp.icmplim=0 > net.inet.icmp.icmplim_output=0 > net.inet.tcp.msl=3000 > net.inet.tcp.sendspace=8192 > net.inet.tcp.recvspace=8192 > kern.maxfilesperproc=65536 > kern.maxfiles=262144 > kern.ipc.maxsockets=32768 > kern.ipc.somaxconn=1024 > kern.ipc.nmbclusters=131072 > net.inet.ip.fw.enable=0 > kern.ipc.somaxconn=10240 > > 2c, > > > Adrian > > 2009/10/15 Hongtao Yin : >> Hi, >> >> Can you try with 64K and up tp 1MB buffers? I see ~1Gbit speeds with my FreeBSD boxes using Broadcom NIC's and cheap Netgear switches. I'm not sure how the original tester got such poor numbers, when my setup is relatively low end, and sustaining Gbit speeds is no major feat. -- Brent Jones brent@servuhome.net From adrian at freebsd.org Mon Oct 19 01:36:37 2009 From: adrian at freebsd.org (Adrian Chadd) Date: Mon Oct 19 01:36:44 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: uhm: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 65536 65536 10.00 862.48 1 megabyte socket buffers threw an error. I'll see why later. Now, as for why 64k socket buffers gave a slower result than 8k socket buffers... ah. If I change the sending end to use 64k socket buffers: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 65536 65536 65536 10.00 916.23 Adrian 2009/10/19 Brent Jones : > On Sun, Oct 18, 2009 at 7:55 AM, Adrian Chadd wrote: >> FYI, I installed netperf on my local p4-D test boxes that I use for >> other testing. >> >> 128 byte send/receive buffers on the client side: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv ? Send ? ?Send >> Socket Socket ?Message ?Elapsed >> Size ? Size ? ?Size ? ? Time ? ? Throughput >> bytes ?bytes ? bytes ? ?secs. ? ?10^6bits/sec >> >> ?8192 ? ?128 ? ?128 ? ?10.00 ? ? 426.17 >> >> 1kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv ? Send ? ?Send >> Socket Socket ?Message ?Elapsed >> Size ? Size ? ?Size ? ? Time ? ? Throughput >> bytes ?bytes ? bytes ? ?secs. ? ?10^6bits/sec >> >> ?8192 ? 1024 ? 1024 ? ?10.00 ? ? 903.39 >> >> 8kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv ? Send ? ?Send >> Socket Socket ?Message ?Elapsed >> Size ? Size ? ?Size ? ? Time ? ? Throughput >> bytes ?bytes ? bytes ? ?secs. ? ?10^6bits/sec >> >> ?8192 ? 8192 ? 8192 ? ?10.00 ? ? 913.71 >> >> Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some >> onboard flavour of the intel e1000 NIC: >> >> ? ?device ? ? = '82573E Intel Corporation 82573E Gigabit Ethernet >> Controller (Copper)' >> >> They are connected via a Cisco 3750G L3 switch. In fact, the traffic >> is routed, rather than switched. >> >> My /etc/sysctl.conf: >> >> net.inet.icmp.icmplim=0 >> net.inet.icmp.icmplim_output=0 >> net.inet.tcp.msl=3000 >> net.inet.tcp.sendspace=8192 >> net.inet.tcp.recvspace=8192 >> kern.maxfilesperproc=65536 >> kern.maxfiles=262144 >> kern.ipc.maxsockets=32768 >> kern.ipc.somaxconn=1024 >> kern.ipc.nmbclusters=131072 >> net.inet.ip.fw.enable=0 >> kern.ipc.somaxconn=10240 >> >> 2c, >> >> >> Adrian >> >> 2009/10/15 Hongtao Yin : >>> Hi, >>> >>> > > Can you try with 64K and up tp 1MB buffers? > > I see ~1Gbit speeds with my FreeBSD boxes using Broadcom NIC's and > cheap Netgear switches. > I'm not sure how the original tester got such poor numbers, when my > setup is relatively low end, and sustaining Gbit speeds is no major > feat. > > > > -- > Brent Jones > brent@servuhome.net > From killing at multiplay.co.uk Mon Oct 19 10:34:51 2009 From: killing at multiplay.co.uk (Steven Hartland) Date: Mon Oct 19 10:36:51 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: Try with something like this, which is the standard set we use on our file serving machines. net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 Regards Steve ----- Original Message ----- From: "Adrian Chadd" To: "Brent Jones" Cc: "Hongtao Yin" ; Sent: Monday, October 19, 2009 2:36 AM Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance uhm: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 65536 65536 10.00 862.48 1 megabyte socket buffers threw an error. I'll see why later. Now, as for why 64k socket buffers gave a slower result than 8k socket buffers... ah. If I change the sending end to use 64k socket buffers: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 65536 65536 65536 10.00 916.23 Adrian 2009/10/19 Brent Jones : > On Sun, Oct 18, 2009 at 7:55 AM, Adrian Chadd wrote: >> FYI, I installed netperf on my local p4-D test boxes that I use for >> other testing. >> >> 128 byte send/receive buffers on the client side: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 8192 128 128 10.00 426.17 >> >> 1kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 8192 1024 1024 10.00 903.39 >> >> 8kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 8192 8192 8192 10.00 913.71 >> >> Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some >> onboard flavour of the intel e1000 NIC: >> >> device = '82573E Intel Corporation 82573E Gigabit Ethernet >> Controller (Copper)' >> >> They are connected via a Cisco 3750G L3 switch. In fact, the traffic >> is routed, rather than switched. >> >> My /etc/sysctl.conf: >> >> net.inet.icmp.icmplim=0 >> net.inet.icmp.icmplim_output=0 >> net.inet.tcp.msl=3000 >> net.inet.tcp.sendspace=8192 >> net.inet.tcp.recvspace=8192 >> kern.maxfilesperproc=65536 >> kern.maxfiles=262144 >> kern.ipc.maxsockets=32768 >> kern.ipc.somaxconn=1024 >> kern.ipc.nmbclusters=131072 >> net.inet.ip.fw.enable=0 >> kern.ipc.somaxconn=10240 >> >> 2c, >> >> >> Adrian >> >> 2009/10/15 Hongtao Yin : >>> Hi, >>> >>> > > Can you try with 64K and up tp 1MB buffers? > > I see ~1Gbit speeds with my FreeBSD boxes using Broadcom NIC's and > cheap Netgear switches. > I'm not sure how the original tester got such poor numbers, when my > setup is relatively low end, and sustaining Gbit speeds is no major > feat. > > > > -- > Brent Jones > brent@servuhome.net > _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From ivoras at freebsd.org Mon Oct 19 11:44:00 2009 From: ivoras at freebsd.org (Ivan Voras) Date: Mon Oct 19 11:44:07 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> Message-ID: Steve Dong wrote: > It looks the jpeg attachments were somehow dropped. Trying again with pdf > attachment. Hopefully it works this time. > Hi, I haven't tried comparing this sort of performance with Linux so your conclusion still might be right, but the fact that you couldn't saturate 1 Gbps on either system even with big packets suggests that there might be an external problem - a bad network card or a bad driver for the network card, or a switch whose line discipline is a bit in conflict with the NIC or the driver. I have previously successfully (and rather trivially) saturated 1 Gbps links with Broadcom cards with FreeBSD 7.x, so it *is* possible. From ivoras at freebsd.org Mon Oct 19 11:48:12 2009 From: ivoras at freebsd.org (Ivan Voras) Date: Mon Oct 19 11:48:18 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> <20091016075336.03eb17f2.wmoran@collaborativefusion.com> <000001ca4f3c$78dc3550$6501a8c0@china.huawei.com> <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> Message-ID: Chuck Swiger wrote: > Hi, Steve-- > > On Oct 17, 2009, at 8:14 AM, Steve Dong wrote: >> If there's a better/lighter way to show these graphics, I'd like to know. > > Sure-- put 'em on a webserver somewhere, and put links to them in your > email to this mailing list. > > If you wanted to do even better than that, set up a simple webpage > describing what you are doing in your comparison, have a link to the > dmesg/boot output for each platform as a .txt file and a description of > any system tweaks & tuning, have a link that points to a description of > the test setup (ie, your ASCII diagram of the switch and 4 machines), > then your graphs, then the raw data (or links to it, depending). You > can then throw in netstat -s output, or NIC driver stats from sysctl, or > switch stats, etc-- anything else that adds useful context. > > There are a fair number of posts in the list archives which describe how > to benchmark reliably, and the people who are most likely to be making > code changes to FreeBSD also tend to like to know whether you've > collected enough data, in a controlled fashion, to have an idea as to > whether your measurements are reproducible. I'm not a purist, and I Also, the OP should take a look at some previous benchmarks and the link to benchmark advices here: http://wiki.freebsd.org/BenchmarkMatrix From ivoras at freebsd.org Mon Oct 19 11:50:17 2009 From: ivoras at freebsd.org (Ivan Voras) Date: Mon Oct 19 11:50:25 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: Steven Hartland wrote: > Try with something like this, which is the standard set we use on our > file serving machines. > > net.inet.tcp.inflight.enable=0 > net.inet.tcp.sendspace=65536 > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 16 MB network buffers? What kind of % impact do you see from them? From killing at multiplay.co.uk Mon Oct 19 13:41:05 2009 From: killing at multiplay.co.uk (Steven Hartland) Date: Mon Oct 19 13:41:11 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: Its max not default, so relies on your configuring each app you want to have high performance to take advantage of it. In our case that means our large transfers easily saturate Gig. Regards Steve ----- Original Message ----- From: "Ivan Voras" To: Sent: Monday, October 19, 2009 12:44 PM Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance > Steven Hartland wrote: >> Try with something like this, which is the standard set we use on our >> file serving machines. >> >> net.inet.tcp.inflight.enable=0 >> net.inet.tcp.sendspace=65536 >> kern.ipc.maxsockbuf=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_max=16777216 > > 16 MB network buffers? What kind of % impact do you see from them? ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From leccine at gmail.com Tue Oct 20 00:17:01 2009 From: leccine at gmail.com (=?UTF-8?Q?Istv=C3=A1n?=) Date: Tue Oct 20 00:17:07 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: On Mon, Oct 19, 2009 at 2:36 AM, Adrian Chadd wrote: > uhm: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 8192 65536 65536 10.00 862.48 > > 1 megabyte socket buffers threw an error. I'll see why later. > > Now, as for why 64k socket buffers gave a slower result than 8k socket > buffers... ah. If I change the sending end to use 64k socket buffers: > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 65536 65536 65536 10.00 916.23 > > > > Adrian > > therefore i like netpipe runs you can see the performance and the latency as well using the packet size as your "x" axis, i think it makes more sense then just 1 number -- the sun shines for all http://l1xl1x.blogspot.com From adrian at freebsd.org Tue Oct 20 02:39:54 2009 From: adrian at freebsd.org (Adrian Chadd) Date: Tue Oct 20 02:40:55 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: 2009/10/20 Istv?n : > therefore i like netpipe runs you can see the performance and the latency as > well using the packet size as your "x" axis, i think it makes more sense > then just 1 number My point was to demonstrate that saturating gigabit ethernet is very doable with FreeBSD, and his limitation is more likely somewhere other than "TCP". I've told him privately to check CPU utilisation. I'll do the same on my boxes when I get some time; I'd like to know why I'm only seeing ~ 800mbit with large buffers. From pathiaki2 at yahoo.com Tue Oct 20 07:05:39 2009 From: pathiaki2 at yahoo.com (Paul Pathiakis) Date: Tue Oct 20 07:05:45 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: <687096.90163.qm@web110506.mail.gq1.yahoo.com> Hi, going to chime in on this one....just trying to help. There's some simple things to get Gb, jumbo frames (MTU > 1500 on both the switch port and the card) is a simple way. However, I'd have to read back on this thread as I haven't had time of late. Basically, and I've seen this on many, many Gb cards, chipsets and Drives make the world of difference. I tried for a few days to try and get an HP DL360 with it's dual on-board Broadcom bge NIC to get to 1 Gb.... just plain no way. If anyone has settings for that, I'd like to know them. Also, this is the same chip set that a lot of vendors use and it is cheap and inexpensive. When I couldn't get the thing to go beyond 720Mb, I tried something simple. I ordered an Intel dual Gb port card and put that in. WITHOUT tuning, this thing started at almost 800 Mb throughput and I almost got it to 850 Mb within a few hours. I wish I could send those settings to this list but it was well over a year ago that I did this. Sadly, most large vendors start with Broadcom chipsets and don't want to spent the extra $10 for the Intel chipset. (No, I am not a fan boy of Intel, more of AMD if anything, but their NICs rock.) P. ________________________________ From: Adrian Chadd To: Istv?n Cc: Hongtao Yin ; freebsd-performance@freebsd.org; Brent Jones Sent: Mon, October 19, 2009 10:39:53 PM Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance 2009/10/20 Istv?n : > therefore i like netpipe runs you can see the performance and the latency as > well using the packet size as your "x" axis, i think it makes more sense > then just 1 number My point was to demonstrate that saturating gigabit ethernet is very doable with FreeBSD, and his limitation is more likely somewhere other than "TCP". I've told him privately to check CPU utilisation. I'll do the same on my boxes when I get some time; I'd like to know why I'm only seeing ~ 800mbit with large buffers. _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From leccine at gmail.com Tue Oct 20 08:13:07 2009 From: leccine at gmail.com (=?UTF-8?Q?Istv=C3=A1n?=) Date: Tue Oct 20 08:13:14 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: i see but there was no debt that it is possible. at least from my side :_) On Tue, Oct 20, 2009 at 3:39 AM, Adrian Chadd wrote: > 2009/10/20 Istv?n : > > > > therefore i like netpipe runs you can see the performance and the latency > as > > well using the packet size as your "x" axis, i think it makes more sense > > then just 1 number > > My point was to demonstrate that saturating gigabit ethernet is very > doable with FreeBSD, and his limitation is more likely somewhere other > than "TCP". > > I've told him privately to check CPU utilisation. I'll do the same on > my boxes when I get some time; I'd like to know why I'm only seeing ~ > 800mbit with large buffers. > -- the sun shines for all http://l1xl1x.blogspot.com From leccine at gmail.com Tue Oct 20 08:14:30 2009 From: leccine at gmail.com (=?UTF-8?Q?Istv=C3=A1n?=) Date: Tue Oct 20 08:14:36 2009 Subject: Comparison of FreeBSD/Linux TCP Throughput performance In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Message-ID: i mean doubt, shouldn't do mailing before the first coffee :))) On Tue, Oct 20, 2009 at 9:13 AM, Istv?n wrote: > i see > but there was no debt that it is possible. at least from my side :_) > > > On Tue, Oct 20, 2009 at 3:39 AM, Adrian Chadd wrote: > >> 2009/10/20 Istv?n : >> >> >> > therefore i like netpipe runs you can see the performance and the >> latency as >> > well using the packet size as your "x" axis, i think it makes more sense >> > then just 1 number >> >> My point was to demonstrate that saturating gigabit ethernet is very >> doable with FreeBSD, and his limitation is more likely somewhere other >> than "TCP". >> >> I've told him privately to check CPU utilisation. I'll do the same on >> my boxes when I get some time; I'd like to know why I'm only seeing ~ >> 800mbit with large buffers. >> > > > > -- > the sun shines for all > > http://l1xl1x.blogspot.com > -- the sun shines for all http://l1xl1x.blogspot.com