From vadim_nuclight at mail.ru Wed Dec 3 03:10:12 2008 From: vadim_nuclight at mail.ru (Vadim Goncharov) Date: Wed Dec 3 03:10:20 2008 Subject: hwpmc granularity and 6.4 network performance References: Message-ID: Hi Adrian Chadd! On Tue, 25 Nov 2008 15:09:19 -0500; Adrian Chadd wrote about 'Re: hwpmc granularity and 6.4 network performance': > * Since you've changed two things - hwpmc _AND_ the kernel version - > you can't easily conclude which one (if any!) has any influence on > Giant showing up in your top output. I suggest recompiling without > hwpmc and seeing if the behaviour changes. This is not so easy to do at the time when I want :) I will check this some weeks later, may be. > * The gprof utility expects something resembling "time" for the > sampling data, but pmcstat doesn't record time, it records "events". > The counts you see in gprof are "events", so change "seconds" to > "events" in your reading of the gprof output. Of course, I know this, but it doesn't change the percentage. > * I don't know if the backported pmc to 6.4 handles stack call graphs > or not. Easy way to check - pmcstat -R sample.out | more ; see if you > just see "sample" lines or "sample" and "callgraph" lines. No. > * I bet that ipfw_chk is a big enough hint. How big is your ipfw ruleset? :) It's not so big in terms of rule count and not so big in terms of exact hint, but it is of course big as a CPU hog :) router# ipfw show | wc -l 70 Surely, not so much, yes? So I want to see which parts are more CPU-intensive, to use as a hint when rewriting ruleset. I've heard about a pmcannotate tool, in -arch@, and I think that it is tool which does the thing exactly what I want, but that requires patch for pmcstat which didn't apply on my 6.4, too much was different :( >> OK, I can conclude from this that I should optimize my ipfw ruleset, but >> that's all. I know from sources that ipfw_chk() is a big function with a >> bunch of 'case's in a large 'switch'. I want to know which parts of that >> switch are executed more often. It says in listing that granularity is >> 4 bytes, I assume that it has a sample for each of 4-byte chunks of binary >> code, so that it must have such information. My kernel is compiled with: >> >> makeoptions DEBUG=-g >> >> so kgdb does know where are instructions for each line of source code. >> How can I obtain this info from profiling? It also would be useful to know >> which places do calls to that bcmp() and rn_match(). -- WBR, Vadim Goncharov. ICQ#166852181 mailto:vadim_nuclight@mail.ru [Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight] From vadim_nuclight at mail.ru Wed Dec 3 03:20:59 2008 From: vadim_nuclight at mail.ru (Vadim Goncharov) Date: Wed Dec 3 03:21:10 2008 Subject: [PATCH] pmcannotate tool References: <3bbf2fe10811230502t3cc52809i6ac91082f780b730@mail.gmail.com> Message-ID: Hi Attilio Rao! On Sun, 23 Nov 2008 14:02:22 +0100; Attilio Rao wrote about '[PATCH] pmcannotate tool': > pmcannotate is a tool that prints out sources of a tool (in C or > assembly) with inlined profiling informations retrieved by a prior > pmcstat analysis. > If compared with things like callgraph generation, it prints out > profiling on a per-instance basis and this can be useful to find, for > example, badly handled caches, too high latency instructions, etc. [...] > objdump is not the only one tool on which pmcannotare rely. Infact, in > order to have it working, pmcstat needs to be present too because we > need to retrieve, from the pmcstat raw output, informations about the > sampled PCs (in particular the name of the function they live within, > its start and ending addresses). As long as currently pmcstat doesn't > return those informations, a new option has been added to the tool > (-m) which can extract (from a raw pmcstat output) all pc sampled, > name of the functions and symbol bundaries they live within. [...] > The patch can be found here: > http://www.freebsd.org/~attilio/pmcannotate.diff/ > where pmcannotate/ dir contains the code and needs to go under > /usr/src/usr.sbin/ and the patch has diffs against pmcstat and > Makefile. It looks like this the thing I need, please see parallel thread "hwpmc granularity and 6.4 network performance", I have CPU hog ipfw_chk() which is 1200+ lines long and want to know which parts are most hogs. But that part of your patch which is for pmcstat isn't applying to 6.4 :( Could you please provide another for this ver? -- WBR, Vadim Goncharov. ICQ#166852181 mailto:vadim_nuclight@mail.ru [Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight] From raykinsella78 at gmail.com Fri Dec 5 07:01:10 2008 From: raykinsella78 at gmail.com (Ray Kinsella) Date: Fri Dec 5 07:01:17 2008 Subject: Thread priority in FreeBSD Message-ID: <584ec6bb0812050701t4bcccb1fr214370f4535ae1d0@mail.gmail.com> Hi all, I have a problem trying to influence thread scheduling in FreeBSD. There are three threads I am interested the priority of:- a. Enqueue Thread: - Enqueues data in a circular buffer b. Interrupt Handler: - Signals that data in the circular buffer has been processed and in ready for removal - Queues an entry on a task queue so the data is removed from the circular buffer asynchronousily - Data is processed by a pci card c. Task Queue Thread: - Dequeues request from a task queue. - Dequeues data from the circular buffer. *Objective: *I want to prioritise the *Enqueue Thread, *such that it will always executes until the circular buffer is exhausted of space. I want the *Task Queue Thread* to have the lowest priority such that it only runs when the *Enqueue Thread* has no work to do. *Observed behaviour: *I set the *Enqueue Thread, *to have a priority of PRI_MAX_KERN (0) I set the *Task Queue Thread,* to have a priority of PRI_MIN_KERN (64) In the main I see the following scheduling:- ... *Enqueue Thread, *Enqueues serveral requests on the circular buffer *Interrupt Handler*, Interrupts the *Enqueue Thread, *puts requests on a taskqueue for async processing. *Task Queue Thread, *Dequeues the request from a taskqueue and the data from the circular buffer *Enqueue Thread, *Enqueues serveral requests on the circular buffer *Interrupt Handler*, Interrupts the *Enqueue Thread**, *puts requests on a taskqueue for async processing. *Task Queue Thread, *Dequeues the request from a taskqueue and the data from the circular buffer .. No matter what I do with thread priorities, the *Task Queue Thread *always follows the *Interrupt Handler. * *Ideal behaviour* Ideally I would get the following behaviour, were *Enqueue Thread *will always run instead of *Task Queue Thread *thread while it has work to do. *Enqueue Thread, *Enqueues serveral requests on the circular buffer *Interrupt Handler*, Interrupts the *Enqueue Thread, *puts requests on a taskqueue for async processing. *Enqueue Thread, *Enqueues serveral requests on the circular buffer *Interrupt Handler*, Interrupts the *Enqueue Thread**, *puts requests on a taskqueue for async processing. *Enqueue Thread, *Enqueues serveral requests on the circular buffer *Interrupt Handler*, Interrupts the *Enqueue Thread**, *puts requests on a taskqueue for async processing. *Enqueue Thread, *yields timeslices as the circular buffer is maxed out *Task Queue Thread, *Dequeues the request from a taskqueue and the data from the circular buffer *Task Queue Thread, *Dequeues the request from a taskqueue and the data from the circular buffer *Task Queue Thread, *Dequeues the request from a taskqueue and the data from the circular buffer Any idea's how to encourage the scheduler to adopt this behaviour ? Thanks Ray Kinsella * * From raykinsella78 at gmail.com Fri Dec 5 07:02:26 2008 From: raykinsella78 at gmail.com (Ray Kinsella) Date: Fri Dec 5 07:02:33 2008 Subject: Thread priority in FreeBSD In-Reply-To: <584ec6bb0812050701t4bcccb1fr214370f4535ae1d0@mail.gmail.com> References: <584ec6bb0812050701t4bcccb1fr214370f4535ae1d0@mail.gmail.com> Message-ID: <584ec6bb0812050702j7dbf5b54k87d2cb4e4f6fa8b4@mail.gmail.com> Apologies, I neglected to clarify, I am of course talking about Kernel Threads. On Fri, Dec 5, 2008 at 3:01 PM, Ray Kinsella wrote: > Hi all, > > I have a problem trying to influence thread scheduling in FreeBSD. > There are three threads I am interested the priority of:- > > a. Enqueue Thread: > - Enqueues data in a circular buffer > > b. Interrupt Handler: > - Signals that data in the circular buffer has been processed and in ready > for removal > - Queues an entry on a task queue so the data is removed from the circular > buffer asynchronousily > - Data is processed by a pci card > > c. Task Queue Thread: > - Dequeues request from a task queue. > - Dequeues data from the circular buffer. > > *Objective: > > *I want to prioritise the *Enqueue Thread, *such that it will always > executes until the circular buffer is exhausted of space. > I want the *Task Queue Thread* to have the lowest priority such that it > only runs when the *Enqueue Thread* has no work to do. > > *Observed behaviour: > > *I set the *Enqueue Thread, *to have a priority of PRI_MAX_KERN (0) > I set the *Task Queue Thread,* to have a priority of PRI_MIN_KERN (64) > > In the main I see the following scheduling:- > > ... > *Enqueue Thread, *Enqueues serveral requests on the circular buffer > *Interrupt Handler*, Interrupts the *Enqueue Thread, *puts requests on a > taskqueue for async processing. > *Task Queue Thread, *Dequeues the request from a taskqueue and the data > from the circular buffer > *Enqueue Thread, *Enqueues serveral requests on the circular buffer > *Interrupt Handler*, Interrupts the *Enqueue Thread**, *puts requests on a > taskqueue for async processing. > *Task Queue Thread, *Dequeues the request from a taskqueue and the data > from the circular buffer > .. > > No matter what I do with thread priorities, the *Task Queue Thread *always > follows the *Interrupt Handler. > * > *Ideal behaviour* > > Ideally I would get the following behaviour, were *Enqueue Thread *will > always run instead > of *Task Queue Thread *thread while it has work to do. > > *Enqueue Thread, *Enqueues serveral requests on the circular buffer > *Interrupt Handler*, Interrupts the *Enqueue Thread, *puts requests on a > taskqueue for async processing. > *Enqueue Thread, *Enqueues serveral requests on the circular buffer > *Interrupt Handler*, Interrupts the *Enqueue Thread**, *puts requests on a > taskqueue for async processing. > *Enqueue Thread, *Enqueues serveral requests on the circular buffer > *Interrupt Handler*, Interrupts the *Enqueue Thread**, *puts requests on a > taskqueue for async processing. > *Enqueue Thread, *yields timeslices as the circular buffer is maxed out > *Task Queue Thread, *Dequeues the request from a taskqueue and the data > from the circular buffer > *Task Queue Thread, *Dequeues the request from a taskqueue and the data > from the circular buffer > *Task Queue Thread, *Dequeues the request from a taskqueue and the data > from the circular buffer > > Any idea's how to encourage the scheduler to adopt this behaviour ? > > Thanks > > Ray Kinsella > > * > > > * > From dudu at dudu.ro Wed Dec 10 07:26:40 2008 From: dudu at dudu.ro (Vlad GALU) Date: Wed Dec 10 07:26:47 2008 Subject: bce(4) and rx errors Message-ID: Hello. Sorry for crossposting, but I wasn't sure which mailing list was the most appropriate for this email. I have an application pulling about 220Kpps from a bce(4) card (details below). At what seems to be random times, errors start showing up on that interface (I'm watching it with netstat -w1 -I), so about 10% of the initial 220Kpps is reported as errors. Bringing the interface down and then back up clears the errors, but they do reappear at a later time. Before they reappear, the systems manages to pull the full 220Kpps as before. This is a temporary setup, we'll very soon use an Intel fiber card, but I thought this issue was worth mentioning, as I don't think it's a hardware problem (the switch also reports no errors). The system is running a fresh (yesterday's) RELENG_7. The card is onboard, on a HP DL380 G5. Here's the pciconf output: -- cut here -- pcib13@pci0:2:0:0: class=0x060400 card=0x00000000 chip=0x01031166 rev=0xc3 hdr=0x01 vendor = 'ServerWorks (Was: Reliance Computer Corp)' device = 'BCM5715 Broadcom dual gigabit, pci bridge' class = bridge subclass = PCI-PCI bce0@pci0:3:0:0: class=0x020000 card=0x7038103c chip=0x164c14e4 rev=0x12 hdr=0x00 vendor = 'Broadcom Corporation' device = '5708C Broadcom NetXtreme II Gigabit Ethernet Adapter' class = network subclass = ethernet -- and here -- Regards, Vlad -- ~/.signature: no such file or directory From pathiaki2 at yahoo.com Thu Dec 18 17:31:20 2008 From: pathiaki2 at yahoo.com (Paul Patterson) Date: Thu Dec 18 17:31:27 2008 Subject: ZFS, NFS and Network tuning Message-ID: <553453.43874.qm@web110514.mail.gq1.yahoo.com> Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul From pathiaki2 at yahoo.com Fri Dec 19 06:48:01 2008 From: pathiaki2 at yahoo.com (Paul Patterson) Date: Fri Dec 19 06:48:17 2008 Subject: ZFS, NFS and Network tuning References: <553453.43874.qm@web110514.mail.gq1.yahoo.com> Message-ID: <15723.22980.qm@web110511.mail.gq1.yahoo.com> Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From pathiaki2 at yahoo.com Fri Dec 19 10:03:16 2008 From: pathiaki2 at yahoo.com (Paul Patterson) Date: Fri Dec 19 10:03:22 2008 Subject: ZFS, NFS and Network tuning References: <553453.43874.qm@web110514.mail.gq1.yahoo.com> <15723.22980.qm@web110511.mail.gq1.yahoo.com> Message-ID: <400826.77992.qm@web110510.mail.gq1.yahoo.com> Hello all, I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 9:47:59 AM Subject: Re: ZFS, NFS and Network tuning Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From pathiaki2 at yahoo.com Fri Dec 19 10:59:55 2008 From: pathiaki2 at yahoo.com (Paul Patterson) Date: Fri Dec 19 11:00:01 2008 Subject: ZFS, NFS and Network tuning References: <553453.43874.qm@web110514.mail.gq1.yahoo.com> <15723.22980.qm@web110511.mail.gq1.yahoo.com> <400826.77992.qm@web110510.mail.gq1.yahoo.com> Message-ID: <309927.87042.qm@web110514.mail.gq1.yahoo.com> Hi, Well, I got some input on things: kern.ipc.somaxconn=32768 net.inet.tcp.mssdflt=1460 And for fstab rw,tcp,intr,noatime,nfsv3,-w=65536,-r=65536 I tried turning on polling with ifconfig bce0 polling, however, I didn't see it in ifconfig bce0 so I don't believe it to be active or the card doesn't support it. aI also removed async from the mounts. These had a detrimental affect on the FreeBSD server. I now get 64K per transfer (system -vm) but I'm still only getting about 4MB/sec on the disks and their utilization has dropped to about 5%. Throughput from both clients is ~8.5MB/sec. The tests were run separately. The NetAPP on each host was over 48.5 MB/sec. The FreeBSD host still has about 2 GB free. Paul ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 1:03:14 PM Subject: Re: ZFS, NFS and Network tuning Hello all, I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 9:47:59 AM Subject: Re: ZFS, NFS and Network tuning Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From mike at sentex.net Fri Dec 19 14:14:48 2008 From: mike at sentex.net (Mike Tancsa) Date: Fri Dec 19 14:14:56 2008 Subject: intel i7 and Hyperthreading Message-ID: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Just got our first board to play around with and unlike in the past, having hyperthreading enabled seems to help performance.... At least in buildworld tests. doing a make -j4 vs -j6 make -j8 vs -j10 gives -j buildworld time % improvement over -j4 4 13:57 6 12:11 13% 8 11:32 18% 10 11:43 17% dmesg below of the hardware... The CPU seems to run fairly cool, but the board has a lot of nasty hot heatsinks eg. running 8 burnP6 procs 0[ns3c]# sysctl -a | grep temperature dev.cpu.0.temperature: 67 dev.cpu.1.temperature: 67 dev.cpu.2.temperature: 65 dev.cpu.3.temperature: 65 dev.cpu.4.temperature: 66 dev.cpu.5.temperature: 66 dev.cpu.6.temperature: 64 dev.cpu.7.temperature: 64 0[ns3c]# vs idle dev.cpu.0.temperature: 46 dev.cpu.1.temperature: 46 dev.cpu.2.temperature: 42 dev.cpu.3.temperature: 42 dev.cpu.4.temperature: 44 dev.cpu.5.temperature: 44 dev.cpu.6.temperature: 40 dev.cpu.7.temperature: 40 Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-PRERELEASE #0: Fri Dec 19 19:48:15 EST 2008 mdtancsa@ns3c.recycle.net:/usr/obj/usr/src/sys/recycle Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2666.78-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x106a4 Stepping = 4 Features=0xbfebfbff Features2=0x98e3bd AMD Features=0x28100000 AMD Features2=0x1 Cores per package: 8 Logical CPUs per core: 2 real memory = 2138992640 (2039 MB) avail memory = 2084880384 (1988 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ACPI Warning (tbfadt-0505): Optional field "Pm2ControlBlock" has zero address or length: 0 450/0 [20070320] ioapic0 irqs 0-23 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 3.0 on pci0 pci2: on pcib2 3ware device driver for 9000 series storage controllers, version: 3.70.05.001 twa0: <3ware 9000 series Storage Controller> port 0x3000-0x30ff mem 0xf4000000-0xf5ffffff,0xe4300000-0xe4300fff irq 16 at device 0.0 on pci2 twa0: [ITHREAD] twa0: INFO: (0x04: 0x003C): Initialize paused: unit=0 twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-2LP, 2 ports, Firmware FE9X 4.06.00.004, BIOS BE9X 4.05.00.015 pcib3: irq 16 at device 7.0 on pci0 pci3: on pcib3 pci0: at device 16.0 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) pci0: at device 20.3 (no driver attached) em0: port 0x40e0-0x40ff mem 0xe4400000-0xe441ffff,0xe4422000-0xe4422fff irq 20 at device 25.0 on pci0 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1c:c0:92:d5:83 uhci0: port 0x40c0-0x40df irq 16 at device 26.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x40a0-0x40bf irq 21 at device 26.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4080-0x409f irq 19 at device 26.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered ehci0: mem 0xe4421000-0xe44213ff irq 18 at device 26.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: on ehci0 usb3: USB revision 2.0 uhub3: on usb3 uhub3: 6 ports with 6 removable, self powered pci0: at device 27.0 (no driver attached) pcib4: irq 17 at device 28.0 on pci0 pci4: on pcib4 pcib5: irq 16 at device 28.1 on pci0 pci5: on pcib5 bge0: mem 0xe4200000-0xe420ffff irq 17 at device 0.0 on pci5 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:10:18:14:15:12 bge0: [ITHREAD] pcib6: irq 17 at device 28.4 on pci0 pci6: on pcib6 atapci0: port 0x2018-0x201f,0x2024-0x2027,0x2010-0x2017,0x2020-0x2023,0x2000-0x200f mem 0xe4100000-0xe41003ff irq 16 at device 0.0 on pci6 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] uhci3: port 0x4060-0x407f irq 23 at device 29.0 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb4: on uhci3 usb4: USB revision 1.0 uhub4: on usb4 uhub4: 2 ports with 2 removable, self powered uhci4: port 0x4040-0x405f irq 19 at device 29.1 on pci0 uhci4: [GIANT-LOCKED] uhci4: [ITHREAD] usb5: on uhci4 usb5: USB revision 1.0 uhub5: on usb5 uhub5: 2 ports with 2 removable, self powered uhci5: port 0x4020-0x403f irq 18 at device 29.2 on pci0 uhci5: [GIANT-LOCKED] uhci5: [ITHREAD] usb6: on uhci5 usb6: USB revision 1.0 uhub6: on usb6 uhub6: 2 ports with 2 removable, self powered ehci1: mem 0xe4420000-0xe44203ff irq 23 at device 29.7 on pci0 ehci1: [GIANT-LOCKED] ehci1: [ITHREAD] usb7: EHCI version 1.0 usb7: companion controllers, 2 ports each: usb4 usb5 usb6 usb7: on ehci1 usb7: USB revision 2.0 uhub7: on usb7 uhub7: 6 ports with 6 removable, self powered pcib7: at device 30.0 on pci0 pci7: on pcib7 vgapci0: port 0x1000-0x10ff mem 0xe2000000-0xe3ffffff,0xe0000000-0xe1ffffff irq 18 at device 2.0 on pci7 fwohci0: mem 0xe4004000-0xe40047ff,0xe4000000-0xe4003fff irq 19 at device 3.0 on pci7 fwohci0: [FILTER] fwohci0: OHCI version 1.10 (ROM=0) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:90:27:00:02:39:70:e3 fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:90:27:39:70:e3 fwe0: Ethernet address: 02:90:27:39:70:e3 fwip0: on firewire0 fwip0: Firewire address: 00:90:27:00:02:39:70:e3 @ 0xfffe00000000, S400, maxrec 2048 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x102c000 sbp0: on firewire0 fwohci0: Initiate bus reset fwohci0: BUS reset fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode isab0: at device 31.0 on pci0 isa0: on isab0 atapci1: port 0x4158-0x415f,0x416c-0x416f,0x4150-0x4157,0x4168-0x416b,0x4130-0x413f,0x4120-0x412f irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] pci0: at device 31.3 (no driver attached) atapci2: port 0x4148-0x414f,0x4164-0x4167,0x4140-0x4147,0x4160-0x4163,0x4110-0x411f,0x4100-0x410f irq 18 at device 31.5 on pci0 atapci2: [ITHREAD] ata6: on atapci2 ata6: [ITHREAD] ata7: on atapci2 ata7: [ITHREAD] cpu0: on acpi0 est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 p4tcc1: on cpu1 cpu2: on acpi0 est2: on cpu2 p4tcc2: on cpu2 cpu3: on acpi0 est3: on cpu3 p4tcc3: on cpu3 cpu4: on acpi0 est4: on cpu4 p4tcc4: on cpu4 cpu5: on acpi0 est5: on cpu5 p4tcc5: on cpu5 cpu6: on acpi0 est6: on cpu6 p4tcc6: on cpu6 cpu7: on acpi0 est7: on cpu7 p4tcc7: on cpu7 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcafff,0xcb000-0xccfff,0xcd800-0xce7ff pnpid ORM0000 on isa0 ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata0: [ITHREAD] ata1 at port 0x170-0x177,0x376 irq 15 on isa0 ata1: [ITHREAD] atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me) firewire0: bus manager 0 (me) ad8: 76319MB at ata4-master SATA150 da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 953664MB (1953103872 512 byte sectors: 255H 63S/T 121575C) lapic3: Forcing LINT1 to edge trigger SMP: AP CPU #3 Launched! lapic1: Forcing LINT1 to edge trigger SMP: AP CPU #1 Launched! lapic4: Forcing LINT1 to edge trigger SMP: AP CPU #4 Launched! lapic6: Forcing LINT1 to edge trigger SMP: AP CPU #6 Launched! lapic5: Forcing LINT1 to edge trigger SMP: AP CPU #5 Launched! lapic2: Forcing LINT1 to edge trigger SMP: AP CPU #2 Launched! lapic7: Forcing LINT1 to edge trigger SMP: AP CPU #7 Launched! ukbd0: on uhub4 kbd2 at ukbd0 uhid0: on uhub4 Trying to mount root from ufs:/dev/da0s1a twa0: INFO: (0x04: 0x000C): Initialize started: unit=0 ukbd0: at uhub4 port 1 (addr 2) disconnected ukbd0: detached uhid0: at uhub4 port 1 (addr 2) disconnected uhid0: detached ukbd0: on uhub4 kbd2 at ukbd0 uhid0: on uhub4 -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike From michelle_li_001 at yahoo.com Sat Dec 20 16:51:46 2008 From: michelle_li_001 at yahoo.com (Michelle Li) Date: Sat Dec 20 16:51:54 2008 Subject: ZFS, NFS and Network tuning (Paul Patterson) In-Reply-To: <20081220120019.BC51510656F4@hub.freebsd.org> Message-ID: <722609.11236.qm@web65412.mail.ac4.yahoo.com> ...and the dmesg? please post freebsd-performance-request@freebsd.org wrote: Send freebsd-performance mailing list submissions to freebsd-performance@freebsd.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.freebsd.org/mailman/listinfo/freebsd-performance or, via email, send a message with subject or body 'help' to freebsd-performance-request@freebsd.org You can reach the person managing the list at freebsd-performance-owner@freebsd.org When replying, please edit your Subject line so it is more specific than "Re: Contents of freebsd-performance digest..." Today's Topics: 1. Re: ZFS, NFS and Network tuning (Paul Patterson) 2. Re: ZFS, NFS and Network tuning (Paul Patterson) 3. Re: ZFS, NFS and Network tuning (Paul Patterson) 4. intel i7 and Hyperthreading (Mike Tancsa) ---------------------------------------------------------------------- Message: 1 Date: Fri, 19 Dec 2008 06:47:59 -0800 (PST) From: Paul Patterson Subject: Re: ZFS, NFS and Network tuning To: Paul Patterson , freebsd-performance@freebsd.org Message-ID: <15723.22980.qm@web110511.mail.gq1.yahoo.com> Content-Type: text/plain; charset=us-ascii Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ------------------------------ Message: 2 Date: Fri, 19 Dec 2008 10:03:14 -0800 (PST) From: Paul Patterson Subject: Re: ZFS, NFS and Network tuning To: Paul Patterson , freebsd-performance@freebsd.org Message-ID: <400826.77992.qm@web110510.mail.gq1.yahoo.com> Content-Type: text/plain; charset=us-ascii Hello all, I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 9:47:59 AM Subject: Re: ZFS, NFS and Network tuning Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ------------------------------ Message: 3 Date: Fri, 19 Dec 2008 10:59:54 -0800 (PST) From: Paul Patterson Subject: Re: ZFS, NFS and Network tuning To: Paul Patterson , freebsd-performance@freebsd.org Message-ID: <309927.87042.qm@web110514.mail.gq1.yahoo.com> Content-Type: text/plain; charset=us-ascii Hi, Well, I got some input on things: kern.ipc.somaxconn=32768 net.inet.tcp.mssdflt=1460 And for fstab rw,tcp,intr,noatime,nfsv3,-w=65536,-r=65536 I tried turning on polling with ifconfig bce0 polling, however, I didn't see it in ifconfig bce0 so I don't believe it to be active or the card doesn't support it. aI also removed async from the mounts. These had a detrimental affect on the FreeBSD server. I now get 64K per transfer (system -vm) but I'm still only getting about 4MB/sec on the disks and their utilization has dropped to about 5%. Throughput from both clients is ~8.5MB/sec. The tests were run separately. The NetAPP on each host was over 48.5 MB/sec. The FreeBSD host still has about 2 GB free. Paul ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 1:03:14 PM Subject: Re: ZFS, NFS and Network tuning Hello all, I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 9:47:59 AM Subject: Re: ZFS, NFS and Network tuning Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ------------------------------ Message: 4 Date: Fri, 19 Dec 2008 17:01:46 -0500 From: Mike Tancsa Subject: intel i7 and Hyperthreading To: freebsd-performance@freebsd.org Message-ID: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Content-Type: text/plain; charset="us-ascii"; format=flowed Just got our first board to play around with and unlike in the past, having hyperthreading enabled seems to help performance.... At least in buildworld tests. doing a make -j4 vs -j6 make -j8 vs -j10 gives -j buildworld time % improvement over -j4 4 13:57 6 12:11 13% 8 11:32 18% 10 11:43 17% dmesg below of the hardware... The CPU seems to run fairly cool, but the board has a lot of nasty hot heatsinks eg. running 8 burnP6 procs 0[ns3c]# sysctl -a | grep temperature dev.cpu.0.temperature: 67 dev.cpu.1.temperature: 67 dev.cpu.2.temperature: 65 dev.cpu.3.temperature: 65 dev.cpu.4.temperature: 66 dev.cpu.5.temperature: 66 dev.cpu.6.temperature: 64 dev.cpu.7.temperature: 64 0[ns3c]# vs idle dev.cpu.0.temperature: 46 dev.cpu.1.temperature: 46 dev.cpu.2.temperature: 42 dev.cpu.3.temperature: 42 dev.cpu.4.temperature: 44 dev.cpu.5.temperature: 44 dev.cpu.6.temperature: 40 dev.cpu.7.temperature: 40 Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-PRERELEASE #0: Fri Dec 19 19:48:15 EST 2008 mdtancsa@ns3c.recycle.net:/usr/obj/usr/src/sys/recycle Timecounter "i8254" frequency 1193182 Hz quality 0 === message truncated === From james.technew at gmail.com Sun Dec 21 06:58:15 2008 From: james.technew at gmail.com (James Chang) Date: Sun Dec 21 06:58:21 2008 Subject: freebsd-performance Digest, Vol 70, Issue 7 In-Reply-To: <20081221120020.BB89F10657A2@hub.freebsd.org> References: <20081221120020.BB89F10657A2@hub.freebsd.org> Message-ID: Hi, Did you have plan to try another NIC (i.e. INTEL em?) and turn on polling mode? I think you can the following thing 1.use asynchronous I/O on Your FreeBSD 7.1 box 2.enable options ZERO_COPY_SOCKETS Best Regards! James Chang 2008/12/21 : > Send freebsd-performance mailing list submissions to > freebsd-performance@freebsd.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > or, via email, send a message with subject or body 'help' to > freebsd-performance-request@freebsd.org > > You can reach the person managing the list at > freebsd-performance-owner@freebsd.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of freebsd-performance digest..." > > > Today's Topics: > > 1. Re: ZFS, NFS and Network tuning (Paul Patterson) (Michelle Li) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 20 Dec 2008 16:25:02 -0800 (PST) > From: Michelle Li > Subject: Re: ZFS, NFS and Network tuning (Paul Patterson) > To: freebsd-performance@freebsd.org > Message-ID: <722609.11236.qm@web65412.mail.ac4.yahoo.com> > Content-Type: text/plain; charset=iso-8859-1 > > ...and the dmesg? > > please post > > freebsd-performance-request@freebsd.org wrote: Send freebsd-performance mailing list submissions to > freebsd-performance@freebsd.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > or, via email, send a message with subject or body 'help' to > freebsd-performance-request@freebsd.org > > You can reach the person managing the list at > freebsd-performance-owner@freebsd.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of freebsd-performance digest..." > > > Today's Topics: > > 1. Re: ZFS, NFS and Network tuning (Paul Patterson) > 2. Re: ZFS, NFS and Network tuning (Paul Patterson) > 3. Re: ZFS, NFS and Network tuning (Paul Patterson) > 4. intel i7 and Hyperthreading (Mike Tancsa) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 19 Dec 2008 06:47:59 -0800 (PST) > From: Paul Patterson > > Subject: Re: ZFS, NFS and Network tuning > To: Paul Patterson > , > freebsd-performance@freebsd.org > Message-ID: <15723.22980.qm@web110511.mail.gq1.yahoo.com> > Content-Type: text/plain; charset=us-ascii > > Hi, > > as promised, the parameter tuning I have on the box (does anyone see anything wrong?) > > /boot/loader.conf > > kern.hz="100" > vm.kmem_size_max="1536M" > vm.kmem_size="1536M" > vfs.zfs.prefetch_disble=1 > > /etc/sysctl.conf > > kern.ipc.maxsockbuf=16777216 > kern.ipc.nmbclusters=32768 > kern.ipc.somaxconn=8192 > kern.maxfiles=65536 > kern.maxfilesperproc=32768 > kern.mxvnodes=600000 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.inflight.enable=0 > net.inet.tcp.path_mtu_discovery=0 > net.inet.tcp.recvbuf_auto=1 > net.inet.tcp.recvbuf_inc=16384 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvspace=65536 > net.inet.tcp.rfc1323=1 > net.inet.tcp.sendbuf_auto=1 > net.inet.tcpsendbuf_inc=8192 > net.inet.tcp.sendspace=65536 > net.inet.udp.maxdgram=57344 > net.inet.udp.recvspace=65536 > net.local.stream.recvspace=65536 > net.inet.tcp.sendbuf_max=16777216 > > > > > > ________________________________ > From: Paul Patterson > > To: freebsd-performance@freebsd.org > Sent: Thursday, December 18, 2008 8:04:37 PM > Subject: ZFS, NFS and Network tuning > > Hi, > > I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) > > Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. > > I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. > > Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. > > The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. > > Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) > > Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) > > I'll post the NetAPP finding tomorrow as I forgot it for now. > > As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime > > I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. > > Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. > > That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) > > Thank you for reading, > > Paul > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > > > > > ------------------------------ > > Message: 2 > Date: Fri, 19 Dec 2008 10:03:14 -0800 (PST) > From: Paul Patterson > > Subject: Re: ZFS, NFS and Network tuning > To: Paul Patterson > , > freebsd-performance@freebsd.org > Message-ID: <400826.77992.qm@web110510.mail.gq1.yahoo.com> > Content-Type: text/plain; charset=us-ascii > > Hello all, > > I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: > > Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent > Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. > > I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) > > I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. > > P. > > > > > ________________________________ > From: Paul Patterson > > To: Paul Patterson > ; freebsd-performance@freebsd.org > Sent: Friday, December 19, 2008 9:47:59 AM > Subject: Re: ZFS, NFS and Network tuning > > > Hi, > > as promised, the parameter tuning I have on the box (does anyone see anything wrong?) > > /boot/loader.conf > > kern.hz="100" > vm.kmem_size_max="1536M" > vm.kmem_size="1536M" > vfs.zfs.prefetch_disble=1 > > /etc/sysctl.conf > > kern.ipc.maxsockbuf=16777216 > kern.ipc.nmbclusters=32768 > kern.ipc.somaxconn=8192 > kern.maxfiles=65536 > kern.maxfilesperproc=32768 > kern.mxvnodes=600000 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.inflight.enable=0 > net.inet.tcp.path_mtu_discovery=0 > net.inet.tcp.recvbuf_auto=1 > net.inet.tcp.recvbuf_inc=16384 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvspace=65536 > net.inet.tcp.rfc1323=1 > net.inet.tcp.sendbuf_auto=1 > net.inet.tcpsendbuf_inc=8192 > net.inet.tcp.sendspace=65536 > net.inet.udp.maxdgram=57344 > net.inet.udp.recvspace=65536 > net.local.stream.recvspace=65536 > net.inet.tcp.sendbuf_max=16777216 > > > > > > ________________________________ > From: Paul Patterson > > To: freebsd-performance@freebsd.org > Sent: Thursday, December 18, 2008 8:04:37 PM > Subject: ZFS, NFS and Network tuning > > Hi, > > I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) > > Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. > > I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. > > Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. > > The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. > > Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) > > Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) > > I'll post the NetAPP finding tomorrow as I forgot it for now. > > As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime > > I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. > > Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. > > That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) > > Thank you for reading, > > Paul > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > > > > ------------------------------ > > Message: 3 > Date: Fri, 19 Dec 2008 10:59:54 -0800 (PST) > From: Paul Patterson > > Subject: Re: ZFS, NFS and Network tuning > To: Paul Patterson > , > freebsd-performance@freebsd.org > Message-ID: <309927.87042.qm@web110514.mail.gq1.yahoo.com> > Content-Type: text/plain; charset=us-ascii > > Hi, > > Well, I got some input on things: > > kern.ipc.somaxconn=32768 > net.inet.tcp.mssdflt=1460 > > And for fstab > > rw,tcp,intr,noatime,nfsv3,-w=65536,-r=65536 > > I tried turning on polling with ifconfig bce0 polling, however, I didn't see it in ifconfig bce0 so I don't believe it to be active or the card doesn't support it. > > aI also removed async from the mounts. These had a detrimental affect on the FreeBSD server. I now get 64K per transfer (system -vm) but I'm still only getting about 4MB/sec on the disks and their utilization has dropped to about 5%. Throughput from both clients is ~8.5MB/sec. The tests were run separately. The NetAPP on each host was over 48.5 MB/sec. > > The FreeBSD host still has about 2 GB free. > > Paul > > > > > ________________________________ > From: Paul Patterson > > To: Paul Patterson > ; freebsd-performance@freebsd.org > Sent: Friday, December 19, 2008 1:03:14 PM > Subject: Re: ZFS, NFS and Network tuning > > Hello all, > > I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: > > Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent > Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. > > I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) > > I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. > > P. > > > > > ________________________________ > From: Paul Patterson > > To: Paul Patterson > ; freebsd-performance@freebsd.org > Sent: Friday, December 19, 2008 9:47:59 AM > Subject: Re: ZFS, NFS and Network tuning > > > Hi, > > as promised, the parameter tuning I have on the box (does anyone see anything wrong?) > > /boot/loader.conf > > kern.hz="100" > vm.kmem_size_max="1536M" > vm.kmem_size="1536M" > vfs.zfs.prefetch_disble=1 > > /etc/sysctl.conf > > kern.ipc.maxsockbuf=16777216 > kern.ipc.nmbclusters=32768 > kern.ipc.somaxconn=8192 > kern.maxfiles=65536 > kern.maxfilesperproc=32768 > kern.mxvnodes=600000 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.inflight.enable=0 > net.inet.tcp.path_mtu_discovery=0 > net.inet.tcp.recvbuf_auto=1 > net.inet.tcp.recvbuf_inc=16384 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvspace=65536 > net.inet.tcp.rfc1323=1 > net.inet.tcp.sendbuf_auto=1 > net.inet.tcpsendbuf_inc=8192 > net.inet.tcp.sendspace=65536 > net.inet.udp.maxdgram=57344 > net.inet.udp.recvspace=65536 > net.local.stream.recvspace=65536 > net.inet.tcp.sendbuf_max=16777216 > > > > > > ________________________________ > From: Paul Patterson > > To: freebsd-performance@freebsd.org > Sent: Thursday, December 18, 2008 8:04:37 PM > Subject: ZFS, NFS and Network tuning > > Hi, > > I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) > > Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. > > I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. > > Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. > > The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. > > Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) > > Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) > > I'll post the NetAPP finding tomorrow as I forgot it for now. > > As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime > > I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. > > Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. > > That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) > > Thank you for reading, > > Paul > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > > > > > ------------------------------ > > Message: 4 > Date: Fri, 19 Dec 2008 17:01:46 -0500 > From: Mike Tancsa > Subject: intel i7 and Hyperthreading > To: freebsd-performance@freebsd.org > Message-ID: <200812192214.mBJMEj2Q009511@lava.sentex.ca> > Content-Type: text/plain; charset="us-ascii"; format=flowed > > Just got our first board to play around with and unlike in the past, > having hyperthreading enabled seems to help performance.... At least > in buildworld tests. > > doing a make -j4 vs -j6 make -j8 vs -j10 gives > > -j buildworld time % improvement over -j4 > 4 13:57 > 6 12:11 13% > 8 11:32 18% > 10 11:43 17% > > > dmesg below of the hardware... The CPU seems to run fairly cool, but > the board has a lot of nasty hot heatsinks > > eg. running 8 burnP6 procs > > 0[ns3c]# sysctl -a | grep temperature > dev.cpu.0.temperature: 67 > dev.cpu.1.temperature: 67 > dev.cpu.2.temperature: 65 > dev.cpu.3.temperature: 65 > dev.cpu.4.temperature: 66 > dev.cpu.5.temperature: 66 > dev.cpu.6.temperature: 64 > dev.cpu.7.temperature: 64 > 0[ns3c]# > > vs idle > > dev.cpu.0.temperature: 46 > dev.cpu.1.temperature: 46 > dev.cpu.2.temperature: 42 > dev.cpu.3.temperature: 42 > dev.cpu.4.temperature: 44 > dev.cpu.5.temperature: 44 > dev.cpu.6.temperature: 40 > dev.cpu.7.temperature: 40 > > Copyright (c) 1992-2008 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.1-PRERELEASE #0: Fri Dec 19 19:48:15 EST 2008 > mdtancsa@ns3c.recycle.net:/usr/obj/usr/src/sys/recycle > Timecounter "i8254" frequency 1193182 Hz quality 0 > > === message truncated === > > > > > > > > > > > > > > > > > > > ------------------------------ > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > End of freebsd-performance Digest, Vol 70, Issue 7 > ************************************************** > From pathiaki2 at yahoo.com Sun Dec 21 08:17:09 2008 From: pathiaki2 at yahoo.com (Paul Patterson) Date: Sun Dec 21 08:17:16 2008 Subject: ZFS, NFS and Network tuning References: <553453.43874.qm@web110514.mail.gq1.yahoo.com> <15723.22980.qm@web110511.mail.gq1.yahoo.com> <400826.77992.qm@web110510.mail.gq1.yahoo.com> <309927.87042.qm@web110514.mail.gq1.yahoo.com> Message-ID: <732531.19977.qm@web110511.mail.gq1.yahoo.com> Hi, well, after tweaking several things and trying several suggestions, things remain the same. There is an interesting thing occurring is that the steady throughput that I'm experiencing is rock solid. There are no longer any serious deviations on throughput. It's constant at 6.5-9.0 MB/sec (this sucks). However, that's at the write and delete speed. There's very little difference. The NetAPP drops dramatically on a random delete. So, someone suggested going with an em (aka Intel card). I'd really like to try that but the machine is off in a data center but I might just take one of my re based cards if I can't get the corporation to buy one and ship it down there. (I'm just of the opinion that if I could go polling the bce might not show the occasional input error. netstat -w 5 -I bce0 ) The last thing that has been suggested is to go with the Intel card, use polling, and enable "options ZERO_COPY_SOCKETS" in the kernel. That's for Monday. (Thank you, James Chang) As for the reasons many people didn't see the suggestions, a lot of people went off the thread and mailed directly. Please don't. It makes it real hard for the thread have integrity. (Heck, many of you are interested in the results. :-) ) P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 1:59:54 PM Subject: Re: ZFS, NFS and Network tuning Hi, Well, I got some input on things: kern.ipc.somaxconn=32768 net.inet.tcp.mssdflt=1460 And for fstab rw,tcp,intr,noatime,nfsv3,-w=65536,-r=65536 I tried turning on polling with ifconfig bce0 polling, however, I didn't see it in ifconfig bce0 so I don't believe it to be active or the card doesn't support it. aI also removed async from the mounts. These had a detrimental affect on the FreeBSD server. I now get 64K per transfer (system -vm) but I'm still only getting about 4MB/sec on the disks and their utilization has dropped to about 5%. Throughput from both clients is ~8.5MB/sec. The tests were run separately. The NetAPP on each host was over 48.5 MB/sec. The FreeBSD host still has about 2 GB free. Paul ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 1:03:14 PM Subject: Re: ZFS, NFS and Network tuning Hello all, I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 9:47:59 AM Subject: Re: ZFS, NFS and Network tuning Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From pathiaki2 at yahoo.com Sun Dec 21 08:50:51 2008 From: pathiaki2 at yahoo.com (Paul Patterson) Date: Sun Dec 21 08:51:00 2008 Subject: ZFS, NFS and Network tuning (Paul Patterson) References: <722609.11236.qm@web65412.mail.ac4.yahoo.com> Message-ID: <726963.81983.qm@web110508.mail.gq1.yahoo.com> Michelle, just found this in the mail. However, it looks like the VPN to work is down. I can't get to the machine right now. I'll mail on Monday. P. ________________________________ From: Michelle Li To: freebsd-performance@freebsd.org Sent: Saturday, December 20, 2008 7:25:02 PM Subject: Re: ZFS, NFS and Network tuning (Paul Patterson) ...and the dmesg? please post freebsd-performance-request@freebsd.org wrote: Send freebsd-performance mailing list submissions to freebsd-performance@freebsd.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.freebsd.org/mailman/listinfo/freebsd-performance or, via email, send a message with subject or body 'help' to freebsd-performance-request@freebsd.org You can reach the person managing the list at freebsd-performance-owner@freebsd.org When replying, please edit your Subject line so it is more specific than "Re: Contents of freebsd-performance digest..." Today's Topics: 1. Re: ZFS, NFS and Network tuning (Paul Patterson) 2. Re: ZFS, NFS and Network tuning (Paul Patterson) 3. Re: ZFS, NFS and Network tuning (Paul Patterson) 4. intel i7 and Hyperthreading (Mike Tancsa) ---------------------------------------------------------------------- Message: 1 Date: Fri, 19 Dec 2008 06:47:59 -0800 (PST) From: Paul Patterson Subject: Re: ZFS, NFS and Network tuning To: Paul Patterson , freebsd-performance@freebsd.org Message-ID: <15723.22980.qm@web110511.mail.gq1.yahoo.com> Content-Type: text/plain; charset=us-ascii Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ------------------------------ Message: 2 Date: Fri, 19 Dec 2008 10:03:14 -0800 (PST) From: Paul Patterson Subject: Re: ZFS, NFS and Network tuning To: Paul Patterson , freebsd-performance@freebsd.org Message-ID: <400826.77992.qm@web110510.mail.gq1.yahoo.com> Content-Type: text/plain; charset=us-ascii Hello all, I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 9:47:59 AM Subject: Re: ZFS, NFS and Network tuning Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ------------------------------ Message: 3 Date: Fri, 19 Dec 2008 10:59:54 -0800 (PST) From: Paul Patterson Subject: Re: ZFS, NFS and Network tuning To: Paul Patterson , freebsd-performance@freebsd.org Message-ID: <309927.87042.qm@web110514.mail.gq1.yahoo.com> Content-Type: text/plain; charset=us-ascii Hi, Well, I got some input on things: kern.ipc.somaxconn=32768 net.inet.tcp.mssdflt=1460 And for fstab rw,tcp,intr,noatime,nfsv3,-w=65536,-r=65536 I tried turning on polling with ifconfig bce0 polling, however, I didn't see it in ifconfig bce0 so I don't believe it to be active or the card doesn't support it. aI also removed async from the mounts. These had a detrimental affect on the FreeBSD server. I now get 64K per transfer (system -vm) but I'm still only getting about 4MB/sec on the disks and their utilization has dropped to about 5%. Throughput from both clients is ~8.5MB/sec. The tests were run separately. The NetAPP on each host was over 48.5 MB/sec. The FreeBSD host still has about 2 GB free. Paul ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 1:03:14 PM Subject: Re: ZFS, NFS and Network tuning Hello all, I guess I've got to send this as I've already had about 5 responses claiming the same thing. This is not a disk bottleneck. The ZFS partition is capable of performing at the theoretical max of the drives. The machine is performing at less than 5 MB combined. I'm assuming that this is a problem with the NFSv3 throughput. I just 'dd' 1000 1MB records (about 1GB) from the clients to their respective servers: Client 1 to NetAPP: 3 tests for 45.9, 45.1, 46.1 Pretty consistent Client 2 to FreeBSD/ZFS: 3 test for 29.7, 12.5, 19.1 NOT consistent (also, the drives were lucky to hit 12% busy. I'm about to mount these servers to each client and see if there's a variation (although they are hw configured the same and bought the same time.) I'll write after this. However, if more people could review the configurations below and see if there's anything glaring.... However, the lack of consistency shows something is wrong network wise. P. ________________________________ From: Paul Patterson To: Paul Patterson ; freebsd-performance@freebsd.org Sent: Friday, December 19, 2008 9:47:59 AM Subject: Re: ZFS, NFS and Network tuning Hi, as promised, the parameter tuning I have on the box (does anyone see anything wrong?) /boot/loader.conf kern.hz="100" vm.kmem_size_max="1536M" vm.kmem_size="1536M" vfs.zfs.prefetch_disble=1 /etc/sysctl.conf kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.mxvnodes=600000 net.inet.tcp.delayed_ack=0 net.inet.tcp.inflight.enable=0 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=65536 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcpsendbuf_inc=8192 net.inet.tcp.sendspace=65536 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216 ________________________________ From: Paul Patterson To: freebsd-performance@freebsd.org Sent: Thursday, December 18, 2008 8:04:37 PM Subject: ZFS, NFS and Network tuning Hi, I just set up my first machine with ZFS. (First, ZFS is nothing short of amazing) I'm running FreeBSD 7.1-RC1 as an NFS server with ZFS striped across two volumes (just testing throughput for now.) Anyhow, I was benching this box, 4GB or RAM, the volume is on 2x146 GB SAS 10K rpm drives and it's an HP Proliant DL360 with dual Gb interfaces. (device bce) Now, I believe that I have tuned this box to the hilt with all the parameters that I can think of (it's at work right now so I'll cut and paste all the sysctls and loader.conf parameters for ZFS and networking) and it still seems to have some type of bottleneck. I have two Debian Linux clients that I use to bench with. I run a script that makes calls that writes to the NFS device and, after about 30 minutes, starts to delete the initial data and follow behind writing and deleting. Here's what's happening: The "other" machine is a NetAPP. It's got 1GB of RAM and it's running RAID DP with 2 parity drives and 6 data drives, all SATA 750 GB 7200 RPM drives with dual Gb interfaces. The benchmark script manages to write lots of little (all less than 30KB) files at a rate of 11,000 per minute, however, after 30 minutes, when it starts deleting, the throughput on write goes to 9500 and deletion is 6000 per minute. If I turn on the second node, I get 17,000 writing combined with about 11,000 deletions combined. One way or another, this will overflow in time. Not good. Now, on to my pet project. :-) The FreeBSD/ZFS server is only able to maintain about 3500 writes per minute but also deletes at the same rate! (I would expect deletion to be at least as fast as writing) The drives are running at only 20-35% while this is going on and only putting down about 4-5 MB/sec each. So, at 1Gb or ~92MB/sec theoretical max (is that about right?) There's something wrong somewhere. I'm assuming it's the network. (I'll post all the tunings tomorrow.) Thinking something wrong, I mounted only one client to each server (they are identical clients and the same configuration as the FreeBSD box). I did a simple stream of: dd if=/dev/zero of=/mnt/nfs bs=1m count=1000. The FreeBSD box wins?! It cranked up the drives to 45-50 MB/sec each and balanced them perfectly on transactions/sec KB/sec, etc from systat -vm. (Woohoo!) The NetAPPs CPU was at over 35-40% constantly, (it does that while benching, too) I'll post the NetAPP finding tomorrow as I forgot it for now. As for the client mounting, it was with the options: nfsvers=3,rsize=32768,wsize=32768,hard,intr,async,noatime I'm trying to figure out why, when running this benchmark, can the NetAPP with WAFL nearly triple the FreeBSD/ZFS box. Also, I'm having something strange happen when I try to mount the disk from the FreeBSD server versus the NetAPP. The FreeBSD server will sometimes RPC timeout. Mounting the NetAPP is instantaneous. That's the beginning. If I have a list of things to check tomorrow, I will. I'd like to see the little machine that could kick the NetAPPs butt. (No offense to NetAPP. :-) ) Thank you for reading, Paul _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ------------------------------ Message: 4 Date: Fri, 19 Dec 2008 17:01:46 -0500 From: Mike Tancsa Subject: intel i7 and Hyperthreading To: freebsd-performance@freebsd.org Message-ID: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Content-Type: text/plain; charset="us-ascii"; format=flowed Just got our first board to play around with and unlike in the past, having hyperthreading enabled seems to help performance.... At least in buildworld tests. doing a make -j4 vs -j6 make -j8 vs -j10 gives -j buildworld time % improvement over -j4 4 13:57 6 12:11 13% 8 11:32 18% 10 11:43 17% dmesg below of the hardware... The CPU seems to run fairly cool, but the board has a lot of nasty hot heatsinks eg. running 8 burnP6 procs 0[ns3c]# sysctl -a | grep temperature dev.cpu.0.temperature: 67 dev.cpu.1.temperature: 67 dev.cpu.2.temperature: 65 dev.cpu.3.temperature: 65 dev.cpu.4.temperature: 66 dev.cpu.5.temperature: 66 dev.cpu.6.temperature: 64 dev.cpu.7.temperature: 64 0[ns3c]# vs idle dev.cpu.0.temperature: 46 dev.cpu.1.temperature: 46 dev.cpu.2.temperature: 42 dev.cpu.3.temperature: 42 dev.cpu.4.temperature: 44 dev.cpu.5.temperature: 44 dev.cpu.6.temperature: 40 dev.cpu.7.temperature: 40 Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-PRERELEASE #0: Fri Dec 19 19:48:15 EST 2008 mdtancsa@ns3c.recycle.net:/usr/obj/usr/src/sys/recycle Timecounter "i8254" frequency 1193182 Hz quality 0 === message truncated === _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From ivoras at freebsd.org Tue Dec 23 01:53:23 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Tue Dec 23 01:53:30 2008 Subject: intel i7 and Hyperthreading In-Reply-To: <200812192214.mBJMEj2Q009511@lava.sentex.ca> References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Message-ID: Mike Tancsa wrote: > Just got our first board to play around with and unlike in the past, > having hyperthreading enabled seems to help performance.... At least in > buildworld tests. > > doing a make -j4 vs -j6 make -j8 vs -j10 gives > > -j buildworld time % improvement over -j4 > 4 13:57 > 6 12:11 13% > 8 11:32 18% > 10 11:43 17% Thanks for posting this! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20081223/ddf25cb0/signature.pgp From pieter at degoeje.nl Tue Dec 23 03:06:53 2008 From: pieter at degoeje.nl (Pieter de Goeje) Date: Tue Dec 23 03:07:02 2008 Subject: intel i7 and Hyperthreading In-Reply-To: References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Message-ID: <200812231122.02892.pieter@degoeje.nl> On Tuesday 23 December 2008, Ivan Voras wrote: > Mike Tancsa wrote: > > Just got our first board to play around with and unlike in the past, > > having hyperthreading enabled seems to help performance.... At least in > > buildworld tests. > > > > doing a make -j4 vs -j6 make -j8 vs -j10 gives > > > > -j buildworld time % improvement over -j4 > > 4 13:57 > > 6 12:11 13% > > 8 11:32 18% > > 10 11:43 17% > > Thanks for posting this! What's missing is build times with hyperthreading disabled. make buildworld -j10 could easily be faster than -j4 even when hyperthreading is disabled, because of increased disk I/O concurrency. 11:43 to build world is very fast indeed :-) -- Pieter de Goeje From mike at sentex.net Tue Dec 23 07:05:06 2008 From: mike at sentex.net (Mike Tancsa) Date: Tue Dec 23 07:05:13 2008 Subject: intel i7 and Hyperthreading In-Reply-To: <200812231122.02892.pieter@degoeje.nl> References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> <200812231122.02892.pieter@degoeje.nl> Message-ID: <200812231505.mBNF50Bu030894@lava.sentex.ca> At 05:22 AM 12/23/2008, Pieter de Goeje wrote: >On Tuesday 23 December 2008, Ivan Voras wrote: > > Mike Tancsa wrote: > > > Just got our first board to play around with and unlike in the past, > > > having hyperthreading enabled seems to help performance.... At least in > > > buildworld tests. > > > > > > doing a make -j4 vs -j6 make -j8 vs -j10 gives > > > > > > -j buildworld time % improvement over -j4 > > > 4 13:57 > > > 6 12:11 13% > > > 8 11:32 18% > > > 10 11:43 17% > > > > Thanks for posting this! > >What's missing is build times with hyperthreading disabled. >make buildworld -j10 could easily be faster than -j4 even when hyperthreading >is disabled, because of increased disk I/O concurrency. > >11:43 to build world is very fast indeed :-) With HT disabled in the BIOS make -j4, 13:48 make -j8, 12:35 so -j4 is about the same, but -j8 does show a bit of an improvement with HT enabled ( ~9%) ... ---Mike From ivoras at freebsd.org Tue Dec 23 08:21:00 2008 From: ivoras at freebsd.org (Ivan Voras) Date: Tue Dec 23 08:21:08 2008 Subject: intel i7 and Hyperthreading In-Reply-To: <200812192214.mBJMEj2Q009511@lava.sentex.ca> References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Message-ID: Mike Tancsa wrote: > FreeBSD 7.1-PRERELEASE #0: Fri Dec 19 19:48:15 EST 2008 > mdtancsa@ns3c.recycle.net:/usr/obj/usr/src/sys/recycle > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2666.78-MHz > 686-class CPU) > Origin = "GenuineIntel" Id = 0x106a4 Stepping = 4 > > Features=0xbfebfbff > > > Features2=0x98e3bd > > AMD Features=0x28100000 > AMD Features2=0x1 > Cores per package: 8 > Logical CPUs per core: 2 > real memory = 2138992640 (2039 MB) > avail memory = 2084880384 (1988 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > cpu4 (AP): APIC ID: 4 > cpu5 (AP): APIC ID: 5 > cpu6 (AP): APIC ID: 6 > cpu7 (AP): APIC ID: 7 I just thought of another thing - can you boot an 8-CURRENT kernel on the machine and report the value of kern.sched.topology_spec sysctl? This is to verify how the ULE sees the HTT topology of the CPUs. From mike at sentex.net Tue Dec 23 08:47:53 2008 From: mike at sentex.net (Mike Tancsa) Date: Tue Dec 23 08:48:01 2008 Subject: intel i7 and Hyperthreading In-Reply-To: References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Message-ID: <200812231647.mBNGlqkh031338@lava.sentex.ca> At 11:20 AM 12/23/2008, Ivan Voras wrote: >I just thought of another thing - can you boot an 8-CURRENT kernel >on the machine and report the value of kern.sched.topology_spec >sysctl? This is to verify how the ULE sees the HTT topology of the CPUs. It will have to wait for the next board as this is going out to a customer today. ---Mike >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From mike at sentex.net Tue Dec 23 13:00:22 2008 From: mike at sentex.net (Mike Tancsa) Date: Tue Dec 23 13:00:29 2008 Subject: intel i7 and Hyperthreading In-Reply-To: References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Message-ID: <200812232100.mBNL0KPP032432@lava.sentex.ca> At 11:20 AM 12/23/2008, Ivan Voras wrote: >Mike Tancsa wrote: > >>FreeBSD 7.1-PRERELEASE #0: Fri Dec 19 19:48:15 EST 2008 >> mdtancsa@ns3c.recycle.net:/usr/obj/usr/src/sys/recycle >>Timecounter "i8254" frequency 1193182 Hz quality 0 >>CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2666.78-MHz >>686-class CPU) >> Origin = "GenuineIntel" Id = 0x106a4 Stepping = 4 >> >>Features=0xbfebfbff >> >> >>Features2=0x98e3bd >> >> AMD Features=0x28100000 >> AMD Features2=0x1 >> Cores per package: 8 >> Logical CPUs per core: 2 >>real memory = 2138992640 (2039 MB) >>avail memory = 2084880384 (1988 MB) >>ACPI APIC Table: >>FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> cpu2 (AP): APIC ID: 2 >> cpu3 (AP): APIC ID: 3 >> cpu4 (AP): APIC ID: 4 >> cpu5 (AP): APIC ID: 5 >> cpu6 (AP): APIC ID: 6 >> cpu7 (AP): APIC ID: 7 > >I just thought of another thing - can you boot an 8-CURRENT kernel >on the machine and report the value of kern.sched.topology_spec >sysctl? This is to verify how the ULE sees the HTT topology of the CPUs. I have a bit of delay, so I can run it with another disk for a day. 0[current]# sysctl -A kern.sched kern.sched.preemption: 1 kern.sched.topology_spec: 0, 1, 2, 3, 4, 5, 6, 7 kern.sched.steal_thresh: 3 kern.sched.steal_idle: 1 kern.sched.steal_htt: 1 kern.sched.balance_interval: 133 kern.sched.balance: 1 kern.sched.affinity: 1 kern.sched.idlespinthresh: 4 kern.sched.idlespins: 10000 kern.sched.static_boost: 160 kern.sched.preempt_thresh: 64 kern.sched.interact: 30 kern.sched.slice: 13 kern.sched.name: ULE 0[current]# I tried to boot off the eSata drive, but HEAD does not understand the controller properly as it never sees the disk. From the BIOS, it says its a marvel thingy and pciconf sees it that way too...Also, the ichwd doesnt seem to work. Its possible its disabled on the motherboard. dmesg and pciconf attached atapci0@pci0:6:0:0: class=0x01018f card=0x4f538086 chip=0x612111ab rev=0xb2 hdr=0x00 vendor = 'Marvell Semiconductor (Was: Galileo Technology Ltd)' device = '6121 SATA2 Controller' class = mass storage subclass = ATA cap 01[48] = powerspec 2 supports D0 D1 D3 current D0 cap 05[50] = MSI supports 1 message cap 10[e0] = PCI-Express 1 legacy endpoint ---Mike -------------- next part -------------- A non-text attachment was scrubbed... Name: dmesg.txt Type: application/octet-stream Size: 64249 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20081223/2a766eb9/dmesg.obj From mike at sentex.net Tue Dec 23 19:20:51 2008 From: mike at sentex.net (Mike Tancsa) Date: Tue Dec 23 19:20:57 2008 Subject: intel i7 and Hyperthreading In-Reply-To: References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> Message-ID: <200812240320.mBO3KoEi033856@lava.sentex.ca> At 11:20 AM 12/23/2008, Ivan Voras wrote: >I just thought of another thing - can you boot an 8-CURRENT kernel >on the machine and report the value of kern.sched.topology_spec >sysctl? This is to verify how the ULE sees the HTT topology of the CPUs. And buildworld from current 4,8 and 10 3287.727u 973.504s 21:18.88 333.1% -4100+2075k 28490+5472io 10433pf+0w 3511.547u 1473.695s 15:30.77 535.6% -2668+2067k 1604+5381io 8741pf+0w 3576.281u 1546.722s 15:38.02 546.1% -2431+2068k 1570+5448io 8711pf+0w ---Mike From ivoras at freebsd.org Wed Dec 24 07:02:45 2008 From: ivoras at freebsd.org (ivoras@freebsd.org) Date: Wed Dec 24 07:02:52 2008 Subject: intel i7 and Hyperthreading In-Reply-To: <200812240320.mBO3KoEi033856@lava.sentex.ca> References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> <200812240320.mBO3KoEi033856@lava.sentex.ca> Message-ID: <9bbcef730812240702t6fd2f9f8wcdda075f70517102@mail.gmail.com> On 24/12/2008, Mike Tancsa wrote: > At 11:20 AM 12/23/2008, Ivan Voras wrote: > >>I just thought of another thing - can you boot an 8-CURRENT kernel >>on the machine and report the value of kern.sched.topology_spec >>sysctl? This is to verify how the ULE sees the HTT topology of the CPUs. > > > And buildworld from current > > 4,8 and 10 > > 3287.727u 973.504s 21:18.88 333.1% -4100+2075k 28490+5472io 10433pf+0w > 3511.547u 1473.695s 15:30.77 535.6% -2668+2067k 1604+5381io 8741pf+0w > 3576.281u 1546.722s 15:38.02 546.1% -2431+2068k 1570+5448io 8711pf+0w > Comparing to your original numbers, it looks like you might have some debugging enabled there: the original 7.x results went from 13:57 to 11:32, this goes from 21:18 to 15:30. I don't think the world is that much larger in -CURRENT. From mike at sentex.net Wed Dec 24 07:07:37 2008 From: mike at sentex.net (Mike Tancsa) Date: Wed Dec 24 07:07:43 2008 Subject: intel i7 and Hyperthreading In-Reply-To: <9bbcef730812240702t6fd2f9f8wcdda075f70517102@mail.gmail.co m> References: <200812192214.mBJMEj2Q009511@lava.sentex.ca> <200812240320.mBO3KoEi033856@lava.sentex.ca> <9bbcef730812240702t6fd2f9f8wcdda075f70517102@mail.gmail.com> Message-ID: <200812241507.mBOF7ZaG036966@lava.sentex.ca> At 10:02 AM 12/24/2008, ivoras@freebsd.org wrote: >Comparing to your original numbers, it looks like you might have some >debugging enabled there: the original 7.x results went from 13:57 to Yes, its a regular kernel and things like malloc are the default and from the dmesg WARNING: WITNESS option enabled, expect reduced performance. The buildworld times were more to compare -j4 vs -j8 on HEAD to see if there are similar differences. ---Mike From pheriko.support at gmail.com Tue Dec 30 02:08:21 2008 From: pheriko.support at gmail.com (Periko Support) Date: Tue Dec 30 02:08:29 2008 Subject: bonnie++ result, can someone tech me how to understand this data and help me tunning my system a little more? In-Reply-To: <5b1d935a0812291750p46b4da74s92f756c646ff4601@mail.gmail.com> References: <5b1d935a0812291750p46b4da74s92f756c646ff4601@mail.gmail.com> Message-ID: <5b1d935a0812291808g6e8b7540sdd42a6357aa6835c@mail.gmail.com> Hi people. I have found a way to show u my results, u can see in the title of the page what flags I give to bonnie++, I test inside /var because is where mysql run. The page is *Bonnie Result s Thanks again for your time!!! * On Mon, Dec 29, 2008 at 5:50 PM, Periko Support wrote: > Hi people. > > I want to learn how to get more from my system, I will run a new > application with mysql 5.1.30, this app is develop en .NET(I'm the > programmer) the clients will be all windows xp boxes, with ps/2 scanner to > add, get, update the data to mysql, querys etc,etc, them reading about > performance, for a database is important the file system, I start googling > around and found tha exist a lot of free app to benchmark the hard disk, one > of the is bonnie++, I have been running several times in my system, now I > have the results, I would like to know what is the meaning and what > parameters can I change to see if I can get better results based on your > point of view. > > All my programs are installed from ports. > > I have the results in one file called bonnie-res.html, I atach the file > here, I don't know if exist a place where I can put this file to be public? > Plus my kernel file called BACULA. > > I was thinking in pastebin, but right now I cannot run bonnie and send > the output to cvs, because my system is running the backup. > > Technical details: > OS: 7.0-RELEASE-p6 i386 (Custom Kernel) > 512RAM > > Motherboard: > Foxconn KM400 Duron 1.6Ghz > > Raid-1 with geom_mirror > > disk 1: ad0: setting PIO4 on 8235 chip > ad0: setting UDMA100 on 8235 chip > ad0: 286168MB at ata0-master UDMA100 > ad0: 586072368 sectors [581421C/16H/63S] 16 sectors/interrupt 1 depth queue > > Disk 2: ad2: setting PIO4 on 8235 chip > ad2: setting UDMA100 on 8235 chip > ad2: 305245MB at ata1-master UDMA100 > ad2: 625142448 sectors [620181C/16H/63S] 16 sectors/interrupt 1 depth queue > > Both disk on different channels. > > This is my make.conf file: > > CPUTYPE?=athlon > CFLAGS= -O2 -fno-strict-aliasing -pipe > CXXFLAGS+= -fconserve-space > MAKE_SHELL?=sh > BDECFLAGS= -W -Wall -ansi -pedantic -Wbad-function-cast -Wcast-align \ > -Wcast-qual -Wchar-subscripts -Winline \ > -Wmissing-prototypes -Wnested-externs -Wpointer-arith \ > -Wredundant-decls -Wshadow -Wstrict-prototypes > -Wwrite-strings > #COPTFLAGS= -O -pipe > ENABLE_SUID_SSH= > PPP_NO_NAT= # do not build with NAT support (see make.conf(5)) > PPP_NO_NETGRAPH= # do not build with Netgraph support > PPP_NO_RADIUS= # do not build with RADIUS support > PPP_NO_SUID= # build with normal permissions > TRACEROUTE_NO_IPSEC= # do not build traceroute(8) with IPSEC support > BOOTWAIT=0 > DOC_LANG= en_US.ISO8859-1 > FORCE_PKG_REGISTER=yes > # added by use.perl 2008-12-14 10:36:46 > PERL_VER=5.8.8 > PERL_VERSION=5.8.8 > > > My src.conf > > WITHOUT_ATM="YES" > WITHOUT_AUTHPF="YES" > WITHOUT_BIND="YES" > WITHOUT_BLUETOOTH="YES" > WITHOUT_GAMES="YES" > WITHOUT_I4B="YES" > WITH_IDEA="NO" > WITHOUT_HTML="YES" > WITHOUT_INET6="YES" > WITHOUT_IPFILTER="YES" > WITHOUT_IPX="YES" > WITHOUT_LPR="YES" > WITHOUT_NCP="YES" > WITHOUT_PF="YES" > WITHOUT_RCMDS="YES" > > Mount info: > > /dev/mirror/gm0s1a on / (ufs, local, soft-updates) > devfs on /dev (devfs, local) > /dev/mirror/gm0s1g on /backups (ufs, local, noatime, soft-updates) > /dev/mirror/gm0s1f on /tmp (ufs, local, noatime, soft-updates) > /dev/mirror/gm0s1d on /usr (ufs, local, noatime, soft-updates) > /dev/mirror/gm0s1e on /var (ufs, local, noatime, soft-updates) > /dev/ad3s1d on /spool (ufs, local, noatime, soft-updates) > > I want to first get if is possible a little more from freebsd latter I will > continue with mysql. > > Thanks all for your time, I normally read this maillist, but now I want to > get a little more from my system. > > I will run iozone + sysbench and let u know my results to, if u need more > info from my system let me know. > > Thanks again and happy new year!!! > > From pheriko.support at gmail.com Tue Dec 30 02:22:51 2008 From: pheriko.support at gmail.com (Periko Support) Date: Tue Dec 30 02:22:58 2008 Subject: bonnie++ result, can someone tech me how to understand this data and help me tunning my system a little more? Message-ID: <5b1d935a0812291750p46b4da74s92f756c646ff4601@mail.gmail.com> Hi people. I want to learn how to get more from my system, I will run a new application with mysql 5.1.30, this app is develop en .NET(I'm the programmer) the clients will be all windows xp boxes, with ps/2 scanner to add, get, update the data to mysql, querys etc,etc, them reading about performance, for a database is important the file system, I start googling around and found tha exist a lot of free app to benchmark the hard disk, one of the is bonnie++, I have been running several times in my system, now I have the results, I would like to know what is the meaning and what parameters can I change to see if I can get better results based on your point of view. All my programs are installed from ports. I have the results in one file called bonnie-res.html, I atach the file here, I don't know if exist a place where I can put this file to be public? Plus my kernel file called BACULA. I was thinking in pastebin, but right now I cannot run bonnie and send the output to cvs, because my system is running the backup. Technical details: OS: 7.0-RELEASE-p6 i386 (Custom Kernel) 512RAM Motherboard: Foxconn KM400 Duron 1.6Ghz Raid-1 with geom_mirror disk 1: ad0: setting PIO4 on 8235 chip ad0: setting UDMA100 on 8235 chip ad0: 286168MB at ata0-master UDMA100 ad0: 586072368 sectors [581421C/16H/63S] 16 sectors/interrupt 1 depth queue Disk 2: ad2: setting PIO4 on 8235 chip ad2: setting UDMA100 on 8235 chip ad2: 305245MB at ata1-master UDMA100 ad2: 625142448 sectors [620181C/16H/63S] 16 sectors/interrupt 1 depth queue Both disk on different channels. This is my make.conf file: CPUTYPE?=athlon CFLAGS= -O2 -fno-strict-aliasing -pipe CXXFLAGS+= -fconserve-space MAKE_SHELL?=sh BDECFLAGS= -W -Wall -ansi -pedantic -Wbad-function-cast -Wcast-align \ -Wcast-qual -Wchar-subscripts -Winline \ -Wmissing-prototypes -Wnested-externs -Wpointer-arith \ -Wredundant-decls -Wshadow -Wstrict-prototypes -Wwrite-strings #COPTFLAGS= -O -pipe ENABLE_SUID_SSH= PPP_NO_NAT= # do not build with NAT support (see make.conf(5)) PPP_NO_NETGRAPH= # do not build with Netgraph support PPP_NO_RADIUS= # do not build with RADIUS support PPP_NO_SUID= # build with normal permissions TRACEROUTE_NO_IPSEC= # do not build traceroute(8) with IPSEC support BOOTWAIT=0 DOC_LANG= en_US.ISO8859-1 FORCE_PKG_REGISTER=yes # added by use.perl 2008-12-14 10:36:46 PERL_VER=5.8.8 PERL_VERSION=5.8.8 My src.conf WITHOUT_ATM="YES" WITHOUT_AUTHPF="YES" WITHOUT_BIND="YES" WITHOUT_BLUETOOTH="YES" WITHOUT_GAMES="YES" WITHOUT_I4B="YES" WITH_IDEA="NO" WITHOUT_HTML="YES" WITHOUT_INET6="YES" WITHOUT_IPFILTER="YES" WITHOUT_IPX="YES" WITHOUT_LPR="YES" WITHOUT_NCP="YES" WITHOUT_PF="YES" WITHOUT_RCMDS="YES" Mount info: /dev/mirror/gm0s1a on / (ufs, local, soft-updates) devfs on /dev (devfs, local) /dev/mirror/gm0s1g on /backups (ufs, local, noatime, soft-updates) /dev/mirror/gm0s1f on /tmp (ufs, local, noatime, soft-updates) /dev/mirror/gm0s1d on /usr (ufs, local, noatime, soft-updates) /dev/mirror/gm0s1e on /var (ufs, local, noatime, soft-updates) /dev/ad3s1d on /spool (ufs, local, noatime, soft-updates) I want to first get if is possible a little more from freebsd latter I will continue with mysql. Thanks all for your time, I normally read this maillist, but now I want to get a little more from my system. I will run iozone + sysbench and let u know my results to, if u need more info from my system let me know. Thanks again and happy new year!!! -------------- next part -------------- bonnie++ -d /var/smack-data/ -c 5 -s "Size" -r "RamSize" -x 5 -u root
Version 1.93dSequential OutputSequential InputRandom
Seeks		Sequential CreateRandom Create
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local550M198985680035480882462398++++++++10637148161186253++++++++++++++++1122547++++++++++++++++
mbx-bacula.MBX.localLatency43031us30491us43955us13946us549us767msLatency99802us2222us226us164ms430us24086us
mbx-bacula.MBX.local550M197975412230408712562398++++++++10111140161656065++++++++++++++++1325354++++++++++++++++
mbx-bacula.MBX.localLatency218ms27055us49718us13473us550us645msLatency110ms586us3555us173ms13249us31276us
mbx-bacula.MBX.local550M193974984827410622561998++++++++10469146161777269++++++++++++++++1290754++++++++++++++++
mbx-bacula.MBX.localLatency220ms35814us42062us13777us205us634msLatency112ms25106us231us202ms4245us28269us
mbx-bacula.MBX.local550M196975255029432972661898++++++++10682147161800769++++++++++++++++1308754++++++++++++++++
mbx-bacula.MBX.localLatency214ms36854us33973us14301us208us156msLatency108ms24634us221us166ms25259us29994us
mbx-bacula.MBX.local550M197975083928413492561698++++++++10535146161835371++++++++++++++++1308654++++++++++++++++
mbx-bacula.MBX.localLatency211ms34809us48628us13846us208us119msLatency110ms24173us224us193ms15659us31076us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5100M196975074129403832560298++++++++7294144161276754++++++++++++++++1269251++++++++++++++++
mbx-bacula.MBX.localLatency214ms37759us44502us25151us566us218msLatency103ms559us240us152ms21851us229us
mbx-bacula.MBX.local5100M197975663632466292962098++++++++7492147161747764++++++++++++++++1382052++++++++++++++++
mbx-bacula.MBX.localLatency202ms35207us45602us24789us544us901msLatency91241us1922us566us167ms25985us576us
mbx-bacula.MBX.local5100M197975687332454682859998++++++++7284143161684863++++++++++++++++1321453++++++++++++++++
mbx-bacula.MBX.localLatency208ms28795us39803us25551us558us793msLatency102ms9668us3658us184ms6295us573us
mbx-bacula.MBX.local5100M195975344530441712761497++++++++7428145161783067++++++++++++++++1260351++++++++++++++++
mbx-bacula.MBX.localLatency234ms39540us44290us30006us529us774msLatency110ms26206us564us244ms4082us27873us
mbx-bacula.MBX.local5100M197974927727454402760198++++++++7243142161746364++++++++++++++++1310252++++++++++++++++
mbx-bacula.MBX.localLatency210ms39444us49439us25713us555us1127msLatency102ms559us3669us167ms1213us33982us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5150M196975421731159729603983572712176950161260148++++++++++++++++1115646++++++++++++++++
mbx-bacula.MBX.localLatency217ms125ms106ms33547us38183us1047msLatency86805us323us3599us149ms660us24919us
mbx-bacula.MBX.local5150M19697555843217412106159849368174698112161243648++++++++++++++++1073642++++++++++++++++
mbx-bacula.MBX.localLatency229ms40441us84757us29206us37307us1011msLatency113ms340us565us151ms2506us26096us
mbx-bacula.MBX.local5150M1969756504321684010601984466715293977161382853++++++++++++++++1070843++++++++++++++++
mbx-bacula.MBX.localLatency228ms32015us79621us21689us57732us1040msLatency107ms31597us562us147ms20608us326us
mbx-bacula.MBX.local5150M1969756583321634110610984362815280174161309850++++++++++++++++1103545++++++++++++++++
mbx-bacula.MBX.localLatency198ms27964us87373us25192us51012us1067msLatency93344us32038us262us141ms20176us581us
mbx-bacula.MBX.local5150M1979757350331648410615984346215267170161276749++++++++++++++++1122445++++++++++++++++
mbx-bacula.MBX.localLatency199ms27776us72803us24043us36276us1092msLatency91552us32201us227us143ms21687us308us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5200M197975676532156809613983815813740.226161284450++++++++++++++++1061843++++++++++++++++
mbx-bacula.MBX.localLatency194ms30671us50800us21443us51974us66493usLatency103ms31975us225us138ms21243us586us
mbx-bacula.MBX.local5200M197975742033161739608983936814741.826161276149++++++++++++++++1132946++++++++++++++++
mbx-bacula.MBX.localLatency200ms33890us75875us45140us25688us77036usLatency108ms323us3770us88695us4441us21784us
mbx-bacula.MBX.local5200M197975670132158959624983718713734.826161341152++++++++++++++++1094244++++++++++++++++
mbx-bacula.MBX.localLatency211ms35944us66448us34283us22131us90519usLatency117ms31480us597us140ms36113us3984us
mbx-bacula.MBX.local5200M195975710233162239611983774013742.026161293350++++++++++++++++1098444++++++++++++++++
mbx-bacula.MBX.localLatency198ms33901us78119us23988us41774us71305usLatency102ms288us226us148ms463us580us
mbx-bacula.MBX.local5200M197975675232163179611983733213727.826161363352++++++++++++++++1131843++++++++++++++++
mbx-bacula.MBX.localLatency222ms37491us90769us32975us41301us78027usLatency115ms21754us1135us143ms19233us24779us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5250M197975742634159639608983786413515.120161271949++++++++++++++++1118044++++++++++++++++
mbx-bacula.MBX.localLatency211ms34257us102ms20662us49607us1215msLatency113ms914us231us141ms21275us251us
mbx-bacula.MBX.local5250M194975736033156129590973761513515.620161292650++++++++++++++++1162348++++++++++++++++
mbx-bacula.MBX.localLatency222ms29760us82561us53349us44795us1186msLatency109ms1811us3628us141ms8810us23862us
mbx-bacula.MBX.local5250M196975781134157119623983781213523.221161240848++++++++++++++++1087644++++++++++++++++
mbx-bacula.MBX.localLatency213ms35043us86354us22693us44057us1166msLatency128ms31821us230us139ms21440us573us
mbx-bacula.MBX.local5250M197975726833159889601983793513521.721161200446++++++++++++++++1154147++++++++++++++++
mbx-bacula.MBX.localLatency197ms37633us88689us45780us40436us1152msLatency176ms323us3625us152ms14814us24615us
mbx-bacula.MBX.local5250M197975769433154198612983726713521.421161268749++++++++++++++++1105343++++++++++++++++
mbx-bacula.MBX.localLatency213ms32185us100ms23405us42917us1161msLatency103ms337us567us147ms7496us285us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5300M199985686933155869624983870713433.018161296850++++++++++++++++1022842++++++++++++++++
mbx-bacula.MBX.localLatency42464us43148us75008us38967us39680us91611usLatency98508us33303us561us169ms20744us257us
mbx-bacula.MBX.local5300M196975682933156609602983732313430.618161300650++++++++++++++++1107745++++++++++++++++
mbx-bacula.MBX.localLatency199ms32231us80816us27387us49840us82895usLatency93681us8317us3650us137ms513us23334us
mbx-bacula.MBX.local5300M196975721233156939604983870613424.018161304950++++++++++++++++1079543++++++++++++++++
mbx-bacula.MBX.localLatency214ms33725us82063us31183us42142us79065usLatency94184us309us222us150ms565us581us
mbx-bacula.MBX.local5300M196975666133154658625983790113428.418161242148++++++++++++++++1108545++++++++++++++++
mbx-bacula.MBX.localLatency224ms36208us85879us26122us46395us83385usLatency136ms7890us576us147ms560us226us
mbx-bacula.MBX.local5300M196975607333155299613973809513420.818161310451++++++++++++++++1057542++++++++++++++++
mbx-bacula.MBX.localLatency201ms43344us107ms28804us42392us102msLatency88264us1734us568us143ms564us230us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5350M200985720933155349610973843813371.316161221347++++++++++++++++1150745++++++++2591065
mbx-bacula.MBX.localLatency42317us40544us79517us35706us51434us1427msLatency108ms7467us3610us142ms458us627us
mbx-bacula.MBX.local5350M199985730533155378619983911113370.316161343052++++++++++++++++1102245++++++++++++++++
mbx-bacula.MBX.localLatency42229us37899us110ms38454us43829us1431msLatency91795us2052us567us145ms21465us344us
mbx-bacula.MBX.local5350M194975746934154968621983844813371.016161175046++++++++++++++++1339652++++++++++++++++
mbx-bacula.MBX.localLatency198ms35737us69751us29958us45207us1360msLatency100ms10528us3656us101ms563us252us
mbx-bacula.MBX.local5350M196975358031153998603983769313371.116161305150++++++++++++++++1097743++++++++++++++++
mbx-bacula.MBX.localLatency198ms47211us78829us29252us41817us1388msLatency97667us7456us226us145ms442us237us
mbx-bacula.MBX.local5350M195975721134156329610983800913371.916161298950++++++++++++++++1075642++++++++++++++++
mbx-bacula.MBX.localLatency225ms35972us78343us37889us39605us1403msLatency136ms10596us566us146ms562us262us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5400M198985726134155599604983821913330.315161162745++++++++++++++++1037442++++++++++++++++
mbx-bacula.MBX.localLatency42387us42425us79064us24617us48069us1432msLatency155ms379us267us143ms7149us571us
mbx-bacula.MBX.local5400M193975760034153759613983837113326.315161300950++++++++++++++++1083343++++++++++++++++
mbx-bacula.MBX.localLatency211ms39602us84547us31763us52587us1434msLatency115ms564us224us142ms5903us247us
mbx-bacula.MBX.local5400M194975750134155419594983746513328.715161187446++++++++++++++++1024841++++++++++++++++
mbx-bacula.MBX.localLatency224ms39136us86727us22977us57265us1505msLatency119ms16296us571us147ms617us25261us
mbx-bacula.MBX.local5400M194975755634153589616983728513329.815161205947++++++++++++++++1130947++++++++++++++++
mbx-bacula.MBX.localLatency212ms29678us95446us22915us42165us1456msLatency195ms315us3632us149ms6227us685us
mbx-bacula.MBX.local5400M195975751534153449579983749013328.815161137644++++++++++++++++1086845++++++++++++++++
mbx-bacula.MBX.localLatency214ms31850us193ms29431us44910us1428msLatency149ms314us564us142ms20731us262us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5450M196975714234155479624983825413305.314161246348++++++++++++++++1075843++++++++++++++++
mbx-bacula.MBX.localLatency194ms41992us182ms24774us47991us1471msLatency138ms1704us571us147ms41594us241us
mbx-bacula.MBX.local5450M195975671033154699594983785113303.314161280350++++++++++++++++916238++++++++++++++++
mbx-bacula.MBX.localLatency197ms44183us77375us31778us45379us1440msLatency127ms32150us224us537ms20737us575us
mbx-bacula.MBX.local5450M197975744234154068599983806013306.614161212047++++++++++++++++1088543++++++++++++++++
mbx-bacula.MBX.localLatency201ms39241us96570us29227us51647us1457msLatency188ms559us234us143ms1671us241us
mbx-bacula.MBX.local5450M195975734234155409620983809813303.814161222948++++++++++++++++1086043++++++++++++++++
mbx-bacula.MBX.localLatency224ms39268us65847us37325us42123us1466msLatency142ms10802us256us143ms6803us231us
mbx-bacula.MBX.local5450M193975731034155929623983800613304.214161187946++++++++++++++++1114043++++++++++++++++
mbx-bacula.MBX.localLatency225ms43284us112ms25743us48983us1445msLatency175ms314us3607us144ms9243us23913us
ConcurrencySizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
mbx-bacula.MBX.local5500M196975645534156159620983825613288.914161215647++++++++++++++++1143247++++++++++++++++
mbx-bacula.MBX.localLatency211ms44718us74548us35281us49885us89259usLatency148ms293us3769us136ms563us23521us
mbx-bacula.MBX.local5500M197975742834154568602983706613289.014161238148++++++++++++++++1070544++++++++++++++++
mbx-bacula.MBX.localLatency213ms46602us91835us27029us50367us89224usLatency115ms7940us563us149ms21136us231us
mbx-bacula.MBX.local5500M195975759234155329622973871613285.714161203947++++++++++++++++1053742++++++++++++++++
mbx-bacula.MBX.localLatency198ms43689us55853us58284us53238us96859usLatency146ms1694us576us144ms581us254us
mbx-bacula.MBX.local5500M196975738034156979615983786313287.814161221547++++++++++++++++1099544++++++++++++++++
mbx-bacula.MBX.localLatency223ms36352us175ms23667us52544us117msLatency145ms13736us232us149ms21686us573us
mbx-bacula.MBX.local5500M196975752234159729609973873813289.614161244148++++++++++++++++1072044++++++++++++++++
mbx-bacula.MBX.localLatency199ms40098us233ms57444us55003us118msLatency146ms31941us566us141ms20071us229us
-------------- next part -------------- A non-text attachment was scrubbed... Name: BACULA Type: application/octet-stream Size: 6297 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20081230/e794366d/BACULA-0001.obj From pheriko.support at gmail.com Tue Dec 30 07:12:29 2008 From: pheriko.support at gmail.com (Periko Support) Date: Tue Dec 30 07:12:36 2008 Subject: bonnie++ result, can someone tech me how to understand this data and help me tunning my system a little more? In-Reply-To: <5b1d935a0812291808g6e8b7540sdd42a6357aa6835c@mail.gmail.com> References: <5b1d935a0812291750p46b4da74s92f756c646ff4601@mail.gmail.com> <5b1d935a0812291808g6e8b7540sdd42a6357aa6835c@mail.gmail.com> Message-ID: <5b1d935a0812292312r2c1dc418v292cee112ba59068@mail.gmail.com> Other thing is that we are using myisam in mysql. Just to correct one thing, "I'm not the programmer". Thanks!!! On Mon, Dec 29, 2008 at 6:08 PM, Periko Support wrote: > Hi people. > > I have found a way to show u my results, u can see in the title of the > page what flags I give to bonnie++, I test inside /var because is where > mysql run. > > The page is *Bonnie Result s > > Thanks again for your time!!! > > * > > On Mon, Dec 29, 2008 at 5:50 PM, Periko Support > wrote: > >> Hi people. >> >> I want to learn how to get more from my system, I will run a new >> application with mysql 5.1.30, this app is develop en .NET(I'm the >> programmer) the clients will be all windows xp boxes, with ps/2 scanner to >> add, get, update the data to mysql, querys etc,etc, them reading about >> performance, for a database is important the file system, I start googling >> around and found tha exist a lot of free app to benchmark the hard disk, one >> of the is bonnie++, I have been running several times in my system, now I >> have the results, I would like to know what is the meaning and what >> parameters can I change to see if I can get better results based on your >> point of view. >> >> All my programs are installed from ports. >> >> I have the results in one file called bonnie-res.html, I atach the file >> here, I don't know if exist a place where I can put this file to be public? >> Plus my kernel file called BACULA. >> >> I was thinking in pastebin, but right now I cannot run bonnie and send >> the output to cvs, because my system is running the backup. >> >> Technical details: >> OS: 7.0-RELEASE-p6 i386 (Custom Kernel) >> 512RAM >> >> Motherboard: >> Foxconn KM400 Duron 1.6Ghz >> >> Raid-1 with geom_mirror >> >> disk 1: ad0: setting PIO4 on 8235 chip >> ad0: setting UDMA100 on 8235 chip >> ad0: 286168MB at ata0-master UDMA100 >> ad0: 586072368 sectors [581421C/16H/63S] 16 sectors/interrupt 1 depth >> queue >> >> Disk 2: ad2: setting PIO4 on 8235 chip >> ad2: setting UDMA100 on 8235 chip >> ad2: 305245MB at ata1-master UDMA100 >> ad2: 625142448 sectors [620181C/16H/63S] 16 sectors/interrupt 1 depth >> queue >> >> Both disk on different channels. >> >> This is my make.conf file: >> >> CPUTYPE?=athlon >> CFLAGS= -O2 -fno-strict-aliasing -pipe >> CXXFLAGS+= -fconserve-space >> MAKE_SHELL?=sh >> BDECFLAGS= -W -Wall -ansi -pedantic -Wbad-function-cast -Wcast-align >> \ >> -Wcast-qual -Wchar-subscripts -Winline \ >> -Wmissing-prototypes -Wnested-externs -Wpointer-arith \ >> -Wredundant-decls -Wshadow -Wstrict-prototypes >> -Wwrite-strings >> #COPTFLAGS= -O -pipe >> ENABLE_SUID_SSH= >> PPP_NO_NAT= # do not build with NAT support (see make.conf(5)) >> PPP_NO_NETGRAPH= # do not build with Netgraph support >> PPP_NO_RADIUS= # do not build with RADIUS support >> PPP_NO_SUID= # build with normal permissions >> TRACEROUTE_NO_IPSEC= # do not build traceroute(8) with IPSEC support >> BOOTWAIT=0 >> DOC_LANG= en_US.ISO8859-1 >> FORCE_PKG_REGISTER=yes >> # added by use.perl 2008-12-14 10:36:46 >> PERL_VER=5.8.8 >> PERL_VERSION=5.8.8 >> >> >> My src.conf >> >> WITHOUT_ATM="YES" >> WITHOUT_AUTHPF="YES" >> WITHOUT_BIND="YES" >> WITHOUT_BLUETOOTH="YES" >> WITHOUT_GAMES="YES" >> WITHOUT_I4B="YES" >> WITH_IDEA="NO" >> WITHOUT_HTML="YES" >> WITHOUT_INET6="YES" >> WITHOUT_IPFILTER="YES" >> WITHOUT_IPX="YES" >> WITHOUT_LPR="YES" >> WITHOUT_NCP="YES" >> WITHOUT_PF="YES" >> WITHOUT_RCMDS="YES" >> >> Mount info: >> >> /dev/mirror/gm0s1a on / (ufs, local, soft-updates) >> devfs on /dev (devfs, local) >> /dev/mirror/gm0s1g on /backups (ufs, local, noatime, soft-updates) >> /dev/mirror/gm0s1f on /tmp (ufs, local, noatime, soft-updates) >> /dev/mirror/gm0s1d on /usr (ufs, local, noatime, soft-updates) >> /dev/mirror/gm0s1e on /var (ufs, local, noatime, soft-updates) >> /dev/ad3s1d on /spool (ufs, local, noatime, soft-updates) >> >> I want to first get if is possible a little more from freebsd latter I >> will continue with mysql. >> >> Thanks all for your time, I normally read this maillist, but now I want to >> get a little more from my system. >> >> I will run iozone + sysbench and let u know my results to, if u need more >> info from my system let me know. >> >> Thanks again and happy new year!!! >> >> > From fjwcash at gmail.com Tue Dec 30 17:09:15 2008 From: fjwcash at gmail.com (Freddie Cash) Date: Tue Dec 30 17:09:22 2008 Subject: bonnie++ result, can someone tech me how to understand this data and help me tunning my system a little more? In-Reply-To: <5b1d935a0812291750p46b4da74s92f756c646ff4601@mail.gmail.com> References: <5b1d935a0812291750p46b4da74s92f756c646ff4601@mail.gmail.com> Message-ID: <200812300850.37429.fjwcash@gmail.com> On December 29, 2008 5:50 pm Periko Support wrote: > I want to learn how to get more from my system, I will run a new > application with mysql 5.1.30, this app is develop en .NET(I'm the Are there specific features in MySQL 5.1.x that you need to use, that are not available in MySQL 5.0.x? If not, you should look at moving back to 5.0.x. Kris Kenneway did several different benchmarks of various versions of MySQL on FreeBSD 6 and 7 and found MySQL 5.0.x to be significantly faster than 5.1.x, especially on 8+ core systems. http://people.freebsd.org/~kris/scaling/mysql.html http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf http://people.freebsd.org/~kris/scaling/Filesystem%20Performance.pdf You should also consider moving to the InnoDB backend instead of MyISAM. The days when MyISAM is faster then InnoDB for all but the simplest of queries is long gone. -- Freddie fjwcash@gmail.com From pheriko.support at gmail.com Tue Dec 30 23:46:15 2008 From: pheriko.support at gmail.com (Periko Support) Date: Tue Dec 30 23:46:21 2008 Subject: bonnie++ result, can someone tech me how to understand this data and help me tunning my system a little more? In-Reply-To: <200812300850.37429.fjwcash@gmail.com> References: <5b1d935a0812291750p46b4da74s92f756c646ff4601@mail.gmail.com> <200812300850.37429.fjwcash@gmail.com> Message-ID: <5b1d935a0812301546u34392c13nd3675349a6e63bca@mail.gmail.com> I'm not the developer, I don't know how to answer u the question. But I will test both version using mysql 5.0/5.1 with InnoDB and see who give better performance. Now, about my bonnie++ results, some one what is the meaning, thanks!!! On Tue, Dec 30, 2008 at 8:50 AM, Freddie Cash wrote: > On December 29, 2008 5:50 pm Periko Support wrote: > > I want to learn how to get more from my system, I will run a new > > application with mysql 5.1.30, this app is develop en .NET(I'm the > > Are there specific features in MySQL 5.1.x that you need to use, that are > not available in MySQL 5.0.x? If not, you should look at moving back to > 5.0.x. Kris Kenneway did several different benchmarks of various versions > of MySQL on FreeBSD 6 and 7 and found MySQL 5.0.x to be significantly > faster > than 5.1.x, especially on 8+ core systems. > > http://people.freebsd.org/~kris/scaling/mysql.html > http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf > http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf > http://people.freebsd.org/~kris/scaling/Filesystem%20Performance.pdf > > You should also consider moving to the InnoDB backend instead of MyISAM. > The days when MyISAM is faster then InnoDB for all but the simplest of > queries is long gone. > > -- > Freddie > fjwcash@gmail.com > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" >